问题描述

部署.NET 应用到App Service中，应用中调用Storage Account的SDK上传文件到Blob中。只是比较高频率在UploadAsync方法中遇见：OutOfMemoryException

image.png

异常信息

ERROR Microsoft.AspNetCore.Server.Kestrel.? [?] - MESSAGE: Connection id "0HN3UB91BK6BS", Request id "0HN3UB91BK6BS:00000585": An unhandled exception was thrown by the application.

** System.OutOfMemoryException: Exception of type 'System.OutOfMemoryException' was thrown.**

at System.GC.AllocateNewArray(IntPtr typeHandle, Int32 length, GC_ALLOC_FLAGS flags)

at System.GC.<AllocateUninitializedArray>g__AllocateNewUninitializedArray|66_0[T](Int32 length, Boolean pinned)

at System.Buffers.TlsOverPerCoreLockedStacksArrayPool`1.Rent(Int32 minimumLength)

at Microsoft.AspNetCore.WebUtilities.HttpResponseStreamWriter..ctor(Stream stream, Encoding encoding, Int32 bufferSize, ArrayPool1 bytePool, ArrayPool1 charPool)

at Microsoft.AspNetCore.Mvc.Infrastructure.MemoryPoolHttpResponseStreamWriterFactory.CreateWriter(Stream stream, Encoding encoding)

at Microsoft.AspNetCore.Mvc.NewtonsoftJson.NewtonsoftJsonResultExecutor.ExecuteAsync(ActionContext context, JsonResult result)

at Microsoft.AspNetCore.Mvc.NewtonsoftJson.NewtonsoftJsonResultExecutor.ExecuteAsync(ActionContext context, JsonResult result)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeNextResultFilterAsync>g__Awaited|30_0[TFilter,TFilterAsync](ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.Rethrow(ResultExecutedContextSealed context)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.InvokeResultFilters()

--- End of stack trace from previous location ---

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeFilterPipelineAsync>g__Awaited|20_0(ResourceInvoker invoker, Task lastTask, State next, Scope scope, Object state, Boolean isCompleted)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)

at Microsoft.AspNetCore.Mvc.Infrastructure.ResourceInvoker.<InvokeAsync>g__Awaited|17_0(ResourceInvoker invoker, Task task, IDisposable scope)

at Microsoft.AspNetCore.Routing.EndpointMiddleware.<Invoke>g__AwaitRequestTask|6_0(Endpoint endpoint, Task requestTask, ILogger logger)

at Microsoft.AspNetCore.Authorization.Policy.AuthorizationMiddlewareResultHandler.HandleAsync(RequestDelegate next, HttpContext context, AuthorizationPolicy policy, PolicyAuthorizationResult authorizeResult)

at Microsoft.AspNetCore.Authorization.AuthorizationMiddleware.Invoke(HttpContext context)

at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware.Invoke(HttpContext context)

at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)

问题解答

从Exception中的Stack Trace上看Out Of Memory发生在GC Allocate New Array, 是在HTTP Process Requests/Response时，需要申请内存遇见内存不足异常。 而异常中信息不充足无法定位。但从App Service的角度，可以考虑修改App Service Platform为64bit (App Service for windows 默认为32bit)。如修改后，依旧出现OutOfMemory异常，则需要考虑收集w3wp.exe 或 dotnet.exe 进程的DUMP文件进行深入分析。

I see the message "Worker Process requested recycle due to 'Percent Memory' limit." How do I address this issue?

The maximum available amount of memory for a 32-bit process (even on a 64-bit operating system) is 2 GB. By default, the worker process is set to 32-bit in App Service (for compatibility with legacy web applications).

Consider switching to 64-bit processes so you can take advantage of the additional memory available in your Web Worker role. This action triggers a web app restart, so schedule accordingly.

Also note that a 64-bit environment requires a Basic or Standard service plan. Free and Shared plans always run in a 32-bit environment.

For more information, see Configure web apps in App Service.

analyzing the memory dump

This article provides guidance about Microsoft Azure App Service debugging features for capturing memory dumps. The capture method that you use is dictated by the scenario in which you capture a memory dump for troubleshooting a performance or availability issue. For example, capturing a memory dump is different for a process that's experiencing excessive memory consumption than for a process that's throwing exceptions or responding slowly. The process in this context is the Internet Information Services (IIS) worker process (W3WP, which runs as w3wp.exe).

参考资料

Capture memory dumps on the Azure App Service platform ： https://learn.microsoft.com/en-us/troubleshoot/azure/app-service/capture-memory-dumps-app-service#next-steps-analyzing-the-memory-dump

Application performance FAQs for Web Apps in Azure : https://learn.microsoft.com/en-us/troubleshoot/azure/app-service/web-apps-performance-faqs

当在复杂的环境中面临问题，格物之道需：浊而静之徐清，安以动之徐生。 云中，恰是如此!