.NET Core 3.1 GC problem (.NET Core not performing GC when low on memory)

[JMan7777]

Hello,

I have a bit of a tricky problem.

I have a Linux web app running on Azure (B4 instance) under .NET Core 3.1.11 which does also perform big batch processing jobs.
So I have a lot of large objects and basically iterating through the objects during the processing.

I can see that the memory usage keeps going up during that processing (and down when an automatic GC kicks in)
but finally comes to a point where the web app restarts as it was exceeding its memory limits.

Now the funny thing is, that when I place a manual GC in the processing loop (let's say perform a GC every 10th large object processing), the memory usage tops out stable around 80% and not even comes close to 100%.

I do the manual GC as follows:

GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;
GC.Collect(2, GCCollectionMode.Forced);

and my GC settings are:

    <TieredCompilation>true</TieredCompilation>
    <TieredCompilationQuickJit>false</TieredCompilationQuickJit>
    <TieredCompilationQuickJitForLoops>false</TieredCompilationQuickJitForLoops>
    <RetainVMGarbageCollection>true</RetainVMGarbageCollection>
    <ServerGarbageCollection>true</ServerGarbageCollection>
    <ConcurrentGarbageCollection>true</ConcurrentGarbageCollection>

Any idea how I can fine-tune the GC settings that I can get rid of the manual GC's.

Thx a lot.

[huoyaoyuan]

It sounds like caused by LOH fragmentation. I'm not sure how the runtime is configured for that, but manually require compaction of LOH should be the right way.

[huoyaoyuan]

Continuously setting LargeObjectHeapCompactionMode without force GC should be OK. By documentation, it means that LOH will be required to be compacted during next GC.

[JMan7777]

Continuously setting LargeObjectHeapCompactionMode without force GC should be OK. By documentation, it means that LOH will be required to be compacted during next GC.

Thx a lot.

I gave it a try but it didn't work so well.

Test settings:

Test A:

1. One time setting at startup:~~

GCSettings.LatencyMode = GCLatencyMode.SustainedLowLatency;

This is to reduce full blocking Gen2 GC's and only run them if really needed.
(Enables garbage collection that tries to minimize latency over an extended period. The collector tries to perform only generation 0, generation 1, and concurrent generation 2 collections. Full blocking collections may still occur if the system is under memory pressure.)

2. Every second as a background task:

GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;

This is to ensure full blocking Gen2 GC's are also cleaning up the large objects.
(The large object heap (LOH) will be compacted during the next blocking generation 2 garbage collection.)

This did not work well. Even under memory pressure full blocking Gen2 GC's did not always happen in time and the app exceeded the memory limits.

Test B:

1. One time setting at startup:~~

GCSettings.LatencyMode = GCLatencyMode.Interactive;

This is the default value by .NET core 3.1 anyway.

2. Every second as a background task:

GCSettings.LargeObjectHeapCompactionMode = GCLargeObjectHeapCompactionMode.CompactOnce;

This is to ensure full blocking Gen2 GC's are also cleaning up the large objects.
(The large object heap (LOH) will be compacted during the next blocking generation 2 garbage collection.)

This did not work well. Even under memory pressure full blocking Gen2 GC's did not always happen in time and the app exceeded the memory limits.

Any other ideas what I can try to get rid of the manual GC's?

[jkotas]

You can try setting HeapHardLimit or HeapHardLimitPercent: https://docs.microsoft.com/en-us/dotnet/core/run-time-config/garbage-collector#heap-limit-percent. It will tell GC to stay under the given limit.

Also, you can try turning off server GC by setting <ServerGarbageCollection>false</ServerGarbageCollection>. The server GC has better throughput, but it also needs more slack space to do a good job.

[JMan7777]

what does COMPlus_GCHeapFreeHardLimit mean?

When currently .NET core processes / app instances are starting they somehow getting the total available memory:

System.GC.HeapHardLimitPercent / COMPlus_GCHeapHardLimitPercent = 80% in a 10GB system will set the COMPlus_GCHeapHardLimit to 8GB.

When running on a non-containerized environment .NET core should be also able to ask the OS about the total free memory.
When running on a containerized environment these free memory metrics should be available in the cgroup filesystem inside each container.
According to: https://docs.docker.com/config/containers/runmetrics

So when .NET core is able the get the free memory then COMPlus_GCHeapFreeHardLimit can be used to check if an app instance is still having enough free memory available or is running in a OOM condition based on the COMPlus_GCHeapFreeHardLimit settings .

Regarding:

(most likely without given any OOM warning inside the app like in my case)

When I wasn't specifying the COMPlus_GCHeapHardLimitPercent value, my .NET core app instance never showed any OOM error. Azure just killed the app with the Azure error "Exceeding memory limits". However when I specified COMPlus_GCHeapHardLimitPercent and set it to a low value I saw OOM errors which are normal as I set the memory restrictions too tight.
I also was surprised that that without COMPlus_GCHeapHardLimitPercent settings .NET Core never gives any OOM warnings. It looks it just requests more memory than the Azure instance is subscribed to and Azure just kills in that case the app instance.

[Maoni0]

every process can read how much free space there is but they can't all use it or you would overcommit. if we can get the equivalent of the memory load of the OS for the container it'll make things better but it's not foolproof, just like running multiple processes on the same machine. the difference is just that folks tend to be a lot more mindful when it comes to containers.

[JMan7777]

if we can get the equivalent of the memory load of the OS for the container it'll make things better but it's not foolproof,

It would be at least better than nothing. At the moment there is no good way available to have auto-scaled app instances available in a container environment without wasting memory. Since memory isn't cheap in Azure I was just hoping there could be some way to make better use of the memory.

If you find a way equivalent of the memory load of the OS for the container, even if not foolproof it would be appreciated by many. Maybe introduce then related new GC settings as an experimental feature. ;)

[Maoni0]

as I mentioned above, we'll definitely take advantage of it if there's a way (I'm chatting with @janvorli about it) :) thanks for bringing this up!

[JMan7777]

You are welcome. Now my problem becomes your problem ;). hahaha