Memory leak on RetrieveMultipleAsync

[lukeswanton]

In the last 3 releases:

• 1.1.22
• 1.1.27
• 1.1.32

The memory leak as described in Issue #207 has returned.

Version 1.1.17 however does not have this issue.

As per #207 this new leak is also only an issue when using RetrieveMultipleAsync, using RetrieveMultiple works as expected

As per [https://github.com//issues/207#issuecomment-983838048](This reply on 207 )
I tested the following and still saw memory consumption continue to rise and never be released:

• Smaller Batch sizes
• Pulling from narrow tables, (The example below is using ActivityParty which is only 33 columns wide)

Here is some sample code from a .NET 8.0 Console App which reproduces the issue:

using Microsoft.PowerPlatform.Dataverse.Client;
using Microsoft.Xrm.Sdk;
using Microsoft.Xrm.Sdk.Query;

internal class Program
{
    private static async Task Main(string[] args)
    {
        string InstanceURL = "<Dynamics URL Here>";
        string ClientId = "<ClientId Here>";
        string ClientSecret = "<ClientSecret Here>";

        ServiceClient.MaxConnectionTimeout = TimeSpan.FromMinutes(60);
        ServiceClient client = new ServiceClient($"AuthType=ClientSecret; Url={InstanceURL}; ClientId={ClientId}; ClientSecret={ClientSecret}")
        {
            EnableAffinityCookie = false,
        };

        QueryExpression query = new()
        {
            PageInfo = new PagingInfo
            {
                PagingCookie = "",
                PageNumber = 1,
                Count = 5000,
            },
            EntityName = "activityparty",
            Distinct = false,
            NoLock = true,
            ColumnSet = new ColumnSet(true),
        };

        bool MoreRecords = false;
        long TotalCount = 0;
        do
        {
            EntityCollection QueryResult = await client.RetrieveMultipleAsync(query);
            TotalCount += QueryResult?.Entities?.Count ?? 0;
            MoreRecords = QueryResult?.MoreRecords ?? false;
            if (MoreRecords)
            {
                query.PageInfo.PagingCookie = QueryResult?.PagingCookie ?? string.Empty;
                query.PageInfo.PageNumber++;
            }
        }
        while (MoreRecords);

        Console.WriteLine($"Downloaded {TotalCount:N0} Records");
    }
}

[MattB-msft]

thanks for reporting, have you configured MaxBufferPool for the client? or are you using defaults?

[MattB-msft]

@lukeswanton Also note: We change the spelling of the config setting from MaxBufferPoolSizeOveride to MaxBufferPoolSizeOverride

(we were missing a "r" in Override)

[lukeswanton]

@MattB-msft everything is using the default settings except where updated as per the example code.

[MattB-msft]

The fix we did for #207 was to open up the access to the MaxBufferPoolSize property to allow as user to tune the buffer sizes to prevent fragmentation in memory and to allow .net GC to handle that better,

Out of curiosity, In your use case, is the "effort" column populated in the activityparty?
And given your paging there how many records are you working with?

[lukeswanton]

@MattB-msft the example provided is not the actual usecase as that is against a custom table with 52 columns and circa 10million rows.

But for reproduction purposes I recreated the issue against an OOB table that has very few columns (no. columns considered due to your reply on #207) and activityparty was a good fit as it is also typically on the larger side.

My concern here is not whether messing with settings can resolve it, but rather the default OOB settings result in a memory leak in the last 3 releases but do not in releases prior.

For my actual usecase I have simply swapped to using the older package version in NuGet and all is working as expected with no memory usage leak.

So it looks like what ever changes where implemented in the 1.1.22 release have caused the recurrence of this leak.

[MattB-msft]

Ok, that help clarify what you're seeing @lukeswanton,
We will have a look now that we understand your seeing it as a general regression and not use case specific.

as for what changed:
in .22, the underlying ServiceModule libs were bumped due to security issues in lower-level transient dependencies.
in .32, the underlying ServiceModule subsystem was forked for .net core and .net framework due to .net core choices that broke backward compatibility.

[iturner100]

I've just started using the Dataverse SDK in a .net8 project, and have run into this same issue with the 1.1.32 version of the package when calling RetrieveMultipleAsync

The application is running in Azure Container Apps and would otherwise comfortably run with low memory constraints (1Gi), but it's throwing InsufficientMemoryException errors when paging through 100k or so Opportunity entities with only about a dozen columns being returned.

Has there been any progress with the issue so far?

Is the recommended workaround to just use the synchronous RetrieveMultiple for now?

[lukeswanton]

@iturner100 , For my use-case I simply swapped to using version 1.1.17, as this is the latest release not effected by this memory leak.
If you are required to use the latest version then yes the only simple work around is to use the synchronous version of the method, although as @MattB-msft mentioned you may be able to get the async method working correctly by messing with the MaxBufferPoolSize setting.

[iturner100]

Thanks @lukeswanton. I did experiment with setting MaxBufferPoolSizeOverride but that didn't appear to stop the heap size from ever-increasing, and that's with forcing a GC before fetching the next page of results as well.

This is my first outing with the Dynamics SDK so just plumped for the latest version. But have just tested with 1.1.17 and it seems to play nicely with the rest of the codebase, so I'll also switch to that for the time-being.

[MateuszPeczek]

@iturner100 what page size are you using? I have similar issue with Dataverse client on Azure Container Apps, though even after going back to 1.1.17 and using ReceiveMultiple instead of ReceiveMultipleAsync I'm still getting "Failed to allocate a managed memory buffer of 268435456 bytes. The amount of available memory may be low."

[iturner100]

@MateuszPeczek I ended up just using the default page size (5,000) and that seems to be working fine using ReceiveMultipleAsync and 1.1.17.

I am only returning small numbers of columns (< 15) in most cases though, so it could be that that is helping

[meghuizen]

Looks like the update of the System.ServiceModel.Http package
Lib version 1.1.17 uses System.ServiceModel.Http version: 4.10.2 , see commit diff: 1.1.17...1.1.21#diff-c7de0b2afb480873c1624871753b95ecade7f835ca3ca43263ce5ce598ef9cd3

[MattB-msft]

There is a split between .net framework and .net 6+ .
The problem is surfacing on .net 6+

1.1.17 is using the pre-split version of .net's WCF layer. if you look at the dependencies here: https://www.nuget.org/packages/Microsoft.PowerPlatform.Dataverse.Client#dependencies-body-tab you can see it between .net 48 and .net 6

[MattB-msft]

Folks... Good news here.
in cooperation with the .net team we found the root cause of the issue, its a fix inside .net itself so it's going into the fix train for .net now.

We are being told the fix will get into the 8.1.2 Nuget packages for https://www.nuget.org/packages/System.ServiceModel.Http
We will pick that up as a dependency shortly after it drops.