Memory leak on RetrieveMultipleAsync

[meghuizen]

I'm running a project in Azure Functions (v4 on .NET 6). This project interfaces with D365 through this library.
Within the project I'm doing calculations on 2,5 million of "transactions" (entities) and writing those totals and averages back to D365.

For that I'm currently using this library (version 0.5.10) and using RetrieveMultipleAsync to get all the items (batched).
Although what I saw happening that my memory build up, however the garbage collector didn't free up, when the objects where out of scope.
When I changed it to the non-async version, this didn't happen.

The items, on which it leaked, looked like internal "WCF" soap messages (BufferedMessages). This has probably to do with some async task scheduling and keeping the context in memory, though my knowledge on how async works is way too limited to give any usefull insights.

By the way, I was using the version without the cancellation token option.

See the attached screenshots for the results I got from the debugger:

[meghuizen]

Here is some example code which I'm using to test this out.
You might see an awefull lot of memory is being used and not everything is cleared, even after GC.Collect();

public static async IAsyncEnumerable<Entity> RetrieveAllAsyncEnumerable(ServiceClient svcClient, QueryExpression query, int pageSize = 5000)
{
    bool moreRecords = false;

    if (!query.TopCount.HasValue)
    {
        query.PageInfo = new PagingInfo
        {
            Count = pageSize,
            PageNumber = 1,
            PagingCookie = null
        };
    }

    do
    {
        EntityCollection result = await svcClient.RetrieveMultipleAsync(query);

        if (result?.Entities == null || !result.Entities.Any())
        {
            yield break;
        }

        foreach (var entity in result.Entities)
        {
            yield return entity;
        }

        moreRecords = result.MoreRecords;

        if (moreRecords && !query.TopCount.HasValue)
        {
            query.PageInfo.PageNumber++;
            query.PageInfo.PagingCookie = result.PagingCookie;
        }
    }
    while (moreRecords);
}

// The executing test function
public static async System.Threading.Tasks.Task TestMultipleAsync(string connectionString)
{
    var crmSvcClient = new Microsoft.PowerPlatform.Dataverse.Client.ServiceClient(connectionString);

    Console.WriteLine("Start fetching all contacts in D365");

    var entityList = RetrieveAllAsyncEnumerable(crmSvcClient, new QueryExpression("contact")
    {
        ColumnSet = new ColumnSet(true),
        TopCount = 5000
    });

    int i = 0;
    await foreach (var e in entityList)
    {
        i++;
    }

    Console.WriteLine($"Fetched {i} contacts from the database");

    entityList = null;
    GC.Collect();

    // do this again
    entityList = RetrieveAllAsyncEnumerable(crmSvcClient, new QueryExpression("contact")
    {
        ColumnSet = new ColumnSet(true),
        TopCount = 5000
    });

    i = 0;
    await foreach (var e in entityList)
    {
        i++;
    }

    Console.WriteLine($"Fetched {i} contacts from the database");

    entityList = null;
    GC.Collect();

    // do this again
    entityList = RetrieveAllAsyncEnumerable(crmSvcClient, new QueryExpression("contact")
    {
        ColumnSet = new ColumnSet(true),
        TopCount = 5000
    });

    i = 0;
    await foreach (var e in entityList)
    {
        i++;
    }

    Console.WriteLine($"Fetched {i} contacts from the database");

    entityList = null;
    GC.Collect();
}

[MattB-msft]

Thanks @meghuizen, our team is taking a look at it to see what we can determine.
We have not yet fully optimized this code, as we have done with CrmServiceClient.

Looking at your code there are a few obvious things there..
Given that you're asking for the full contact record (which is a very wide table) and a row count of 5k, your requests are very likely exceeding 512K for each response. That will trigger the WCF stack to use the heap to store them which has the byproduct of creating memory fragmentation which surfaces as memory leaks. You're not seeing with the sync method because its able to process the end-to-end request in one action vs having to juggle multiple connections concurrently

You could adjust the behavior right now in 2 ways..
First, do not use new ColumnSet(true), provide the query a specific list of columns like 1k, or 500.
Second, reduce the return size from 5K to something

if you do the first, you can play with the second to see the level you can get to that will not trip the memory leak.

There are a few things we can do inside the client to optimize the way WCF treats this, however it boils down to a tradeoff between perf and footprint.

Please let us know if the suggestions ease this for you, and we will take a look at trying to optimize this in the shorter term.

[meghuizen]

Thanks @MattB-msft for your comment. Well, the ColumnSet(true) was just to give you some example of a query I'm using, but in reallity I'm using a different query, with linked entities and specific column set.

For me the use case is, doing calculations on all my "transaction" records, with aggregates, averages and calculations which have a time window (last year, 2 years ago, etc.)

However I've fixed it for now, by not using the Async calls for this, and that helps for now. But even there I see more memory usage than what I would expect.

For my code I'm trying to see if I can change this to a direct Web API call with HttpClient. However that will not solve this issue.

The heap size thing sounds indeed like something I should consider. Maybe experimenting with the page size might be helpfull. 'll try it out.

[MattB-msft]

Please do keep in mind that the Dataverse API is somewhat verbose. It includes formatted values, translations of option sets and such to full names in addition to a lot of other data that is used for forming the entities with correct typing's. Without seeing specifically what your doing and the size of data in question I cannot really comment of what the correct expectation should be for memory consumption.

The WebAPI on the other hand does give you control of content in degree's by using annotations to include the "by default" context data that is include in the organization api. Thus you can get a smaller return payload with the tradeoff of the client needed to have much more pre-knowledge of the context of the operation.

in your case, for what your doing, it may make sense to use the webAPI to get the more concise list, though I would discourage you from trying to fit it back into entities at that point.

[MattB-msft]

@meghuizen, as we are getting closer to the 1.0 we have started to look into this more deeply.
Sorry about the long time responding here.

In ordered to isolate the issue between large block allocation inside WCF vs something in our implementation, we set up a repro similar to what you have described, using a single client + multiple requests + a request that consistently returned a fixed size message.

We observed that that a minor memory creep (not using the term leak here) that is well within the expected buffering range of .net core and .net framework (<> 4megs of flux). This memory was returned after a period of inactivity in the running client.

We modified the system to return a significant amount of data in a single request (+10meg) and we began seeing the buffer creep your describing, this is more in line with a leak type behavior however the same pattern as before occurred where we observed a +- of about 7megs per batch after GC.
We manipulated the MaxBufferPool setting that WCF exposes, and observed an improvement in the overall buffering behavior, the key being that the client surrendered the memory over time.

Now this is not directly noticeable in the system process viewer as .net reports using more memory than its really using on average. We are using .net memory profilers and a few 3rd party memory monitoring tools to verify what we are seeing. As the .net team says "trust the GC" :)

We have included the MaxBufferPool configuration setting in the next drop of the Dataverse ServiceClient to allow you to tune it for your workload.

thanks for your report,
We will close this with the next drop of the client.