NullReferenceException when cloning service client

[svejdo1]

Scenario:

1. attempt to get CDS client, when the connection to CDS times out on creation

var foo = new ServiceClient(instanceUrl, tokenProvider);

This sometimes times out when CDS is busy - e.g. following is returned (which is fine).

Microsoft.PowerPlatform.Dataverse.Client.Utils.DataverseConnectionException: Failed to connect to Dataverse ---> System.AggregateException: One or more errors occurred. (The request channel timed out attempting to send after 00:01:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout.) ---> System.TimeoutException: The request channel timed out attempting to send after 00:01:00. Increase the timeout value passed to the call to Request or increase the SendTimeout value on the Binding. The time allotted to this operation may have been a portion of a longer timeout. ---> System.TimeoutException: The HTTP request to 'https://sg867tp1crmhubfshorg-00135-psme.api.crm10.dynamics.com/XRMServices/2011/Organization.svc/web?SDKClientVersion=9.0.46.3291' has exceeded the allotted timeout of 00:01:00. The time allotted to this operation may have been a portion of a longer timeout. at System.ServiceModel.Channels.HttpChannelFactory1.HttpClientRequestChannel.HttpClientChannelAsyncRequest.SendRequestAsync(Message message, TimeoutHelper timeoutHelper)
at System.ServiceModel.Channels.RequestChannel.RequestAsync(Message message, TimeSpan timeout)
--- End of inner exception stack trace ---
at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.InitServiceAsync()
--- End of inner exception stack trace ---
at System.Threading.Tasks.Task1.GetResultCore(Boolean waitCompletionNotification) at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.GetCachedService(ConnectionService& ConnectionObject) at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.IntilizeService(ConnectionService& ConnectionObject) at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection(Object externalOrgServiceProxy, AuthenticationType requestedAuthType, String hostName, String port, String orgName, NetworkCredential credential, String userId, SecureString password, String domain, String Geo, String claimsHomeRealm, Boolean useSsl, Boolean useUniqueInstance, OrganizationDetail orgDetail, String clientId, Uri redirectUri, PromptBehavior promptBehavior, OrganizationWebProxyClient externalOrgWebProxyClient, String certificateThumbPrint, StoreName certificateStoreName, X509Certificate2 certificate, Uri instanceUrl, Boolean isCloned, Boolean useDefaultCreds, Version incomingOrgVersion, ILogger externalLogger) --- End of inner exception stack trace --- at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.CreateServiceConnection(Object externalOrgServiceProxy, AuthenticationType requestedAuthType, String hostName, String port, String orgName, NetworkCredential credential, String userId, SecureString password, String domain, String Geo, String claimsHomeRealm, Boolean useSsl, Boolean useUniqueInstance, OrganizationDetail orgDetail, String clientId, Uri redirectUri, PromptBehavior promptBehavior, OrganizationWebProxyClient externalOrgWebProxyClient, String certificateThumbPrint, StoreName certificateStoreName, X509Certificate2 certificate, Uri instanceUrl, Boolean isCloned, Boolean useDefaultCreds, Version incomingOrgVersion, ILogger externalLogger) at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient..ctor(Uri instanceUrl, Func2 tokenProviderFunction, Boolean useUniqueInstance, ILogger logger)`

2. However when we kick the retry (T+1minute within the same process) we get the client successfully; however the cloning fails on

var foo = new ServiceClient(instanceUrl, tokenProvider); var bar = foo.Clone();

System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.PowerPlatform.Dataverse.Client.ConnectionService.SetClonedProperties(ServiceClient sourceClient) at Microsoft.PowerPlatform.Dataverse.Client.ServiceClient.Clone(Assembly strongTypeAsm, ILogger logger)

Which is not ok

[MattB-msft]

To clarify..
Your Connected, and running fine..
The pattern your using is a root connection, then a clone per thread.
At some point you hit a timeout error / connection crash,
Then you wait, and attempt to create a new connection. this works.
Then you try to create a clone from the new connection == Null Ref.

is that correct?

thanks

[MattB-msft]

@svejdo1
As an update, we added some additional logs to the current drop to try to nail this down.. however in unrelated testing we discovered something that may actually be the root cause.

Out of curiosity are you using the onboard authentication flow or are you managing that yourself?
We have discovered an issue related to MSAL token refresh logic not managing its own connection timeout mechanics properly, and we believe that his may be what your seeing, if your using the onboard auth system. We are working on running this issue down now.

[ThreeSevenths]

@MattB-msft - we saw some similar issues with the tokens sticking in the client. We manage it ourselves using Microsoft.Identity.Web and ITokenAquisition. We ended up tearing down our service clients and thus our entire stack below at half-token expiration in order to get a stable enough implementation. It seems that once you set the HeaderToken property of the OrganizationWebProxyClient, you cannot change it.

[MattB-msft]

@ThreeSevenths, if your using the onboard auth system, it will update the header token each time a request goes out,
we have done some updates in a pending release that will optimize that further to take the burden off the in memory token cache system MSAL has

We do not expose the OrganizationWebProxyClient in the DataverseCleint, and I would suggest you avoid using it if your building new code based on some future directions.

That said, you can change the OrgnaizationWebProxyClient.headertoken property at any time.

Can you outline the process your using that your seeing it be not settable? or share some code?