ServerChannel loses telemetry items on errors/timeouts as retry does not handle it correctly

[cijothomas]

There is a default timeout on outbound Http call to Ai backend- if the Send was not successful within this time window, it is aborted.
https://github.com/Microsoft/ApplicationInsights-dotnet/blob/develop/src/Microsoft.ApplicationInsights/Channel/Transmission.cs#L26

However, the retry mechanism does not handle this scenario, and items are dropped. The ErrorHandlingTransmissionPolicy retries only for those failures in which a Response has been received with status codes 500, 503

Proposal is to deploy retry mechanisms on a white listed set of response codes, and when no response is received due to client timeout errror.

Also, on NetCore - no retry policies are implemented,as Retry mechanisms were based on 'WebException' being thrown, but this is not thrown by HttpClient. HttpClient.SendAsync() does not throw Exception, instead populates HttpResponse with correct error codes. (This is different from HttpWebRequest, which threw WebException from which HttpResponse had to be parsed out)

Given that current code is now completely based on HttpClient, all the retry logics/scenarios are to be modified to ensure Telemetry is not lost.

[cijothomas]

Also, on NetCore - no retry policies are implemented,as Retry mechanisms were based on 'WebException' being thrown, but this is not thrown by HttpClient. HttpClient.SendAsync() does not throw Exception, instead populates HttpResponse with correct error codes. (This is different from HttpWebRequest, which threw WebException from which HttpResponse had to be parsed out)

Given that current code is now completely based on HttpClient, all the retry logics/scenarios are to be modified to ensure Telemetry is not lost.