cupertino_http: Failure calling Dart_PostCObject_DL

[brianquinlan]

From @lavinov-mercury:

Hi! Unfortunately, we are facing crash using cupertino_http. I couldn't find specific steps to reproduce it, for us it just happens sometimes after returning from the background or using Hot Restart. I tried to integrate it in 2 ways: with runWithClient (literally copied snippet from the README), and by replacing client in-place:

final _client = SentryHttpClient(
      client: Platform.isIOS
          ? CupertinoClient.defaultSessionConfiguration()
          : http.Client(),
      );

In both cases after some time playing around, i could face the following crash:

*** Assertion failure in -[CUPHTTPClientDelegate URLSession:dataTask:didReceiveResponse:completionHandler:], CUPHTTPClientDelegate.m:120
*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Dart_PostCObject_DL failed.'
*** First throw call stack:
(0x181a1405c 0x199f2ef54 0x1832d599c 0x10532e10c 0x18220cc70 0x181684c04 0x181686950 0x18168e0ac 0x18168ec44 0x181699318 0x1f1e511b0 0x1f1e50f50)
libc++abi: terminating with uncaught exception of type NSException
* thread #65, queue = 'com.apple.NSURLSession-delegate', stop reason = signal SIGABRT
    frame #0: 0x00000001b84659c4 libsystem_kernel.dylib`__pthread_kill + 8
libsystem_kernel.dylib`__pthread_kill:
->  0x1b84659c4 <+8>:  b.lo   0x1b84659e4               ; <+40>
    0x1b84659c8 <+12>: pacibsp
    0x1b84659cc <+16>: stp    x29, x30, [sp, #-0x10]!
    0x1b84659d0 <+20>: mov    x29, sp
Target 0: (Runner) stopped.

The failed line is:
https://github.com/dart-lang/http/blob/master/pkgs/cupertino_http/src/CUPHTTPClientDelegate.m#L120

[brianquinlan]

unregisterTask doesn't seem to be called anywhere.

[brianquinlan]

This is a plausible explanation: https://developer.apple.com/forums/thread/47252

[brianquinlan]

This was fixed in 0.0.6. Could you please try it out?

[delfme]

I was just hit by it. It never occurred since today and it occurs on Hot Restart, almost at every app launch.

Log

*** Assertion failure in -[CUPHTTPClientDelegate URLSession:dataTask:didReceiveResponse:completionHandler:], CUPHTTPClientDelegate.m:116
*** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Dart_PostCObject_DL failed.'
*** First throw call stack:
(0x1d5d7ce38 0x1cef138d8 0x1d062b8ec 0x105a780c4 0x1d6f774c8 0x1d018d060 0x1d01611b0 0x1d0161140 0x1d01224f4 0x1d0122228 0x1d0127940 0x1dd3a4114 0x1dd394f88 0x1dd398418 0x1dd397a80 0x1dd3a6a6c 0x1dd3a7284 0x223ff4dbc 0x223ff4b98)
libc++abi: terminating with uncaught exception of type NSException

Flutter doctor, testing on iphone 13 pro ios 16.3.1

Doctor summary (to see all details, run flutter doctor -v):
[!] Flutter (Channel [user-branch], 3.13.0-0.2.pre, on macOS 12.6.5 21G531 darwin-arm64, locale en-IT)
    ! Flutter version 3.13.0-0.2.pre on channel [user-branch] at /Users/delf/Development/flutter
      Currently on an unknown channel. Run `flutter channel` to switch to an official channel.
      If that doesn't fix the issue, reinstall Flutter by following instructions at https://flutter.dev/docs/get-started/install.
    ! Upstream repository unknown source is not a standard remote.
      Set environment variable "FLUTTER_GIT_URL" to unknown source to dismiss this error.
[!] Android toolchain - develop for Android devices (Android SDK version 31.0.0)
    ✗ cmdline-tools component is missing
      Run `path/to/sdkmanager --install "cmdline-tools;latest"`
      See https://developer.android.com/studio/command-line for more details.
    ✗ Android license status unknown.
      Run `flutter doctor --android-licenses` to accept the SDK licenses.
      See https://flutter.dev/docs/get-started/install/macos#android-setup for more details.
[✓] Xcode - develop for iOS and macOS (Xcode 14.0.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.2)
[✓] Connected device (3 available)
[✓] Network resources

[delfme]

Updated from 1.0.0 to 1.1.0 and issue still occurs.

[MateusHBR]

Why can't we have concurrent operations regarding network calls? Is that a limitation of the Dart threading model? I am building a client with Cupertino HTTP, trying to improve my upload speed by making multiple concurrent calls, but this is limited due to the lack of this configuration.

[brianquinlan]

Why can't we have concurrent operations regarding network calls? Is that a limitation of the Dart threading model? I am building a client with Cupertino HTTP, trying to improve my upload speed by making multiple concurrent calls, but this is limited due to the lack of this configuration.

Hi @MateusHBR

Could you explain in some more detail what you mean?

[MateusHBR]

Hey @brianquinlan

I'm just trying to understand why we can't customize the NSOperationQueue by adding a custom maxConcurrentOperationCount. Is that the case, as mentioned here https://github.com/dart-lang/language/tree/main/working/333%20-%20shared%20memory%20multithreading, that the native code communicates through an arbitrary thread and Dart doesn't know how to handle it?

My scenario is that I'm trying to promote concurrency over NSUrlSession Calls, and this setting that can't be updated prevents me from doing that.

Implemented on this PR: #1033

[brianquinlan]

@MateusHBR

maxConcurrentOperationCount controls the number of callbacks that can be in flight at once, not the number of concurrent HTTP requests.

We set maxConcurrentOperationCount to 1 to prevent race conditions in callbacks e.g. the response being complete callback arriving before data callbacks.

Could you explain in detail what actual issue you are seeing? Maybe with some example code that doesn't behave as you'd want it to?

[MateusHBR]

Hey @brianquinlan

During my reading of the thread: https://developer.apple.com/forums/thread/47252

I noticed that it could be customized the networkOperationQueue, which by default, NSOperationQueue will return a concurrent queue, that is, a queue on which multiple operations can run concurrently. However If I want the operations to be serialised, which is the only sane way to deal with NSURLSession callbacks, set the queue’s maxConcurrentOperationCount to 1.

That reduces the throughput of my HTTP calls because they can be called from different threads, so I was trying to customize this setting together with my current settings that I already have defined:

  late final http.BaseClient _client = () {
    final config = URLSessionConfiguration.defaultSessionConfiguration();
    config.httpMaximumConnectionsPerHost = 8;
    config.allowsExpensiveNetworkAccess = true;
    config.allowsCellularAccess = true;
    config.networkServiceType =
        NSURLRequestNetworkServiceType.NSURLNetworkServiceTypeResponsiveData;
    config.waitsForConnectivity = true;
    config.multipathServiceType = NSURLSessionMultipathServiceType
        .NSURLSessionMultipathServiceTypeHandover;
    config.requestCachePolicy =
        NSURLRequestCachePolicy.NSURLRequestReloadIgnoringLocalCacheData;

    // It would be better to use a custom queue, but it's not possible to create
    // a custom queue in the current version of the cupertino_http package.
    // https://developer.apple.com/forums/thread/47252
    // final queue = NSOperationQueue()
    //   ..maxConcurrentOperationCount = 4
    //   ..name = 'cupertino_http.NSURLSessionDelegateQueue'.toNSString();

    final nativeClient = CupertinoClient.fromSessionConfiguration(config);

    final client = retry.RetryClient(
      nativeClient,
      when: (resp) => resp.statusCode >= 500,
      whenError: (error, _) {
        final commonExceptions = [
          'SocketException',
          'HttpException',
          'HandshakeException',
          'TimeoutException',
        ];

        final err = error.runtimeType.toString();
        return commonExceptions.any(err.contains);
      },
      delay: (c) => Duration(seconds: c * 2),
    );

    return SentryHttpClient(client: client);
  }();

[brianquinlan]

Hey @MateusHBR

I think that I'm still not understand you. The reason that cupertino_http doesn't work with a larger maxConcurrentOperationCount is because it allows delegate callbacks to be executed out-of-order. For example:

If we have a NSTask that queues these delegate calls:
...didReceiveData:
...didCompleteWithError:

If those are executed concurrently and cupertino_http receives the ...didCompleteWithError: callback first then it will close the Stream that communicates the received data into Dart before that ...didReceiveData: callback was processed.