[node] Distributed Tracing not working on Express server running Apollo Federation

[serglom21]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Sentry Saas (sentry.io)

Which SDK are you using?

@sentry/node

SDK Version

7.54.0

Framework Version

No response

Link to Sentry event

https://sergios-test-org.sentry.io/performance/graphql-backend:11412188f8714bef8f38ee9c39ebcdf5/?project=4505321195241472&query=http.method%3APOST&referrer=performance-transaction-summary&statsPeriod=14d&transaction=POST+%2Fgraphql&unselectedSeries=p100%28%29

SDK Setup

Sentry.init({
    dsn: "{DSN}",
    integrations: [
      new Sentry.Integrations.Http({ tracing: true }),
      new Sentry.Integrations.Undici(),
      new Sentry.Integrations.Express({ app }),
      new Sentry.Integrations.Apollo(),
      new Sentry.Integrations.GraphQL()
    ],
    tracesSampleRate: 1.0,
})

Steps to Reproduce

1. Clone https://github.com/sergiosentry/apollo-graphql-sentry
2. Follow README to install package dependencies and run services

Expected Result

I expect the trace to be propagated from the gateway service to the accounts service via the /graphql request

These services should then be linked in Sentry with the same trace.

There is a request made to localhost:4001/ that was not traced correctly to that downstream service

Actual Result

The trace is not propagated to the downstream service (accounts/index.js)

It seems that we are not generating spans after the request is initialized and redirected to the GraphQL service.

[AbhiPrasad]

Hey @sergiosentry - I wasn't able to reproduce this. See: https://sentry-test.sentry.io/performance/abhi-node-gateway:9769c9be5c24405f8234f8f126071ccd/?transaction=POST+%2Fgraphql

I'm running the gateway service at http://localhost:4000/graphql and the account service is at http://localhost:4001/graphql

[serglom21]

@AbhiPrasad Based on that event that was linked, it seems that this was tested on node 20.
I tested it on node 16 but based on user reports it is also happening on v18. It just might be the case that as this comment points out, this is fixed on node 20

[danielkhan]

Can we please revisit this.
I see the reproducer by @sergiosentry and if it may be node 16 only, can someone verify this?

[AbhiPrasad]

I verified that distributed tracing was working on Apollo Foundation v3 and for all Node versions.

What is left to verify is Apollo Foundation v4 - but from early testing I don't see any issues.

[danielkhan]

@AbhiPrasad can we look at the data? Can we infer what might be missing at the customer? What are the next steps to fix this?
@sergiosentry if I understand this correctly, you even have a reproducer, correct? Can we try this out and see what's different here?

[serglom21]

@danielkhan we were able to reproduce it last week with the latest changes pushed to the repo. Abhi is looking at the new examples

In order to reproduce it, we added an explicit POST request using undici to the graphQL endpoint. The only difference that I can see between the example that works and the one that does not - Is that this sample is using node 20.0.0 and this sample (which does not work) is using node 18.0.0

[serglom21]

Hi @AbhiPrasad!
Have there been any updates on this investigation/were you able to find out what is going wrong here?

Any additional context/repro data I can provide?

Thanks

[AbhiPrasad]

Hey @sergiosentry - assigning @onurtemizkan to work on this.

[onurtemizkan]

Hi all, I have tried to trace back this problem between Node versions. And it seems the problem exists between versions 18.0.0 - 18.6.0 (inclusive). I can confirm that distributed tracing works fine on 18.7.0 onwards.

I did not check the specific update that fixed it, but I suspect it's nodejs/node@aac97c2.

Reproduction on 18.6.0 - Transaction

Reproduction on 18.7.0 - Transaction

[danielkhan]

Thank you @onurtemizkan. @sergiosentry let me know if this settles it.

[serglom21]

Update here: The customer was able to fix Distributed Tracing by manually appending the sentry-trace and baggage headers to the downstream services requests.

This should be good now! Thanks all!