Google Cloud Function: missing Stackdriver logs when Sentry captures exceptions

[bsiddiqui]

Package + Version

• @sentry/serverless

Version:

6.1.0

Description

Sentry for google cloud functions swallows logs (not sent to stackdriver) if an exception is captured.

We have some code that captures an error with Sentry.captureException which somehow is leading to the logs from that run of the function not appearing in Stackdriver. Based on the docs I assumed Sentry would include "A link to the Stackdriver logs", implying it wouldn't swallow logs.

Example issue:
https://sentry.io/organizations/upandup/issues/2324315838/?project=5632160

[kamilogorek]

Can you provide an example of how you use the SDK? Stackdriver logs links are only provided for the automatically captured exceptions through the wrapHttpFunction wrapper.

[bsiddiqui]

@kamilogorek really basic code

module.exports = Sentry.GCPFunction.wrapEventFunction((data, context, callback) => {
    console.log('Logging some tests')
    console.log('Logging some more tests')
    throw new Error('Testing')
})

Here's a link to an issue in our Sentry project: https://sentry.io/organizations/upandup/issues/2330399156/?project=5632160

We have 2 issues.

1. Major: we have no logs in Stackdriver for the above event - whenever the wrapped function captures our logs it removes them from Stackdriver (it doesn't do this if an error isn't reported). This is very unexpected.

2. Minor: Are Stackdriver logs only linked by wrapHttpFunction, not wrapEventFunction and wrapCloudEventFunction? The docs say "Issue reports automatically include: A link to the Stackdriver logs"

[bsiddiqui]

Ok to make this even more odd...

This is how we might write a non-Sentry wrapped func

module.exports = (data, context, callback) => {
  try {
    // do something
    callback() // no error
  } catch (err) {
    callback(err) // let GCF no there was an error
  } 
}

In the wrapped version of the above function, Sentry doesn't catch the error returned by callback(err)

If I remove the callback and instead in the catch throw err, the error is reported in Sentry but it does not show in Stackdriver and the function never returns so the function crashes from a timeout

[bsiddiqui]

I put this all together in this one snippet to show all the unexpected cases

module.exports = Sentry.GCPFunction.wrapEventFunction(
  async (event, context, callback) => {
    let data, dataString
    try {
      dataString = Buffer.from(event.data, 'base64').toString()
      data = JSON.parse(dataString)
    } catch (err) {
      return callback(new Error(`Error parsing string: ${dataString}`))
    }

    console.log('Logging some tests')
    console.log('Logging some more tests')

    if (data.error) {
      // If this is called an issue will show in Sentry but the function
      // will not exit so it will crash with a timeout (unexpected).
      // When the func crashes the error will show in Stackdriver
      throw new Error('Testing')
    }

    if (data.callbackError) {
      const error = new Error('Testing')
      // If this is called the issue will log in Stackdriver and the
      // function will exit with an error status but it will not show
      // in Sentry (unexpected)
      callback(error)
    }

    if (data.callback) {
      // If this is called the function will log in Stackdriver (expected)
      callback()
    }

    // If returned without the callback the function will crash because
    // of a timeout (unexpected
  }
)

[bsiddiqui]

@kamilogorek @ahmedetefy can we get an update here? Pretty big issue that's been open over a week without a response

[ahmedetefy]

@bsiddiqui Apologies for the late response!
I will look into further within the week and get back to you

[bsiddiqui]

@ahmedetefy any update? it's been 2 weeks since your last message

we're at a place where we have to remove sentry if we can't figure out what's going on and i'd like to avoid that if possible

[bsiddiqui]

@ahmedetefy @kamilogorek it's been over 2 months without an update - is this going to be resolved?

[alexbchr]

We have the exact same issue on our end. We have a GCP function subscribing to a Pubsub topic and crashes in it are correctly sent in Sentry. However, even though the Function crashed, Stackdriver logs do not contain any errors and the faulty executions appears like they ended correctly in StackDriver, which is clearly wrong.

[github-actions]

This issue has gone three weeks without activity. In another week, I will close it.

But! If you comment or otherwise update it, I will reset the clock, and if you label it Status: Backlog or Status: In Progress, I will leave it alone ... forever!

---

"A weed is but an unloved flower." ― Ella Wheeler Wilcox 🥀

[kirbysayshi]

This is absolutely still an issue. I ran into the same problem today.

Sentry swallows errors and does not report anything to stackdriver/cloud logging.

Most importantly: GCP will consider the function successfully completed!

Example stackdriver output from a function that threw (similar to the examples above): "Function execution took 910 ms, finished with status: 'ok'"

[alexbchr]

I just updated the SDK to the latest version (6.17.8) and there is a slight improvement. I still get absolutely no StackDriver logs for the crash, but I see the function terminated with a "crash" status.

However, there are still a couple issues:

• Still no error/crash logs in Stackdriver
• The Sentry exception logged has basically no relevant GCP information stored with it. Before updating, at least I had some details like which Function the crash was originating from, the HTTP Url, and other details. Right now I get absolutely nothing:
  

For info, updated from @sentry/serverless and @sentry/tracing 6.12.0.

[alexbchr]

Sentry team, can we reopen this issue? Those are really important issues, as basic functionalities are straight up not working...

[JonasLaux]

We have the same issue, forcing us to not use Sentry.wrapCloudEventFunction.

and

The diff between those two is passing the cloud handler with Sentry:
wrappedHandler = Sentry.wrapCloudEventFunction(handler);

And without Sentry (just take handler as is)

[mydea]

Could you share your handler code, possibly (at least the critical part of how the handler is set up), and possibly an example of how/where an error is thrown?

[JonasLaux]

Sure, to give you the most important parts (can't public much of the business logic):
Handler:

export async function globalHandler(evt: CloudEvent<{ message: { data: string } }>) {
  const rootLogger = getLogger().child({ evt });
  try {
    const logger = rootLogger.child({ providerId });
    await doSomeThings(logger);
  } catch (err) {
    rootLogger.error('Reconciler error:', err);
    throw err;
  }
}
registerCloudEventHandler(globalHandler);

registerCloudEventHandler:

export function registerCloudEventHandler<T>(handler: CloudEventFunction<T>, options?: RegisterCloudEventHandlerOptions) {
  const hardFailOptions = { ...defaultHardFailOptions, ...options?.hardFail };
  let wrappedHandler: CloudEventFunction<T> = handler;

 Sentry.init({
    ...
  });
  wrappedHandler = Sentry.wrapHttpFunction(withSentryErrorHandling(Sentry, handler));
  functions.cloudEvent(HANDLER, exitProcessOnTimeout(hardFailWhenNeeded(wrappedHandler, hardFailOptions)));
}

withSentryErrorHandling:

export function withSentryErrorHandling<A extends Array<unknown>, R>(Sentry: typeof SentryGoogleCloudServerless, handler: (...args: A) => Promise<R | undefined>): (...args: A) => Promise<R | undefined> {
  return async (...args) => {
    try {
      return await handler(...args);
    } catch (error) {
      Sentry.captureException(error);
      throw error;
    }
  };
}

[mydea]

Hey, so some things about this setup:

1. You write that the problem is from adding Sentry.wrapCloudEventFunction() but your example uses Sentry.wrapHttpFunction(). Which one is it (or both?)
2. I would recommend against calling Sentry.init in this way, you should call this on the top level of your application. E.g.:

Sentry.init({...});

export async function globalHandler(evt: CloudEvent<{ message: { data: string } }>) {
  const rootLogger = getLogger().child({ evt });
  try {
    const logger = rootLogger.child({ providerId });
    await doSomeThings(logger);
  } catch (err) {
    rootLogger.error('Reconciler error:', err);
    throw err;
  }
}
registerCloudEventHandler(globalHandler);

// somewhere else
export function registerCloudEventHandler<T>(handler: CloudEventFunction<T>, options?: RegisterCloudEventHandlerOptions) {
  const hardFailOptions = { ...defaultHardFailOptions, ...options?.hardFail };
  const wrappedHandler = Sentry.wrapHttpFunction(handler);
  functions.cloudEvent(HANDLER, exitProcessOnTimeout(hardFailWhenNeeded(wrappedHandler, hardFailOptions)));
}

Importantly, you should not use/need the custom withSentryErrorHandling code, as this is done by the wrapHttpFunction itself already.