Unable to empty the buffer of events when running @sentry/node on AWS Lambda

[khozzy]

Package + Version

• @sentry/node

Version:

6.12.0

Description

I'm running sentry/node on the AWS Lambda service. The intention is that when a certain file is created on an S3 bucket, its content is parsed and send to Sentry.

The main parts of the code look as follow:

Sentry.init({
  dsn: process.env.SENTRY_DSN,
  environment: process.env.ENVIRONMENT,
  tracesSampleRate: 1.0,
  logLevel: LogLevel.Verbose,
});

export const handler = (event: S3Event) => event.Records.map(handle);

const handle = async (s3Record: S3EventRecord) => {
  // get file from s3 and parse it's content save them to variable events
  let events = []; // assume that there is about 150 events (only 30 of them actually land in Sentry)

  try {
    return Promise.all(events.map(sendEventToSentry));
  } catch (err) {
    console.error(err);
    return err;
  }
}

const sendEventToSentry = async (event: ValidationError) => {
  console.log(event);

  Sentry.captureEvent({
    message: event.message,
    timestamp: Date.parse(event.timestamp) / 1000,
    level: Sentry.Severity.Error,
    platform: event.appId,
    user: {
      ip_address: event.userIp,
    },
    extra: {
      "schema": event.schema,
      "properties": {
        "accountId": event.accountId,
        "profileId": event.profileId,
        "payload": event.properties
      }
    }
  });
  await Sentry.flush(2000);
}

The problem is that every time the lambda is fired it sends a fixed number of 30 events (files contain more events to send).
In the logs, all items are listed. I'm running out of ideas why not all items are being picked up by the Sentry collector.

I tried troubleshooting it like in #1449, but with no success.

[kamilogorek]

It's because our request buffer is configured to send up to 30 events concurrently. After it's full, any overflowing events are dropped. There is currently no way to configure the size of the buffer (we will introduce this in the future soon).

If you want to send all the events, you need to limit the Promise.all to have a concurrency of 30, which should allow for buffer to process events in time.

You can use any of these packages to achieve this:
https://github.com/sindresorhus/p-all
https://github.com/sindresorhus/p-map
https://github.com/sindresorhus/p-limit

[khozzy]

@kamilogorek thanks for the response. I've tried two approaches based on your suggestions but still without any luck successfully ingesting all events.

Line replaced:

return Promise.all(events.map(sendEventToSentry));

Option 1)

import asyncPool from 'tiny-async-pool';

const SENTRY_EVENT_BUFFER = 30
return await asyncPool(SENTRY_EVENT_BUFFER, parse(body), sendEventToSentry)

No difference observed - 30 events ingested.

Option 2)

const SENTRY_EVENT_BUFFER = 30
let validationErrors = parse(body);

while (validationErrors.length) {
  await Promise.all(validationErrors.splice(0, SENTRY_EVENT_BUFFER).map(sendEventToSentry))
}
return await getCurrentHub().getClient().close(SENTRY_FLUSH_TIMEOUT_MS);

This one behaves better. I can observe 1-4 buffer flushes (emitting 30-120 events per lambda run). The performance is however not deterministic.

Do you maybe have any other ideas about what can be adjusted?

[kamilogorek]

@khozzy I was able to make it work using your exact code with asyncPool. There are 2 things that you need to most likely updated.

1. There is a chance that your events are grouped together due to message similarity. Eg. this event has the same message + different int, so it will be grouped together:

(just int)

(string interpolation with int)

If that's the case, you need to force Sentry to "ungroup" them using fingerprints https://docs.sentry.io/platforms/node/usage/sdk-fingerprinting/

In your case, it will be as simple as (example trimmed to make it more readable):

const sendEventToSentry = async event => {
  Sentry.captureEvent({
    message: event.message,
    fingerprint: event.message
  });

  return Sentry.flush(2000); // you can return the flush, as async function will await the return value
};

2. You need to disable spike protection. Going from 0 to 100s of events will trigger security measures and it will drop some of the events:

You can do this in Settings => Subscriptions

Changing this setting takes few minutes to kick in, so wait like ~5 minutes before retrying.

Hope that will fix that.

[AbhiPrasad]

Closing as it seems the issue has been addressed. Please re-open if you have any other questions or feedback. Thanks!