Massive issue influx from single client

[bobvandevijver]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Self-hosted/on-premise

Which package are you using?

@sentry/vue

SDK Version

7.19.0

Framework Version

7.19.0

Link to Sentry event

No response

Steps to Reproduce

With our latest production update yesterday, we went from Sentry 7.17.4 to 7.19.0. Shortly afterwards, our installation was flooded by events from a single IP-address, causing all kinds of JS errors that are not related to the webpage at all, but are clear result from an automated scan. Examples are:

• ReferenceError: createPopper is not defined
• ReferenceError: Swiper is not defined
• ReferenceError: PrefixFree is not defined

The common divider here is that all those event are caused by an anonymous script that is probably injected by the user itself.

This single user (with a Chinese IP-address) singlehandedly triggered about 500 events (of which 200x an ingress limited event). A second (again Chinese) IP-address just triggered 2.4k of events (of which 1.4k ingress limited events).

This has not happened before, so maybe something broke with the detection of these kinds of events? I am not seeing anything related in the release notes though...

Expected Result

I believe these events should be filtered before being posted to Sentry.

Actual Result

Complete chaos 😨

[Lms24]

Hi @bobvandevijver thanks for writing in! This seems strange to me, as we didn't change anything SDK-wise w.r.t filtering out known, problematic errors. Is this still happening or was it just at this one time?

We do try filtering out certain errors on the server-side, in Relay. For example crawler errors or browser extension errors. You can enable or disable automatic filtering, as well as apply custom rules under your project's settings tab in Sentry: "Project Settings" -> "Inbound Filters". It might be worth trying to filter out specific error messages that you observed were only triggered by this one user.

Do you by any chance have access to the user-agent sent by this user? Perhaps it's a new crawler or something we could add to our filters.

[bobvandevijver]

We've seen it three times, but I've blocked the source IPs on the application firewall to at least prevent them from further scanning.

For example crawler errors or browser extension errors.

Yeah, both are enabled (by default I believe?)

It might be worth trying to filter out specific error messages that you observed were only triggered by this one user.

They are basically all unique errors, all reference errors for a lot of different items making that quite impossible. The one I mentioned in the opening post (2.4k events, 1.4k ingress events) triggered 1k different error events!

Here is a screenshot of just the first page of the last time it happened:

Here's a screenshot of one of the events:

Most curious for me is the error when trying to resolve that anonymous URL actually. When trying to reproduce this by triggering something manually with setTimeout(() => UnknownSomething.call(), 1000) Sentry didn't bother trying to resolve the anonymous line at all...

Do you by any chance have access to the user-agent sent by this user? Perhaps it's a new crawler or something we could add to our filters.

Absolutely, here is is:

Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/599.37 (KHTML, like Gecko) Chrome/55.0.2590 Safari/537.36

It seems to be using an ancient Chrome version...

[thallada]

We are also seeing the massive influx of errors. I keep trying to block them by IP but they keep using new IPs so it's a never ending cat and mouse. Can they get added to the crawler blocking? (we have both crawlers and errors from extensions blocked).

User agents I've seen (this is definitely not all of them, just a random sample from some of the spam errors):

• Mozilla/5.0 (Windows NT 7_2_2; Win64; x64) AppleWebKit/577.46 (KHTML, like Gecko) Chrome/105.0.1775 Safari/537.36
• Mozilla/5.0 (Macintosh; Intel Mac OS X 9_1) AppleWebKit/602.42 (KHTML, like Gecko) Chrome/54.0.2748 Safari/537.36
• Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/579.45 (KHTML, like Gecko) Chrome/58.0.605 Safari/537.36
• Mozilla/5.0 (Macintosh; Intel Mac OS X 9_0_1) AppleWebKit/577.54 (KHTML, like Gecko) Chrome/69.0.2215 Safari/537.36

Since these user agents also seems to be frequently changing, it might not be a good way to block them (even if I could do that, doesn't seem possible, see getsentry/sentry#12753).

The only thing consistent between all of them is that the IP addresses are geolocated to somewhere in China. Also they all have stack traces that start with at ? (<anonymous>:1:28).

This started happening for us around November 4th.

Edit: I added a new Issue Grouping Fingerprint Rule that I hope will target these spam errors specifically:

error.type:ReferenceError stack.abs_path:"<anonymous>" -> spam-reference-error

It only applies to future errors, so I won't know if it will work until they spam us again. If it does I should be able to permanently ignore the "spam-reference-error" issue group.

Edit 2: It worked. I can now ignore all of these spam errors grouped together.

[Lms24]

Hi @bobvandevijver and @thallada ,
can you confirm that this only happens with newer SDK versions? Does it still happen if you downgrade the SDK to 7.17.4 or older? We're curious to see if this is just a bad timing or if there's in fact some problem with newer SDK versions.
Overall, to us these user agents look pretty normal in the sense that you can't really identify the user as a crawler by them. Which makes filtering them on our end pretty hard.

[bobvandevijver]

@Lms24 Actually, the instance I placed the screenshots of was also the last time it happened to us (for now). I updated a couple projects to the same release at the same time (as it was our deployment evening), but it only happened at that particular project. So, I'm starting to think it was indeed just bad timing, also due to the comment of @thallada (thanks for the grouping rule by the way, I just configured it for our project as well!)

I will keep monitoring this, but I am still curious about the cause of Sentry trying to resolve that 'anonymous' url... I have not seen that before, also not with my own anonymous functions triggering similar stack traced.

[thallada]

can you confirm that this only happens with newer SDK versions?

The two apps I'm getting these errors from are both on a pretty old version of @sentry/nextjs actually: 6.19.7

I'll see if upgrading changes anything.

[Lms24]

I'm going to close this issue for the time being. Please feel free to reply here in case these problems continue to appear.

[HuyAms]

Hi, I installed Sentry on my site built with Remix.

I got spam with many errors that from external scripts (such as tracking script, google analytics?, etc). How could I possibly filter out those errors?

[meotimdihia]

Hi, I have the same problem as @HuyAms
New errors always appear in the ad scripts.
And I have to manually do it every time.
Do we have any configuration to do it? these errors from external scripts

[Lms24]

Hi @HuyAms and @meotimdihia, really sorry to hear that. Is it coming from the same IP addresses as described above?

In any case, you can add patterns of error messages in ignoreErrors or configure beforeSend to filter out these errors based on all the data (e.g. stack trace) we collect when the SDK captures the errors.

[meotimdihia]

@Lms24 Sorry, I think our problem is different with this issue, but when I saw @HuyAms posted it and I replied:

We came from this issue: #6374

[HuyAms]

@Lms24 do we have any example of filtering out the noise? It seems to be a quite common issue. Do we have a document/post somewhere?

[Lms24]

@HuyAms have you tried the issue grouping rule mentioned in this comment above?

[dlindenkreuz]

We are also receiving a bunch of similar events where a small number of IP addresses attempts to detect a bunch of global variables, probably as part of some vulnerability scanning.

I added the Issue Grouping Fingerprint Rule suggested above.

error.type:ReferenceError stack.abs_path:"<anonymous>" -> spam-reference-error

It would be great if Sentry looked into building some sort of spam filter. Alternatively, the docs could mention some approaches to combat spam reports on a more individual basis.

[enapupe]

I'm having the same issue as all, a surge of errors coming from the "same" user.
I thought adding my own JS source URL to allowUrls would fix it but anonymous errors are still being reported.

This is what my current config looks like, I'm using latest @sentry/browser

init({
    dsn: appId,
    release: appVersion,
    allowUrls: [STATICS_URL],
    ignoreErrors: [
      // Random plugins/extensions
      'top.GLOBALS',
      // See: http://blog.errorception.com/2012/03/tale-of-unfindable-js-error.html
      'originalCreateNotification',
      'canvas.contentDocument',
      'MyApp_RemoveAllHighlights',
      'http://tt.epicplay.com',
      "Can't find variable: ZiteReader",
      'jigsaw is not defined',
      'ComboSearch is not defined',
      'http://loading.retry.widdit.com/',
      'atomicFindClose',
      // Facebook borked
      'fb_xd_fragment',
      // ISP "optimizing" proxy - `Cache-Control: no-transform` seems to reduce this. (thanks @acdha)
      // See http://stackoverflow.com/questions/4113268/how-to-stop-javascript-injection-from-vodafone-proxy
      'bmi_SafeAddOnload',
      'EBCallBackMessageReceived',
      // See http://toolbar.conduit.com/Developer/HtmlAndGadget/Methods/JSInjection.aspx
      'conduitPage',
      // Generic error code from errors outside the security sandbox
      // You can delete this if using raven.js > 1.0, which ignores these automatically.
      'Script error.',
      // Avast extension error
      '_avast_submit',
    ],
    denyUrls: [
      // Google Adsense
      /pagead\/js/i,
      // Facebook flakiness
      /graph\.facebook\.com/i,
      // Facebook blocked
      /connect\.facebook\.net\/en_US\/all\.js/i,
      // Woopra flakiness
      /eatdifferent\.com\.woopra-ns\.com/i,
      /static\.woopra\.com\/js\/woopra\.js/i,
      // Chrome extensions
      /extensions\//i,
      /^chrome:\/\//i,
      // Other plugins
      /127\.0\.0\.1:4001\/isrunning/i, // Cacaoweb
      /webappstoolbarba\.texthelp\.com\//i,
      /metrics\.itunes\.apple\.com\.edgesuite\.net\//i,
    ],
  })```