Skip to content

Reqest entity too large #874

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
buremba opened this issue Mar 1, 2017 · 16 comments
Closed

Reqest entity too large #874

buremba opened this issue Mar 1, 2017 · 16 comments

Comments

@buremba
Copy link

buremba commented Mar 1, 2017

Do you want to request a feature or report a bug?
Bug

What is the current behavior?
If the user flow involves some big HTTP requests and operations, the error logging fails with "Reqest entity too large" because Sentry tries to send full stack-trace and HTTP request logs to the backend server.

What is the expected behavior?
The SDK should truncate the body if it's too large.

Which versions of Raven.js, and which browser and OS are affected by this issue?
I'm using the version 3.6 with your SaaS offering.

@benvinegar
Copy link
Contributor

benvinegar commented Mar 1, 2017

because Sentry tries to send full stack-trace and HTTP request logs to the backend server.

Raven.js does not send HTTP request logs, unless you're doing something custom.

The SDK should truncate the body if it's too large.

It truncates frames. It doesn't send HTTP request bodies so it wouldn't know to do this.

Are you appending your own data manually or using a plugin like redux-raven-middleware? Please attach a copy of a sample outbound report and I can act on this.

I'm using the version 3.6 with your SaaS offering.

This version is pretty out of date now.

@buremba
Copy link
Author

buremba commented Mar 2, 2017

I use Angular plugin with default configurations but you're right, the version is pretty out of date. I upgraded to the last version and let you know if it occurs again. Thanks.

Here is a sample:
curl 'https://app.getsentry.com/api/app_id/store/?sentry_version=7&sentry_client=raven-js%2F3.9.2&sentry_key=KEY' -H 'Pragma: no-cache' -H 'Origin: https://app.rakam.io' -H 'Accept-Encoding: gzip, deflate, br' -H 'Accept-Language: tr-TR,tr;q=0.8,en-US;q=0.6,en;q=0.4' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36' -H 'Content-Type: text/plain;charset=UTF-8' -H 'Accept: */*' -H 'Cache-Control: no-cache' -H 'Connection: keep-alive' -H 'DNT: 1' --data-binary $'{"project":"nope","logger":"javascript","platform":"javascript","request":{"headers":{"User-Agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36"},"url":"https://app.rakam.io/report-playground?query=select%201,2,3&options=%7B%22chartType%22:%22geo-cluster%22,%22limit%22:1000,%22enableStatistics%22:true,%22tableOptions%22:%7B%22enabled%22:true%7D,%22chartOptions%22:%7B%22latitude%22:0,%22longitude%22:1,%22title%22:2%7D,%22variables%22:%5B%5D,%22segments%22:%5B%5D,%22measures%22:%5B0,1,2%5D,%22dimensions%22:%5B%5D%7D&queryOptions=%7B%7D"},"exception":{"values":[{"type":"TypeError","value":"Failed to execute \'appendChild\' on \'Node\': parameter 1 is not of type \'Node\'.","stacktrace":{"frames":[{"filename":"https://app.rakam.io/static/components/raven-js/dist/raven.js","lineno":372,"colno":29,"function":"HTMLImageElement.wrapped","in_app":false},{"filename":"https://app.rakam.io/static/dist/scripts/1.1-353348d0887eb92338a9.js","lineno":3,"colno":20753,"function":"HTMLImageElement.o [as _leaflet_click27]","in_app":true},{"filename":"https://app.rakam.io/static/dist/scripts/1.1-353348d0887eb92338a9.js","lineno":2,"colno":17248,"function":"e._onMouseClick","in_app":true},{"filename":"https://app.rakam.io/static/dist/scripts/1.1-353348d0887eb92338a9.js","lineno":1,"colno":5028,"function":"e.fireEvent","in_app":true},{"filename":"https://app.rakam.io/static/dist/scripts/1.1-353348d0887eb92338a9.js","lineno":2,"colno":23910,"function":"e.togglePopup","in_app":true},{"filename":"https://app.rakam.io/static/dist/scripts/1.1-353348d0887eb92338a9.js","lineno":2,"colno":23724,"function":"e.openPopup","in_app":true},{"filename":"https://app.rakam.io/static/dist/scripts/1.1-353348d0887eb92338a9.js","lineno":2,"colno":23431,"function":"e.openPopup","in_app":true},{"filename":"https://app.rakam.io/static/dist/scripts/1.1-353348d0887eb92338a9.js","lineno":1,"colno":21249,"function":"e.addLayer","in_app":true},{"filename":"https://app.rakam.io/static/dist/scripts/1.1-353348d0887eb92338a9.js","lineno":1,"colno":29748,"function":"e._layerAdd","in_app":true},{"filename":"https://app.rakam.io/static/dist/scripts/1.1-353348d0887eb92338a9.js","lineno":2,"colno":19069,"function":"e.onAdd","in_app":true},{"filename":"https://app.rakam.io/static/dist/scripts/1.1-353348d0887eb92338a9.js","lineno":2,"colno":20008,"function":"e.update","in_app":true},{"filename":"https://app.rakam.io/static/dist/scripts/1.1-353348d0887eb92338a9.js","lineno":2,"colno":21514,"function":"e._updateContent","in_app":true}]}}]},"culprit":"https://app.rakam.io/static/dist/scripts/1.1-353348d0887eb92338a9.js","extra":{"cluster":"4c8e91d0-471e-4041-a11f-f818b2b757a1","name":"rakam-prod","type":"ui","session:duration":3993424},"breadcrumbs":{"values":[{"timestamp":1488365533.259,"category":"ui.click","message":"td > i.margin-horizontal-5.pointer.fa.fa-play-circle.green"},{"timestamp":1488365533.335,"type":"http","category":"xhr","data":{"method":"GET","url":"https://api.mixpanel.com/track/?data=%3D&ip=1&_=1488365533262","status_code":200}},{"timestamp":1488365533.497,"type":"http","category":"xhr","data":{"method":"POST","url":"https://webapp-1290649538.us-west- .......... TONS OF HTTP REQUESTS

@benvinegar
Copy link
Contributor

benvinegar commented Mar 2, 2017

TONS OF HTTP REQUESTS

So ... those look like breadcrumbs, and we limit those to 50 per page. If your URLs are explosively large, they could present a problem. So you could set maxBreadcrumbs: 20 or something.

@buremba
Copy link
Author

buremba commented Mar 2, 2017

They're quite large because these Mixpanel calls have a big base64 encoded query parameter that includes the event body. I believe that the best option here is to discard some third party analytics service calls such as Mixpanel or truncate the url if it's too large but it seems that both of them requires some work do be done in this repo, right?

@benvinegar
Copy link
Contributor

Ah, that's very helpful. Yes, we should truncate this client-side.

@buremba
Copy link
Author

buremba commented Mar 2, 2017

It looks like I also removed Mixpanel HTTP calls in the snippet since they may include some user sensitive data but that part is particularly large.

@benvinegar
Copy link
Contributor

Note, in the meantime you can use breadcrumbCallback to truncate (or disable) HTTP breadcrumbs. See config.

@derekrprice
Copy link

We are seeing this today with many large breadcrumbs, and it crashes the browser. I would argue that Sentry.io should NEVER crash the browser regardless of what the user does. Could you please add some sort of automatic client-side checking and truncation of the packet size before throwing an error in the client, like Sentry's Python integration did.

@derekrprice
Copy link

I think this issue needs to be reopened.

@kamilogorek
Copy link
Contributor

What do you mean by "crash the browser"? Like... an actual crash? How did that happen? Can you provide some more details?

@derekrprice
Copy link

It threw an error in the browser, effectively crashing the JS application running there.

@derekrprice
Copy link

As for details on how to reproduce, log a lot of large breadcrumbs, then hit another error. Raven throws its own exception and does not log the original error, either to the console or to the server.

@derekrprice
Copy link

Ok, looking closer at this on my end, I may have been a bit alarmist. Sentry's thrown error is not crashing the application since it starts with our own JS error, but it IS true that the large number of breadcrumbs are preventing sentry from logging the error to the server. I think truncating the breadcrumbs in the client to ensure that the stack trace gets logged to the server should be standard. I would rather see the stack trace with a note about truncated breadcrumbs than nothing at all when my users are encountering real errors.

@kamilogorek
Copy link
Contributor

kamilogorek commented Apr 9, 2018

We'll rethink the approach to this issue in the next major release. Right now, however, it'd be too much overhead to do this operating for every single event and it may impact the performance of our SDK.

We do provide all the required tools so that developers can do that themselves if they are willing to - https://github.com/getsentry/raven-js/blob/deff2f06b40d10389705167c25efc2e737240e3e/packages/raven-js/src/utils.js#L486-L553

If you want to calculate the size of the breadcrumbs every single time and truncate them if they are too heavy, feel free to do that. Here's how you'd approach something like this:

const utils = require('../node_modules/raven-js/src/utils'); // or just extract it to your own file, its up to you

const MAX_SIZE = 50 * 1024;

function truncateBreadcrumbs (data) {
    // Handle edge-case to prevent an infinite loop
    if (utils.jsonSize(utils.stringify(data)) > MAX_SIZE) {
      data.breadcrumbs = data.breadcrumbs.split(0, -1);
      return truncateBreadcrumbs(data)
    } else {
      return data;
    }
}

Raven.config('__YOUR_DSN__', {
  dataCallback: function (data) {
    return truncateBreadcrumbs(data);
  }
}).install();

@CaelanStewart
Copy link

CaelanStewart commented Jul 12, 2019

Edit
I had a read through the docs, and I found that the Raven Vue integrations deprecated. I switched to the new centralised integrations package. I noticed that the option attachProps: true is there by default, setting that to false solved the 413 issue.

There isn't much visibility that I could find searching the docs about that option, and that it may be set to false to reduce payload size. It feels like it should be more prominent in your docs. A lot is left to the deduction of the developer.

Now to look into how to dynamically send or not send props depending on if the payload size exceeds the maximum. It would be great if there were a third value for attachProps such as 'auto' that would check the payload size, and if there is space enough, add the prop data.


@kamilogorek - we're getting this error - the app is intrinsically data heavy.

Your code refers to functions that no longer exist.

Would I be right in assuming the only thing those non-existent functions do is allow you to encode JSON with circular references?

If so, if I were to choose another library and use their stringification function, in place of your old utilities, would your code sample work still?

Also, is there a better, built in solution to this? We could probably get away without sending any app state at all, just console log, the error in question and other minutiae.

Otherwise errors gobbled up by Vue don't make it up and aren't reported without the Raven/Vue plugin - unless I've setup the Sentry browser client incorrectly. As only an error allow to propagate uncaught appeared in Sentry.

Cheers

@kamilogorek
Copy link
Contributor

kamilogorek commented Jul 23, 2019

Your code refers to functions that no longer exist.

@CaelanStewart It does, but here (@sentry/utils package) -

export function normalizeToSize<T>(
object: { [key: string]: any },
// Default Node.js REPL depth
depth: number = 3,
// 100kB, as 200kB is max payload size, so half sounds reasonable
maxSize: number = 100 * 1024,
): T {
const serialized = normalize(object, depth);
if (jsonSize(serialized) > maxSize) {
return normalizeToSize(object, depth - 1, maxSize);
}
return serialized as T;
}

Would I be right in assuming the only thing those non-existent functions do is allow you to encode JSON with circular references?

Kinda, but also some more:

/**
 * normalize()
 *
 * - Creates a copy to prevent original input mutation
 * - Skip non-enumerablers
 * - Calls `toJSON` if implemented
 * - Removes circular references
 * - Translates non-serializeable values (undefined/NaN/Functions) to serializable format
 * - Translates known global objects/Classes to a string representations
 * - Takes care of Error objects serialization
 * - Optionally limit depth of final output
 */

Also, is there a better, built in solution to this? We could probably get away without sending any app state at all, just console log, the error in question and other minutiae.

You can investigate vue's context (and optional alter/remove it) with beforeSend callback.

Sentry.init({
  beforeSend(event) {
    // event.contexts.vue has all the metadata
    return event;
  }
})

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants