Request Entity Too Large #604

pehrlich · 2015-04-29T01:59:26Z

Recently started getting the follow error, without any explicit change to Raven or Sentry configuration. Google results are unrevealing. Is there something known that can cause this?

Any proper ways of logging more fully the original AttributeError, which caused the Sentry error, besides disabling Sentry?

INFO:werkzeug:172.17.42.1 - - [29/Apr/2015 01:02:51] "POST / HTTP/1.1" 500 -
ERROR:sentry.errors:Unable to reach Sentry log server: HTTP Error 413: Request Entity Too Large (url: https://app.getsentry.com/api/39671/store/, body: b'<html>\r\n<head><title>413 Request Entity Too Large</title></head>\r\n<body bgcolor="white">\r\n<center><h1>413 Request Entity Too Large</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n')
Traceback (most recent call last):
  File "/usr/local/lib/python3.4/site-packages/raven/transport/threaded.py", line 159, in send_sync
    super(ThreadedHTTPTransport, self).send(data, headers)
  File "/usr/local/lib/python3.4/site-packages/raven/transport/http.py", line 49, in send
    ca_certs=self.ca_certs,
  File "/usr/local/lib/python3.4/site-packages/raven/utils/http.py", line 62, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.4/urllib/request.py", line 461, in open
    response = meth(req, response)
  File "/usr/local/lib/python3.4/urllib/request.py", line 571, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/local/lib/python3.4/urllib/request.py", line 499, in error
    return self._call_chain(*args)
  File "/usr/local/lib/python3.4/urllib/request.py", line 433, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.4/urllib/request.py", line 579, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 413: Request Entity Too Large
ERROR:sentry.errors:Failed to submit message: "AttributeError: 'dict' object has no attribute 'encode'"

The text was updated successfully, but these errors were encountered:

dcramer · 2015-04-29T02:09:18Z

Is this getsentry.com or are you hosting it yourself?

pehrlich · 2015-04-29T03:03:56Z

Getsentry.com

On Tuesday, April 28, 2015, David Cramer [email protected] wrote:

Is this getsentry.com or are you hosting it yourself?

—
Reply to this email directly or view it on GitHub
#604 (comment)
.

dcramer · 2015-04-29T06:50:05Z

This would mean the packet is very large. I think we should do something on our end around logging, but it's a little complicated. We definitely should log a reasonable version of the message (beyond just what we give you here). Unfortunately without changes to the client I don't have a great answer.

pehrlich · 2015-04-29T07:03:03Z

Hm, ok. What composes the packet? I am working with a number large files,
perhaps one is getting itself attached to the error report somehow, and we
definetly wouldn't want or need that.

Happy to test out or make changes as advised.

On Tuesday, April 28, 2015, David Cramer [email protected] wrote:

This would mean the packet is very large. I think we should do
something on our end around logging, but it's a little complicated. We
definitely should log a reasonable version of the message (beyond just what
we give you here). Unfortunately without changes to the client I don't have
a great answer.

—
Reply to this email directly or view it on GitHub
#604 (comment)
.

dcramer · 2015-04-30T19:50:43Z

Closing this out as it was (potentially) resolved elsewhere

pehrlich · 2015-04-30T19:54:54Z

nice, thank you

jacobsvante · 2015-08-27T09:56:07Z

It was? I'm still experiencing 413 responses from getsentry.com when POSTing a 5.3MB payload.
Perhaps you could log something about the 413 in Sentry when this happens so one knows an error occurred. Took some digging in syslogs to find the error.

dcramer · 2015-08-27T17:48:47Z

@jmagnusson do you see this in the 5.6 release?

jacobsvante · 2015-08-28T06:35:11Z

Yup, still. Sentry responded with an error: HTTP Error 413: Request Entity Too Large (url: https://app.getsentry.com/api/.../store/)

pehrlich · 2015-09-23T19:30:48Z

So this is still an issue, although thanks to your help earlier/elsewhere, it is being logged sufficiently locally.

A believe a couple actions should be taken:

Upon receiving a too large entity, the server should log a proxy exception, alerting the user that an event happened which was un-capturable. This would prevent silent doom.
An API should be devised to all the automatic or selective truncation of large strings when errors are thrown. E.g., if I have an attribute error on a dictionary with a large string key, in many cases I don't have any need to know the contents of that key, beyond perhaps the first 100 chars to indicate its further contents.

jhubert · 2016-03-09T06:29:34Z

Seeing this as well from the error dump of a complex elasticsearch query.

jacobsvante · 2016-03-09T09:01:04Z

It still hasn't been resolved?

ruimarinho · 2016-04-11T10:34:00Z

Just got hit by this issue (not related to the python lib though). What's the maximum body size allowed by sentry?

dcramer · 2016-04-11T16:32:46Z

Most of the time this is due to the configuration of the server. If this is getsentry.com the request limit is extremely high and there's no way you should be hitting it. If you are, then it likely means that the client side variable trimming isn't working correctly, or has been re-configured to send data that is too large.

ruimarinho · 2016-04-11T22:42:40Z

This is getsentry.com and the issue is that the metadata I've submitted is simply too large and not due to an error in trimming (it exceeds 1Mb). @pehrlich mentioned that the sentry server could log an exception whenever a request that contains a large body is discarded, but that would require processing the whole message to find out the id which would defeat the original protection in place - unless, of course, the id can be obtained in another way (query parameter?) while still discarding an offending body.

The second @pehrlich's suggestion is inline we something that I wanted to suggest too - let the client preemptively calculate the body size and remove certain information (e.g. all metadata?) so that at least the most relevant pieces still won't go unnoticed.

evdoks · 2016-06-24T13:27:08Z

Any news here? We are still getting this error 413 with getsentry.com.

jakubzitny · 2016-06-27T11:18:30Z

We were having the same problem on self-hosted sentry. Of course, the problem was in request size limit in Nginx.

joar · 2016-10-05T07:42:48Z

We are experiencing this issue with raven==5.36.0, using raven.transport.http.HTTPTransport, on https://app.getsentry.com/api/32671/store/.

In our case, it's also seems to be related to elasticsearch logging.

jacobsvante · 2017-10-06T09:09:34Z

Still getting request entity too large:

2017-10-06 04:02:45,810|sentry.errors|ERROR|Sentry responded with an error: HTTP Error 413: Request Entity Too Large (url: https://sentry.io/api/<id>/store/) (base.py:682)

Any chance this could get prioritized? I'm missing exceptions from some of our most important endpoints because of it.

mitsuhiko · 2017-10-06T17:46:53Z

I think we will need to do some better pruning for the breadcrumbs which I assume are triggering this in the case mentioned above.

jacobsvante · 2017-11-02T16:03:02Z

@mitsuhiko Any hints as to when this might be fixed?

mitsuhiko · 2017-11-02T18:29:50Z

I'm going to take a look at this.

geuben · 2017-11-21T21:42:45Z

I am also experiencing this issue. The excessive size in my case is certainly coming from the breadcrumbs

mitsuhiko · 2017-12-06T21:07:52Z

(that would be sql queries etc. Just want to get an idea where to trim here)

fcoury · 2017-12-07T15:59:17Z

I am running into the same issue on a React/Redux app:

Content-Length:183107

Is 183kb considred too large?

philtrep · 2017-12-07T16:17:40Z

@fcoury The limit seems to be 100kb https://docs.sentry.io/learn/quotas/#attributes-limits

@mitsuhiko mistag 😉 Currently investigating, should have more info later today or tomorrow.

notatestuser · 2017-12-18T04:13:55Z

Having a request that is around 111KB dropped by sentry.io/api. That limit seems a little low - we simply log all of our redux actions and send them as breadcrumbs.

whisller · 2018-01-03T10:31:45Z

Experiencing same problem with raven==6.1.0 + raven-python-lambda==0.1.4.

Sentry responded with an error: HTTP Error 413: Request Entity Too Large (url: https://sentry.io/api/211040/store/)

peterbe · 2018-01-25T20:13:54Z

I have raven==0.6.4 and Django 1.17. Saw these errors in my uwsgi log too. No idea what to do about it.
I injected a little logging into the lib/python3.5/site-packages/raven/utils/http.py just to get an insight into what that data is. I still don't know what it is other than it's some binary blob. It's size hovers around 100KB.
I'm using sentry.io.

jacobsvante · 2018-02-01T20:31:54Z

This issue almost 3 years old now. Will it ever be fixed?

I've always loved Sentry and felt confident in that it will notify us of any exceptions that occur in production, but this issue scuffs that feeling a bit.

dcramer · 2018-02-01T20:32:53Z

@jmagnusson it should be very rare you'd see this, and might be more commonly happening with on-premise installs since max size is configurable. If you're seeing it on sentry.io let us know and we can probably just duct tape packet size in the mean time.

jacobsvante · 2018-02-01T20:33:22Z

@dcramer We're using your hosted solution

peterbe · 2018-02-01T20:59:32Z

@dcramer I'm using sentry.io and I'm getting it too.

dcramer · 2018-02-01T22:08:42Z

@jmagnusson @peterbe are you still experiencing this?

peterbe · 2018-02-02T13:35:16Z

are you still experiencing this?

As of 8 days ago I was. See my previous comment.
I can provide you more details but in private in case I do something silly that reveals too much.

dcramer · 2018-02-02T16:44:21Z

I meant as in right now. We made changes yesterday which should have reduced the impact.

peterbe · 2018-02-02T20:00:20Z

@dcramer Yes, but because I don't know how to corner this, here's what I did:
(Note: I'm doing this on a side-project so attention is restricted)

I injected some print in site-packages/raven/utils/http.py towards the end:

-   return opener.open(url, data, timeout)
+   try:
+        return opener.open(url, data, timeout)
+   except Exception:
+        print("EXCEPTION!!!!", url, url.get_full_url(), data[:1000])
+        raise

Then I hawkishly watch my uwsgi log file and I get this:

EXCEPTION!!!! <urllib.request.Request object at 0x7fd9103efb38> https://sentry.io/api/196160/store/ b'x\x9c\xec\xbdiw\xe2H\xb20\xfcW\xd4\xf5~p\xd5)\x97\xd9\xb7\xea;\xb7\x0f\x8b0xe\xf3\x82\xa7\xe6\xf4MP"\xc9\x08\t$
Sentry responded with an error: HTTP Error 413: Request Entity Too Large (url: https://sentry.io/api/196160/store/)
Traceback (most recent call last):
  File "/var/lib/django/songsearch/venv/lib/python3.5/site-packages/raven/transport/threaded.py", line 165, in send_sync
    super(ThreadedHTTPTransport, self).send(url, data, headers)
  File "/var/lib/django/songsearch/venv/lib/python3.5/site-packages/raven/transport/http.py", line 43, in send
    ca_certs=self.ca_certs,
  File "/var/lib/django/songsearch/venv/lib/python3.5/site-packages/raven/utils/http.py", line 67, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 472, in open
    response = meth(req, response)
  File "/usr/lib/python3.5/urllib/request.py", line 582, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python3.5/urllib/request.py", line 510, in error
    return self._call_chain(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/usr/lib/python3.5/urllib/request.py", line 590, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 413: Request Entity Too Large
['OSError: write error']

dcramer · 2018-02-02T20:07:33Z

@peterbe any chance you could try to capture data to a file and email that to david at sentry.io? This might be something where its genuinely creating a gigantic packet. The SDK is supposed to limit the length of everything, but thats not always been true.

dcramer · 2018-02-02T20:08:28Z

Aside, you can sign with PGP key if you're concerned: https://sentry.io/security/

peterbe · 2018-02-02T20:21:14Z

@dcramer I don't think it needs to be signed up I've written a little hack now to generate these data files. I'll email you one if I get another exception. At the time of writing I haven't had one happen in a while.

dcramer · 2018-02-02T20:39:39Z

The problem in this specific example is breadcrumbs are not truncating log messages. In this case theres an elasticsearch logger (not sure if yours or the official library), and an unbelievably large debug message which seems to be the full payload being inserted.

peterbe · 2018-02-02T21:12:27Z

Stock Django 1.11 with no fancy logging specifically for elasticsearch_dsl. I appreciate that sometimes Elasticsearch barfs but there should still be an error sent to Sentry. I do use this little snippet for calling elasticsearch_dsl:

from elasticsearch.exceptions import (
    ConnectionTimeout,
    ConnectionError,
    TransportError,
    NotFoundError,
)

_es_operational_exceptions = (
    ConnectionTimeout,
    ConnectionError,
    TransportError,
)


def es_retry(callable, *args, **kwargs):
    sleep_time = kwargs.pop('_sleep_time', 1)
    attempts = kwargs.pop('_attempts', 10)
    verbose = kwargs.pop('_verbose', False)
    try:
        return callable(*args, **kwargs)
    except _es_operational_exceptions as exception:
        if attempts:
            attempts -= 1
            if verbose:
                print("ES Retrying ({} {}) {}".format(
                    attempts,
                    sleep_time,
                    exception
                ))
            time.sleep(sleep_time)
        else:
            raise

I don't see those ES Retrying prints in the uwsgi logs.

dcramer · 2018-02-02T23:29:32Z

@peterbe sorry I should have been clear that i was intending to fix it, but the behavior (of using debug messages to log e.g. raw packets) is poor in general. ES certainly isnt alone in that.

danieledapo · 2018-02-09T10:29:06Z

any chance of deploying a new version of raven which includes the fix? This issue has been a real pain for a long time for us and we really want to try this fix.

ashwoods · 2018-02-09T20:33:03Z

@d-dorazio will be out by tomorrow.

ryechus · 2020-05-05T22:26:33Z

I am still experiencing this issue even after updating to version 6.10.0.

davidpaley · 2020-08-04T20:20:21Z

I am having this error working with next.js and "@sentry/react": version 5.20.1 with the default configuration of React

Empact · 2020-11-06T05:55:36Z

For reference, @davidpaley apparently solved his problem here: getsentry/sentry-javascript#2798

dcramer mentioned this issue Apr 30, 2015

Improve failure handling and logging #605

Merged

dcramer closed this as completed Apr 30, 2015

jacobsvante mentioned this issue Aug 27, 2015

Limit request data to a maximum of 4MB #645

Closed

dcramer reopened this Oct 6, 2017

dcramer assigned mitsuhiko Oct 6, 2017

mitsuhiko assigned ashwoods Oct 6, 2017

ashwoods added the Breadcrumbs label Nov 24, 2017

peterbe mentioned this issue Jan 25, 2018

Sentry responded with an error: HTTP 413: Request Entity Too Large #981

Closed

dcramer mentioned this issue Feb 2, 2018

fix(breadcrumbs): Ensure maximum lengths on several attributes #1194

Merged

ashwoods closed this as completed in #1194 Feb 3, 2018

derekrprice mentioned this issue Apr 6, 2018

Reqest entity too large getsentry/sentry-javascript#874

Closed

Empact mentioned this issue Nov 6, 2020

413 Request Entity Too Large on Breadcrumbs getsentry/sentry-javascript#3036

Closed

Uh oh!

Request Entity Too Large #604

Request Entity Too Large #604

Comments

pehrlich commented Apr 29, 2015

dcramer commented Apr 29, 2015

Uh oh!

pehrlich commented Apr 29, 2015

Uh oh!

dcramer commented Apr 29, 2015

Uh oh!

pehrlich commented Apr 29, 2015

Uh oh!

dcramer commented Apr 30, 2015

Uh oh!

pehrlich commented Apr 30, 2015

Uh oh!

jacobsvante commented Aug 27, 2015

Uh oh!

dcramer commented Aug 27, 2015

Uh oh!

jacobsvante commented Aug 28, 2015

Uh oh!

pehrlich commented Sep 23, 2015

Uh oh!

jhubert commented Mar 9, 2016

Uh oh!

jacobsvante commented Mar 9, 2016

Uh oh!

ruimarinho commented Apr 11, 2016

Uh oh!

dcramer commented Apr 11, 2016

Uh oh!

ruimarinho commented Apr 11, 2016

Uh oh!

evdoks commented Jun 24, 2016

Uh oh!

jakubzitny commented Jun 27, 2016

Uh oh!

joar commented Oct 5, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jacobsvante commented Oct 6, 2017

Uh oh!

mitsuhiko commented Oct 6, 2017

Uh oh!

jacobsvante commented Nov 2, 2017

Uh oh!

mitsuhiko commented Nov 2, 2017

Uh oh!

geuben commented Nov 21, 2017

Uh oh!

mitsuhiko commented Dec 6, 2017

Uh oh!

fcoury commented Dec 7, 2017

Uh oh!

philtrep commented Dec 7, 2017

Uh oh!

notatestuser commented Dec 18, 2017

Uh oh!

whisller commented Jan 3, 2018

Uh oh!

peterbe commented Jan 25, 2018

Uh oh!

jacobsvante commented Feb 1, 2018

Uh oh!

dcramer commented Feb 1, 2018

Uh oh!

jacobsvante commented Feb 1, 2018

Uh oh!

peterbe commented Feb 1, 2018

Uh oh!

dcramer commented Feb 1, 2018

Uh oh!

peterbe commented Feb 2, 2018

Uh oh!

dcramer commented Feb 2, 2018

Uh oh!

peterbe commented Feb 2, 2018

Uh oh!

joar commented Oct 5, 2016 •

edited

Loading

dcramer commented Feb 2, 2018 •

edited

Loading

dcramer commented Feb 2, 2018 •

edited

Loading