Odd crash in gthread after page served

[dale-wahl]

I just started to see this after rebuilding an older app after making some small changes.

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 609, in spawn_worker
    worker.init_process()
  File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/gthread.py", line 95, in init_process
    super().init_process()
  File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
    self.run()
  File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/gthread.py", line 232, in run
    self.murder_keepalived()
  File "/usr/local/lib/python3.8/site-packages/gunicorn/workers/gthread.py", line 176, in murder_keepalived
    self.poller.unregister(conn.sock)
  File "/usr/local/lib/python3.8/selectors.py", line 366, in unregister
    key = super().unregister(fileobj)
  File "/usr/local/lib/python3.8/selectors.py", line 249, in unregister
    key = self._fd_to_key.pop(self._fileobj_lookup(fileobj))
  File "/usr/local/lib/python3.8/selectors.py", line 225, in _fileobj_lookup
    return _fileobj_to_fd(fileobj)
  File "/usr/local/lib/python3.8/selectors.py", line 42, in _fileobj_to_fd
    raise ValueError("Invalid file descriptor: {}".format(fd))
ValueError: Invalid file descriptor: -1

I was not sure if I was causing it, so I started up a very simple Flask app and was able to reproduce. It serves the home page fine, but a moment later I see the above and the worker restarts.

Command
gunicorn --worker-tmp-dir /dev/shm --workers=1 --threads=4 --worker-class=gthread --log-level=debug --access-logfile=gunicorn_access.log --reload --bind 0.0.0.0:4000 frontend:app

Environment

blinker==1.6.2
click==8.1.5
Flask==2.3.2
gunicorn==21.0.1
importlib-metadata==6.8.0
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.3
packaging==23.1
Werkzeug==2.3.6
zipp==3.16.2

Will revert to a previous version and see if the error persists.

[benoitc]

i will try. That maybe related to the python version

[dale-wahl]

I tried on both 3.7 and 3.8 and received the same error. Probably time to upgrade my python version!

Downgrading to gunicorn 20 resolved the issue for me.

[tari]

I noticed this occurring intermittently for one of my servers today as well, and the timing seems consistent with the first build of a container image that uses gunicorn 21.

It doesn't occur on every request, but when it occurs my downstream proxy (nginx) drops the backend for a bit, saying recv() failed (104: Connection reset by peer) while reading response header from upstream which causes it to temporarily disable the backend and return 502s for a bit which is visible to my monitoring as occasional errors. The observed error rate is around 10%, but actual rate per request is probably lower because the backend gets disabled for 10 seconds after each failure.

I'm using Python 3.11 and running a Django app, but the logs show the same traceback as Dale shared.

[benoitc]

I reproduced it thanks; This should be fixed in 86d85cb

[benoitc]

21.1.0 has been released with the fix #3031