`--reload` option not working

[ahmedsaoudi]

Upon using Gunicorn for my Django app, I tried to enable the --reload option to make my work easier, avoiding manual restarting for every change.

Even though I tried the option on the command line (gunicron --reload --bind ...), I normally run Gunicorn as a systemd service. Here the config file:

[Unit]
Description='gunicorn daemon'
After=network.target

[Service]
User=me
Group=www-data
WorkingDirectory=path/to/djangoapp
ExecStart=/path/to/Envs/venv/bin/gunicorn --access-logfile - --error-logfile /path/to/gunicorn-error.log --workers 3 --reload --bind unix:/path/to/djangoapp/appname.sock appname.wsgi:application
ExecReload=/bin/kill -s HUP $MAINPID
ExecStop=/bin/kill -s TERM $MAINPID

[Install]
WantedBy=multi-user.target

As you can see, I'm using Gunicorn from a Virtualenv, as the rest of the app; but I don't know if that's related.

And apparently, I'm not alone; somebody else has the same issue: https://unix.stackexchange.com/questions/375292/how-to-restart-systemd-service-after-files-update-under-gunicorn-server

[benoitc]

is there any log that can be useful?

[Code0x58]

Does it just never work? The way the reloading works might not be quite as simple as you expect, see the code:

• if inotify isn't installed (or you use --reload-engine=poll) then gunicorn will only look for changes in your loaded python modules (including those added after each worker starts), and files added using --reload-extra-file

  • if you don't use a .py file extension and your file name ends with pyo/pyc, then it won't be watched

• otherwise the directories of python modules which were loaded when the watcher started, and of the files provided with --reload-extra-file will be watched for changes using inotify

  • files added to directories which were created after the watcher was created won't be watched
  • subdirectories of those watched are not automatically watched
  • if your editor uses memory mapping to modify the file, then it the changes won't be detected

  Modules that only imported in functions or inside something like if __name__ == '__main__':, unless they are loaded as a result of your gunicorn configuration file, or WSGI entry point.
  You could use inotifywait to see if the issue is to do with patchy watching, or inotify limitations.

Both implementations will have issues with your templates and static content also won't be monitored for changes.

As you are using Django, it may be reasonable for you to just use manage.py runserver, although that may also not be as good as using a hand configured inotifywait to manage reloads.

[tilgovi]

It might not work if you don't load your application code during configuration phase. I addressed that in my #1565 change. I can port that change to master.
https://github.com/benoitc/gunicorn/pull/1565/files#diff-454219f8d55400a662c02ab2a83186ae

[tilgovi]

A few separate issues:

• The .py file that corresponds to a .pyc file or .pyo file is watched. It's true that if you were to change the .pyc file it would not trigger a reload, but in the common case a .py file is changed and only if it's reloaded will a new .pyc be written. I suppose if you are vendoring .pyc files and you want to update them, or shipping pre-bytecode-compiled packages and wanting a reload when they're updated, this doesn't serve your use right now. We can open an issue specifically for that. I'm not immediately sure the best way to address it. What you say about watching directories might make sense.

• If your editor uses memory mapping to modify the file, it's not detected? Why not? When the editor saves, shouldn't the file be changed?

• I think we need to start the reloader after loading the wsgi entry point, or we might miss directories. Again, I think your recursive directory approach sounds okay, too, though.

[Code0x58]

• the regex that converts the name will change sampyo to sampy as well (module file names don't require a .py suffix, and the regex doesn't check for \. ). Admittedly very unlikely, especially with .pyo generated for imported modules. There would also be the issue with pre-compiled byte code that you mentioned too
• see the limitations of inotify under notes on the man page. I think this may also be brought up in the python module's documentation about editors
• I agree. I am not sure if changing the order will lead to issues with user/gunicorn code being executed in a different order and causing issues for some users upgrading

[Code0x58]

I just thought of another possible reason which would explain it not working as expected: the reload works by killing the worker processes and then letting the arbiter fork new workers, so any modules loaded pre-fork won't be reloaded by the python interpreter.

If this is happening then you'll see in the logs that workers are firing up for requests after a change, but the app itself hasn't changed/reloaded.

[tilgovi]

The docs for reload say that it's incompatible with preloading.
http://docs.gunicorn.org/en/latest/settings.html#reload

I think we do not cache any modules imported while loading a Python config file, so that should not break reloading either.

[tilgovi]

Maybe if the user imports application code for the arbiter hooks, that could break reloading.

[javabrett]

Running gunicorn 19.7.1, I have an application which imports a Flask app from webapp.py:

from webapp import app

...

class StandaloneApplication(gunicorn.app.base.BaseApplication):

...

    def __init__(self, app, options=None):
        self.options = options or {}
        self.application = app
        super(StandaloneApplication, self).__init__()

    def load_config(self):
        config = dict([(key, value) for key, value in iteritems(self.options)
                       if key in self.cfg.settings and value is not None])

        for key, value in iteritems(config):
            self.cfg.set(key.lower(), value)

    def load(self):
        return self.application

if __name__ == '__main__':
    options = {
        'bind': ...,
        'workers': 8,
        'accesslog': '/....',
        'errorlog': '/....',
        'capture_output': True,
        'loglevel': 'debug',
        'reload': True,
    }
    # make Flask propagate exceptions in middleware back to us for logging
    app.config['PROPAGATE_EXCEPTIONS'] = True
    StandaloneApplication(app, options).run()

I don't believe I have inotify installed and am probably polling with reload-mode auto.

When I modify source-code, I see this in the logs:

[2018-02-14 03:01:22 +0000] [534] [INFO] Worker reloading: /...../health_check.py modified
[2018-02-14 03:01:22 +0000] [531] [INFO] Worker reloading: /...../health_check.py modified
...
[2018-02-14 03:01:26 +0000] [529] [INFO] Worker exiting (pid: 529)
[2018-02-14 03:01:26 +0000] [527] [INFO] Worker exiting (pid: 527)
[2018-02-14 03:01:27 +0000] [899] [INFO] Booting worker with pid: 899
[2018-02-14 03:01:27 +0000] [29546] [DEBUG] 8 workers
[2018-02-14 03:01:27 +0000] [901] [INFO] Booting worker with pid: 901

This was my second attempt to get the health-check code to return a different dict - I removed a variable and replaced it with a string-constant. A GET serviced by health_check.py still returns the old values.

Any idea why Gunicorn is able to detect the source-change, but nothing changes in the response (other than a timestamp i.e. the response is not cached).

[tilgovi]

You would need to delay the import until the load() call.

def load(self):
    from wsgiapp import app
    return app

With the way you have it, the webapp code is imported by the Gunicorn arbiter at startup. Even when workers are killed and the arbiter forks new workers the old code is already imported and in the Python module cache. If you defer the import until load() it will be loaded by each of the workers after they fork.

[berkerpeksag]

Perhaps we could improve the example at http://docs.gunicorn.org/en/stable/custom.html or add a note about how to import the WSGI app?

[javabrett]

@tilgovi thanks, indeed with the late import reloading works perfectly.

I wonder if there is any way to detect this sort of error when reload is true? Otherwise yes @berkerpeksag updated example code would also be useful.

Anyway, this works for me now.

[billzhong]

I have the exactly same issue @javabrett mentioned. Thanks @tilgovi for the solution.
An updated document or example will be great as @berkerpeksag suggested.

[qcho]

Hi,

I think I'm having the same issue. The difference I'm not using any "Custom App" not I have gunicorn dependency on my proyect. My idea is to provide a WSGI app and then in production mounit it over gunicorn.

Also the idea is to have similar dev environment thats why i'm using --reload option setting it up in my gunicorn config file. In docker the only difference between prod and dev are some debugging params ^_^

Is there any way to define the "load" function in the GUNICORN Conf file so it tells future workers to perform the reload later?

It seems the process /usr/local/bin/python /usr/local/bin/gunicorn is caching the code.

I too receive the notifications properly:

app_1  | [2019-04-07 18:33:09 +0000] [18] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:09 +0000] [20] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:09 +0000] [26] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:09 +0000] [30] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:09 +0000] [10] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:09 +0000] [12] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:09 +0000] [14] [INFO] Worker reloading: /app/apis/auth.py modified
app_1  | [2019-04-07 18:33:10 +0000] [16] [INFO] Worker reloading: /app/apis/auth.py modified

This is what i have running:

root         7  0.2  1.1 104588 23148 ?        S    18:32   0:00 /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        10  0.5  1.8 367552 37436 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        12  0.5  1.8 367296 37408 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        14  0.5  1.8 367296 37412 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        16  0.6  1.8 367204 37416 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        18  0.5  1.8 367200 37200 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        20  0.5  1.8 367296 37204 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        26  0.5  1.8 367296 37204 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app
root        30  0.5  1.8 367208 37324 ?        Sl   18:32   0:00  \_ /usr/local/bin/python /usr/local/bin/gunicorn -k egg:meinheld#gunicorn_worker -c /app/gunicorn_conf.py main:app

RUN INFO:

 ❯ docker-compose up                                                                                                                                             [15:32:36]
Recreating nerd_app_1 ... done
Attaching to nerd_app_1
app_1  | Checking for script in /app/prestart.sh
app_1  | There is no script /app/prestart.sh
app_1  | [2019-04-07 18:32:40 +0000] [7] [DEBUG] Current configuration:
app_1  |   config: /app/gunicorn_conf.py
app_1  |   bind: ['0.0.0.0:80']
app_1  |   backlog: 2048
app_1  |   workers: 8
app_1  |   worker_class: egg:meinheld#gunicorn_worker
app_1  |   threads: 1
app_1  |   worker_connections: 1000
app_1  |   max_requests: 0
app_1  |   max_requests_jitter: 0
app_1  |   timeout: 30
app_1  |   graceful_timeout: 30
app_1  |   keepalive: 120
app_1  |   limit_request_line: 4094
app_1  |   limit_request_fields: 100
app_1  |   limit_request_field_size: 8190
app_1  |   reload: True
app_1  |   reload_engine: auto
app_1  |   reload_extra_files: []
app_1  |   spew: False
app_1  |   check_config: False
app_1  |   preload_app: False
app_1  |   sendfile: None
app_1  |   reuse_port: False
app_1  |   chdir: /app
app_1  |   daemon: False
app_1  |   raw_env: []
app_1  |   pidfile: None
app_1  |   worker_tmp_dir: None
app_1  |   user: 0
app_1  |   group: 0
app_1  |   umask: 0
app_1  |   initgroups: False
app_1  |   tmp_upload_dir: None
app_1  |   secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
app_1  |   forwarded_allow_ips: ['127.0.0.1']
app_1  |   accesslog: None
app_1  |   disable_redirect_access_to_syslog: False
app_1  |   access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
app_1  |   errorlog: -
app_1  |   loglevel: debug
app_1  |   capture_output: False
app_1  |   logger_class: gunicorn.glogging.Logger
app_1  |   logconfig: None
app_1  |   logconfig_dict: {}
app_1  |   syslog_addr: udp://localhost:514
app_1  |   syslog: False
app_1  |   syslog_prefix: None
app_1  |   syslog_facility: user
app_1  |   enable_stdio_inheritance: False
app_1  |   statsd_host: None
app_1  |   statsd_prefix:
app_1  |   proc_name: None
app_1  |   default_proc_name: main:app
app_1  |   pythonpath: None
app_1  |   paste: None
app_1  |   on_starting: <function OnStarting.on_starting at 0x7f01a395d268>
app_1  |   on_reload: <function OnReload.on_reload at 0x7f01a395d378>
app_1  |   when_ready: <function WhenReady.when_ready at 0x7f01a395d488>
app_1  |   pre_fork: <function Prefork.pre_fork at 0x7f01a395d598>
app_1  |   post_fork: <function Postfork.post_fork at 0x7f01a395d6a8>
app_1  |   post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f01a395d7b8>
app_1  |   worker_int: <function worker_int at 0x7f01a3971c80>
app_1  |   worker_abort: <function WorkerAbort.worker_abort at 0x7f01a395d9d8>
app_1  |   pre_exec: <function PreExec.pre_exec at 0x7f01a395dae8>
app_1  |   pre_request: <function PreRequest.pre_request at 0x7f01a395dbf8>
app_1  |   post_request: <function PostRequest.post_request at 0x7f01a395dc80>
app_1  |   child_exit: <function ChildExit.child_exit at 0x7f01a395dd90>
app_1  |   worker_exit: <function WorkerExit.worker_exit at 0x7f01a395dea0>
app_1  |   nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f01a3971048>
app_1  |   on_exit: <function OnExit.on_exit at 0x7f01a3971158>
app_1  |   proxy_protocol: False
app_1  |   proxy_allow_ips: ['127.0.0.1']
app_1  |   keyfile: None
app_1  |   certfile: None
app_1  |   ssl_version: 2
app_1  |   cert_reqs: 0
app_1  |   ca_certs: None
app_1  |   suppress_ragged_eofs: True
app_1  |   do_handshake_on_connect: False
app_1  |   ciphers: TLSv1
app_1  |   raw_paste_global_conf: []
app_1  | [2019-04-07 18:32:40 +0000] [7] [INFO] Starting gunicorn 19.9.0
app_1  | [2019-04-07 18:32:40 +0000] [7] [DEBUG] Arbiter booted
app_1  | [2019-04-07 18:32:40 +0000] [7] [INFO] Listening at: http://0.0.0.0:80 (7)
app_1  | [2019-04-07 18:32:40 +0000] [7] [INFO] Using worker: egg:meinheld#gunicorn_worker
app_1  | [2019-04-07 18:32:40 +0000] [10] [INFO] Booting worker with pid: 10
app_1  | [2019-04-07 18:32:40 +0000] [12] [INFO] Booting worker with pid: 12
app_1  | [2019-04-07 18:32:40 +0000] [14] [INFO] Booting worker with pid: 14
app_1  | [2019-04-07 18:32:40 +0000] [16] [INFO] Booting worker with pid: 16
app_1  | [2019-04-07 18:32:40 +0000] [18] [INFO] Booting worker with pid: 18
app_1  | [2019-04-07 18:32:40 +0000] [20] [INFO] Booting worker with pid: 20
app_1  | [2019-04-07 18:32:40 +0000] [26] [INFO] Booting worker with pid: 26
app_1  | [2019-04-07 18:32:40 +0000] [30] [INFO] Booting worker with pid: 30