Running Dagster as a Service in Windows

[yuhan]

Credits

This note was written by @jragbeer about running Dagster as a service in Windows. Thanks for the contribution!

Guide

Suppose you’re running on Windows 10 and want to run Dagster continuously with a bit more resiliency.

By running the dagster-daemon run and dagit commands in the command prompt (cmd.exe), you can get going with Dagster and enjoy all of its features. If those windows close, Dagster no longer runs and your neither do your scheduled processes. If there’s an unexpected restart event, Dagster doesn’t restart.

The way to ensure that your Dagster process runs regardless of open windows or restarts is run it as a service. Using NSSM, that is easy to do. Following these steps, I was able to get Dagster running as a service, which would survive unexpected restarts and logouts. I’ve only tested this method with Windows 10, but I’ve confirmed it’s worked well on 3 different PCs.

1. Download the latest version of Dagster and Dagit from pip by typing pip install dagster dagit in a Command Prompt (with admin privileges).
2. Download NSSM from https://nssm.cc/download. I chose follow the warning on the first line and downloaded prelease build 2.24-101.
3. Once downloaded, put the file found inside the Win64 folder from the nssm-2.24-101-xxxx.zip file (nssm.exe) into your DAGSTER_HOME directory.
4. In the DAGSTER_HOME folder, create a .bat file named “dagster_service.bat”.

• Include in the file the following 4 lines:

  @echo off
  cd *DAGSTER_HOME*  - full path to the DAGSTER_HOME folder
  dagster-daemon run
  pause
  

5. Create another .bat file named "dagit_service.bat" in the DAGSTER_HOME folder.

• Include in the file the following 4 lines:

  @echo off
  cd *DAGSTER_HOME*  - full path to the DAGSTER_HOME folder
  dagit 
  pause
  

6. Once in the DAGSTER_HOME folder, open a Command Prompt (with Admin privileges) and type nssm install dagster_service – dagster_service in this case is the name of the service you’ll see in the Services app in Windows.
7. This brings up a GUI that you can use to create the service. On the first tab (Application) click on the browse (…) button beside Path. This allows you to select a file in the pop-up window, select "dagster_service.bat" inside the DAGSTER_HOME folder.
8. In the Details tab, fill in the Display Name and Description boxes. These will be displayed in the Services app in Windows. Leave Startup type on "Automatic".
9. At this point, you can press the "Install service" button near the bottom of the GUI.
10. Repeat steps 5-8. Replace nssm install dagster_service with nssm install dagit_service and "dagster_service.bat" with "dagit_service.bat".
11. Open Task Manager and then to the Services tab. Find "dagster_service" and "dagit_service". Right-click on "dagster_service" and select "Start". After it has been in the Running Status for a second or two, right-click on "dagit_service" and select "Start".
12. Navigate to 127.0.0.1:3000 in your browser and you should see Dagit running.

[kennyb-qb]

when you want to deploy a new version of the code base (like added a new job or changed existing job configuration), do you need to re-install?

[jragbeer]

I've not found that you've needed to. Simply refresh using the "Reload All" button on the Workspace page.
When issues arise (I haven't had any across multiple deployments so far, but I did during testing) I have restarted the service via Task Manager. Last case scenario would be to uninstall/re-install.

[Alexisbiancotti]

Hi!

Have you made additional steps to use python in the cmd? if yes, which ones? I get this error following from nssm with the steps that you specify:

"'dagit' is not recognized as an internal or external command, operable program or batch file."

I already made the steps that I find in another generic article https://qr.ae/pveLQ0 and if I run the bat it works just fine (the problem is with nssm)

[kevorkkeheian]

Try to change the logon user in the service to an administrator user. It worked with me

[sashan90s]

How do I locate DAGSTER_HOME folder?

[BeigeOne]

to follow this guide in 2024 you need to know that dagit is now known as dagster-webserver

#14873

Also worth noting that using nssm (which acts as a permanent wrapper for the .bat) avoids windows “Error 1053 the service did not respond to the start or control request in a timely fashion”

[dfish8124]

Hello,

I have dagster running as a service out of my DAGSTER_HOME folder, how do I configure the service to use a dagster project not in the DAGSTER_HOME folder?

[BeigeOne]

I think it wil just use whichever project is in your current working directory, so go to that directory and then execute dagster whilst in there. Does that help?

[dfish8124]

I want the bat files to live in my DAGSTER_HOME directory and dagster projects to live outside of that directory. When I run the bat file to start the daemon it is using the pyrpoject.toml in DAGSTER_HOME to find modules. I have tried to change pyproject.toml in DAGSTER_HOME to point to other directories but I can't get it working.

[BeigeOne]

I have it working in a similarish way. I can try to help. Do you need the pyproject.toml in DAGSTER_HOME? Perhaps that is taking priority over the one in current working directiry. I only have dagster.yaml in DAGSTER_HOME (I have the bat files one level higher. I can't really experiment on my set-up atm.

[dfish8124]

Thanks for helping!

I get the error "Error: No arguments given and no [tool.dagster] block in pyproject.toml found" when running the bat in DAGSTER_HOME without a pyproject.toml file. If I run that bat from my working directory I get the same error:

[dfish8124]

I have solved my issue. I updated the bat files to cd to the working directory instead of DAGSTER_HOME and things are working now. Thanks for you help.

[Marcodenijs]

In case people can't get it to work because they try to run it in a virtual environment try this:

@echo off
cd *DAGSTER_HOME*  - full path to the DAGSTER_HOME folder
call venv\scripts\activate
dagster-daemon run
pause

You have to put call in front of the venv activation. When you run venv\scripts\activate in a batch script, it is run in a new command shell, and once the activation is done, the shell exits. This means the environment changes (like activating the virtual environment) do not persist in the original command shell where your batch script is running.

[DominikRoB]

Is there a specific reason to use NSSM instead of a service configured via pywin32?
NSSM seems to be abandonded - so either it'll become a security hazard or stop working eventually on new windows versions.

[KarMishRMS]

any updates on this? can we get some help with creating dagster service via pywin32?