Dagster Nomad

This is an example Dagster project with the dagster-nomad integration.

Setup

1. Install a Nomad cluster following the Nomad documentation.

2. Install Terraform following the Terraform documentation.

3. Install just for launching commands Just documentation.

4. Install Dagster and dagster-nomad in a python virtual environment:

   python3.11 -m venv .venv
   just setup-dev

5. Install Docker compose

Run example

Build an image for the RunLauncher to use and adjust the tag in tf/dagster-executor.hcl

docker build . --tag dagster-nomad-example:local

In a terminal run the following

just docker-up
just nomad-up

Create a bucket dagster in minio In another terminal run the following.

just terraform-setup
just dagster-dev

Open in your browser dagster-webserver and nomad-ui

Multiple Code Locations example

Following the convention for the DockerRunLauncher and K8sRunLauncher, the DAGSTER_CURRENT_IMAGE environment variable can used to specify the Docker image for the NomadRunLauncher to use.

Build the different images for your code locations.

docker build . --tag dagster-nomad-example:local

Launch the multiple code locations:

DAGSTER_CURRENT_IMAGE=dagster-nomad-example:local  just dagster-grpc

Uncomment the load_from.grpc_server in the workspace.yaml files

just dagster-dev