Self hosted Dagster performance

[marekkiraly-intellis]

Hi, I am currenlty finddling with self hosted dagster performance. I have been trying to setup my docker-compose.yml in a way, that distributes the resources between dagster-api, dagster-db, dagster-daemon, dagster-webserver and other services. I am interested in understadning - Which service is the one, that actually runs the jobs ? I would like to give it the most amount of RAM to ensure smooth runs, but I am not sure whethet to go for the dagster-daemon ro dagster-api. Thanks for any tips on this

[zyd14]

It really depends on what run launcher you're using, as that's the component that determines how a run is created / launched. If you're using the DefaultRunLauncher then the jobs are executing within the code location server, which I suspect is what your dagster-api entity is. If you're using the DockerRunLauncher (which is what I'd suggest you use if you're not already) then none of these resources is carrying out the job because a new container is being spun up for your job, the CPU / memory resources for which can be configured in the DockerRunLauncher config.

You might find this doc helpful to better understand the role of the different components