Reload code location not working

[kennycontreras]

Hello community 👋

I'd like some insights into what could be wrong with my local Dagster setup. When I try to reload the code location, it doesn't work, even though I can see in the logs that the modules are being loaded again.

I have a local docker image running Dagster.

Before I was using the @repository decorator, the reload always worked fine. Recently, I migrated the code to use Definitions, and the issue started.

Here are some of my files:

Dockerfile

FROM base_dagster_image

RUN mkdir -p /opt/dagster/app
WORKDIR /opt/dagster/app

COPY . .
RUN pip install -r requirements.txt

EXPOSE 4000

CMD ["dagster", "api", "grpc", "-h", "0.0.0.0", "-p", "4000", "-w", "workspace.yaml"]
#CMD ["dagster", "code-server", "start", "-h", "0.0.0.0", "-p", "4000", "-w", "workspace.yaml"]

I have tried with code-server and grpc

workspace.yaml

load_from:
  - python_file:
      relative_path: "definitions.py"
      attribute: "batch"
      location_name: "batch"

docker-compose.yml

version: '3.4'
services:
  dagster:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "3000:3000"
    command: >
      bash -c "dagster-webserver -h 0.0.0.0 -w /opt/dagster/app/workspace.yaml"
    working_dir: /opt/dagster/app/
    volumes:
      - ./:/opt/dagster/app
    environment:
      - DAGSTER_HOME=/tmp

Based on the logs, I know that my definitions.py is reloading the modules, but even then, the actual code is not the one with the latest changes. For example, If one pipeline has an error that I fixed. I click reload, but the error is still there.

The reloading string there is coming from the definitions.py implementation.

definitions.py

from dagster import Definitions, AssetsDefinition, ScheduleDefinition, SensorDefinition, JobDefinition
from dagster._core.definitions.partitioned_schedule import UnresolvedPartitionedAssetScheduleDefinition
from common.utils import recursively_find_instances

dagster_assets = recursively_find_instances(root="batch", types=(AssetsDefinition,))
dagster_schedules = recursively_find_instances(root="batch", types=(ScheduleDefinition,
                                                                    UnresolvedPartitionedAssetScheduleDefinition))
dagster_sensors = recursively_find_instances(root="batch", types=(SensorDefinition,))
dagster_jobs = recursively_find_instances(root="batch", types=(JobDefinition,))

batch = Definitions(
    assets=dagster_assets,
    schedules=dagster_schedules,
    sensors=dagster_sensors,
    jobs=dagster_jobs
    # resources={...}
)

Update:

To provide more context, as I continue debugging what may be the issue. The hot reloading for the container is working.
If I make some local changes, they are also changed in the container. I double-checked this by getting inside the container and doing a simple cat to the files. Even then, Dagster doesn't acknowledge the changes with the reload button. Could it be some catch when the importlib for modules happens?

Any insights or recommendations would be really appreciated.

Thanks!

[kennycontreras]

Edit: Fixed the problem with importlib.reload(module_name) and it's working perfectly.

[C-Loftus]

@kennycontreras Could you clarify where you put that line importlib.reload(module_name)? I am also having issues with this and am wondering if I have the same issue.

i.e. if I have code that throws an error on it being loaded, I can remove that reload the module and the error won't be there, but at runtime it still appears; the new code appears to not be reloaded in some parts of the dagster runtime at least

Also see that you have the following commented out. Any reason for that? I thought you need for reloading: #14709

#CMD ["dagster", "code-server", "start", "-h", "0.0.0.0", "-p", "4000", "-w", "workspace.yaml"]

[jessebrowning]

Hello friends. @C-Loftus did you end up figuring this out? @kennycontreras I've looked through your repos to see if I could find your implementation but didn't have any luck.

[C-Loftus]

I have generally changed my workflow to always use dagster dev outside of the container when developing. I generally find that developing inside of a container has lots of random bugs that I was never able to resolve and that dagster dev is really the intended tool for developing. If you want an example I have all my code here publicly https://github.com/internetofwater/scheduler

[jessebrowning]

@C-Loftus Appreciate the reply. Moving to that as well. Thanks for the repo!

[kennycontreras]

I'm sorry all! I totally missed the notifications, but here's my utility module that helps me to reload the jobs, sensors, etc. I hope it's helpful for you.

def _find_instance_in_module(file: str, types: List[type]) -> Dict[str, object]:
    if file.endswith(".py"):
        module_name = file[:-3].replace("/", ".")
        module = importlib.import_module(module_name)
        importlib.reload(module)
        return {k: v for k, v in module.__dict__.items() if not k.startswith("__") and type(v) in types}


def recursively_find_instances(root: str, types: Sequence[type]) -> List[object]:
    modules = list()

    def recurse(path: str):
        if os.path.isfile(path):
            instances = _find_instance_in_module(path, types)
            if instances:
                for instance in instances.values():
                    modules.append(instance)
            return
        else:
            for child in os.listdir(path):
                if not child.startswith("__"):
                    recurse(f"{path}/{child}")

    recurse(root)
    return modules

[brodarnikola]

Can you please @kennycontreras link here a simple github project for this auto reload inside docker ?

I tried your solution, but it does not want to work

[brodarnikola]

@C-Loftus @jessebrowning @kennycontreras

I checked your code @C-Loftus ..

Inside entrypoint.sh, you are using this code

" set -x

if [ "$DAGSTER_DEBUG" = "true" ]; then
echo "Starting dagster in debug mode and waiting for connection to debugpy"
exec python -m debugpy --configure-subProcess true --listen 0.0.0.0:5678 -m dagster dev -h 0.0.0.0 -p 3000 -m userCode.main
else
echo "Starting dagster code server"
exec dagster code-server start -h 0.0.0.0 -p 4000 -m userCode.main
fi " ,,

to refresh code, bussinnes logic inside Dagster.

If you are in Debug mode, then you are using "dagster dev" ..

My use case is that Docker container should always run .. It would not be good to restart Docker container to apply, accept code change, busines logic

When I use this " dagster code-server start -h 0.0.0.0 -p 4000 -m userCode.main " ,, then I always need first to restart docker container, then I need to start again docker container to see the change in code, in busines logic.

Also checked the files with "cat" inside running docker container .. The code change, business logic is visible, applied inside this file, but for some reason it does not to refresh, change without restart of the docker container.

[C-Loftus]

The code you are referencing above is moreso for me to attach a debugger into the user code container.

Things certainly may have changed since I last worked on this but my belief at the moment is:

• If you want hot reloading for development you should use dagster dev outside of the container on the host.
• If you want reloading in a production environment, it probably makes sense to rebuild and reload a new container
  • It would be nice for reloading in a production an environment to be an option, but from what I saw when I last investigated, I don't believe it is a possibility How can I reload definitions within the Dagster UI without needing to redeploy my code? #14709 (comment) without adding with custom importlib stuff
    • Regardless, it is probably advisable for a container to be immutable in production and dynamically reloading Python individual modules in a production environment could be a security liability.
  • If you need high availability where at least one user code container is always running in production, you can accomplish this by using a second code location or using a container orchestrator like k8s.
  • In my opinion if you are looking to make frequent changes to a running container, that is probably a sign that it should be done in development mode and not in your production environment to begin with.