Deploy to a process worker and run flow code with uv

[wigging]

I have some Python code as follows:

# runner.py

from prefect import flow

if __name__ == "__main__":
    flow.from_source(
        source="https://github.com/wigging/prefect-orc.git",
        entrypoint="02_deploy_local.py:my_flow",
    ).deploy(
        name="my-deploy-flow",
        work_pool_name="cpu-process"
    )

This code pulls down the GitHub repo to a worker that is in a process pool and runs the flow code that is located in the entrypoint Python script. The GitHub repo is setup as a uv project and contains a pyproject.toml file.

How is the flow code being executed on the worker? Is it using uv to call the Python script like uv run 02_deploy_local.py or is it using the Python command like python 02_deploy_local.py?

If the worker is using the Python command to run the script, then how do I tell it to use the uv run command? The worker has uv installed on it so I would like it to use uv to run the Python script.

[zzstoatzz]

hi @wigging - we have some friction to resolve here, but to answer your question

• to start the subprocess where the flow run executes, the worker runs the command from the work pool's base job template, that you can override on a per-deployment basis or on the work pool itself
• you can additionally modify the processes python environment via the pull step from within that subprocess, though as mentioned, this doesn't play as nice with uv today as it could

so there's a sequencing friction to smooth out, because you should be able to clone your pyproject.toml / uv.lock and use that to fully define the python environment used by the subprocess the worker creates, but today that's not possible without low-level customization of prefect (writing your own worker or similar that clones before the subprocess begins)

so the immediate workaround for a process worker is to override the command job variable to do something like uv run --with git+https://github.com/org/repo.git#subdirectory=path/to/project -m prefect.engine if you can, or otherwise use an appropriate --with flag

[zzstoatzz]

you'd modify the command job variable like I linked above except in the job_variables dictionary instead of yaml. the command is meant to invoke prefect.engine, not a specific file like 02_deploy_local.py

the point of using uv here would be to use the --with flag to install deps for the subprocess without polluting the parent worker process

Or is there a way to define this in the Prefect server UI so that all flows sent to this worker automatically use uv without having to define it in the Python file?

you'd set the command i've been mentioning on the work pool itself, so that all workers connected to that work pool would use it, but that only works if the command correctly installs the dependencies that each flow run would need

[wigging]

So that's similar to what I had earlier but it would include --with for grabbing the dependencies. But the project that is deployed already has a pyproject.toml file where the dependencies are listed. Therefore uv should automatically install those dependencies when uv run is invoked. I don't understand why --with would be needed here. I'll just run this when I get back to my work computer and see what happens. To be continued...

[zzstoatzz]

Therefore uv should automatically install those dependencies when uv run is invoked.

because as i mentioned there's a sequencing problem. your pyproject.toml / uv.lock are cloned during the pull steps that occur in the subprocess that starts via the command

i.e. uv run has no way to know what your deps are until you clone them to disk (or use --with), but the clone happens in the pull steps in the very subprocess that uv run would start, unless you use the --with flag to tell it to start the subprocess with the right deps

[wigging]

I tried this

job_variables={
    "command": "uv run --with numpy python -m prefect.engine"
}

but it still doesn't work. I'll just try a Docker worker because this isn't working well. And even if it did work, it's not utilizing the pyproject.toml file. If I have a lot of dependencies then it's not feasible to define all of them using the --with option. That's what the pyproject.toml file is for.

[zzstoatzz]

yea if you're using a process worker this is the workaround

the immediate workaround for a process worker is to override the command job variable to do something like uv run --with git+https://github.com/org/repo.git#subdirectory=path/to/project -m prefect.engine if you can

otherwise a containerized work pool type (like docker) is a good option as you mention