[Hiring] Windmill Expert – Need Help Eliminating Cold Starts in Multi-Step Crawler Flow

[MichaelBallas]

We run a large-scale multi-step crawler inside Windmill (~14 steps/flow, thousand jobs per hour), but are bottlenecked by cold starts and job fragmentation. Looking to hire an expert to help refactor this to persistent worker execution with batch orchestration. Budget is available. DM or respond here if you’re experienced with scaling Windmill or similar infra.

[rubenfiszel]

Hi Michael,

In our roadmap, we plan to be able to eliminate most cold start by extending dedicated workers to be dedicated to a full workspace instead of just a specific script or flows. Timeline is a few months.

In the meantime, feel free to open discussions with some specifics and you might get some good answers. Some of what you want to understand first is where the cold start come from:

• loading imports
• running chromium driver
• etc
  Based on the answers, we may be able to give some directions to focus on