Eternal Durable function with multiple slots (multiple task hubs)

[dxynnez]

Hi team,

We have a use case where we need to pull things indefinitely (hence long running) and persist state periodically.

Originally this was implemented with multiple timer functions (pull from different sources). But because of the nature of timer function & the underlying singleton lock implementation, most of the timers are running on the same worker instance most of the time, and hence causing imbalance on our workers.

We are now looking into using eternal durable function to replace the heavy-lifting work of the timer to load-balance the workers. But we have a few questions that need clarification:

1. as this is eventually a durable function, the history is stored per taskhub. We use different taskHubs in different slots and during deployment (deploy to staging and swap to prod), looks like it's possible to have 2 eternal durable functions instances running on different slots even if we use the same instance Id (e.g., the old PROD is using taskhub A and is shutting down gracefully; and the new PROD is using a new taskhub and hence is unaware of any instanceId in the old taskhub and decides to enqueue & process the instance)?

2. would the ContinueAsNew actually complete the durable function instance, or it would just wipe out the history without completing the instance? To make sure we never lose the eternal durable function, we will still have a timer to periodically schedule the eternal durable using the same instanceId (if it exists already, this would then be a no-op), but if the timer runs and tries to start a new durable function around the same time as the ContinueAsNew of the existing running eternal durable, what would happen? Is there any race that I would end up with 2 eternal durables with the same instanceId, even in the same taskhub?

[jianawu]

I did some experiments to check the Question#2:
When I use eternal durable function and keeps triggering timer function to periodically schedule the durable function using the same InstanceId.

The eternal durable function will work for 30mins and the timer function triggered every 20s. In the timer function, it checks status of durable function and restart durable function with the same instance id if needed.

In the most of the case, I got:

Time	log	notes
2025-11-19 12:44:24.695	XXXOrchestratorFunction started	First line in durable function
2025-11-19 12:44:40.031	Start to trigger orchestrator	First line in timer function
2025-11-19 12:44:40.073	Orchestrator instance status for ID: Experiment_Instance is Completed	Check durable function status in timer function
2025-11-19 12:44:40.171	Successfully triggered xxxx orchestrator for ID: Experiment_ServiceInstance	In timer function
2025-11-19 12:44:48.928	XXXOrchestratorFunction started	First line in durable function
2025-11-19 12:44:48.959	Orchestrator Function continues to work	In durable function

It seems both timer and eternal triggered durable function successfully. but finally, only one orchestrator function is running.

In the experiment for around 12 hours, I got one different log like:

Time	log	notes
2025-11-19 12:13:05.969	XXXOrchestratorFunction started	First line in durable function
2025-11-19 12:13:20.000	Start to trigger orchestrator	First line in timer function
2025-11-19 12:13:20.011	Orchestrator instance status for ID: Experiment_ServiceInstance is Completed	Check durable function status in timer function
2025-11-19 12:13:20.044	Successfully triggered xxx orchestrator for ID: Experiment_ServiceInstance	In timer function
2025-11-19 12:13:30.751	XXXOrchestratorFunction started	First line in durable function
	Orchestrator starts to work	In durable function
2025-11-19 12:14:00.236	Start to trigger orchestrator	First line in timer function
2025-11-19 12:14:00.496	Orchestrator instance status for ID: Experiment_ServiceInstance is not existed	Check durable function status in timer function
2025-11-19 12:14:00.722	Successfully triggered orchestrator for ID: Experiment_ServiceInstance	In timer function
2025-11-19 12:14:24.309	Orchestrator seems restarted

It seems sometime, the durable function lost and the timer function did not found the target durable function with the instance id.

Based on the experiments, I also want to ask questions:

1. I feel like ContinueAsNew stopped the durable function and triggered it again. In this case, actually in a very short period of time, we lost the function. Am I right?
2. I also find that the timer will also trigger the eternal durable function. Although, in my experiments, the durable function runs correctly. (sometimes, it restarted unexpectedly like in my experiment sample). Could we confirm that it is safe to use a eternal durable function with a timer as a periodically scheduler to ensure the existence of the durable function?
3. Do we have any suggestions on how to trigger and how to maintain (keep checking and reschedule if needed) the existence of eternal durable functions?

[dxynnez]

Hi @cgillum ,

Is this something you can help to answer?

Due to the implementation limit, it's a bit hard for us to make the function processing truly idempotent (e.g., 'concurrent' executions for the same instanceId would cause problems). We understand that durable function doesn't guarantee exactly-once execution, but we do want to lower the chance of duplicate executions (concurrent & duplicate to be more specific) as much as possible. With multiple slots (multiple taskhubs), it sounds like duplicate executions is more likely to happen whenever there is a slot-swap & there are in-flight eternal durable functions. The behavior of ContinueAsNew also matters here as that also affect what kinds of 'singleton' guarantee we actually have within the same taskhub.

And as @jianawu mentioned, it's a bit unclear to us that what would be the recommended way to trigger an eternal durable as they are designed to run forever.