Parallel (Sub) workflows: you get to decide!

Whats the issue?

As you may recall: during the 'developers days' last September a group of us spent some time discussing what parallel (sub)workflows should look like and how they may be used. @ the Berkeley code sprint we were able to succesfully test a first pass of the feature. We are however still not sure whether this feature is worthwhile implementing!

We need your feedback in deciding whether to move forward with this feature as it is a significant investment in time and effort

Parallel workflows in a nutshell

Lots of people know the unofficial way of executing workflows in parallel. You basically use a workflow to spawn a large amount of other tasks or workflows that run independently from its parent. This approach has a number of drawbacks:

• Accounting: it's not easy to link sub workflows to the parent tasks. It's easy to create a gap in your accounting as you may not be able to see why a workflow was started.
• It's easy to cause queue saturation when generating large amounts of tasks
• No official documented api or pattern of implementing this "safely". It's up to you to implement it correctly
• Not possible to easily create compound workflows that execute in parallel.

Our POC implementation has solved the above issues

Key design choices made:

• Build parallelism based on Parallel StepLists
• Parallelism based on a iterator: prepare a list states and pass them to StepLists that get executed in parallel

l = [{"a": 1}, {"a": 2}, ...]
state = {"iterator": l }

@workflow("My For Each Parallel wf")
def foreachparallel_wf() -> StepList:
    """Assume `state` is the state in this workflow"""
    return (
      begin
      >> step_a 
      #  foreach_parallel(`name of the step`, `key in the state with iterator`, `StepList`)                  
      >> foreach_parallel("Name of my parallel step", "iterator", begin >> step_b >> step_c)
      end
    )

Diagram
Dynamic parallel (foreach_parallel()) — N branches from a runtime list:

    ┌──────────────────────┐
    │   sequential steps   │
    │  (produces items=[…])│
    └──────────┬───────────┘
               │  items = [item_0, item_1, …, item_N-1]
      ┌────────┼────────┐       FORK (dynamic — N branches)
      │        │        │
 ┌────▼──┐ ┌───▼──┐ ┌───▼──┐
 │item_0 │ │item_1│ │item_N│   PARALLEL (same branch template)
 │ seed  │ │ seed │ │ seed │   each seeded with its item
 └────┬──┘ └───┬──┘ └───┬──┘
      │        │        │       Results persisted in DB (not merged)
      └────────┼────────┘       JOIN (barrier — worst status wins)
               │
    ┌──────────▼───────────┐
    │   sequential steps   │    Receives pre-parallel state
    └──────────────────────┘

• Parallelism based on distinct StepLists enabling compound workflows.

from workflow_a import stepfunc_a
from workflow_b import stepfunc_b

@workflow("My Parallel wf")
def parallel_wf() -> StepList:
    """Assume `state` is the state in this workflow"""
    return (
      begin
      >> step_a 
      #  parallel(`name of the step`, StepList, StepList ...)                  
      >> parallel("Name of my parallel step", stepfunc_a(), stepfunc_b(), ...)
      end
    )

Diagram
Static parallel (parallel() / | operator) — fixed branches, defined at definition time:

    ┌──────────────────────┐
    │   sequential steps   │
    └──────────┬───────────┘
               │
         ┌─────┴─────┐          FORK (static — 2 branches)
         │           │
    ┌────▼────┐ ┌────▼────┐
    │branch 0 │ │branch 1 │     PARALLEL (each gets deep copy of state)
    │ step_a  │ │ step_c  │
    │ step_b  │ │ step_d  │
    └────┬────┘ └────┬────┘
         │           │          Results persisted in DB (not merged)
         └─────┬─────┘          JOIN (barrier — worst status wins)
               │
    ┌──────────▼───────────┐
    │   sequential steps   │    Receives pre-parallel state
    └──────────────────────┘

Dilemma

We think most people will think this is a neat feature, however will people use it? There are a few things we are concerned about.

• Major design and refactoring of the FE (can we make do with a minimal implementation?)
• Complex code that may only serve small user group or niche usecase
• Will people actually use the feature and not only think it's neat

What will we do with the outcome of this poll?

The results of the poll will help steer the technical committee in their choice whether to implement this feature in the API and how far we should go in supporting this feature in the frontend.

More information?

Check out the design and first implementation here: Here on GitHub

[jamesharr]

We're just getting started with WFO, but I think we've already identified at least a few different use-cases:

• When we do maintenance tasks on a given Port, we may want to make changes on all subscriptions on that port (IE: eBGP graceful shutdown)
• When we onboard new routers, we may need to setup the other side of iBGP sessions on other routers. In the case of a full-mesh, that's N other routers; in the case of a route-reflector design it might be 3 other routers.
• If we were to model a private L3VPN service as a single subscription (with multiple connections), we might want to take an action on each connection independently.
• If we're implementing a decoupled design with L3VPNs and cloud services (IE: AWS DirectConnect, Azure ExpressRoute, etc), we might have one subscription to deal with the L2VPN/L3VPN configuration and a separate subscriptions to interact with the cloud providers and provide SAPs for the L3VPN to use. In many cases, a change to an L3VPN service may trigger a change/workflow in the cloud subscription or vice/versa. In some cases, it can make sense to execute parts of workflows in parallel.