WorkerManager/Worker Terminology

[sharpener6]

• Controller: controller is top level component in scheduler, like WorkerController, TaskController, etc.

• Capabilities: dictionary mapping from capability string to numerical count of resources (no unit defined, it can means any unit), e.g.: {"cpu": 10, "memory": 1000}

• Capability Types: set of capability strings like {"cpu", "memory"}

• Worker (individual working unit that can accept queue of tasks), each worker have it's own capabilities

• Worker Type: a string the defines the worker type, this is isolated from capabilities, each worker manager will given a worker type when start, and when worker

• Worker Manager ID: a unique identify worker manager, e.g.: "aws_1"

• Worker Manager (Nanny process in charge of creating new workers)

  • when start worker manager, the user need provide the capabilities (include capability types and max allowed number of capabilities)
  • when start worker manager, the user need provide Worker Manager ID and Worker Type, so each worker started with will attach this worker type
  • in Scheduler Side, it has WorkerManagerController, each worker manager send worker manager heartbeat, WorkerManagerController will hold worker type to worker manager ids information

• NativeWorkerManager

  • --fixed mode:

    • no WorkerManager Loop created, no WorkerManager heartbeat to scheduler
    • we called this mode as dummy worker manager, it doesn't turn up/down workers
    • only do spawn fixed number of workers, passing worker type to newly created workers
    • join the worker processes
    • this mode is used when using ORB/kubernets WorkerManager
    • ORB/Kubernets worker manager need pass this command to create workers with dummy worker managers like:
      • "native_worker_manager --fixed 10 --worker-type "orb" tcp://x.x.x.x:1234"

  • WorkerManager mode:

    • WorkerManager loop created, and send heartbeat to scheduler
    • WorkerManager is responsible for the lifecycle of workers he spawned

Policy Engine Config Sample

policy_engine_type = "waterfall_v1"
policy_content = """
# priority, worker_manager_id, max_task_concurrency
         1,               wm1,                   10  # 10 concurrency for this worker manager
         1,               wm2,                   20  # 20 concurrency for this worker manager
         1,               wm3,                   40  # 40 concurrency for this worker manager
         1,               wm4,                   64  # 64 concurrency for this worker manager 
         1,               wm5,                 1000  # 1000 concurrency for this worker manager       
"""

waterfall v2

# each worker manager will send scheduler max number of concurrency,
# so each worker type will sum up number of max number of concurrency
policy_engine_type = "waterfall_v2"
policy_content = """
# priority,       worker_type 
         1,            native   # 1 worker manager with 10 max number of cucurrency, so 10 for this worker type
         1,           native1   # 2 worker managers with worker_type="native1" and 10 for each, total 20 workers * 1
         1,               sym   # 1 worker manager with 10 max number of cucurrency, 1 fake worker * 10 
         1,               aws   # 1 worker manager with 64 max number of cucurrency, 64 workers * 1
         1,             batch   # 1 worker manager with 1000 max number of cucurrency, 1 fake worker * 1000         
"""