Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,3 +43,4 @@ harbor run -p harbor_cookbook/recipes/<name> -a claude-code -m anthropic/claude-
| [tinker&#8209;rl](harbor_cookbook/tinker_rl/) | RL training on Harbor tasks using the Tinker Cookbook integration. |
| [prime&#8209;rl](harbor_cookbook/prime_rl/) | RL training on Harbor tasks using Prime RL and Verifiers. |
| [sky&#8209;rl](harbor_cookbook/sky_rl/) | RL training on Harbor tasks using SkyRL. |
| [mosaic&#8209;rl](harbor_cookbook/mosaic_rl/) | Grouped RL rollouts on Mosaic Sandbox using Harbor's released Job API. |
81 changes: 81 additions & 0 deletions harbor_cookbook/mosaic_rl/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# mosaic-rl

Grouped Harbor rollouts on Mosaic Sandbox using the released Harbor 0.22
`Job`/`JobConfig` API.

## Prerequisites

```bash
pip install 'mosaic-sandbox[harbor]'
mos login
```

You also need a model provider key. Set `MOSAIC_API_TOKEN` instead of using
`mos login` if preferred. A `DAYTONA_API_KEY` is needed only for the optional
comparison below.

## Quick start

```bash
uv run harbor_cookbook/mosaic_rl/rollout.py \
--dataset terminal-bench@2.0 \
--group-size 4 \
--n-concurrent 16 \
--agent terminus-2 \
--model hosted_vllm/my-model
```

Use `--task ./path/to/task` instead of `--dataset`, repeating `--task` for
multiple local tasks. The script prints each task's `k` rewards and writes
grouped rollouts to `rollouts.json`. Token IDs and masks are required by
default; add `--reward-only` for evaluation-only runs.

## Why group rollouts on Mosaic?

A GRPO-style step samples `k` attempts for one task. Mosaic builds that task's
environment once into a content-addressed snapshot, then restores an isolated
sandbox for each attempt. The build cost is therefore amortised across the
group rather than paid once per attempt.

The script currently selects Mosaic through
`import_path="mosaic_sandbox.integrations.harbor:MosaicEnvironment"`, which
works with stock Harbor 0.22. Once the upstream provider lands, `--env mosaic`
will be available as a native Harbor environment selection.

This example intentionally targets the released Harbor `Job`/`JobConfig` API.
The sibling `harbor_rl` example targets the `harbor.rl` interface from open
upstream PR [#1291](https://github.com/harbor-framework/harbor/pull/1291);
the two examples use different APIs on purpose.

## Mosaic and Daytona comparison

The script accepts `--env mosaic`, `--env daytona`, or another Harbor
environment import path. Run both arms with the same task set, `k`, concurrency,
agent, and model:

```bash
uv run harbor_cookbook/mosaic_rl/rollout.py \
--dataset terminal-bench@2.0 \
--group-size 4 \
--n-concurrent 16 \
--agent terminus-2 \
--model hosted_vllm/my-model \
--env mosaic \
--jobs-dir jobs/mosaic

DAYTONA_API_KEY=... uv run harbor_cookbook/mosaic_rl/rollout.py \
--dataset terminal-bench@2.0 \
--group-size 4 \
--n-concurrent 16 \
--agent terminus-2 \
--model hosted_vllm/my-model \
--env daytona \
--jobs-dir jobs/daytona
```

Record wall-clock time, rollouts per second, and mean per-trial wall time from
each run. Separately record the first-attempt cost and the subsequent-attempt
cost: use a fresh snapshot/task for the first-attempt observation and inspect
the provider and trial timestamps for the grouped run so the one-time build
and per-attempt restore costs remain visible. Do not compare runs with
different task content, model, agent, `k`, or concurrency.
97 changes: 97 additions & 0 deletions harbor_cookbook/mosaic_rl/rollout.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,97 @@
# /// script
# dependencies = ["mosaic-sandbox[harbor]>=0.12.0"]
# requires-python = ">=3.12"
# ///
"""Generate grouped Harbor rollouts with Mosaic or another provider."""

from __future__ import annotations

import argparse
import asyncio
import json
import time
from pathlib import Path

from mosaic_sandbox.integrations.harbor_rl import (
MOSAIC_ENVIRONMENT_IMPORT_PATH,
generate_rollouts,
rollout_group_to_dict,
)


def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--dataset", help="Harbor dataset name, such as terminal-bench@2.0"
)
parser.add_argument(
"--task",
action="append",
type=Path,
help="Local Harbor task directory; repeat for multiple tasks",
)
parser.add_argument("--group-size", type=int, default=4)
parser.add_argument("--n-concurrent", type=int, default=4)
parser.add_argument("--agent", default="terminus-2")
parser.add_argument("--model", required=True)
parser.add_argument("--env", default="mosaic")
parser.add_argument("--jobs-dir", type=Path, default=Path("jobs"))
parser.add_argument("--output", type=Path, default=Path("rollouts.json"))
parser.add_argument(
"--reward-only",
action="store_true",
help="Skip token extraction for evaluation-only runs",
)
args = parser.parse_args()
if not args.dataset and not args.task:
parser.error("one of --dataset or --task is required")

environment_import_path = None
environment_type = None
if args.env == "mosaic":
environment_import_path = MOSAIC_ENVIRONMENT_IMPORT_PATH
elif args.env == "daytona":
environment_type = "daytona"
else:
environment_import_path = args.env

started = time.perf_counter()
groups = asyncio.run(
generate_rollouts(
args.task,
dataset=args.dataset,
agent=args.agent,
model=args.model,
group_size=args.group_size,
n_concurrent=args.n_concurrent,
jobs_dir=args.jobs_dir,
reward_only=args.reward_only,
environment_import_path=environment_import_path,
environment_type=environment_type,
)
)
elapsed = time.perf_counter() - started

args.output.write_text(
json.dumps([rollout_group_to_dict(group) for group in groups], indent=2) + "\n"
)
for group in groups:
print(f"{group.task_name}: {[attempt.reward for attempt in group.attempts]}")

n_rollouts = sum(len(group) for group in groups)
print(f"Rollouts: {n_rollouts}")
print(
f"Wall-clock: {elapsed:.3f}s\nRollouts/sec: {n_rollouts / elapsed:.3f}"
if elapsed
else "Wall-clock: 0.000s\nRollouts/sec: inf"
)
print(
f"Mean per-trial wall time: {elapsed / n_rollouts:.3f}s"
if n_rollouts
else "Mean per-trial wall time: n/a"
)
print(f"Wrote grouped rollouts to {args.output}")


if __name__ == "__main__":
main()