You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
harbor: reward.json without a reward key is scored as the sum of its fields #5641
Found while running SkillEvaluator's Tier 3 task folders on Gym. Gym cc8991e4 (main, 2026-11-04), Docker sandbox.
Summary. If a Harbor verifier writes a multi-metric reward.json with no reward key, the trial's reward is the sum of every field. SkillEvaluator's verifier writes six 0–1 metrics plus overall, so its trials score up to about 7.
Expected. A way to name the field that is the score (here overall), so per-metric scores are kept as metrics and not added up.
Proposed fix. Opt-in reward_key in dataset.toml ([gym.defaults], overridable under [gym.tasks."<id>".verifier]), read by select_reward in environment_servers/harbor/verifier.py; a missing key is harbor:invalid_reward. The default (sum) is unchanged. Related: #2137.
Activity
feat(harbor): let a dataset name the reward.json key that is the score