Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 30 additions & 2 deletions .github/workflows/config/.secrets.baseline

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions ATTRIBUTIONS.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,7 @@ All third-party software is used as obtained, without modification, unless other
| **MCP Python SDK** | Model Context Protocol SDK | MIT | Anthropic, PBC | https://github.com/modelcontextprotocol/python-sdk |
| **OmegaConf** | Config library | BSD-3-Clause | Omry Yadan | https://github.com/omry/omegaconf |
| **OpenAI Python** | API client | Apache-2.0 | OpenAI | https://github.com/openai/openai-python |
| **PinchBench** | Agentic benchmark (cloned & patched at image build) | MIT | PinchBench | https://github.com/pinchbench/skill |
| **spaCy Model (en_core_web_sm)** | NLP embedding model | MIT | Explosion AI | https://github.com/explosion/spacy-models/releases/tag/en_core_web_sm-3.8.0 |
| **Uvicorn** | ASGI server | BSD-3-Clause | Encode OSS Ltd. | https://github.com/encode/uvicorn |
| **uvloop** | Async event loop | Apache-2.0 OR MIT | MagicStack Inc. | https://github.com/MagicStack/uvloop |
Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -283,6 +283,7 @@ The Dataset column links to publicly available datasets (e.g., on HuggingFace).
| Openenv | games | Maze navigation environment via OpenEnv. Agent navigates an 8x8 grid to find the exit. | - | - | - | - | <a href='resources_servers/openenv/configs/openenv_maze.yaml'>openenv_maze.yaml</a> | - |
| Over Refusal Detection | | - | - | ✓ | - | TBD | <a href='resources_servers/over_refusal_detection/configs/over_refusal_detection.yaml'>over_refusal_detection.yaml</a> | - |
| Physics Judge | math | Physics QA verified by NeMo Skills' physics judge LLM with math-verify symbolic fallback | Score open-ended physics benchmarks (e.g. PHYSICS) where the judge emits [Correct] / [Incorrect] verdicts | - | - | - | <a href='resources_servers/physics_judge/configs/physics_judge.yaml'>physics_judge.yaml</a> | - |
| Pinchbench | | - | - | - | - | - | <a href='responses_api_agents/pinchbench/configs/pinchbench.yaml'>pinchbench.yaml</a> | - |
| Polymath | math | PolyMath multilingual math benchmark with weighted (difficulty) and per-language metrics | Improve multilingual math reasoning across 18 languages and 4 difficulty tiers | - | - | - | <a href='resources_servers/polymath/configs/polymath.yaml'>polymath.yaml</a> | - |
| Proof Genselect | math | Pairwise proof selection with binary correctness reward | - | - | - | - | <a href='resources_servers/proof_genselect/configs/proof_genselect.yaml'>proof_genselect.yaml</a> | - |
| Proof Judge | math | Theorem proving with verifier + meta-verifier judge (combined env) | - | - | - | - | <a href='resources_servers/proof_judge/configs/proof_judge.yaml'>proof_judge.yaml</a> | - |
Expand Down
4 changes: 4 additions & 0 deletions responses_api_agents/pinchbench/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
.venv/
__pycache__/
*.pyc
results/
50 changes: 50 additions & 0 deletions responses_api_agents/pinchbench/Dockerfile.benchmark
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
# PinchBench runner image: Node 22 + OpenClaw + python/uv.
#
# The PinchBench skill is NOT vendored in this repo. At build time we clone it at a
# pinned tag and apply the NVIDIA integration patch (custom OpenAI-compatible endpoint,
# JUDGE_BASE_URL/JUDGE_API_KEY judge routing, transcript resolution) — mirroring how
# harbor_agent / mini_swe_agent pin a framework commit instead of vendoring. run_task.sh
# (our per-task entrypoint) is baked in at /opt/run_task.sh.
#
# Build context = responses_api_agents/pinchbench/ (see setup_scripts/build_image.sh).
FROM node:22-bookworm

LABEL org.opencontainers.image.source="https://github.com/pinchbench/skill"
LABEL org.opencontainers.image.description="PinchBench runner (OpenClaw + NVIDIA endpoint patch)"

RUN apt-get update && apt-get install -y \
python3 python3-pip python3-venv git curl jq \
&& rm -rf /var/lib/apt/lists/*

# uv (PinchBench runs benchmark.py via `uv run --with pyyaml`)
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
ENV PATH="/root/.local/bin:$PATH"

# OpenClaw + Brave web-search plugin (pinned). The explicit plugin link makes the
# runtime discover Brave as provider "brave" instead of falling back to SearXNG/Kimi.
RUN npm install -g openclaw@2026.6.5 @openclaw/brave-plugin@2026.6.5 \
&& (openclaw plugins install --link /usr/local/lib/node_modules/@openclaw/brave-plugin \
|| echo "WARN: brave plugin link failed; web search disabled")

# Clone the PinchBench skill at the validated tag, then apply the NVIDIA integration
# patch (see setup_scripts/nvidia-pinchbench.patch — touches scripts/lib_agent.py only).
ARG PINCHBENCH_SKILL_REF=v2.0.0
RUN git clone --depth 1 --branch "$PINCHBENCH_SKILL_REF" https://github.com/pinchbench/skill /opt/pinchbench-skill
COPY setup_scripts/nvidia-pinchbench.patch /tmp/nvidia-pinchbench.patch
RUN cd /opt/pinchbench-skill && git apply /tmp/nvidia-pinchbench.patch && echo "applied nvidia-pinchbench.patch"

# Sandbox isolation: benchmark.py hardcodes its run-root to /tmp/pinchbench, but under
# the apptainer provider /tmp is the *host's* dir SHARED across concurrent instances
# (only the per-sandbox /sandbox bind is isolated). Make the run-root honor
# $PINCHBENCH_RUN_ROOT so run_task.sh can point it at the isolated working mount.
RUN cd /opt/pinchbench-skill && \
sed -i 's#run_root = Path("/tmp/pinchbench")#run_root = Path(os.environ.get("PINCHBENCH_RUN_ROOT", "/tmp/pinchbench"))#' scripts/benchmark.py && \
sed -i 's#agent_workspace = Path(f"/tmp/pinchbench/{run_id}/agent_workspace")#agent_workspace = Path(f"{run_root}/{run_id}/agent_workspace")#' scripts/benchmark.py && \
grep -q 'PINCHBENCH_RUN_ROOT' scripts/benchmark.py

# Bake in our per-task entrypoint.
COPY run_task.sh /opt/run_task.sh

RUN mkdir -p /root/.openclaw
ENTRYPOINT ["/bin/bash", "-c"]
CMD ["openclaw --version && echo 'Ready for benchmarks'"]
21 changes: 21 additions & 0 deletions responses_api_agents/pinchbench/LICENSE
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
MIT License

Copyright (c) 2026 PinchBench

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
Loading
Loading