Repository navigation
feat(nooa): add the SWE-bench Pro session recipe #3845
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
44 commits
Select commit
Hold shift + click to select a range
31a7d72
fix(nooa): forward invocation correlation from the sandbox
init-nikhil e10a4ac
feat(nooa): persist partial sandbox checkpoints at execution boundaries
init-nikhil fa61ac0
fix(nooa): import the relocated process supervisor
init-nikhil c54b13d
fix(nooa): recover partial evidence after sandbox worker failure
init-nikhil 82090b9
refactor(nooa): use shared sandbox sessions
init-nikhil 7690229
test(nooa): remove unused collector test setup
init-nikhil 83dd1a7
fix(nooa): record capture gaps for unconfirmed cleanup
init-nikhil cdb801a
fix(nooa): recover checkpoints before optional diagnostics
init-nikhil 9d15dea
[wip] feat(nooa): wire sandbox execution into native sessions
init-nikhil 2233a26
fix(nooa): preserve sandbox session close evidence
init-nikhil 135041d
fix(nooa): remove the default SWE benchmark call limit
init-nikhil 09f21ee
chore(nooa): sync PR 3850 with current main
init-nikhil 5f82d13
chore(nooa): sync PR 3851 with its updated parent
init-nikhil a9a1672
chore(nooa): sync PR 3849 with its updated parent
init-nikhil 53196dc
chore(nooa): sync PR 3845 with its updated parent
init-nikhil 3c332f4
fix(nooa): distinguish partial and final worker evidence
init-nikhil 98f59e5
fix(nooa): retain sandbox capture failures and worker diagnostics
init-nikhil 0b4ad6b
chore(nooa): integrate sandbox supervision review fixes
init-nikhil d82a4d0
fix(nooa): keep sandbox session seeding responsive
init-nikhil 18d8d09
chore(nooa): carry entrypoint review fixes into supervision
init-nikhil bf7e086
test(nooa): verify SWE routing and correct setup guidance
init-nikhil ec52176
chore(nooa): carry entrypoint review fixes into sessions
init-nikhil 08399ab
docs(nooa): describe configurable worker supervision budget
init-nikhil 318bd29
chore(nooa): carry reviewed session fixes into SWE recipe
init-nikhil 096e5d1
chore(nooa): carry configurable budget wording into sessions
init-nikhil 9c06350
ch(nooa): carry budget wording into SWE recipe
init-nikhil dcc8bbd
test(nooa): assert close-race HTTP responses in lifecycle checks
init-nikhil d147360
chore(nooa): carry close-race regression updates into PR 3845
init-nikhil 307a969
fix(nooa): retain worker diagnostics after capture failures
init-nikhil c229e72
Merge commit '307a96916f3823cb95fcbbeb25506cf2b0262bab' into HEAD
init-nikhil 0b4b97b
Merge commit 'c229e723be82bffd40c4b44611f2a4eebfb1ccd3' into HEAD
init-nikhil 309e8d5
fix(nooa): drain ordered background checkpoint writes
init-nikhil 45f4649
Merge commit 'cac55a5fca93baf73e202770f71097b2d952f046' into HEAD
init-nikhil ee433d0
Merge commit '45f4649854fedd039fe64d1ad8d9d6d6d632abcf' into HEAD
init-nikhil 3d8c38e
Merge commit 'ee433d0914fc8b94ad81636897819f1b55765899' into HEAD
init-nikhil 843b65a
Merge commit '3d8c38e17' into HEAD
init-nikhil 4d2d3e6
chore(nooa): merge current main into sandbox sessions
init-nikhil c847a73
Merge updated sandbox sessions into SWE-bench Pro recipe
init-nikhil 63e6886
docs(nooa): clarify worker and episode timeout boundaries
init-nikhil 46ab918
Merge clarified NOOA timeout documentation into SWE-bench Pro recipe
init-nikhil a7a6ba5
fix(nooa): extend SWE-bench Pro sandbox TTL beyond run budgets
init-nikhil a174558
test(nooa): use supervisor filename in session fixture
init-nikhil c570b13
chore(stack): merge updated parent into #3845
init-nikhil fa9fd09
chore(stack): merge main into SWE-bench Pro recipe
init-nikhil File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
| config_paths: | ||
| - resources_servers/swebench_pro/configs/swebench_pro.yaml | ||
| - responses_api_agents/nooa_agent/configs/nooa_agent.yaml | ||
| - environment_servers/single_agent_turn_legacy/configs/single_agent_turn_legacy.yaml | ||
|
|
||
| # This flat-row recipe binds its own environment instead of the generic NOOA environment. | ||
| nooa_environment_server: | ||
| _delete_key: environment_servers | ||
|
|
||
| swebench_pro_nooa_resources_server: | ||
| _inherit_from: swebench_pro_resources_server | ||
| resources_servers: | ||
| swebench_pro: | ||
| allowed_agents: [nooa_agent] | ||
| sandbox_config: | ||
| # Outlive the episode, patch extraction, and final cleanup budgets. | ||
| ttl_s: 25200 | ||
| datasets: | ||
| - name: swebench_pro | ||
| type: benchmark | ||
| jsonl_fpath: benchmarks/swebench/data/swebench_pro_benchmark.jsonl | ||
| prepare_script: benchmarks/swebench/pro/prepare.py | ||
| num_repeats: 1 | ||
|
|
||
| swebench_pro_nooa_agent: | ||
| _inherit_from: nooa_agent | ||
| responses_api_agents: | ||
| nooa_agent: | ||
| resources_server: | ||
| type: resources_servers | ||
| name: swebench_pro_nooa_resources_server | ||
| nooa: | ||
| execution_mode: sandboxed | ||
| max_policy_calls: null | ||
|
init-nikhil marked this conversation as resolved.
Resolved
|
||
|
|
||
| swebench_pro_nooa: | ||
| _inherit_from: single_agent_turn_legacy | ||
| environment_servers: | ||
| single_agent_turn_legacy: | ||
| resources_server: | ||
| type: resources_servers | ||
| name: swebench_pro_nooa_resources_server | ||
| agent_server: | ||
| type: responses_api_agents | ||
| name: swebench_pro_nooa_agent | ||
104 changes: 104 additions & 0 deletions
104
fern/versions/latest/pages/evaluation-tutorials/nooa-swe-bench-pro.mdx
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,104 @@ | ||
| --- | ||
|
init-nikhil marked this conversation as resolved.
Resolved
|
||
| title: "NOOA sessions and SWE-bench Pro" | ||
| description: "Run NOOA through single_agent_turn and a Resources-owned sandbox." | ||
| position: 5 | ||
| --- | ||
|
|
||
| # NOOA sessions and SWE-bench Pro | ||
|
|
||
| NOOA uses Gym's agent sessions and existing `single_agent_turn` Environment Server. It seeds Resources, | ||
| opens and invokes NOOA, closes the agent to collect evidence and final Resources cookies, asks Resources to | ||
| verify, and closes Resources. Call the Environment Server's `/run`; NOOA | ||
| no longer implements `/run` or assigns rewards. Sessions support one activation, with identical retries sharing it. | ||
| Use one agent-server worker; independent sessions can still execute concurrently. | ||
|
|
||
| The default harness is `nooa_bench.bench_agent:BenchAgent`, invoked through | ||
| `responses_api_agents.nooa_agent.bench_agent_adapter:invoke_bench_agent`. The standard NOOA requirements | ||
| install `nooa`, `nooa-cli`, and `nooa-bench` at the same revision. Normal `gym env prefetch` prepares | ||
| these dependencies for the agent server; sandbox runtime preparation installs the same pinned packages. | ||
|
|
||
| ## Execution modes | ||
|
|
||
| `nooa.execution_mode: embedded` runs NOOA in the agent server. Its Python execution can access files, processes, | ||
| and network on that host; embedded mode provides no sandbox protection. Custom embedded invocation adapters | ||
| must close their own processes and resources before returning; agent session close cannot track arbitrary | ||
| host-side effects. | ||
|
|
||
| `nooa.execution_mode: sandboxed` runs the complete NOOA agent in the Resources-provided sandbox. Resources must | ||
| return direct `SandboxAccess`; NOOA never provisions or destroys the sandbox. NOOA prepares its runtime under | ||
| `/opt/nemo-gym-nooa`, keeps session files under `/tmp/nemo-gym-nooa`, and executes in the supplied task workdir. | ||
| NOOA closes its worker and session before Resources verifies; the Resources-owned sandbox remains available. | ||
| Verification can inspect persisted artifacts after agent cleanup. Tasks whose verifier requires a | ||
| NOOA-started service to remain live are not supported by this lifecycle. | ||
|
|
||
| Cold runtime setup requires Linux x86_64 or aarch64 with glibc, `bash`, `tar`, `gzip`, `sha256sum`, `flock`, | ||
| `tee`, `mv` with `-T` support, and writable `/opt/nemo-gym-nooa` and `/tmp`. The controller supplies Python; | ||
| the sandbox needs outbound package and build-dependency access. Alpine/musl images are not supported. | ||
| Model and Resources URLs must be reachable from the sandbox. A host-loopback URL is not automatically | ||
| rewritten; configure `use_absolute_ip` / `default_host` for the deployment. No host virtual environment | ||
| or provider credentials are copied into the runner. | ||
|
|
||
| ## Resource tools and observations | ||
|
|
||
| Direct HTTP tool grants arrive through the agent session's `tool_accesses`. Generated `self.*` resource methods, `GymResponsesLLM`, | ||
| and `GymTraceHooks` execute alongside NOOA. Model and resource cookies remain separate; final resource cookies | ||
| return on close for verification. Model/resource infrastructure failures fail the episode instead of becoming a | ||
| reward-zero response. Policy-budget exhaustion remains a gradable partial response. | ||
|
|
||
| Hooks capture Python execution and nested resource calls. Sandbox results are exported on completion or handled | ||
| errors; a hard kill can lose in-memory events. Sandbox-written results are agent-reported evidence, not trusted | ||
| training records. Resources remains responsible for rewards. | ||
|
|
||
| ## Stateful resource example | ||
|
|
||
| Use `resources_servers/example_session_state_mgmt/configs/example_session_state_mgmt_nooa.yaml` for an embedded | ||
| real-model smoke of the instance-method resource bindings. Its materialized task rows are in | ||
| `resources_servers/example_session_state_mgmt/data/example_nooa_tasks.jsonl`; other agents retain the original rows. | ||
| The smoke requires `POLICY_BASE_URL`, `POLICY_API_KEY`, and `POLICY_MODEL_NAME`: | ||
|
|
||
| ```bash | ||
| bash responses_api_agents/nooa_agent/smoke/run_stateful_e2e.sh | ||
| ``` | ||
|
|
||
| The script routes `nooa-counter` tasks to `example_session_state_mgmt_environment_server` and checks the server-side | ||
| counter reward, model usage, resource calls, and trajectory. This example does not exercise sandbox provisioning. | ||
|
|
||
| ## SWE-bench Pro | ||
|
|
||
| The recipe reuses the existing Resources-owned task sandbox, patch extraction, and separate grading sandbox: | ||
|
init-nikhil marked this conversation as resolved.
Resolved
|
||
|
|
||
| ```bash | ||
| python benchmarks/swebench/pro/prepare.py | ||
| gym env start --config benchmarks/swebench/pro/nooa.yaml --config model-provider.yaml | ||
|
|
||
| gym eval run --no-serve \ | ||
| --config benchmarks/swebench/pro/nooa.yaml --config model-provider.yaml \ | ||
| --input benchmarks/swebench/data/swebench_pro_benchmark.jsonl \ | ||
| --output /tmp/nooa-swebench-pro.jsonl --limit 1 --concurrency 1 \ | ||
| +agent_name=swebench_pro_nooa_agent | ||
| ``` | ||
|
|
||
| Use `model-provider.yaml` to supply the policy model and sandbox provider settings and credentials. Pass the | ||
| same recipe and overlay to both commands: `--no-serve` does not inherit routing from the running head server. | ||
| For Docker, configure Gym's published model | ||
| and Resources addresses to be reachable from the container before running. Preparation preserves upstream grading | ||
| metadata in the shared flat dataset. The existing `single_agent_turn_legacy` adapter separates that metadata | ||
| from the Responses request before invoking NOOA. Swap the provider configuration for | ||
| remote infrastructure without changing the agent recipe. | ||
|
|
||
| The overall deadline belongs to the Environment Server (`default_episode_timeout_seconds`). The standard | ||
| task contract has no separate NOOA agent deadline. This recipe sets the task sandbox TTL to **25,200 seconds | ||
| (seven hours)** through | ||
| `swebench_pro_nooa_resources_server.resources_servers.swebench_pro.sandbox_config.ttl_s`. | ||
| Override that Resources key when changing the budget; provider overlays do not set this TTL. | ||
| Keep the worker ceiling plus patch extraction and cleanup below `ttl_s`, and also keep the episode deadline | ||
| plus final cleanup below it. The defaults are a 21,420-second worker ceiling and a 21,600-second episode timeout. | ||
| Patch generation and patch-file removal each use the sandbox command timeout (180 seconds); stopping the task | ||
| sandbox has a 120-second budget. Patch download has no separate timeout; the episode deadline bounds the caller's wait. | ||
| Final cleanup allows 180 seconds per callback, with at most three registered callbacks including the Resources retry. | ||
| The episode budget includes setup, so it can expire before the worker reaches its ceiling. Protected grading uses a | ||
| separate sandbox after patch extraction; a model-call limit alone is not a wall-clock guarantee. | ||
| NOOA has no model-call limit by default (`max_policy_calls: null`). Set a positive `max_policy_calls` to opt into a | ||
| shared model-call budget for the rollout. The optional `resources_server` agent reference identifies the Resources | ||
| server that owns the dataset and its schema. Runtime tool and sandbox access comes from the standard environment's | ||
| agent session grants. The old `run_timeout_secs` agent configuration field is removed. | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,159 @@ | ||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | ||
| # SPDX-License-Identifier: Apache-2.0 | ||
|
|
||
| import json | ||
| from inspect import signature | ||
| from pathlib import Path | ||
| from unittest.mock import MagicMock | ||
|
|
||
| import pytest | ||
|
|
||
| from environment_servers.single_agent_turn.app import SingleAgentTurnEnvironmentServerConfig | ||
| from environment_servers.single_agent_turn_legacy.app import SingleAgentTurnLegacyEnvironmentServer | ||
| from nemo_gym.server_utils import ServerClient | ||
|
|
||
|
|
||
| ROOT = Path(__file__).resolve().parents[3] | ||
|
|
||
|
|
||
| def test_swe_shared_preparation_preserves_grading_metadata_through_legacy_adapter(tmp_path: Path) -> None: | ||
| from benchmarks.swebench.pro.prepare import prepare, render_prompt | ||
|
|
||
| task = { | ||
| "instance_id": "repo-1", | ||
| "dockerhub_tag": "fixture:1", | ||
| "problem_statement": "Fix the task.", | ||
| "requirements": "Keep the public interface.", | ||
| "interface": "main()", | ||
| "patch": "private ground truth", | ||
| } | ||
| for name in ( | ||
| "run_scripts/repo-1/run_script.sh", | ||
| "run_scripts/repo-1/parser.py", | ||
| "dockerfiles/base_dockerfile/repo-1/Dockerfile", | ||
| "dockerfiles/instance_dockerfile/repo-1/Dockerfile", | ||
| ): | ||
| asset = tmp_path / "upstream" / name | ||
| asset.parent.mkdir(parents=True, exist_ok=True) | ||
| asset.write_text("private grading asset") | ||
| output = prepare( | ||
| dataset=[task], | ||
| upstream_root=tmp_path / "upstream", | ||
| output_fpath=tmp_path / "shared.jsonl", | ||
| image_digest_resolver=lambda tag: "sha256:" + "a" * 64, | ||
| ) | ||
| original = output.read_bytes() | ||
| row = json.loads(original) | ||
| adapter = SingleAgentTurnLegacyEnvironmentServer( | ||
| config=SingleAgentTurnEnvironmentServerConfig( | ||
| name="swebench_pro_nooa", | ||
| host="localhost", | ||
| port=8000, | ||
| entrypoint="app.py", | ||
| cleanup_timeout_seconds=180, | ||
| resources_server={"type": "resources_servers", "name": "swebench_pro_nooa_resources_server"}, | ||
| agent_server={"type": "responses_api_agents", "name": "swebench_pro_nooa_agent"}, | ||
| ), | ||
| server_client=MagicMock(spec=ServerClient), | ||
| ) | ||
| request = adapter._episode_request_from_row(row | {"_ng_task_index": 0, "_ng_rollout_index": 0}) | ||
| assert request.task.task_id.taskset == "swebench_pro_nooa_resources_server" | ||
| assert request.task.task_id.task_id == "repo-1" | ||
| assert request.task.task_input.task_data == {k: v for k, v in row.items() if k != "responses_create_params"} | ||
| params = request.task.task_input.responses_create_params | ||
| assert params.input[0].content == render_prompt(task) | ||
| assert "private ground truth" not in params.model_dump_json() | ||
| assert "private grading asset" not in params.model_dump_json() | ||
| for task_source in (adapter.config.name, adapter.config.agent_server.name, adapter.config.resources_server.name): | ||
| routed = adapter._episode_request_from_row( | ||
| row | ||
| | { | ||
| "_ng_task_index": 0, | ||
| "_ng_rollout_index": 0, | ||
| "task_source": task_source, | ||
| "agent_ref": {"name": adapter.config.agent_server.name}, | ||
| } | ||
| ) | ||
| assert routed.task.task_id.taskset == task_source | ||
| assert routed.task.task_input.responses_create_params == params | ||
| with pytest.raises(ValueError, match="does not match configured agent server"): | ||
| adapter._episode_request_from_row(row | {"agent_ref": {"name": "different_agent"}}) | ||
| with pytest.raises(ValueError, match="names none of this Environment Server"): | ||
| adapter._episode_request_from_row(row | {"task_source": "different_resources"}) | ||
| assert output.read_bytes() == original | ||
|
|
||
|
|
||
| def test_swe_recipe_uses_borrowed_sandbox_and_single_agent_turn_environment() -> None: | ||
| from omegaconf import OmegaConf | ||
|
|
||
| from environment_servers.single_agent_turn.app import SingleAgentTurnEnvironmentServerConfig | ||
| from nemo_gym.global_config import GlobalConfigDictParser | ||
| from nemo_gym.rollout_collection import RolloutCollectionHelper | ||
| from nemo_gym.sandbox import AsyncSandbox | ||
| from resources_servers.swebench_pro.app import SWEBenchProResourcesServerConfig | ||
| from responses_api_agents.nooa_agent.config import NOOAAgentConfig | ||
|
|
||
| parser = GlobalConfigDictParser() | ||
| _, configs = parser.load_extra_config_paths([str(ROOT / "benchmarks/swebench/pro/nooa.yaml")]) | ||
| config = OmegaConf.merge(*configs) | ||
| parser._recursively_swap_keys(config) | ||
| environment = SingleAgentTurnEnvironmentServerConfig( | ||
| name="swebench_pro_nooa", | ||
| host="localhost", | ||
| port=8000, | ||
| **OmegaConf.to_container(config.swebench_pro_nooa.environment_servers.single_agent_turn_legacy, resolve=True), | ||
| ) | ||
| agent = NOOAAgentConfig( | ||
| name=environment.agent_server.name, | ||
| host="localhost", | ||
| port=8001, | ||
| **OmegaConf.to_container(config[environment.agent_server.name].responses_api_agents.nooa_agent, resolve=True), | ||
| ) | ||
| assert agent.nooa.execution_mode == "sandboxed" | ||
| assert agent.num_workers == 1 | ||
| assert agent.model_server.name == "policy_model" | ||
| assert agent.max_policy_calls is None | ||
| assert environment.resources_server.name == "swebench_pro_nooa_resources_server" | ||
| assert not environment.resources_tool_transports | ||
| resources = config[environment.resources_server.name].resources_servers.swebench_pro | ||
|
init-nikhil marked this conversation as resolved.
Resolved
|
||
| assert resources.allowed_agents == ["nooa_agent"] | ||
| assert resources.datasets[0].jsonl_fpath == "benchmarks/swebench/data/swebench_pro_benchmark.jsonl" | ||
| assert resources.datasets[0].prepare_script == "benchmarks/swebench/pro/prepare.py" | ||
| assert resources.get("task_network_policy") is None | ||
| resource_config = SWEBenchProResourcesServerConfig( | ||
| name=environment.resources_server.name, | ||
| host="localhost", | ||
| port=8002, | ||
| **OmegaConf.to_container(resources, resolve=True), | ||
| ) | ||
| ttl = resource_config.sandbox_config["ttl_s"] | ||
| # Patch extraction and patch-file removal each use AsyncSandbox.exec's default timeout. | ||
| command_timeout = signature(AsyncSandbox.exec).parameters["timeout_s"].default | ||
| assert isinstance(command_timeout, (int, float)) | ||
| assert resource_config.verification_stop_timeout is not None | ||
| assert environment.default_episode_timeout_seconds is not None | ||
| # SingleAgentTurn registers Resources, agent, and post-verification Resources retry callbacks. | ||
| final_cleanup_budget = 3 * environment.cleanup_timeout_seconds | ||
| assert ttl > ( | ||
| agent.sandbox_runner_timeout_seconds | ||
| + 2 * command_timeout | ||
| + resource_config.verification_stop_timeout | ||
| + final_cleanup_budget | ||
| ) | ||
| # The episode deadline bounds the caller's wait for setup and patch download, which has no separate timeout. | ||
| assert ttl > environment.default_episode_timeout_seconds + final_cleanup_budget | ||
| instances = parser.filter_for_server_instance_configs(config) | ||
| assert {(instance.name, instance.SERVER_TYPE) for instance in instances} == { | ||
| ("swebench_pro_nooa_agent", "responses_api_agents"), | ||
| ("swebench_pro_nooa_resources_server", "resources_servers"), | ||
| ("swebench_pro_nooa", "environment_servers"), | ||
| } | ||
| rows = [{"agent_ref": {"name": agent.name}, "task_source": environment.resources_server.name}] | ||
| RolloutCollectionHelper._validate_agent_pairings(rows, config) | ||
| resources.allowed_agents = ["different_agent"] | ||
| with pytest.raises(ValueError, match="does not accept the agent"): | ||
| RolloutCollectionHelper._validate_agent_pairings(rows, config) | ||
| environments = [instance for instance in instances if instance.SERVER_TYPE == "environment_servers"] | ||
| assert [(instance.name, set(instance.environment_servers)) for instance in environments] == [ | ||
| ("swebench_pro_nooa", {"single_agent_turn_legacy"}) | ||
| ] | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.