Skip to content

Add benchmark: HumanEval - #889

Merged
Ying1123 merged 3 commits into
mainfrom
add-human-eval
Aug 2, 2024
Merged

Ying1123 merged 3 commits into
mainfrom
add-human-eval

Conversation

@Ying1123

@Ying1123 Ying1123 commented Aug 2, 2024

Copy link
Copy Markdown
Contributor

No description provided.

@Ying1123
Ying1123 merged commit e90e3a5 into main Aug 2, 2024
@Ying1123
Ying1123 deleted the add-human-eval branch August 2, 2024 07:46
timethink pushed a commit to timethink/sglang that referenced this pull request Mar 9, 2025
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 26, 2026
…own merges moved

The merge-regression run over the day's ticket suites turned up exactly one
failure outside the recorded standing set:
test_rejected_evidence_pins.py::PpWithSpecEvidenceTest::
test_evidence_cites_land_on_BOTH_halves_of_the_guard.

Not a defect in any merged fix. sgl-project#898 re-pinned this row against
be56c4e, and the same day's server_args.py additions -- sgl-project#889, sgl-project#893,
sgl-project#894, sgl-project#897 and sgl-project#887/sgl-project#890 -- then pushed the guard down 44 lines. The
verdict did not change; its coordinates did. :19436/:19451 -> :19480/:19495,
both re-read in the merged tree rather than taken from the failure text.

This is the guard doing its job: a line pin is a coordinate into a file
other tickets edit, so it goes stale at MERGE time, not at write time, and
only a train run can see it. #704b re-pinned the same row for the same
reason, so the note now says so instead of reading as a one-off.

CLASS. test_rejected_evidence_pins.py is the only line-pin validator in the
tree (grep for _LINE_REF / "cite has drifted"), and its other five cases
pass, so no sibling pin drifted unseen behind a missing guard. 6 passed.
efschu pushed a commit to efschu/htsglang that referenced this pull request Aug 26, 2026
The merge-regression pass runs ruff with the repo's OWN selection
(.pre-commit-config.yaml: --select=F401,F821,UP037), not ruff's defaults --
the default run reports 575 findings on these files and measures a gate
nobody enforces.

Under the enforced selection: 0 F401, 0 F821, 22 UP037. Nineteen were
already there at 0cd27d9, measured rather than assumed by running the
same selection against the base blobs of the same files; three are new,
all in phase_policy.py, all forward references introduced by sgl-project#889/sgl-project#893
(effective_pp_exit_term, stranded_decode_s, superseded_pp_bound_warning).

Fixed all 22 rather than the 3. Both files carry `from __future__ import
annotations`, so the quotes are inert and their removal is the autofix the
hook itself applies; fixing only the new three would leave the file split
between two conventions, and leaving all 22 hands the next toucher a
pre-commit failure they did not cause.

91 passed across the suites over both files (sgl-project#889, sgl-project#893, sgl-project#896, the arm
outcome suite, sgl-project#838 layout conformance) after the rewrite.

Codespell, for the record, is NOT clean and is not made clean here: 12
hits on train-touched files against 11 at the same base. The single new one
is "oder" in test_one_chunk_tp_prefill_887.py:8 -- inside a verbatim German
user directive the test quotes as its premise, the same class as the
"egal" quote that has sat at phase_policy.py since before this train. A
quote is not a typo, and rewriting one to please a dictionary would falsify
the record it exists to preserve.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant