Fix file corruption when server ignores Range header on download retry

[XciD]

Summary

• Fix file corruption in http_get() when a download retry's Range header is ignored by the server
• When a download is interrupted (timeout/connection error), the retry sends a Range header. CloudFront (and other CDNs) ignore Range when Accept-Encoding: gzip is present, returning 200 OK with the full file instead of 206 Partial Content. The code didn't check the response status, so it appended the full file to the existing partial data, corrupting the file
• The fix detects this case (resume_size > 0 but status_code == 200) and truncates the file before writing

Context

Discovered via a CI failure where tokenizer.json (expected 1,843,320 bytes) ended up as 3,920,264 bytes after two retries:

233,624 (partial) + 1,843,320 (full, appended) + 1,843,320 (full, appended) = 3,920,264

The root cause is that Python's httpx (and requests) automatically sends Accept-Encoding: gzip, deflate. CloudFront cannot serve a Range response when content-encoding is applied, so it ignores the Range header and returns 200 with the full (compressed) content. The client transparently decompresses it and appends it to the partial file.

Confirmed with curl:

# With Accept-Encoding: gzip → Range is IGNORED → 200
curl -H 'Accept-Encoding: gzip, deflate' -H 'Range: bytes=500-999' <cdn-url>  # → 200

# Without Accept-Encoding → Range is HONORED → 206
curl -H 'Range: bytes=500-999' <cdn-url>  # → 206

---

Note

Low Risk
Small, localized change to retry/download write behavior plus a targeted unit test; low risk aside from potential edge cases around resume/truncate semantics for unusual servers.

Overview
Fixes a corruption case in http_get retries: if a resumed download sends a Range header but the server responds with 200 OK (ignoring the range), the partially written temp file is now truncated and the download restarts from byte 0 instead of appending full content onto partial data.

Adds a regression test covering this scenario by simulating an interrupted download followed by a retry where the server returns 200 with the full body, asserting the final file length/content are correct (not oversized from appends).

^{Written by Cursor Bugbot for commit 9bc947e. This will update automatically on new commits. Configure here.}

[XciD]

Full Investigation Report

Incident: AMD GPU CI Build 4130

• Failed test: compile/fullgraph/test_basic_correctness.py::test_compile_correctness[test_setting1]
• Model: TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ
• Error:

  OSError: Consistency check failed: file should be of size 1843320 but has size 3920264 (tokenizer.json).
  

• CI User: vllm-ci (hf_hub/0.36.1, python/3.12.12, torch/2.9.1)
• CI Runner IP: 137.220.63.153 (Choopa/Vultr, Elk Grove Village, Illinois)

Timeline (2026-02-04 UTC)

Time	Event	Source
07:59:09	HEAD request → 200, Content-Length: 1,843,320	CloudFront
07:59:10	GET #1 (no Range header) → 200, cs_bytes: 548	CloudFront
07:59:22	ReadTimeout after ~12s, partial data received	CI log
07:59:23	GET #2 (with Range header) → 200 (not 206), cs_bytes: 571 (+23 bytes = Range header)	CloudFront
~07:59:35	ReadTimeout again on retry	Inferred
07:59:46	GET #3 (with Range header) → 200 (not 206), cs_bytes: 571	CloudFront
07:59:46+	Consistency check fails: 3,920,264 ≠ 1,843,320	CI log

CloudFront Evidence (Athena)

All requests served from edge ORD58-P13 (Chicago, closest to Illinois CI runner).

Request	cs_bytes	sc_bytes	sc_status	Range header
HEAD	346	1,240	200	No
GET #1	548	487,440	200	No
GET #2 (retry)	571	487,448	200	Yes
GET #3 (retry)	571	487,431	200	Yes

Key observations:

• cs_bytes increases by exactly 23 bytes on retries — that's the Range: bytes=N- header being sent
• sc_status is 200 on all requests, never 206 — CloudFront ignored the Range header
• sc_bytes (~487KB) is consistent across all GETs — full gzip-compressed file every time

Mathematical Proof

The file is served gzip-compressed by CloudFront (~487KB on the wire). iter_content() / iter_bytes() transparently decompresses, yielding 1,843,320 bytes.

GET #1 partial (before timeout):    233,624 bytes
GET #2 full (appended):           1,843,320 bytes
GET #3 full (appended):           1,843,320 bytes
─────────────────────────────────────────────
Total:                            3,920,264 bytes  ✓  (matches error exactly)

Root Cause

CloudFront ignores Range when Accept-Encoding: gzip is present. This is per HTTP spec — Range applies to the entity body, but Content-Encoding transforms are applied after Range, making them incompatible.

Python's httpx (and requests) automatically sends Accept-Encoding: gzip, deflate on every request. When CloudFront sees both headers, it serves the full gzip-compressed response with 200 OK instead of 206 Partial Content.

Confirmed with curl:

# With Accept-Encoding: gzip → Range is IGNORED → 200
curl -s -o /dev/null -w '%{http_code}' \
  -H 'Accept-Encoding: gzip, deflate' \
  -H 'Range: bytes=500-999' \
  'https://huggingface.co/api/resolve-cache/models/TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ/05835c52707fff57cecd16a364de2bc65c9bf102/tokenizer.json'
# → 200

# Without Accept-Encoding → Range is HONORED → 206
curl -s -o /dev/null -w '%{http_code}' \
  -H 'Range: bytes=500-999' \
  'https://huggingface.co/api/resolve-cache/models/TheBloke/TinyLlama-1.1B-Chat-v0.3-GPTQ/05835c52707fff57cecd16a364de2bc65c9bf102/tokenizer.json'
# → 206

The Bug

In http_get(), when a ReadTimeout or ConnectionError occurs during streaming, the code retries recursively with resume_size=new_resume_size, which adds a Range header. But it never checks if the response is 206 (Partial Content) vs 200 (OK). Since temp_file is in append mode, the full response gets appended to existing partial data.

The Fix

4 lines: after the GET request, if resume_size > 0 but the server returned 200 (not 206), truncate the file since we're receiving the complete content, not a partial response.

[bot-ci-comment]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 76.62%. Comparing base (1daa48b) to head (9bc947e).
⚠️ Report is 67 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3778      +/-   ##
==========================================
+ Coverage   75.00%   76.62%   +1.61%     
==========================================
  Files         145      153       +8     
  Lines       13978    15090    +1112     
==========================================
+ Hits        10484    11562    +1078     
- Misses       3494     3528      +34     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Wauplin]

Very nice catch! This issue has being around for the last 2 years but never caught/reproduced 😄

Let's close #1498, #1549, #2372. Thanks @XciD !