Use HTTP range requests for uncompressed layer fetches in libindex

[ushaket]

Summary

Use HTTP range requests for uncompressed layer fetches in libindex instead of downloading the full blob up front. This lets claircore access large uncompressed layers lazily through io.ReaderAt, while keeping the existing full-download behavior for compressed layers.

---

Related Issues

• Related motivation: quay/clair-action#275 Presigned layer URLs expire before fetch when images have large layers

---

Changes

HTTP range-backed layer access

• Added internal/httputil.RangeReaderAt, an io.ReaderAt implementation backed by HTTP Range requests
• Added read-ahead buffering to reduce the number of small HTTP requests generated while tarfs walks tar headers
• Reused existing compression detection logic by exporting zreader.DetectCompression

Fetcher behavior

• Updated libindex/fetcher.go to probe layers with Range: bytes=0-15
• If the layer is uncompressed and the registry responds cleanly to the probe, libindex now initializes the layer directly from RangeReaderAt
• If the layer is compressed or the registry does not cleanly support the range path, the fetcher falls back to the existing full-download path

Test coverage

• Added unit tests for RangeReaderAt
• Added integration tests for:
  • uncompressed range-backed fetches
  • compressed fallback
  • non-range fallback
  • malformed probe fallback
  • transport failure fallback
  • follow-up reads where a server ignores a later Range request
  • same digest accessed through different source/auth contexts

---

Context

Claircore already builds tarfs from io.ReaderAt, which means scanners do not require a fully downloaded layer stream up front. By switching uncompressed layers to an HTTP range-backed reader, the existing indexing and scanning pipeline can stay unchanged while avoiding unnecessary transfer of large blob contents.

This is especially useful for large uncompressed model/data layers, where scanners typically only need tar metadata or a small number of files.

---

Testing

In-repo automated tests

• internal/httputil/rangereaderat_test.go
• libindex/fetcher_range_test.go

Real-image validation

Validated the behavior against live registries and real images:

• registry.redhat.io modelcar images:
  • uncompressed layers used the range-backed path
  • compressed layers used the existing fallback path
  • large .safetensors layers were accessed through small ranged reads without full blob downloads
• Common public images:
  • Python scanner validated on python:3.12-slim
  • Java scanner validated on tomcat:10.1-jdk17-temurin
  • RPM scanner validated on centos:stream9

A/B equivalence validation

Ran a fixture-based comparison of old vs new fetch behavior and confirmed equivalent scanner results for:

• Python
• Java
• RPM

The same fixture runs also showed that irrelevant uncompressed blob content was fully downloaded with the old path and avoided with the new range-backed path.

---

Examples

Uncompressed layer

Before:

• blob is fully downloaded before scanning

After:

• fetcher probes the blob with a small range request
• uncompressed layers are read lazily through RangeReaderAt
• tarfs and scanners request only the byte ranges they need

Compressed layer

Before:

• full download + decompress + spool

After:

• probe detects compression
• fetcher follows the same full-download path as before

---

Assisted-by: Claude Opus 4.6