Skip to content

chore: bump opendal to 0.59 - #2979

Merged
pavelzw merged 2 commits into
conda:mainfrom
wolfv:chore/opendal-0.59
Oct 23, 2026
Merged

pavelzw merged 2 commits into
conda:mainfrom
wolfv:chore/opendal-0.59

Conversation

@wolfv

@wolfv wolfv commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Description

opendal 0.57 pulls in quick-xml 0.39.4, which has two DoS advisories (RUSTSEC-2026-0194, RUSTSEC-2026-0195). opendal 0.59 uses quick-xml 0.42. I split this out of #2977 because of the raw API rewrite below.

The library side needs two changes:

  • Operator::new now returns the Operator, so .finish() goes away (rattler_index, rattler_upload).
  • opendal 0.59 no longer installs an HTTP transport by itself, so every S3 request failed with default HTTP transport is not installed. opendal::install_default() would fix that, but it installs the transport process-wide on a first-installed-wins basis, and a library shouldn't make that choice for the application. Instead, the new S3CredentialSource::opendal_operator() in rattler_s3 builds the operator and attaches its own reqwest transport (opendal-http-transport-reqwest). rattler_index and rattler_upload now use it instead of opendal_builder() + Operator::new.

Most of the diff is the ETag memory backend in the rattler_index tests. opendal 0.59 replaced the raw Access trait with Service, and read/write/list/delete are now synchronous and take an OperationContext. What moved:

  • Reads go through an oio::StreamReader. Its open() does what read() used to do (hooks, if_match/if_none_match/if_unmodified_since), and it also honors the requested byte range.
  • Writes: the writer can't await when it's created, so it takes the entry lock and checks the write conditions on its first write() or close(). The BeforeWrite hook fires at that point too. The lock is still held until close(), so concurrent writes to one path are serialized the same way as before.
  • Listing collects entries on the first next(). The listing logic itself didn't change; I only moved it into ETagMemoryBackend::list_entries.
  • Metadata is built through MetadataBuilder (FileEntry::metadata()).

The backend changes are in test code only. Nothing changes for users apart from the newer opendal.

How Has This Been Tested?

  • cargo nextest run -p rattler_index -p rattler_upload -p rattler_s3 --all-features: 85/85 pass, the same 85 tests that main lists.
  • To check that the port didn't weaken the conditional-write tests, I disabled the if_match check on writes in the backend. test_conditional_write_with_if_match fails both here and on main, and nothing else does on either, so the coverage is unchanged. (The concurrency tests don't depend on that particular check, before or after.)
  • pixi run e2e-s3-rustfs passes locally (upload, index, cache-control headers, install). That's the test that caught the missing transport, which the unit tests can't see because they only use the memory and fs backends.
  • clippy is clean, cargo check passes in py-rattler/, and cargo audit no longer reports quick-xml.

AI Disclosure

  • This PR contains AI-generated content.
    • I have tested any AI-generated content in my PR.
    • I take responsibility for any AI-generated content in my PR.
      Tools: Claude Code
Make a separate PR for opendal bump pls

Checklist:

  • I have performed a self-review of my own code

🤖 Generated with Claude Code

opendal 0.57 depends on quick-xml 0.39, which has two DoS advisories
(RUSTSEC-2026-0194, RUSTSEC-2026-0195); 0.59 uses quick-xml 0.42.

- Operator::new returns the Operator directly, so drop .finish().
- Port the ETag memory test backend from the removed raw Access trait to
  Service: read/write/list/delete are now synchronous, so conditional
  read checks move into a StreamReader, the writer takes the entry lock
  and checks write conditions on its first write/close, and the lister
  collects entries on its first next().

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
opendal 0.59 no longer installs an HTTP transport, so every S3 request
failed with "default HTTP transport is not installed". Rather than
installing one process-wide from library code (first-installed-wins,
which would take the choice away from applications), add
S3CredentialSource::opendal_operator, which attaches a reqwest
transport to the operator it builds, and use it in rattler_index and
rattler_upload.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@wolfv
wolfv requested a review from pavelzw October 23, 2026 10:39
@pavelzw
pavelzw merged commit afe9bd2 into conda:main Oct 23, 2026
22 of 23 checks passed
@octo-sts octo-sts Bot mentioned this pull request Oct 23, 2026
wolfv added a commit that referenced this pull request Oct 23, 2026
sigstore 0.15 (#2977) and opendal 0.59 (#2979) change types that are
part of the public API of these crates. cargo-semver-checks doesn't
flag dependency type changes like these, so bump by hand:

- rattler_sigstore 0.3.0: re-exports sigstore types
- rattler 0.55.0: Installer::with_attestation_policy takes a
  rattler_sigstore VerificationPolicy
- rattler_virtual_package_detectors 0.3.0: EnvironmentError::Install
  wraps rattler's InstallerError
- rattler_s3 0.4.0: opendal_builder returns an opendal S3 builder
- rattler_index 0.34.0: takes opendal Operators, exposes an opendal
  Timestamp
- rattler_upload 0.14.0: S3Opts::credentials is a rattler_s3 type

The zip 9 baselines are published (rattler_package_streaming 0.27.15),
so turn semver_check back on. Exclude the release-plz generated crate
changelogs from typos, since they quote old commit messages verbatim.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Signed-off-by: Wolf Vollprecht <w.vollprecht@gmail.com>
@octo-sts octo-sts Bot mentioned this pull request Oct 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants