Skip to content

fix(copy): re-push referenced manifests when the destination reports them missing#1212

Merged
TerryHowe merged 2 commits into
oras-project:mainfrom
Orexii:fix/copy-selfheal-missing-referenced-manifests
Jul 15, 2026
Merged

fix(copy): re-push referenced manifests when the destination reports them missing#1212
TerryHowe merged 2 commits into
oras-project:mainfrom
Orexii:fix/copy-selfheal-missing-referenced-manifests

Conversation

@Orexii

@Orexii Orexii commented Jun 10, 2026

Copy link
Copy Markdown
Contributor

What this does

Makes CopyGraph self-heal a partially-copied graph. When the destination rejects a manifest push because it is missing referenced content (MANIFEST_BLOB_UNKNOWN / BLOB_UNKNOWN / MANIFEST_UNKNOWN), the copy now re-pushes the node's manifest successors and retries the push once, instead of returning the error.

Why

CopyGraph skips a node whose sub-DAG it believes already exists ("skip if a rooted sub-DAG exists"). If a prior copy was interrupted after a child manifest's blobs were uploaded but before the child manifest was committed, the destination can report the child as present (Exists/HEAD) yet reject the parent that references it. For a multi-arch image index this means the index push fails with MANIFEST_BLOB_UNKNOWN, and since the child is skipped on every run, re-running the same oras cp --recursive never recovers.

This was hit in production mirroring a multi-arch image (mongo:7.0.34) between two registries; the only manual workaround was copying each child manifest by digest and re-pushing the index.

How

  • copyGraph: wrap the node push; on a "missing referenced content" error, re-copy the manifest successors unconditionally (bypassing the existence short-circuit) and retry the push once. No-op on the success path; blob successors are not re-pushed.
  • isMissingReferencedContentError: typed classification via errcode (handles both a single errcode.Error and a multi-error errcode.Errors); no string matching.

Testing

  • TestCopyGraph_SelfHealsInterruptedIndexCopy: end-to-end reproduction against a destination that emulates the interrupted state — fails without this change (MANIFEST_BLOB_UNKNOWN), passes with it.
  • TestIsMissingReferencedContentError: unit test for the classifier (100% covered).
  • go test -race ./... passes; gofmt / go vet clean; new files carry the Apache license header.

Notes for reviewers

  • Design choice: detection currently uses the registry-specific errcode package from transport-agnostic copyGraph. An alternative is a sentinel in errdef that the remote manifestStore maps MANIFEST_BLOB_UNKNOWN / BLOB_UNKNOWN onto. Happy to switch if you prefer that layering.
  • Scope: recovers the immediate manifest successors (the index -> child-manifest case). No API or signature changes; no breaking changes.

Fixes #1211

…them missing

copyGraph skips a node when dst.Exists reports it present ("skip if a rooted
sub-DAG exists"). If a previous copy was interrupted after a referenced
manifest's blobs were uploaded but before that manifest itself was committed,
the destination can report the child manifest as existing while still rejecting
any manifest that references it. The push of the parent (e.g. an image index)
then fails with MANIFEST_BLOB_UNKNOWN, and because the child is skipped on every
subsequent run the recursive copy never self-heals.

When a node push fails because the destination is missing referenced content
(MANIFEST_BLOB_UNKNOWN / BLOB_UNKNOWN / MANIFEST_UNKNOWN), re-copy the node's
manifest successors unconditionally and retry the push once. This lets an
interrupted copy recover instead of failing identically on every retry. The
retry is a no-op on the success path.

Adds TestCopyGraph_SelfHealsInterruptedIndexCopy (end-to-end reproduction) and
TestIsMissingReferencedContentError (classifier unit test).

Signed-off-by: Malcolm Morgan <Malcolm.Morgan@Microsoft.com>
@codecov

codecov Bot commented Jun 12, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.20690% with 4 lines in your changes missing coverage. Please review.
✅ Project coverage is 83.09%. Comparing base (2d37097) to head (307ce2c).

Files with missing lines Patch % Lines
copy.go 86.20% 2 Missing and 2 partials ⚠️
Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1212   +/-   ##
=======================================
  Coverage   83.09%   83.09%           
=======================================
  Files          82       82           
  Lines        5808     5834   +26     
=======================================
+ Hits         4826     4848   +22     
- Misses        603      605    +2     
- Partials      379      381    +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@TerryHowe TerryHowe left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/lgtm

@Orexii

Orexii commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@TerryHowe anything needed to do on my end for this? Please let me know. Thanks

@TerryHowe
TerryHowe merged commit e1399f8 into oras-project:main Jul 15, 2026
9 checks passed
TerryHowe added a commit that referenced this pull request Jul 15, 2026
…them missing (#1213)

Backport of #1212 to the `v2` branch.

## What this does
Makes `CopyGraph` self-heal a partially-copied graph. When the
destination rejects a manifest push because it is missing referenced
content (`MANIFEST_BLOB_UNKNOWN` / `BLOB_UNKNOWN` / `MANIFEST_UNKNOWN`),
the copy now re-pushes the node's manifest successors and retries the
push once, instead of returning the error.

## Why
`CopyGraph` skips a node whose sub-DAG it believes already exists ("skip
if a rooted sub-DAG exists"). If a prior copy was interrupted after a
child manifest's blobs were uploaded but before the child manifest was
committed, the destination can report the child as present
(`Exists`/HEAD) yet reject the parent that references it. For a
multi-arch image index this means the index push fails with
`MANIFEST_BLOB_UNKNOWN`, and since the child is skipped on every run,
re-running the same `oras cp --recursive` never recovers. This is the
line that reaches the ORAS CLI.

This was hit in production mirroring a multi-arch image (`mongo:7.0.34`)
between two registries; the only manual workaround was copying each
child manifest by digest and re-pushing the index.

## How
- `copyGraph`: wrap the node push; on a "missing referenced content"
error, re-copy the manifest successors unconditionally (bypassing the
existence short-circuit) and retry the push once. No-op on the success
path; blob successors are not re-pushed.
- `isMissingReferencedContentError`: typed classification via `errcode`
(handles both a single `errcode.Error` and a multi-error
`errcode.Errors`); no string matching.

## Testing
- `TestCopyGraph_SelfHealsInterruptedIndexCopy`: end-to-end reproduction
against a destination that emulates the interrupted state — fails
without this change (`MANIFEST_BLOB_UNKNOWN`), passes with it.
- `TestIsMissingReferencedContentError`: unit test for the classifier
(100% covered).
- `go test -race ./...` passes; `gofmt` / `go vet` clean; new files
carry the Apache license header.

The diff is identical to #1212 apart from the module import paths
(`github.com/oras-project/oras-go/v3` -> `oras.land/oras-go/v2`).

Relates to #1211.

Signed-off-by: Malcolm Morgan <Malcolm.Morgan@Microsoft.com>
Co-authored-by: Malcolm Morgan <Malcolm.Morgan@Microsoft.com>
Co-authored-by: Terry Howe <terrylhowe@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CopyGraph fails to self-heal a partially-copied multi-arch index (MANIFEST_BLOB_UNKNOWN on retry)

2 participants