refactor(test/blackbox): extract shared push/pull helpers by andaaron · Pull Request #4132 · project-zot/zot

andaaron · 2026-06-14T12:44:47Z

See commit messages. Closes #3727.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

codecov · 2026-06-14T13:06:58Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.55%. Comparing base (c18a4a9) to head (af735a7).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4132      +/-   ##
==========================================
- Coverage   91.58%   91.55%   -0.04%     
==========================================
  Files         200      200              
  Lines       29356    29356              
==========================================
- Hits        26886    26877       -9     
- Misses       1590     1598       +8     
- Partials      880      881       +1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot

Pull request overview

This PR refactors the blackbox BATS push/pull-style test suites by extracting shared logic into reusable helper scripts, reducing duplication across the standard, FIPS140, upgrade, and authenticated variants.

Changes:

Introduces shared helper libraries for push/pull flows (helpers_pushpull.bash) and authenticated flows (helpers_pushpull_authn.bash) and rewires existing BATS files to use them.
Simplifies helpers_upgrade.bash by delegating common push/pull operations to the shared helpers.
Splits out an explicit “delete image index” test case in the relevant suites.

Reviewed changes

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
test/blackbox/upgrade.bats	Uses shared pagination helper and adds a delete-image-index test hook.
test/blackbox/upgrade_minimal.bats	Adds a delete-image-index test hook.
test/blackbox/pushpull.bats	Switches to shared push/pull helpers and adds delete-image-index coverage.
test/blackbox/pushpull_authn.bats	Switches to shared authenticated push/pull helpers.
test/blackbox/helpers_upgrade.bash	Delegates common push/pull operations to `helpers_pushpull`.
test/blackbox/helpers_pushpull.bash	New shared helper library for push/pull-style blackbox tests.
test/blackbox/helpers_pushpull_authn.bash	New shared helper library for authenticated push/pull blackbox tests.
test/blackbox/fips140.bats	Switches to shared push/pull helpers (FIPS mode).
test/blackbox/fips140_authn.bats	Switches to shared authenticated helpers (FIPS mode) and deduplicates auth/push/pull logic.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 3 comments.

Move duplicated push/pull/regclient/oras/helm helper functions out of fips140.bats, pushpull.bats and helpers_upgrade.bash into a single helpers_pushpull.bash, then have all three suites load from there. The helpers now share verify_prerequisites, get_zot_port, common assertion helpers (catalog/tag presence, OCI index ref name) and the regclient pagination listing. helpers_upgrade.bash keeps only the test_release_* and test_new_* wrappers that compose those helpers. Net effect: ~1000 lines of duplicated test scaffolding removed across the four files; behavior of the existing test cases is preserved. Refs: project-zot#3727 Signed-off-by: Akash Kumar <meakash7902@gmail.com>

@test

…lpers Build on the shared helpers_pushpull.bash to remove more duplicated blackbox scaffolding: - Add pushpull_setup_file/pushpull_teardown/pushpull_teardown_file keyed on PUSHPULL_FIPS_MODE so pushpull.bats and fips140.bats only set the flag and delegate lifecycle, instead of each carrying a near-identical setup_file/teardown. - Add helpers_pushpull_authn.bash (loaded by pushpull_authn.bats and fips140_authn.bats) for shared htpasswd setup, FIPS vs non-FIPS config and teardown, and the regclient/OCI/ML test helpers; both authn suites collapse to one-line @test bodies keyed on PUSHPULL_AUTHN_FIPS_MODE. - Make each authn helper self-sufficient by performing its own regctl login, removing the hidden dependency on the first test having logged in. - Split the implicit manifest delete out of helper_pull_image_index_and_delete into a standalone helper_delete_manifest, surfaced as its own "delete image index" @test in the pushpull, fips140 and upgrade suites, so the delete is explicit and no longer an order-fragile side effect of a pull. Refs: project-zot#3727 Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

Set org.opencontainers.image.title on oras artifact push and verify pull both via oras pull and manifest/blob fetch. Add retry_until_success and use it for periodic notation/cosign signature sync checks on slow CI. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

Use curl --fail for manifest deletes so HTTP errors fail the test. Remove the duplicate regctl --format flag in helper_push_manifest_with_regclient. Harden helper_authn_ml_artifacts with run and status checks, using a binary-safe shell redirect for the ONNX artifact round-trip. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Write oras artifact and docker build files under BATS_TEST_TMPDIR instead of the test working directory, and check git/docker in verify_prerequisites for clearer failures when running bats directly. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 47 out of 47 changed files in this pull request and generated 1 comment.

vrajashkr

Looks good overall. Thanks for taking a look at this. Left some questions and comments.

vrajashkr · 2026-06-14T19:25:29Z

We could consider adding "Closes" for issue #3727

Drop the FIPS vs non-FIPS split between zot-log.json and zot.log in pushpull, authn, and upgrade helpers. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

Replace repeated command -v checks with a single loop over curl, jq, git, and docker. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

Call helper_* directly from upgrade.bats and upgrade_minimal.bats, load helpers_pushpull in those suites, and move teardown there. Remove helpers_upgrade.bash now that the release/new wrappers are gone. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

Drop parameter defaults from shared push/pull helpers and pass image, repository, and pagination values explicitly at each call site. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

Rename helper_push_docker_image to helper_build_docker_image_push_and_pull and update test titles to reflect build plus expected push/pull failures without the docker compatibility extension. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

regctl persists login and TLS settings on disk, so push/pull blackbox tests could leak state into ~/.regctl/config.json across suites. Point REGCTL_CONFIG at BATS_FILE_TMPDIR for pushpull, authn, and upgrade suites, configure TLS once in authn setup, and drop redundant logout and per-login TLS setup. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

Copilot

Pull request overview

Copilot reviewed 47 out of 47 changed files in this pull request and generated 2 comments.

Touch zot-log.json during upgrade setup and only cat it in teardown when the file exists, so a missing log cannot mask the underlying test failure. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

Copilot

Pull request overview

Copilot reviewed 47 out of 47 changed files in this pull request and generated no new comments.

Copilot

Pull request overview

Copilot reviewed 47 out of 47 changed files in this pull request and generated no new comments.

vrajashkr

LGTM

andaaron requested a review from Copilot June 14, 2026 13:03

Copilot started reviewing on behalf of andaaron June 14, 2026 13:03 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

Comment thread test/blackbox/helpers_pushpull.bash Outdated

Comment thread test/blackbox/helpers_pushpull.bash Outdated

Comment thread test/blackbox/helpers_pushpull_authn.bash Outdated

andaaron force-pushed the refactor/blackbox-dedupe-3727 branch from aac36e8 to b28744a Compare June 14, 2026 13:36

andaaron mentioned this pull request Jun 14, 2026

refactor(test/blackbox): extract shared push/pull helpers #4018

Closed

2 tasks

andaaron requested a review from Copilot June 14, 2026 13:45

Copilot started reviewing on behalf of andaaron June 14, 2026 13:46 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

Comment thread test/blackbox/helpers_pushpull_authn.bash Outdated

Comment thread test/blackbox/helpers_pushpull_authn.bash Outdated

andaaron force-pushed the refactor/blackbox-dedupe-3727 branch from b28744a to 521f9d3 Compare June 14, 2026 13:57

andaaron requested a review from Copilot June 14, 2026 13:58

Copilot started reviewing on behalf of andaaron June 14, 2026 13:58 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

Comment thread test/blackbox/helpers_pushpull.bash

Comment thread test/blackbox/helpers_pushpull.bash Outdated

Comment thread test/blackbox/helpers_pushpull.bash Outdated

AkashKumar7902 and others added 4 commits June 14, 2026 14:33

andaaron force-pushed the refactor/blackbox-dedupe-3727 branch from 34bf877 to c9a0621 Compare June 14, 2026 14:33

andaaron requested a review from Copilot June 14, 2026 14:34

Copilot started reviewing on behalf of andaaron June 14, 2026 14:34 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

Comment thread test/blackbox/helpers_pushpull_authn.bash

andaaron force-pushed the refactor/blackbox-dedupe-3727 branch from c9a0621 to 746d03f Compare June 14, 2026 14:50

andaaron requested a review from Copilot June 14, 2026 14:51

Copilot started reviewing on behalf of andaaron June 14, 2026 14:51 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

Comment thread test/blackbox/helpers_pushpull_authn.bash Outdated

Comment thread test/blackbox/helpers_pushpull.bash Outdated

andaaron force-pushed the refactor/blackbox-dedupe-3727 branch from 746d03f to 423a3f5 Compare June 14, 2026 15:07

andaaron requested a review from Copilot June 14, 2026 15:07

Copilot started reviewing on behalf of andaaron June 14, 2026 15:08 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

andaaron requested a review from Copilot June 14, 2026 15:26

Copilot started reviewing on behalf of andaaron June 14, 2026 15:26 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

Comment thread test/blackbox/helpers_pushpull_authn.bash

andaaron marked this pull request as ready for review June 14, 2026 15:50

andaaron requested a review from rchincha as a code owner June 14, 2026 15:50

vrajashkr reviewed Jun 14, 2026

View reviewed changes

andaaron added 7 commits June 14, 2026 19:45

fix(test/blackbox): unify blackbox log path to zot/zot-log.json

164ce40

Drop the FIPS vs non-FIPS split between zot-log.json and zot.log in pushpull, authn, and upgrade helpers. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

refactor(test/blackbox): loop over tools in verify_prerequisites

50c9938

Replace repeated command -v checks with a single loop over curl, jq, git, and docker. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

refactor(test/blackbox): fix exit code for retry_until_success

418b9c2

Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

refactor(test/blackbox): require explicit args on pushpull helpers

ed789cc

Drop parameter defaults from shared push/pull helpers and pass image, repository, and pagination values explicitly at each call site. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

andaaron requested a review from Copilot June 14, 2026 21:00

Copilot started reviewing on behalf of andaaron June 14, 2026 21:00 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

Comment thread test/blackbox/upgrade.bats

Comment thread test/blackbox/upgrade_minimal.bats

fix(test/blackbox): harden upgrade suite log dump on failure

af735a7

Touch zot-log.json during upgrade setup and only cat it in teardown when the file exists, so a missing log cannot mask the underlying test failure. Signed-off-by: Andrei Aaron <andreifdaaron@gmail.com>

andaaron requested a review from Copilot June 14, 2026 21:20

Copilot started reviewing on behalf of andaaron June 14, 2026 21:21 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

andaaron requested a review from vrajashkr June 14, 2026 21:37

rchincha requested a review from Copilot June 15, 2026 01:50

Copilot started reviewing on behalf of rchincha June 15, 2026 01:50 View session

rchincha approved these changes Jun 15, 2026

View reviewed changes

Copilot AI reviewed Jun 15, 2026

View reviewed changes

andaaron merged commit 95ce68c into project-zot:main Jun 15, 2026
44 of 45 checks passed

andaaron deleted the refactor/blackbox-dedupe-3727 branch June 15, 2026 07:56

vrajashkr reviewed Jun 15, 2026

View reviewed changes

andaaron mentioned this pull request Jun 17, 2026

refactor(storage)!: add a common blobstore to store all blobs (#4136) #4137

Draft

Uh oh!

Conversation

andaaron commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jun 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

vrajashkr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vrajashkr commented Jun 14, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

andaaron commented Jun 14, 2026 •

edited

Loading

codecov Bot commented Jun 14, 2026 •

edited

Loading