feat(jwt-asm): support AWS Secrets Manager for JWT verification by matheuscscp · Pull Request #3763 · project-zot/zot

matheuscscp · 2026-02-01T20:46:54Z

Closes: #3756

The e2e test uses LocalStack to run in CI. I also ran the script locally with the --use-real-aws flag and it works:

matheuscscp · 2026-02-01T21:04:29Z

I have also added a temporary GH Actions Workflow to test the nightly job in this PR and removed it:

https://github.com/project-zot/zot/actions/runs/21570015300/job/62147554369?pr=3763

matheuscscp · 2026-02-01T21:10:51Z

AWS Secrets Manager Bearer Auth E2E Test Script

This commit adds an e2e test script (examples/aws-secrets-manager-bearer-auth.sh) that validates JWT bearer authentication using EdDSA keys stored in AWS Secrets Manager. The script uses LocalStack by default (no real AWS credentials needed), with an opt-in --use-real-aws flag for testing against real AWS.

How It Works

Starts a LocalStack container (or uses real AWS with --use-real-aws)
Stores a hardcoded Ed25519 JWKS keypair in AWS Secrets Manager
Builds and starts zot configured with awsSecretsManager bearer auth
Signs JWTs with different access claim scopes using a Go helper (examples/jwt-sign/main.go)
Runs 10 tests using curl and crane to validate authentication and authorization
Cleans up all resources (secret, container, zot process, temp files)

Flags for Local Iteration

--use-real-aws - Use real AWS credentials instead of LocalStack
--skip-build - Skip building zot (reuse existing binary)
--keep-resources - Keep resources running after exit for debugging
--region REGION - AWS region (default: us-east-1)

Test Cases

Test	Type	Description	What it Proves
1	curl	Auth succeeds with valid token	Valid EdDSA JWT accepted via ASM public key (HTTP 200)
2	curl	Auth fails without token	Unauthenticated requests are blocked (HTTP 401)
3	curl	Auth fails with invalid token	Malformed tokens are rejected (HTTP 401)
4	crane	Push image with push+pull token	Real OCI push succeeds with `push,pull` access claim
5	crane	List tags with pull-only token	Real OCI tag listing succeeds with `pull` access claim
6	crane	Pull manifest with pull-only token	Real OCI manifest read succeeds with `pull` access claim
7	crane	Push fails with pull-only token	Write operation rejected when token only grants `pull` (HTTP 401)
8	crane	Operations fail without token	Unauthenticated crane operations fail
9	curl	Token with wrong kid rejected	JWT signed with unknown kid is rejected (HTTP 401)
10	curl	Access to unauthorized repo fails	Token scoped to `test-repo` cannot access `other-repo` (HTTP 401)

Key validations:

Tests 1-3, 9-10 use curl to validate basic authentication and token verification
Tests 4-8 use the crane CLI with registryToken in Docker config for direct Bearer token auth
Tests 7 and 10 validate that authorization is enforced via JWT access claims (scope-based, not accessControl config)
Test 9 validates kid-based key selection from the JWKS stored in Secrets Manager

Raw Test Logs - LocalStack (click to expand)

[INFO] Using LocalStack mode (no real AWS credentials needed)
[INFO] Starting LocalStack...
e61eb8b6b434baf319006f0bc7c9c61d6f38afb31b8fa97f401727c0dfb32210
[INFO] Waiting for LocalStack to be ready...
[INFO] LocalStack is ready
[INFO] Creating secret 'zot/e2e-test-jwt-keys-1769980144' in Secrets Manager (us-east-1)...
{
  "ARN": "arn:aws:secretsmanager:us-east-1:000000000000:secret:zot/e2e-test-jwt-keys-1769980144-aMlAOa",
  "Name": "zot/e2e-test-jwt-keys-1769980144",
  "VersionId": "9e997e48-495c-4c30-89ae-efcce06729ee"
}
[INFO] Secret created successfully
[INFO] Verifying secret retrieval...
[INFO] Secret verification passed
[INFO] Building zot...
[INFO] Zot built: /tmp/zot-asm-test
[INFO] Creating zot configuration...
[INFO] Zot configuration:
{
  "distSpecVersion": "1.1.1",
  "storage": {
    "rootDirectory": "/tmp/zot-asm-storage"
  },
  "http": {
    "address": "127.0.0.1",
    "port": "5000",
    "compat": [
      "docker2s2"
    ],
    "auth": {
      "bearer": {
        "realm": "zot",
        "service": "zot-service",
        "awsSecretsManager": {
          "region": "us-east-1",
          "secretName": "zot/e2e-test-jwt-keys-1769980144",
          "refreshInterval": "30s"
        }
      }
    }
  },
  "log": {
    "level": "debug"
  }
}
[INFO] Starting zot...
[INFO] Zot started with PID 3946748
[INFO] Waiting for zot to be ready...
{"time":"2026-02-01T21:09:14.077039827Z","level":"info","message":"configuration settings","params":{"distSpecVersion":"1.1.1","GoVersion":"","Commit":"","ReleaseTag":"","BinaryType":"","Storage":{"RootDirectory":"/tmp/zot-asm-storage","Dedupe":true,"RemoteCache":false,"GC":true,"Commit":false,"GCDelay":3600000000000,"GCInterval":3600000000000,"Retention":{"DryRun":false,"Delay":3600000000000,"Policies":null},"StorageDriver":null,"CacheDriver":null,"GCMaxSchedulerDelay":0,"SubPaths":null},"HTTP":{"Address":"127.0.0.1","ExternalURL":"","Port":"5000","AllowOrigin":"","TLS":null,"Auth":{"FailDelay":0,"HTPasswd":{"Path":""},"LDAP":null,"Bearer":{"Realm":"zot","Service":"zot-service","Cert":"","awsSecretsManager":{"region":"us-east-1","secretName":"zot/e2e-test-jwt-keys-1769980144","refreshInterval":30000000000}},"OpenID":null,"APIKey":false,"SessionKeysFile":"","SessionDriver":null},"AccessControl":null,"Realm":"","Ratelimit":null,"Compat":["docker2s2"]},"Log":{"Level":"debug","Output":"","Audit":""},"Extensions":null,"scheduler":null,"cluster":null},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/controller.go:284","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-02-01T21:09:14.077250566Z","level":"info","message":"jwt bearer authentication","enabled":true,"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/controller.go:288","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-02-01T21:09:14.077292216Z","level":"info","message":"oidc bearer authentication","enabled":false,"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/controller.go:289","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-02-01T21:09:14.07732969Z","level":"info","message":"basic authentication (htpasswd)","enabled":false,"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/controller.go:290","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-02-01T21:09:14.077366717Z","level":"info","message":"basic authentication (LDAP)","enabled":false,"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/controller.go:291","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-02-01T21:09:14.077404604Z","level":"info","message":"basic authentication (API key)","enabled":false,"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/controller.go:292","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-02-01T21:09:14.077442197Z","level":"info","message":"OpenID authentication","enabled":false,"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/controller.go:293","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-02-01T21:09:14.077479836Z","level":"info","message":"mutual TLS authentication","enabled":false,"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/controller.go:294","func":"zotregistry.dev/zot/v2/pkg/api.(*Controller).Init","goroutine":1}
{"time":"2026-02-01T21:09:14.077631992Z","level":"info","message":"runtime params","cpus":14,"max. open files":1048576,"listen backlog":"4096","max. inotify watches":"524288","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/runtime.go:18","func":"zotregistry.dev/zot/v2/pkg/api.DumpRuntimeParams","goroutine":1}
{"time":"2026-02-01T21:09:14.077739681Z","level":"info","message":"events disabled in configuration","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/extensions/extension_events_disabled.go:16","func":"zotregistry.dev/zot/v2/pkg/extensions.NewEventRecorder","goroutine":1}
{"time":"2026-02-01T21:09:14.077788255Z","level":"warn","message":"lint extension is disabled because given zot binary doesn't include this feature please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/extensions/extensions_lint_disabled.go:12","func":"zotregistry.dev/zot/v2/pkg/extensions.GetLinter","goroutine":1}
{"time":"2026-02-01T21:09:14.114514759Z","level":"debug","message":"started","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/common/healthz.go:63","func":"zotregistry.dev/zot/v2/pkg/common.(*Healthz).Started","goroutine":1}
{"time":"2026-02-01T21:09:14.115461872Z","level":"warn","message":"skipping enabling search extension because given zot binary doesn't include this feature,please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/extensions/extension_search_disabled.go:31","func":"zotregistry.dev/zot/v2/pkg/extensions.EnableSearchExtension","goroutine":1}
{"time":"2026-02-01T21:09:14.115631861Z","level":"warn","message":"skipping enabling metrics extension because given zot binary doesn't include this feature,please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/extensions/extension_metrics_disabled.go:18","func":"zotregistry.dev/zot/v2/pkg/extensions.EnableMetricsExtension","goroutine":1}
{"time":"2026-02-01T21:09:14.115692497Z","level":"warn","message":"skipping enabling scrub extension because given zot binary doesn't include this feature,please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/extensions/extension_scrub_disabled.go:16","func":"zotregistry.dev/zot/v2/pkg/extensions.EnableScrubExtension","goroutine":1}
{"time":"2026-02-01T21:09:14.115740666Z","level":"warn","message":"skipping enabling sync extension because given zot binary doesn't include this feature,please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/extensions/extension_sync_disabled.go:18","func":"zotregistry.dev/zot/v2/pkg/extensions.EnableSyncExtension","goroutine":1}
{"time":"2026-02-01T21:09:14.115806184Z","level":"warn","message":"skipping adding to the scheduler a generator for updating signatures validity because given binary doesn't include this feature, please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/extensions/extension_image_trust_disabled.go:26","func":"zotregistry.dev/zot/v2/pkg/extensions.EnableImageTrustVerification","goroutine":1}
{"time":"2026-02-01T21:09:14.11583055Z","level":"debug","message":"generator is done","component":"scheduler","generator":"GCTaskGenerator","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/scheduler/scheduler.go:478","func":"zotregistry.dev/zot/v2/pkg/scheduler.(*generator).generate","goroutine":43}
{"time":"2026-02-01T21:09:14.116550784Z","level":"info","message":"the AWS Secrets Manager JWT verification was enabled","region":"us-east-1","secretName":"zot/e2e-test-jwt-keys-1769980144","refreshInterval":30000000000,"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/bearer_aws_secrets_manager.go:102","func":"zotregistry.dev/zot/v2/pkg/api.NewAWSSecretsManager","goroutine":1}
{"time":"2026-02-01T21:09:14.118745578Z","level":"warn","message":"skipping enabling swagger because given zot binary doesn't include this feature, please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/debug/swagger/swagger_disabled.go:19","func":"zotregistry.dev/zot/v2/pkg/debug/swagger.SetupSwaggerRoutes","goroutine":1}
{"time":"2026-02-01T21:09:14.118858883Z","level":"warn","message":"skipping enabling graphql playground extension because given zot binary doesn't include this feature, please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/debug/gqlplayground/gqlplayground_disabled.go:16","func":"zotregistry.dev/zot/v2/pkg/debug/gqlplayground.SetupGQLPlaygroundRoutes","goroutine":1}
{"time":"2026-02-01T21:09:14.118941699Z","level":"warn","message":"skipping enabling pprof extension because given zot binary doesn't include this feature, please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/debug/pprof/pprof_disabled.go:15","func":"zotregistry.dev/zot/v2/pkg/debug/pprof.SetupPprofRoutes","goroutine":1}
{"time":"2026-02-01T21:09:14.119066474Z","level":"warn","message":"skipping setting up search routes because given zot binary doesn't include this feature,please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/extensions/extension_search_disabled.go:39","func":"zotregistry.dev/zot/v2/pkg/extensions.SetupSearchRoutes","goroutine":1}
{"time":"2026-02-01T21:09:14.119131502Z","level":"warn","message":"skipping setting up image trust routes because given zot binary doesn't include this feature,please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/extensions/extension_image_trust_disabled.go:19","func":"zotregistry.dev/zot/v2/pkg/extensions.SetupImageTrustRoutes","goroutine":1}
{"time":"2026-02-01T21:09:14.119213349Z","level":"warn","message":"skipping setting up mgmt routes because given zot binary doesn't include this feature,please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/extensions/extension_mgmt_disabled.go:17","func":"zotregistry.dev/zot/v2/pkg/extensions.SetupMgmtRoutes","goroutine":1}
{"time":"2026-02-01T21:09:14.119275891Z","level":"warn","message":"userprefs extension is disabled because given zot binary doesn't include this feature please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/extensions/extension_userprefs_disable.go:20","func":"zotregistry.dev/zot/v2/pkg/extensions.SetupUserPreferencesRoutes","goroutine":1}
{"time":"2026-02-01T21:09:14.119346082Z","level":"warn","message":"skipping setting up ui routes because given zot binary doesn't include this feature,please build a binary that does so","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/extensions/extension_ui_disabled.go:15","func":"zotregistry.dev/zot/v2/pkg/extensions.SetupUIRoutes","goroutine":1}
{"time":"2026-02-01T21:09:14.119922177Z","level":"debug","message":"ready","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/common/healthz.go:58","func":"zotregistry.dev/zot/v2/pkg/common.(*Healthz).Ready","goroutine":1}
{"time":"2026-02-01T21:09:14.215783211Z","level":"info","message":"no repositories found in storage, finished.","component":"dedupe","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/storage/common/common.go:943","func":"zotregistry.dev/zot/v2/pkg/storage/common.(*DedupeTaskGenerator).Next","goroutine":43}
{"time":"2026-02-01T21:09:14.215951657Z","level":"debug","message":"generator is done","component":"scheduler","generator":"DedupeTaskGenerator","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/scheduler/scheduler.go:478","func":"zotregistry.dev/zot/v2/pkg/scheduler.(*generator).generate","goroutine":43}
{"time":"2026-02-01T21:09:14.941368428Z","level":"debug","message":"bearer token authorization failed","error":"no bearer token given","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func2.1","goroutine":114}
{"time":"2026-02-01T21:09:14.941794347Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49070","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept":["*/*"],"User-Agent":["curl/8.5.0"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":114}
[INFO] Zot is responding (HTTP 401)
[INFO] Signing test JWTs...
[INFO] Signed JWT for push+pull to test-repo
[INFO] Signed JWT for pull-only to test-repo
[INFO] Signed JWT with no access claims
[INFO] TEST 1: Verifying basic authentication with valid token...
{"time":"2026-02-01T21:09:15.34309259Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49080","method":"GET","path":"/v2/","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Accept":["*/*"],"Authorization":["******"],"User-Agent":["curl/8.5.0"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":116}
[INFO] TEST 1 PASSED: Authentication succeeded (HTTP 200)
[INFO] TEST 2: Verifying authentication fails without token...
{"time":"2026-02-01T21:09:15.359645481Z","level":"debug","message":"bearer token authorization failed","error":"no bearer token given","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func2.1","goroutine":25}
{"time":"2026-02-01T21:09:15.359868779Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49082","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept":["*/*"],"User-Agent":["curl/8.5.0"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":25}
[INFO] TEST 2 PASSED: No token correctly rejected (HTTP 401)
[INFO] TEST 3: Verifying authentication fails with invalid token...
{"time":"2026-02-01T21:09:15.377254749Z","level":"error","message":"failed to parse Authorization header","error":"invalid bearer token given: token is malformed: could not JSON decode header: invalid character '\\u008a' looking for beginning of value","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/authn.go:647","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func2.1","goroutine":27}
{"time":"2026-02-01T21:09:15.377480024Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49094","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":182,"headers":{"Accept":["*/*"],"Authorization":["******"],"User-Agent":["curl/8.5.0"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":27}
[INFO] TEST 3 PASSED: Invalid token correctly rejected (HTTP 401)
[INFO] TEST 4: Pushing OCI image using crane with push+pull token...
{"time":"2026-02-01T21:09:16.195329387Z","level":"debug","message":"bearer token authorization failed","error":"no bearer token given","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func2.1","goroutine":124}
{"time":"2026-02-01T21:09:16.195569452Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49110","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":124}
{"time":"2026-02-01T21:09:16.197730419Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49110","method":"HEAD","path":"/v2/test-repo/manifests/v1","statusCode":404,"latency":"0s","bodySize":212,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json"],"Authorization":["******"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":124}
{"time":"2026-02-01T21:09:16.199657625Z","level":"warn","message":"not found in cache","error":"cache miss","digest":"sha256:54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/storage/imagestore/imagestore.go:1360","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).CheckBlob","goroutine":124}
{"time":"2026-02-01T21:09:16.199820662Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49110","method":"HEAD","path":"/v2/test-repo/blobs/sha256:54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971","statusCode":404,"latency":"0s","bodySize":405,"headers":{"Authorization":["******"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":124}
{"time":"2026-02-01T21:09:16.20003098Z","level":"warn","message":"not found in cache","error":"cache miss","digest":"sha256:aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/storage/imagestore/imagestore.go:1360","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).CheckBlob","goroutine":48}
{"time":"2026-02-01T21:09:16.200188638Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49114","method":"HEAD","path":"/v2/test-repo/blobs/sha256:aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066","statusCode":404,"latency":"0s","bodySize":405,"headers":{"Authorization":["******"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":48}
{"time":"2026-02-01T21:09:16.200837077Z","level":"warn","message":"not found in cache","error":"cache miss","digest":"sha256:54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/storage/imagestore/imagestore.go:1360","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).CheckBlob","goroutine":124}
{"time":"2026-02-01T21:09:16.201902105Z","level":"warn","message":"not found in cache","error":"cache miss","digest":"sha256:aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/storage/imagestore/imagestore.go:1360","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).CheckBlob","goroutine":48}
{"time":"2026-02-01T21:09:16.201917441Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49110","method":"POST","path":"/v2/test-repo/blobs/uploads/?from=google-containers%2Fbusybox&mount=sha256%3A54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971&origin=gcr.io","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["0"],"Content-Type":["application/json"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":124}
{"time":"2026-02-01T21:09:16.202246219Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49114","method":"POST","path":"/v2/test-repo/blobs/uploads/?from=google-containers%2Fbusybox&mount=sha256%3Aaab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066&origin=gcr.io","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["0"],"Content-Type":["application/json"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":48}
{"time":"2026-02-01T21:09:16.820613988Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49110","method":"PATCH","path":"/v2/test-repo/blobs/uploads/cf08ff17-a2db-45c7-bb7b-69568d5310db","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["1497"],"Content-Type":["application/octet-stream"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":124}
{"time":"2026-02-01T21:09:16.82221099Z","level":"debug","message":"dedupe begin","src":"/tmp/zot-asm-storage/test-repo/.uploads/cf08ff17-a2db-45c7-bb7b-69568d5310db","dstDigest":"sha256:54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971","dst":"/tmp/zot-asm-storage/test-repo/blobs/sha256/54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/storage/imagestore/imagestore.go:1154","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).DedupeBlob","goroutine":124}
{"time":"2026-02-01T21:09:16.842243012Z","level":"debug","message":"rename","src":"/tmp/zot-asm-storage/test-repo/.uploads/cf08ff17-a2db-45c7-bb7b-69568d5310db","dst":"/tmp/zot-asm-storage/test-repo/blobs/sha256/54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971","component":"dedupe","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/storage/imagestore/imagestore.go:1180","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).DedupeBlob","goroutine":124}
{"time":"2026-02-01T21:09:16.842466139Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49110","method":"PUT","path":"/v2/test-repo/blobs/uploads/cf08ff17-a2db-45c7-bb7b-69568d5310db?digest=sha256%3A54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971","statusCode":201,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["0"],"Content-Type":["application/octet-stream"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":124}
{"time":"2026-02-01T21:09:16.913148814Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49114","method":"PATCH","path":"/v2/test-repo/blobs/uploads/5188ac1f-0060-4600-86b6-61fc494ae3fa","statusCode":202,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["692336"],"Content-Type":["application/octet-stream"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":48}
{"time":"2026-02-01T21:09:16.915975736Z","level":"debug","message":"dedupe begin","src":"/tmp/zot-asm-storage/test-repo/.uploads/5188ac1f-0060-4600-86b6-61fc494ae3fa","dstDigest":"sha256:aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066","dst":"/tmp/zot-asm-storage/test-repo/blobs/sha256/aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/storage/imagestore/imagestore.go:1154","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).DedupeBlob","goroutine":48}
{"time":"2026-02-01T21:09:16.927650871Z","level":"debug","message":"rename","src":"/tmp/zot-asm-storage/test-repo/.uploads/5188ac1f-0060-4600-86b6-61fc494ae3fa","dst":"/tmp/zot-asm-storage/test-repo/blobs/sha256/aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066","component":"dedupe","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/storage/imagestore/imagestore.go:1180","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).DedupeBlob","goroutine":48}
{"time":"2026-02-01T21:09:16.927859288Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49114","method":"PUT","path":"/v2/test-repo/blobs/uploads/5188ac1f-0060-4600-86b6-61fc494ae3fa?digest=sha256%3Aaab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066","statusCode":201,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["0"],"Content-Type":["application/octet-stream"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":48}
{"time":"2026-02-01T21:09:16.930201587Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49114","method":"PUT","path":"/v2/test-repo/manifests/v1","statusCode":201,"latency":"0s","bodySize":0,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["527"],"Content-Type":["application/vnd.docker.distribution.manifest.v2+json"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":48}
[INFO] TEST 4 PASSED: crane copy succeeded with push+pull token
[INFO] Output: 2026/02/01 21:09:16 pushed blob: sha256:54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971
2026/02/01 21:09:16 pushed blob: sha256:aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066
2026/02/01 21:09:16 127.0.0.1:5000/test-repo:v1: digest: sha256:545e6a6310a27636260920bc07b994a299b6708a1b26910cfefd335fdfb60d2b size: 527
[INFO] TEST 5: Listing tags using crane with pull-only token...
{"time":"2026-02-01T21:09:16.959674736Z","level":"debug","message":"bearer token authorization failed","error":"no bearer token given","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func2.1","goroutine":33}
{"time":"2026-02-01T21:09:16.959904924Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49138","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":33}
{"time":"2026-02-01T21:09:16.961405196Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49138","method":"GET","path":"/v2/test-repo/tags/list?n=1000","statusCode":200,"latency":"0s","bodySize":34,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":33}
[INFO] TEST 5 PASSED: crane ls succeeded and found tag 'v1'
[INFO] Tags: v1
[INFO] TEST 6: Pulling manifest using crane with pull-only token...
{"time":"2026-02-01T21:09:16.984820994Z","level":"debug","message":"bearer token authorization failed","error":"no bearer token given","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func2.1","goroutine":165}
{"time":"2026-02-01T21:09:16.985043732Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49150","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":165}
{"time":"2026-02-01T21:09:16.986468476Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49150","method":"GET","path":"/v2/test-repo/manifests/v1","statusCode":200,"latency":"0s","bodySize":527,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json"],"Accept-Encoding":["gzip"],"Authorization":["******"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":165}
[INFO] TEST 6 PASSED: crane manifest succeeded with pull token
[INFO] Manifest preview: {
   "schemaVersion": 2,
   "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
   "config": {
      "mediaType": "application/vnd.docker.container.image.v1+json",
[INFO] TEST 7: Verifying push fails with pull-only token...
{"time":"2026-02-01T21:09:17.731269984Z","level":"debug","message":"bearer token authorization failed","error":"no bearer token given","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func2.1","goroutine":148}
{"time":"2026-02-01T21:09:17.731518657Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49172","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":148}
{"time":"2026-02-01T21:09:17.73290658Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49172","method":"HEAD","path":"/v2/test-repo/manifests/v2","statusCode":404,"latency":"0s","bodySize":212,"headers":{"Accept":["application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json"],"Authorization":["******"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":148}
{"time":"2026-02-01T21:09:17.734462002Z","level":"debug","message":"blob path found","blob path":"/tmp/zot-asm-storage/test-repo/blobs/sha256/54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/storage/imagestore/imagestore.go:1344","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).CheckBlob","goroutine":148}
{"time":"2026-02-01T21:09:17.734632642Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49172","method":"HEAD","path":"/v2/test-repo/blobs/sha256:54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Authorization":["******"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":148}
{"time":"2026-02-01T21:09:17.734710826Z","level":"debug","message":"blob path found","blob path":"/tmp/zot-asm-storage/test-repo/blobs/sha256/aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/storage/imagestore/imagestore.go:1344","func":"zotregistry.dev/zot/v2/pkg/storage/imagestore.(*ImageStore).CheckBlob","goroutine":131}
{"time":"2026-02-01T21:09:17.734844054Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49184","method":"HEAD","path":"/v2/test-repo/blobs/sha256:aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066","statusCode":200,"latency":"0s","bodySize":0,"headers":{"Authorization":["******"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":131}
{"time":"2026-02-01T21:09:17.735676203Z","level":"debug","message":"bearer token authorization failed","error":"bearer token does not have sufficient scope","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func2.1","goroutine":131}
{"time":"2026-02-01T21:09:17.73578756Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49184","method":"PUT","path":"/v2/test-repo/manifests/v2","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["527"],"Content-Type":["application/vnd.docker.distribution.manifest.v2+json"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":131}
{"time":"2026-02-01T21:09:17.737319937Z","level":"debug","message":"bearer token authorization failed","error":"bearer token does not have sufficient scope","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func2.1","goroutine":152}
{"time":"2026-02-01T21:09:17.737448824Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49190","method":"PUT","path":"/v2/test-repo/manifests/v2","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"Authorization":["******"],"Content-Length":["527"],"Content-Type":["application/vnd.docker.distribution.manifest.v2+json"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":152}
[INFO] TEST 7 PASSED: Push correctly rejected with pull-only token
[INFO] Output: 2026/02/01 21:09:17 existing blob: sha256:aab39f0bc16d3c109d7017bcbc13ee053b9b1b1c6985c432ec9b5dde1eb0d066
Error: PUT http://127.0.0.1:5000/v2/test-repo/manifests/v2: unexpected status code 401 Unauthorized: {"code":"UNAUTHORIZED","message":"authentication required","detail":{"description":"The access controller was unable to authenticate the client. Often this will be accompanied by a WWW-Authenticate HTTP response header indicating how to authenticate."}}
[INFO] TEST 8: Verifying crane operations fail without token...
{"time":"2026-02-01T21:09:17.764760102Z","level":"debug","message":"bearer token authorization failed","error":"no bearer token given","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func2.1","goroutine":155}
{"time":"2026-02-01T21:09:17.765023194Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49218","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept-Encoding":["gzip"],"User-Agent":["crane/0.20.6 go-containerregistry/0.20.6"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":155}
[INFO] TEST 8 PASSED: crane ls failed without token
[INFO] Output: Error: reading tags for 127.0.0.1:5000/test-repo: Get "zot?scope=repository%3Atest-repo%3Apull&service=zot-service": unsupported protocol scheme ""
[INFO] TEST 9: Verifying token with wrong kid is rejected...
{"time":"2026-02-01T21:09:17.928315639Z","level":"error","message":"failed to parse Authorization header","error":"invalid bearer token given: token is unverifiable: error while executing keyfunc: invalid bearer token given: no public key found for kid wrong-kid-does-not-exist","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/authn.go:647","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func2.1","goroutine":178}
{"time":"2026-02-01T21:09:17.928627336Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49234","method":"GET","path":"/v2/","statusCode":401,"latency":"0s","bodySize":182,"headers":{"Accept":["*/*"],"Authorization":["******"],"User-Agent":["curl/8.5.0"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":178}
[INFO] TEST 9 PASSED: Wrong kid correctly rejected (HTTP 401)
[INFO] TEST 10: Verifying access to unauthorized repository fails...
{"time":"2026-02-01T21:09:17.944225834Z","level":"debug","message":"bearer token authorization failed","error":"bearer token does not have sufficient scope","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/authn.go:639","func":"zotregistry.dev/zot/v2/pkg/api.bearerAuthHandler.func2.1","goroutine":134}
{"time":"2026-02-01T21:09:17.944412516Z","level":"info","message":"HTTP API","module":"http","component":"session","clientIP":"127.0.0.1:49250","method":"GET","path":"/v2/other-repo/tags/list","statusCode":401,"latency":"0s","bodySize":253,"headers":{"Accept":["*/*"],"Authorization":["******"],"User-Agent":["curl/8.5.0"]},"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/session.go:93","func":"zotregistry.dev/zot/v2/pkg/api.SessionLogger.func1.1","goroutine":134}
[INFO] TEST 10 PASSED: Access to unauthorized repo correctly rejected (HTTP 401)
[INFO] ==========================================
[INFO] All AWS Secrets Manager Bearer Auth tests PASSED!
[INFO] ==========================================
[INFO] 
[INFO] Options:
[INFO]   --use-real-aws    Use real AWS (default: LocalStack)
[INFO]   --skip-build      Skip building zot
[INFO]   --keep-resources  Keep resources after exit
[INFO]   --region REGION   AWS region (default: us-east-1)
[INFO] Cleaning up...
{"time":"2026-02-01T21:09:17.951924173Z","level":"info","message":"received signal","signal":15,"caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/cli/server/config_reloader.go:45","func":"zotregistry.dev/zot/v2/pkg/cli/server.signalHandler","goroutine":53}
{"time":"2026-02-01T21:09:17.951940112Z","level":"debug","message":"htpasswd watcher terminating...","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/api/htpasswd.go:216","func":"zotregistry.dev/zot/v2/pkg/api.(*HTPasswdWatcher).Run.func1","goroutine":46}
{"time":"2026-02-01T21:09:17.965317347Z","level":"debug","message":"received stop signal, gracefully shutting down...","component":"scheduler","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/scheduler/scheduler.go:270","func":"zotregistry.dev/zot/v2/pkg/scheduler.(*Scheduler).RunScheduler.func1","goroutine":43}
{"time":"2026-02-01T21:09:17.965633373Z","level":"error","message":"failed to start controller, exiting","error":"http: Server closed","caller":"/home/matheus/Documents/github.com/project-zot/zot/pkg/cli/server/root.go:87","func":"zotregistry.dev/zot/v2/pkg/cli/server.NewServerRootCmd.newServeCmd.func2","goroutine":1}
localstack-zot-asm-test

codecov · 2026-02-01T21:34:07Z

Codecov Report

❌ Patch coverage is 94.82759% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.63%. Comparing base (7f629b5) to head (26254e8).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/api/authn.go	71.42%	5 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3763      +/-   ##
==========================================
+ Coverage   91.60%   91.63%   +0.02%     
==========================================
  Files         189      190       +1     
  Lines       26932    27040     +108     
==========================================
+ Hits        24672    24778     +106     
- Misses       1460     1462       +2     
  Partials      800      800

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

matheuscscp · 2026-02-01T23:06:05Z

Edit: Go script removed.

About the binary size increase, this time it's tiny: 1.8%

Copilot

Pull request overview

This PR adds support for loading JWT bearer verification keys from AWS Secrets Manager, enables JWKS-based public keys for bearer auth, and wires this into the bearer auth middleware plus CI E2E coverage.

Changes:

Extend auth configuration and bearer auth wiring to support AWS Secrets Manager as a dynamic source of JWT verification keys, alongside static certs.
Introduce AWSSecretsManagerAuthorizer plus JWKS-aware key loading, refactor BearerAuthorizer to use a context-aware key callback, and add comprehensive unit tests (including Ed25519/JWKS scenarios).
Add a jwt-sign helper, an AWS Secrets Manager bearer-auth E2E script (LocalStack or real AWS), documentation, and a new nightly workflow job.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
`pkg/api/config/config.go`	Adds `AWSSecretsManagerConfig` and related helpers so bearer auth can be configured to load verification keys from AWS Secrets Manager.
`pkg/api/bearer.go`	Refactors `BearerAuthorizer` to use a `BearerAuthorizerKeyFunc` with `context.Context`, enabling dynamic key lookup (including from AWS Secrets Manager) and JWKS-based verification.
`pkg/api/authn.go`	Integrates the new AWS Secrets Manager authorizer and JWKS-aware key loading into the bearer-auth middleware, enforces mutual exclusivity between `cert` and `awsSecretsManager`, and passes request contexts into `BearerAuthorizer`.
`pkg/api/bearer_aws_secrets_manager.go`	Implements `AWSSecretsManagerAuthorizer`, encapsulating config validation, AWS SDK client construction, key retrieval/caching, and `kid`-based key selection.
`pkg/api/bearer_test.go`	Updates bearer tests to the new context-aware key func, and adds Ed25519/JWKS tests (including a pre-signed external token) to validate JWKS support.
`pkg/api/bearer_aws_secrets_manager_test.go`	Provides unit tests and end-to-end tests for AWS Secrets Manager integration, covering config validation, caching, JWKS vs PEM parsing, error propagation, and end-to-end bearer auth with `kid` selection.
`examples/jwt-sign/main.go`	Adds a small CLI to sign Ed25519 JWTs from a JWKS payload, producing tokens with the `access` claim structure expected by `BearerAuthorizer`.
`examples/aws-secrets-manager-bearer-auth.sh`	Adds an E2E script that provisions a secret in AWS Secrets Manager (or LocalStack), starts Zot configured for ASM-backed bearer auth, signs tokens, and validates push/pull flows with `crane`.
`examples/README-AWS-SECRETS-MANAGER-BEARER-AUTH.md`	Documents how to configure Zot for AWS Secrets Manager-based JWT verification, secret formats (PEM and JWKS), key rotation, IAM permissions, and troubleshooting.
`.github/workflows/nightly.yaml`	Adds a nightly job that runs the new AWS Secrets Manager bearer-auth E2E script, ensuring this integration remains healthy in CI.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

matheuscscp · 2026-02-02T10:08:59Z

@rchincha Can you please retrigger the two red test jobs? I'm not sure it's related

rchincha · 2026-02-03T04:58:22Z

@matheuscscp the PR actually lgtm.
Let's wait for other reviewers for their comments.

Copilot

Pull request overview

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Signed-off-by: Matheus Pimenta <matheuscscp@gmail.com>

matheuscscp · 2026-02-03T09:03:51Z

Thanks @rchincha! I have rebased and fixed one of copilot's comments 🙏

matheuscscp requested review from andaaron and rchincha as code owners February 1, 2026 20:46

matheuscscp force-pushed the jwt-asm-verification branch 2 times, most recently from 96a9a29 to f6e67af Compare February 1, 2026 21:04

matheuscscp force-pushed the jwt-asm-verification branch from a47a331 to e0b418e Compare February 1, 2026 21:14

matheuscscp force-pushed the jwt-asm-verification branch 3 times, most recently from 04c599b to a1b9cba Compare February 1, 2026 22:38

rchincha requested a review from Copilot February 2, 2026 05:01

Copilot started reviewing on behalf of rchincha February 2, 2026 05:01 View session

rchincha reviewed Feb 2, 2026

View reviewed changes

Comment thread examples/jwt-sign/main.go Outdated

Copilot AI reviewed Feb 2, 2026

View reviewed changes

Comment thread examples/jwt-sign/main.go Outdated

Comment thread pkg/api/bearer_aws_secrets_manager.go

Comment thread pkg/api/bearer_aws_secrets_manager.go Outdated

Comment thread examples/aws-secrets-manager-bearer-auth.sh Outdated

matheuscscp force-pushed the jwt-asm-verification branch from a1b9cba to 831181f Compare February 2, 2026 09:40

matheuscscp requested a review from rchincha February 2, 2026 13:17

matheuscscp force-pushed the jwt-asm-verification branch 2 times, most recently from 604d7d0 to f0ab21d Compare February 2, 2026 16:31

andaaron reviewed Feb 2, 2026

View reviewed changes

Comment thread pkg/api/authn.go Outdated

Comment thread pkg/api/bearer_aws_secrets_manager.go Outdated

Comment thread pkg/api/bearer_aws_secrets_manager.go

matheuscscp force-pushed the jwt-asm-verification branch 2 times, most recently from 379d75a to ee25a30 Compare February 2, 2026 22:59

matheuscscp requested a review from andaaron February 2, 2026 23:03

rchincha reviewed Feb 3, 2026

View reviewed changes

Comment thread pkg/api/authn.go

rchincha requested a review from Copilot February 3, 2026 04:57

Copilot started reviewing on behalf of rchincha February 3, 2026 04:58 View session

rchincha requested a review from vrajashkr February 3, 2026 04:58

Copilot AI reviewed Feb 3, 2026

View reviewed changes

Comment thread pkg/api/bearer_aws_secrets_manager.go

Comment thread pkg/api/config/config.go

feat(jwt-asm): support AWS Secrets Manager for JWT verification

26254e8

Signed-off-by: Matheus Pimenta <matheuscscp@gmail.com>

matheuscscp force-pushed the jwt-asm-verification branch from ee25a30 to 26254e8 Compare February 3, 2026 09:03

andaaron approved these changes Feb 3, 2026

View reviewed changes

matheuscscp requested a review from rchincha February 3, 2026 14:44

rchincha merged commit 0e5a339 into project-zot:main Feb 3, 2026
43 of 44 checks passed

Uh oh!

Conversation

matheuscscp commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

matheuscscp commented Feb 1, 2026

Uh oh!

matheuscscp commented Feb 1, 2026

AWS Secrets Manager Bearer Auth E2E Test Script

How It Works

Flags for Local Iteration

Test Cases

Uh oh!

codecov Bot commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

matheuscscp commented Feb 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

matheuscscp commented Feb 2, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rchincha commented Feb 3, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

matheuscscp commented Feb 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

matheuscscp commented Feb 1, 2026 •

edited

Loading

codecov Bot commented Feb 1, 2026 •

edited

Loading

matheuscscp commented Feb 1, 2026 •

edited

Loading

matheuscscp commented Feb 3, 2026 •

edited

Loading