Skip to content

Commit a1b9cba

Browse files
matheuscscpmatheuscscp
committed
feat(jwt-asm): support AWS Secrets Manager for JWT verification
Signed-off-by: Matheus Pimenta <matheuscscp@gmail.com>
1 parent b9aad15 commit a1b9cba

10 files changed

Lines changed: 1845 additions & 23 deletions

‎.github/workflows/nightly.yaml‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,21 @@ jobs:
228228
sudo ./scripts/enable_userns.sh
229229
./examples/kind/kind-oidc-workload-identity.sh
230230
231+
aws-secrets-manager-bearer:
232+
name: AWS Secrets Manager Bearer Auth E2E
233+
runs-on: ubuntu-latest
234+
steps:
235+
- uses: actions/checkout@v6
236+
- uses: actions/setup-go@v6
237+
with:
238+
go-version: 1.25.x
239+
- name: Install crane
240+
run: |
241+
go install github.com/google/go-containerregistry/cmd/crane@latest
242+
- name: Run AWS Secrets Manager bearer auth tests
243+
run: |
244+
./examples/aws-secrets-manager-bearer-auth.sh
245+
231246
cloud-scale-out:
232247
name: s3+dynamodb scale-out
233248
runs-on: oracle-vm-16cpu-64gb-x86-64

‎examples/README-AWS-SECRETS-MANAGER-BEARER-AUTH.md‎

Lines changed: 279 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,279 @@
1+
# AWS Secrets Manager Bearer Authentication
2+
3+
This document describes how to configure Zot to retrieve JWT verification keys from AWS Secrets Manager, enabling dynamic key rotation without restarting the registry.
4+
5+
## Overview
6+
7+
AWS Secrets Manager bearer authentication allows Zot to retrieve public keys for JWT verification from AWS Secrets Manager instead of loading them from a static file on disk. This is useful when keys need to be rotated without downtime, or when the same keys are shared across multiple Zot instances.
8+
9+
Zot periodically refreshes the keys from AWS Secrets Manager, so key rotations are picked up automatically.
10+
11+
## Benefits
12+
13+
- **Dynamic Key Rotation**: Rotate public keys in AWS Secrets Manager without restarting Zot
14+
- **Centralized Key Management**: Manage verification keys for multiple Zot instances in a single place
15+
- **Lazy Loading**: Keys are fetched on first authentication, so startup is not blocked by network issues
16+
- **Caching**: Keys are cached locally and refreshed periodically to minimize API calls
17+
- **Multiple Keys**: Support multiple key IDs (`kid`) for seamless key rotation
18+
19+
## Configuration
20+
21+
### Basic Configuration
22+
23+
Add AWS Secrets Manager configuration to your bearer authentication settings:
24+
25+
```json
26+
{
27+
"http": {
28+
"auth": {
29+
"bearer": {
30+
"realm": "zot",
31+
"service": "zot-service",
32+
"awsSecretsManager": {
33+
"region": "us-east-1",
34+
"secretName": "zot/jwt-verification-keys"
35+
}
36+
}
37+
}
38+
}
39+
}
40+
```
41+
42+
### Configuration Options
43+
44+
- **`region`** (required): The AWS region where the secret is stored.
45+
- Example: `"us-east-1"`
46+
47+
- **`secretName`** (required): The name or ARN of the secret in AWS Secrets Manager.
48+
- Example: `"zot/jwt-verification-keys"`
49+
- Example: `"arn:aws:secretsmanager:us-east-1:123456789012:secret:zot/keys-AbCdEf"`
50+
51+
- **`refreshInterval`** (optional): How often to refresh keys from AWS Secrets Manager. Default: `1m` (1 minute).
52+
- Example: `"5m"` (5 minutes)
53+
- Example: `"30s"` (30 seconds)
54+
55+
## Secret Format
56+
57+
The secret stored in AWS Secrets Manager must be a JSON object where each key is a key ID (`kid`) and each value is either a PEM-encoded public key or a JWKS key set with a single key:
58+
59+
### PEM Format
60+
61+
```json
62+
{
63+
"key-id-1": "-----BEGIN PUBLIC KEY-----\nMCowBQYDK2VwAyEA...\n-----END PUBLIC KEY-----\n",
64+
"key-id-2": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG...\n-----END PUBLIC KEY-----\n"
65+
}
66+
```
67+
68+
### JWKS Format
69+
70+
Each value can also be a JWKS key set containing a single key:
71+
72+
```json
73+
{
74+
"key-id-1": "{\"keys\":[{\"kty\":\"OKP\",\"crv\":\"Ed25519\",\"x\":\"...\"}]}"
75+
}
76+
```
77+
78+
### Supported Key Types
79+
80+
- **Ed25519** (EdDSA) - Recommended for new deployments
81+
- **RSA** (RS256, RS384, RS512, PS256, PS384, PS512)
82+
- **ECDSA** (ES256, ES384, ES512)
83+
84+
## Complete Example
85+
86+
```json
87+
{
88+
"distSpecVersion": "1.1.1",
89+
"storage": {
90+
"rootDirectory": "/tmp/zot"
91+
},
92+
"http": {
93+
"address": "127.0.0.1",
94+
"port": "8080",
95+
"auth": {
96+
"bearer": {
97+
"realm": "zot",
98+
"service": "zot-service",
99+
"awsSecretsManager": {
100+
"region": "us-east-1",
101+
"secretName": "zot/jwt-verification-keys",
102+
"refreshInterval": "5m"
103+
}
104+
}
105+
},
106+
"accessControl": {
107+
"repositories": {
108+
"**": {
109+
"policies": [
110+
{
111+
"users": ["service-account-1"],
112+
"actions": ["read", "create", "update"]
113+
}
114+
]
115+
}
116+
}
117+
}
118+
},
119+
"log": {
120+
"level": "info"
121+
}
122+
}
123+
```
124+
125+
## JWT Token Requirements
126+
127+
JWTs presented to Zot must include a `kid` (Key ID) header that matches one of the key IDs in the secret. This is how Zot selects the correct public key for verification.
128+
129+
### Example JWT Header
130+
131+
```json
132+
{
133+
"alg": "EdDSA",
134+
"kid": "key-id-1",
135+
"typ": "JWT"
136+
}
137+
```
138+
139+
### Example JWT Payload
140+
141+
```json
142+
{
143+
"iss": "https://auth.example.com",
144+
"sub": "service-account-1",
145+
"aud": ["zot"],
146+
"exp": 1705258800,
147+
"iat": 1705255200,
148+
"access": [
149+
{
150+
"type": "repository",
151+
"name": "my-app",
152+
"actions": ["pull", "push"]
153+
}
154+
]
155+
}
156+
```
157+
158+
## Key Rotation
159+
160+
To rotate keys without downtime:
161+
162+
1. **Add the new key** to the secret in AWS Secrets Manager alongside the existing key(s):
163+
```json
164+
{
165+
"old-key-id": "-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----\n",
166+
"new-key-id": "-----BEGIN PUBLIC KEY-----\n...\n-----END PUBLIC KEY-----\n"
167+
}
168+
```
169+
170+
2. **Update your token issuer** to sign new tokens with the new key ID.
171+
172+
3. **Wait for the refresh interval** to elapse. Zot will automatically pick up the new key.
173+
174+
4. **Remove the old key** from the secret once all tokens signed with the old key have expired.
175+
176+
## Compatibility
177+
178+
### With OIDC Workload Identity
179+
180+
AWS Secrets Manager bearer authentication can coexist with OIDC workload identity. If both are configured, Zot will try OIDC authentication first, then fall back to traditional bearer token authentication using keys from AWS Secrets Manager:
181+
182+
```json
183+
{
184+
"http": {
185+
"auth": {
186+
"bearer": {
187+
"realm": "zot",
188+
"service": "zot-service",
189+
"awsSecretsManager": {
190+
"region": "us-east-1",
191+
"secretName": "zot/jwt-verification-keys"
192+
},
193+
"oidc": [
194+
{
195+
"issuer": "https://kubernetes.default.svc.cluster.local",
196+
"audiences": ["zot"]
197+
}
198+
]
199+
}
200+
}
201+
}
202+
}
203+
```
204+
205+
### With Static Certificate
206+
207+
AWS Secrets Manager and the static `cert` option are mutually exclusive. Zot will refuse to start if both are configured.
208+
209+
## AWS Authentication
210+
211+
Zot uses the default AWS credential chain to authenticate with AWS Secrets Manager. This means you can use any of the following:
212+
213+
- **Environment variables**: `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`
214+
- **Shared credentials file**: `~/.aws/credentials`
215+
- **IAM role** (for EC2, ECS, or EKS workloads)
216+
- **Web identity token** (for EKS with IRSA)
217+
218+
### Required IAM Permissions
219+
220+
The IAM principal used by Zot needs the following permission:
221+
222+
```json
223+
{
224+
"Version": "2012-10-17",
225+
"Statement": [
226+
{
227+
"Effect": "Allow",
228+
"Action": "secretsmanager:GetSecretValue",
229+
"Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:zot/jwt-verification-keys-*"
230+
}
231+
]
232+
}
233+
```
234+
235+
## Troubleshooting
236+
237+
### Enable Debug Logging
238+
239+
Set log level to `debug` to see detailed authentication logs:
240+
241+
```json
242+
{
243+
"log": {
244+
"level": "debug"
245+
}
246+
}
247+
```
248+
249+
### Common Issues
250+
251+
1. **"region must be specified"**: The `region` field is required in the configuration.
252+
253+
2. **"secret name must be specified"**: The `secretName` field is required in the configuration.
254+
255+
3. **"failed to load AWS configuration"**: Check that AWS credentials are available and valid.
256+
257+
4. **"failed to retrieve secret"**: Verify the secret exists in the specified region and the IAM principal has `secretsmanager:GetSecretValue` permission.
258+
259+
5. **"failed to parse secret JSON"**: The secret value must be a valid JSON object mapping key IDs to PEM-encoded public keys.
260+
261+
6. **"no public key found for kid"**: The JWT's `kid` header does not match any key ID in the secret. Check that the key ID in the JWT matches one of the keys in the secret.
262+
263+
7. **"token missing 'kid' header"**: JWTs must include a `kid` header when using AWS Secrets Manager keys.
264+
265+
8. **"cannot configure both cert and AWS Secrets Manager"**: The static `cert` and `awsSecretsManager` options are mutually exclusive. Remove one of them.
266+
267+
## Security Considerations
268+
269+
1. **Least Privilege**: Grant only `secretsmanager:GetSecretValue` permission, scoped to the specific secret ARN.
270+
271+
2. **Secret Encryption**: AWS Secrets Manager encrypts secrets at rest using KMS. Use a customer-managed KMS key for additional control.
272+
273+
3. **Refresh Interval**: Use a short refresh interval (e.g., 1-5 minutes) to pick up key rotations quickly. The default of 1 minute is appropriate for most use cases.
274+
275+
4. **Key Types**: Prefer Ed25519 (EdDSA) keys for new deployments. They are faster and produce smaller signatures than RSA.
276+
277+
5. **TLS**: Always use TLS for all communication to protect tokens in transit.
278+
279+
6. **Access Control**: Always configure access control policies to limit what authenticated clients can do.

0 commit comments

Comments
 (0)