Commit ef3a0ff
authored
feat: per-project envelope encryption, IDOR protection, material URL proxy, and E2E test suite (#460)
* feat(api): per-project envelope encryption, material URL proxy, and IDOR protection
Encryption architecture:
- AES-256-GCM envelope encryption with per-object random DEKs
- Zero-knowledge key hierarchy: master key embedded in API token
(sk-ac-{authSecret}.{encryptedMasterKey}), never stored server-side
- HKDF-SHA256 key derivation from auth secret + pepper
- Idempotent RewrapDEK for crash-safe key rotation (O(1), no S3 re-encryption)
- Redis cache encryption with prefix byte scheme (0x00=plain, 0x01=encrypted)
- DB content encryption via EncodeContent/DecodeContent binary framing
Material URL proxy:
- Replaces presigned S3 URLs with Redis-backed decryption proxy
- GET /api/v1/material/{token} — unauthenticated, token is the credential
- Enables transparent download of encrypted S3 objects
IDOR protection:
- Cross-project ownership validation on all resource endpoints
- Defense-in-depth checks at both handler and service layers
- diskRepo.GetByProjectAndID, session.ProjectID guards, S3 key prefix checks
Admin encryption lifecycle:
- POST /admin/v1/project/encrypt — idempotent batch encryption
- POST /admin/v1/project/decrypt — batch decryption
- PUT /admin/v1/project/secret_key (Bearer) — preserves master key
- PUT /admin/v1/project/:id/secret_key (JWT) — blocked for encrypted projects
* feat(core): thread user_kek through Python CORE for end-to-end encryption
- Add crypto.py: AES-256-GCM envelope encryption (Go-compatible wire format)
- S3 client: auto-encrypt/decrypt with user_kek on upload/download
- MQ consumers: hard-fail on invalid KEK (never silent plaintext fallback)
- Skill learner: SkillLearnerCtx carries KEK to all LLM tool handlers
- Artifact data: encode_content/decode_content with binary framing (mirrors Go)
- Sandbox backends: user_kek parameter on all 5 backends (E2B, CF, AWS, Novita, base)
- ORM: encryption_enabled column synced with Go GORM model
- Unit tests: 35+ cases for artifact data including encryption round-trips
* feat(dashboard): encryption settings UI, API key localStorage, and status indicators
- Encryption toggle in Project Settings with confirmation dialogs
- API key save/show/copy/delete via useApiKeyStorage localStorage hook
- ShieldCheck/ShieldX encryption status icons in project selector
- Bearer-authenticated encrypt/decrypt/rotate server actions
- Key rotation auto-saves new key when previous key is stored
* docs: add end-to-end encryption guide
- Envelope encryption architecture and API key format
- Enable/disable encryption steps
- Key rotation (zero-rewrap design) instructions
- Warnings: key loss, disabled text search, disabled dedup
* test: comprehensive E2E test suite and CI improvements
E2E tests (6 new test files, 1864 lines):
- test_encryption.py: 17 tests — lifecycle, upload/download, material URLs,
key rotation, Redis cache inspection (verifies no plaintext leakage)
- test_project_isolation.py: 20 IDOR regression tests across all endpoints
- test_disk_artifact.py: disk CRUD, artifact ls/grep/glob, session configs
- test_agent_skills.py: ZIP upload, pagination, file download, deletion
- test_learning_spaces.py: CRUD, skill associations, meta JSONB filtering
- test_users.py: user listing, resource counts, cascade delete
CI improvements:
- Parallel Docker image builds (4 images with GHA layer cache)
- .dockerignore for API and CORE to reduce build context
- Dockerfile.e2e: thin test runner image with tests as volume mount
- docker-compose.test.yml: 8-service stack with mock LLM, disabled Argon2
* fix: scope Redis cache by project_id, align KEK hard-fail, and add OSS encryption page
- Scope Redis message parts cache key by project_id to prevent
cross-project cache collisions (message:parts:{projectID}:{sha256})
- Add update_session_status("failed") in session_message.py on invalid
KEK decode, matching skill_learner.py's hard-fail pattern
- Add encryption page to OSS dashboard (src/server/ui) with API key
input and encrypt/decrypt toggle
- Include encryption_enabled in /api/v1/project/configs response
* fix(docker): use encryption-capable default token with pepper
- Update docker-compose default ROOT_API_BEARER_TOKEN to new format
(auth_secret.encrypted_master_key) so encryption works out of the box
- Add ROOT_SECRET_PEPPER default in docker-compose
- Update EnsureDefaultProjectExists to parse new token format (extract
auth_secret before the dot for HMAC/PHC hashing)
* fix: align default token with pepper, fix encryption page load error
- Use default pepper "your-secret-pepper" (viper default) for token
generation instead of a separate pepper — no need for ROOT_SECRET_PEPPER
- Update viper default apiBearerToken to new format matching docker-compose
- Fix encryption page: hide card when initial status fetch fails (e.g.
Unauthorized), only show error alert
* fix(docker): add ROOT_SECRET_PEPPER to docker-compose
config.yaml uses ${ROOT_SECRET_PEPPER} which os.ExpandEnv resolves to
empty string when the env var is unset, overriding the viper default.
This caused HMAC mismatch (token generated with "your-secret-pepper"
but server used "") resulting in 401 Unauthorized.
* feat(api): compact token format with AES Key Wrap (RFC 3394)
Compress API token from 130 to 76 chars (body) by:
- Reducing auth_secret from 32 to 16 bytes (128-bit, still secure)
- Using AES Key Wrap (RFC 3394) instead of AES-GCM for master key
wrapping (40 bytes vs 60, no random nonce needed)
- Binary packing version byte + auth + wrapped_mk into single base64url
New format: sk-ac-{base64url(0x01 | auth_16B | aes_kw_40B)} = 82 chars
Old dot-separated and legacy formats remain fully supported.
Includes RFC 3394 test vectors (Section 4.1, 4.3, 4.6).
* refactor: remove dot-separated token format, keep only compact + legacy
The dot-separated format (sk-ac-{auth}.{encrypted_mk}) was never
deployed. Remove it to simplify the codebase:
- Remove WrapMasterKey/UnwrapMasterKey (AES-GCM token wrapping)
- Remove EncryptedMasterKey field from ParsedToken
- Remove dot-parsing branch in ParseProjectToken
- Remove dot-format test cases
- Remove generateRandomSecret (unused after compact format)
- Update Python E2E tests to use compact format with AES Key Wrap
S3 envelope encryption (WrapDEK/UnwrapDEK via AES-GCM) is unchanged.
* chore: update stale token references in auth comment and .env.example
* fix(e2e): update tests for compact token format and project-scoped Redis keys
- test_key_rotation_plain_project: assert 76-char compact body instead of dot
- test_redis_cache_*: include project_id in Redis key lookup to match
the project-scoped cache key format (message:parts:{project_id}:{sha256})
* fix: change LearningSpaceSession.SessionID from uniqueIndex to index
The uniqueIndex caused a harmless but noisy GORM AutoMigrate error on
every startup: DROP CONSTRAINT uni_learning_space_sessions_session_id
fails because the constraint doesn't exist in the database.
A session can be learned by multiple learning spaces, so uniqueIndex
was semantically wrong. Changed to plain index.
* fix(auth): use dedicated cache struct for Redis project auth to preserve secret fields
model.Project uses json:"-" on SecretKeyHMAC and SecretKeyHashPHC to
prevent API leakage, but this caused these fields to be silently dropped
when cached in Redis. First request (DB hit) succeeded, but subsequent
requests (Redis hit) failed with 401 because Argon2 verification ran
against empty strings.
Introduce projectAuthCache struct with explicit JSON tags for Redis
serialization. Add guard (SecretKeyHMAC != "") to reject stale entries
from the old format.
* feat: expose encrypt/decrypt endpoints on standard API for OSS compatibility
Move encrypt/decrypt logic into shared package-level functions in
handler/encryption.go. Both AdminHandler and ProjectHandler delegate
to these functions, eliminating code duplication.
Register POST /api/v1/project/encrypt and POST /api/v1/project/decrypt
on the standard API router so OSS Docker deployments (which don't run
the admin binary) can use encryption features.
Admin binary retains /admin/v1/project/encrypt and /admin/v1/project/decrypt
for backward compatibility.
E2E tests updated to call the standard API endpoints.
* fix(ui): replace server action file uploads with route handlers
Server actions cannot reliably serialize File objects in Next.js
standalone/docker builds, causing ERR_INCOMPLETE_CHUNKED_ENCODING 500.
- Add route handlers for disk, agent skills, and session message uploads
- Remove File parameters from server actions
- Fix encryption endpoints to use /api/v1 instead of /admin/v1
* fix(docker): set APP_EXTERNALURL default so Load Preview works out of the box
Without a default, buildURL() falls back to container-internal hostname,
producing material URLs unreachable from the browser. Default to
http://localhost:${API_EXPORT_PORT:-8029} for the CLI docker-compose.
* fix(docker): set APP_EXTERNALURL default in server docker-compose
Same fix as the CLI docker-compose — default to
http://localhost:${API_EXPORT_PORT:-8029} instead of empty string.1 parent 168b086 commit ef3a0ff
138 files changed
Lines changed: 9912 additions & 877 deletions
File tree
- .github/workflows
- dashboard
- app/project/[id]
- api-keys
- settings/general
- components
- lib
- acontext/operations
- hooks
- types
- docs/content/docs/(guides)
- security
- store
- (features)
- src
- client/acontext-cli/internal/docker
- server
- api/go
- cmd
- admin
- server
- configs
- internal
- bootstrap
- config
- infra
- assetrefwriter
- blob
- crypto
- httpclient
- middleware
- modules
- handler
- model
- repo
- serializer
- service
- pkg/utils/tokens
- router
- core
- acontext_core
- infra
- sandbox/backend
- llm
- agent
- tool/skill_learner_lib
- schema
- api
- mq
- orm
- service
- controller
- data
- routers
- tests/service
- tests
- e2e
- ui
- app
- agent_skills
- api
- agent_skills/upload
- disk/upload
- session/messages
- disk
- encryption
- session
- [sessionId]/messages
- components
- messages
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
35 | | - | |
| 35 | + | |
36 | 36 | | |
37 | | - | |
38 | | - | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
39 | 72 | | |
40 | 73 | | |
41 | 74 | | |
| |||
47 | 80 | | |
48 | 81 | | |
49 | 82 | | |
50 | | - | |
51 | | - | |
52 | | - | |
53 | | - | |
54 | | - | |
55 | | - | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
101 | | - | |
| 101 | + | |
| 102 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
4 | 4 | | |
5 | 5 | | |
6 | 6 | | |
7 | | - | |
| 7 | + | |
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
62 | | - | |
| 62 | + | |
63 | 63 | | |
64 | 64 | | |
65 | 65 | | |
| |||
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
88 | | - | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
89 | 91 | | |
90 | | - | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
91 | 98 | | |
92 | 99 | | |
93 | 100 | | |
| |||
Lines changed: 129 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
11 | 16 | | |
12 | 17 | | |
13 | 18 | | |
| |||
54 | 59 | | |
55 | 60 | | |
56 | 61 | | |
| 62 | + | |
57 | 63 | | |
58 | 64 | | |
59 | 65 | | |
| |||
88 | 94 | | |
89 | 95 | | |
90 | 96 | | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
91 | 102 | | |
92 | 103 | | |
93 | 104 | | |
| |||
153 | 164 | | |
154 | 165 | | |
155 | 166 | | |
156 | | - | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
157 | 171 | | |
158 | 172 | | |
159 | 173 | | |
160 | 174 | | |
161 | 175 | | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
162 | 181 | | |
163 | 182 | | |
164 | 183 | | |
| |||
429 | 448 | | |
430 | 449 | | |
431 | 450 | | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
| 480 | + | |
| 481 | + | |
| 482 | + | |
| 483 | + | |
| 484 | + | |
| 485 | + | |
| 486 | + | |
| 487 | + | |
| 488 | + | |
| 489 | + | |
| 490 | + | |
| 491 | + | |
| 492 | + | |
| 493 | + | |
| 494 | + | |
| 495 | + | |
| 496 | + | |
| 497 | + | |
| 498 | + | |
| 499 | + | |
| 500 | + | |
| 501 | + | |
| 502 | + | |
| 503 | + | |
| 504 | + | |
| 505 | + | |
| 506 | + | |
| 507 | + | |
| 508 | + | |
| 509 | + | |
| 510 | + | |
| 511 | + | |
| 512 | + | |
| 513 | + | |
| 514 | + | |
| 515 | + | |
| 516 | + | |
| 517 | + | |
| 518 | + | |
| 519 | + | |
| 520 | + | |
| 521 | + | |
| 522 | + | |
| 523 | + | |
| 524 | + | |
| 525 | + | |
| 526 | + | |
| 527 | + | |
| 528 | + | |
| 529 | + | |
| 530 | + | |
| 531 | + | |
| 532 | + | |
| 533 | + | |
| 534 | + | |
| 535 | + | |
| 536 | + | |
| 537 | + | |
| 538 | + | |
| 539 | + | |
| 540 | + | |
| 541 | + | |
| 542 | + | |
| 543 | + | |
| 544 | + | |
| 545 | + | |
| 546 | + | |
| 547 | + | |
| 548 | + | |
| 549 | + | |
| 550 | + | |
| 551 | + | |
| 552 | + | |
| 553 | + | |
| 554 | + | |
| 555 | + | |
| 556 | + | |
| 557 | + | |
| 558 | + | |
| 559 | + | |
432 | 560 | | |
433 | 561 | | |
434 | 562 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
146 | 146 | | |
147 | 147 | | |
148 | 148 | | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
149 | 205 | | |
150 | 206 | | |
151 | 207 | | |
| |||
0 commit comments