Skip to content

Commit ef3a0ff

Browse files
authored
feat: per-project envelope encryption, IDOR protection, material URL proxy, and E2E test suite (#460)
* feat(api): per-project envelope encryption, material URL proxy, and IDOR protection Encryption architecture: - AES-256-GCM envelope encryption with per-object random DEKs - Zero-knowledge key hierarchy: master key embedded in API token (sk-ac-{authSecret}.{encryptedMasterKey}), never stored server-side - HKDF-SHA256 key derivation from auth secret + pepper - Idempotent RewrapDEK for crash-safe key rotation (O(1), no S3 re-encryption) - Redis cache encryption with prefix byte scheme (0x00=plain, 0x01=encrypted) - DB content encryption via EncodeContent/DecodeContent binary framing Material URL proxy: - Replaces presigned S3 URLs with Redis-backed decryption proxy - GET /api/v1/material/{token} — unauthenticated, token is the credential - Enables transparent download of encrypted S3 objects IDOR protection: - Cross-project ownership validation on all resource endpoints - Defense-in-depth checks at both handler and service layers - diskRepo.GetByProjectAndID, session.ProjectID guards, S3 key prefix checks Admin encryption lifecycle: - POST /admin/v1/project/encrypt — idempotent batch encryption - POST /admin/v1/project/decrypt — batch decryption - PUT /admin/v1/project/secret_key (Bearer) — preserves master key - PUT /admin/v1/project/:id/secret_key (JWT) — blocked for encrypted projects * feat(core): thread user_kek through Python CORE for end-to-end encryption - Add crypto.py: AES-256-GCM envelope encryption (Go-compatible wire format) - S3 client: auto-encrypt/decrypt with user_kek on upload/download - MQ consumers: hard-fail on invalid KEK (never silent plaintext fallback) - Skill learner: SkillLearnerCtx carries KEK to all LLM tool handlers - Artifact data: encode_content/decode_content with binary framing (mirrors Go) - Sandbox backends: user_kek parameter on all 5 backends (E2B, CF, AWS, Novita, base) - ORM: encryption_enabled column synced with Go GORM model - Unit tests: 35+ cases for artifact data including encryption round-trips * feat(dashboard): encryption settings UI, API key localStorage, and status indicators - Encryption toggle in Project Settings with confirmation dialogs - API key save/show/copy/delete via useApiKeyStorage localStorage hook - ShieldCheck/ShieldX encryption status icons in project selector - Bearer-authenticated encrypt/decrypt/rotate server actions - Key rotation auto-saves new key when previous key is stored * docs: add end-to-end encryption guide - Envelope encryption architecture and API key format - Enable/disable encryption steps - Key rotation (zero-rewrap design) instructions - Warnings: key loss, disabled text search, disabled dedup * test: comprehensive E2E test suite and CI improvements E2E tests (6 new test files, 1864 lines): - test_encryption.py: 17 tests — lifecycle, upload/download, material URLs, key rotation, Redis cache inspection (verifies no plaintext leakage) - test_project_isolation.py: 20 IDOR regression tests across all endpoints - test_disk_artifact.py: disk CRUD, artifact ls/grep/glob, session configs - test_agent_skills.py: ZIP upload, pagination, file download, deletion - test_learning_spaces.py: CRUD, skill associations, meta JSONB filtering - test_users.py: user listing, resource counts, cascade delete CI improvements: - Parallel Docker image builds (4 images with GHA layer cache) - .dockerignore for API and CORE to reduce build context - Dockerfile.e2e: thin test runner image with tests as volume mount - docker-compose.test.yml: 8-service stack with mock LLM, disabled Argon2 * fix: scope Redis cache by project_id, align KEK hard-fail, and add OSS encryption page - Scope Redis message parts cache key by project_id to prevent cross-project cache collisions (message:parts:{projectID}:{sha256}) - Add update_session_status("failed") in session_message.py on invalid KEK decode, matching skill_learner.py's hard-fail pattern - Add encryption page to OSS dashboard (src/server/ui) with API key input and encrypt/decrypt toggle - Include encryption_enabled in /api/v1/project/configs response * fix(docker): use encryption-capable default token with pepper - Update docker-compose default ROOT_API_BEARER_TOKEN to new format (auth_secret.encrypted_master_key) so encryption works out of the box - Add ROOT_SECRET_PEPPER default in docker-compose - Update EnsureDefaultProjectExists to parse new token format (extract auth_secret before the dot for HMAC/PHC hashing) * fix: align default token with pepper, fix encryption page load error - Use default pepper "your-secret-pepper" (viper default) for token generation instead of a separate pepper — no need for ROOT_SECRET_PEPPER - Update viper default apiBearerToken to new format matching docker-compose - Fix encryption page: hide card when initial status fetch fails (e.g. Unauthorized), only show error alert * fix(docker): add ROOT_SECRET_PEPPER to docker-compose config.yaml uses ${ROOT_SECRET_PEPPER} which os.ExpandEnv resolves to empty string when the env var is unset, overriding the viper default. This caused HMAC mismatch (token generated with "your-secret-pepper" but server used "") resulting in 401 Unauthorized. * feat(api): compact token format with AES Key Wrap (RFC 3394) Compress API token from 130 to 76 chars (body) by: - Reducing auth_secret from 32 to 16 bytes (128-bit, still secure) - Using AES Key Wrap (RFC 3394) instead of AES-GCM for master key wrapping (40 bytes vs 60, no random nonce needed) - Binary packing version byte + auth + wrapped_mk into single base64url New format: sk-ac-{base64url(0x01 | auth_16B | aes_kw_40B)} = 82 chars Old dot-separated and legacy formats remain fully supported. Includes RFC 3394 test vectors (Section 4.1, 4.3, 4.6). * refactor: remove dot-separated token format, keep only compact + legacy The dot-separated format (sk-ac-{auth}.{encrypted_mk}) was never deployed. Remove it to simplify the codebase: - Remove WrapMasterKey/UnwrapMasterKey (AES-GCM token wrapping) - Remove EncryptedMasterKey field from ParsedToken - Remove dot-parsing branch in ParseProjectToken - Remove dot-format test cases - Remove generateRandomSecret (unused after compact format) - Update Python E2E tests to use compact format with AES Key Wrap S3 envelope encryption (WrapDEK/UnwrapDEK via AES-GCM) is unchanged. * chore: update stale token references in auth comment and .env.example * fix(e2e): update tests for compact token format and project-scoped Redis keys - test_key_rotation_plain_project: assert 76-char compact body instead of dot - test_redis_cache_*: include project_id in Redis key lookup to match the project-scoped cache key format (message:parts:{project_id}:{sha256}) * fix: change LearningSpaceSession.SessionID from uniqueIndex to index The uniqueIndex caused a harmless but noisy GORM AutoMigrate error on every startup: DROP CONSTRAINT uni_learning_space_sessions_session_id fails because the constraint doesn't exist in the database. A session can be learned by multiple learning spaces, so uniqueIndex was semantically wrong. Changed to plain index. * fix(auth): use dedicated cache struct for Redis project auth to preserve secret fields model.Project uses json:"-" on SecretKeyHMAC and SecretKeyHashPHC to prevent API leakage, but this caused these fields to be silently dropped when cached in Redis. First request (DB hit) succeeded, but subsequent requests (Redis hit) failed with 401 because Argon2 verification ran against empty strings. Introduce projectAuthCache struct with explicit JSON tags for Redis serialization. Add guard (SecretKeyHMAC != "") to reject stale entries from the old format. * feat: expose encrypt/decrypt endpoints on standard API for OSS compatibility Move encrypt/decrypt logic into shared package-level functions in handler/encryption.go. Both AdminHandler and ProjectHandler delegate to these functions, eliminating code duplication. Register POST /api/v1/project/encrypt and POST /api/v1/project/decrypt on the standard API router so OSS Docker deployments (which don't run the admin binary) can use encryption features. Admin binary retains /admin/v1/project/encrypt and /admin/v1/project/decrypt for backward compatibility. E2E tests updated to call the standard API endpoints. * fix(ui): replace server action file uploads with route handlers Server actions cannot reliably serialize File objects in Next.js standalone/docker builds, causing ERR_INCOMPLETE_CHUNKED_ENCODING 500. - Add route handlers for disk, agent skills, and session message uploads - Remove File parameters from server actions - Fix encryption endpoints to use /api/v1 instead of /admin/v1 * fix(docker): set APP_EXTERNALURL default so Load Preview works out of the box Without a default, buildURL() falls back to container-internal hostname, producing material URLs unreachable from the browser. Default to http://localhost:${API_EXPORT_PORT:-8029} for the CLI docker-compose. * fix(docker): set APP_EXTERNALURL default in server docker-compose Same fix as the CLI docker-compose — default to http://localhost:${API_EXPORT_PORT:-8029} instead of empty string.
1 parent 168b086 commit ef3a0ff

138 files changed

Lines changed: 9912 additions & 877 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.github/workflows/e2e-test.yaml

Lines changed: 36 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -32,10 +32,43 @@ jobs:
3232
- name: Set up Docker Buildx
3333
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd
3434

35-
- name: Build Images
35+
- name: Build Docker Images (parallel)
3636
run: |
37-
cd src/server
38-
docker compose -f docker-compose.test.yml build
37+
docker buildx build \
38+
--load \
39+
--cache-from type=gha,scope=e2e-core \
40+
--cache-to type=gha,mode=max,scope=e2e-core \
41+
-t acontext-e2e-test-core:latest \
42+
./src/server/core &
43+
PID_CORE=$!
44+
45+
docker buildx build \
46+
--load \
47+
--cache-from type=gha,scope=e2e-api \
48+
--cache-to type=gha,mode=max,scope=e2e-api \
49+
-t acontext-e2e-test-api:latest \
50+
./src/server/api/go &
51+
PID_API=$!
52+
53+
docker buildx build \
54+
--load \
55+
--file ./src/server/api/go/Dockerfile.admin \
56+
--cache-from type=gha,scope=e2e-admin \
57+
--cache-to type=gha,mode=max,scope=e2e-admin \
58+
-t acontext-e2e-test-admin:latest \
59+
./src/server/api/go &
60+
PID_ADMIN=$!
61+
62+
docker buildx build \
63+
--load \
64+
--cache-from type=gha,scope=e2e-runner \
65+
--cache-to type=gha,mode=max,scope=e2e-runner \
66+
-t acontext-e2e-test-runner:latest \
67+
-f ./src/server/tests/Dockerfile.e2e \
68+
./src/server/tests &
69+
PID_RUNNER=$!
70+
71+
wait $PID_CORE $PID_API $PID_ADMIN $PID_RUNNER
3972
4073
- name: Run E2E Test
4174
run: |
@@ -47,9 +80,3 @@ jobs:
4780
run: |
4881
cd src/server
4982
docker compose -f docker-compose.test.yml logs
50-
51-
- name: Teardown
52-
if: always()
53-
run: |
54-
cd src/server
55-
docker compose -f docker-compose.test.yml down --timeout 10

AGENTS.md

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -98,4 +98,5 @@ All workflows create a GitHub Release with a path-scoped changelog generated by
9898
### Update Dashboard and Dashboard OSS at the same time
9999
- **Dashboard** (`dashboard/`): The hosted/commercial dashboard with authentication (Supabase), organizations, projects, and a mixin-based `AcontextClient`. Routes are scoped under `/project/[id]/...`.
100100
- **Dashboard OSS** (`src/server/ui/`): The open-source, self-hosted dashboard.
101-
- When modifying UI features (types, server actions, page components), always apply the equivalent change to both dashboards.
101+
- When modifying UI features (types, server actions, page components), always apply the equivalent change to both dashboards.
102+
- **Exception:** Encryption features (encryption settings, API key localStorage, key rotation UI) are commercial-only and do NOT need to be added to Dashboard OSS.

dashboard/.env.example

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ NEXT_PUBLIC_BASE_PATH=""
44
NEXT_PUBLIC_SUPABASE_URL=your-project-url
55
NEXT_PUBLIC_SUPABASE_PUBLISHABLE_KEY=sb_publishable_... or anon key
66

7-
ACONTEXT_API_BEARER_TOKEN=your-root-api-bearer-token
7+
ACONTEXT_API_BEARER_TOKEN=AaGyw9Tl9qe4ydDh8qO0xdZNkrobQvwHWFRsnp5a3QtfbaDSDJQeRHxXPr4bGpc0g130EqBSjRNF
88
ACONTEXT_PROJECT_BEARER_TOKEN_PREFIX=sk-ac-
99

1010
NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY=xxx

dashboard/app/project/[id]/api-keys/actions.ts

Lines changed: 10 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -59,7 +59,7 @@ export async function getSecretKeyHistory(projectId: string) {
5959
* Rotate (generate new) secret key for a project
6060
* Returns the full key for one-time display
6161
*/
62-
export async function rotateSecretKey(projectId: string) {
62+
export async function rotateSecretKey(projectId: string, apiKey?: string) {
6363
// Get current user (will redirect if not authenticated)
6464
const user = await getCurrentUser();
6565

@@ -85,9 +85,16 @@ export async function rotateSecretKey(projectId: string) {
8585
}
8686

8787
try {
88-
// Call API to generate new secret key
88+
// When an API key is available, use the Bearer route (preserves master key
89+
// for encrypted projects). Otherwise fall back to the admin route, which
90+
// will reject the request server-side if the project has encryption enabled.
8991
const client = new AcontextClient();
90-
const fullSecretKey = await client.updateProjectSecretKey(projectId);
92+
let fullSecretKey: string;
93+
if (apiKey) {
94+
fullSecretKey = await client.rotateProjectSecretKey(apiKey);
95+
} else {
96+
fullSecretKey = await client.rotateProjectSecretKeyAdmin(projectId);
97+
}
9198

9299
if (!fullSecretKey) {
93100
return { error: "Failed to generate secret key" };

dashboard/app/project/[id]/api-keys/api-keys-page-client.tsx

Lines changed: 129 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,11 @@ import {
88
RefreshCw,
99
AlertTriangle,
1010
KeyRound,
11+
Save,
12+
Trash2,
13+
Eye,
14+
EyeOff,
15+
Info,
1116
} from "lucide-react";
1217
import { toast } from "sonner";
1318
import { useTopNavStore } from "@/stores/top-nav";
@@ -54,6 +59,7 @@ import {
5459
import { rotateSecretKey } from "./actions";
5560
import { Tabs, TabsContent, TabsList, TabsTrigger } from "@/components/ui/tabs";
5661
import { CodeEditor } from "@/components/code-editor";
62+
import { useApiKeyStorage } from "@/lib/hooks/use-api-key-storage";
5763

5864
const SERVICE_URL = "api.acontext.app/api/v1";
5965

@@ -88,6 +94,11 @@ export function ApiKeysPageClient({
8894
"python"
8995
);
9096

97+
// Saved API key in localStorage
98+
const { apiKey: savedApiKey, hasApiKey: hasSavedApiKey, saveApiKey, removeApiKey } = useApiKeyStorage(project.id);
99+
const [apiKeyInput, setApiKeyInput] = useState("");
100+
const [showSavedKey, setShowSavedKey] = useState(false);
101+
91102
const isOwner = role === "owner";
92103
const hasExistingKey = keyRotations.length > 0;
93104
const latestRotation = keyRotations[0] || null;
@@ -153,12 +164,20 @@ console.log(await client.ping());`,
153164

154165
const performKeyGeneration = () => {
155166
startTransition(async () => {
156-
const result = await rotateSecretKey(project.id);
167+
const result = await rotateSecretKey(
168+
project.id,
169+
savedApiKey ?? undefined,
170+
);
157171
if (result.error) {
158172
toast.error(result.error);
159173
} else if (result.secretKey) {
160174
setNewlyGeneratedKey(result.secretKey);
161175
setShowKeyDialog(true);
176+
// Auto-update localStorage if the user had a saved key
177+
if (hasSavedApiKey) {
178+
saveApiKey(result.secretKey);
179+
toast.info("Saved API key in your browser has been updated with the new key.");
180+
}
162181
router.refresh();
163182
toast.success(
164183
hasExistingKey
@@ -429,6 +448,115 @@ IMPORTANT: Store this key securely. It will not be shown again.
429448
</CardContent>
430449
</Card>
431450

451+
{/* Saved API Key Card */}
452+
<Card>
453+
<CardHeader>
454+
<CardTitle className="flex items-center gap-2">
455+
<Save className="h-5 w-5" />
456+
Saved API Key
457+
</CardTitle>
458+
<CardDescription>
459+
Save your API key in your browser for encryption features.
460+
Your API key is stored only in your browser and is never sent to our servers.
461+
</CardDescription>
462+
</CardHeader>
463+
<CardContent className="space-y-4">
464+
{hasSavedApiKey ? (
465+
<div className="space-y-3">
466+
<div className="space-y-2">
467+
<Label>Current Saved Key</Label>
468+
<div className="flex items-center gap-2">
469+
<Input
470+
value={showSavedKey ? (savedApiKey ?? "") : "****************************************"}
471+
readOnly
472+
className="font-mono text-sm"
473+
/>
474+
<Button
475+
variant="outline"
476+
size="icon"
477+
onClick={() => setShowSavedKey(!showSavedKey)}
478+
title={showSavedKey ? "Hide key" : "Show key"}
479+
>
480+
{showSavedKey ? (
481+
<EyeOff className="h-4 w-4" />
482+
) : (
483+
<Eye className="h-4 w-4" />
484+
)}
485+
</Button>
486+
<Button
487+
variant="outline"
488+
size="icon"
489+
onClick={() => {
490+
if (savedApiKey) {
491+
navigator.clipboard.writeText(savedApiKey);
492+
toast.success("Saved API key copied to clipboard");
493+
}
494+
}}
495+
title="Copy saved key"
496+
>
497+
<Copy className="h-4 w-4" />
498+
</Button>
499+
<Button
500+
variant="destructive"
501+
size="icon"
502+
onClick={() => {
503+
removeApiKey();
504+
setShowSavedKey(false);
505+
toast.success("Saved API key removed from browser");
506+
}}
507+
title="Remove saved key"
508+
>
509+
<Trash2 className="h-4 w-4" />
510+
</Button>
511+
</div>
512+
</div>
513+
<Alert>
514+
<Info className="h-4 w-4" />
515+
<AlertDescription>
516+
Your API key is stored only in your browser. It is used for encryption/decryption operations.
517+
</AlertDescription>
518+
</Alert>
519+
</div>
520+
) : (
521+
<div className="space-y-3">
522+
<div className="space-y-2">
523+
<Label htmlFor="api-key-input">API Key</Label>
524+
<div className="flex items-center gap-2">
525+
<Input
526+
id="api-key-input"
527+
type="password"
528+
value={apiKeyInput}
529+
onChange={(e) => setApiKeyInput(e.target.value)}
530+
placeholder="Paste your API key here"
531+
className="font-mono text-sm"
532+
/>
533+
<Button
534+
onClick={() => {
535+
if (apiKeyInput.trim()) {
536+
saveApiKey(apiKeyInput.trim());
537+
setApiKeyInput("");
538+
toast.success("API key saved to browser");
539+
}
540+
}}
541+
disabled={!apiKeyInput.trim()}
542+
>
543+
<Save className="h-4 w-4" />
544+
Save
545+
</Button>
546+
</div>
547+
</div>
548+
<Alert>
549+
<Info className="h-4 w-4" />
550+
<AlertDescription>
551+
Your API key is stored only in your browser. It is used for encryption/decryption operations.
552+
Save your API key here after generating or rotating it.
553+
</AlertDescription>
554+
</Alert>
555+
</div>
556+
)}
557+
</CardContent>
558+
</Card>
559+
432560
{/* Key History Card */}
433561
{hasExistingKey && (
434562
<Card>

dashboard/app/project/[id]/settings/general/actions.ts

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -146,6 +146,62 @@ export async function getProjectConfigs(
146146
}
147147
}
148148

149+
async function toggleProjectEncryption(
150+
projectId: string,
151+
apiKey: string,
152+
action: "encrypt" | "decrypt"
153+
): Promise<{ success?: boolean; error?: string }> {
154+
try {
155+
await getCurrentUser();
156+
157+
const project = await getProject(projectId);
158+
if (!project) {
159+
return { error: "Project not found" };
160+
}
161+
162+
const membership = await getOrganizationMembershipForCurrentUser(
163+
project.organization_id,
164+
"role"
165+
);
166+
if (!membership) {
167+
return { error: "Project not found or access denied" };
168+
}
169+
if (membership.role !== "owner") {
170+
return { error: `Only organization owners can ${action === "encrypt" ? "enable" : "disable"} encryption` };
171+
}
172+
173+
const client = new AcontextClient();
174+
if (action === "encrypt") {
175+
await client.encryptProject(projectId, apiKey);
176+
} else {
177+
await client.decryptProject(projectId, apiKey);
178+
}
179+
180+
const encodedProjectId = encodeId(projectId);
181+
revalidatePath(`/project/${encodedProjectId}`, "layout");
182+
183+
return { success: true };
184+
} catch (error) {
185+
return {
186+
error: `Failed to ${action} project: ${error instanceof Error ? error.message : "Unknown error"}`,
187+
};
188+
}
189+
}
190+
191+
export async function encryptProjectAction(
192+
projectId: string,
193+
apiKey: string
194+
): Promise<{ success?: boolean; error?: string }> {
195+
return toggleProjectEncryption(projectId, apiKey, "encrypt");
196+
}
197+
198+
export async function decryptProjectAction(
199+
projectId: string,
200+
apiKey: string
201+
): Promise<{ success?: boolean; error?: string }> {
202+
return toggleProjectEncryption(projectId, apiKey, "decrypt");
203+
}
204+
149205
export async function updateProjectConfigs(
150206
projectId: string,
151207
configs: Partial<ProjectConfig>

0 commit comments

Comments
 (0)