test(e2e) + ci: harden UI e2e suite against the double-prefix regression

[yuneng-berri]

Relevant issues

Follow-up hardening for the UI double-prefix bug (post-fix from commits ba24e4a1b3 remove next env and aab3ef8988 chore: update Next.js build artifacts). The source fix — removing NEXT_PUBLIC_BASE_URL="ui/" from ui/litellm-dashboard/.env.production and rebuilding the bundle under litellm/proxy/_experimental/out/ — already landed on litellm_internal_staging. This PR addresses the three unresolved "Next Steps" from the investigation:

1. Why did CircleCI e2e_ui_testing (pipeline 74034 / job 1526026) pass with the broken bundle?
2. Make e2e_ui_testing an actual merge gate.
3. Add strict URL assertions to the Playwright suite so this class of bug fails loudly in the future.

Changes

CI (.circleci/config.yml)

• Add - e2e_ui_testing to requires: on build_and_test and db_migration_disable_update_check, so a failing UI e2e run blocks the two jobs that function as the main-path gate. Intentionally scoped narrow so the rest of the DAG still runs in parallel with e2e_ui_testing.
• Add a lightweight Run guardedPage fixture meta-tests step inside e2e_ui_testing that runs before booting the proxy, so fixture breakage is caught fast and cheap.

Playwright fixture — ui/litellm-dashboard/e2e_tests/fixtures/guarded-page.ts

Re-exports test from @playwright/test with the default page fixture wrapped in a URL guard. Specs only change their import source; no per-test code changes. The guard installs a page.on('request') listener and fails any test whose browser issues a request matching:

1. /ui/ui/ — the exact double-prefix regression. Applies to every resource type (document, script, xhr, fetch, …).
2. /ui/<api-verb>/... for xhr/fetch requests only, where <api-verb> is one of the top-level paths pulled from ${proxyBaseUrl}/... calls in networking.tsx (key, team, user, model, global, spend, customer, health, sso, callbacks, budget, tag, etc.).

Document navigations to /ui/guardrails, /ui/usage, /ui/prompts etc. — real UI routes whose first segment collides with API verb names — are explicitly NOT flagged. Static assets under /ui/_next/*, /ui/assets/*, *.{html,js,css,map,png,…} are allow-listed.

Spec migration

Every spec under e2e_tests/tests/** (including the skipped users specs, for consistency) now imports test/expect from ../../fixtures/guarded-page instead of @playwright/test. Because the fixture overrides the page fixture name, existing ({ page }) callbacks are automatically protected with no code changes.

globalSetup.ts

Tighten the post-login waitForURL predicate so a redirect to /ui/ui/... fails setup immediately, instead of silently authenticating against a broken bundle.

Meta-tests

Two Playwright specs under e2e_tests/tests/meta/ (run via playwright.meta.config.ts, excluded from the main run):

• guard.spec.ts — liveness test. Spins up an ephemeral HTTP server and verifies the guarded page's request listener fires on a real navigation, plus runtime sanity-checks isForbiddenRequestUrl().
• guard_triggers.spec.ts — negative test. Intentionally fires /ui/ui/ and /ui/key/info XHRs with test.fail(true, ...) annotations so each spec passes iff the guard trips. If someone breaks the fixture wiring, these flip red.

Unit tests — ui/litellm-dashboard/tests/e2e_guard.test.ts

84 vitest cases covering double-prefix detection, API-verb-under-/ui/ for xhr/fetch, legitimate UI routes (including API-verb-named routes like /ui/guardrails), static asset allow-list, root-level API paths, cross-origin, malformed URLs, and the resource-type boundary. All 84 pass.

Phase 1 — Why did the original CircleCI run pass?

I could not retrieve logs for pipeline 74034 / job 1526026 from this sandbox — CircleCI's public API returns "Project not found" (the project is private) and neither the CircleCI MCP tool nor the GitHub Status API exposes the run from here.

Based on static analysis of ui/litellm-dashboard/e2e_tests/tests/** and networking.tsx:

• Coverage gap. Every active Playwright spec asserts on visible UI text or URL shape, never on XHR URLs. waitForLoadState("networkidle") treats 404 responses as "completed". Several pages render from client state even when API calls fail.
• Path resolution. With the broken bundle, relative fetch("ui/project/list") resolves against the current URL. The Next.js export's canonical URL is /ui (no trailing slash), so the browser replaces the last segment → /ui/project/list. In Docker (behind a reverse proxy that serves under /ui/), the current URL ends with /, so the same relative URL resolves to /ui/ui/project/list → 404.

In other words: the source, the build, and the bundle were all identical between Docker and CircleCI — what differed was the current-URL-with-trailing-slash, and whether the proxy's static-asset handler happened to fall through to the API router for /ui/<verb>/... paths. This PR's guarded fixture catches both variants since the UI still issues raw /ui/<api-verb>/... XHRs from the relative URL resolution, regardless of whether they 404 or get routed.

Pre-Submission checklist

• I have added testing in the appropriate directory (ui/litellm-dashboard/tests/e2e_guard.test.ts, ui/litellm-dashboard/e2e_tests/tests/meta/*.spec.ts).
• Full vitest sweep passes: 384 files, 3889 tests.
• Meta-specs pass under Playwright's chromium runner: 4/4 (two positive, two negative with test.fail(true, ...) inversion).
• CircleCI config validated (python3 -c "import yaml; yaml.safe_load(...)").
• Scope isolated: no source changes in litellm/, no schema changes, no dep changes. Only CI config, Playwright e2e infrastructure, and matching vitest coverage.

Type

✅ Test
🚄 Infrastructure

Screenshots / Proof of Fix

Vitest:

Test Files  384 passed (384)
     Tests  3889 passed (3889)
  Duration  399.75s

Meta-tests (Playwright chromium, standalone):

Running 4 tests using 1 worker
  ✘  1 [chromium] › tests/meta/guard_triggers.spec.ts › fires when a /ui/ui/ XHR is made (91ms)
  ✘  2 [chromium] › tests/meta/guard_triggers.spec.ts › fires when an /ui/<api-verb> XHR is made (67ms)
  ✓  3 [chromium] › tests/meta/guard.spec.ts › should fire request listeners on real HTTP navigation (38ms)
  ✓  4 [chromium] › tests/meta/guard.spec.ts › should classify forbidden and allowed URLs at runtime (2ms)
  4 passed (1.2s)

✘ 1 and ✘ 2 are the negative tests — they expect the guard to trip (via test.fail(true, ...)), so the run-level result is 4 passed.

Follow-ups not in this PR

• Branch-protection: GitHub admin change to mark the CircleCI e2e_ui_testing status as a required check on litellm_internal_staging / main. Must be done in the repo settings UI; cannot be changed from code.

  

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!