Skip to content

DQX Studio: Replace apx framework with first-party build/dev scripts#1223

Merged
mwojtyczka merged 14 commits into
mainfrom
app/remove_apx_dependency
Jun 10, 2026
Merged

DQX Studio: Replace apx framework with first-party build/dev scripts#1223
mwojtyczka merged 14 commits into
mainfrom
app/remove_apx_dependency

Conversation

@berrybluecode

@berrybluecode berrybluecode commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

The app previously relied on the unpublished apx package (github.com/databricks-solutions/apx) to drive its build and dev workflows. Drop the dependency and reimplement what we need ourselves:

  • scripts/build_app.py: replaces apx build. Generates the FastAPI OpenAPI schema, runs orval (now configured via orval.config.ts) to regenerate ui/lib/api.ts, runs vite build, and produces the application wheel + app.yml + requirements.txt under .build/. The wheel carries a build-tag local-version segment (e.g. .b20260530t012345) baked into both the filename and the internal METADATA, so successive deploys at the same git commit always reinstall fresh code in the Databricks Apps persistent venv. This obsoletes scripts/_align_wheel_version.py (now removed), which used to repair apx's filename-only mangling after the fact.

  • scripts/dev.py: replaces apx dev start. Runs uvicorn with --reload on a fixed internal port and serves the SPA via Vite on :9001, proxying /api and /docs to the backend. SIGINT is forwarded to both children and either child crashing tears the other down, so no orphan processes are left behind.

  • pyproject.toml: drop the apx dependency-group entry and the [tool.uv.sources].apx git source. Rename the metadata table from [tool.apx.metadata] to [tool.dqx_app.metadata] (same schema, new namespace) and document its consumers (build_app.py, vite.config.ts). Add basedpyright to the dev group.

  • databricks.yml: rewrite artifacts.default.build to call scripts/build_app.py instead of apx build. Add explicit Mode A (bundle-managed) vs Mode B (external/reuse) documentation on the sql_warehouse_id variable; default to empty so a target that forgets to set it fails fast.

  • post_deploy_grants.sh: extend the warehouse grant path so external warehouses (Mode B) get CAN_USE granted to the app SP, job SP, and the users group via the warehouses permissions API — the bundle doesn't own the warehouse in that mode, so the bundle's permissions: block can't do it.

  • Documentation refresh across DEPLOYMENT.md, DEVELOPMENT.md, README.md, AGENTS.md, the UI/backend CLAUDE.md files, and docs/dqx/docs/{installation,dev/contributing}.mdx to reflect the new build/dev entry points and warehouse modes.

No runtime behaviour change in the app itself: app/src/...backend/app.py is touched only to pick up the new metadata namespace.

Changes

Linked issues

Resolves #..

Tests

  • manually tested
  • added unit tests
  • added integration tests
  • added end-to-end tests
  • added performance tests

Documentation and Demos

  • added/updated demos
  • added/updated docs
  • added/updated agent skills

The app previously relied on the unpublished ``apx`` package
(github.com/databricks-solutions/apx) to drive its build and dev
workflows. Drop the dependency and reimplement what we need ourselves:

- scripts/build_app.py: replaces ``apx build``. Generates the FastAPI
  OpenAPI schema, runs orval (now configured via orval.config.ts) to
  regenerate ui/lib/api.ts, runs ``vite build``, and produces the
  application wheel + ``app.yml`` + ``requirements.txt`` under
  ``.build/``. The wheel carries a build-tag local-version segment
  (e.g. ``.b20260530t012345``) baked into both the filename and the
  internal METADATA, so successive deploys at the same git commit
  always reinstall fresh code in the Databricks Apps persistent venv.
  This obsoletes ``scripts/_align_wheel_version.py`` (now removed),
  which used to repair apx's filename-only mangling after the fact.

- scripts/dev.py: replaces ``apx dev start``. Runs uvicorn with
  ``--reload`` on a fixed internal port and serves the SPA via Vite
  on :9001, proxying ``/api`` and ``/docs`` to the backend. SIGINT is
  forwarded to both children and either child crashing tears the
  other down, so no orphan processes are left behind.

- pyproject.toml: drop the ``apx`` dependency-group entry and the
  ``[tool.uv.sources].apx`` git source. Rename the metadata table from
  ``[tool.apx.metadata]`` to ``[tool.dqx_app.metadata]`` (same
  schema, new namespace) and document its consumers (build_app.py,
  vite.config.ts). Add ``basedpyright`` to the dev group.

- databricks.yml: rewrite ``artifacts.default.build`` to call
  ``scripts/build_app.py`` instead of ``apx build``. Add explicit
  Mode A (bundle-managed) vs Mode B (external/reuse) documentation
  on the ``sql_warehouse_id`` variable; default to empty so a target
  that forgets to set it fails fast.

- post_deploy_grants.sh: extend the warehouse grant path so external
  warehouses (Mode B) get ``CAN_USE`` granted to the app SP, job SP,
  and the ``users`` group via the warehouses permissions API — the
  bundle doesn't own the warehouse in that mode, so the bundle's
  ``permissions:`` block can't do it.

- Documentation refresh across DEPLOYMENT.md, DEVELOPMENT.md,
  README.md, AGENTS.md, the UI/backend CLAUDE.md files, and
  docs/dqx/docs/{installation,dev/contributing}.mdx to reflect the
  new build/dev entry points and warehouse modes.

No runtime behaviour change in the app itself: app/src/...backend/app.py
is touched only to pick up the new metadata namespace.
@codecov

codecov Bot commented Jun 5, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.14%. Comparing base (45bd3b1) to head (fea954e).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1223      +/-   ##
==========================================
+ Coverage   92.13%   92.14%   +0.01%     
==========================================
  Files         101      101              
  Lines        9603     9603              
==========================================
+ Hits         8848     8849       +1     
+ Misses        755      754       -1     
Flag Coverage Δ
anomaly 53.51% <ø> (ø)
anomaly-serverless 53.52% <ø> (ø)
integration 50.05% <ø> (+1.24%) ⬆️
integration-serverless 50.70% <ø> (-0.08%) ⬇️
unit 55.92% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

github-actions Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

✅ 748/748 passed, 41 skipped, 4h27m56s total

Running from acceptance #4805

@github-actions

github-actions Bot commented Jun 5, 2026

Copy link
Copy Markdown
Contributor

✅ 182/182 passed, 1 flaky, 5h11m16s total

Flaky tests:

  • 🤪 test_anomaly_check_with_custom_info_column_name (39.919s)

Running from anomaly #919

@mwojtyczka mwojtyczka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strong refactor — dropping the unpublished/un-pinned apx git dependency is a clear supply-chain win, and the scripts are unusually well-documented (the process-group signal handling in dev.py and the Mode A/B warehouse docs in databricks.yml are excellent). A few notes inline, plus two non-anchored points:

  • PR claims "added unit tests" but the diff contains none. build_app.py (_read_app_metadata missing-key path, _write_metadata_module formatting, newest-wheel/stale-wheel logic) and dev.py's shutdown logic are testable. Please add a couple or correct the checklist.
  • Removed apxDevProxyGuard from vite.config.ts looks correct given the new topology (Vite is now itself the public-facing proxy rather than sitting behind apx), so the origin guard no longer applies — just flagging it as a conscious drop of a (dev-only) control rather than an accident.

Nothing blocking on correctness; I'd want the grant-failure propagation and the JSON-construction hardening addressed in this deploy-privileged script before merge.

Comment thread app/scripts/post_deploy_grants.sh Outdated
Comment thread app/scripts/post_deploy_grants.sh
Comment thread app/scripts/build_app.py Outdated

@mwojtyczka mwojtyczka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

left some comments

@mwojtyczka mwojtyczka added the under-review This PR is currently being reviewed by one of DQX maintainers. label Jun 5, 2026
* post_deploy_grants.sh: validate UC identifiers / UUIDs / warehouse
  IDs before they reach a SQL warehouse or URL path; build all JSON
  payloads with jq -n --arg so embedded quotes/backticks/backslashes
  can no longer escape the request envelope; track a FAILURES counter
  and exit non-zero so individual GRANT failures surface to CI instead
  of being swallowed under a misleading "Done" message.
* build_app.py: fix stale [tool.apx.metadata] reference in module
  docstring (now [tool.dqx_app.metadata], matching _read_app_metadata)
  and align the example build tag to lowercase t to match
  time.strftime("b%Y%m%dt%H%M%S").
* databricks.yml: add Mode B target dqx-app-demo for the
  fevm-tsh-dqx-app-demo workspace, reusing the pre-existing
  dqx-studio-sql-warehouse (e864734d584d98a6) and
  dqx-studio-lakebase-v2 instance from a prior apx-era deploy.
Comment thread app/src/databricks_labs_dqx_app/ui/CLAUDE.md Outdated
Comment thread app/src/databricks_labs_dqx_app/ui/lib/i18n/locales/de.json Outdated
* Rename frontend folder `components/apx/` → `components/layout/`
  (9 components: Navbar, SidebarLayout, theme-provider, mode-toggle,
  Logo, HeaderUserMenu, SidebarUserFooter, PageBreadcrumb,
  LanguageSelector) and update all 18 importing files.

* Strip "replaces apx X" / "matching the legacy apx Y" comments and
  docstrings from orval.config.ts, vite.config.ts, scripts/build_app.py,
  scripts/dev.py, and pyproject.toml. The code itself was already
  apx-free; this removes the breadcrumbs pointing back to it.

* Update ui/CLAUDE.md to drop the historical-artifact note on the
  renamed folder.

* Drop ui/lib/i18n/locales/de.json.

Verified with make fmt, make test (1141 passed), make app-test
(591 passed), and make app-check (TypeScript + basedpyright clean).

@mwojtyczka mwojtyczka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mwojtyczka mwojtyczka added Approved to Merge When PR is reviewed and approved. To be merged once all tests pass and removed under-review This PR is currently being reviewed by one of DQX maintainers. labels Jun 10, 2026
@mwojtyczka mwojtyczka mentioned this pull request Jun 16, 2026
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Approved to Merge When PR is reviewed and approved. To be merged once all tests pass

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants