DQX Studio: Replace apx framework with first-party build/dev scripts#1223
Conversation
The app previously relied on the unpublished ``apx`` package
(github.com/databricks-solutions/apx) to drive its build and dev
workflows. Drop the dependency and reimplement what we need ourselves:
- scripts/build_app.py: replaces ``apx build``. Generates the FastAPI
OpenAPI schema, runs orval (now configured via orval.config.ts) to
regenerate ui/lib/api.ts, runs ``vite build``, and produces the
application wheel + ``app.yml`` + ``requirements.txt`` under
``.build/``. The wheel carries a build-tag local-version segment
(e.g. ``.b20260530t012345``) baked into both the filename and the
internal METADATA, so successive deploys at the same git commit
always reinstall fresh code in the Databricks Apps persistent venv.
This obsoletes ``scripts/_align_wheel_version.py`` (now removed),
which used to repair apx's filename-only mangling after the fact.
- scripts/dev.py: replaces ``apx dev start``. Runs uvicorn with
``--reload`` on a fixed internal port and serves the SPA via Vite
on :9001, proxying ``/api`` and ``/docs`` to the backend. SIGINT is
forwarded to both children and either child crashing tears the
other down, so no orphan processes are left behind.
- pyproject.toml: drop the ``apx`` dependency-group entry and the
``[tool.uv.sources].apx`` git source. Rename the metadata table from
``[tool.apx.metadata]`` to ``[tool.dqx_app.metadata]`` (same
schema, new namespace) and document its consumers (build_app.py,
vite.config.ts). Add ``basedpyright`` to the dev group.
- databricks.yml: rewrite ``artifacts.default.build`` to call
``scripts/build_app.py`` instead of ``apx build``. Add explicit
Mode A (bundle-managed) vs Mode B (external/reuse) documentation
on the ``sql_warehouse_id`` variable; default to empty so a target
that forgets to set it fails fast.
- post_deploy_grants.sh: extend the warehouse grant path so external
warehouses (Mode B) get ``CAN_USE`` granted to the app SP, job SP,
and the ``users`` group via the warehouses permissions API — the
bundle doesn't own the warehouse in that mode, so the bundle's
``permissions:`` block can't do it.
- Documentation refresh across DEPLOYMENT.md, DEVELOPMENT.md,
README.md, AGENTS.md, the UI/backend CLAUDE.md files, and
docs/dqx/docs/{installation,dev/contributing}.mdx to reflect the
new build/dev entry points and warehouse modes.
No runtime behaviour change in the app itself: app/src/...backend/app.py
is touched only to pick up the new metadata namespace.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1223 +/- ##
==========================================
+ Coverage 92.13% 92.14% +0.01%
==========================================
Files 101 101
Lines 9603 9603
==========================================
+ Hits 8848 8849 +1
+ Misses 755 754 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
✅ 748/748 passed, 41 skipped, 4h27m56s total Running from acceptance #4805 |
|
✅ 182/182 passed, 1 flaky, 5h11m16s total Flaky tests:
Running from anomaly #919 |
mwojtyczka
left a comment
There was a problem hiding this comment.
Strong refactor — dropping the unpublished/un-pinned apx git dependency is a clear supply-chain win, and the scripts are unusually well-documented (the process-group signal handling in dev.py and the Mode A/B warehouse docs in databricks.yml are excellent). A few notes inline, plus two non-anchored points:
- PR claims "added unit tests" but the diff contains none.
build_app.py(_read_app_metadatamissing-key path,_write_metadata_moduleformatting, newest-wheel/stale-wheel logic) anddev.py's shutdown logic are testable. Please add a couple or correct the checklist. - Removed
apxDevProxyGuardfromvite.config.tslooks correct given the new topology (Vite is now itself the public-facing proxy rather than sitting behind apx), so the origin guard no longer applies — just flagging it as a conscious drop of a (dev-only) control rather than an accident.
Nothing blocking on correctness; I'd want the grant-failure propagation and the JSON-construction hardening addressed in this deploy-privileged script before merge.
* post_deploy_grants.sh: validate UC identifiers / UUIDs / warehouse
IDs before they reach a SQL warehouse or URL path; build all JSON
payloads with jq -n --arg so embedded quotes/backticks/backslashes
can no longer escape the request envelope; track a FAILURES counter
and exit non-zero so individual GRANT failures surface to CI instead
of being swallowed under a misleading "Done" message.
* build_app.py: fix stale [tool.apx.metadata] reference in module
docstring (now [tool.dqx_app.metadata], matching _read_app_metadata)
and align the example build tag to lowercase t to match
time.strftime("b%Y%m%dt%H%M%S").
* databricks.yml: add Mode B target dqx-app-demo for the
fevm-tsh-dqx-app-demo workspace, reusing the pre-existing
dqx-studio-sql-warehouse (e864734d584d98a6) and
dqx-studio-lakebase-v2 instance from a prior apx-era deploy.
…slabs/dqx into app/remove_apx_dependency
* Rename frontend folder `components/apx/` → `components/layout/` (9 components: Navbar, SidebarLayout, theme-provider, mode-toggle, Logo, HeaderUserMenu, SidebarUserFooter, PageBreadcrumb, LanguageSelector) and update all 18 importing files. * Strip "replaces apx X" / "matching the legacy apx Y" comments and docstrings from orval.config.ts, vite.config.ts, scripts/build_app.py, scripts/dev.py, and pyproject.toml. The code itself was already apx-free; this removes the breadcrumbs pointing back to it. * Update ui/CLAUDE.md to drop the historical-artifact note on the renamed folder. * Drop ui/lib/i18n/locales/de.json. Verified with make fmt, make test (1141 passed), make app-test (591 passed), and make app-check (TypeScript + basedpyright clean).
…slabs/dqx into app/remove_apx_dependency
The app previously relied on the unpublished
apxpackage (github.com/databricks-solutions/apx) to drive its build and dev workflows. Drop the dependency and reimplement what we need ourselves:scripts/build_app.py: replaces
apx build. Generates the FastAPI OpenAPI schema, runs orval (now configured via orval.config.ts) to regenerate ui/lib/api.ts, runsvite build, and produces the application wheel +app.yml+requirements.txtunder.build/. The wheel carries a build-tag local-version segment (e.g..b20260530t012345) baked into both the filename and the internal METADATA, so successive deploys at the same git commit always reinstall fresh code in the Databricks Apps persistent venv. This obsoletesscripts/_align_wheel_version.py(now removed), which used to repair apx's filename-only mangling after the fact.scripts/dev.py: replaces
apx dev start. Runs uvicorn with--reloadon a fixed internal port and serves the SPA via Vite on :9001, proxying/apiand/docsto the backend. SIGINT is forwarded to both children and either child crashing tears the other down, so no orphan processes are left behind.pyproject.toml: drop the
apxdependency-group entry and the[tool.uv.sources].apxgit source. Rename the metadata table from[tool.apx.metadata]to[tool.dqx_app.metadata](same schema, new namespace) and document its consumers (build_app.py, vite.config.ts). Addbasedpyrightto the dev group.databricks.yml: rewrite
artifacts.default.buildto callscripts/build_app.pyinstead ofapx build. Add explicit Mode A (bundle-managed) vs Mode B (external/reuse) documentation on thesql_warehouse_idvariable; default to empty so a target that forgets to set it fails fast.post_deploy_grants.sh: extend the warehouse grant path so external warehouses (Mode B) get
CAN_USEgranted to the app SP, job SP, and theusersgroup via the warehouses permissions API — the bundle doesn't own the warehouse in that mode, so the bundle'spermissions:block can't do it.Documentation refresh across DEPLOYMENT.md, DEVELOPMENT.md, README.md, AGENTS.md, the UI/backend CLAUDE.md files, and docs/dqx/docs/{installation,dev/contributing}.mdx to reflect the new build/dev entry points and warehouse modes.
No runtime behaviour change in the app itself: app/src/...backend/app.py is touched only to pick up the new metadata namespace.
Changes
Linked issues
Resolves #..
Tests
Documentation and Demos