Skip to content

feat(databricks): add databricks-metric-view target, plus two cross-dialect fixes it surfaced - #14

Merged
andrebaaij merged 22 commits into
mainfrom
databricks-metric-view-main
Jul 21, 2026
Merged

andrebaaij merged 22 commits into
mainfrom
databricks-metric-view-main

Conversation

@andrebaaij

@andrebaaij andrebaaij commented Jul 20, 2026 •

Copy link
Copy Markdown
Contributor

Adds a databricks-metric-view target, plus two changes that started as findings against it and turned out to affect every dialect.

Three separable pieces of work, in the order they were built. Reviewing them in that order is easiest, since 2 and 3 both came out of reviewing 1.


1. The databricks-metric-view target (commits 1 to 17)

Transpiles a dbt semantic layer into Unity Catalog metric views, the YAML semantic layer Databricks AI/BI Genie grounds its answers on. Genie resolves metrics from metric views rather than inferring them, and auto-imports their agent metadata (display_name, synonyms), which makes metric views the Databricks analog of the existing cortex target.

One <table>.yaml per IR table, rooted at that table, with direct IR relationships as joins.

aspect choice
Output Raw metric-view YAML, one file per table (parallels cortex)
Identity --database is the Unity Catalog catalog, --schema the source schema. Required, so a missing catalog fails fast rather than emitting an unqualified source
Measures Metrics first, then raw ir.Measures not already covered by name or expression, else a synthesised count(1)
Fields Own dimensions bare, joined dimensions prefixed by join name, deduped
Degrades Window/conversion metrics, cross-grain derived metrics, and unrenderable aggregations become a note rather than SQL we cannot stand behind

Four details that are easy to get wrong, each covered by a test:

  • The on join key is emitted double quoted. Databricks parses metric-view YAML as YAML 1.1, where a bare on key is the boolean true, silently corrupting the join.
  • Source columns in measure expressions are emitted bare. The source relation is aliased source, not its physical name, so sum(fct_orders.order_gross) would not resolve.
  • A field whose name collides with a measure is dropped. Databricks requires measure and dimension names to be unique within a view and rejects the whole view otherwise. The computed measure is canonical.
  • Aggregations are validated before rendering. sum_boolean is mapped to sum(case when x then 1 else 0 end); percentile and an omitted agg are skipped with a note, because one invalid measure expression loses the entire view.

Requires Databricks Runtime 17.3+ (display_name / synonyms); documented in the design doc and dialect/README.md.

Validated against a live workspace, not just goldens

Deployed to a real Unity Catalog with the full 59 table ecomm marts loaded:

  • 38/38 metric views created, zero failures
  • Measures match hand written SQL to 6 decimal places over 308k orders (orders, net_revenue, aov, refund_rate)
  • Joins resolve without fan out: segment totals sum exactly to the base row count
  • Derived ratios re-aggregate correctly rather than averaging averages

That reaches parity with cortex, snowflake-semantic-view and supersimple, which all emit 38 tables for this project.

Bugs the live run caught that the tests did not

Each fix adds the missing shape to the fixture so it cannot regress silently.

  1. Measures-only tables were dropped. A table with dbt measures but no metric referencing them (obt_sales) vanished, while cortex carried it.
  2. 32 of 38 tables were dropped. The fact-selection rule skipped every dimension-only table. A metric view requires at least one measure, hence the synthesised count(1), which asserts no business semantics and so carries no source information.
  3. Raw measures were dropped from tables that had metrics, losing 10 real measures across 5 views on the real dataset (discounts, tax, clicks, impressions, refund totals).
  4. A measure with no expr emitted sum(), which Databricks rejects, losing the whole view. expr is optional in MetricFlow and defaults to the measure name; the parser now defaults it, which also fixes cortex emitting expr: "".
  5. A name collision (roas) was rejected by Databricks, where a precomputed column shares a metric's name.

A structural invariant test now asserts that every emitted measure expression, in every view, is built only from known-safe aggregate calls. That guards the defect class rather than the five instances.


2. Role-playing dimensions, all dialects (commit 18)

A fact table referencing the same dimension twice (ship-to and bill-to customer, order_date and ship_date to a shared date dimension) is ordinary star-schema modelling, and the IR carries both FKs correctly. Every emitter that names relationships keyed them by the related table alone, so the second FK was either silently dropped or emitted under a duplicate name the consumer rejects.

target before
databricks-metric-view second FK silently dropped
supersimple second FK silently dropped (its relations map collided on the key)
cortex both emitted, duplicate name
snowflake-semantic-view both emitted, duplicate name, so the DDL is invalid (Snowflake requires unique relationship names)

A shared relRoleSuffix helper makes all four disambiguate identically, by the relationship's own left column(s). A table pair with a single relationship keeps its existing name, so no existing golden churns.

nao-context-rules was already correct (it writes column-qualified prose) and nao-yaml does not model relationships.

Adds fct_orders.billing_customer_sk to the shared fixture and TestRolePlayingDimensionAllTargets, which pins the behaviour for all five relationship-carrying targets with name and column assertions rather than counts.


3. Warnings in the Emitter contract (commits 20 to 22)

Emit(m *ir.Model, dir string) (warnings []string, err error)

Emitters degrade or drop constructs they cannot represent, but those decisions only ever reached the emitted artifact (a comment, custom_instructions, NOTES.md). Nothing reached the person running the CLI, and one case reached nothing at all: a table skipped for having no emittable dimensions produced no file and therefore no note.

The channel could not be added by accumulating state on the emitter, because every dialect is registered as a value singleton (Register(cortex{})) with value receivers on Emit, so state would either not persist or contaminate a process-wide instance.

Emitters return only their own degrade notes, not the m.Notes they also fold into the artifact, so model-level notes do not double-print alongside the CLI's existing block. A test pins that.

This also deletes a workaround: main.go had a hardcoded if spec.TargetDialect == "cortex" branch calling the exported CortexTypeGaps, which existed precisely because this channel was missing. Cortex returns those as warnings and the special case is gone; CortexTypeGaps is unexported.

Zero golden churn, verified by regenerating every golden and getting an empty diff. This changes only what the CLI prints, never an emitted artifact.


Testing

  • 20 unit tests in dialect/databricks_metric_view_test.go, including the measure-expression invariant and the zero-fields skip with a control case
  • Golden and structure tests for the new target; TestRolePlayingDimensionAllTargets across five targets
  • Fixture extended with the shapes that hid the bugs: a measures-only table, a precomputed column colliding with a metric name, and a second FK to the same dimension
  • go test ./..., go vet ./..., gofmt clean

Known limitations

Documented in docs/design-databricks-metric-view.md, not hidden: joined dimensions with compound expressions are skipped with a note rather than emitted invalid; time-dimension granularity is not expressed as date_trunc; cumulative and conversion metric degrade paths lack fixture coverage in three targets (#15).

Genie itself has not been validated end to end. The claim that metric views are what Genie grounds on comes from Databricks documentation, not from our own observation.

🤖 Generated with Claude Code

andrebaaij and others added 4 commits July 20, 2026 22:04
Port from worktree-databricks-target (tip b3cf596), adapted for the
dialect package (renamed from layer on main). Emits one Unity Catalog
metric view YAML per IR table, with joins, fields, and measures sourced
from metrics (or raw measures for measures-only tables), synthesising a
row-count measure when a table has none, and dropping a dimension that
collides with a same-named measure.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename snowflakeTargets -> warehouseTargets and add
databricks-metric-view, so a profile with that target-dialect and no
database fails fast in loadProfile instead of emitting an unqualified
source reference.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the shared ecommerce fixture with obt_sales (measures-only OBT,
no metrics.yml entry) and a precomputed fct_orders.aov column that
collides with the computed aov metric, then port the
databricks-metric-view integration tests (emitDatabricks,
TestDatabricksMetricViewStructure, TestDatabricksMetricViewGolden) and
regenerate goldens. Other targets' goldens churn additively from the
new fixture rows; TestEcommerceCortexStructure's expected table/
relationship lists are updated to include obt_sales.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Port docs/design-databricks-metric-view.md and
docs/plans/2026-07-20-databricks-metric-view.md from
worktree-databricks-target, updating layer/ path references to
dialect/ and flag-based CLI examples (--target-type, --database) to
the profiles-only model (semglot.yaml profile, loadProfile). Add a
databricks-metric-view row to dialect/README.md's IR-mapping table and
mention the new target in README.md's dialect list and feature table.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
andrebaaij and others added 18 commits July 20, 2026 23:03
buildView was all-or-nothing: a table with any metric emitted measures
from metrics only, silently dropping every raw ir.Measure even ones no
metric covered. fct_order_lines lost net_line_revenue this way.

Emit metric-derived measures first, then append raw measures not
already represented by lowercased name OR by rendered expression, so a
near-duplicate raw measure (same expr as a metric, different name,
e.g. orders_count beside an orders metric) is still suppressed while
genuinely uncovered raw measures survive.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dbxStripSourceQualifier matched strings.ToLower(table)+"." against
expressions that renderSQL emits with the table's original case
(ir.Col.Table/ir.Agg.Table preserve whatever case the dbt YAML used).
A semantic model named e.g. FCT_Orders therefore kept its qualifier in
the rendered measure (sum(FCT_Orders.order_gross)); the metric view's
source relation is aliased `source`, so Databricks cannot resolve it
and rejects the entire view, losing every other measure with it.

Match case-insensitively with a compiled regexp anchored on a leading
word boundary, so the qualifier strips regardless of the source
table's case.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both docs described the original fact-selection rule (one view per
table with >=1 metric, pure-dimension tables skipped) — a bug fixed
during implementation that emitted only 6 of 38 views on the real
dataset. The shipped rule is one metric view per IR table, skipping
only a table left with zero fields.

Update design-databricks-metric-view.md, the artefact readers cite, to
describe current behaviour: measures come from metrics plus uncovered
raw measures (by name or expression), a synthesised row_count when a
table would otherwise have none, and a field dropped when its name
collides with a measure.

The plan doc is a historical record of an executed plan, so leave it
otherwise unedited: add a dated erratum near the top and mark the
specific lines that assert the superseded rule as shipped, rather than
rewriting them.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MetricFlow's `expr:` on a measure is optional and defaults to the
measure name, exactly like the entity and dimension loops just above
this one in dialect/dbt.go — but the measure loop had no such
fallback, so `expr: sum()` was emitted for the databricks target
(rejected outright: WRONG_NUM_ARGS.WITHOUT_SUGGESTION) and `expr: ""`
for cortex. Fixed in the parser so every target benefits.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The expression dedup in the raw-measure loop existed only to suppress
a raw measure that near-duplicates a METRIC (same expr, different
name — e.g. orders_count beside an orders metric). Extending it to
raw-vs-raw silently dropped legitimately distinct raw measures that
happen to share an agg+column (e.g. revenue and net_revenue, both
sum(amount)), even though they carry different names, labels,
descriptions and synonyms. Raw-vs-raw is now deduped by name only;
raw-vs-metric still by name or expression.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
renderSQL preserves a metric's rendered column case verbatim
(count(distinct ORDER_ID) for an uppercase dbt column), while the raw
measure side always lowercases its column via aggExpr. The
near-duplicate dedup (raw measure suppressed when a metric already
emits its expression) compared these directly, so a project with
uppercase column exprs let the near-duplicate through under a case
mismatch. The lookup now normalises both sides to lowercase; the
expression actually written to YAML is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A joined dimension is qualified as <join>.<expr> to reference it
against the source alias, unconditionally. That only produces valid
SQL when Expr is a bare column — a compound expression (e.g.
coalesce(region, 'unknown'), date_trunc('month', ordered_at)) became
"dim_customer.coalesce(region, 'unknown')", which Databricks rejects,
taking the entire view down with it. Now only bare-identifier exprs
are prefixed and emitted; a compound one is skipped and noted in the
view comment instead. Documented as a v1 limitation already; this
just stops it from failing the whole view.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
usedNames was populated only AFTER the metric loop finished, so two
metrics on the same table whose lowercased names collide (e.g. AOV
and aov) both emitted — Databricks rejects the entire view for a
duplicate name, the very failure the field/measure dedup elsewhere
was written to prevent. usedNames is now checked and populated inside
the loop; a colliding metric is skipped and noted, first one wins.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aggExpr fell through to `agg+"("+col+")"` for any agg it didn't know,
so dbt/MetricFlow shapes it doesn't handle explicitly (sum_boolean,
percentile, an omitted agg or expr) rendered invalid Databricks SQL.
One invalid measure rejects the ENTIRE metric view, not just that
measure, losing every other measure/dimension/join on the table.

- aggExpr (dialect/dbt.go) now lowers sum_boolean to an explicit CASE
  (MetricFlow's "sum a boolean as 1/0" semantics) and gives median its
  own case instead of an accidental passthrough.
- dbxValidMeasureExpr (dialect/databricks_metric_view.go) is a single
  choke point both the metric-derived and raw-measure paths in
  buildView run every rendered expr through: it requires every
  function call in the expr to be a known-safe aggregate
  (sum/count/avg/min/max/median) with a non-empty argument. A measure
  that fails is skipped and noted rather than emitted.
- Add a structural invariant test (the point of this round, after
  three review rounds of "an expression Databricks would reject
  reached the YAML"): it builds a table with sum_boolean, percentile,
  an omitted agg, an omitted expr, and a normal sum, then generically
  parses every measures[].expr in every emitted view and asserts each
  one is built only from known-safe aggregate calls, rather than
  hardcoding the five reproduction cases.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
buildView only read m.Relationships from the model, so a note semglot
attaches when it can't transpile something (e.g. an unresolved dbt
measure reference) reached Cortex, supersimple, snowflake-semantic-
view, nao-yaml and nao-context-rules but silently never reached
databricks-metric-view. Genie was systematically told less than every
sibling target about what semglot could not transpile.

Seed buildView's notes slice with a clone of m.Notes (Emit stays
read-only over m) so every emitted view carries every model-level
note, alongside that view's own degrade notes. Chosen over attaching a
note only to the view whose table name it mentions: simpler, and a
note is never silently absent just because a user opened one view
instead of another.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
display_name and synonyms are emitted unconditionally whenever the IR
carries a label or synonyms, but Databricks requires Runtime 17.3+ for
those keys (the base metric-view YAML v1.1 shape needs 17.2+). On an
older warehouse the view is rejected outright. Document the floor
rather than gate the keys behind an option, in the design doc,
dialect/README.md's target row, and the top-level README's dialect-
support notes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- dbxDegrade's returned reason strings still used an em-dash, unlike
  every other note string in this file; land verbatim in emitted
  comment copy, so match the established phrasing style.
- dbxField.DisplayName could never be assigned (ir.Field has no Label
  to source it from) and nothing set it; remove rather than carry a
  dead struct field.
- design-databricks-metric-view.md: the target is implemented and
  live-validated now, not "pending implementation"; fix the raw-
  measure dedup paragraph, which still said raw measures dedup by
  both name and expression (the rule since the raw-vs-raw dedup fix:
  raw-vs-metric by name or expression, raw-vs-raw by name only).
- dialect/README.md: classify the two unclassified `--` cells for the
  new databricks-metric-view column (data types, primary keys) under
  Gaps vs. limits, consistent with how the other dialects' cells are
  written.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…twice

A role-playing dimension (fct_orders referencing dim_customer as both ship-to
and bill-to) is ordinary star-schema modelling, and the IR carries both FKs
correctly. Every emitter that names relationships keyed them by the related
table alone, so the second FK was either silently dropped
(databricks-metric-view, supersimple, whose relations map collided on the key)
or emitted under a duplicate name the consumer rejects (cortex,
snowflake-semantic-view; Snowflake requires unique relationship names).

Add a shared relRoleSuffix helper so all four disambiguate identically, by the
relationship's own left column(s). A table pair with a single relationship
keeps its existing name, so no existing golden churns. nao-context-rules was
already correct (it writes column-qualified prose) and nao-yaml does not model
relationships.

Adds the shape to the shared ecommerce fixture (fct_orders.billing_customer_sk)
and TestRolePlayingDimensionAllTargets, which pins the behaviour for all five
relationship-carrying targets.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… table

Measures are built before fields and seed the field-dedup set, so a table whose
every dimension collides with a measure name ends up with no fields. A metric
view requires at least one dimension, so no view is written. That drop had no
test, in an emitter where five separate silent-drop defects have already been
found.

Adds a control case so the assertion cannot pass vacuously.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Emit(m, dir) error becomes Emit(m, dir) (warnings []string, err error) across
all seven dialects, so a source construct an emitter degrades or drops
(a metric with no target primitive, a measure with an unrenderable
aggregation, a whole table with no dimension) has a channel out other than
being written into the artifact. Each emitter still writes its own degrade
notes plus m.Notes into the artifact exactly as before; only its OWN notes are
returned, since the CLI already prints m.Notes separately and returning the
combined slice would double-print it. cortex additionally folds
cortexTypeGaps(m) into its returned warnings, replacing the CLI's hardcoded
special case. databricks-metric-view also now reports the one path that
silently dropped a whole table (zero dimensions survive the measure/dimension
name collision), naming the table and why.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…and call sites

main.go captures emitter.Emit's warnings and prints them after the existing
model.Notes block, then deletes the hardcoded
`if spec.TargetDialect == "cortex" { ... CortexTypeGaps ... }` branch it was
standing in for. CortexTypeGaps has no callers left outside the dialect
package, so it's unexported to cortexTypeGaps.

Every other .Emit( call site (test fixtures, integration tests, the registry's
fake emitter) is updated mechanically to `_, err := ...Emit(...)` to match the
new two-return signature; none of these needed behavioral changes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TestDatabricksMetricViewZeroFieldsSkipped now asserts a warning naming the
table it silently dropped, via a new emitDbxW test helper (emitDbx plus
warnings) alongside the existing emitDbx used everywhere else.
TestNaoYamlWarningsExcludeModelNotes guards the double-print regression: an
emitter's returned warnings must carry only its own degrade notes, never
model.Notes verbatim, while the emitted artifact still carries both.
TestBuildCmdSurfacesEmitterWarnings exercises the real CLI binary path end to
end and confirms the type-gap warnings that used to come from main.go's
deleted cortex special case still reach stderr through the generic channel.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@andrebaaij andrebaaij changed the title feat(databricks): add databricks-metric-view target feat(databricks): add databricks-metric-view target, plus two cross-dialect fixes it surfaced Jul 21, 2026
@andrebaaij
andrebaaij merged commit 901452e into main Jul 21, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant