Repository navigation
feat(databricks): add databricks-metric-view target, plus two cross-dialect fixes it surfaced - #14
Merged
Merged
Conversation
Port from worktree-databricks-target (tip b3cf596), adapted for the dialect package (renamed from layer on main). Emits one Unity Catalog metric view YAML per IR table, with joins, fields, and measures sourced from metrics (or raw measures for measures-only tables), synthesising a row-count measure when a table has none, and dropping a dimension that collides with a same-named measure. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Rename snowflakeTargets -> warehouseTargets and add databricks-metric-view, so a profile with that target-dialect and no database fails fast in loadProfile instead of emitting an unqualified source reference. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extend the shared ecommerce fixture with obt_sales (measures-only OBT, no metrics.yml entry) and a precomputed fct_orders.aov column that collides with the computed aov metric, then port the databricks-metric-view integration tests (emitDatabricks, TestDatabricksMetricViewStructure, TestDatabricksMetricViewGolden) and regenerate goldens. Other targets' goldens churn additively from the new fixture rows; TestEcommerceCortexStructure's expected table/ relationship lists are updated to include obt_sales. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Port docs/design-databricks-metric-view.md and docs/plans/2026-07-20-databricks-metric-view.md from worktree-databricks-target, updating layer/ path references to dialect/ and flag-based CLI examples (--target-type, --database) to the profiles-only model (semglot.yaml profile, loadProfile). Add a databricks-metric-view row to dialect/README.md's IR-mapping table and mention the new target in README.md's dialect list and feature table. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
buildView was all-or-nothing: a table with any metric emitted measures from metrics only, silently dropping every raw ir.Measure even ones no metric covered. fct_order_lines lost net_line_revenue this way. Emit metric-derived measures first, then append raw measures not already represented by lowercased name OR by rendered expression, so a near-duplicate raw measure (same expr as a metric, different name, e.g. orders_count beside an orders metric) is still suppressed while genuinely uncovered raw measures survive. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
dbxStripSourceQualifier matched strings.ToLower(table)+"." against expressions that renderSQL emits with the table's original case (ir.Col.Table/ir.Agg.Table preserve whatever case the dbt YAML used). A semantic model named e.g. FCT_Orders therefore kept its qualifier in the rendered measure (sum(FCT_Orders.order_gross)); the metric view's source relation is aliased `source`, so Databricks cannot resolve it and rejects the entire view, losing every other measure with it. Match case-insensitively with a compiled regexp anchored on a leading word boundary, so the qualifier strips regardless of the source table's case. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both docs described the original fact-selection rule (one view per table with >=1 metric, pure-dimension tables skipped) — a bug fixed during implementation that emitted only 6 of 38 views on the real dataset. The shipped rule is one metric view per IR table, skipping only a table left with zero fields. Update design-databricks-metric-view.md, the artefact readers cite, to describe current behaviour: measures come from metrics plus uncovered raw measures (by name or expression), a synthesised row_count when a table would otherwise have none, and a field dropped when its name collides with a measure. The plan doc is a historical record of an executed plan, so leave it otherwise unedited: add a dated erratum near the top and mark the specific lines that assert the superseded rule as shipped, rather than rewriting them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
MetricFlow's `expr:` on a measure is optional and defaults to the measure name, exactly like the entity and dimension loops just above this one in dialect/dbt.go — but the measure loop had no such fallback, so `expr: sum()` was emitted for the databricks target (rejected outright: WRONG_NUM_ARGS.WITHOUT_SUGGESTION) and `expr: ""` for cortex. Fixed in the parser so every target benefits. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The expression dedup in the raw-measure loop existed only to suppress a raw measure that near-duplicates a METRIC (same expr, different name — e.g. orders_count beside an orders metric). Extending it to raw-vs-raw silently dropped legitimately distinct raw measures that happen to share an agg+column (e.g. revenue and net_revenue, both sum(amount)), even though they carry different names, labels, descriptions and synonyms. Raw-vs-raw is now deduped by name only; raw-vs-metric still by name or expression. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
renderSQL preserves a metric's rendered column case verbatim (count(distinct ORDER_ID) for an uppercase dbt column), while the raw measure side always lowercases its column via aggExpr. The near-duplicate dedup (raw measure suppressed when a metric already emits its expression) compared these directly, so a project with uppercase column exprs let the near-duplicate through under a case mismatch. The lookup now normalises both sides to lowercase; the expression actually written to YAML is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A joined dimension is qualified as <join>.<expr> to reference it
against the source alias, unconditionally. That only produces valid
SQL when Expr is a bare column — a compound expression (e.g.
coalesce(region, 'unknown'), date_trunc('month', ordered_at)) became
"dim_customer.coalesce(region, 'unknown')", which Databricks rejects,
taking the entire view down with it. Now only bare-identifier exprs
are prefixed and emitted; a compound one is skipped and noted in the
view comment instead. Documented as a v1 limitation already; this
just stops it from failing the whole view.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
usedNames was populated only AFTER the metric loop finished, so two metrics on the same table whose lowercased names collide (e.g. AOV and aov) both emitted — Databricks rejects the entire view for a duplicate name, the very failure the field/measure dedup elsewhere was written to prevent. usedNames is now checked and populated inside the loop; a colliding metric is skipped and noted, first one wins. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
aggExpr fell through to `agg+"("+col+")"` for any agg it didn't know,
so dbt/MetricFlow shapes it doesn't handle explicitly (sum_boolean,
percentile, an omitted agg or expr) rendered invalid Databricks SQL.
One invalid measure rejects the ENTIRE metric view, not just that
measure, losing every other measure/dimension/join on the table.
- aggExpr (dialect/dbt.go) now lowers sum_boolean to an explicit CASE
(MetricFlow's "sum a boolean as 1/0" semantics) and gives median its
own case instead of an accidental passthrough.
- dbxValidMeasureExpr (dialect/databricks_metric_view.go) is a single
choke point both the metric-derived and raw-measure paths in
buildView run every rendered expr through: it requires every
function call in the expr to be a known-safe aggregate
(sum/count/avg/min/max/median) with a non-empty argument. A measure
that fails is skipped and noted rather than emitted.
- Add a structural invariant test (the point of this round, after
three review rounds of "an expression Databricks would reject
reached the YAML"): it builds a table with sum_boolean, percentile,
an omitted agg, an omitted expr, and a normal sum, then generically
parses every measures[].expr in every emitted view and asserts each
one is built only from known-safe aggregate calls, rather than
hardcoding the five reproduction cases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
buildView only read m.Relationships from the model, so a note semglot attaches when it can't transpile something (e.g. an unresolved dbt measure reference) reached Cortex, supersimple, snowflake-semantic- view, nao-yaml and nao-context-rules but silently never reached databricks-metric-view. Genie was systematically told less than every sibling target about what semglot could not transpile. Seed buildView's notes slice with a clone of m.Notes (Emit stays read-only over m) so every emitted view carries every model-level note, alongside that view's own degrade notes. Chosen over attaching a note only to the view whose table name it mentions: simpler, and a note is never silently absent just because a user opened one view instead of another. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
display_name and synonyms are emitted unconditionally whenever the IR carries a label or synonyms, but Databricks requires Runtime 17.3+ for those keys (the base metric-view YAML v1.1 shape needs 17.2+). On an older warehouse the view is rejected outright. Document the floor rather than gate the keys behind an option, in the design doc, dialect/README.md's target row, and the top-level README's dialect- support notes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- dbxDegrade's returned reason strings still used an em-dash, unlike every other note string in this file; land verbatim in emitted comment copy, so match the established phrasing style. - dbxField.DisplayName could never be assigned (ir.Field has no Label to source it from) and nothing set it; remove rather than carry a dead struct field. - design-databricks-metric-view.md: the target is implemented and live-validated now, not "pending implementation"; fix the raw- measure dedup paragraph, which still said raw measures dedup by both name and expression (the rule since the raw-vs-raw dedup fix: raw-vs-metric by name or expression, raw-vs-raw by name only). - dialect/README.md: classify the two unclassified `--` cells for the new databricks-metric-view column (data types, primary keys) under Gaps vs. limits, consistent with how the other dialects' cells are written. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…twice A role-playing dimension (fct_orders referencing dim_customer as both ship-to and bill-to) is ordinary star-schema modelling, and the IR carries both FKs correctly. Every emitter that names relationships keyed them by the related table alone, so the second FK was either silently dropped (databricks-metric-view, supersimple, whose relations map collided on the key) or emitted under a duplicate name the consumer rejects (cortex, snowflake-semantic-view; Snowflake requires unique relationship names). Add a shared relRoleSuffix helper so all four disambiguate identically, by the relationship's own left column(s). A table pair with a single relationship keeps its existing name, so no existing golden churns. nao-context-rules was already correct (it writes column-qualified prose) and nao-yaml does not model relationships. Adds the shape to the shared ecommerce fixture (fct_orders.billing_customer_sk) and TestRolePlayingDimensionAllTargets, which pins the behaviour for all five relationship-carrying targets. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… table Measures are built before fields and seed the field-dedup set, so a table whose every dimension collides with a measure name ends up with no fields. A metric view requires at least one dimension, so no view is written. That drop had no test, in an emitter where five separate silent-drop defects have already been found. Adds a control case so the assertion cannot pass vacuously. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Emit(m, dir) error becomes Emit(m, dir) (warnings []string, err error) across all seven dialects, so a source construct an emitter degrades or drops (a metric with no target primitive, a measure with an unrenderable aggregation, a whole table with no dimension) has a channel out other than being written into the artifact. Each emitter still writes its own degrade notes plus m.Notes into the artifact exactly as before; only its OWN notes are returned, since the CLI already prints m.Notes separately and returning the combined slice would double-print it. cortex additionally folds cortexTypeGaps(m) into its returned warnings, replacing the CLI's hardcoded special case. databricks-metric-view also now reports the one path that silently dropped a whole table (zero dimensions survive the measure/dimension name collision), naming the table and why. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…and call sites
main.go captures emitter.Emit's warnings and prints them after the existing
model.Notes block, then deletes the hardcoded
`if spec.TargetDialect == "cortex" { ... CortexTypeGaps ... }` branch it was
standing in for. CortexTypeGaps has no callers left outside the dialect
package, so it's unexported to cortexTypeGaps.
Every other .Emit( call site (test fixtures, integration tests, the registry's
fake emitter) is updated mechanically to `_, err := ...Emit(...)` to match the
new two-return signature; none of these needed behavioral changes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TestDatabricksMetricViewZeroFieldsSkipped now asserts a warning naming the table it silently dropped, via a new emitDbxW test helper (emitDbx plus warnings) alongside the existing emitDbx used everywhere else. TestNaoYamlWarningsExcludeModelNotes guards the double-print regression: an emitter's returned warnings must carry only its own degrade notes, never model.Notes verbatim, while the emitted artifact still carries both. TestBuildCmdSurfacesEmitterWarnings exercises the real CLI binary path end to end and confirms the type-gap warnings that used to come from main.go's deleted cortex special case still reach stderr through the generic channel. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a
databricks-metric-viewtarget, plus two changes that started as findings against it and turned out to affect every dialect.Three separable pieces of work, in the order they were built. Reviewing them in that order is easiest, since 2 and 3 both came out of reviewing 1.
1. The
databricks-metric-viewtarget (commits 1 to 17)Transpiles a dbt semantic layer into Unity Catalog metric views, the YAML semantic layer Databricks AI/BI Genie grounds its answers on. Genie resolves metrics from metric views rather than inferring them, and auto-imports their agent metadata (
display_name,synonyms), which makes metric views the Databricks analog of the existingcortextarget.One
<table>.yamlper IR table, rooted at that table, with direct IR relationships asjoins.cortex)--databaseis the Unity Catalog catalog,--schemathe source schema. Required, so a missing catalog fails fast rather than emitting an unqualifiedsourceir.Measures not already covered by name or expression, else a synthesisedcount(1)Four details that are easy to get wrong, each covered by a test:
onjoin key is emitted double quoted. Databricks parses metric-view YAML as YAML 1.1, where a bareonkey is the booleantrue, silently corrupting the join.source, not its physical name, sosum(fct_orders.order_gross)would not resolve.sum_booleanis mapped tosum(case when x then 1 else 0 end);percentileand an omittedaggare skipped with a note, because one invalid measure expression loses the entire view.Requires Databricks Runtime 17.3+ (
display_name/synonyms); documented in the design doc anddialect/README.md.Validated against a live workspace, not just goldens
Deployed to a real Unity Catalog with the full 59 table ecomm marts loaded:
orders,net_revenue,aov,refund_rate)That reaches parity with
cortex,snowflake-semantic-viewandsupersimple, which all emit 38 tables for this project.Bugs the live run caught that the tests did not
Each fix adds the missing shape to the fixture so it cannot regress silently.
obt_sales) vanished, whilecortexcarried it.count(1), which asserts no business semantics and so carries no source information.expremittedsum(), which Databricks rejects, losing the whole view.expris optional in MetricFlow and defaults to the measure name; the parser now defaults it, which also fixescortexemittingexpr: "".roas) was rejected by Databricks, where a precomputed column shares a metric's name.A structural invariant test now asserts that every emitted measure expression, in every view, is built only from known-safe aggregate calls. That guards the defect class rather than the five instances.
2. Role-playing dimensions, all dialects (commit 18)
A fact table referencing the same dimension twice (ship-to and bill-to customer, order_date and ship_date to a shared date dimension) is ordinary star-schema modelling, and the IR carries both FKs correctly. Every emitter that names relationships keyed them by the related table alone, so the second FK was either silently dropped or emitted under a duplicate name the consumer rejects.
relationsmap collided on the key)nameA shared
relRoleSuffixhelper makes all four disambiguate identically, by the relationship's own left column(s). A table pair with a single relationship keeps its existing name, so no existing golden churns.nao-context-ruleswas already correct (it writes column-qualified prose) andnao-yamldoes not model relationships.Adds
fct_orders.billing_customer_skto the shared fixture andTestRolePlayingDimensionAllTargets, which pins the behaviour for all five relationship-carrying targets with name and column assertions rather than counts.3. Warnings in the
Emittercontract (commits 20 to 22)Emitters degrade or drop constructs they cannot represent, but those decisions only ever reached the emitted artifact (a
comment,custom_instructions,NOTES.md). Nothing reached the person running the CLI, and one case reached nothing at all: a table skipped for having no emittable dimensions produced no file and therefore no note.The channel could not be added by accumulating state on the emitter, because every dialect is registered as a value singleton (
Register(cortex{})) with value receivers onEmit, so state would either not persist or contaminate a process-wide instance.Emitters return only their own degrade notes, not the
m.Notesthey also fold into the artifact, so model-level notes do not double-print alongside the CLI's existing block. A test pins that.This also deletes a workaround:
main.gohad a hardcodedif spec.TargetDialect == "cortex"branch calling the exportedCortexTypeGaps, which existed precisely because this channel was missing. Cortex returns those as warnings and the special case is gone;CortexTypeGapsis unexported.Zero golden churn, verified by regenerating every golden and getting an empty diff. This changes only what the CLI prints, never an emitted artifact.
Testing
dialect/databricks_metric_view_test.go, including the measure-expression invariant and the zero-fields skip with a control caseTestRolePlayingDimensionAllTargetsacross five targetsgo test ./...,go vet ./..., gofmt cleanKnown limitations
Documented in
docs/design-databricks-metric-view.md, not hidden: joined dimensions with compound expressions are skipped with a note rather than emitted invalid; time-dimension granularity is not expressed asdate_trunc; cumulative and conversion metric degrade paths lack fixture coverage in three targets (#15).Genie itself has not been validated end to end. The claim that metric views are what Genie grounds on comes from Databricks documentation, not from our own observation.
🤖 Generated with Claude Code