Skip to content

feat(ossie): Apache Ossie as a source and target dialect - #19

Merged
andrebaaij merged 41 commits into
mainfrom
feat/ossie-dialect
Aug 13, 2026
Merged

andrebaaij merged 41 commits into
mainfrom
feat/ossie-dialect

Conversation

@andrebaaij

Copy link
Copy Markdown
Contributor

What and why

Adds ossie — a source and target dialect for the Apache Ossie Core Metadata Specification (core-spec 0.2.0.dev0), the vendor-neutral semantic-model interchange format.

Because everything routes through the IR, this makes every existing emit-only dialect reachable from an OSI file, and makes semglot's IR reachable from any tool that exports OSI. It is also semglot's second source dialect, which turned out to be the interesting part: a lot of code quietly assumed dbt was the only thing that could produce IR.

Also in here

Four IR fields, each added because Ossie carries something the IR had no slot for, and each wired across all nine dialects — structurally where a target has a slot, folded into prose or warned where it doesn't:

  • Table.Synonyms — Ossie datasets carry them; so do Snowflake semantic views and Cortex, which semglot simply wasn't filling. Closes the supersimple synonyms gap dialect/README.md recorded.
  • Table.Source — without it, ossie → ossie silently relocated every table: PROD.SALES.orders_v1 came back as WAREHOUSE.PUBLIC.orders.
  • Relationship.Name and Relationship.Synonyms — the declared join name was regenerated from <from>_to_<to>, and relationship ai_context wasn't parsed at all, so join hints like [who bought, what was sold] vanished before any code saw them.

Plus a shared SQL-expression parser extracted from dbt.go, and a shared emit-side helper that hoists inlined aggregates so dbt, snowflake-semantic-view, lightdash and supersimple can express SUM(a)/SUM(b) through their existing reference-handling paths.

dbt_emit also learned to warn. It was the only emitter in the codebase with zero degradation sites — safe while dbt was both the sole source and a target, wrong the moment a second source existed.

Testing

Four layers, three built on Apache Ossie's own material (vendored at pinned commit 88e0011, ASF headers retained, provenance recorded):

  • Parse conformance against their example and converter fixtures.
  • Cross-implementation agreement — semglot's dbt → ossie output compared semantically against what Ossie's own dbt converter produces. Exactly one root divergence survives: their MetricFlow-inherited COUNT(x) → SUM(CASE WHEN x IS NOT NULL THEN 1 ELSE 0 END) desugaring, whose upstream sibling rule is commented as legacy behaviour being phased out.
  • Information-loss reporting — an IR differ asserting a round trip loses only what OSI's format inherently cannot carry, pinned as exact tuples.
  • Schema conformance — walks every $defs property in osi-schema.json against the Go structs, so a spec property with no Go counterpart fails the build. This exists because two such gaps shipped undetected.

97 tests before, 275 after.

Known limits

Documented in dialect/README.md: OSI has no measure concept, so an unpublished dbt measure returns from a round trip as a published metric; metric expressions containing scalar functions (NULLIF) are outside the six-aggregate grammar and are noted rather than transpiled; unique_keys and custom_extensions have no IR home and are reported.

Design: docs/superpowers/specs/2026-08-12-ossie-dialect-design.md

Checklist

  • This PR is based on main (not stacked on another unmerged branch)
  • If it changes emitted output, I regenerated goldens (UPDATE_GOLDEN=1 go test ./...) and reviewed the diff (don't bless output you haven't read)
  • Docs updated if behavior or a dialect mapping changed (README.md, dialect/README.md)

🤖 Generated with Claude Code

andrebaaij and others added 30 commits August 12, 2026 11:18
Adds the design for `ossie`, semglot's second source dialect and eighth
target, reading and writing the Apache Ossie Core Metadata Specification
(core-spec 0.2.0.dev0).

Records the two decisions taken during design: a plain-aggregation OSI
metric parses into both an ir.Measure and a simple ir.Metric (matching
what dbt.Parse produces), and IR concepts OSI has no slot for degrade to
prose rather than to a vendor custom_extensions block.

Also records the divergences found against Ossie's own reference
converters, which the planned differential tests must tolerate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ossie carries ai_context.synonyms on datasets, and ir.Table has no
synonyms field — only ir.Field and ir.Metric do. Rather than degrade
dataset synonyms to prose on parse, the design now adds Table.Synonyms
to the IR and wires it across every dialect: structurally for ossie,
dbt, snowflake-semantic-view and cortex, and folded into descriptions
via the existing appendClause helper elsewhere.

This is a pre-existing gap rather than an ossie quirk — Snowflake
semantic views already accept table synonyms and svSynonyms already
renders the clause, it is simply never called for a table. Wiring
supersimple also closes the synonyms gap dialect/README.md records.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve tasks: ir.Table.Synonyms across all nine dialects, the shared
SQL-expression parser extraction, ossie Parse and Emit, then the three
test layers (vendored parse conformance, agreement with Apache Ossie's
reference dbt converter, and round-trip information-loss reporting).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ir.Table had no synonyms field while ir.Field and ir.Metric both do,
so an entity's alternative names had nowhere structural to live. dbt
reads and writes them as a model-level meta.synonyms block, mirroring
the existing column-level convention.
Both formats have a table-level synonyms slot semglot was not filling.
svSynonyms already rendered the clause and documented that Snowflake
accepts it on tables; it was simply never called for one.

Verified Cortex Analyst supports table-level synonyms: the official
Snowflake-Labs/semantic-model-generator proto (semantic_model.proto)
defines Table.synonyms (repeated string, field 2, "A list of other
term/phrases used to refer to this table."), so cortex took the
Supported branch rather than deferring to prose-folding in Task 3.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…xists

nao-yaml, nao-context-rules, lightdash, databricks-metric-view and
supersimple have no table-level synonyms slot, so they degrade to prose
via the existing synonymClause/appendClause helpers — the same way they
already degrade column synonyms. Closes the supersimple synonyms gap
dialect/README.md recorded.
Pure move, no behaviour change: derivedParser becomes exprParser with a
pluggable leaf rule, so ossie can reuse the grammar without dbt
inheriting a permissive one. The existing dbt tests are the proof.
parseAggExpr reads OSI metric expressions - aggregate calls over
columns, combined arithmetically - into ir.Agg/Binary/Col trees. An
aggregate argument that is not a plain column becomes ir.Raw, which is
how Ossie's own converter renders a filtered measure.
DISTINCT-stripping and *-as-Arg:nil were applied to any known
aggregate, not just count. SUM(DISTINCT x) silently dropped DISTINCT
and SUM(*) silently accepted invalid SQL. Both are now gated on
fn == "count"; for other aggregates DISTINCT falls through to the
existing Raw fallback (preserving it verbatim) and a bare * fails the
parse outright.
Reads the Apache Ossie core-spec semantic_model layer into the IR:
datasets become tables, fields become dimensions or time dimensions per
the spec's is_time resolution rule, and ai_context.synonyms round-trips
structurally at both table and field level.
Fix round 1: unique_keys, field label, ai_context.examples (model/dataset/
field), and model-level ai_context.synonyms were unmarshalled but never
consumed, vanishing without a trace. Label now folds into the field
description via appendClause; the rest surface as notes on the model, since
the IR has no slot for them. Also drop the unused ossie receiver on
mergeModel, making it a package-level function like the file's other
non-method helpers.
OSI's flat metric list distributes onto the IR's per-table metrics by
homing each expression on its first referenced dataset, the same rule
dbt uses for cross-table derived metrics. A plain aggregation over one
column additionally synthesises an ir.Measure, matching what dbt.Parse
produces for a type: simple metric.

Also notes a relationship whose from_columns/to_columns are empty
rather than dropping it silently, per the project's no-silent-drop
constraint.
setAggTable stamped the metric's single resolved home table onto every
Agg node and every Raw arg's Columns, regardless of which table a given
sub-expression actually belonged to. A cross-table metric like
SUM(orders.amount) / COUNT(CASE WHEN customers.active THEN 1 END) left
the non-home operand's columns unqualified in rendered SQL with no note
- a plausible-looking but wrong IR.

Replace it with resolveAggTables, which re-runs metricHome against each
Agg node's own Arg: a Raw arg's Columns and a bare Col arg's Table are
filled from that node's own resolved table, matching the
always-qualified Col invariant dbt.go maintains. When a node's argument
can't be attributed (unknown or ambiguous column), it's left unqualified
and a note names the metric and the sub-expression, instead of silently
inheriting the wrong table.
Measure columns are emitted as fields because OSI defines fields as the
operands of metric expressions - the same shape Ossie's own dbt
converter produces. Enum values and table grain have no OSI slot and
fold into descriptions.

Also warns (rather than silently omitting datatype) when a field's SQL
type is set but has no OSI portable-vocabulary equivalent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OSI documents field name as unique within a dataset, but Dimensions,
TimeDimensions, and Measures were appended into one fields list with no
dedup and no warning - a silently wrong result when two entries shared
a name. First occurrence now wins (Dimensions, then TimeDimensions,
then Measures); a same-name/same-expression duplicate is dropped
silently (nothing lost), a same-name/different-expression collision is
dropped with a returned warning naming both expressions.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measures and metrics both land in OSI's single flat metrics list; a name
clash resolves in the metric's favour with a warning, matching the
precedent dialect/README.md records for lightdash. Relationship names
use relRoleSuffix so role-playing dimensions stay unique, as OSI
requires.
Parses Apache Ossie's own example and converter fixtures, covering
multi-dialect expressions, composite keys, dataset synonyms and
cross-dataset metrics. Fixtures are copied verbatim at a pinned commit
with their ASF headers retained; see the vendor README for provenance.
TestOssieParsesUpstreamFixtures logged len(m.Relationships) via t.Logf
but never asserted it, and never counted ir.Measures at all. A
regression that silently broke relationship extraction or measure
synthesis across every fixture would still have passed. Pin exact
per-fixture counts, derived from the upstream files and from
simpleAgg's synthesis rule, so a partial regression is caught rather
than just a total-zero case.
Compares semglot's dbt -> ossie output against the OSI documents Ossie's
own converter produces, semantically rather than textually. The accepted
divergences are enumerated in referenceDivergences; a new one appearing
is a regression signal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
summarize compared fields by name alone, so the field-level half of the
COUNT-desugaring divergence went unasserted: their converter declares
field `order_count` as `CASE WHEN order_id IS NOT NULL THEN 1 ELSE 0 END`
where semglot declares it as `order_id`.

Fields are now compared as `name = expression`. The field-level form has
no SUM wrapper, which frees rather than costs an anchor: countDesugarBare
is anchored on the whole string, so it can only rewrite a field whose
entire expression is that exact idiom over a bare identifier. countDesugar
and parenWrapped are unchanged.

Also records that divergence 4's edge-case behaviour cuts both ways —
COUNT(x) is 0 over empty input where their SUM(CASE ...) is NULL, but as a
ratio denominator that reverses, their NULL propagating where COUNT's 0
risks a divide-by-zero — and that the rule originates in dbt Labs'
metricflow_semantic_interfaces as legacy behaviour due for removal.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lossReport diffs two IR models and lists what a round-trip dropped or
gained, asserted against an allowlist of documented format limits.
dbt -> ossie -> dbt loses unpublished measures because OSI has no
measure concept; ossie -> ossie tolerates the same gap (a measure's
field declaration returns as a dimension, since OSI's fields[] has no
marker distinguishing a metric operand from a real one).

Writing the report surfaced two real defects in dialect/ossie.go,
fixed rather than allowlisted: metricHome's ir.Raw case ignored an
explicit `table.column` qualifier and fell back to colOwner's
column-name lookup, which is ambiguous whenever two datasets declare
the same column name, silently dropping the metric; and the
measure-to-metric emit path qualified a compound expression by naive
string concatenation, producing invalid SQL for anything other than a
bare column. Both are fixed in dialect/ossie.go and
dialect/ossie_emit.go.

Documents the ossie mapping and its limits in dialect/README.md, and
reconciles the design doc's divergences list and Cortex table-synonym
verification note with what shipped.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round 1 review findings on the round-trip loss report:

- tableColumns (dialect/ossie_emit.go) omitted measure operand columns
  from the qualification set, so a measure referencing another
  measure's bare-identifier column emitted with that column
  unqualified in OSI's cross-dataset metrics list -- silently wrong
  SQL. Now folds in every measure's Expr when it's a bare identifier,
  mirroring dbt.go's own column-set construction.

- allowedLoss matching moved from strings.Contains over formatted text
  to exact-tuple equality over a structured lossLine (Table, Category,
  Dir, Name). Substring matching let "dimensions: gained" mask "time
  dimensions: gained", and "measures: lost" mask a total loss of every
  measure rather than just the one documented case -- both
  demonstrated by mutation testing. allowedLoss now pins the exact
  tuple each documented format limit produces, so an unexpected
  Category or Name no longer hides behind an unrelated allowed line.
  Reproduced both mutations against the fix: both now fail
  TestDBTToOssieLoss.

- Added the ossie column to the top-level README's feature matrix
  (it's a target now, the table's own caption says so).

- Corrected dialect/README.md's "field label" claim: ir.Field has no
  Label at all, so there is nothing for ossie's emit path to fold --
  OSI's field label only ever flows the other way, into description on
  parse. Same correction applied to the design doc.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…drop

Closes the six dialect-side findings from the final whole-branch review.

resolveAggTables descended only Agg and Binary, so an OSI metric whose
expression is a bare column parsed to an ir.Col with no Table and was
emitted unqualified into OSI's MODEL-scoped metrics list, where it names
nothing in particular — silently wrong, and contradicting the function's
own always-qualified-Col doc comment. A top-level Col is now resolved by
the same rule the nested case uses: stamped when it resolves, left
unqualified WITH a note when it does not.

Datasets were never merged by name, so two files or two semantic_model[]
entries each declaring `orders` produced two same-named ir.Tables:
tableIndex homed every metric on the first, and Emit wrote two datasets:
entries under one name — an invalid OSI document, silently. They now
merge, as dbt.Parse merges models: and semantic_models: by name. Fields
and synonyms union; description and primary key fill in when the first
declaration left them empty. A conflicting redeclaration keeps the first
and notes what was dropped, the same first-wins-plus-warn rule addField
applies in the emit direction. Emit gained the matching guard for two IR
tables publishing one metric name.

Four constructs parsed into the OSI structs and then vanished. Each now
leaves a note naming its level and owner: ai_context.instructions below
the model level (both the object and the bare-string form), metric-level
ai_context.examples, custom_extensions at all five levels the spec allows
(only vendor_name is decoded — the payload is vendor-private and the
design's prose-only degradation stands, so nothing is ever emitted), and
a non-simpleAgg metric's datatype, which has no carrier at all since
ir.Metric has no DataType and only a synthesised measure could hold it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ossie was the only one of eight targets with no checked-in golden for the
repo's own ecommerce fixture — no byte-level regression lock and no
reviewable emitted artifact. TestOssieGolden follows the pattern the other
seven use (emitTarget + UPDATE_GOLDEN=1 + byte compare). The golden was
generated, then read in full and checked for well-formed OSI: one
semantic_model entry, six datasets each with a database.schema.table
source and an explicit dimension.is_time on every field, datatypes only
from the logical enum, field names unique per dataset and metric names
unique across the model, model-scoped metric expressions fully
table-qualified against dataset-scoped field expressions that are
correctly not, and named relationships with the role-playing dimension
disambiguated.

TestOssieParsesUpstreamFixtures logged m.Notes but never asserted them,
although the design's test section claims it asserts no unexpected notes
are produced. Notes are how this branch honours its no-silent-drop
ruling, so they are the conformance signal, not debug output. wantNotes
pins the exact set per fixture, compared sorted so note order stays an
implementation detail while content does not, and catches a regression in
both directions: one that stops noting something and one that starts
noting everything. The t.Logf dump stays — it is how a future reader
re-does the triage.

The relationship and measure counts, the reference-converter divergence
allowlist, and the 25 allowedLoss tuples are all unchanged. Only
TestOssieRoundTrip's comment moved: its per-file parsing is still right,
but not for the stale reason it gave — same-named datasets now merge
explicitly and note every disagreement, so the real reason is that
merging four unrelated upstream documents yields a chimera whose
round-trip says nothing about any one of them.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dbt/Snowflake projects type most numeric columns "number", but irTypes had
no entry for it, so the ecommerce fixture lost datatype on 20 of 32 emitted
fields (19 warnings). "numeric" is already mapped to Decimal, so number's
omission looked accidental. Adding it recovers 26 datatype: lines in the
ossie golden and drops the CLI warning count to 0; osiTypes (the reverse
map) is untouched, so Decimal still spells back to "decimal".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OSI's fields: is a curated list of group/filter-able attributes, not a
complete column index, so a metric's aggregate argument routinely names a
physical column no dataset declares (e.g. fixtureA_ossie.yaml's own
`SUM(o_totalprice)`, commented "fact columns are bare in measures"), and
COUNT(*) names no column at all. Both cases previously made metricHome fail
and the metric was skipped with a note, giving 0/2 metrics on each of Apache
Ossie's own Databricks fixtures.

Add metricHomeOrFact: a qualified reference or a colOwner-resolvable column
still wins outright; only when that fails does it fall back to the dataset
with no incoming relationship — the fact table, per Apache Ossie's own
documented convention and their Databricks converter's own rule. The
fallback fires only after normal resolution is exhausted, never overriding
declared information, and every fallback attribution is recorded in a note
naming the inference, since it is read out of the model's shape rather than
the file.

Threads the same fact-derived home through resolveAggTables (including the
COUNT(*) case, which never stamped Table before) only when the metric's own
top-level decision already required the fallback, so a genuinely
ambiguous/undisclosed sub-expression elsewhere in the same tree keeps its
original left-unqualified-and-noted behaviour untouched.

Updates test/ossie_conformance_test.go's wantMeasures (0/0/0/3 -> 1/1/2/3)
and wantNotes, and adds three allowedLoss entries in test/loss_test.go for
the same already-documented "measure's synthesised field resurfaces as a
dimension" format limit, now also hit by fixtureA/B/tpcds_ossie's newly
homed metrics.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…al address

An OSI dataset's `source` is a fully-qualified physical table reference
that nothing downstream can recover once discarded — unlike dbt's
`model: ref()`, which dbt itself resolves. Parsing threw it away and
every physical-address emitter (ossie, cortex, snowflake-semantic-view,
databricks-metric-view, supersimple) fabricated a replacement from the
profile's database/schema instead, silently relocating every table and
collapsing cross-database models into one database with zero warnings.

ir.Table gains a Source field; each emitter now prefers it when set and
falls back to the existing profile reconstruction when empty (the dbt
path, which has no physical source of its own). cortex's BaseTable needs
Database/Schema/Table split from the single Source string; a source that
doesn't split into exactly three clean parts (a two-part name, or a
query, which the OSI spec permits) falls back to reconstruction AND
warns, rather than half-applying it. mergeTable now notes a Source
conflict the same way it already does for Description/PrimaryKey, and
lossReport gained a (deliberately asymmetric) Source comparison to guard
the new round-trip fidelity.

The ecommerce fixture is dbt-sourced, so Source stays empty throughout
it and no existing golden changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…r-target

Fix round 1 on task 16. Two defects found in review:

1. databricks-metric-view's join Source still built from the joining
   view's own catalog/schema (dbxQualify(catalog, schema, r.Right)),
   ignoring the JOINED table's own declared Source even though
   tableByName was already in scope. A view's own `source:` could show
   the right declared address while `joins[].source:` pointed at a
   fabricated, unrelated location for the same entity — unwarned.
   Extracted dbxSourceFor(t, catalog, schema, addNote) and call it for
   both the view's own Source and every join's Source.

2. The "exactly 3 dot-parts or fall back+warn" gate only needs to apply
   to cortex, whose BaseTable has separate Database/Schema/Table YAML
   fields. snowflake-semantic-view, databricks-metric-view, and
   supersimple each hold their physical reference as one string and can
   take Table.Source verbatim regardless of part count — a two-part
   schema.table resolves fine against the session database / current
   catalog. The uniform gate was rejecting a usable two-part source in
   favour of a same-shaped FABRICATED one (supersimple: warned
   "CRM.customer_master could not be expressed as supersimple's ...
   shape" then substituted the equally two-part PUBLIC.CUSTOMER_MASTER).
   Replaced the 3-part check for those three targets with
   looksLikeQuery, which only rejects a source that reads as a SQL
   query (whitespace, parens, or a leading SELECT) — the one shape the
   OSI spec permits that none of them can hold as an address. cortex is
   unchanged.

Warning wording reworded per target (querySourceWarning vs the
unsplittable one) to say what actually couldn't be expressed.

No existing golden moved; no pinned expectation (referenceDivergences,
allowedLoss, wantRelationships/wantMeasures/wantNotes) needed a change.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dbt_emit was the only emitter with no degradation sites. That was safe while
dbt was both the sole source and a target — dbt -> dbt is lossless by
construction — but ossie is a second source now, and two bare `continue`s in
emitMetrics were dropping metrics with no warning and exit code 0.

(a) Both misses now return a warning naming the metric and why dbt has no form
for it, bringing dbt to parity with snowflake-semantic-view, supersimple and
lightdash. The contract comment on Emit is corrected.

(b) hoistInlineAggs (dialect/agg_names.go) recovers the shape rather than only
reporting it. An ir.Binary over inlined ir.Agg leaves is a legitimate IR shape —
cortex, databricks-metric-view, nao-yaml, nao-context-rules and ossie all render
it correctly — but dbt, snowflake-semantic-view, lightdash and supersimple all
express a derived metric as arithmetic over metrics REFERENCED BY NAME, and an
inlined aggregate has no name. The helper synthesises a metric per leaf (plus,
for dbt alone, the backing measure its metric->measure indirection needs),
rewrites the tree to ir.Ref, and lets each emitter's existing Ref-handling path
run unchanged. Its second job is the mirror image: a COUNT(*) or compound
aggregate that measureFor cannot match gets a backing measure under the metric's
own name, COUNT(*) as dbt's idiomatic `agg: count` over `expr: 1`.

Only those four emitters call it, so no synthesised name reaches the five that
inline correctly. Names are `<column>_<agg>` (or `<metric>_<agg>` when the
aggregate names no column), collision-checked against the table's metric names
and every dimension/measure name AND physical column — Lightdash keys columns[]
by column and resolves a clash in favour of the dimension, so a careless name
would delete the metric being rescued — with a `_2`, `_3`, … fallback. An
aggregate the model already publishes under a name reuses it. Emit stays
read-only over the model: every list is freshly built.

Reach, measured on the vendored Apache Ossie fixtures (metrics reaching dbt
output, before -> after): fixtureA 1/2 -> 2/2, fixtureB 1/2 -> 2/2, tpcds_ossie
2/2 -> 2/2, tpcds_semantic_model 3/4 -> 4/4. snowflake-semantic-view and
supersimple gain TPC-DS's cross-table customer_lifetime_value; lightdash
degrades it loudly, which is its documented same-model restriction, and
supersimple still degrades non-division arithmetic. No golden moved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fix round 1 for task 15.

1. The collision check was per-table, but metric names are MODEL-global for
   three of the four consumers: dbt's metrics: list is one flat project-wide
   block, a semantic view's metrics () clause is one namespace (metricTableOf
   is keyed by bare name and last-wins), and supersimple resolves ratio
   operands through one global map. A second table could therefore mint a name
   the first already publishes — shadowing that metric in supersimple's map
   with no NOTES.md entry and exit 0, duplicating a name in dbt's flat list so
   a numerator became ambiguous, and misqualifying a derived metric in
   Snowflake. One metric-name set, seeded from every table and shared by every
   table's minter, closes all three. Dimension, measure and physical-column
   reservations stay per-table, which is their real scope.

2. renderSQL rendered a nil Agg.Arg as the empty string, so COUNT(*) came out
   as `count()` — not valid SQL in any dialect, shipping to cortex,
   databricks-metric-view, nao-yaml, nao-context-rules, ossie and
   snowflake-semantic-view alike. A nil Arg is COUNT(*) and nothing else
   (sqlexpr.go is its only producer and rejects `*` for any other function;
   dbt.go always sets an Arg), so the star belongs in the renderer. This is in
   render_sql.go, shared by every emitter and outside the helper, so it is
   unaffected by the ring-fence on who may call hoistInlineAggs. A filtered
   nil-Arg aggregate counts 1 per matching row rather than splicing `*` into a
   CASE — a shape no shipped path builds, guarded so it cannot become one.

3. A synthesised component is now emitted BEFORE the metric referencing it,
   and snowflake-semantic-view emits every component ahead of its metrics ()
   clause, so a component homed on a later table no longer trails its
   referent either. Whether Snowflake requires definition-before-reference is
   undocumented; ordering costs nothing and removes the question.

Tests: model-wide collision (the reviewer's orders/lineitem amount_sum repro,
asserting each metric name appears once model-wide) and COUNT(*) parse ->
render -> parse. Both mutation-verified. No golden moved; no pinned
expectation changed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
andrebaaij and others added 11 commits August 12, 2026 18:19
cortex split ir.Table.Source into base_table's separate database/schema/
table keys on dot count alone. The OSI spec permits `source` to be "either
database_name.schema_name.table_name or query", and the canonical query
form has exactly the two dots a three-part address has, so

    source: SELECT * FROM prod.raw.orders WHERE deleted = false

"split" cleanly and emitted database "SELECT * FROM prod", schema raw,
table "orders WHERE deleted = false" — a plausible, wrong, unwarned
address, at exit code 0. The block's own comment already promised the
opposite: that a query falls back to the profile reconstruction WITH a
warning.

splitSource now rejects anything looksLikeQuery accepts, before it splits.
The guard lives there rather than only at cortex's call site so no future
caller can reintroduce it — the doc comment already forbade half-applying
a failed split, and a query is now a failed split. cortex tests the query
shape first so the warning says a query was declared (querySourceWarning,
the wording the three string-valued targets already use) rather than that
a split failed; a two-part name still reports unsplittableSourceWarning.

The existing test passed against the broken emitter because its probe
query had a single dot and so was rejected by the part count before the
query shape was ever considered. Both shapes are pinned now.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every metric view resolves the physical source of every table it joins as
well as its own, and a role-playing dimension joins the same table twice,
so one query-shaped source was reported once per resolution — three
identical warnings for a table that is also a join target.

addNote collapses repeats within a view, so the artifact comment does not
carry the same sentence twice, and Emit collapses identical text across
views, since the returned warnings are per BUILD. Each emitted file still
carries the note once: a reader of one view must see why its join source
was reconstructed. The text names the table and the source it is about, so
a second copy carries nothing the first does not.

Also keeps the notes of a table skipped for having no emittable field.
The view that would have carried them is never written, so they reached
nobody; each names a further source construct that did not survive, which
the one-line skip warning does not restate.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Table.Source — the physical address an ossie dataset declares, and the one
thing in an OSI document nothing downstream can reconstruct — reached only
five of the nine emitters. dbt, lightdash, nao-yaml and nao-context-rules
discarded it with no warning and no comment saying why, so `ossie -> dbt`
and `ossie -> nao-yaml` dropped a declared address at exit code 0. Table
synonyms got a disciplined nine-of-nine sweep (structural in five,
prose-folded in four); this brings Source to the same bar.

  nao-context-rules  carries it verbatim, in the Table reference entry.
                     Prose is the only form that holds BOTH legal OSI
                     shapes — an address and a query — with nothing to
                     choose between. Folded into the entry body, so a
                     table declaring nothing else now gets an entry.
  nao-yaml           carries it as prose in notes:, beside the synonyms,
                     and deliberately NOT in a metric's source.table.
                     That key is not an address slot: nao pins database
                     and schema in its directory scaffolding and
                     source.table selects a table within that pair, so a
                     qualified address there would restate or contradict
                     the scaffolding — the same plausible-but-wrong
                     address the cortex base_table split is guarded
                     against.
  dbt, lightdash     cannot, and now warn. Both emit a dbt schema file: a
                     models: entry annotates a model dbt already builds
                     and resolves through ref(), and Lightdash reads that
                     compiled relation from dbt's manifest. The one
                     adjacent slot, config's database/schema/alias, does
                     not describe an address — it RELOCATES the table dbt
                     builds — so writing one there would rewrite the
                     project rather than record a fact about it.

TestEveryEmitterCarriesOrWarnsAboutDeclaredSource is driven by Names(), so
a dialect added later is covered automatically. For both source shapes it
asserts, per emitter, that the address either reaches the artifact or is
named in a warning that also names the table — and that no emitter pasted
a query into an address slot.

Lightdash also stops emitting hoist-minted metrics undescribed and
unannounced. ldMetric gains description, set for a minted metric only, so
a name nobody authored explains itself and names the metric it was minted
for. A component whose referencing metric degraded anyway is KEPT — it is
a real aggregate, and on a dimension-only table the only metric there — but
reported as an orphan, computed model-wide because the component and the
metric that degraded sit on different tables, which is why it degraded.

dialect/README.md gains the missing "Table source" row across all nine
columns, and its false "no source-table identity in ossie on parse" limit
is replaced by the two that are actually true. Same correction in the
top-level README, with a Physical table source row.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An OSI relationship declares a `name` — the spec's "unique identifier for
the relationship" — and an `ai_context`. semglot carried neither: the
parser read `r.Name` only to build note text and threw it away, and
osiRelationship had no AIContext field at all, so yaml.v3 discarded the
key before any code could even note it dropped. On the TPC-DS fixture
`store_sales_to_date` came back as `store_sales_to_date_dim`, and four
relationships' worth of `synonyms` ("when sale occurred", "who bought")
vanished silently — the phrases an agent uses to pick a join from a
natural-language question.

ir.Relationship gains Name and Synonyms, shaped like Table.Source and
Table.Synonyms. ossie parses both, wires instructionsNote/examplesNote
for the one ai_context level they were never wired to, and emits the
declared name plus ai_context.synonyms.

Every emitter that names a relationship (cortex, snowflake-semantic-view,
databricks-metric-view, ossie) now prefers the declared name through the
shared relationshipNames, and falls back to today's generated name WITH a
warning when it is absent, collides with another relationship's declared
or generated name, or is not valid for the target — relRoleSuffix's
role-playing-dimension disambiguation must not lose to a declared name
that would emit a duplicate. Targets with a join but no name slot (dbt,
lightdash, supersimple) report the loss; the prose targets (nao-yaml,
nao-context-rules) fold name and synonyms into their text, as they
already do for a table's source and synonyms.

lossReport now compares a relationship's declared name and synonyms,
matched by endpoints+columns, flagged only when the BEFORE model declared
one — the same rule the Source comparison uses, since a dbt relationship
is anonymous. No allowedLoss entry was added or relaxed; no golden moved.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two spec properties were lost because nobody enumerated the schema's own
property list against the Go structs: Relationship.ai_context had no
field at all (an unparsed key is silence, not a test failure), and
CustomExtension.data was never decoded. Make that enumeration a test so
it cannot regress.

core-spec/osi-schema.json is vendored from apache/ossie at the pinned
commit 88e0011 alongside the existing fixtures (it carries no licence
header — JSON has no comment syntax and upstream embeds none; the README
now says so, so a reader does not think one was stripped).

TestOssieStructsCoverSchema walks the document root plus every mapped
$def, unions AIContext's oneOf branches, and asserts each declared
property has a yaml tag on the mapped struct, naming the property and the
struct on failure. It checks the reverse too: a tag the schema does not
declare would produce a document the spec's additionalProperties: false
rejects. TestOssieSchemaMappingIsComplete pins the mapping itself, so a
new $def must be mapped or explicitly recorded as a scalar rather than
silently skipped. All eleven struct-backed types pass.

osiCustomExtension gains Data to match. It is decoded and deliberately
never carried into the IR or emitted — custom extensions stay prose-only
degradation — but decoding it keeps the struct a faithful mirror of the
spec, which is the whole point of the check.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Preferring a declared relationship name renamed a metric view's joins
from store_sales_to_customer's old generated alias `customer` — and
silently broke every cross-table measure expression, which renderSQL
qualifies with the IR TABLE name. store_sales.yaml emitted
`count(distinct customer.c_customer_sk)` against a view whose only
relations were `source` and store_sales_to_*, and Databricks rejects the
ENTIRE view for a dangling relation. That the two agreed at all was an
accident: the alias used to be strings.ToLower(r.Right), byte-identical
to the qualifier, so the reconciliation was a no-op nobody wrote down.

dbxJoin.Name is now the single source of truth. dbxJoinAliases reads the
aliases back out of mv.Joins and dbxRewriteQualifiers rewrites every
<table>.<column> qualifier through it — self stripped (the base relation
is aliased `source`), a joined table replaced by its alias, and anything
else reported as unresolvable so the measure is dropped with a note
rather than emitted against a relation that does not exist. Both ends
carry a comment naming the coupling, since it broke for want of one.

dbxStripSourceQualifier goes with it: its claim that only the source
qualifier can appear here was demonstrably false (dbxCrossGrain checks
metric REFERENCES, not column qualifiers), and tokenising via sqlTokens
also closes the string-literal weakness its comment admitted to. A
role-playing dimension — two joins to one table, both suffixed, so a bare
qualifier matched neither even before this branch — is now detected as
ambiguous and degraded instead of emitted dangling.

Also: relationshipNames takes an `emits` predicate so a column-less
relationship no target writes can no longer force a real one onto the
fallback path; the "relationship name" loss line sets Table as its doc
says; nao-context-rules prints a join's identity clause once rather than
per column pair.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… one

relationshipNames warned "emitted as %q instead" even when the fallback it
names IS the declared name. Reachable in the commonest star shape — two facts
joining one dimension, one of them declaring the dimension's own name — so the
warning fired on conforming input while the artifact was correct, which is how
readers learn to ignore warnings.

The EqualFold(generated[i], n) guard that already existed for the collision
COUNT is now a short-circuit for the whole switch: every branch falls back to
generated[i], so when that equals the declared name there is nothing to fall
back to. Any duplication that survives is a property of the fallback generator
and would appear identically had nothing been declared, which is silent today.

Also adds relNotEmittedWarning/relEndpointMissing, the shared wording for a
relationship that reaches a target's artifact NOT AT ALL — deliberately
distinct from relNameWarning's "the join is there, its name is not", which
would tell a reader the join was written when it was not. Used by the emitter
fixes that follow.

And repairs splitSource's duplicated doc phrase.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KDt9y2dppWsK6r14ofrwcY
Three defects in one file, all in how a metric view reconciles the model's
relationships with the ONE table each view is rooted at.

CRITICAL: the joins loop is gated on r.Left == t.Name and is the only place
relationshipNames' warnings and relSynonymsWarning are consulted, so a
relationship whose Left endpoint names no table in the model was iterated by
no view at all: no join, no note, no warning, nothing in any artifact's
comment, and its declared name and synonyms went with it. An OSI document with
`from: ghost_fact` produced ZERO databricks warnings while all eight other
emitters carried or reported it. Emit now sweeps for those before the table
loop, returns them as warnings and threads them into every view's comment (the
new modelNotes seed) so a reader of any artifact sees the join this target
could not place. The emits predicate handed to relationshipNames now requires
the Left table to exist too, for the reason relHasColumns already encodes: a
relationship no artifact writes occupies no join name and must not push a real
one onto the fallback path.

IMPORTANT: dbxRewriteQualifiers treated every `a.b` as a table qualifier, so a
struct or map access inside a measure was dropped with a note —
SUM(store_sales.payload.amount) became "references payload ..., skipped" —
while the SAME view's fields: emitted `expr: payload.amount` happily. It now
resolves a qualifier as a relation only when it names a table of the model;
anything else is a nested column and is written verbatim. A model table this
view does not join (or joins twice) still fails, which is the case the
qualifier rewrite exists for.

MINOR: a join whose target table is not in the model fabricated
analytics.<schema>.<table> with no warning — the "plausible, wrong, unwarned
address" splitSource's own doc comment forbids. The join is still written, but
the reconstruction is now named.

Also fixes dbxJoin.RightTable's now-false "used only internally" comment and
gives the joins loop the missing back-pointer to dbxJoinAliases /
dbxRewriteQualifiers, the coupling that broke once already.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KDt9y2dppWsK6r14ofrwcY
…writes

dbt and lightdash warned over all m.Relationships before any table filtering,
so both reported "its declared name has no slot on a join" for joins their
artifact never contains — telling a reader the join was written when it was
not. supersimple, which warns inside its emitted loop, was the model.

Scoping them exposes the other half: a relationship whose relevant endpoint
names no table in the model is iterated by nothing, so scoping alone would
have turned a misleading warning into a silent drop. Each emitter now reports
that as the whole loss it is (relNotEmittedWarning), naming the declared name
and the synonyms that go with it:

  - dbt: no model to hang the `relationships` test on, or no join columns;
  - lightdash: no model to write meta.joins on;
  - supersimple: no PARENT model to write the relation on. This one was a live
    silent drop of the same class as the databricks one, found by sweeping both
    endpoint directions rather than just the one the finding named — supersimple
    hangs its relation off the right endpoint, so `to: ghost_dim` vanished
    there exactly as `from: ghost_fact` vanished from databricks.

Tests: the orphan sweep now runs every registered emitter over both directions,
and a second test forbids any of the four table-loop targets from reporting the
sub-loss for a join it never wrote. Both mutation-verified against each fix.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KDt9y2dppWsK6r14ofrwcY
This branch ADDED ldMetric.Description (on main ldMetric has only Type/SQL) and
filled it only for the components semglot mints. So a metric nobody authored
explained itself while every source-authored metric went out bare: OSI's
total_sales, described "Total sales revenue across all transactions" with
synonyms [total revenue, gross sales, sales amount], emitted as
`total_sales: {type: sum}` with neither the description nor the synonyms in
that build's 28-item warning list.

While there was no slot this was a documented format limit. With the slot
filled for semglot's own metrics and not the user's, it was indefensible.

The description is now carried on all three routes a metric takes into the
file — the column-level simple metric, the model-level derived metric, and a
raw ir.Measure promoted to one (databricks-metric-view has always emitted a
measure's description as its comment:, so lightdash was the outlier). Synonyms
have no key on a Lightdash metric, so they fold into that same description via
ldMetricText, exactly as a dimension's synonyms already fold into the column
description one screen above.

The minted branch no longer sets met.Description: a minted metric's own
Description IS its synthesizedNote, so the general carry covers it (verified
byte-for-byte in the emitted artifact).

Golden moves by 7 description lines, every one the source's own text; the three
metrics lightdash already degrades gain nothing, and a metric the source did
not describe stays bare.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KDt9y2dppWsK6r14ofrwcY
Treating an unrecognised `a.b` as a nested column access is the right
reading — a struct field must survive as `sum(payload.amount)`, matching
what the same view's fields: already emits — but it is indistinguishable
from a table qualifier the model never declares. ir.Col.Table is assigned
purely syntactically, so nothing downstream can tell the two apart.

Emitting the nested-column reading silently meant a phantom qualifier
shipped as if valid. Databricks cannot resolve it and rejects the entire
view, taking every other measure, dimension and join in the file with it —
so a noted single-measure drop became an unwarned whole-file failure.

Keep the expression, and warn. Matches the posture dbxQualify already
takes when it reconstructs an absent join's address: emit it, but name it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@andrebaaij
andrebaaij merged commit 529a741 into main Aug 13, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant