Repository navigation
feat(ossie): Apache Ossie as a source and target dialect - #19
Merged
Merged
Conversation
Adds the design for `ossie`, semglot's second source dialect and eighth target, reading and writing the Apache Ossie Core Metadata Specification (core-spec 0.2.0.dev0). Records the two decisions taken during design: a plain-aggregation OSI metric parses into both an ir.Measure and a simple ir.Metric (matching what dbt.Parse produces), and IR concepts OSI has no slot for degrade to prose rather than to a vendor custom_extensions block. Also records the divergences found against Ossie's own reference converters, which the planned differential tests must tolerate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ossie carries ai_context.synonyms on datasets, and ir.Table has no synonyms field — only ir.Field and ir.Metric do. Rather than degrade dataset synonyms to prose on parse, the design now adds Table.Synonyms to the IR and wires it across every dialect: structurally for ossie, dbt, snowflake-semantic-view and cortex, and folded into descriptions via the existing appendClause helper elsewhere. This is a pre-existing gap rather than an ossie quirk — Snowflake semantic views already accept table synonyms and svSynonyms already renders the clause, it is simply never called for a table. Wiring supersimple also closes the synonyms gap dialect/README.md records. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve tasks: ir.Table.Synonyms across all nine dialects, the shared SQL-expression parser extraction, ossie Parse and Emit, then the three test layers (vendored parse conformance, agreement with Apache Ossie's reference dbt converter, and round-trip information-loss reporting). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ir.Table had no synonyms field while ir.Field and ir.Metric both do, so an entity's alternative names had nowhere structural to live. dbt reads and writes them as a model-level meta.synonyms block, mirroring the existing column-level convention.
Both formats have a table-level synonyms slot semglot was not filling. svSynonyms already rendered the clause and documented that Snowflake accepts it on tables; it was simply never called for one. Verified Cortex Analyst supports table-level synonyms: the official Snowflake-Labs/semantic-model-generator proto (semantic_model.proto) defines Table.synonyms (repeated string, field 2, "A list of other term/phrases used to refer to this table."), so cortex took the Supported branch rather than deferring to prose-folding in Task 3. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…xists nao-yaml, nao-context-rules, lightdash, databricks-metric-view and supersimple have no table-level synonyms slot, so they degrade to prose via the existing synonymClause/appendClause helpers — the same way they already degrade column synonyms. Closes the supersimple synonyms gap dialect/README.md recorded.
Pure move, no behaviour change: derivedParser becomes exprParser with a pluggable leaf rule, so ossie can reuse the grammar without dbt inheriting a permissive one. The existing dbt tests are the proof.
parseAggExpr reads OSI metric expressions - aggregate calls over columns, combined arithmetically - into ir.Agg/Binary/Col trees. An aggregate argument that is not a plain column becomes ir.Raw, which is how Ossie's own converter renders a filtered measure.
DISTINCT-stripping and *-as-Arg:nil were applied to any known aggregate, not just count. SUM(DISTINCT x) silently dropped DISTINCT and SUM(*) silently accepted invalid SQL. Both are now gated on fn == "count"; for other aggregates DISTINCT falls through to the existing Raw fallback (preserving it verbatim) and a bare * fails the parse outright.
Reads the Apache Ossie core-spec semantic_model layer into the IR: datasets become tables, fields become dimensions or time dimensions per the spec's is_time resolution rule, and ai_context.synonyms round-trips structurally at both table and field level.
Fix round 1: unique_keys, field label, ai_context.examples (model/dataset/ field), and model-level ai_context.synonyms were unmarshalled but never consumed, vanishing without a trace. Label now folds into the field description via appendClause; the rest surface as notes on the model, since the IR has no slot for them. Also drop the unused ossie receiver on mergeModel, making it a package-level function like the file's other non-method helpers.
OSI's flat metric list distributes onto the IR's per-table metrics by homing each expression on its first referenced dataset, the same rule dbt uses for cross-table derived metrics. A plain aggregation over one column additionally synthesises an ir.Measure, matching what dbt.Parse produces for a type: simple metric. Also notes a relationship whose from_columns/to_columns are empty rather than dropping it silently, per the project's no-silent-drop constraint.
setAggTable stamped the metric's single resolved home table onto every Agg node and every Raw arg's Columns, regardless of which table a given sub-expression actually belonged to. A cross-table metric like SUM(orders.amount) / COUNT(CASE WHEN customers.active THEN 1 END) left the non-home operand's columns unqualified in rendered SQL with no note - a plausible-looking but wrong IR. Replace it with resolveAggTables, which re-runs metricHome against each Agg node's own Arg: a Raw arg's Columns and a bare Col arg's Table are filled from that node's own resolved table, matching the always-qualified Col invariant dbt.go maintains. When a node's argument can't be attributed (unknown or ambiguous column), it's left unqualified and a note names the metric and the sub-expression, instead of silently inheriting the wrong table.
Measure columns are emitted as fields because OSI defines fields as the operands of metric expressions - the same shape Ossie's own dbt converter produces. Enum values and table grain have no OSI slot and fold into descriptions. Also warns (rather than silently omitting datatype) when a field's SQL type is set but has no OSI portable-vocabulary equivalent. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OSI documents field name as unique within a dataset, but Dimensions, TimeDimensions, and Measures were appended into one fields list with no dedup and no warning - a silently wrong result when two entries shared a name. First occurrence now wins (Dimensions, then TimeDimensions, then Measures); a same-name/same-expression duplicate is dropped silently (nothing lost), a same-name/different-expression collision is dropped with a returned warning naming both expressions. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measures and metrics both land in OSI's single flat metrics list; a name clash resolves in the metric's favour with a warning, matching the precedent dialect/README.md records for lightdash. Relationship names use relRoleSuffix so role-playing dimensions stay unique, as OSI requires.
Parses Apache Ossie's own example and converter fixtures, covering multi-dialect expressions, composite keys, dataset synonyms and cross-dataset metrics. Fixtures are copied verbatim at a pinned commit with their ASF headers retained; see the vendor README for provenance.
TestOssieParsesUpstreamFixtures logged len(m.Relationships) via t.Logf but never asserted it, and never counted ir.Measures at all. A regression that silently broke relationship extraction or measure synthesis across every fixture would still have passed. Pin exact per-fixture counts, derived from the upstream files and from simpleAgg's synthesis rule, so a partial regression is caught rather than just a total-zero case.
Compares semglot's dbt -> ossie output against the OSI documents Ossie's own converter produces, semantically rather than textually. The accepted divergences are enumerated in referenceDivergences; a new one appearing is a regression signal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
summarize compared fields by name alone, so the field-level half of the COUNT-desugaring divergence went unasserted: their converter declares field `order_count` as `CASE WHEN order_id IS NOT NULL THEN 1 ELSE 0 END` where semglot declares it as `order_id`. Fields are now compared as `name = expression`. The field-level form has no SUM wrapper, which frees rather than costs an anchor: countDesugarBare is anchored on the whole string, so it can only rewrite a field whose entire expression is that exact idiom over a bare identifier. countDesugar and parenWrapped are unchanged. Also records that divergence 4's edge-case behaviour cuts both ways — COUNT(x) is 0 over empty input where their SUM(CASE ...) is NULL, but as a ratio denominator that reverses, their NULL propagating where COUNT's 0 risks a divide-by-zero — and that the rule originates in dbt Labs' metricflow_semantic_interfaces as legacy behaviour due for removal. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
lossReport diffs two IR models and lists what a round-trip dropped or gained, asserted against an allowlist of documented format limits. dbt -> ossie -> dbt loses unpublished measures because OSI has no measure concept; ossie -> ossie tolerates the same gap (a measure's field declaration returns as a dimension, since OSI's fields[] has no marker distinguishing a metric operand from a real one). Writing the report surfaced two real defects in dialect/ossie.go, fixed rather than allowlisted: metricHome's ir.Raw case ignored an explicit `table.column` qualifier and fell back to colOwner's column-name lookup, which is ambiguous whenever two datasets declare the same column name, silently dropping the metric; and the measure-to-metric emit path qualified a compound expression by naive string concatenation, producing invalid SQL for anything other than a bare column. Both are fixed in dialect/ossie.go and dialect/ossie_emit.go. Documents the ossie mapping and its limits in dialect/README.md, and reconciles the design doc's divergences list and Cortex table-synonym verification note with what shipped. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Round 1 review findings on the round-trip loss report: - tableColumns (dialect/ossie_emit.go) omitted measure operand columns from the qualification set, so a measure referencing another measure's bare-identifier column emitted with that column unqualified in OSI's cross-dataset metrics list -- silently wrong SQL. Now folds in every measure's Expr when it's a bare identifier, mirroring dbt.go's own column-set construction. - allowedLoss matching moved from strings.Contains over formatted text to exact-tuple equality over a structured lossLine (Table, Category, Dir, Name). Substring matching let "dimensions: gained" mask "time dimensions: gained", and "measures: lost" mask a total loss of every measure rather than just the one documented case -- both demonstrated by mutation testing. allowedLoss now pins the exact tuple each documented format limit produces, so an unexpected Category or Name no longer hides behind an unrelated allowed line. Reproduced both mutations against the fix: both now fail TestDBTToOssieLoss. - Added the ossie column to the top-level README's feature matrix (it's a target now, the table's own caption says so). - Corrected dialect/README.md's "field label" claim: ir.Field has no Label at all, so there is nothing for ossie's emit path to fold -- OSI's field label only ever flows the other way, into description on parse. Same correction applied to the design doc. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…drop Closes the six dialect-side findings from the final whole-branch review. resolveAggTables descended only Agg and Binary, so an OSI metric whose expression is a bare column parsed to an ir.Col with no Table and was emitted unqualified into OSI's MODEL-scoped metrics list, where it names nothing in particular — silently wrong, and contradicting the function's own always-qualified-Col doc comment. A top-level Col is now resolved by the same rule the nested case uses: stamped when it resolves, left unqualified WITH a note when it does not. Datasets were never merged by name, so two files or two semantic_model[] entries each declaring `orders` produced two same-named ir.Tables: tableIndex homed every metric on the first, and Emit wrote two datasets: entries under one name — an invalid OSI document, silently. They now merge, as dbt.Parse merges models: and semantic_models: by name. Fields and synonyms union; description and primary key fill in when the first declaration left them empty. A conflicting redeclaration keeps the first and notes what was dropped, the same first-wins-plus-warn rule addField applies in the emit direction. Emit gained the matching guard for two IR tables publishing one metric name. Four constructs parsed into the OSI structs and then vanished. Each now leaves a note naming its level and owner: ai_context.instructions below the model level (both the object and the bare-string form), metric-level ai_context.examples, custom_extensions at all five levels the spec allows (only vendor_name is decoded — the payload is vendor-private and the design's prose-only degradation stands, so nothing is ever emitted), and a non-simpleAgg metric's datatype, which has no carrier at all since ir.Metric has no DataType and only a synthesised measure could hold it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ossie was the only one of eight targets with no checked-in golden for the repo's own ecommerce fixture — no byte-level regression lock and no reviewable emitted artifact. TestOssieGolden follows the pattern the other seven use (emitTarget + UPDATE_GOLDEN=1 + byte compare). The golden was generated, then read in full and checked for well-formed OSI: one semantic_model entry, six datasets each with a database.schema.table source and an explicit dimension.is_time on every field, datatypes only from the logical enum, field names unique per dataset and metric names unique across the model, model-scoped metric expressions fully table-qualified against dataset-scoped field expressions that are correctly not, and named relationships with the role-playing dimension disambiguated. TestOssieParsesUpstreamFixtures logged m.Notes but never asserted them, although the design's test section claims it asserts no unexpected notes are produced. Notes are how this branch honours its no-silent-drop ruling, so they are the conformance signal, not debug output. wantNotes pins the exact set per fixture, compared sorted so note order stays an implementation detail while content does not, and catches a regression in both directions: one that stops noting something and one that starts noting everything. The t.Logf dump stays — it is how a future reader re-does the triage. The relationship and measure counts, the reference-converter divergence allowlist, and the 25 allowedLoss tuples are all unchanged. Only TestOssieRoundTrip's comment moved: its per-file parsing is still right, but not for the stale reason it gave — same-named datasets now merge explicitly and note every disagreement, so the real reason is that merging four unrelated upstream documents yields a chimera whose round-trip says nothing about any one of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dbt/Snowflake projects type most numeric columns "number", but irTypes had no entry for it, so the ecommerce fixture lost datatype on 20 of 32 emitted fields (19 warnings). "numeric" is already mapped to Decimal, so number's omission looked accidental. Adding it recovers 26 datatype: lines in the ossie golden and drops the CLI warning count to 0; osiTypes (the reverse map) is untouched, so Decimal still spells back to "decimal". Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OSI's fields: is a curated list of group/filter-able attributes, not a complete column index, so a metric's aggregate argument routinely names a physical column no dataset declares (e.g. fixtureA_ossie.yaml's own `SUM(o_totalprice)`, commented "fact columns are bare in measures"), and COUNT(*) names no column at all. Both cases previously made metricHome fail and the metric was skipped with a note, giving 0/2 metrics on each of Apache Ossie's own Databricks fixtures. Add metricHomeOrFact: a qualified reference or a colOwner-resolvable column still wins outright; only when that fails does it fall back to the dataset with no incoming relationship — the fact table, per Apache Ossie's own documented convention and their Databricks converter's own rule. The fallback fires only after normal resolution is exhausted, never overriding declared information, and every fallback attribution is recorded in a note naming the inference, since it is read out of the model's shape rather than the file. Threads the same fact-derived home through resolveAggTables (including the COUNT(*) case, which never stamped Table before) only when the metric's own top-level decision already required the fallback, so a genuinely ambiguous/undisclosed sub-expression elsewhere in the same tree keeps its original left-unqualified-and-noted behaviour untouched. Updates test/ossie_conformance_test.go's wantMeasures (0/0/0/3 -> 1/1/2/3) and wantNotes, and adds three allowedLoss entries in test/loss_test.go for the same already-documented "measure's synthesised field resurfaces as a dimension" format limit, now also hit by fixtureA/B/tpcds_ossie's newly homed metrics. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…al address An OSI dataset's `source` is a fully-qualified physical table reference that nothing downstream can recover once discarded — unlike dbt's `model: ref()`, which dbt itself resolves. Parsing threw it away and every physical-address emitter (ossie, cortex, snowflake-semantic-view, databricks-metric-view, supersimple) fabricated a replacement from the profile's database/schema instead, silently relocating every table and collapsing cross-database models into one database with zero warnings. ir.Table gains a Source field; each emitter now prefers it when set and falls back to the existing profile reconstruction when empty (the dbt path, which has no physical source of its own). cortex's BaseTable needs Database/Schema/Table split from the single Source string; a source that doesn't split into exactly three clean parts (a two-part name, or a query, which the OSI spec permits) falls back to reconstruction AND warns, rather than half-applying it. mergeTable now notes a Source conflict the same way it already does for Description/PrimaryKey, and lossReport gained a (deliberately asymmetric) Source comparison to guard the new round-trip fidelity. The ecommerce fixture is dbt-sourced, so Source stays empty throughout it and no existing golden changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…r-target Fix round 1 on task 16. Two defects found in review: 1. databricks-metric-view's join Source still built from the joining view's own catalog/schema (dbxQualify(catalog, schema, r.Right)), ignoring the JOINED table's own declared Source even though tableByName was already in scope. A view's own `source:` could show the right declared address while `joins[].source:` pointed at a fabricated, unrelated location for the same entity — unwarned. Extracted dbxSourceFor(t, catalog, schema, addNote) and call it for both the view's own Source and every join's Source. 2. The "exactly 3 dot-parts or fall back+warn" gate only needs to apply to cortex, whose BaseTable has separate Database/Schema/Table YAML fields. snowflake-semantic-view, databricks-metric-view, and supersimple each hold their physical reference as one string and can take Table.Source verbatim regardless of part count — a two-part schema.table resolves fine against the session database / current catalog. The uniform gate was rejecting a usable two-part source in favour of a same-shaped FABRICATED one (supersimple: warned "CRM.customer_master could not be expressed as supersimple's ... shape" then substituted the equally two-part PUBLIC.CUSTOMER_MASTER). Replaced the 3-part check for those three targets with looksLikeQuery, which only rejects a source that reads as a SQL query (whitespace, parens, or a leading SELECT) — the one shape the OSI spec permits that none of them can hold as an address. cortex is unchanged. Warning wording reworded per target (querySourceWarning vs the unsplittable one) to say what actually couldn't be expressed. No existing golden moved; no pinned expectation (referenceDivergences, allowedLoss, wantRelationships/wantMeasures/wantNotes) needed a change. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dbt_emit was the only emitter with no degradation sites. That was safe while dbt was both the sole source and a target — dbt -> dbt is lossless by construction — but ossie is a second source now, and two bare `continue`s in emitMetrics were dropping metrics with no warning and exit code 0. (a) Both misses now return a warning naming the metric and why dbt has no form for it, bringing dbt to parity with snowflake-semantic-view, supersimple and lightdash. The contract comment on Emit is corrected. (b) hoistInlineAggs (dialect/agg_names.go) recovers the shape rather than only reporting it. An ir.Binary over inlined ir.Agg leaves is a legitimate IR shape — cortex, databricks-metric-view, nao-yaml, nao-context-rules and ossie all render it correctly — but dbt, snowflake-semantic-view, lightdash and supersimple all express a derived metric as arithmetic over metrics REFERENCED BY NAME, and an inlined aggregate has no name. The helper synthesises a metric per leaf (plus, for dbt alone, the backing measure its metric->measure indirection needs), rewrites the tree to ir.Ref, and lets each emitter's existing Ref-handling path run unchanged. Its second job is the mirror image: a COUNT(*) or compound aggregate that measureFor cannot match gets a backing measure under the metric's own name, COUNT(*) as dbt's idiomatic `agg: count` over `expr: 1`. Only those four emitters call it, so no synthesised name reaches the five that inline correctly. Names are `<column>_<agg>` (or `<metric>_<agg>` when the aggregate names no column), collision-checked against the table's metric names and every dimension/measure name AND physical column — Lightdash keys columns[] by column and resolves a clash in favour of the dimension, so a careless name would delete the metric being rescued — with a `_2`, `_3`, … fallback. An aggregate the model already publishes under a name reuses it. Emit stays read-only over the model: every list is freshly built. Reach, measured on the vendored Apache Ossie fixtures (metrics reaching dbt output, before -> after): fixtureA 1/2 -> 2/2, fixtureB 1/2 -> 2/2, tpcds_ossie 2/2 -> 2/2, tpcds_semantic_model 3/4 -> 4/4. snowflake-semantic-view and supersimple gain TPC-DS's cross-table customer_lifetime_value; lightdash degrades it loudly, which is its documented same-model restriction, and supersimple still degrades non-division arithmetic. No golden moved. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Fix round 1 for task 15. 1. The collision check was per-table, but metric names are MODEL-global for three of the four consumers: dbt's metrics: list is one flat project-wide block, a semantic view's metrics () clause is one namespace (metricTableOf is keyed by bare name and last-wins), and supersimple resolves ratio operands through one global map. A second table could therefore mint a name the first already publishes — shadowing that metric in supersimple's map with no NOTES.md entry and exit 0, duplicating a name in dbt's flat list so a numerator became ambiguous, and misqualifying a derived metric in Snowflake. One metric-name set, seeded from every table and shared by every table's minter, closes all three. Dimension, measure and physical-column reservations stay per-table, which is their real scope. 2. renderSQL rendered a nil Agg.Arg as the empty string, so COUNT(*) came out as `count()` — not valid SQL in any dialect, shipping to cortex, databricks-metric-view, nao-yaml, nao-context-rules, ossie and snowflake-semantic-view alike. A nil Arg is COUNT(*) and nothing else (sqlexpr.go is its only producer and rejects `*` for any other function; dbt.go always sets an Arg), so the star belongs in the renderer. This is in render_sql.go, shared by every emitter and outside the helper, so it is unaffected by the ring-fence on who may call hoistInlineAggs. A filtered nil-Arg aggregate counts 1 per matching row rather than splicing `*` into a CASE — a shape no shipped path builds, guarded so it cannot become one. 3. A synthesised component is now emitted BEFORE the metric referencing it, and snowflake-semantic-view emits every component ahead of its metrics () clause, so a component homed on a later table no longer trails its referent either. Whether Snowflake requires definition-before-reference is undocumented; ordering costs nothing and removes the question. Tests: model-wide collision (the reviewer's orders/lineitem amount_sum repro, asserting each metric name appears once model-wide) and COUNT(*) parse -> render -> parse. Both mutation-verified. No golden moved; no pinned expectation changed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
cortex split ir.Table.Source into base_table's separate database/schema/
table keys on dot count alone. The OSI spec permits `source` to be "either
database_name.schema_name.table_name or query", and the canonical query
form has exactly the two dots a three-part address has, so
source: SELECT * FROM prod.raw.orders WHERE deleted = false
"split" cleanly and emitted database "SELECT * FROM prod", schema raw,
table "orders WHERE deleted = false" — a plausible, wrong, unwarned
address, at exit code 0. The block's own comment already promised the
opposite: that a query falls back to the profile reconstruction WITH a
warning.
splitSource now rejects anything looksLikeQuery accepts, before it splits.
The guard lives there rather than only at cortex's call site so no future
caller can reintroduce it — the doc comment already forbade half-applying
a failed split, and a query is now a failed split. cortex tests the query
shape first so the warning says a query was declared (querySourceWarning,
the wording the three string-valued targets already use) rather than that
a split failed; a two-part name still reports unsplittableSourceWarning.
The existing test passed against the broken emitter because its probe
query had a single dot and so was rejected by the part count before the
query shape was ever considered. Both shapes are pinned now.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every metric view resolves the physical source of every table it joins as well as its own, and a role-playing dimension joins the same table twice, so one query-shaped source was reported once per resolution — three identical warnings for a table that is also a join target. addNote collapses repeats within a view, so the artifact comment does not carry the same sentence twice, and Emit collapses identical text across views, since the returned warnings are per BUILD. Each emitted file still carries the note once: a reader of one view must see why its join source was reconstructed. The text names the table and the source it is about, so a second copy carries nothing the first does not. Also keeps the notes of a table skipped for having no emittable field. The view that would have carried them is never written, so they reached nobody; each names a further source construct that did not survive, which the one-line skip warning does not restate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Table.Source — the physical address an ossie dataset declares, and the one
thing in an OSI document nothing downstream can reconstruct — reached only
five of the nine emitters. dbt, lightdash, nao-yaml and nao-context-rules
discarded it with no warning and no comment saying why, so `ossie -> dbt`
and `ossie -> nao-yaml` dropped a declared address at exit code 0. Table
synonyms got a disciplined nine-of-nine sweep (structural in five,
prose-folded in four); this brings Source to the same bar.
nao-context-rules carries it verbatim, in the Table reference entry.
Prose is the only form that holds BOTH legal OSI
shapes — an address and a query — with nothing to
choose between. Folded into the entry body, so a
table declaring nothing else now gets an entry.
nao-yaml carries it as prose in notes:, beside the synonyms,
and deliberately NOT in a metric's source.table.
That key is not an address slot: nao pins database
and schema in its directory scaffolding and
source.table selects a table within that pair, so a
qualified address there would restate or contradict
the scaffolding — the same plausible-but-wrong
address the cortex base_table split is guarded
against.
dbt, lightdash cannot, and now warn. Both emit a dbt schema file: a
models: entry annotates a model dbt already builds
and resolves through ref(), and Lightdash reads that
compiled relation from dbt's manifest. The one
adjacent slot, config's database/schema/alias, does
not describe an address — it RELOCATES the table dbt
builds — so writing one there would rewrite the
project rather than record a fact about it.
TestEveryEmitterCarriesOrWarnsAboutDeclaredSource is driven by Names(), so
a dialect added later is covered automatically. For both source shapes it
asserts, per emitter, that the address either reaches the artifact or is
named in a warning that also names the table — and that no emitter pasted
a query into an address slot.
Lightdash also stops emitting hoist-minted metrics undescribed and
unannounced. ldMetric gains description, set for a minted metric only, so
a name nobody authored explains itself and names the metric it was minted
for. A component whose referencing metric degraded anyway is KEPT — it is
a real aggregate, and on a dimension-only table the only metric there — but
reported as an orphan, computed model-wide because the component and the
metric that degraded sit on different tables, which is why it degraded.
dialect/README.md gains the missing "Table source" row across all nine
columns, and its false "no source-table identity in ossie on parse" limit
is replaced by the two that are actually true. Same correction in the
top-level README, with a Physical table source row.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An OSI relationship declares a `name` — the spec's "unique identifier for
the relationship" — and an `ai_context`. semglot carried neither: the
parser read `r.Name` only to build note text and threw it away, and
osiRelationship had no AIContext field at all, so yaml.v3 discarded the
key before any code could even note it dropped. On the TPC-DS fixture
`store_sales_to_date` came back as `store_sales_to_date_dim`, and four
relationships' worth of `synonyms` ("when sale occurred", "who bought")
vanished silently — the phrases an agent uses to pick a join from a
natural-language question.
ir.Relationship gains Name and Synonyms, shaped like Table.Source and
Table.Synonyms. ossie parses both, wires instructionsNote/examplesNote
for the one ai_context level they were never wired to, and emits the
declared name plus ai_context.synonyms.
Every emitter that names a relationship (cortex, snowflake-semantic-view,
databricks-metric-view, ossie) now prefers the declared name through the
shared relationshipNames, and falls back to today's generated name WITH a
warning when it is absent, collides with another relationship's declared
or generated name, or is not valid for the target — relRoleSuffix's
role-playing-dimension disambiguation must not lose to a declared name
that would emit a duplicate. Targets with a join but no name slot (dbt,
lightdash, supersimple) report the loss; the prose targets (nao-yaml,
nao-context-rules) fold name and synonyms into their text, as they
already do for a table's source and synonyms.
lossReport now compares a relationship's declared name and synonyms,
matched by endpoints+columns, flagged only when the BEFORE model declared
one — the same rule the Source comparison uses, since a dbt relationship
is anonymous. No allowedLoss entry was added or relaxed; no golden moved.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two spec properties were lost because nobody enumerated the schema's own property list against the Go structs: Relationship.ai_context had no field at all (an unparsed key is silence, not a test failure), and CustomExtension.data was never decoded. Make that enumeration a test so it cannot regress. core-spec/osi-schema.json is vendored from apache/ossie at the pinned commit 88e0011 alongside the existing fixtures (it carries no licence header — JSON has no comment syntax and upstream embeds none; the README now says so, so a reader does not think one was stripped). TestOssieStructsCoverSchema walks the document root plus every mapped $def, unions AIContext's oneOf branches, and asserts each declared property has a yaml tag on the mapped struct, naming the property and the struct on failure. It checks the reverse too: a tag the schema does not declare would produce a document the spec's additionalProperties: false rejects. TestOssieSchemaMappingIsComplete pins the mapping itself, so a new $def must be mapped or explicitly recorded as a scalar rather than silently skipped. All eleven struct-backed types pass. osiCustomExtension gains Data to match. It is decoded and deliberately never carried into the IR or emitted — custom extensions stay prose-only degradation — but decoding it keeps the struct a faithful mirror of the spec, which is the whole point of the check. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Preferring a declared relationship name renamed a metric view's joins from store_sales_to_customer's old generated alias `customer` — and silently broke every cross-table measure expression, which renderSQL qualifies with the IR TABLE name. store_sales.yaml emitted `count(distinct customer.c_customer_sk)` against a view whose only relations were `source` and store_sales_to_*, and Databricks rejects the ENTIRE view for a dangling relation. That the two agreed at all was an accident: the alias used to be strings.ToLower(r.Right), byte-identical to the qualifier, so the reconciliation was a no-op nobody wrote down. dbxJoin.Name is now the single source of truth. dbxJoinAliases reads the aliases back out of mv.Joins and dbxRewriteQualifiers rewrites every <table>.<column> qualifier through it — self stripped (the base relation is aliased `source`), a joined table replaced by its alias, and anything else reported as unresolvable so the measure is dropped with a note rather than emitted against a relation that does not exist. Both ends carry a comment naming the coupling, since it broke for want of one. dbxStripSourceQualifier goes with it: its claim that only the source qualifier can appear here was demonstrably false (dbxCrossGrain checks metric REFERENCES, not column qualifiers), and tokenising via sqlTokens also closes the string-literal weakness its comment admitted to. A role-playing dimension — two joins to one table, both suffixed, so a bare qualifier matched neither even before this branch — is now detected as ambiguous and degraded instead of emitted dangling. Also: relationshipNames takes an `emits` predicate so a column-less relationship no target writes can no longer force a real one onto the fallback path; the "relationship name" loss line sets Table as its doc says; nao-context-rules prints a join's identity clause once rather than per column pair. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… one relationshipNames warned "emitted as %q instead" even when the fallback it names IS the declared name. Reachable in the commonest star shape — two facts joining one dimension, one of them declaring the dimension's own name — so the warning fired on conforming input while the artifact was correct, which is how readers learn to ignore warnings. The EqualFold(generated[i], n) guard that already existed for the collision COUNT is now a short-circuit for the whole switch: every branch falls back to generated[i], so when that equals the declared name there is nothing to fall back to. Any duplication that survives is a property of the fallback generator and would appear identically had nothing been declared, which is silent today. Also adds relNotEmittedWarning/relEndpointMissing, the shared wording for a relationship that reaches a target's artifact NOT AT ALL — deliberately distinct from relNameWarning's "the join is there, its name is not", which would tell a reader the join was written when it was not. Used by the emitter fixes that follow. And repairs splitSource's duplicated doc phrase. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDt9y2dppWsK6r14ofrwcY
Three defects in one file, all in how a metric view reconciles the model's relationships with the ONE table each view is rooted at. CRITICAL: the joins loop is gated on r.Left == t.Name and is the only place relationshipNames' warnings and relSynonymsWarning are consulted, so a relationship whose Left endpoint names no table in the model was iterated by no view at all: no join, no note, no warning, nothing in any artifact's comment, and its declared name and synonyms went with it. An OSI document with `from: ghost_fact` produced ZERO databricks warnings while all eight other emitters carried or reported it. Emit now sweeps for those before the table loop, returns them as warnings and threads them into every view's comment (the new modelNotes seed) so a reader of any artifact sees the join this target could not place. The emits predicate handed to relationshipNames now requires the Left table to exist too, for the reason relHasColumns already encodes: a relationship no artifact writes occupies no join name and must not push a real one onto the fallback path. IMPORTANT: dbxRewriteQualifiers treated every `a.b` as a table qualifier, so a struct or map access inside a measure was dropped with a note — SUM(store_sales.payload.amount) became "references payload ..., skipped" — while the SAME view's fields: emitted `expr: payload.amount` happily. It now resolves a qualifier as a relation only when it names a table of the model; anything else is a nested column and is written verbatim. A model table this view does not join (or joins twice) still fails, which is the case the qualifier rewrite exists for. MINOR: a join whose target table is not in the model fabricated analytics.<schema>.<table> with no warning — the "plausible, wrong, unwarned address" splitSource's own doc comment forbids. The join is still written, but the reconstruction is now named. Also fixes dbxJoin.RightTable's now-false "used only internally" comment and gives the joins loop the missing back-pointer to dbxJoinAliases / dbxRewriteQualifiers, the coupling that broke once already. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01KDt9y2dppWsK6r14ofrwcY
…writes
dbt and lightdash warned over all m.Relationships before any table filtering,
so both reported "its declared name has no slot on a join" for joins their
artifact never contains — telling a reader the join was written when it was
not. supersimple, which warns inside its emitted loop, was the model.
Scoping them exposes the other half: a relationship whose relevant endpoint
names no table in the model is iterated by nothing, so scoping alone would
have turned a misleading warning into a silent drop. Each emitter now reports
that as the whole loss it is (relNotEmittedWarning), naming the declared name
and the synonyms that go with it:
- dbt: no model to hang the `relationships` test on, or no join columns;
- lightdash: no model to write meta.joins on;
- supersimple: no PARENT model to write the relation on. This one was a live
silent drop of the same class as the databricks one, found by sweeping both
endpoint directions rather than just the one the finding named — supersimple
hangs its relation off the right endpoint, so `to: ghost_dim` vanished
there exactly as `from: ghost_fact` vanished from databricks.
Tests: the orphan sweep now runs every registered emitter over both directions,
and a second test forbids any of the four table-loop targets from reporting the
sub-loss for a join it never wrote. Both mutation-verified against each fix.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KDt9y2dppWsK6r14ofrwcY
This branch ADDED ldMetric.Description (on main ldMetric has only Type/SQL) and
filled it only for the components semglot mints. So a metric nobody authored
explained itself while every source-authored metric went out bare: OSI's
total_sales, described "Total sales revenue across all transactions" with
synonyms [total revenue, gross sales, sales amount], emitted as
`total_sales: {type: sum}` with neither the description nor the synonyms in
that build's 28-item warning list.
While there was no slot this was a documented format limit. With the slot
filled for semglot's own metrics and not the user's, it was indefensible.
The description is now carried on all three routes a metric takes into the
file — the column-level simple metric, the model-level derived metric, and a
raw ir.Measure promoted to one (databricks-metric-view has always emitted a
measure's description as its comment:, so lightdash was the outlier). Synonyms
have no key on a Lightdash metric, so they fold into that same description via
ldMetricText, exactly as a dimension's synonyms already fold into the column
description one screen above.
The minted branch no longer sets met.Description: a minted metric's own
Description IS its synthesizedNote, so the general carry covers it (verified
byte-for-byte in the emitted artifact).
Golden moves by 7 description lines, every one the source's own text; the three
metrics lightdash already degrades gain nothing, and a metric the source did
not describe stays bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KDt9y2dppWsK6r14ofrwcY
Treating an unrecognised `a.b` as a nested column access is the right reading — a struct field must survive as `sum(payload.amount)`, matching what the same view's fields: already emits — but it is indistinguishable from a table qualifier the model never declares. ir.Col.Table is assigned purely syntactically, so nothing downstream can tell the two apart. Emitting the nested-column reading silently meant a phantom qualifier shipped as if valid. Databricks cannot resolve it and rejects the entire view, taking every other measure, dimension and join in the file with it — so a noted single-measure drop became an unwarned whole-file failure. Keep the expression, and warn. Matches the posture dbxQualify already takes when it reconstructs an absent join's address: emit it, but name it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What and why
Adds
ossie— a source and target dialect for the Apache Ossie Core Metadata Specification (core-spec0.2.0.dev0), the vendor-neutral semantic-model interchange format.Because everything routes through the IR, this makes every existing emit-only dialect reachable from an OSI file, and makes semglot's IR reachable from any tool that exports OSI. It is also semglot's second source dialect, which turned out to be the interesting part: a lot of code quietly assumed dbt was the only thing that could produce IR.
Also in here
Four IR fields, each added because Ossie carries something the IR had no slot for, and each wired across all nine dialects — structurally where a target has a slot, folded into prose or warned where it doesn't:
Table.Synonyms— Ossie datasets carry them; so do Snowflake semantic views and Cortex, which semglot simply wasn't filling. Closes thesupersimplesynonyms gapdialect/README.mdrecorded.Table.Source— without it,ossie → ossiesilently relocated every table:PROD.SALES.orders_v1came back asWAREHOUSE.PUBLIC.orders.Relationship.NameandRelationship.Synonyms— the declared join name was regenerated from<from>_to_<to>, and relationshipai_contextwasn't parsed at all, so join hints like[who bought, what was sold]vanished before any code saw them.Plus a shared SQL-expression parser extracted from
dbt.go, and a shared emit-side helper that hoists inlined aggregates sodbt,snowflake-semantic-view,lightdashandsupersimplecan expressSUM(a)/SUM(b)through their existing reference-handling paths.dbt_emitalso learned to warn. It was the only emitter in the codebase with zero degradation sites — safe while dbt was both the sole source and a target, wrong the moment a second source existed.Testing
Four layers, three built on Apache Ossie's own material (vendored at pinned commit
88e0011, ASF headers retained, provenance recorded):dbt → ossieoutput compared semantically against what Ossie's own dbt converter produces. Exactly one root divergence survives: their MetricFlow-inheritedCOUNT(x)→SUM(CASE WHEN x IS NOT NULL THEN 1 ELSE 0 END)desugaring, whose upstream sibling rule is commented as legacy behaviour being phased out.$defsproperty inosi-schema.jsonagainst the Go structs, so a spec property with no Go counterpart fails the build. This exists because two such gaps shipped undetected.97 tests before, 275 after.
Known limits
Documented in
dialect/README.md: OSI has no measure concept, so an unpublished dbt measure returns from a round trip as a published metric; metric expressions containing scalar functions (NULLIF) are outside the six-aggregate grammar and are noted rather than transpiled;unique_keysandcustom_extensionshave no IR home and are reported.Design:
docs/superpowers/specs/2026-08-12-ossie-dialect-design.mdChecklist
main(not stacked on another unmerged branch)UPDATE_GOLDEN=1 go test ./...) and reviewed the diff (don't bless output you haven't read)README.md,dialect/README.md)🤖 Generated with Claude Code