Skip to content

feat(lightdash): Lightdash emitter, merged with main and collision-safe - #17

Merged
andrebaaij merged 15 commits into
mainfrom
fix/lightdash-metric-dimension-collision
Aug 12, 2026
Merged

andrebaaij merged 15 commits into
mainfrom
fix/lightdash-metric-dimension-collision

Conversation

@andrebaaij

Copy link
Copy Markdown
Contributor

Brings the lightdash target onto main, and fixes a bug that made it silently
lose metrics when deployed.

The bug

Deploying semglot's generated schema.yml to Lightdash Cloud produced:

- obt_marketing_daily: Skipped metric "roas" because a dimension with the same
  name exists. Dimensions take priority.
- obt_marketing_daily: Skipped metric "attributed_revenue" because a dimension
  with the same name exists. Dimensions take priority.

The emitter emitted a metric whose name matched a dimension it also emitted on
the same model. Lightdash resolves the clash in favour of the dimension and
silently drops the metric, so the semantic layer quietly loses the metric it
exists to publish. Downstream this does not look like a semglot bug, it looks
like the analytics agent failing to find a metric.

attributed_revenue was the emitter manufacturing its own collision: it is not
a dimension in the IR at all, the column entry existed only because the emitter
hung a column-level metric on it, and Lightdash turns any columns[] entry
into a dimension.

The fix

The computed metric is canonical; the colliding column is not emitted as a
dimension. Metrics it hosted are re-homed at model level with an explicit
sql: ${TABLE}.<col>. This matches what the sibling emitters already do —
snowflake_semantic_view.go has a TestSVDedupsDimensionMetricNameCollision
using roas on obt_marketing_daily as its example, and
databricks_metric_view.go seeds seen with measure names so the field is the
one dropped. Keeping semglot's answer to "roas" identical across targets is the
point of routing everything through one IR.

Exception: a column named by meta.primary_key or by a join's sql_on is
resolved structurally and Lightdash rejects the whole explore if it is missing,
so there the dimension stays and the metric degrades to a note. Both branches
emit a note, so nothing is silent either way.

Rejected alternatives: renaming the metric (the name is the downstream
contract), and meta.dimension.hidden: true (a hidden dimension still occupies
the name, so the metric is still skipped).

Merged with main

The branch was 25 commits behind and predated the Emit signature change that
threads degrade warnings back to the CLI. lightdash.Emit now returns
([]string, error); the notes are still also written into schema.yml as a
leading comment block, since that file is committed and deployed and its reader
should not have to go back to the build log.

README conflicted because both sides added a dialect — kept both, and spliced
the lightdash column into main's capability matrix.

Verified against a live workspace

Not just unit tests. Regenerated over the real 38-model marts source and
redeployed to Lightdash Cloud:

before after
explores compiled 37 SUCCESS + 1 PARTIAL 38 SUCCESS
skipped-metric warnings 2 0

That also settles the one thing that could not be verified from tests alone:
Lightdash accepts the re-homed model-level metric form.

go test ./... green, go vet clean, gofmt clean. Golden diff is additions
only.

andrebaaij and others added 15 commits July 20, 2026 08:05
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Refine the spec: derived metrics require refs to emitted same-table simple
metrics (no dangling ${...}); drop time_intervals (no IR interval list);
count(*) degrades.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The design section still described the older flag/identity config; the
implementation (and plan) use a semglot.yaml profile field. Match the doc
to what was built.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lightdash gives dimensions and metrics one namespace, and every entry in
columns[] becomes a dimension -- including a column the emitter creates
only to host a column-level metric. When a metric carried the same name,
Lightdash kept the dimension and skipped the metric:

  obt_marketing_daily: Skipped metric "roas" because a dimension with the
  same name exists. Dimensions take priority.
  obt_marketing_daily: Skipped metric "attributed_revenue" because a
  dimension with the same name exists. Dimensions take priority.

The skip is a line in the deploy log and nothing in the emitted file, so
downstream the metric just looks like it was never defined -- an analytics
agent asking for roas finds only a raw per-row column and answers a
different question. Over the ecommerce marts both collisions were on
obt_marketing_daily: sum(attributed_revenue) named attributed_revenue
(the metric's own backing column is the collision) and the derived
roas = attributed_revenue / ad_spend against a precomputed roas column.

Resolve it the way snowflake-semantic-view and databricks-metric-view
already resolve the same clash: the computed metric is canonical, the
colliding field goes. The colliding column is no longer emitted, and any
metric it hosted is re-homed at model level with an explicit
sql: ${TABLE}.<col> -- the same aggregation, spelled out instead of
inferred from the column it hung on. Re-homing moves a metric and never
removes one, so a derived ${ref} to it still resolves.

Exception: a column named by meta.primary_key or by a join's sql_on is
resolved structurally and Lightdash rejects the whole explore when that
dimension is missing, so there the dimension stays and the metric degrades
to a note. Both branches note, so no metric disappears silently.

Goldens are unchanged: the ecommerce fixture has no colliding names.
The branch was 25 commits behind main and predated the Emit signature change
that threads non-fatal degrade warnings back to the CLI, so it no longer
satisfied the Emitter interface.

  - lightdash.Emit now returns ([]string, error). The notes are still ALSO
    written into schema.yml as a leading `# semglot:` comment block: that file
    is committed and deployed by `lightdash deploy`, so a reader of the
    artifact should see what was dropped without going back to the build log.
  - README: both sides added a dialect. Kept both rows and spliced the
    lightdash column into main's capability matrix rather than choosing one.
  - Golden regenerated. The diff is additions only: main's new fixture content
    (obt_sales, billing_customer_sk, the dim_customer join) plus one new
    collision note, `aov`, which the merged fixture now triggers — the fix
    firing on a case main introduced.

go test ./... green, go vet clean, gofmt clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XdPy5mkvNRuGZ59rShLUag
The emitter branch added the target but never touched this file, so the mapping
reference described every dialect except lightdash. Adds it to the output table
and as a column in the IR-concept mapping, and records its gaps and limits:
composite primary keys have nowhere to go (meta.primary_key is a single name),
synonyms and enum values fold into the column description, and a Lightdash
metric cannot express a filtered aggregate or reference a metric on another
model.

Also documents a failure mode the `--` notation cannot express. Lightdash puts
dimensions and metrics in one namespace per model and resolves a clash in
favour of the dimension, silently dropping the metric — so a metric named after
a column is not a gap, it is a correctness trap: the layer deploys cleanly and
quietly lacks the metric it exists to publish. That is the bug this branch
fixes, and the reference should say so rather than leaving the next person to
rediscover it from a deploy warning.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XdPy5mkvNRuGZ59rShLUag
"Style" implied an aesthetic preference. This is a compatibility switch, and
its values are literally the YAML key path the meta block is written under
("meta" / "config.meta"), so naming it for the key path makes the name and the
value validate each other: `dbt-meta-key-path: config.meta` needs no comment,
where `meta-style: config.meta` read like a category error.

The profile key moves with it (meta-style -> dbt-meta-key-path).

Also records WHY the option exists rather than only what it does. It cannot be
derived: semglot writes a file that someone else's dbt parses later, so the
emitting process never sees the dbt that will read it. Top-level meta still
parses without warning on dbt 1.11 (verified with --show-all-deprecations
against the real 38-model schema), so "" stays the default; when dbt removes
it, flipping the default to "config.meta" makes the option vestigial and
deletable.

docs/plans/ is left alone: it records what was decided at the time.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XdPy5mkvNRuGZ59rShLUag
config.meta has been dbt's preferred form since 1.10 (2025-06-16), with two
minors shipped since, and 1.10 deprecated top-level meta. Anyone starting today
is on 1.10+, so the legacy form is the one that should need asking for.

Only an explicit "meta" now opts back into top-level `meta:`, for a project
pinned to dbt 1.9 or earlier. The option stays because it cannot be derived:
semglot writes a file that someone else's dbt parses later.

This changes emitted output for every caller that did not set the option, hence
its own commit.

Tests: the content tests read ldDoc's `meta:` shape and assert WHAT is emitted,
not where it nests, so emitLightdash pins them to the legacy path rather than
having a default change break all of them. Placement is owned by
TestLightdashConfigDbtMetaKeyPath, which now builds the emitter directly to
check the real default and gained a case for the "meta" opt-out. The golden is
generated with default options, so it exercises config.meta end to end.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01XdPy5mkvNRuGZ59rShLUag
@andrebaaij
andrebaaij merged commit 32950c1 into main Aug 12, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant