Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
0386459
feat(databricks): add databricks-metric-view emitter
andrebaaij Jul 20, 2026
c8699a1
feat(cli): require a catalog for databricks-metric-view
andrebaaij Jul 20, 2026
6b6607a
test(databricks): golden + structure tests for metric-view target
andrebaaij Jul 20, 2026
34bb5fe
docs: port databricks-metric-view design, plan, and dialect mapping
andrebaaij Jul 20, 2026
f0f971e
fix(databricks): stop dropping raw measures from tables with metrics
andrebaaij Jul 20, 2026
326e96a
fix(databricks): strip source qualifier case-insensitively
andrebaaij Jul 20, 2026
30f20e4
docs(databricks): correct fact-selection rule to match shipped behaviour
andrebaaij Jul 20, 2026
4e74630
fix(dbt): default a measure's expr to its name when omitted
andrebaaij Jul 20, 2026
733d7f9
fix(databricks): stop deduping distinct raw measures by expression
andrebaaij Jul 20, 2026
8d703b0
fix(databricks): compare measure expressions case-insensitively
andrebaaij Jul 20, 2026
2b6b42d
fix(databricks): skip joined dimensions with a compound expr, note it
andrebaaij Jul 20, 2026
413dda1
fix(databricks): dedup metric names inside the metric loop
andrebaaij Jul 20, 2026
58793e9
style(databricks): drop em-dashes from emitted note copy
andrebaaij Jul 20, 2026
e697077
fix(databricks): validate agg before it reaches the metric-view YAML
andrebaaij Jul 20, 2026
4883050
fix(databricks): fold model.Notes into every emitted view's comment
andrebaaij Jul 20, 2026
948dba2
docs(databricks): document the Databricks Runtime 17.3+ requirement
andrebaaij Jul 20, 2026
22f1faf
style(databricks): finish the loose ends from earlier rounds
andrebaaij Jul 20, 2026
21278fc
fix(dialects): keep every FK when a table references the same target …
andrebaaij Jul 21, 2026
633286f
test(databricks): pin the zero-fields skip, the one path that drops a…
andrebaaij Jul 21, 2026
0c47b08
feat(dialect): let emitters return non-fatal degrade warnings
andrebaaij Jul 21, 2026
01d4cac
refactor(cmd,dialect): thread the new Emit signature through the CLI …
andrebaaij Jul 21, 2026
9f841ac
test(dialect,cmd): pin the new warning channel end to end
andrebaaij Jul 21, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 22 additions & 17 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,8 +17,8 @@

Where [`sqlglot`](https://github.com/tobymao/sqlglot) translates across SQL
dialects, **semglot** translates across **semantic-layer dialects** (dbt
semantic models, Snowflake Cortex, Snowflake semantic views, and more) through
one neutral intermediate representation (IR).
semantic models, Snowflake Cortex, Snowflake semantic views, Databricks metric
views, and more) through one neutral intermediate representation (IR).

You point it at a source layer, pick a target dialect, and it writes the
equivalent layer out. Because everything routes through the IR, adding a dialect
Expand All @@ -37,6 +37,7 @@ both, so `dbt` to `dbt` is a lossless round-trip.
| `supersimple` | | ✓ |
| `nao-yaml` | | ✓ |
| `nao-context-rules` | | ✓ |
| `databricks-metric-view` | | ✓ |

Adding a dialect is small, self-contained work: implement the `Dialect` interface
(`Parse`, `Emit`, or both) and register it, and every conversion to and from it
Expand Down Expand Up @@ -129,7 +130,7 @@ profiles:
source-dialect: dbt # optional. default: dbt
target-dialect: snowflake-semantic-view # required
output: ./out/view # required. directory to write into
database: ANALYTICS # required for Snowflake targets (cortex, snowflake-semantic-view)
database: ANALYTICS # required for warehouse targets (cortex, snowflake-semantic-view, databricks-metric-view)
schema: SEM # optional. default: MAIN (schema of the source tables)
view-schema: SEM_VIEWS # optional. schema for the emitted semantic-view object; defaults to schema
model-name: catalog # optional. default: source dir name
Expand All @@ -142,27 +143,27 @@ profiles:
`MAIN`, and `model-name` is the source directory name.
- `build` fails clearly when the config is missing or unparseable, the `--profile`
is not found, a required field (`source`, `target-dialect`, `output`) is absent,
or a Snowflake target has no `database`.
or a warehouse target has no `database`.

## Dialect support

Every dialect maps to the same neutral IR, but targets differ in how much of it
they can express. This is what each **target** emits today (`dbt` is currently
the only source).

| Feature | `dbt` | `cortex` | `snowflake-semantic-view` | `supersimple` | `nao-yaml` | `nao-context-rules` |
|-------------------------|:-----:|:--------:|:-------------------------:|:-------------:|:----------:|:-------------------:|
| Tables | ✓ | ✓ | ✓ | ✓ | | ~ |
| Columns | ✓ | ✓ | ✓ | ✓ | ✓ | ~ |
| Time dimensions | ✓ | ✓ | ~ | ✓ | ✓ | ~ |
| Descriptions | ✓ | ✓ | ✓ | ✓ | ~ | ✓ |
| Data types | ✓ | ✓ | | ✓ | | |
| Primary keys | ✓ | ✓ | ✓ | ✓ | | |
| Relationships | ✓ | ✓ | ✓ | ✓ | | ✓ |
| Metrics (aggregations) | ✓ | ✓ | ✓ | ✓ | ~ | ✓ |
| Ratio & derived metrics | ✓ | ✓ | ✓ | ~ | ✓ | ✓ |
| Synonyms | ~ | ✓ | | | ≈ | ≈ |
| Enums / allowed values | ✓ | ~ | ≈ | ≈ | ✓ | ✓ |
| Feature | `dbt` | `cortex` | `snowflake-semantic-view` | `supersimple` | `nao-yaml` | `nao-context-rules` | `databricks-metric-view` |
|-------------------------|:-----:|:--------:|:-------------------------:|:-------------:|:----------:|:-------------------:|:------------------------:|
| Tables | ✓ | ✓ | ✓ | ✓ | | ~ | ✓ |
| Columns | ✓ | ✓ | ✓ | ✓ | ✓ | ~ | ✓ |
| Time dimensions | ✓ | ✓ | ~ | ✓ | ✓ | ~ | ~ |
| Descriptions | ✓ | ✓ | ✓ | ✓ | ~ | ✓ | ✓ |
| Data types | ✓ | ✓ | | ✓ | | | |
| Primary keys | ✓ | ✓ | ✓ | ✓ | | | |
| Relationships | ✓ | ✓ | ✓ | ✓ | | ✓ | ✓ |
| Metrics (aggregations) | ✓ | ✓ | ✓ | ✓ | ~ | ✓ | ✓ |
| Ratio & derived metrics | ✓ | ✓ | ✓ | ~ | ✓ | ✓ | ✓ |
| Synonyms | ~ | ✓ | | | ≈ | ≈ | ✓ |
| Enums / allowed values | ✓ | ~ | ≈ | ≈ | ✓ | ✓ | ≈ |

`✓` structured · `≈` rolled up as text in a description or comment · `~` partial · blank not emitted.

Expand All @@ -179,6 +180,10 @@ drops it:
- **`nao-yaml`** is a flat, model-global document, so it has no table grouping.
- **`nao-context-rules`** is prose, so it lists only elements that carry a
description or synonyms.
- **`databricks-metric-view`** requires Databricks Runtime 17.2+ for the base
metric-view YAML shape; `display_name` and `synonyms` are emitted whenever
the IR carries a label or synonyms and require Runtime 17.3+. On an older
warehouse, a view containing them is rejected.

## License

Expand Down
9 changes: 5 additions & 4 deletions cmd/semglot/config.go
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,10 @@ type buildSpec struct {
Description string
}

// snowflakeTargets emit into a physical Snowflake database and therefore require
// a database; without one they'd emit invalid, unqualified DDL.
var snowflakeTargets = map[string]bool{"cortex": true, "snowflake-semantic-view": true}
// warehouseTargets emit into a physical warehouse (Snowflake, or a Databricks
// Unity Catalog) and therefore require a resolved database/catalog; without one
// they'd emit invalid, unqualified DDL.
var warehouseTargets = map[string]bool{"cortex": true, "snowflake-semantic-view": true, "databricks-metric-view": true}

// loadProfile reads configPath, selects the named profile, applies defaults, and
// validates it into a buildSpec.
Expand Down Expand Up @@ -105,7 +106,7 @@ func loadProfile(configPath, name string) (buildSpec, error) {
if spec.ModelName == "" {
spec.ModelName = defaultModelName(spec.Sources)
}
if snowflakeTargets[spec.TargetDialect] && spec.Database == "" {
if warehouseTargets[spec.TargetDialect] && spec.Database == "" {
return buildSpec{}, fmt.Errorf("profile %q: target-dialect %s requires a database", name, spec.TargetDialect)
}
return spec, nil
Expand Down
13 changes: 13 additions & 0 deletions cmd/semglot/config_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,19 @@ func TestLoadProfileSnowflakeRequiresDatabase(t *testing.T) {
}
}

func TestLoadProfileDatabricksRequiresDatabase(t *testing.T) {
cfg := writeConfig(t, `profiles:
p:
source: /x
target-dialect: databricks-metric-view
output: ./out
`)
_, err := loadProfile(cfg, "p")
if err == nil {
t.Fatal("want error: databricks-metric-view target with no database")
}
}

func TestDefaultModelName(t *testing.T) {
for in, want := range map[string]string{
"/a/b/ecommerce": "ecommerce",
Expand Down
13 changes: 6 additions & 7 deletions cmd/semglot/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,8 @@ func buildCmd(args []string) int {
fmt.Fprintln(os.Stderr, "build: parse:", err)
return 1
}
if err := emitter.Emit(model, spec.Output); err != nil {
warnings, err := emitter.Emit(model, spec.Output)
if err != nil {
fmt.Fprintln(os.Stderr, "build: emit:", err)
return 1
}
Expand All @@ -89,12 +90,10 @@ func buildCmd(args []string) int {
fmt.Fprintln(os.Stderr, " - "+n)
}
}
if spec.TargetDialect == "cortex" {
if gaps := dialect.CortexTypeGaps(model); len(gaps) > 0 {
fmt.Fprintf(os.Stderr, "warning: %d Cortex column(s) had no source data_type; inferred a type (add data_type in dbt to fix):\n", len(gaps))
for _, g := range gaps {
fmt.Fprintln(os.Stderr, " - "+g)
}
if len(warnings) > 0 {
fmt.Fprintf(os.Stderr, "warning: %d item(s) degraded or dropped by the %s target:\n", len(warnings), spec.TargetDialect)
for _, w := range warnings {
fmt.Fprintln(os.Stderr, " - "+w)
}
}
fmt.Printf("wrote to %s (%s -> %s)\n", spec.Output, spec.SourceDialect, spec.TargetDialect)
Expand Down
52 changes: 52 additions & 0 deletions cmd/semglot/main_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,58 @@ func TestBuildCmdEndToEnd(t *testing.T) {
}
}

// TestBuildCmdSurfacesEmitterWarnings proves the CLI prints an emitter's
// returned warnings (not just model.Notes): the shared dbt fixture declares no
// data_type anywhere, so the cortex target must infer one for every column,
// and cortex.Emit returns those as warnings (dialect/cortex.go's
// cortexTypeGaps). This is what replaced the old hardcoded
// `if spec.TargetDialect == "cortex" { ... CortexTypeGaps ... }` branch.
func TestBuildCmdSurfacesEmitterWarnings(t *testing.T) {
out := t.TempDir()
src, err := filepath.Abs("../../dialect/testdata/dbt")
if err != nil {
t.Fatal(err)
}
cfg := filepath.Join(t.TempDir(), "semglot.yaml")
body := fmt.Sprintf(`profiles:
ecom:
source: %s
target-dialect: cortex
output: %s
database: ANALYTICS
schema: MAIN
model-name: eval_marts
`, src, out)
if err := os.WriteFile(cfg, []byte(body), 0o644); err != nil {
t.Fatal(err)
}

r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
origStderr := os.Stderr
os.Stderr = w
code := buildCmd([]string{"--profile", "ecom", "--config", cfg})
w.Close()
os.Stderr = origStderr
stderr, err := io.ReadAll(r)
if err != nil {
t.Fatal(err)
}

if code != 0 {
t.Fatalf("buildCmd exit code = %d, want 0\nstderr:\n%s", code, stderr)
}
got := string(stderr)
if !strings.Contains(got, "degraded or dropped by the cortex target") {
t.Fatalf("stderr = %q, want a warning block naming the cortex target", got)
}
if !strings.Contains(got, "inferred") {
t.Fatalf("stderr = %q, want a type-gap warning (column had no source data_type, a type was inferred)", got)
}
}

func TestBuildCmdMissingProfile(t *testing.T) {
if code := buildCmd([]string{}); code != 2 {
t.Fatalf("missing --profile should exit 2, got %d", code)
Expand Down
41 changes: 25 additions & 16 deletions dialect/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,26 +28,27 @@ Each emitter writes:
| `supersimple` | one `<TABLE>.yaml` per model, plus `NOTES.md` for anything deferred |
| `nao-yaml` | `semantic.yaml` |
| `nao-context-rules` | `RULES.md` (prose) |
| `databricks-metric-view` | one `<table>.yaml` metric view per model table, with direct joins to referenced tables (requires Databricks Runtime 17.2+; `display_name`/`synonyms` require 17.3+) |

## Mapping

Cells name the construct the IR concept becomes. Notation: `<->` dbt reads it
back into the IR too; `text` the value survives only as prose folded into a
description or comment; `--` not emitted (see [Gaps vs. limits](#gaps-vs-limits)).

| IR concept | `dbt` | `cortex` | `snowflake-semantic-view` | `supersimple` | `nao-yaml` | `nao-context-rules` |
|---|---|---|---|---|---|---|
| Table | `models:` + `semantic_models:` `<->` | `tables[].base_table` | `tables (...)` | one file per model | `--` | "Table reference" (if described) |
| Column / dimension | column + `dimensions type: categorical` `<->` | `dimensions[]` | `dimensions (...)` | `properties` | `dimensions[]` (deduped) | listed if described |
| Time dimension | `dimensions type: time` + `agg_time_dimension` `<->` | `time_dimensions[]` | plain dimension (not marked as time) | `properties` (Date) | `dimensions type: date` | with dimensions |
| Data type | column `data_type` `<->` | `data_type` | `--` | property `type` | `--` | `--` |
| Primary key | `primary_key` constraint + primary entity `<->` | `primary_key` | `primary key (...)` | `primary_key` | `--` | `--` |
| Relationship / join | `relationships` test on the FK column `<->` | `relationships[]` | `relationships (...) references` | `relations` (hasMany, join_key) | `--` | "Joins & routing" |
| Description | `description` `<->` | `description` | `comment='...'` | `description` | `description` (field/metric) | prose |
| Synonyms | `meta.synonyms` on the column `<->` | `synonyms:` | `with synonyms (...)` | `--` (gap) | `text` (into description) | `text` (into description) |
| Enum / allowed values | `accepted_values` test + `meta.enum` `<->` | `sample_values` + `text` | `text` (into comment) | `text` (into description) | `values:` | "Allowed values" |
| Simple metric (aggregation) | `measures` + `metrics type: simple` `<->` | `facts[]` | `metrics (...)` | metric aggregation | metric `source{table,column,aggregation}` | "Key metrics reference" |
| Ratio / derived metric | `type: ratio` / `type: derived` `<->` | `expr` (rendered SQL) | inline SQL in `metrics (...)` | division ratio -> pipeline; other arithmetic -> `NOTES.md` | `type: derived`, `formula` | rendered SQL |
| IR concept | `dbt` | `cortex` | `snowflake-semantic-view` | `supersimple` | `nao-yaml` | `nao-context-rules` | `databricks-metric-view` |
|---|---|---|---|---|---|---|---|
| Table | `models:` + `semantic_models:` `<->` | `tables[].base_table` | `tables (...)` | one file per model | `--` | "Table reference" (if described) | `source` (+ `joins[].source` for referenced tables) |
| Column / dimension | column + `dimensions type: categorical` `<->` | `dimensions[]` | `dimensions (...)` | `properties` | `dimensions[]` (deduped) | listed if described | `fields[]` |
| Time dimension | `dimensions type: time` + `agg_time_dimension` `<->` | `time_dimensions[]` | plain dimension (not marked as time) | `properties` (Date) | `dimensions type: date` | with dimensions | plain `fields[]` entry (not marked as time) |
| Data type | column `data_type` `<->` | `data_type` | `--` | property `type` | `--` | `--` | `--` |
| Primary key | `primary_key` constraint + primary entity `<->` | `primary_key` | `primary key (...)` | `primary_key` | `--` | `--` | `--` |
| Relationship / join | `relationships` test on the FK column `<->` | `relationships[]` | `relationships (...) references` | `relations` (hasMany, join_key) | `--` | "Joins & routing" | `joins[]` (quoted `"on":` condition) |
| Description | `description` `<->` | `description` | `comment='...'` | `description` | `description` (field/metric) | prose | `comment` (field/measure/view) |
| Synonyms | `meta.synonyms` on the column `<->` | `synonyms:` | `with synonyms (...)` | `--` (gap) | `text` (into description) | `text` (into description) | `synonyms:` (capped at 10) |
| Enum / allowed values | `accepted_values` test + `meta.enum` `<->` | `sample_values` + `text` | `text` (into comment) | `text` (into description) | `values:` | "Allowed values" | `text` (into comment) |
| Simple metric (aggregation) | `measures` + `metrics type: simple` `<->` | `facts[]` | `metrics (...)` | metric aggregation | metric `source{table,column,aggregation}` | "Key metrics reference" | `measures[]` |
| Ratio / derived metric | `type: ratio` / `type: derived` `<->` | `expr` (rendered SQL) | inline SQL in `metrics (...)` | division ratio -> pipeline; other arithmetic -> `NOTES.md` | `type: derived`, `formula` | rendered SQL | inline SQL in `measures[].expr` |

## Gaps vs. limits

Expand Down Expand Up @@ -79,9 +80,17 @@ and nao dimensions carry a structured `values:` list
notes. Verified against nao's own docs and the eval's real `semantic.yaml`, so
there is nowhere structured to emit tables or relationships. (nao-context-rules,
being prose, does emit joins.)
- **Data types in `snowflake-semantic-view` and the nao dialects.** Omitted on
purpose: these formats lean on the synced source-table schema for types rather
than restating them in the semantic doc.
- **Data types in `snowflake-semantic-view`, the nao dialects, and
`databricks-metric-view`.** Omitted on purpose: these formats lean on the
synced source-table schema for types rather than restating them in the
semantic doc. A metric view's `fields`/`measures` carry no `data_type` key at
all ([docs](https://docs.databricks.com/en/metric-views/index.html)).
- **Primary keys in `nao-yaml`, `nao-context-rules`, and
`databricks-metric-view`.** None of these formats has a primary-key slot: the
nao dialects have no per-table construct to hang one on, and a Databricks
metric view declares `source`/`joins`/`fields`/`measures` only, no
primary-key key; Unity Catalog constraints on the underlying table are the
authority there instead.

## Adding or changing a mapping

Expand Down
Loading
Loading