Repository navigation
chore: move the in-development version to 0.3.0.dev0 - #691
Conversation
The next release is 0.3.0-incubating, so the in-development version becomes 0.3.0.dev0. There is no 0.2.0 release. - Set 0.3.0.dev0 in the schemas, spec.yaml, the Python package, the specification and ontology documents, the examples, the compliance fixtures and the tests. - Record in the version history that 0.3.0.dev0 replaces 0.2.0.dev0, and that documents declaring 0.2.0.dev0 must declare 0.3.0.dev0. - Reject 0.2.0.dev0 as an embedded model version in the ontology tests. The expression language revision table keeps its dated 0.2.0.dev entry.
christianeu-db
left a comment
There was a problem hiding this comment.
looks good - left a few notes to consider but feel free to address them as followups
|
|
||
| def _doc(metrics=None, datasets=None): | ||
| document = {"version": "0.2.0.dev0", "name": "t"} | ||
| document = {"version": "0.3.0.dev0", "name": "t"} |
There was a problem hiding this comment.
Not a blocker, but no workflow runs anything under compliance/. validation-ci.yml and python-ci.yml do not list it in their paths, and neither runs the grader tests or validates the fixtures. So the green checks on this PR do not cover this file or the nine *.ossie.yaml fixtures. Locally, the 13 grader tests pass and all nine fixtures validate.
This matters more after the rename. #631 adds five more portability fixtures that declare 0.2.0.dev0, and #310 adds an ontology example that no CI step validates. Either one can merge green after this PR and leave a document on main that the schema rejects. Should we add compliance/** to the Validation CI paths, with steps that run the grader tests and validate the fixtures? A follow-up PR is fine too.
| ### Migrating earlier document shapes | ||
|
|
||
| This is a breaking change in the unreleased `0.2.0.dev0` specification. Earlier | ||
| This is a breaking change in the unreleased `0.3.0.dev0` specification. Earlier |
There was a problem hiding this comment.
nit: the break is still in the 0.2.0.dev0 snapshot. 0.3.0.dev0 should always have the flattening applied.
(non-blocking side note: perhaps these kinds of migration notes should in a separate file rather than the core -spec. the core spec just states the current state w/o reflecting on the past)
|
@jbonofre do you mind updating core-spec/wide_table_semantics.md. This could also be a followup PR (if so, I can take it) |
apache/ossie#691 renamed the in-development version of the schema; there will be no 0.2.0 release. The vendored apache-ossie moves to d700cde, and the TPC-DS fixture is again an exact copy of the example on Ossie's main. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
apache/ossie#691 renamed the in-development version of the schema; there will be no 0.2.0 release. The vendored apache-ossie moves to d700cde, and the TPC-DS fixture is again an exact copy of the example on Ossie's main. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…filter Resolved the sole conflict in core-spec/spec.md Version History by keeping apache#691's 0.3.0.dev0 heading and replacement note, and re-applying apache#704's Dataset.source table-identifier changelog bullet. Bumped examples/dataset_sources.yaml from 0.2.0.dev0 to 0.3.0.dev0 to match main.
apache/ossie#691 moved the spec's version to 0.3.0.dev0; there will be no 0.2.0. dactopus-ossie-clickhouse 0.15.0 and newer read only the new version, so CI moves its pin from 0.14.0 to 0.16.0, and AGENTS.md drops the old name custom_extensions it kept until the pin moved. Nothing else in the models changes. The DuckDB extension (0.3.0) reads the new version and answers as before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PR #691 moved the in-development version to 0.3.0.dev0 everywhere the version is declared, but the Layer 3 "Wide Table Query Interface" spec (core-spec/wide_table_semantics.md) kept its draft-spec badge at 0.2.0.dev0. Its Layer 2 sibling (relational_semantics.md) was bumped in that PR, so this straggler left the layered spec set inconsistent. Bump the badge to 0.3.0.dev0 to match. Co-authored-by: Isaac <no-reply@databricks.com>
main moved the in-development version to 0.3.0.dev0 (apache#691), and the schema pins it, so the fixtures this PR adds declare it too. Co-authored-by: Isaac <no-reply@databricks.com>
main moved the in-development version to 0.3.0.dev0 (apache#691), and the schema pins it, so the fixtures this PR adds declare it too. Co-authored-by: Isaac <no-reply@databricks.com>
…form of M3 (#631) * test(compliance): add portability rung M6, date and time scalars One dataset with YEAR and DATE_TRUNC fields and an AVG(DATEDIFF(...)) metric, using the part-first DATEDIFF(part, start, end) order from expression_language.md. Adds one construct on top of the baseline. Co-authored-by: Isaac <no-reply@databricks.com> * test(compliance): add the FILTER (WHERE ...) form to portability rung M3 expression_language.md makes the aggregate FILTER (WHERE ...) clause REQUIRED. Two variants (qualified and bare) express the same filtered aggregate as the existing CASE WHEN fixtures, so a difference between the two forms isolates handling of the clause itself. Co-authored-by: Isaac <no-reply@databricks.com> * test(compliance): add portability rung M4, a cross-dataset metric Two datasets joined by a declared relationship and one model-scoped metric that aggregates across the join (SUM over one dataset divided by a COUNT DISTINCT over the other), referencing fields as dataset.field. Co-authored-by: Isaac <no-reply@databricks.com> * test(compliance): add portability rung M5, metric-to-metric composition A model-scoped metric (avg_order_value) composed from two other model-scoped metrics referenced by name, per Metric Scoping in core-spec/spec.md. Co-authored-by: Isaac <no-reply@databricks.com> * docs(compliance): describe portability rungs M0-M6 Lists the M3 FILTER (WHERE ...) form and rungs M4-M6 in the README table, notes that M7 and M8 wait on the window ORDER BY NULL-placement default, and records that the fixtures live in this repository. Co-authored-by: Isaac <no-reply@databricks.com> * test(compliance): cover M4-M6 in the identity test, name every element by kind and name Follow-ups from the review of #631: - The identity regression test looped over M0-M3 only; it now covers M0-M6, and the grader README says so. - M5's reference_form_primary said "qualified", but its only variant (the composite under test) references metric names; it is now "metric-name". - M6 named "fields and metric" with a comma-joined name, and M2 named "metrics" without a name, so a lookup by kind and name found neither. Every rung now lists elements_under_test, each with one kind (metric or field) and one name. Two new tests keep the manifest honest: every listed element is found by kind and name in each variant of its rung, and each primary variant uses the rung's primary reference form. Both fail on the previous manifest. Co-authored-by: Isaac <no-reply@databricks.com> * test(compliance): carry M5's composition inlined, as the spec requires A metric expression cannot refer to other metrics (core-spec/spec.md, Metric Scoping; expression_language.md), and since #608 validate.py warns when one does. M5's composite, avg_order_value = total_amount / order_count, referenced the two metrics by name. The "Referenced as metric_name" row in the Metric Scoping table is how a query addresses a metric, not what an expression may contain. M5 now carries the composition the way a converter has to emit it when the source tool composes measures: the parts inlined, over qualified dataset.field references, SUM(sales.amount) / NULLIF(COUNT(sales.order_id), 0), NULL when there are no orders. The two base metrics stay alongside it. The fixture validates with no warnings on main's validator, and its primary reference form is qualified again. Co-authored-by: Isaac <no-reply@databricks.com> * test(compliance): write M3's CASE form without ELSE 0, as the FILTER form means expression_language.md (Conditional Aggregations) states that FILTER (WHERE ...) is equivalent to a CASE with no ELSE for an aggregate that ignores NULL, and that an ELSE 0 "would count as a real row". M3's CASE fixtures wrote ELSE 0, so on rows that exist but do not match they give 0 where the FILTER variants added here give NULL; they were not the same aggregate the rung describes. Found by idk-arsh's execution check on apache/ossie-converters#24. Both CASE fixtures now read SUM(CASE WHEN ... THEN amount END), which also matches DAX (CALCULATE over no rows is BLANK). All four M3 fixtures validate on main. Co-authored-by: Isaac <no-reply@databricks.com> * test(compliance): carry the bare variants as dataset-scoped metrics, grade them Since #486 a model-scoped metric must reference fields as dataset.field, and since #608 validate.py warns on the bare variants of M0, M2 and M3 ("is not qualified"). Metric Scoping in core-spec/spec.md gives the bare form its place: a dataset-scoped metric (datasets[].metrics) references its own dataset's fields by single-part name. The four bare fixtures now carry their metrics there, and validate without warnings on main. The grader read only the model-root metrics, so a dataset-scoped metric was not graded at all, and the identity test still passed. It now collects datasets[].metrics too, keyed dataset.metric as the spec references them, and the identity test asserts that every metric in a fixture is graded. New tests: a dataset-scoped metric graded PORTABLE, dropped, and moved to the model (REWRITTEN, identity not preserved). All four fail without the grader change. Co-authored-by: Isaac <no-reply@databricks.com> * test(compliance): name manifest elements by the grader's keys; spec spelling in M6 Follow-ups from the second review of #631: - The manifest named fields bare (amount_with_tax, order_year, order_month_start), but the grader reports a field as dataset.field, so matching manifest elements to findings found no result for M1 or M6. Fields are now named sales.<field>, and a variant that carries the rung's metrics as dataset-scoped metrics says so with metrics_in: sales (the grader reports those as sales.<metric>). The manifest test now uses the grader's own _collect keys instead of building a name set of its own, so the two cannot drift apart; it fails on the previous manifest. - M6 wrote DATEDIFF('day', ...), the Tableau spelling in the mapping table; the spec writes the part as a keyword, DATEDIFF(day, ...). Co-authored-by: Isaac <no-reply@databricks.com> * docs(compliance): point the portability ladder at apache/ossie-converters#24 Issue #576 was transferred to apache/ossie-converters#24 with the converters; the ladder, grader and manifest still cited the old number, which now resolves to nothing. Co-authored-by: Isaac <no-reply@databricks.com> * fix(compliance): keep the dataset qualifier on a field two datasets declare M4 is the first rung with two datasets, and customer_id is declared by both sales and customers. The grader stripped every table qualifier before comparing trees, so a converter that re-pointed COUNT(DISTINCT customers.customer_id) to sales.customer_id, or dropped the qualifiers, was graded as a cosmetic rewrite, and _repair_by_expression paired such a renamed metric as expression-equivalent. _ambiguous_fields() reads the field names that more than one input dataset declares; for those the qualifier is kept, for all others it is stripped as before (M0-M3 have one dataset each, so their grades do not change). Three regression tests; two fail on the previous grader. Suggested-by: Jean-Baptiste Onofré <jbonofre@apache.org> Co-authored-by: Isaac <no-reply@databricks.com> * test(compliance): move the M3-M6 fixtures to spec version 0.3.0.dev0 main moved the in-development version to 0.3.0.dev0 (#691), and the schema pins it, so the fixtures this PR adds declare it too. Co-authored-by: Isaac <no-reply@databricks.com> * fix(compliance): resolve a dataset qualifier against the dataset that declares the field Review of #631. The shared-field rule kept or dropped a qualifier by the field name alone, so a field requalified with its own dataset needed review when another dataset declared the name too, and a metric that named a dataset not declaring the field was cosmetic. Each column is now resolved to the dataset it names before the trees are compared: a bare name in a field expression or dataset-scoped metric is the element's own dataset, elsewhere the one dataset that declares it, and a written qualifier is kept as written. Co-authored-by: Isaac <no-reply@databricks.com> * fix(compliance): treat a returned dataset with an authored dataset's source as that dataset A converter that names a dataset after its source table (lightdash returns orders for an authored sales over samples.sales.orders) renamed it. Since qualifiers are kept as written, its metrics and fields were graded as dropped plus an unrelated addition. Map the renamed dataset's qualifiers on the returned side to the authored dataset before resolving, when exactly one authored dataset has that source. The 120 cases of the latest Interop matrix run now grade as on main. Adds a test built from lightdash's real M0 return. Co-authored-by: Isaac <no-reply@databricks.com> * docs(compliance): M5 carries composition inlined, without claiming metrics cannot reference metrics Since #703 a model-scoped metric may evaluate a dataset-scoped metric with AGG(dataset.metric). Reword the M5 fixture comment, the manifest construct and the README row. Co-authored-by: Isaac <no-reply@databricks.com> --------- Co-authored-by: Isaac <no-reply@databricks.com>
Summary
The next release is
0.3.0-incubating, so the in-development version becomes0.3.0.dev0. There will be no 0.2.0 release.0.3.0.dev0everywhere the current version is declared. That coversossie-schema.json,ontology.json,mapping.json,spec.yaml, the Python package (pyproject.toml,uv.lock,models.py),spec.md,ontology.md,relational_semantics.md,docs/index.md, the examples, the compliance fixtures and the tests.const, so documents that declare0.2.0.dev0no longer validate. The version history inspec.mdsays they must declare0.3.0.dev0, and the ontology tests now reject0.2.0.dev0. Because it changes the schemas, this counts as a specification change, which needs three binding +1 votes.apache/ossie-convertersneeds a matching change. It writes0.2.0.dev0in about 30 files.0.2.0.deventry.