[FEATURE]: Versioning of rules

[mwojtyczka]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

Tracibility is important for large organization. Ability to track which rules were applied to historical data is especially important in regulated orgs.

Proposed Solution

Loose ideas, e.g. add version to the rules and quality results. For quality results we may leverage user_metadata field. Add version to rules stored in a table, and option to provide version number for retrieval. The version should be part of the summary table as well.

Additional Context

Come up with a proposal.

[hski-bayer]

Maybe enable Change Data Feed on the rules definition table?

[mwojtyczka]

we had a request to make this available directly in the results

[STEFANOVIVAS]

I don't know if I got it. It will be something like to create a unique identification for every data quality check saved in the "checks_table" and list all check id's that were applied to the input dataframe in that specific run?

[STEFANOVIVAS]

Hi @mwojtyczka! Sorry to bother you, but is this feature a short-term priority? Or would you prefer I tackle something else?

[mwojtyczka]

@STEFANOVIVAS feel free to work on this, it would make sense to come up with a design first, before implementing. The idea was to have a way to add version to the rules, so that when the rule is updated (e.g. threshold in the rule is changed), the version can be bumped by the user. Currently if a rule is applied but then changed later, we actually don't know which rule definition was used at that time. The check results don't provide info today, about the version of the rule, only info which rule function was used.

[STEFANOVIVAS]

Hi, @mwojtyczka! Does this feature for all rules applied in a given execution of apply_checks, or only to those that cause errors and warnings?
Maybe the easiest way to solve this problem could be just to show all the information about the check in the output dataframe, instead of creating a versioning rule mechanism?

[mwojtyczka]

when you say output dataframe, do you mean in the _error & _warning columns?

[STEFANOVIVAS]

Exactly, @mwojtyczka. We already have some of the check rule attributes in these columns, like columns, filter, and check function.

[mwojtyczka]

i think this is more relevant to those checks that cause errors/warnings. We should include the version in the summary table as well and these would capture the version whether there are errors/warnings or not. We could either extend the result with a new field or use the user_metadata field which is a map field.

My idea was to include the rule version in the rule definition and emit the rule version with every result. We could think of new field, or reuse user_metadata, although the user metadata field is a bit hard to read as it is a map. For detecting rules drift, we should also calculate hash of a rule and store it in the result to get exact technical fingerprint. If someone forget to bump the version, one could use the hash value to detect silent drift.

I would propose the following:

• New field in DQRule: version
• New fields in the results: rule_version (value from the DQRule.version), rule_fingerprint: hash of the rule definition, excluding version

We should version individual rules.

[STEFANOVIVAS]

Hi, @mwojtyczka! Just one more thing: if we're only going to display the rule version number in the results, I suppose we'll need a table to store all possible rule versions, right? Because in situations with historical data, we wouldn't be able to remember the rule definition. Should we reuse the checks_table, maybe adding two more columns ("hash" and "version")?
The advantage of reusing user metadata is that it the rule version and hash would be reflected automatically in the user metadata column of the checks_table too, along with the result dataframe.

[mwojtyczka]

If the version is a new field in the rule definition, then it should be a new column in the checks table. But I'm more inclined to have the version as part of user_metadata because not everyone need rules versioning and it will minimize the changes. In that case the version would just be part of the existing user metadata field / column. We can think maybe of extending the checks load/save method with a version (e.g. load "latest" or "v1.1.0).

In general, versioning makes most sense when checks are stored in a table, because user can centralize checks definitions in one place. In that case all rules and versions are stored centrally so I don't really see a need to introduce a new table to track versions. The only question is on how to track the results back to the definition. When using summary table and end-to-end methods, this would be available in the summary table. When not using summary table and end-to-end methods, I suppose the only option currently is to store location as another field in the user metadata:

- criticality: warn
      check:
        function: is_not_null_and_not_empty
        arguments:
          column: col5
      user_metadata:
        version: 0.1.1  # when wanting to version the rules
        location: /Volume/path/checks_0_1_1.yaml  # locaiton of the rules for tracing purposes

[STEFANOVIVAS]

If we want to detect silent drift in the checks rules I think that we could update the APPEND mode behavior or extend save_checks method to permit a something like MERGE mode ("For detecting rules drift, we should also calculate hash of a rule and store it in the result to get exact technical fingerprint. If someone forget to bump the version, one could use the hash value to detect silent drift"). If there is no check rule for that function and input table, insert the new check rule. If there is already a check rule for that function and input table, and any other column change, insert a new row in the checks_table with a +1 rule version.

Regarding the issue of tracking the results back to the definition we can use a hash or unique ID in the checks_table definitions and exibit it in the results as you thought (rule_fingerprint), so we can join the results and checks_table by this rule or unique ID.

When you say "summary table" do you mean Summary metrics table, right? In this table we have a run-level metrics information and not a data quality check-level granularity. In fact, I proposed to enhance the summary metrics information to show metrics quantity by check rule, so we can see which check had the most impact (#943). After that we could associate the rule definition by performing a join with the checks_table to take the rule definition.

[mwojtyczka]

I think we should focus on a solution that works well with checks stored in a table. In that context, using a rule_fingerprint as the version identifier may be better. It’s hard to imagine users manually defining versions for each rule. In my opinion, versioning should be as automatic as possible.

While it would be possible to use a MERGE-based approach to automatically bump versions, I think this would be problematic. In particular, it becomes non-trivial to determine whether a change represents an update to an existing rule, or the addition of a new rule.

Instead, we can introducing created_at and rule_fingerprint columns to the checks table. When loading checks for a given run_config_name, we would always load the latest set of rules based on created_at. When saving checks the default behavior should be "append", so history is preserved automatically. A new set of checks would be stored with a new created_at timestamp. The rule_fingerprint would be computed automatically at save time. Users could optionally switch to an "overwrite" mode, with a clear warning that history will not be preserved.

When checks are applied, the corresponding rule_fingerprint should be saved in the results preferably as a dedicated field (rather than in user_metadata), since this makes tracking, joining, and auditing much easier. This approach does require providing the full set of rules when saving checks, but that is already how the system behaves today, so it doesn’t introduce a new constraint.

For tracking purposes, I agree that check location + rule_fingerprint is a reliable identifier. The current approach using check location + name + run_config_name is not always sufficient, especially since name is optional.

By "summary table" I meant a summary metrics table. As noted, this level of aggregation does not provide enough granularity for detailed tracking. Check location alone is insufficient without a stable, versioned identifier such as the rule fingerprint. For detailed rule tracking, we can can rule it out. I mentioned this, because the summary metrics table contain information about the checks location. For tracking we need both the checks location and rule fingerprint or version.

Maybe it is not a bad idea to save the checks_location in the row-level results as well, along the the rule_fingerprints. It will make the results bigger, but it would make tracking much easier.