[FEATURE]: Add guidance on using DQX in a production environment

[mwojtyczka]

Is there an existing issue for this?

• I have searched the existing issues

Problem statement

Customers want to know best practices for using DQX in production. These are currently not documented.

Proposed Solution

Add new page in docs.

Example of best practices:

• Store checks in a delta table
• Have common set of checks for group of tables, and specific checks for individual tables
• Prefer row-level checks over dataset-level checks
• For critical use cases, quarantine bad data to ensure it is not used by downstream processes
• Use DAB for deployment
• Use apply methods that run checks across multiple tables for scaling
• Add missing checks as soon as issue in production system is idenfied
• Type of checks and when to use what: rule-based, AI-assisted, anomaly detection, UC (DQM)
• Usage of workflows:
  • use "overwrite" mode for output and quarantine configs for workflows as they run on the complete table
  • use custom installation folder
• Deployment: how to move/deploy checks across envs (check into git and make deployment alongside the pipeline)
• It is very likely that the data can be of high quality for one task but low for another. Therefore, different rules may need to be defined for the same data depending on the consumers
• Usage of profiler to get initial set of quality rules
• Data quality best practices,e.g. Focus on critical fields to concentrate data quality effort where it matter most
• Embed in jobs or run as background process
• Apply all checks in one pass (row-level, dataset-level, rule-based, anomaly detection-based)
• Actions: quarantine or output depending on data criticality
• Always pin to specific version when installing DQX:
  pip install databricks-labs-dqx==0.9.3
  databricks labs install dqx@v0.9.3
• Review breaking changes before upgrading: BREAKING CHANGES! section in https://github.com/databrickslabs/dqx/releases
• Versioning of rules,e.g. as part of user_metadata in the results.
• Tracing of the rules by including location and run config name in the user metadata (provide this user_metadata): https://databrickslabs.github.io/dqx/docs/guide/additional_configuration/#adding-user-metadata-to-the-results-of-all-checks
  (adding rules versioning to dqx is planned)

- criticality: warn
      check:
        function: is_not_null_and_not_empty
        arguments:
          column: col5
      user_metadata:
        run_config_name: sales_v0.1.0
        location: /Volume/path/checks_0_1_1.yaml  # locaiton of the rules for tracing purposes

Additional Context

No response