Add configurable criticality for profiler job

[mwojtyczka]

Changes

Added a parameter to allow users to specify the default criticality for generated rules (default 'error').

Linked issues

Resolves #1102

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests
• added performance tests

Documentation and Demos

• added/updated demos
• added/updated docs

[github-actions]

✅ 700/700 passed, 18 flaky, 39 skipped, 5h27m10s total

Flaky tests:

• 🤪 test_profiler_rounding_midnight_behavior (178ms)
• 🤪 test_profiler_non_TEST_SCHEMA_profile_options (252ms)
• 🤪 test_get_installation (214ms)
• 🤪 test_profiler_non_TEST_SCHEMA_profile_options_remove_outliers_no_outlier_columns (158ms)
• 🤪 test_profiler_with_pk_detection (183ms)
• 🤪 test_load_checks_when_global_installation_missing (10.006s)
• 🤪 test_profiler_non_TEST_SCHEMA_profile_options_with_rounding_enabled (10.005s)
• 🤪 test_profile_table_with_pk_detection (10.005s)
• 🤪 test_dashboard_repeated_workspace_installation (10.005s)
• 🤪 test_profile_tables_for_patterns_with_exclude_patterns (10.005s)
• 🤪 test_profile_tables_for_patterns_with_pk_detection (5.083s)
• 🤪 test_profile_tables_include_patterns (5.085s)
• 🤪 test_load_checks_when_user_installation_missing (10.005s)
• 🤪 test_profiler_empty_df (10.005s)
• 🤪 test_profile_tables_no_pattern_match (8.579s)
• 🤪 test_merge_info_columns_single_column (8.547s)
• 🤪 test_profiler_when_numeric_field_is_empty (10.005s)
• 🤪 test_observer_check_metrics_change_between_runs (4.706s)

_{Running from acceptance #4407}

[github-actions]

✅ 117/117 passed, 8h12m49s total

_{Running from anomaly #521}

[codecov]

Codecov Report

❌ Patch coverage is 66.66667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.98%. Comparing base (aa31ae6) to head (74039ac).
⚠️ Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
...rc/databricks/labs/dqx/profiler/profiler_runner.py	50.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #1117       +/-   ##
===========================================
+ Coverage   55.62%   91.98%   +36.35%     
===========================================
  Files         101      101               
  Lines        9418     9419        +1     
===========================================
+ Hits         5239     8664     +3425     
+ Misses       4179      755     -3424     

Flag	Coverage Δ
anomaly	0.00% <0.00%> (?)
anomaly-serverless	53.99% <33.33%> (?)
integration	50.24% <33.33%> (?)
integration-serverless	50.36% <33.33%> (?)
unit	55.63% <33.33%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ghanse]

LGTM