AI-Assisted rules generation from data profiles

[souravg-db2]

Changes

Added generation of rules from data profiles generated by profiler.

Linked issues

Resolves #481

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests
• added performance tests

[github-actions]

✅ 499/499 passed, 6 flaky, 48 skipped, 4h21m48s total

Flaky tests:

• 🤪 test_quality_checker_workflow_with_ref (169ms)
• 🤪 test_quality_checker_workflow_for_patterns (185ms)
• 🤪 test_quality_checker_workflow_for_patterns_exclude_patterns (55ms)
• 🤪 test_quality_checker_workflow_for_patterns_exclude_output (105ms)
• 🤪 test_quality_checker_workflow_for_patterns_table_checks_storage (10.009s)
• 🤪 test_profiler_workflow_for_patterns (10.004s)

_{Running from acceptance #3470}

[codecov]

Codecov Report

❌ Patch coverage is 90.62500% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 90.48%. Comparing base (16e23c3) to head (1b05bf8).
⚠️ Report is 4 commits behind head on main.

Files with missing lines	Patch %	Lines
src/databricks/labs/dqx/llm/llm_core.py	86.66%	4 Missing ⚠️
src/databricks/labs/dqx/llm/llm_utils.py	95.83%	1 Missing ⚠️
src/databricks/labs/dqx/profiler/generator.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #963      +/-   ##
==========================================
+ Coverage   90.45%   90.48%   +0.03%     
==========================================
  Files          64       64              
  Lines        6350     6476     +126     
==========================================
+ Hits         5744     5860     +116     
- Misses        606      616      +10     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[Copilot]

Pull request overview

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[mwojtyczka]

LGTM