Adding detailed telemetry

[mwojtyczka]

Changes

• Added detailed telemetry to track usage of different methods
• Made spark connect column optional to support DQX execution without spark connect
• Minor docs changes

Linked issues

Resolves #378

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests
• added performance tests

[Copilot]

Pull Request Overview

This PR adds comprehensive telemetry tracking to monitor usage of different DQX methods and storage operations. It introduces a telemetry framework to capture method calls for analytics and monitoring purposes.

• Added a new telemetry module with decorator-based logging functionality
• Instrumented engine methods and storage handlers with telemetry decorators
• Updated documentation to use "in-transit" terminology instead of "in-flight"

Reviewed Changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
src/databricks/labs/dqx/telemetry.py	New telemetry module with logging functions and decorator
src/databricks/labs/dqx/engine.py	Added telemetry decorators to engine methods and individual check logging
src/databricks/labs/dqx/checks_storage.py	Added telemetry decorators to all storage handler methods
pyproject.toml	Added "ui/*" to ruff exclude list
docs/dqx/docs/reference/engine.mdx	Minor documentation fixes (trailing spaces)
docs/dqx/docs/guide/index.mdx	Changed "in-flight" to "in-transit" terminology
demos/dqx_demo_pii_detection.py	Changed "in-flight" to "in-transit" terminology

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[github-actions]

✅ 295/295 passed, 4 flaky, 1 skipped, 3h1m33s total

Flaky tests:

• 🤪 test_save_results_in_table_in_user_installation_missing_output_and_quarantine_table (4.476s)
• 🤪 test_e2e_workflow (13m49.895s)
• 🤪 test_save_load_checks_from_table_in_user_installation (9.044s)
• 🤪 test_e2e_workflow (10m54.893s)

_{Running from acceptance #2293}

[Copilot]

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

[Copilot]

Pull Request Overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 2 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[alexott]

small comments, otherwise looks good to me

[Copilot]

Pull Request Overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated 3 comments.

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}