Skip to content

Document for private PYPI and Auto Upload dependencies for workspace installation#930

Merged
mwojtyczka merged 27 commits into
mainfrom
ft/927_support_pr_pypi
Nov 18, 2025
Merged

Document for private PYPI and Auto Upload dependencies for workspace installation#930
mwojtyczka merged 27 commits into
mainfrom
ft/927_support_pr_pypi

Conversation

@dinbab1984

Copy link
Copy Markdown
Contributor

Changes

  • Documentation of steps for using private PYPI for installing DQX
  • Included Upload dependencies for workspace installation automatically

Linked issues

This feature addresses #927

Tests

  • manually tested
  • added unit tests
  • added integration tests
  • added end-to-end tests
  • added performance tests

@github-actions

github-actions Bot commented Nov 12, 2025

Copy link
Copy Markdown
Contributor

✅ 427/427 passed, 24 flaky, 30 skipped, 3h32m27s total

Flaky tests:

  • 🤪 test_engine_without_observer_no_metrics_saved[apply_checks_by_metadata_and_save_in_table] (10.004s)
  • 🤪 test_apply_checks_by_metadata_and_save_in_table_raises_error_for_sparkconnect[apply_checks_and_save_in_table] (10.004s)
  • 🤪 test_apply_checks_by_metadata_and_save_in_table_raises_error_for_sparkconnect[apply_checks_by_metadata_and_save_in_table] (10.009s)
  • 🤪 test_save_results_in_table_raises_error_for_sparkconnect (10.004s)
  • 🤪 test_load_checks_from_table_saved_from_dict_with_unresolved_for_each_column (10.006s)
  • 🤪 test_installation_with_upload_dependencies (10.005s)
  • 🤪 test_workflow_deployment_uploads_dependencies (10.005s)
  • 🤪 test_dependency_discovery_includes_extras (10.005s)
  • 🤪 test_quality_checker_workflow_with_custom_check_func_rel_path (10.006s)
  • 🤪 test_config_upload_dependencies_persists (10.004s)
  • 🤪 test_end_to_end_installation_and_workflow_with_upload_dependencies (10.005s)
  • 🤪 test_list_tables (10.004s)
  • 🤪 test_e2e_workflow_for_patterns (10.006s)
  • 🤪 test_list_tables_extended (10.006s)
  • 🤪 test_list_tables_with_exclude_patterns (10.005s)
  • 🤪 test_list_tables_no_matching (10.006s)
  • 🤪 test_e2e_workflow_for_patterns_exclude_patterns (10.008s)
  • 🤪 test_e2e_workflow_for_patterns_exclude_output (10.004s)
  • 🤪 test_e2e_workflow_for_patterns_table_checks_storage (10.013s)
  • 🤪 test_row_rule_executor_apply (10.011s)
  • 🤪 test_dataset_rule_executor_apply (10.013s)
  • 🤪 test_fresh_global_config_installation (10.005s)
  • 🤪 test_e2e_workflow_serverless (8m20.685s)
  • 🤪 test_engine_without_observer_no_metrics_saved[apply_checks_and_save_in_table] (3.209s)

Running from acceptance #3185

@codecov

codecov Bot commented Nov 13, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 81.08108% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.81%. Comparing base (4ed822d) to head (e219c18).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
...atabricks/labs/dqx/installer/workflow_installer.py 79.41% 7 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #930      +/-   ##
==========================================
- Coverage   89.91%   89.81%   -0.11%     
==========================================
  Files          60       60              
  Lines        5184     5221      +37     
==========================================
+ Hits         4661     4689      +28     
- Misses        523      532       +9     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@mwojtyczka mwojtyczka requested a review from Copilot November 13, 2025 08:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds support for installing DQX in workspaces with restricted internet access by enabling automatic upload of dependencies and documenting private PyPI usage.

  • Adds upload_dependencies configuration flag to control automatic dependency uploads during installation
  • Implements dynamic dependency discovery from package metadata with fallback support
  • Documents installation procedures for private PyPI mirrors and offline scenarios

Reviewed Changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 5 comments.

File Description
src/databricks/labs/dqx/config.py Adds upload_dependencies boolean field to WorkspaceConfig
src/databricks/labs/dqx/installer/config_provider.py Prompts user about internet access and sets upload_dependencies flag
src/databricks/labs/dqx/installer/workflow_installer.py Implements dependency extraction, discovery, and upload logic
docs/dqx/docs/installation.mdx Documents private PyPI and offline installation procedures

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/databricks/labs/dqx/installer/workflow_installer.py Outdated
Comment thread docs/dqx/docs/installation.mdx Outdated
Comment thread docs/dqx/docs/installation.mdx Outdated
Comment thread docs/dqx/docs/installation.mdx Outdated
Comment thread docs/dqx/docs/installation.mdx Outdated
@mwojtyczka mwojtyczka linked an issue Nov 13, 2025 that may be closed by this pull request
1 task

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread docs/dqx/docs/installation.mdx Outdated
Comment thread docs/dqx/docs/installation.mdx Outdated
Comment thread src/databricks/labs/dqx/installer/workflow_installer.py Outdated
* improved telemetry

@mwojtyczka mwojtyczka left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@mwojtyczka mwojtyczka mentioned this pull request Dec 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEATURE]: Add support for installing DQX with company hosted PYPI mirror

3 participants