Adding performance benchmarks

[mwojtyczka]

Changes

• Adding perf benchmarks for applying checks to ensure the performance does not degrade with time. Allow 25% degradation. Covering all check functions including foreach column rule.
• Publish perf benchmarks details as report in the docs. The benchmark report is published/updated in docs automatically (docs/dqx/docs/reference/benchmarks.mdx) using the baseline (tests/perf/.benchmarks/baseline.json), whenever the baseline changes.
• Run performance tests on each PR and as part of nightly github workflow.
• Baseline is created/updated automatically under tests/perf/.benchmarks/baseline.json. If the baseline file is removed or any test is removed or added, the baseline is updated automatically and commited to git. Note that the commits are not signed so user must re-add the commit if baseline is updated. Instructions in the contributing guide are provided.
• Updated docs

Tests specification:

• 100 million rows are used for each test, with 5 rounds.
• Tests are run sequentially to reduce variability.
• Benchmarks are created using Databricks Serverless cluster.
• 25% performance degradation allowed.

Example output for perf tests:

• Failing: https://github.com/databrickslabs/dqx/actions/runs/17294830624/job/49090378224
• Passing: https://github.com/databrickslabs/dqx/actions/runs/17296158556/job/49094753814

This PR is continuation of #525

Linked issues

Resolves #56

Tests

• manually tested
• added unit tests
• added integration tests
• added end-to-end tests
• added performance tests

[github-actions]

✅ 66/66 passed, 5m56s total

_{Running from perf_test #92}

[github-actions]

✅ 295/295 passed, 5 flaky, 1 skipped, 2h45m42s total

Flaky tests:

• 🤪 test_profiler_workflow_class (30.77s)
• 🤪 test_profiler_serverless (2m36.337s)
• 🤪 test_e2e_workflow (11m7.933s)
• 🤪 test_e2e_workflow_serverless (7m29.176s)
• 🤪 test_e2e_workflow (7m57.014s)

_{Running from acceptance #2194}

[github-actions]

✅ 66/66 passed, 6m15s total

_{Running from perf_test #4}

[mwojtyczka]

@cornzyblack @ghanse i moved the previous PR here so that we can run the tests directly. It is complete, please review.

[Copilot]

Pull Request Overview

This PR implements comprehensive performance benchmarks for the DQX library to monitor performance regression over time and ensure the library maintains optimal performance as it evolves.

• Adds pytest-benchmark based performance tests covering all check functions including row, dataset, and foreach column rules
• Implements automated performance testing pipeline with GitHub Actions that runs on PRs and nightly builds
• Creates automated reporting system that generates markdown documentation from benchmark results

Reviewed Changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
tests/perf/test_apply_checks.py	Comprehensive benchmark tests covering all check functions with 100 million row datasets
tests/perf/generate_md_report.py	Script to generate markdown reports from baseline benchmark results
tests/perf/conftest.py	Test configuration and fixtures for performance tests
tests/perf/.benchmarks/baseline.json	Baseline performance metrics for comparison
.github/workflows/performance.yml	GitHub Actions workflow for PR performance testing
.github/workflows/nightly.yml	Added nightly performance testing job
pyproject.toml	Added performance testing dependencies and hatch script
docs/dqx/docs/reference/benchmarks.mdx	Auto-generated benchmark documentation
docs/dqx/docs/dev/contributing.mdx	Updated contributing guidelines with performance testing info
Makefile	Added performance testing target
.github/pull_request_template.md	Added performance testing checkbox

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}

[cornzyblack]

Hi @mwojtyczka, all checks in check_funcs have been covered. I can see you added some tests with DQForEachColRule. Should I add more DQForEachColRule tests as additional foreach tests for the covered ones?

[mwojtyczka]

Hi @mwojtyczka, all checks in check_funcs have been covered. I can see you added some tests with DQForEachColRule. Should I add more DQForEachColRule tests as additional foreach tests for the covered ones?

That would be great. We should cover all check funcs with foreach. Check if you can commit to the branch.

[cornzyblack]

Hi @mwojtyczka, all checks in check_funcs have been covered. I can see you added some tests with DQForEachColRule. Should I add more DQForEachColRule tests as additional foreach tests for the covered ones?

That would be great. We should cover all check funcs with foreach. Check if you can commit to the branch.

Alright will add them, so I can't commit and push to this branch directly, but I can open a PR to this branch and maybe you can trigger the performance from your end?

[mwojtyczka]

Hi @mwojtyczka, all checks in check_funcs have been covered. I can see you added some tests with DQForEachColRule. Should I add more DQForEachColRule tests as additional foreach tests for the covered ones?

That would be great. We should cover all check funcs with foreach. Check if you can commit to the branch.

Alright will add them, so I can't commit and push to this branch directly, but I can open a PR to this branch and maybe you can trigger the performance from your end?

You can fork it and create PR to merge it to this branch. Alternatively, you could just drop the code here, but it would be great if your contribution is visible. I plan to merge on Monday, if you can complete by Monday go ahead, otherwise, I will add the missing tests.

[cornzyblack]

Hi @mwojtyczka, all checks in check_funcs have been covered. I can see you added some tests with DQForEachColRule. Should I add more DQForEachColRule tests as additional foreach tests for the covered ones?

That would be great. We should cover all check funcs with foreach. Check if you can commit to the branch.

Alright will add them, so I can't commit and push to this branch directly, but I can open a PR to this branch and maybe you can trigger the performance from your end?

You can fork it and create PR to merge it to this branch. Alternatively, you could just drop the code here, but it would be great if your contribution is visible. I plan to merge on Monday, if you can complete by Monday go ahead, otherwise, I will add the missing tests.

Sure, will add it to this branch (PR) before Monday.

[cornzyblack]

Hi @mwojtyczka I have made the changes in this PR #556

@ghanse I added the custom ways to generate dataset types in conftest for this comment
in conftest.py. Was this what you were expecting?

[ghanse]

Overall LGTM. Left a minor comment on GPG signing.

[ghanse]

Is there any way to avoid this? Could we add a GPG key in the repo secrets and sign with that key when the benchmarks are committed?

[mwojtyczka]

I haven't found a way because the commit is on behalf of the user that triggers the workflow. I don't want to use a specific user.