Add ARM64 (AArch64) benchmark coverage (ASV runner) to monitor perf regressions and gaps (incl. Kunpeng)

[113xiaoji]

Research

• I have searched the [pandas] tag on StackOverflow for similar questions.

• I have asked my usage related question on StackOverflow.

Link to question on StackOverflow

None

Question about pandas

Hi pandas maintainers,

I’m working on improving Python data ecosystem performance on ARM64 servers (Huawei Kunpeng, also Ampere/Neoverse/Graviton-class). I’d like to ask about pandas’ approach to ongoing ARM64 performance monitoring and whether the project would welcome additional ARM64 benchmarking coverage.

Why this matters

Performance changes may impact architectures differently. If we only run regular benchmarks on x86, ARM64 regressions or widening gaps can slip in as the codebase evolves—even when changes are well-intentioned.

pandas already uses ASV and has runner automation, so adding ARM64 could be a natural extension to help:

• catch ARM64 regressions early (nightly/periodic),
• understand cross-arch impact (x86 improves, ARM64 regresses, or vice versa),
• provide reproducible data for optimization work and PR reviews.

Questions for maintainers

1. Does pandas currently run ASV benchmarks on ARM64 on a regular schedule? If yes, where are the results published?
2. If not, would you be open to adding an ARM64 runner (or accepting externally-run ARM64 results)?
3. What is the preferred workflow for adding a new benchmark environment?
   • via existing runner infrastructure (if applicable),
   • or via a separate community-managed runner that publishes a dashboard?
4. Would maintainers value tracking ARM64 vs x86 ratios (to detect widening gaps), or just per-arch regressions?
5. Are there priority benchmark groups you’d like to ensure are covered on ARM64 (groupby, merge/join, indexing, IO, strings, datetime, etc.)?

What I can contribute (long-term, bare-metal)

• Access to stable bare-metal ARM64 machines (Kunpeng), dedicated to benchmarking (not VMs, not shared workloads).
• Reproducible environment controls (CPU frequency/power policy, BIOS settings, kernel, pinned toolchain), so results are stable and comparable.
• I’m willing to take long-term ownership of runner operations (automation, maintenance, upgrades, incident handling), so this won’t become a temporary/unmaintained runner.
• Investigation and actionable reports for regressions, with potential fixes upstream.

Operational commitments (so maintainers can trust sustainability)

• Cadence
  • Nightly run on main (latest SHA each night).
  • Weekly run on the latest maintenance/release branch(es) if useful.
  • Optional: monthly “deep run” (more repetitions / expanded suite) to confirm trends.
• Reproducibility
  • Dedicated bare-metal host, minimal background noise.
  • Pinned OS/kernel/toolchain; documented dependency versions (NumPy/Arrow/optional deps as applicable).
  • Fixed CPU governor / power mode and recorded machine metadata each run.
• Publishing
  • Publish an ASV dashboard (or whichever mechanism pandas prefers) with a clearly named environment (e.g., aarch64-kunpeng-baremetal).
  • Keep logs/metadata for debugging (dependency versions, build flags, ASV machine info).
• Retention
  • Keep raw results for ≥ 2 years to support trend analysis across releases.
• Regression policy
  • Use a triage threshold (example: >3–5% sustained regression on key groups, confirmed via rerun) to open actionable follow-ups.
  • Confirm regressions with reruns to reduce noise.
• Failure handling
  • Automatic retry on transient failures; alert on repeated failures with an owner/contact.
• Continuity
  • If I can’t maintain the runner, I’ll provide advance notice and help transfer scripts/docs to another maintainer/volunteer.

Happy to follow whatever process you prefer. If this belongs in a different repo (benchmark infra/runner), please redirect me.

Thanks!
<Your Name / Org>
<Contact / GitHub handle>
(ARM64 HW summary: Kunpeng model, cores, memory, OS/kernel; optional)

[rhshadrach]

Thanks for you're interest here! cc @pandas-dev/pandas-core

Does pandas currently run ASV benchmarks on ARM64 on a regular schedule?

No.

If not, would you be open to adding an ARM64 runner (or accepting externally-run ARM64 results)?

Certainly open to something being setup external to pandas-dev (I would have no right to not be open to that 😆). But I'd start there and if it is going well, we can move it inside of pandas-dev if preferred.

For accepting externally-run results, I think it would be suitable to open issues on performance degradation for ARM64 if they have been vetted by a human. For this, we'll also want to make sure the commit has not been flagged by https://github.com/pandas-dev/asv-runner/ (that part can be automated).

What is the preferred workflow for adding a new benchmark environment?

You can fork asv-runner if desired, but since this is external to pandas-dev and you'd be putting the work it I think this is your call to make.

Would maintainers value tracking ARM64 vs x86 ratios (to detect widening gaps), or just per-arch regressions?

If this is an easy lift, I don't see a reason not to do it. However, I also don't see a lot of particular value in it: a regression in ARM is a regression regardless of how x86 changes.

Are there priority benchmark groups you’d like to ensure are covered on ARM64 (groupby, merge/join, indexing, IO, strings, datetime, etc.)?

If you're subsetting the pandas ASVs, I would think IO has lower priority but would be tough to make judgement calls on the rest.

Weekly run on the latest maintenance/release branch(es) if useful.

Since these aren't changing, I'm not sure I see the point.

If I can’t maintain the runner, I’ll provide advance notice and help transfer scripts/docs to another maintainer/volunteer.

Just want to mention that I'd expect if this were to happen, there would be no one to hand it off to. I don't think that should stop you here, but wanted to make sure this was known.