Skip to content

Move the benchmark entry point to numpy_groupies.benchmarks #125

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
ml31415 merged 6 commits into from
Oct 16, 2026
Merged

Move the benchmark entry point to numpy_groupies.benchmarks #125

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
627e6f4
Move the benchmark entry point to numpy_groupies.benchmarks
Oct 16, 2026
6642ba3
Print the benchmark table as Markdown and stream the single run
Oct 16, 2026
c550e9f
Make the benchmark columns opt-in and explain them in --help
Oct 16, 2026
f6373ca
Time the purepy column once regardless of --repeat
Oct 16, 2026
59cb36c
Time the grouploop column once regardless of --repeat
Oct 16, 2026
ec6db6f
Refresh the benchmark tables in README and docs/benchmarks.md
Oct 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 15 additions & 15 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,26 +208,26 @@ All implementations share the calling syntax and produce the same results up to

## Performance

Best of five timed runs, in milliseconds, for 500,000 values in 1,000 groups (lower is better):
Best of three passes of five timed runs each, in milliseconds, for 500,000 values in 1,000 groups (lower is better):

| function | numpy | numba | numbagg | pandas |
| -------- | -----: | -----: | ------: | -----: |
| `sum` | 0.92 | 0.43 | 0.49 | 15.89 |
| `mean` | 1.56 | 0.51 | 0.56 | 15.99 |
| `max` | 1.71 | 0.64 | 0.88 | 15.15 |
| `std` | 3.09 | 0.67 | 0.68 | 16.82 |
| `nansum` | 4.17 | 0.44 | 1.76 | 22.26 |
| `median` | 48.41 | 11.49 | --- | 27.70 |
| `sort` | 135.15 | 25.99 | --- | --- |
| `cumsum` | 44.84 | 0.58 | --- | 7.32 |
| custom callable | 144.10 | 42.97 | --- | 108.59 |
| function | numpy | numba | numbagg | pandas |
| --------------- | -----: | -----: | ------: | -----: |
| `sum` | 0.95 | 0.44 | 0.47 | 15.96 |
| `mean` | 1.52 | 0.51 | 0.55 | 16.98 |
| `max` | 1.68 | 0.63 | 0.85 | 16.26 |
| `std` | 3.09 | 0.66 | 0.69 | 17.02 |
| `nansum` | 4.24 | 0.45 | 1.78 | 21.71 |
| `median` | 48.77 | 11.52 | --- | 28.57 |
| `sort` | 136.16 | 25.96 | --- | --- |
| `cumsum` | 45.88 | 0.58 | --- | 7.57 |
| custom callable | 142.54 | 43.16 | --- | 109.89 |

In short: the NumPy implementation is about 5–24× faster than pandas on common reductions, and Numba adds another 2–10× on top — and about 77× for `cumsum`. numbagg is par with us on the light scans; we lead on the nan-heavy accumulations (up to 4×), and numbagg has no `median`, cumulatives, `trapezoid` or `sort`. `median`, `sort` and custom callables are the expensive cases everywhere. Absolute numbers depend on your machine, so treat the ratios as the takeaway.
In short: the NumPy implementation is about 5–18× faster than pandas on common reductions, and Numba adds another 2–10× on top — and about 79× for `cumsum`. numbagg is par with us on the light scans; we lead on the nan-heavy accumulations (up to 4×), and numbagg has no `median`, cumulatives, `trapezoid` or `sort`. `median`, `sort` and custom callables are the expensive cases everywhere. Absolute numbers depend on your machine, so treat the ratios as the takeaway.

The complete table and the benchmark setup are in [docs/benchmarks.md](https://github.com/ml31415/numpy-groupies/blob/master/docs/benchmarks.md). To run it yourself, from the repository root (add `--pandas --numbagg` for those columns, `--purepy` for the pure-Python one):
The complete table and the benchmark setup are in [docs/benchmarks.md](https://github.com/ml31415/numpy-groupies/blob/master/docs/benchmarks.md). To run it yourself, from the repository root — `--help` explains which columns to ask for:

```sh
python -m numpy_groupies.benchmarks.generic --pandas --numbagg
python -m numpy_groupies.benchmarks --numbagg --pandas
```

The `numpy` and `numba` columns above are not the whole story: NumPy 1.25 brought major [speed improvements to ufuncs](https://numpy.org/doc/stable/release/1.25.0-notes.html), which narrowed the gap between the NumPy and Numba implementations considerably. The authors hope that `ufunc.at` or an equivalent in NumPy or SciPy will eventually become fast enough to make this package redundant.
Expand Down
114 changes: 56 additions & 58 deletions docs/benchmarks.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,85 +7,83 @@ Complete benchmark results for `aggregate`. A short summary is in the [README](.
- 500,000 indices, uniformly drawn from `[0, 1000)`.
- The values of `a` are uniform on `[0, 1)`, and everything above `0.8` is then set to `0`, so that there are falsy values for the boolean functions.
- For the `nan…` functions, of the remaining values those below `0.2` are set to `nan` (about 20%), leaving the rest on the interval `[0.2, 0.8]`.
- Times in **milliseconds**, taking the minimum over 5 timed runs after one untimed warm-up call (`repeat=5` in `numpy_groupies/benchmarks/generic.py`). Lower is better.
- Times in **milliseconds**, the fastest of five timed executions after one untimed warm-up call, over three complete passes of the table (`--repeat=5 --runs=3` of `numpy_groupies.benchmarks`). Lower is better.
- Software: Python 3.14.7, NumPy 2.5.4, Numba 0.68.0, pandas 3.0.6, numbagg 0.9.6 (Linux, x86_64).

Absolute numbers depend heavily on the machine, Python and library versions, so rely on the ratios between columns rather than on the values. On shared machines single runs wobble - `--runs=3` takes the per-cell minimum of three complete runs. Re-run on your own hardware, from the repository root:
Absolute numbers depend heavily on the machine, Python and library versions, so rely on the ratios between columns rather than on the values. On shared machines single runs wobble, so `--runs=3` measures everything three times and reports the fastest measurement of each cell. Re-run on your own hardware, from the repository root:

```sh
python -m numpy_groupies.benchmarks.generic --pandas --numbagg
python -m numpy_groupies.benchmarks --ufunc --numbagg --pandas --runs=3
```

Without `--pandas` the pandas column is omitted, and the pure-Python implementation only appears with `--purepy`. The optional numbagg column (from the `numbagg` package, included in the dev extras) appears with `--numbagg`. numbagg ships nan-guarded grouped reductions only, so its plain-func numbers run through the nan twins - on the benchmark's nan-free plain data the results coincide.

Columns: `ufunc` is the `ufunc.at`-based implementation (incomplete, for benchmarking only), `numpy` the default NumPy implementation, `numba` the Numba implementation, `numbagg` the numbagg grouped-reduction wrapper, and `pandas` the pandas `groupby` wrapper. `----` means the function is not available in that implementation. The row `arbitrary` is a custom Python callable passed as `func`.
Which columns are measured by default, what the optional ones are and how the output arrives is what `--help` explains. `----` in the table below means the function is not available in that implementation, the row `arbitrary` is a custom Python callable passed as `func`.

## Results

| function | ufunc | numpy | numba | numbagg | pandas |
| ------------ | -----: | ------: | -----: | ------: | ------: |
| sum | 0.866 | 0.924 | 0.434 | 0.485 | 15.891 |
| prod | 0.920 | 0.928 | 0.468 | 0.444 | 15.191 |
| min | 1.730 | 1.687 | 0.611 | 0.815 | 15.018 |
| max | 1.712 | 1.713 | 0.638 | 0.876 | 15.154 |
| len | 0.942 | 0.830 | 0.373 | 0.471 | 14.100 |
| all | 32.574 | 1.805 | 0.823 | 1.567 | 15.725 |
| any | 30.603 | 2.506 | 0.644 | 1.568 | 15.675 |
| anynan | 5.221 | 0.890 | 0.623 | 2.674 | 15.258 |
| allnan | 7.607 | 2.122 | 0.826 | 3.255 | 15.315 |
| mean | ---- | 1.559 | 0.511 | 0.556 | 15.985 |
| median | ---- | 48.409 | 11.493 | ---- | 27.700 |
| trapezoid | ---- | 3.895 | 0.667 | ---- | ---- |
| std | ---- | 3.094 | 0.673 | 0.677 | 16.823 |
| var | ---- | 3.187 | 0.675 | 0.675 | 16.696 |
| first | ---- | 1.765 | 0.362 | 0.313 | 15.390 |
| last | ---- | 0.952 | 0.373 | 0.431 | 15.573 |
| argmax | ---- | 2.671 | 0.606 | 0.888 | 15.260 |
| argmin | ---- | 4.992 | 0.586 | 0.838 | 14.827 |
| nansum | ---- | 4.169 | 0.443 | 1.755 | 22.259 |
| nanprod | ---- | 4.115 | 0.479 | 1.770 | 20.854 |
| nanmin | ---- | 4.875 | 0.977 | 2.011 | 21.029 |
| nanmax | ---- | 4.945 | 1.010 | 2.085 | 21.752 |
| nanlen | ---- | 2.477 | 0.465 | 1.712 | 20.736 |
| nanall | ---- | 4.985 | 1.058 | 3.249 | 21.618 |
| nanany | ---- | 5.664 | 0.634 | 2.659 | 21.525 |
| nanmean | ---- | 4.678 | 1.634 | 1.758 | 22.228 |
| nanmedian | ---- | 42.432 | 10.681 | ---- | 30.505 |
| nantrapezoid | ---- | 6.558 | 1.772 | ---- | ---- |
| nanvar | ---- | 6.255 | 1.860 | 1.921 | 22.432 |
| nanstd | ---- | 6.216 | 1.893 | 1.929 | 22.450 |
| nanfirst | ---- | 5.077 | 0.393 | 0.368 | 22.225 |
| nanlast | ---- | 4.370 | 0.434 | 1.627 | 21.577 |
| nanargmin | ---- | 6.385 | 0.774 | 1.958 | 16.259 |
| nanargmax | ---- | 3.998 | 0.807 | 2.028 | 16.571 |
| cumsum | ---- | 44.838 | 0.583 | ---- | 7.322 |
| cumprod | ---- | ---- | 0.627 | ---- | 13.258 |
| cummax | ---- | ---- | 0.758 | ---- | 13.085 |
| cummin | ---- | ---- | 0.786 | ---- | 12.785 |
| arbitrary | ---- | 144.095 | 42.973 | ---- | 108.594 |
| sort | ---- | 135.146 | 25.992 | ---- | ---- |
| function | numpy | numba | ufunc | numbagg | pandas |
| ------------ | -----------: | -----------: | -----------: | -----------: | -----------: |
| sum | 0.953 | 0.444 | 0.903 | 0.472 | 15.961 |
| prod | 0.886 | 0.456 | 0.909 | 0.443 | 15.274 |
| min | 1.675 | 0.618 | 1.687 | 0.837 | 16.073 |
| max | 1.684 | 0.632 | 1.767 | 0.849 | 16.259 |
| len | 0.845 | 0.382 | 0.929 | 0.488 | 15.132 |
| all | 1.756 | 0.819 | 32.878 | 1.575 | 16.028 |
| any | 2.465 | 0.650 | 30.555 | 1.562 | 16.047 |
| anynan | 0.920 | 0.614 | 5.350 | 2.658 | 15.452 |
| allnan | 2.109 | 0.831 | 7.601 | 3.289 | 15.752 |
| mean | 1.524 | 0.511 | ---- | 0.545 | 16.978 |
| median | 48.770 | 11.517 | ---- | ---- | 28.566 |
| trapezoid | 3.814 | 0.653 | ---- | ---- | ---- |
| std | 3.093 | 0.656 | ---- | 0.693 | 17.020 |
| var | 3.133 | 0.667 | ---- | 0.685 | 16.854 |
| first | 1.790 | 0.293 | ---- | 0.308 | 15.966 |
| last | 0.938 | 0.366 | ---- | 0.425 | 15.951 |
| argmax | 2.625 | 0.624 | ---- | 0.908 | 15.760 |
| argmin | 5.136 | 0.583 | ---- | 0.847 | 15.562 |
| nansum | 4.237 | 0.446 | ---- | 1.778 | 21.705 |
| nanprod | 4.159 | 0.449 | ---- | 1.741 | 21.129 |
| nanmin | 4.844 | 0.936 | ---- | 1.996 | 21.081 |
| nanmax | 4.923 | 1.005 | ---- | 2.083 | 21.219 |
| nanlen | 2.495 | 0.448 | ---- | 1.713 | 20.246 |
| nanall | 4.929 | 1.057 | ---- | 3.270 | 21.970 |
| nanany | 5.698 | 0.642 | ---- | 2.656 | 21.667 |
| nanmean | 4.705 | 1.650 | ---- | 1.797 | 21.271 |
| nanmedian | 42.596 | 10.618 | ---- | ---- | 30.263 |
| nantrapezoid | 6.610 | 1.771 | ---- | ---- | ---- |
| nanvar | 6.111 | 1.860 | ---- | 1.922 | 21.491 |
| nanstd | 6.135 | 1.863 | ---- | 1.934 | 21.645 |
| nanfirst | 4.872 | 0.348 | ---- | 0.317 | 21.347 |
| nanlast | 4.224 | 0.449 | ---- | 1.610 | 21.495 |
| nanargmin | 6.409 | 0.758 | ---- | 1.939 | 16.622 |
| nanargmax | 3.947 | 0.794 | ---- | 2.019 | 16.535 |
| cumsum | 45.878 | 0.583 | ---- | ---- | 7.567 |
| cumprod | ---- | 0.563 | ---- | ---- | 13.840 |
| cummax | ---- | 0.750 | ---- | ---- | 14.388 |
| cummin | ---- | 0.761 | ---- | ---- | 13.083 |
| arbitrary | 142.543 | 43.163 | ---- | ---- | 109.890 |
| sort | 136.157 | 25.961 | ---- | ---- | ---- |

## Reading the table

- **numba vs numpy:** Numba is faster for every function both provide - by 1.4–13× across the reductions (`sum` 2.1×, `argmin` 8.5×, `nanfirst` 13×), by 5.2× on `sort` (26 ms against 135 ms), and by far the most on `cumsum` (about 77×).
- **numba vs numbagg:** par on the light scans (`sum` 0.43 vs 0.49 ms, `nanfirst` 0.39 vs 0.37 ms, and `first` even a sixth behind); numba leads wherever the nan handling branches per value (`anynan` 4.3×, `nansum` 4.0×, `nanlast` 3.8×, `nanargmax` 2.5×), and is the only one of the two with `median`, the cumulatives, `trapezoid` and `sort`.
- **numpy vs pandas:** the NumPy implementation is about 5× (`var`, `std`) to 17× (`sum`, `anynan`, `len`; `prod` 16×) faster than pandas on common reductions. The exceptions are `median`, where pandas (28 ms) beats NumPy (48 ms) but not Numba (11 ms), and the cumulatives, where pandas is six times faster than NumPy (`cumsum` 7.3 against 44.8 ms) and still twelve times slower than Numba (0.58 ms).
- **ufunc:** `ufunc.at` was historically slow; NumPy 1.25 narrowed the gap considerably, but this implementation remains incomplete and `all`/`any` are still very slow.
- **numba vs numpy:** Numba is faster for every function both provide - by 1.5-6.1× across the plain reductions (`sum` 2.1×, `first` 6.1×), by 2.9-14× on the nan ones (`nanmean` 2.9×, `nanfirst` 14×), by 4.2-8.8× on the argreductions (`argmin` 8.8×), by 5.2× on `sort` (26 ms against 136 ms), and by far the most on `cumsum` (about 79×).
- **numba vs numbagg:** par on the light scans (`sum` 0.44 against 0.47 ms, `first` 0.29 against 0.31 ms, while `prod` and `nanfirst` go to numbagg by a few per cent); numba leads wherever the nan handling branches per value (`anynan` 4.3×, `nansum` 4.0×, `nanlen` 3.8×, `nanlast` 3.6×, `nanargmax` 2.5×), and is the only one of the two with `median`, the cumulatives, `trapezoid` and `sort`.
- **numpy vs pandas:** the NumPy implementation is 5.4× (`var`) to 17.9× (`len`) faster on the plain reductions, up to 8.1× on the nan ones (`nanlen`) and 2.6-6.0× on the argreductions. The exceptions are `median` and `nanmedian`, where pandas (28.6 ms and 30.3 ms) beats NumPy (48.8 ms and 42.6 ms) but not Numba (11.5 ms and 10.6 ms), and the cumulatives, where pandas is six times faster than NumPy (`cumsum` 7.6 against 45.9 ms) and still thirteen times slower than Numba (0.58 ms).
- **ufunc:** `ufunc.at` was historically slow; NumPy 1.25 brought [major speed improvements](https://numpy.org/doc/stable/release/1.25.0-notes.html) and on the arithmetic scans the column now stands level with NumPy (`sum` 0.90 against 0.95 ms, `prod`, `min`, `max` and `len` within a few per cent). It is six times behind on `anynan` and twelve to nineteen times on `all` and `any`, and the implementation remains incomplete.
- **Expensive everywhere:** `median`, `sort` and custom callables.

## Sorting across group counts

`sort` emits one value per input item and orders every group on its own, so its cost turns on the group *size* - a few large blocks to order against many small ones. `--sort` measures that function alone over a hundredth, one, ten and a hundred times the group count of the table above, on the same 500,000 values:

```sh
python -m numpy_groupies.benchmarks.generic --sort --runs=3
python -m numpy_groupies.benchmarks --sort --runs=3
```

| groups | values per group | numpy | numba |
| -----: | ---------------: | ------: | -----: |
| 100 | 5000 | 124.895 | 34.421 |
| 1000 | 500 | 141.278 | 26.088 |
| 10000 | 50 | 159.103 | 21.794 |
| 100000 | 5 | 167.581 | 11.088 |
| 100 | 5000 | 126.200 | 34.335 |
| 1000 | 500 | 143.293 | 26.167 |
| 10000 | 50 | 157.599 | 21.341 |
| 100000 | 5 | 167.970 | 12.095 |

NumPy slows down monotonically as the groups multiply. The Numba column dips in the middle: with few large groups the comparison sort of a whole block is what costs, while many tiny groups are ordered by insertion sort in the flat array, without the per-group view that a library sort has to build - which is three times the speed it had before at five values per group. The other implementations have no column here because none of them provides `sort`.
NumPy slows down monotonically as the groups multiply, from 126 to 168 ms. The Numba column falls the whole way down, to 12 ms: with few large groups the comparison sort of a whole block is what costs, while many tiny groups are ordered by insertion sort in the flat array, without the per-group view that a library sort has to build - which is almost three times the speed of the five-thousand-value groups. The other implementations have no column here because none of them provides `sort`.
Loading
Loading