You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 5fec1d9
Browse filesBrowse the repository at this point in the historyBrowse files
Move diagrams into docs/, fix paths and stale claims in the docs
- Point the README images and docs/functions.md at docs/diagrams/.
- docs/benchmarks.md: the timings are the minimum of 5 runs (repeat=5), the
values above 0.8 are zeroed (not those below 0.2), and the pandas and
pure-Python columns need --pandas and --purepy.
- README: the pure-Python implementation imports NumPy, so it is not a
dependency-free fallback; report best-of-five runs rather than a median;
use the ratios the benchmark table actually shows.
- utils.label_contiguous_1d: the docstring warning meant "liable to change",
and drop the README note pointing at it.
- pyproject.toml: BSD-2-Clause, matching LICENSE.txt.
- Add rumdl as markdown linter/formatter (config in [tool.rumdl]) and keep
ruff-format off .md, since it collapses the hand-aligned example output.
Copy file name to clipboardExpand all lines: README.md
+11-11Lines changed: 11 additions & 11 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -23,7 +23,7 @@ npg.aggregate(group_idx, a, func="sum", fill_value=0)
23
23
24
24
Group 2 never occurs in `group_idx`, so its slot is filled with `fill_value`.
25
25
26
-

26
+

pip install "numpy_groupies[fast]"# + Numba for the fastest implementation
65
65
conda install -c conda-forge numpy_groupies
66
66
```
67
67
68
-
NumPy is the only declared dependency. Numba is optional, and the pure-Python implementation needs neither (see [Implementations](#implementations)).
68
+
NumPy is the only declared dependency. Numba is optional and only the fastest implementation needs it; the pure-Python implementation runs without Numba but still imports NumPy (see [Implementations](#implementations)).
69
69
70
70
If you only want one implementation, you can copy a single file (e.g. `aggregate_numpy.py`) into your project: paste the contents of `utils.py` at its top, replacing the `from .utils import (...)` line.
| 4 | 2-D, shape *(d, n)*| 1-D, len *n*| — |*d*-dimensional; `group_idx[:, i]` is the position of `a[i]`|
150
150
| 5 | 2-D | scalar | — | like form 4, scalar broadcast |
151
151
152
-

152
+

153
153
154
154
Output size defaults to `max(group_idx) + 1` per dimension; pass `size=` to fix it. Full parameter descriptions, plus notes on memory layout and performance, are in the [reference](https://github.com/ml31415/numpy-groupies/blob/master/docs/functions.md).
@@ -287,15 +287,15 @@ All implementations share the calling syntax and produce the same results up to
287
287
| -------------- | ------------ | ----- |
288
288
|**numba**| numpy, numba | Fastest. Default if numba is installed. Lacks `sort` and `array`. |
289
289
|**numpy**| numpy | Based on `np.bincount` and indexing tricks. Default without numba. Most complete. |
290
-
|**pure python**|nothing| Standard library only. Very slow; a last resort when NumPy is unavailable. |
290
+
|**pure python**|numpy| Plain Python loops, no Numba. Very slow; useful as a fallback or for porting. |
291
291
| numpy ufunc | numpy | For benchmarking only: built on `ufunc.at` (`np.add.at`, …). Incomplete. |
292
292
| pandas | numpy, pandas | For reference only: wraps `groupby`. Skips NaN even in the plain functions (except `median`, `cumsum`). |
293
293
294
294
> **Gotchas:** with Numba installed, `aggregate(..., func="sort")` and `func="array"` raise `NotImplementedError`, and custom callables must be Numba-compilable (numeric code). For those cases call the NumPy implementation: `npg.aggregate_np(group_idx, a, func="sort")`.
295
295
296
296
## Performance
297
297
298
-
Median time in milliseconds for 500,000 values in 1,000 groups (lower is better), taken from the maintainers' benchmark on an Intel i7-7560U laptop CPU (Linux, Python 3.14, NumPy 2.5, Numba 0.67, pandas 3.0):
298
+
Best of five timed runs, in milliseconds, for 500,000 values in 1,000 groups (lower is better), taken from the maintainers' benchmark on an Intel i7-7560U laptop CPU (Linux, Python 3.14, NumPy 2.5, Numba 0.67, pandas 3.0):
299
299
300
300
| function | numpy | numba | pandas |
301
301
| -------- | -----: | -----: | -----: |
@@ -308,19 +308,19 @@ Median time in milliseconds for 500,000 values in 1,000 groups (lower is better)
308
308
|`cumsum`| 53.28 | 1.15 | 13.43 |
309
309
| custom callable | 161.54 | 50.94 | 131.86 |
310
310
311
-
In short: the NumPy implementation is about 4–13× faster than pandas on common reductions, and Numba adds another 1.5–4.5× on top — and about 46× for `cumsum`. `median` and custom callables are the expensive cases everywhere. Absolute numbers depend on your machine, so treat the ratios as the takeaway.
311
+
In short: the NumPy implementation is about 4–12× faster than pandas on common reductions, and Numba adds another 1.5–5× on top — and about 46× for `cumsum`. `median` and custom callables are the expensive cases everywhere. Absolute numbers depend on your machine, so treat the ratios as the takeaway.
312
312
313
-
The complete table and the benchmark setup are in [docs/benchmarks.md](https://github.com/ml31415/numpy-groupies/blob/master/docs/benchmarks.md). To run it yourself, from the repository root:
313
+
The complete table and the benchmark setup are in [docs/benchmarks.md](https://github.com/ml31415/numpy-groupies/blob/master/docs/benchmarks.md). To run it yourself, from the repository root (add `--pandas` for the pandas column, `--purepy` for the pure-Python one):
314
314
315
-
```
315
+
```sh
316
316
python -m numpy_groupies.benchmarks.generic
317
317
```
318
318
319
319
The `numpy` and `numba` columns above are not the whole story: NumPy 1.25 brought major [speed improvements to ufuncs](https://numpy.org/doc/stable/release/1.25.0-notes.html), which narrowed the gap between the NumPy and Numba implementations considerably. The authors hope that `ufunc.at` or an equivalent in NumPy or SciPy will eventually become fast enough to make this package redundant.
Copy file name to clipboardExpand all lines: docs/benchmarks.md
+9-7Lines changed: 9 additions & 7 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -5,16 +5,18 @@ Complete benchmark results for `aggregate`. A short summary is in the [README](.
5
5
## Setup
6
6
7
7
- 500,000 indices, uniformly drawn from `[0, 1000)`.
8
-
- The values of `a` are uniform on `[0, 1)`, and everything below`0.2` is then set to `0`, so that there are falsy values for the boolean functions.
9
-
- For the `nan…` functions another 20% of the values are set to `nan`, leaving the rest on the interval `[0.2, 0.8)`.
10
-
- Times in **milliseconds**, taking the minimum over 7 runs after discarding a warm-up run. Lower is better.
8
+
- The values of `a` are uniform on `[0, 1)`, and everything above`0.8` is then set to `0`, so that there are falsy values for the boolean functions.
9
+
- For the `nan…` functions, of the remaining values those below `0.2`are set to `nan` (about 20%), leaving the rest on the interval `[0.2, 0.8]`.
10
+
- Times in **milliseconds**, taking the minimum over 5 timed runs after one untimed warm-up call (`repeat=5` in `numpy_groupies/benchmarks/generic.py`). Lower is better.
11
11
- Machine: Intel i7-7560U at 2.40 GHz, Linux (x86_64), Python 3.14.2, NumPy 2.5.3, Numba 0.67.0, pandas 3.0.6.
12
12
13
13
Absolute numbers depend heavily on the machine, Python and library versions, so rely on the ratios between columns rather than on the values. Re-run on your own hardware, from the repository root:
Without `--pandas` the pandas column is omitted, and the pure-Python implementation only appears with `--purepy`.
18
20
19
21
Columns: `ufunc` is the `ufunc.at`-based implementation (incomplete, for benchmarking only), `numpy` the default NumPy implementation, `numba` the Numba implementation, and `pandas` the pandas `groupby` wrapper. `----` means the function is not available in that implementation. The row `arbitrary` is a custom Python callable passed as `func`.
20
22
@@ -65,7 +67,7 @@ Columns: `ufunc` is the `ufunc.at`-based implementation (incomplete, for benchma
65
67
66
68
## Reading the table
67
69
68
-
-**numba vs numpy:** Numba is faster for every function both provide, typically by 1.5–4.5× on reductions. The largest gap is `cumsum` (about 46×).
69
-
-**numpy vs pandas:** the NumPy implementation is about 4× (`std`, `var`) to 13× (`sum`, `prod`, `len`) faster than pandas on common reductions. The exception is `median`, where pandas (24 ms) beats NumPy (53 ms) but not Numba (12 ms).
70
+
-**numba vs numpy:** Numba is faster for every function both provide, typically by 1.5–5× on reductions (up to 5.1× for `argmin`). The largest gap is `cumsum` (about 46×).
71
+
-**numpy vs pandas:** the NumPy implementation is about 4× (`std`, `var`) to 12× (`len`; `sum` 11×, `prod` 9×) faster than pandas on common reductions. The exception is `median`, where pandas (24 ms) beats NumPy (53 ms) but not Numba (12 ms).
70
72
-**ufunc:**`ufunc.at` was historically slow; NumPy 1.25 narrowed the gap considerably, but this implementation remains incomplete and `all`/`any` are still very slow.
71
73
-**Expensive everywhere:**`median`, `sort` and custom callables.
0 commit comments