Skip to content

Commit 5fec1d9

Browse files
committed
Move diagrams into docs/, fix paths and stale claims in the docs
- Point the README images and docs/functions.md at docs/diagrams/. - docs/benchmarks.md: the timings are the minimum of 5 runs (repeat=5), the values above 0.8 are zeroed (not those below 0.2), and the pandas and pure-Python columns need --pandas and --purepy. - README: the pure-Python implementation imports NumPy, so it is not a dependency-free fallback; report best-of-five runs rather than a median; use the ratios the benchmark table actually shows. - utils.label_contiguous_1d: the docstring warning meant "liable to change", and drop the README note pointing at it. - pyproject.toml: BSD-2-Clause, matching LICENSE.txt. - Add rumdl as markdown linter/formatter (config in [tool.rumdl]) and keep ruff-format off .md, since it collapses the hand-aligned example output.
1 parent 837a3c8 commit 5fec1d9

13 files changed

Lines changed: 39 additions & 21 deletions

‎.pre-commit-config.yaml‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,9 @@ repos:
2121
- id: ruff
2222
args: [--fix]
2323
- id: ruff-format
24+
# The docs keep hand-aligned example output (`# array([...])`) and small matrices,
25+
# which ruff-format collapses; the markdown layout is owned by rumdl instead.
26+
exclude: \.md$
2427

2528
- repo: https://github.com/astral-sh/uv-pre-commit
2629
rev: 0.12.23
@@ -31,3 +34,9 @@ repos:
3134
rev: v2.4.3
3235
hooks:
3336
- id: codespell
37+
38+
- repo: https://github.com/rvben/rumdl-pre-commit
39+
rev: v0.2.78
40+
hooks:
41+
- id: rumdl-check
42+
args: [--fix]

‎README.md‎

Lines changed: 11 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -23,7 +23,7 @@ npg.aggregate(group_idx, a, func="sum", fill_value=0)
2323

2424
Group 2 never occurs in `group_idx`, so its slot is filled with `fill_value`.
2525

26-
![Diagram of aggregate: values are collected by group label and reduced](https://github.com/ml31415/numpy-groupies/raw/master/diagrams/aggregate.png)
26+
![Diagram of aggregate: values are collected by group label and reduced](https://github.com/ml31415/numpy-groupies/raw/master/docs/diagrams/aggregate.png)
2727

2828
**Contents:**
2929
[Why numpy-groupies?](#why-numpy-groupies) ·
@@ -59,13 +59,13 @@ npg.aggregate(inverse, values) # array([5., 2., 3.]) -> groups "a", "b",
5959

6060
## Installation
6161

62-
```
62+
```sh
6363
pip install numpy_groupies # NumPy implementation
6464
pip install "numpy_groupies[fast]" # + Numba for the fastest implementation
6565
conda install -c conda-forge numpy_groupies
6666
```
6767

68-
NumPy is the only declared dependency. Numba is optional, and the pure-Python implementation needs neither (see [Implementations](#implementations)).
68+
NumPy is the only declared dependency. Numba is optional and only the fastest implementation needs it; the pure-Python implementation runs without Numba but still imports NumPy (see [Implementations](#implementations)).
6969

7070
If you only want one implementation, you can copy a single file (e.g. `aggregate_numpy.py`) into your project: paste the contents of `utils.py` at its top, replacing the `from .utils import (...)` line.
7171

@@ -149,7 +149,7 @@ pd.Series(a).groupby(group_idx).sum().reindex(range(group_idx.max() + 1), fill_v
149149
| 4 | 2-D, shape *(d, n)* | 1-D, len *n* | — | *d*-dimensional; `group_idx[:, i]` is the position of `a[i]` |
150150
| 5 | 2-D | scalar | — | like form 4, scalar broadcast |
151151

152-
![Diagram of the five input forms](https://github.com/ml31415/numpy-groupies/raw/master/diagrams/aggregate_dims.png)
152+
![Diagram of the five input forms](https://github.com/ml31415/numpy-groupies/raw/master/docs/diagrams/aggregate_dims.png)
153153

154154
Output size defaults to `max(group_idx) + 1` per dimension; pass `size=` to fix it. Full parameter descriptions, plus notes on memory layout and performance, are in the [reference](https://github.com/ml31415/numpy-groupies/blob/master/docs/functions.md).
155155

@@ -241,7 +241,7 @@ npg.label_contiguous_1d(np.array([0, 3, 3, 0, 0, 5, 5, 5, 1, 1, 0, 2]))
241241
# array([0, 1, 1, 0, 0, 2, 2, 2, 3, 3, 0, 4])
242242
```
243243

244-
The output is a ready-made `group_idx` for "aggregate over each run of …" questions. The docstring flags the API of this function as not final.
244+
The output is a ready-made `group_idx` for "aggregate over each run of …" questions.
245245

246246
### `relabel_groups_unique` / `relabel_groups_masked`
247247

@@ -287,15 +287,15 @@ All implementations share the calling syntax and produce the same results up to
287287
| -------------- | ------------ | ----- |
288288
| **numba** | numpy, numba | Fastest. Default if numba is installed. Lacks `sort` and `array`. |
289289
| **numpy** | numpy | Based on `np.bincount` and indexing tricks. Default without numba. Most complete. |
290-
| **pure python** | nothing | Standard library only. Very slow; a last resort when NumPy is unavailable. |
290+
| **pure python** | numpy | Plain Python loops, no Numba. Very slow; useful as a fallback or for porting. |
291291
| numpy ufunc | numpy | For benchmarking only: built on `ufunc.at` (`np.add.at`, …). Incomplete. |
292292
| pandas | numpy, pandas | For reference only: wraps `groupby`. Skips NaN even in the plain functions (except `median`, `cumsum`). |
293293

294294
> **Gotchas:** with Numba installed, `aggregate(..., func="sort")` and `func="array"` raise `NotImplementedError`, and custom callables must be Numba-compilable (numeric code). For those cases call the NumPy implementation: `npg.aggregate_np(group_idx, a, func="sort")`.
295295
296296
## Performance
297297

298-
Median time in milliseconds for 500,000 values in 1,000 groups (lower is better), taken from the maintainers' benchmark on an Intel i7-7560U laptop CPU (Linux, Python 3.14, NumPy 2.5, Numba 0.67, pandas 3.0):
298+
Best of five timed runs, in milliseconds, for 500,000 values in 1,000 groups (lower is better), taken from the maintainers' benchmark on an Intel i7-7560U laptop CPU (Linux, Python 3.14, NumPy 2.5, Numba 0.67, pandas 3.0):
299299

300300
| function | numpy | numba | pandas |
301301
| -------- | -----: | -----: | -----: |
@@ -308,19 +308,19 @@ Median time in milliseconds for 500,000 values in 1,000 groups (lower is better)
308308
| `cumsum` | 53.28 | 1.15 | 13.43 |
309309
| custom callable | 161.54 | 50.94 | 131.86 |
310310

311-
In short: the NumPy implementation is about 4–13× faster than pandas on common reductions, and Numba adds another 1.5–4.5× on top — and about 46× for `cumsum`. `median` and custom callables are the expensive cases everywhere. Absolute numbers depend on your machine, so treat the ratios as the takeaway.
311+
In short: the NumPy implementation is about 4–12× faster than pandas on common reductions, and Numba adds another 1.5–5× on top — and about 46× for `cumsum`. `median` and custom callables are the expensive cases everywhere. Absolute numbers depend on your machine, so treat the ratios as the takeaway.
312312

313-
The complete table and the benchmark setup are in [docs/benchmarks.md](https://github.com/ml31415/numpy-groupies/blob/master/docs/benchmarks.md). To run it yourself, from the repository root:
313+
The complete table and the benchmark setup are in [docs/benchmarks.md](https://github.com/ml31415/numpy-groupies/blob/master/docs/benchmarks.md). To run it yourself, from the repository root (add `--pandas` for the pandas column, `--purepy` for the pure-Python one):
314314

315-
```
315+
```sh
316316
python -m numpy_groupies.benchmarks.generic
317317
```
318318

319319
The `numpy` and `numba` columns above are not the whole story: NumPy 1.25 brought major [speed improvements to ufuncs](https://numpy.org/doc/stable/release/1.25.0-notes.html), which narrowed the gap between the NumPy and Numba implementations considerably. The authors hope that `ufunc.at` or an equivalent in NumPy or SciPy will eventually become fast enough to make this package redundant.
320320

321321
## Development
322322

323-
```
323+
```sh
324324
git clone https://github.com/ml31415/numpy-groupies
325325
cd numpy-groupies
326326
pip install -e ".[dev]" # pytest, numba, pandas

‎docs/benchmarks.md‎

Lines changed: 9 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -5,16 +5,18 @@ Complete benchmark results for `aggregate`. A short summary is in the [README](.
55
## Setup
66

77
- 500,000 indices, uniformly drawn from `[0, 1000)`.
8-
- The values of `a` are uniform on `[0, 1)`, and everything below `0.2` is then set to `0`, so that there are falsy values for the boolean functions.
9-
- For the `nan…` functions another 20% of the values are set to `nan`, leaving the rest on the interval `[0.2, 0.8)`.
10-
- Times in **milliseconds**, taking the minimum over 7 runs after discarding a warm-up run. Lower is better.
8+
- The values of `a` are uniform on `[0, 1)`, and everything above `0.8` is then set to `0`, so that there are falsy values for the boolean functions.
9+
- For the `nan…` functions, of the remaining values those below `0.2` are set to `nan` (about 20%), leaving the rest on the interval `[0.2, 0.8]`.
10+
- Times in **milliseconds**, taking the minimum over 5 timed runs after one untimed warm-up call (`repeat=5` in `numpy_groupies/benchmarks/generic.py`). Lower is better.
1111
- Machine: Intel i7-7560U at 2.40 GHz, Linux (x86_64), Python 3.14.2, NumPy 2.5.3, Numba 0.67.0, pandas 3.0.6.
1212

1313
Absolute numbers depend heavily on the machine, Python and library versions, so rely on the ratios between columns rather than on the values. Re-run on your own hardware, from the repository root:
1414

15+
```sh
16+
python -m numpy_groupies.benchmarks.generic --pandas
1517
```
16-
python -m numpy_groupies.benchmarks.generic
17-
```
18+
19+
Without `--pandas` the pandas column is omitted, and the pure-Python implementation only appears with `--purepy`.
1820

1921
Columns: `ufunc` is the `ufunc.at`-based implementation (incomplete, for benchmarking only), `numpy` the default NumPy implementation, `numba` the Numba implementation, and `pandas` the pandas `groupby` wrapper. `----` means the function is not available in that implementation. The row `arbitrary` is a custom Python callable passed as `func`.
2022

@@ -65,7 +67,7 @@ Columns: `ufunc` is the `ufunc.at`-based implementation (incomplete, for benchma
6567

6668
## Reading the table
6769

68-
- **numba vs numpy:** Numba is faster for every function both provide, typically by 1.5–4.5× on reductions. The largest gap is `cumsum` (about 46×).
69-
- **numpy vs pandas:** the NumPy implementation is about 4× (`std`, `var`) to 13× (`sum`, `prod`, `len`) faster than pandas on common reductions. The exception is `median`, where pandas (24 ms) beats NumPy (53 ms) but not Numba (12 ms).
70+
- **numba vs numpy:** Numba is faster for every function both provide, typically by 1.5–5× on reductions (up to 5.1× for `argmin`). The largest gap is `cumsum` (about 46×).
71+
- **numpy vs pandas:** the NumPy implementation is about 4× (`std`, `var`) to 12× (`len`; `sum` 11×, `prod` 9×) faster than pandas on common reductions. The exception is `median`, where pandas (24 ms) beats NumPy (53 ms) but not Numba (12 ms).
7072
- **ufunc:** `ufunc.at` was historically slow; NumPy 1.25 narrowed the gap considerably, but this implementation remains incomplete and `all`/`any` are still very slow.
7173
- **Expensive everywhere:** `median`, `sort` and custom callables.

0 commit comments

Comments
 (0)