Skip to content

Add iterable_match_keys for key-based list matching per path (#664), fix #665 - #666

Merged
seperman merged 1 commit into
devfrom
deepdiff-664
Nov 16, 2026
Merged

seperman merged 1 commit into
devfrom
deepdiff-664

Conversation

@seperman

Copy link
Copy Markdown
Member

Closes #664. Closes #665.

What

New DeepDiff parameter iterable_match_keys: maps paths (exact or glob, same syntax as exclude_paths) to a key spec, and pairs the items of those lists by key instead of by position.

DeepDiff(t1, t2, iterable_match_keys={
    "root['users']": "id",                                        # one key
    "root['orders'][*]['items']": ("sku", "warehouse"),           # composite key
    "root['events']": lambda item: (item["product"]["id"], item["ts"]),  # nested / computed
})

Matching happens inside _get_matching_pairs: t2 is indexed by key and t1 is walked once, so it's O(n+m). Lists stay lists, so results keep real indexes and Delta round-trips.

Semantics, per the issue

  • An exact path beats globs. Two globs matching the same list raise ValueError. A path without root ("users") works like it does in exclude_paths.
  • Duplicate keys pair by occurrence; extra items are reported as added or removed.
  • Items have no key when the lookup raises KeyError/IndexError/TypeError or CannotCompare, or the key is unhashable. They are paired in order among themselves.
  • 1 and 1.0 are the same key. True vs 1 and "1" vs 1 are not. This applies to each part of a composite key and to keys a callable returns.
  • Records with different keys at the same index stay iterable_item_added + iterable_item_removed. A new DiffLevel.paired_by_key flag keeps mutual_add_removes_to_become_value_changes from merging them.
  • For the lists it selects, it takes precedence over iterable_compare_func. The compare function still applies to other lists.
  • Composes with exclude_paths/include_paths globs, ignore_order/ignore_order_func, custom_operators, view='tree'.

Decisions worth reviewing

  1. ignore_order=True: keyed lists are still paired by key, but a record at a different index is not reported as iterable_item_moved and paths aren't swapped to t2. That matches other ignore_order lists and what Delta expects under ignore_order. For the same reason, _iterable_compare_func_was_used is not set in that case.
  2. Delta with mixed lists: non-keyed scalar lists still use difflib. When _iterable_compare_func_was_used is set, Delta previously applied every values_changed at new_path, which is wrong for items of opcode-diffed lists. Delta now skips that swap when the item's parent list has _iterable_opcodes.
  3. Lists inside a moved record skip difflib and are diffed pair by pair, as with iterable_compare_func. Their opcodes would otherwise be stored under a path that mixes t1 and t2 indexes. Lists inside records that didn't move still use difflib.
  4. Multiprocessing: _subtree_parallel_safe() returns False when iterable_match_keys is set, because workers only see subtree-relative paths. Distance workers get the parameter through _parameters; an unpicklable spec, such as a lambda, falls back to serial. A slow test checks that the parallel result equals the serial one.

#665

_get_matching_pairs now tracks matched t2 items by index instead of by DeepHash, which also removes the DeepHash calls from that loop. There are regression tests for every row of the table in #665, plus tree view, Delta round-trip and ignore_order.

Not changed (pre-existing, same with iterable_compare_func)

  • t2 - Delta(diff, bidirectional=True) and Delta(flat_dict_list=...) don't reproduce reorders even for a simple compare-func reorder. The acceptance criterion t1 + Delta(diff) == t2 holds.
  • Delta logs Unable to get the item at ... for items added to a nested list inside a moved record, and the result is still correct. This is identical with iterable_compare_func.

Tests and docs

  • tests/test_diff_iterable_match_keys.py: 62 tests covering the acceptance criteria.
  • Full suite: 1371 passed with --runslow.
  • pyright is clean on match_keys.py, delta.py and the test file. The changed Python lines are within 120 chars.
  • New docs page iterable_match_keys.rst with a REST-payload example. Its examples pass under doctest and the Sphinx build has no warnings.
  • Also updated: the parameter entry in diff_doc.rst, a pointer from the iterable_compare_func docs, multi_processing.md, and CHANGELOG.md.

DeepDiff(t1, t2, iterable_match_keys={path: key_spec}) pairs the items
of the selected lists by key instead of by position. Paths use the
exclude_paths glob syntax; a key spec is a key name, a tuple of key
names, or a callable. Matching runs inside _get_matching_pairs in
O(n + m), so results keep real list indexes and Delta round-trips.

- Duplicate keys pair by occurrence; items without a key pair in order.
- 1 and 1.0 are the same key; booleans and strings never match numbers.
- Records with different keys at the same index stay added + removed
  instead of being merged into values_changed.
- With ignore_order, keyed lists are still paired by key, without
  reporting moves.
- Lists inside a moved record skip difflib, and Delta no longer swaps
  paths for items of opcode-diffed lists, so mixed diffs round-trip.
- Subtree multiprocessing falls back to serial, since workers can't
  see absolute paths.

Also fixes iterable_compare_func dropping an added item whose content
equals an already matched item: matched t2 items are now tracked by
index, not content hash.

Closes #664
Closes #665
@codecov

codecov Bot commented Nov 16, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.33775% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 95.21%. Comparing base (5e7b5ef) to head (79cec36).

Files with missing lines Patch % Lines
deepdiff/diff.py 97.14% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##              dev     #666      +/-   ##
==========================================
+ Coverage   95.09%   95.21%   +0.11%     
==========================================
  Files          18       19       +1     
  Lines        4915     5054     +139     
==========================================
+ Hits         4674     4812     +138     
- Misses        241      242       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@seperman
seperman merged commit 9f713b3 into dev Nov 16, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant