Repository navigation
Add iterable_match_keys for key-based list matching per path (#664), fix #665 - #666
Merged
Merged
Conversation
DeepDiff(t1, t2, iterable_match_keys={path: key_spec}) pairs the items
of the selected lists by key instead of by position. Paths use the
exclude_paths glob syntax; a key spec is a key name, a tuple of key
names, or a callable. Matching runs inside _get_matching_pairs in
O(n + m), so results keep real list indexes and Delta round-trips.
- Duplicate keys pair by occurrence; items without a key pair in order.
- 1 and 1.0 are the same key; booleans and strings never match numbers.
- Records with different keys at the same index stay added + removed
instead of being merged into values_changed.
- With ignore_order, keyed lists are still paired by key, without
reporting moves.
- Lists inside a moved record skip difflib, and Delta no longer swaps
paths for items of opcode-diffed lists, so mixed diffs round-trip.
- Subtree multiprocessing falls back to serial, since workers can't
see absolute paths.
Also fixes iterable_compare_func dropping an added item whose content
equals an already matched item: matched t2 items are now tracked by
index, not content hash.
Closes #664
Closes #665
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## dev #666 +/- ##
==========================================
+ Coverage 95.09% 95.21% +0.11%
==========================================
Files 18 19 +1
Lines 4915 5054 +139
==========================================
+ Hits 4674 4812 +138
- Misses 241 242 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #664. Closes #665.
What
New
DeepDiffparameteriterable_match_keys: maps paths (exact or glob, same syntax asexclude_paths) to a key spec, and pairs the items of those lists by key instead of by position.Matching happens inside
_get_matching_pairs: t2 is indexed by key and t1 is walked once, so it's O(n+m). Lists stay lists, so results keep real indexes andDeltaround-trips.Semantics, per the issue
ValueError. A path withoutroot("users") works like it does inexclude_paths.KeyError/IndexError/TypeErrororCannotCompare, or the key is unhashable. They are paired in order among themselves.1and1.0are the same key.Truevs1and"1"vs1are not. This applies to each part of a composite key and to keys a callable returns.iterable_item_added+iterable_item_removed. A newDiffLevel.paired_by_keyflag keepsmutual_add_removes_to_become_value_changesfrom merging them.iterable_compare_func. The compare function still applies to other lists.exclude_paths/include_pathsglobs,ignore_order/ignore_order_func,custom_operators,view='tree'.Decisions worth reviewing
ignore_order=True: keyed lists are still paired by key, but a record at a different index is not reported asiterable_item_movedand paths aren't swapped to t2. That matches otherignore_orderlists and what Delta expects underignore_order. For the same reason,_iterable_compare_func_was_usedis not set in that case._iterable_compare_func_was_usedis set, Delta previously applied everyvalues_changedatnew_path, which is wrong for items of opcode-diffed lists. Delta now skips that swap when the item's parent list has_iterable_opcodes.iterable_compare_func. Their opcodes would otherwise be stored under a path that mixes t1 and t2 indexes. Lists inside records that didn't move still use difflib._subtree_parallel_safe()returns False wheniterable_match_keysis set, because workers only see subtree-relative paths. Distance workers get the parameter through_parameters; an unpicklable spec, such as a lambda, falls back to serial. A slow test checks that the parallel result equals the serial one.#665
_get_matching_pairsnow tracks matched t2 items by index instead of by DeepHash, which also removes the DeepHash calls from that loop. There are regression tests for every row of the table in #665, plus tree view, Delta round-trip andignore_order.Not changed (pre-existing, same with
iterable_compare_func)t2 - Delta(diff, bidirectional=True)andDelta(flat_dict_list=...)don't reproduce reorders even for a simple compare-func reorder. The acceptance criteriont1 + Delta(diff) == t2holds.Unable to get the item at ...for items added to a nested list inside a moved record, and the result is still correct. This is identical withiterable_compare_func.Tests and docs
tests/test_diff_iterable_match_keys.py: 62 tests covering the acceptance criteria.--runslow.match_keys.py,delta.pyand the test file. The changed Python lines are within 120 chars.iterable_match_keys.rstwith a REST-payload example. Its examples pass under doctest and the Sphinx build has no warnings.diff_doc.rst, a pointer from theiterable_compare_funcdocs,multi_processing.md, andCHANGELOG.md.