Skip to content

iterable_compare_func does not seem to work on nested lists #307

Open

Description

@jvacek

Please checkout the F.A.Q page before creating a bug ticket to make sure it is not already addressed.

Describe the bug
I cannot get the compare function to work properly on lists that are not 1st-level.

To Reproduce
Define the two following objects. They are identical except that:

  • the version attribute went 0.0.0->0.0.1
  • in the second object has a number field removed in the numberfield_set list, and a new one added. UUID4s are used to track which one should be correlated with which, much like ids. in the example in the docs.
self_json= b'{"stringfield_set":[],"numberfield_set":[{"uuid":"fa0c87e8-01f5-43f5-8e63-24886f72ffd0","name":"field_1","address":1,"documentation":"","first_bit_offset":2,"size_in_bits":3,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""},{"uuid":"332e5cfe-886d-41cc-b4d9-0fc1296b3ea0","name":"field_2","address":5,"documentation":"","first_bit_offset":5,"size_in_bits":6,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""}],"enumfield_set":[],"version":"0.0.0","version_date":null,"version_commit":null,"ros_link":null,"documentation":"","definition":42,"information":[],"users":[]}'
other_json=b'{"stringfield_set":[],"numberfield_set":[{"uuid":"fa0c87e8-01f5-43f5-8e63-24886f72ffd0","name":"field_1","address":1,"documentation":"","first_bit_offset":2,"size_in_bits":3,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""},{"uuid":"056429c5-812b-4f49-9aae-1b52bd40aacd","name":"field_3","address":7,"documentation":"","first_bit_offset":8,"size_in_bits":9,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""}],"enumfield_set":[],"version":"0.0.1","version_date":null,"version_commit":null,"ros_link":null,"documentation":"","definition":42,"information":[],"users":[]}'

make the iterable compare func

def field_diff_function(x, y, level=None):
    try:
        return x["uuid"] == y["uuid"]

    except Exception:
        raise CannotCompare() from None

run the compare:

        diff_native = DeepDiff(
            json.loads(self_json),
            json.loads(other_json),
            iterable_compare_func=field_diff_function,
            ignore_order=True,
        )

the following diff is the result:

{
  "values_changed": {
    "root['numberfield_set'][1]['uuid']": {
      "new_value": "056429c5-812b-4f49-9aae-1b52bd40aacd",
      "old_value": "332e5cfe-886d-41cc-b4d9-0fc1296b3ea0"
    },
    "root['numberfield_set'][1]['name']": {
      "new_value": "field_3",
      "old_value": "field_2"
    },
    "root['numberfield_set'][1]['address']": {
      "new_value": 7,
      "old_value": 5
    },
    "root['numberfield_set'][1]['first_bit_offset']": {
      "new_value": 8,
      "old_value": 5
    },
    "root['numberfield_set'][1]['size_in_bits']": {
      "new_value": 9,
      "old_value": 6
    },
    "root['version']": {
      "new_value": "0.0.1",
      "old_value": "0.0.0"
    }
  }
}

Expected behavior
As the fields should be matched using the uuid attribute, it should show that one has been added, and the other has been removed, and not that they changed.

OS, DeepDiff version and Python version (please complete the following information):

  • Python Version: 3.9.10
  • deepdiff version: 5.8.0

Additional context
I'm definitely not ruling out there's a problem somewhere between the chair and the keyboard

Activity

  1. jvacek commented on Apr 29, 2022

    @jvacek
    ContributorAuthor

    Upon further inspection, slapping a print() statement inside the compare function shows that the function is actually never run.

  2. LizardBlizzard commented on Jun 25, 2022

    @LizardBlizzard

    The iterable_compare_func function is not called when you set ignore_order parameter to True from what I know

  3. havardthom commented on Jan 17, 2023

    @havardthom
    Contributor

    The iterable_compare_func function is not called when you set ignore_order parameter to True from what I know

    Is this a bug or intended behaviour? It is not mentioned in the docs atleast.

    I'm doing diffs on nested dicts with list of dicts which have id keys where iterable_compare_func helps comparing correct dicts to eachother, however the performance is not very good. Setting ignore_order=True helps performance alot but wrong dicts are compared sometimes since iterable_compare_func is not used.

  4. Omar-Abdul-Azeez commented on Mar 3, 2023

    @Omar-Abdul-Azeez

    I'm having the same issue and was about to create an issue...
    I also done some more testing and debugging...

    I think there are actually 2 bugs in this:

    1. This is exclusive to ignore_order=True. There's a len > 1 check that prevents the fuction from being called.
      only if there are two or more different items in both lists does the function actually get called. ie. If there's 1 addition and 1 removal it doesn't call the function to check whether they are pairs or not.
      I'm almost certain this is incorrect behaviour. I edited it locally and got this fixed (you can mimic by just introducing another element in both lists with differences). Instead of getting key 'id' changed from a to b, I'm now getting dict {'id': a} changed to dict {'id': b} ie. instead of considering k, v changes, the entire dicts are considered changes.
      https://github.com/seperman/deepdiff/blob/8ab1c8dbf19bb87177c10029a518051d6622532a/deepdiff/diff.py#L1094

    2. Irrelevent of ignore_order. The result of the custom compare function is ignored for nested dicts
      I've tested this on a list of dicts, inside each of these dicts is a key whose value is also a list of dicts. each dict has an id key. The example below sets ignore_order=True but it works for both cases, just make sure the order is correct.

    [
    {'id': '1010', 'g': [ {'id': '2020'} ] },
    {'id': '73', 'g': [ {'id': '101'}, {'id': '6790'} ] }
    ]
    
    [
    {'id': '73', 'g': [ {'id': '202'},  {'id': '15294'} ] },
    {'id': '1012', 'g': [ {'id': '2020'} ] }
    ]
    

    I use this as compare function. Except the prints, taken straight from the docs.

    def compare_func(x, y, level=None):
        print(x)
        print(y)
        print('in')
        if not isinstance(x, dict) or not isinstance(y, dict):
            print('not dict')
            raise CannotCompare
        if x['id'] == y['id']:
            print('match')
            return True
        print('not match')
        return False
    

    I found out that it gets called 8 times. 4 times comparing the 2 top level dicts to each other. 4 times comparing the 2 nested dicts to each other (the ones in the dict with the same id)
    the prints were as expected for both the top level and nested dicts. but the results are different... 'id': '1010' and 'id': '1012' are not pairs and appeared in iterable_item_added and iterable_item_removed accordingly. but nested dicts were paired with eachother and entered in values_changed DESPITE the function returning False for all pairing checks between them...

  5. Omar-Abdul-Azeez commented on Mar 3, 2023

    @Omar-Abdul-Azeez

    I did some more debugging and I found out that the dicts are reported as iterable_item_removed and not as values_changed
    2023-03-04-03-38-43
    which led me to think the problem is in the this line
    https://github.com/seperman/deepdiff/blob/8ab1c8dbf19bb87177c10029a518051d6622532a/deepdiff/diff.py#L309
    up to this point all differences found are under removed or added but once this function gets called it changes to changed for some reason... taking a quick look inside that function there's a suspicious mutual_add_removes_to_become_value_changes() call that makes me believe it doesn't respect the custom comparing function... I took a quick peek inside it and it seems to do exactly that. I think I need a break...
    https://github.com/seperman/deepdiff/blob/8ab1c8dbf19bb87177c10029a518051d6622532a/deepdiff/diff.py#L1551

  6. seperman commented on Nov 19, 2023

    @seperman
    Member

    Hi @jvacek @Omar-Abdul-Azeez @havardthom @LizardBlizzard
    Somehow this ticket was lost among other tickets I was paying attention to.
    @Omar-Abdul-Azeez Thank you for diving into it already. I am going to take a look soon.

  7. pascalstindt commented on Jan 9, 2024

    @pascalstindt

    I think I'm running into the same issue. I'm using a custom iterable_compare_func function to match on id.
    All the results BUT ONE are correctly reported either as iterable_item_added or iterable_item_removed. The single incorrect one is reported as values_changed with clearly a non matching id.

    @seperman and @Omar-Abdul-Azeez have you been able to fix this? Thanks a lot.

  8. amyasnikov commented on Jul 5, 2024

    @amyasnikov

    Running into the same issue even on the latest (7.0.1) version.
    @seperman it would be very nice if you could get back to this issue

  9. cdiproperzio1 commented on Sep 12, 2024

    @cdiproperzio1

    Having the same issue. Thanks!

  10. bongi23 commented on Feb 10, 2025

    @bongi23

    Similar issue here. I am comparing dicts with a field that contains list of dicts. Dicts in these lists have an "_id" field that I would like to use to compare items.

    Example

    obj_1 = {
        "name": "Andrea",
        "addresses": [
            "_id": 1
           "AddressName": "name1"
        ]
    }
    
    obj_2 = {
        "name": "Andrea",
        "addresses": [
            "_id": 1
           "AddressName": "name2"
        ]
    }
    

    I would like to get a value_change event for root[addresses][0][AddressName] but the comparator that returns True for a match on the _id filed does not get called.

    I have both ignore_order and ignore_repetition set to True

  11. dtorres-sf commented on Apr 18, 2025

    @dtorres-sf
    Contributor

    This issue can now be closed. This is fixed in deepdiff 8+, I believe with this PR:

    #473

    Tested the following:

            self_json = b'{"stringfield_set":[],"numberfield_set":[{"uuid":"fa0c87e8-01f5-43f5-8e63-24886f72ffd0","name":"field_1","address":1,"documentation":"","first_bit_offset":2,"size_in_bits":3,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""},{"uuid":"332e5cfe-886d-41cc-b4d9-0fc1296b3ea0","name":"field_2","address":5,"documentation":"","first_bit_offset":5,"size_in_bits":6,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""}],"enumfield_set":[],"version":"0.0.0","version_date":null,"version_commit":null,"ros_link":null,"documentation":"","definition":42,"information":[],"users":[]}'
            other_json = b'{"stringfield_set":[],"numberfield_set":[{"uuid":"fa0c87e8-01f5-43f5-8e63-24886f72ffd0","name":"field_1","address":1,"documentation":"","first_bit_offset":2,"size_in_bits":3,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""},{"uuid":"056429c5-812b-4f49-9aae-1b52bd40aacd","name":"field_3","address":7,"documentation":"","first_bit_offset":8,"size_in_bits":9,"is_signed":false,"is_lsb_left":false,"value_multiply_by":1.0,"value_divide_by":1.0,"value_increase_by":0.0,"value_unit":""}],"enumfield_set":[],"version":"0.0.1","version_date":null,"version_commit":null,"ros_link":null,"documentation":"","definition":42,"information":[],"users":[]}'
    
            def field_diff_function(x, y, level=None):
                try:
                    return x["uuid"] == y["uuid"]
    
                except Exception:
                    raise CannotCompare() from None
            diff_native = DeepDiff(
                json.loads(self_json),
                json.loads(other_json),
                iterable_compare_func=field_diff_function,
                report_repetition=True,
                ignore_order=True,
            )
            print(json.dumps(diff_native, indent=2))
    

    Got this result:

    {
      "values_changed": {
        "root['version']": {
          "new_value": "0.0.1",
          "old_value": "0.0.0"
        }
      },
      "iterable_item_added": {
        "root['numberfield_set'][1]": {
          "uuid": "056429c5-812b-4f49-9aae-1b52bd40aacd",
          "name": "field_3",
          "address": 7,
          "documentation": "",
          "first_bit_offset": 8,
          "size_in_bits": 9,
          "is_signed": false,
          "is_lsb_left": false,
          "value_multiply_by": 1.0,
          "value_divide_by": 1.0,
          "value_increase_by": 0.0,
          "value_unit": ""
        }
      },
      "iterable_item_removed": {
        "root['numberfield_set'][1]": {
          "uuid": "332e5cfe-886d-41cc-b4d9-0fc1296b3ea0",
          "name": "field_2",
          "address": 5,
          "documentation": "",
          "first_bit_offset": 5,
          "size_in_bits": 6,
          "is_signed": false,
          "is_lsb_left": false,
          "value_multiply_by": 1.0,
          "value_divide_by": 1.0,
          "value_increase_by": 0.0,
          "value_unit": ""
        }
      }
    }
    
  12. Omar-Abdul-Azeez commented on Jun 8, 2025

    @Omar-Abdul-Azeez

    while the example in the original post may be fixed (I haven't tested), the one in my comment isn't. reposting for ease.

    Irrelevent of ignore_order. The result of the custom compare function is ignored for nested dicts
    I've tested this on a list of dicts, inside each of these dicts is a key whose value is also a list of dicts. each dict has an id key. The example below sets ignore_order=True but it works for both cases, just make sure the order is correct.

    [
    {'id': '1010', 'g': [ {'id': '2020'} ] },
    {'id': '73', 'g': [ {'id': '101'}, {'id': '6790'} ] }
    ]
    [
    {'id': '73', 'g': [ {'id': '202'},  {'id': '15294'} ] },
    {'id': '1012', 'g': [ {'id': '2020'} ] }
    ]
    

    I use this as compare function. Except the prints, taken straight from the docs.

    def compare_func(x, y, level=None):
        print(x)
        print(y)
        print('in')
        if not isinstance(x, dict) or not isinstance(y, dict):
            print('not dict')
            raise CannotCompare
        if x['id'] == y['id']:
            print('match')
            return True
        print('not match')
        return False
    

    actually checking this example now yields no prints(???) and the result being value changes for root[0] (id:1010 with id:73) and for root[1] (id:73 with id:1012) which is clearly wrong.

    Image

    and it is still due to this function not respecting the custom comparison.
    https://github.com/seperman/deepdiff/blob/ecc823a3fdb9c094b7392ca40f859eab8cc8572a/deepdiff/diff.py#L1754

    before the function is called there are additions for id: 73, 1012 and removals for id: 1010, and 73. I'm not sure if 73 should be paired beforehand or this function pairs them and finds out that the change is in a nested dict inside rather than the top level so I can't tell if it will even find out the difference correctly or not.

  13. seperman commented on Nov 16, 2026

    @seperman
    Member

    Thanks everyone for the detailed reports, and sorry this sat for so long. Earlier PRs (#333, #454, #473) fixed parts of this, but the latest example in this thread still fails on dev (5e7b5ef) and on the released 9.1.0. I traced it to two separate causes.

    Reproduction

    from deepdiff import DeepDiff
    from deepdiff.helper import CannotCompare
    
    def compare_func(x, y, level=None):
        if not isinstance(x, dict) or not isinstance(y, dict):
            raise CannotCompare
        return x["id"] == y["id"]
    
    t1 = [{"id": "1010", "g": [{"id": "2020"}]}, {"id": "73", "g": [{"id": "101"}, {"id": "6790"}]}]
    t2 = [{"id": "73", "g": [{"id": "202"}, {"id": "15294"}]}, {"id": "1012", "g": [{"id": "2020"}]}]

    Without ignore_order: the top level is now correct (73 is matched, 1010 removed, 1012 added). But the nested items are reported as changed even though compare_func returned False for every nested pair:

    "root[0]['g'][0]": {'old_value': {'id': '101'}, 'new_value': {'id': '202'}}   # values_changed
    "root[0]['g'][1]": {'old_value': {'id': '6790'}, 'new_value': {'id': '15294'}}

    With ignore_order=True: compare_func is never called (0 calls), and records with different ids are paired by position:

    'root[0]': {'old_value': {'id': '1010', ...}, 'new_value': {'id': '73', ...}}    # values_changed
    'root[1]': {'old_value': {'id': '73', ...},   'new_value': {'id': '1012', ...}}

    Cause 1: removed and added at the same index are merged into values_changed

    _get_matching_pairs handles the nested list correctly: compare_func rejects every pair, so each item is reported as removed or added. But when results are built, DeepDiff._get_view_results (diff.py:2243) runs TreeResult.mutual_add_removes_to_become_value_changes() (model.py:59) on the whole tree whenever report_repetition is off. That step turns any removed and added pair at the same path into a values_changed, overriding the compare function's decision.

    Cause 2: with ignore_order, pairing is skipped when most items changed

    _diff_iterable_with_deephash (diff.py:1738) only pairs items when the changed fraction, (added + removed) / (len(t1) + len(t2) + 1), is at most cutoff_intersection_for_pairs (default 0.7). In this example no item is identical, so the fraction is 4/5 = 0.8. Pairing is skipped, and compare_func with it, so everything is added or removed. Cause 1 then merges those by position into the wrong values_changed.

    Confirmation: passing cutoff_intersection_for_pairs=1 makes compare_func get called and the top level comes out right. The nested items are then still merged by cause 1.

    Workaround until this is fixed: with ignore_order=True, pass cutoff_intersection_for_pairs=1 when using iterable_compare_func. The nested values_changed from cause 1 remains.

    Candidate fixes (tried on a scratch copy of dev, not merged)

    1. Cause 2: don't apply the cutoff_intersection_for_pairs shortcut when iterable_compare_func is set. The compare function is an explicit instruction for how to pair items, and here the shortcut exists only to avoid the cost of pairing. Trade-off: large, mostly changed lists now pay O(n·m) compare calls.
    2. Cause 1: don't let the merge step override a compare-function decision. The blunt version, skipping the merge whenever iterable_compare_func is set, fixes this example in both modes, and the existing test suite passes. But it's too broad. With ignore_order=True it also changes lists the compare function doesn't handle (it raises CannotCompare), e.g. ["a", "b"] → ["x", "y"] becomes added/removed instead of values_changed. While applying a Delta built from the result, it also logs Unable to get the item at root[0]['g'][0], although the round-trip still succeeds. The right fix is narrower: mark the levels that the compare function explicitly rejected, and have the merge skip only those.

    Two smaller things found while tracing:

    • In the ignore_order path, the compare function is called as compare_func(t2_item, t1_item) with no level (distance.py:116). The in-order path calls compare_func(t1_item, t2_item, level). A compare function that is not symmetric, or that relies on level, behaves differently between the two modes.
    • Paths under a matched parent differ between modes. Without ignore_order, nested paths use the parent's t2 index (root[0]['g']); with it, the t1 index (root[1]['g']).

    Related: #665 (another iterable_compare_func matching bug: an unmatched t2 item is dropped when it equals a matched one), #664 (planned key-based list matching, which builds on this code), and #290, #414, #489 (ignore_order together with iterable_compare_func).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions