Skip to content

Ignoring order for deeply nested lists of dictionaries. #393

Closed as not planned

Description

@explorigin

Is your feature request related to a problem? Please describe.
I have a dataset that looks like:

{
	"_links": {
		"self": {
			"href": "/accounts/036aa770-b6fe-11ed-b489-b625a1e46b37/messages"
		}
	},
	"_embedded": {
		"messages": [
			{
				"id": "c589614c-b6cb-11ed-9083-7cb91a971123",
				"channel": "email",
				"contact": {},
				"status": "delivered",
			},
			{
				"id": "c5892790-b6cb-11ed-8a7c-a8944afb6fba",
				"channel": "email",
				"contact": {},
				"status": "delivered"
			}
		]
	}
}

I'd like group_by to work with deep lists such as this.

Describe the solution you'd like
Calling like DeepDiff(old, new, ignore_order=True, group_by={"root['_embedded']['messages']": "id"}) would be able to apply different grouping to different areas in the dataset.

Describe alternatives you've considered
I'll just have to manipulate my data myself prior to comparison. The disadvantage of this is that I don't get a diff that reflects the level of nesting of the list.

Activity

  1. seperman commented on May 1, 2023

    @seperman
    Member

    Hi @explorigin
    This is going to be tricky. It seems like you want to diff at the "root['_embedded']['messages']" level. Then group by id.
    So what happens to the rest of the items in your nested dictionary that don't fall under "root['_embedded']['messages']" ?

  2. explorigin commented on May 1, 2023

    @explorigin
    Author

    I'd like to convert the list at "root['_embedded']['messages']" to an object keyed by id and then diff the whole data tree.

  3. seperman commented on May 10, 2023

    @seperman
    Member

    @explorigin Ah, I see. Ok, that makes more sense than what I mentioned above. I'm curious if this is a very common use case or not.

  4. explorigin commented on May 30, 2023

    @explorigin
    Author

    It's common for me. :-) The example is a common REST API reply where you have a result and some metadata in one package.

  5. seperman commented on Nov 19, 2023

    @seperman
    Member

    We now allow group_by to take 2 elements. In your case, it needs to take 3.
    I think we should make group_by take arbitrary number of elements.

  6. seperman commented on Nov 16, 2026

    @seperman
    Member

    An update on this one, and thanks to everyone for the patience.

    Rather than extending group_by to accept a {path: key} mapping, we're adding a dedicated parameter for key-based matching of nested lists, tracked in #664. group_by will stay as it is (root level), so it remains simple.

    With the new parameter (working name), the original example would look like:

    DeepDiff(old, new, iterable_match_keys={"root['_embedded']['messages']": "id"})

    Records in that list are paired by id, so inserts, removals and reorders no longer cascade into false changes. Globs work too, e.g. root['orders'][*]['items'].

    One difference from what was asked here: the list is not converted into an object keyed by id. Results keep real list paths such as root['_embedded']['messages'][1]['status']. That's what lets Delta keep working and lets the new parameter combine with existing options like exclude_paths.

    Comments on the design are welcome in #664.

  7. seperman commented on Nov 16, 2026

    @seperman
    Member

    Closing in favour of #664, which adds key-based matching for nested lists as a dedicated parameter while keeping group_by as it is. Follow #664 for progress.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions