Add a built-in way to compare sequences ignoring order

[Delgan]

What's the problem this feature will solve?

A convenient way to assert that two sequences contains the exact same elements regardless of their order.

Describe the solution you'd like

assert [1, 2, 3] == pytest.unordered([2, 1, 3])

For example, I could use it when order of elements are not part of the specification of a function:

# Per "Path.iterdir()" documentation: "The children are yielded in arbitrary order"
assert list(tmp_path.iterdir()) == pytest.unordered([tmp_path / "some_file.txt", tmp_path / "another_file.txt"])

Alternative Solutions

Sorting the sequence manually, using unittest.TestCase.assertCountEqual, using pytest-unordered plugin.

Additional context

This issue duplicates:

• Provide an implementation of UnitTest.assertCountEqual #5548
• assert helper to compare list contents, ignoring order #7899

These tickets were closed with the suggestion to first create an external plugin and testing it during a few years. Following this, @utapyngo created the pytest-unordered plugin. It has been several years. I would like to bring back the suggestion of a built-in pytest.unordered() helper similar to pytest.approx() because I think pytest users would greatly benefit from it.

I don't think it opens the room for too many assertion helpers: with approx() and unordered() all main cases are covered. For comparison, here is the list of helpers provided by the Catch2 C++ testing library. Python provides many built-in functions to ease assertion, but unordered() is missing and is non-trivial.

What do you think?

[RonnyPfannschmidt]

When order doesn't matter, comparing the result of sorted is orders of magnitude simpler to comprehend.

If unique values should be folded, comparing sets is order of magnitude simpler to comprehend.

So far i haven't seen a setup where unordered is necessary and more comprehensive.

I'd like to see one before adding such a non trivial tool.

[Delgan]

I think the most convincing argument is that the sorted() workaround requires elements to be sortable.
This is not the case by default for custom classes, dict or dataclass, for example.
Converting to a set requires elements to be hashable and may silently ignore issues due to duplicated objects.

Maybe we can take a look at Github search results for from pytest_unordered import unordered and assertCountEqual to get an overview of the real-world use cases.

[kalekundert]

If it helps to have an example of unordered in a real test, here's one I wrote just the other day:
stepwise/tests/test_reaction.py. This is exactly one of the cases that @Delgan described. I'm trying to compare two lists-of-dicts that each represent the edges of a graph data structure. Because dicts are neither sortable nor hashable, neither sorted nor set can work here.

I use unordered in almost every package I write tests for, and I think it's very useful. I'd love to see it added to pytest proper.

[utapyngo]

I wrote pytest_unordered especially for testing APIs that may return data in arbitrary order. It is tedious and often impossible to sort or convert a part of a complex json to a set.

In the project I have been working on for more than 8 years, we started using the initial implementation of pytest_unordered in 2019, before it was even published on PyPI, and are still using it today.

[utapyngo]

@RonnyPfannschmidt, do you want me to create a pull request into pytest?

[RonnyPfannschmidt]

There is currently an attempt to get an idea of matcher combination, so that we have a library with matchers for not just unordered, but other qualities as well, currently I don't have any bandwith to work on it however

[utapyngo]

There is currently an attempt to get an idea of matcher combination, so that we have a library with matchers for not just unordered, but other qualities as well, currently I don't have any bandwith to work on it however

I haven't found an issue mentioning matchers. Where can I read more about this idea?

[RonnyPfannschmidt]

A few weeks ago i had a call with @asottile about creating a DSL for assertion /check writing that combines tools like re_asserrt, unordered, approx and a a few more for structural matching

Unfortunately i only got around making a repo for it, now I'm on parental leave

[utapyngo]

A few weeks ago i had a call with @asottile about creating a DSL for assertion /check writing that combines tools like re_asserrt, unordered, approx and a a few more for structural matching

Unfortunately i only got around making a repo for it, now I'm on parental leave

It sounds interesting, I would like to collaborate.

[Pheric]

Hello!

Will this issue be moving forward?

[utapyngo]

May we at least add pytest-unordered under pytest-dev?

[The-Compiler]

Another source of inspiration for assertion helper functions in general: jwodder/anys: Matchers for pytest