Fix unbounded memory usage in pickle environment

[vpoughonEZ]

In my project we are using incremental builds and building often in parallel with sphinx-build -j auto. We use sphinx-needs a lot and we've noticed in this configuration huge memory usage during compilation, sometimes so high (30+ GB) that it crashes my laptop. After some debugging, it turns out it comes from the size of the pickled environment that's used to store information between incremental builds.

This file is typically found in your build directory as environment.pickle. The problem is that it grows after every incremental parallel build. This can be seen by rebuilding our project with something like:

grep -Rl "\.\. need" | xargs touch
sphinx-build -j auto [...]

When doing this, we notice that the file size of environment.pickle increases every time. Of course when a build is run, this file is unpickled in RAM by each sphinx-build process. Looking more closely at the cause, it comes from env.needs_all_docs["all"]. This list is correctly appended to only if the element is not already in it in add_doc() :

sphinx-needs/sphinx_needs/utils.py

Lines 575 to 576 in 6d7740f

	if docname not in env.needs_all_docs["all"]:
	env.needs_all_docs["all"].append(docname)

However, merge_data() appends both lists without checking for duplicates. This can result in unbounded increase in the size of the list in the build environment:

sphinx-needs/sphinx_needs/needs.py

Lines 730 to 731 in 6d7740f

	if other_key in objects:
	objects[other_key] += other_value

This can also be seen by running this simple script in your build directory:

import pickle

with open("path/to/your/environment.pickle", "rb") as f:
    env = pickle.load(f)

print("{} = {}".format('len(env.needs_all_docs["all"])', len(env.needs_all_docs["all"])))

This PR fixes this issue by merging lists without duplicates in merge_data().

[danwos]

Thanks for the PR and the great explanation 👍

Will merge it as soon as the CI is green.

[danwos]

Ok, the CI probs are not your fault.
A new flake8 version introduced some new checks.
Will solve them soon.