Meilisearch auto-batching uses a lot of space on /tmp, "out of space" error

[groentebroer]

Describe the bug
When adding new sets of documents to the index through the API, the indexer stops indexing after a certain amount of time. The /tasks
API endpoints shows "No space left on device (os error 28)". The partition where the index should be placed has plenty of space available. The /tmp has limited amount of space, because that was not needed (until now maybe?)...

Adding --disable-auto-batching makes 0.29.0 work like 0.28.1, which still indexes all items, so the exhausive use of /tmp seems to be related to the auto-batch feature which is enabled by default in 0.29.0.

Is there any use of auto-batching when adding new tasks with over 400 items at a time?
When adding/updating single items, the batching makes sense, but in this case the documents are already offered in "batches"...

To Reproduce

1. Create a clean install and data folder of 0.29.0.
2. I have a dataset with about 150.000 articles.
3. The dataset is divided in JSON files from 400 articles to 25.000 articles per file.
4. Have a relatively small /tmp partition (4GB at my case).
5. Add all dataset JSON files through the API to make Meilisearch index those.
6. Wait until the CPU stops working, without completing any of the tasks created by the addition of the JSON files.

Expected behavior
Successful indexing, dispite the "small" (4GB) /tmp size.

Meilisearch version:
{
"commitSha": "fa315352da8100e43da0f345712fb43cdf0271cb",
"commitDate": "2022-09-13T16:07:06Z",
"pkgVersion": "0.29.0"
}

Additional context
Linux, Intel

[curquiza]

Hello @groentebroer 👋

I think this discussion can help you: #2765

Let me know if I'm wrong!

[groentebroer]

Thanks @curquiza for your response. Looks like we are running into the same sort of problem indeed.
For me the trigger is that it behaves much more different compared to v0.28.1, because of the auto-batching feature.
I know how to overcome the issue and can create a larger /tmp partition, (after doing so I am indeed very happy with the indexing speed improvements, which are related!).

But maybe:

1. We can include in the documentation that the auto-batch feature might consume more /tmp space during the indexing process?
2. Specify "a best guess" of /tmp space, based on a dataset size of some sort in the documentation? (Difficult, to do a proper estimate, for sure.)
3. Need to verify a minimum amount of /tmp space related to the "size" of the data in the tasks to be sure there is enough space? Fallback would be to disable the auto-batch for that set of tasks?
4. Use the same space for "tmp" data, as the "index" space, so people do not have to have multiple "big" partitions
5. Throw an error on the console, about the diskspace (now it is only shown in the tasks list).
6. Do nothting and get "issues" like this in github ;-).

[curquiza]

Hello @groentebroer sorry for the delay.

This is something I will transfer soon to the docs team.

Regarding the other solutions, it would be really awesome if you open a discussion on the product side, with your favorite solution (depending on your needs I suppose). Knowing your use case and issues would help us choose the best solutions, and other users could join the conversation.

EDIT: just noticed there is already an issue opened on the docs side! meilisearch/documentation#1903

[curquiza]

Hello again!

I updated my previous point, but people following this issue do not receive notification for this: the doc team already opened an issue for this. meilisearch/documentation#1903

I discussed this with the team, and FYI we already tried to avoid using tmp for our temporary files, but this drastically impacted the performance during indexation for people using Docker. Knowing more than 75% of our users are on Docker, we couldn't change this.
Meaning we cannot change the default behavior. However, for any product improvement, for customized behavior for example, everyone is more than welcome to open a discussion on the product side: this repository is to help us gather feedback based on concrete use case our users have. Let us know about your needs!

Thanks in advance! I close this issue since it's more a product discussion now

I