Merge #1864

1864: V0.29: updates to auto-batching r=guimachiavelli a=maryamsulemani97

closes #1838

Co-authored-by: Maryam Sulemani <[email protected]>
Co-authored-by: Maryam <[email protected]>

Author: bors[bot]

.vuepress/config.js

@@ -271,21 +271,6 @@ module.exports = {
             },
           ],
         },
-        {
-          title: '🧪 Experimental',
-          collapsable: false,
-          path: '/learn/experimental/overview.html',
-          children: [
-            {
-              title: 'Overview',
-              path: '/learn/experimental/overview',
-            },
-            {
-              title: 'Auto-batching',
-              path: '/learn/experimental/auto-batching',
-            },
-          ],
-        },
         {
           title: '👐 Contributing',
           path: '/learn/contributing/overview.html',

.vuepress/public/_redirects

@@ -196,3 +196,6 @@
 
 # Rename indexation to indexing
 /learn/advanced/indexation.html                     /learn/advanced/indexing.html
+
+# Remove autobatching
+/learn/experimental/auto-batching.html              /learn/core_concepts/documents.html

learn/advanced/indexing.md

@@ -28,16 +28,14 @@ Multi-threading is unfortunately not possible in machines with only one processo
 
 ## Improving indexing performance
 
-If you encounter performance issues during the indexing we recommend trying the following points:
+If you encounter performance issues during indexing, we recommend trying the following:
 
 - Make sure you are using the latest [stable version of Meilisearch](https://github.com/meilisearch/meilisearch/releases). New releases often include performance improvements that can significantly increase indexing speed.
 
-- indexing is a memory-intensive and multi-threaded operation. This means **the more memory and processor cores available, the faster Meilisearch will index new documents**. When trying to improve indexing speed, using a machine with more processor cores is more effective than increasing RAM.
+- Indexing is a memory-intensive and multi-threaded operation. This means **the more memory and processor cores available, the faster Meilisearch will index new documents**. When trying to improve indexing speed, using a machine with more processor cores is more effective than increasing RAM.
 
 - **Bigger HTTP payloads are processed more quickly than smaller payloads**. For example, adding the same 100,000 documents in two batches of 50,000 documents will be quicker than adding them in four batches of 25,000 documents. By default, Meilisearch sets the maximum payload size to 100MB, but [you can change this value if necessary](/learn/configuration/instance_options.md#payload-limit-size). That said, **the bigger the payload is, the higher the memory consumption will be**. An instance may crash if it requires more RAM than is currently available in a machine.
 
-  - If you want to speed up indexing but don't wish to batch documents manually, consider giving our [experimental auto-batcher](/learn/experimental/auto-batching.md) a try.
-
 - **Meilisearch should not be your main database**. The more documents you add, the longer will indexing and search take, so you should only index documents you want to retrieve when searching.
 
 - By default, all document fields are searchable. We strongly recommend changing this by [updating the `searchableAttributes` list](/reference/api/settings.md#update-searchable-attributes) so it only contains fields you want to search in. The fewer fields Meilisearch needs to index, the faster is the indexing process.

learn/configuration/instance_options.md

@@ -106,6 +106,19 @@ If no master key is provided in a `development` environment, all routes will be
 
 [Learn more about Meilisearch's use of security keys.](/learn/security/master_api_keys.md)
 
+### Disable auto-batching
+
+::: warning
+🚩 This is a CLI flag and does not take any values. Assigning a value will throw an error. 🚩
+:::
+
+**Environment variable**: `MEILI_DISABLE_AUTO_BATCHING`
+**CLI option**: `--disable-auto-batching`
+
+Deactivates auto-batching when provided.
+
+[Learn more about auto-batching.](/learn/core_concepts/documents.md#auto-batching)
+
 ### Disable analytics
 
 ::: warning

learn/core_concepts/documents.md

@@ -124,3 +124,19 @@ Since CSV does not support arrays or nested objects, `cast` cannot be converted
 ::: note
 If you don't specify the data type for an attribute, it will default to `:string`.
 :::
+
+### Auto-batching
+
+Auto-batching combines consecutive document addition requests into a single batch and processes them together. This significantly speeds up the indexing process.
+
+Meilisearch batches document addition requests when they:
+
+- Target the same index
+- Have the same update method (i.e., [POST](/reference/api/documents.md#add-or-replace-documents) or [PUT](/reference/api/documents.md#add-or-update-documents))
+- Are immediately consecutive
+
+Tasks within the same batch share the same values for `startedAt`, `finishedAt`, and `duration`.
+
+If a task fails due to an invalid document, it will be removed from the batch. The rest of the batch will still process normally. If an [`internal`](/reference/api/overview.md#errors) error occurs, the whole batch will fail and all tasks within it will share the same `error` object.
+
+You can deactivate auto-batching using the `--disable-auto-batching` command-line flag or the `MEILI_DISABLE_AUTO_BATCHING` environment variable. This is useful in cases where you want to avoid any potential bugs in the feature or reduce visibility latency. When auto-batching is disabled, the whole queue takes longer to process, but each individual task will be processed earlier (until a certain number of processed tasks).

learn/experimental/auto-batching.md

@@ -181,7 +181,9 @@ Meilisearch stores data in the form of discrete records, called [documents](/lea
 Meilisearch currently only accepts data in JSON, NDJSON, and CSV formats. You can read more about this in our [documents guide](/learn/core_concepts/documents.md#dataset-format).
 :::
 
-The previous command added documents from `movies.json` to a new index called `movies`. After adding documents, you should receive a response like this:
+The previous command added documents from `movies.json` to a new index called `movies`.
+
+By default, Meilisearch combines consecutive document requests into a single batch and processes them together. This process is called auto-batching, and it significantly speeds up indexing. After adding documents, you should receive a response like this:
 
 ```json
 {