From d69973aba8e3c84c6632f6144360404224d65419 Mon Sep 17 00:00:00 2001 From: Paula Date: Mon, 18 Dec 2023 14:50:12 +1100 Subject: [PATCH 1/2] compression of requests and responses --- .../version-3.12/whats-new-in-3-12.md | 79 ++++++++++++++++--- 1 file changed, 70 insertions(+), 9 deletions(-) diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md index 9258bbae15..d019cf2515 100644 --- a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md @@ -301,15 +301,58 @@ the queue might grow and eventually overflow. You can configure the upper bound of the queue with this option. If the queue is full, log entries are written synchronously until the queue has space again. -## Client tools - -### arangodump - -_arangodump_ now supports a `--ignore-collection` startup option that you can -specify multiple times to exclude the specified collections from a dump. - -It cannot be used together with the existing `--collection` option for specifying -collections to include. +### Transparent compression of requests and responses between ArangoDB server and client tools + +The following options have been added to all [client tools](../../components/tools/_index.md) +and can be used to enable transparent compression of the data that is sent +between the client tools and the ArangoDB server: +- `--compress-transfer` +- `--compress-request-threshold` + +If the `--compress-transfer` option is set to `true`, the client tools add an +extra `Accept-Encoding:deflate` HTTP header to all requests made to the server. +This allows the server to compress its responses before sending them back to the +client tools. The clients also transparently compress their own requests to the +server if the size of the request body (in bytes) is at least the value of the +`--compress-request-threshold` option. The default value is `0`, which disables +the compression of the request bodies in the client tools. To opt in to sending +compressed data, the option must be set to a value greater than `0`. +The request body compression is performed using the deflate compression algorithm. +The client tools also add a `Content-Encoding: deflate` header to the request +when the request body was compressed. + +The following option has been added to the ArangoDB server: +- `--http.compress-response-threshold` +- `--http.handle-content-encoding-for-unauthenticated-requests` + +The value of the `--http.compress-response-threshold` option specifies the threshold value (in bytes) from which on +response bodies are sent out compressed by the server. The default value is `0`, +which disables sending out compressed response bodies. To enable compression, +the option should be set to a value greater than `0`. The selected value should +be large enough to justify the compression overhead. +Regardless of the value of this option, response body compression only happens +when the client signals that it expects a compressed response body by sending +an `Accept-Encoding: gzip` or `Accept-Encoding: deflate` header with its request. +If that header is missing, no response compression is performed by the server. + +If the `--http.handle-content-encoding-for-unauthenticated-requests` +option is set to `true`, the ArangoDB server will automatically +uncompress incoming HTTP requests with `Content-Encodings: gzip` and +`Content-Encoding: deflate` even if the request is not authenticated. +If the option is set to `false`, any unauthenticated request that has a +`Content-Encoding` header set is rejected. This is the default setting. + +{{< info >}} +As compression uses CPU cycles, it should be activated only when the network +communication between the server and clients is slow and there is enough CPU +capacity left for the extra compression/decompression work. + +Furthermore, requests and responses should only be compressed when they exceed a +certain minimum size, i.e. 250 bytes. + +Request and response compression is only supported for HTTP/1.1 and HTTP/2 +responses, and not when using the VST protocol. +{{< /info >}} ## Miscellaneous changes @@ -437,6 +480,14 @@ The following metric as been added: ### arangodump +### `--ignore-collection` startup option + +_arangodump_ now supports a `--ignore-collection` startup option that you can +specify multiple times to exclude the specified collections from a dump. + +It cannot be used together with the existing `--collection` option for specifying +collections to include. + #### Improved dump performance and size From version 3.12 onward, _arangodump_ has extended parallelization capabilities @@ -512,5 +563,15 @@ _arangodump_ operations on the server: dump thread was blocked because it honored the server-side memory limit for dumps. +### Transparent compression of requests and responses + +The following options have been added to all [client tools](../../components/tools/_index.md) +and can be used to enable transparent compression of the data that is sent +between the client tools and the ArangoDB server: +- `--compress-transfer` +- `--compress-request-threshold` + +For more information, see the [dedicated feature section](#transparent-compression-of-requests-and-responses-between-arangodb-server-and-client-tools). + ## Internal changes From 42ae9668f1a53002bc0e4805c54f37987131a0e2 Mon Sep 17 00:00:00 2001 From: Simran Spiller Date: Wed, 20 Dec 2023 13:38:02 +0100 Subject: [PATCH 2/2] Review --- .../tools/arangorestore/examples.md | 11 ++ .../version-3.12/whats-new-in-3-12.md | 121 +++++++++--------- 2 files changed, 72 insertions(+), 60 deletions(-) diff --git a/site/content/3.12/components/tools/arangorestore/examples.md b/site/content/3.12/components/tools/arangorestore/examples.md index 0169401a35..005a4ca627 100644 --- a/site/content/3.12/components/tools/arangorestore/examples.md +++ b/site/content/3.12/components/tools/arangorestore/examples.md @@ -191,6 +191,17 @@ also restored or already present on the server. See [_arangodump_](../arangodump/examples.md#encryption) for details. +## Compression + +You can optionally let *arangorestore* compress the data for the network transfer +with the `--compress-transfer` startup option. This can reduce the traffic and +thus save time. Set the `--compress-request-threshold` startup option to define +the minimum size for request bodies (in bytes) at which compression is applied. + +``` +arangorestore --input-directory "dump" --compress-transfer --compress-request-threshold 250 +``` + ## Reloading Data into a different Collection _arangorestore_ restores documents and edges with the exact same `_key`, diff --git a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md index d019cf2515..e35c25c102 100644 --- a/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md +++ b/site/content/3.12/release-notes/version-3.12/whats-new-in-3-12.md @@ -193,6 +193,63 @@ The previously fixed limit of 128 MiB for [Stream Transactions](../../develop/tr can now be configured with the new `--transaction.streaming-max-transaction-size` startup option. The default value remains 128 MiB. +### Transparent compression of requests and responses between ArangoDB servers and client tools + +The following startup options have been added to all +[client tools](../../components/tools/_index.md) (except the ArangoDB Starter) +and can be used to enable transparent compression of the data that is sent +between a client tool and an ArangoDB server: + +- `--compress-transfer` +- `--compress-request-threshold` + +If the `--compress-transfer` option is set to `true`, the client tool adds an +extra `Accept-Encoding: deflate` HTTP header to all requests made to the server. +This allows the server to compress its responses before sending them back to the +client tool. + +The client also transparently compresses its own requests to the server if the +size of the request body (in bytes) is at least the value of the +`--compress-request-threshold` startup option. The default value is `0`, which +disables the compression of the request bodies in the client tool. To opt in to +sending compressed data, set the option to a value greater than `0`. +The client tool adds a `Content-Encoding: deflate` HTTP header to the request +if the request body is compressed using the deflate compression algorithm. + +The following options have been added to the ArangoDB server: + +- `--http.compress-response-threshold` +- `--http.handle-content-encoding-for-unauthenticated-requests` + +The value of the `--http.compress-response-threshold` startup option specifies +the threshold value (in bytes) from which on response bodies are sent out +compressed by the server. The default value is `0`, which disables sending out +compressed response bodies. To enable compression, the option should be set to a +value greater than `0`. The selected value should be large enough to justify the +compression overhead. Regardless of the value of this option, the client has to +signal that it expects a compressed response body by sending an +`Accept-Encoding: gzip` or `Accept-Encoding: deflate` HTTP header with its request. +If that header is missing, no response compression is performed by the server. + +If the `--http.handle-content-encoding-for-unauthenticated-requests` +startup option is set to `true`, the ArangoDB server automatically decompresses +incoming HTTP requests with `Content-Encodings: gzip` or +`Content-Encoding: deflate` HTTP header even if the request is not authenticated. +If the option is set to `false`, any unauthenticated request that has a +`Content-Encoding` header set is rejected. This is the default setting. + +{{< info >}} +As compression uses CPU cycles, it should be activated only when the network +communication between the server and clients is slow and there is enough CPU +capacity left for the extra compression/decompression work. + +Furthermore, requests and responses should only be compressed when they exceed a +certain minimum size, e.g. 250 bytes. + +Request and response compression is only supported for responses that use the +HTTP/1.1 or HTTP/2 protocol, and not when using the VelocyStream (VST) protocol. +{{< /info >}} + ### LZ4 compression for values in the in-memory edge cache Introduced in: v3.11.2 @@ -301,59 +358,6 @@ the queue might grow and eventually overflow. You can configure the upper bound of the queue with this option. If the queue is full, log entries are written synchronously until the queue has space again. -### Transparent compression of requests and responses between ArangoDB server and client tools - -The following options have been added to all [client tools](../../components/tools/_index.md) -and can be used to enable transparent compression of the data that is sent -between the client tools and the ArangoDB server: -- `--compress-transfer` -- `--compress-request-threshold` - -If the `--compress-transfer` option is set to `true`, the client tools add an -extra `Accept-Encoding:deflate` HTTP header to all requests made to the server. -This allows the server to compress its responses before sending them back to the -client tools. The clients also transparently compress their own requests to the -server if the size of the request body (in bytes) is at least the value of the -`--compress-request-threshold` option. The default value is `0`, which disables -the compression of the request bodies in the client tools. To opt in to sending -compressed data, the option must be set to a value greater than `0`. -The request body compression is performed using the deflate compression algorithm. -The client tools also add a `Content-Encoding: deflate` header to the request -when the request body was compressed. - -The following option has been added to the ArangoDB server: -- `--http.compress-response-threshold` -- `--http.handle-content-encoding-for-unauthenticated-requests` - -The value of the `--http.compress-response-threshold` option specifies the threshold value (in bytes) from which on -response bodies are sent out compressed by the server. The default value is `0`, -which disables sending out compressed response bodies. To enable compression, -the option should be set to a value greater than `0`. The selected value should -be large enough to justify the compression overhead. -Regardless of the value of this option, response body compression only happens -when the client signals that it expects a compressed response body by sending -an `Accept-Encoding: gzip` or `Accept-Encoding: deflate` header with its request. -If that header is missing, no response compression is performed by the server. - -If the `--http.handle-content-encoding-for-unauthenticated-requests` -option is set to `true`, the ArangoDB server will automatically -uncompress incoming HTTP requests with `Content-Encodings: gzip` and -`Content-Encoding: deflate` even if the request is not authenticated. -If the option is set to `false`, any unauthenticated request that has a -`Content-Encoding` header set is rejected. This is the default setting. - -{{< info >}} -As compression uses CPU cycles, it should be activated only when the network -communication between the server and clients is slow and there is enough CPU -capacity left for the extra compression/decompression work. - -Furthermore, requests and responses should only be compressed when they exceed a -certain minimum size, i.e. 250 bytes. - -Request and response compression is only supported for HTTP/1.1 and HTTP/2 -responses, and not when using the VST protocol. -{{< /info >}} - ## Miscellaneous changes ### Active AQL query cursors metric @@ -565,13 +569,10 @@ _arangodump_ operations on the server: ### Transparent compression of requests and responses -The following options have been added to all [client tools](../../components/tools/_index.md) -and can be used to enable transparent compression of the data that is sent -between the client tools and the ArangoDB server: -- `--compress-transfer` -- `--compress-request-threshold` - -For more information, see the [dedicated feature section](#transparent-compression-of-requests-and-responses-between-arangodb-server-and-client-tools). +Startup options to enable transparent compression of the data that is sent +between a client tool and the ArangoDB server have been added. See the +[Server options](#transparent-compression-of-requests-and-responses-between-arangodb-servers-and-client-tools) +section above that includes a description of the added client tool options. ## Internal changes