You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/guides/ingesters-with-wal.md
+4-4Lines changed: 4 additions & 4 deletions
Original file line number
Diff line number
Diff line change
@@ -5,7 +5,7 @@ weight: 5
5
5
slug: ingesters-with-wal
6
6
---
7
7
8
-
Currently the ingesters without the TSDB stores all the data in the memory and if it happens to crash for any reason, there will be a loss of data. WAL helps fill this gap of reliability.
8
+
Currently the ingesters running in the chunks storage mode, store all their data in memory. If there is a crash, there could be loss of data. WAL helps fill this gap in reliability.
9
9
10
10
To use WAL, there are some changes that needs to be made in the deployment.
11
11
@@ -14,17 +14,17 @@ To use WAL, there are some changes that needs to be made in the deployment.
14
14
1. Since ingesters need to have the same persistent volume across restarts/rollout, all the ingesters should be run on [statefulset](https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/) with fixed volumes.
15
15
16
16
2. Following flags needs to be set
17
-
*`--ingester.wal-dir` to the directory where the WAL data should be stores and/or recovered from.
17
+
*`--ingester.wal-dir` to the directory where the WAL data should be stores and/or recovered from. Note that this should be on the mounted volume.
18
18
*`--ingester.wal-enabled` to `true` which enables writing to WAL during ingestion.
19
19
*`--ingester.checkpoint-enabled` to `true` to enable checkpointing of in-memory chunks to disk. This is optional which helps in speeding up the replay process.
20
20
*`--ingester.checkpoint-duration` to the interval at which checkpoints should be created. Default is `30m`, and depending on the number of series, it can be brought down to `15m` if there are less series per ingester (say 1M).
21
21
*`--ingester.recover-from-wal` to `true` to recover data from an existing WAL. The data is recovered even if WAL is disabled and this is set to `true`. The WAL dir needs to be set for this.
22
22
* If you are going to enable WAL, it is advisable to always set this to `true`.
23
-
*`--ingester.tokens-file-path` should be set to the filepath where the tokens should be stored. Why this is required is described below.
23
+
*`--ingester.tokens-file-path` should be set to the filepath where the tokens should be stored. Note that this should be on the mounted volume. Why this is required is described below.
24
24
25
25
## Changes in lifecycle when WAL is enabled
26
26
27
-
1. Flushing of data to chunk store during rollouts or scale down is disabled. This is because during a rollout of statefulset there are no ingesters that are simultaneously leaving and joining, rather the same ingester is shut down and broght back again with updated config. Hence flushing is skipped and the data is recovered from the WAL.
27
+
1. Flushing of data to chunk store during rollouts or scale down is disabled. This is because during a rollout of statefulset there are no ingesters that are simultaneously leaving and joining, rather the same ingester is shut down and brought back again with updated config. Hence flushing is skipped and the data is recovered from the WAL.
28
28
29
29
2. As there are no transfers between ingesters, the tokens are stored and recovered from disk between rollout/restarts. This is [not a new thing](https://github.com/cortexproject/cortex/pull/1750) but it is effective when using statefulsets.
0 commit comments