mito: Write cache for remote object store

[evenyag]

What type of enhancement is this?

Performance

What does the enhancement do?

Now the mito storage engine writes SST files to remote object stores directly.

greptimedb/src/mito2/src/sst/parquet/writer.rs

Lines 35 to 43 in 5f8c175

	pub struct ParquetWriter {
	/// SST output file path.
	file_path: String,
	/// Input data source.
	source: Source,
	/// Region metadata of the source and the target SST.
	metadata: RegionMetadataRef,
	object_store: ObjectStore,
	}

We have to fetch the object from the object store again if we want to access the file. If we implement a write-through cache for parquet files, we don't need to download the object again.

Implementation challenges

This might increase the cost of uploading an object and the memory pressure of memtables.

• A better approach is to release the memtable once we flush the file to the write cache.
• We update the manifest after the object is fully uploaded to the remote object store

To implement async upload, we need to store other metadata such as flushed sequence and region id for files in the write cache. The region edit also requires memtable ids to remove flushed memtables. We should switch to using the minimum sequence of memtable as the memtable id is incremented globally.

For simplicity, we can implement the sync version first, which returns after files are uploaded.

Steps

• Implements a sync write cache
  • WriteCache skeleton feat(mito): Add WriteCache struct and write SSTs to write cache #2999
  • Uploads files to remote object stores feat: write and upload sst #3106
  • A file cache for the write cache feat(mito): Implement file cache #3022
  • Do not use write cache if using fs
  • Integrate to the datanode feat(mito): Init the write cache in datanode #3100
  • Fix file cache (use get instead of contains_key) feat(mito): Init the write cache in datanode #3100
  • Reads from the write cache feat: read data from write cache #3128
  • Add tests for write cache in region level test: engine with write cache #3163
  • Support writing index file to the write cache feat(mito): support write cache for index file #3144
  • Read page for the metadata from the write cache feat: read metadata from write cache #3224
• Implements an async write cache
  • Write metadata of an upload to the local file system
  • Add a volatile FileHandle for files in the write cache to Version
  • Uploads files to remote object stores in background
  • Updates manifest and version after uploading files
  • Recovers upload tasks?
  • Handle alter should not only consider whether memtable is empty
• Benchmarks on remote object stores
  • S3
  • OSS
  • COS

Further discussions

• If the engine opens a region in a fresh env without the write cache, replaying the wal might cause oom
• We can trigger a flush during replay to avoid OOM and upload it once we have write permission
• Delays uploading level 0 files we might compact them later

Related Issues

It should be part of #2516