Refactor: Consider removing cluster id from code base

[v0y4g3r]

What type of enhancement is this?

Refactor

What does the enhancement do?

The cluster_id field was originally intended to isolate nodes from different clusters, allowing metasrv to be shared among various GreptimeDB clusters. However, since the cluster ID is not assigned anywhere and is largely ignored, it can lead to confusion. We need to remove this field because we will not rely on it for tenant isolation.

Implementation challenges

The main challenge is metadata compatibility in that many key values in metasrv contains a cluster id field, for example:

greptimedb/src/meta-srv/src/key/datanode.rs

Lines 44 to 47 in 4045298

	pub struct DatanodeLeaseKey {
	pub cluster_id: ClusterId,
	pub node_id: u64,
	}

[killme2008]

Question: how to allow metasrv to be shared among various GreptimeDB clusters?

[v0y4g3r]

Question: how to allow metasrv to be shared among various GreptimeDB clusters?

The problem is, even if with cluster id, metasrv cluster cannot be shared since that field is almost always hardcoded to 0. I discussed with @fengjiachun yesterday and we don't think it's a good idea to share metasrv instances. Instead, we can share the underlying metadata store (etcd, pg, etc.).

[killme2008]

Question: how to allow metasrv to be shared among various GreptimeDB clusters?

The problem is, even if with cluster id, metasrv cluster cannot be shared since that field is almost always hardcoded to 0. I discussed with @fengjiachun yesterday and we don't think it's a good idea to share metasrv instances. Instead, we can share the underlying metadata store (etcd, pg, etc.).

Can we confirm that the underlying metadata storage could be shared between clusters? By the root path or something like that?

[v0y4g3r]

Question: how to allow metasrv to be shared among various GreptimeDB clusters?

The problem is, even if with cluster id, metasrv cluster cannot be shared since that field is almost always hardcoded to 0. I discussed with @fengjiachun yesterday and we don't think it's a good idea to share metasrv instances. Instead, we can share the underlying metadata store (etcd, pg, etc.).

Can we confirm that the underlying metadata storage could be shared between clusters? By the root path or something like that?

For all kinds of metadata store, we can use store_key_prefix for different clusters.

And specifically for PostgreSQL metadata store, we have an additional meta_table_name for namespace isolation.