Introduce namespace-scoped keyspace identity for API V3

[disksing]

Background

Agent-oriented applications tend to create many small, short-lived, and highly customized logical clusters. They may also create several clusters concurrently for exploration and retain only the best result. This workload requires a much larger total number of cluster creations and surviving logical clusters than API V2 was designed for.

API V2 uses a 24-bit keyspace ID, limiting one physical cluster to about 2^24 keyspaces. API V3 introduces a namespace above keyspace so the canonical tenant identity becomes (namespace_id, keyspace_id). A namespace can map to an organization and provide its own 24-bit keyspace ID space, expanding the logical address space to 2^32 × 2^24 = 2^56 identities.

Concepts

• Namespace: a logical grouping in which multiple keyspaces may share physical TiKV Regions. An organization can map to one namespace, or a namespace may contain a single keyspace when stronger physical isolation is preferred.
• Keyspace: a logical cluster with virtualization semantics similar to an API V2 keyspace.
• Keyspace identity: the pair (namespace_id, keyspace_id). A numeric keyspace ID alone is not globally unique in API V3.

The corresponding TiKV physical-key proposal uses a fixed, aligned 8-byte prefix:

+-----------+-------------------+-------------------+
| Mode (1B) | Namespace ID (4B) | Keyspace ID (3B)  |
+-----------+-------------------+-------------------+

This layout keeps key parsing deterministic, allows a single 64-bit prefix comparison, and lets related keyspaces share fewer Regions to reduce PD scheduling and heartbeat pressure. The physical encoding itself is implemented outside kvproto; kvproto defines the identity carried across component boundaries.

Goals

• Add API V3 and a public KeyspaceIdentity containing namespace_id and keyspace_id.
• Add namespace metadata and lifecycle RPCs.
• Use the complete identity in KV, PD, TSO, GC, resource management, AutoID, MPP, and other tenant-scoped protobuf APIs.
• Make it explicit that V3 callers must not treat a bare keyspace ID or keyspace name as globally unique.
• Preserve existing V1/V2 keyspace ID fields where legacy protocol compatibility is required.
• Regenerate Go/Rust/C++ bindings and protobuf lock metadata for downstream consumers.

Compatibility and rollout

API V3 is intended to be selected at build time in TiKV so its 8-byte key layout can evolve independently without risking regressions in the API V2 4-byte path. A transparent online API V2-to-V3 migration is not required in this phase. Components using V3 must propagate the complete identity consistently; mixing V3 requests with legacy ID-only routing must be rejected or handled explicitly.

Non-goals

• Solving large-scale cluster metadata scalability in this phase.
• Providing an online or transparent API V2-to-V3 migration path.
• Defining TiKV storage-engine adapters, compaction migration, backup/restore, or CDC implementation details in kvproto.

Implementation status

The initial kvproto schema and generated bindings were implemented in #1458. Follow-up protocol refinements are tracked separately.