Support Kubernetes MCS API (ServiceExport / ServiceImport) in Linkerd

[kahirokunn]

What problem are you trying to solve?

Currently, multi-cluster service discovery in Linkerd typically relies on Linkerd-specific mechanisms.
However, this can make interoperability difficult with other tools and clusters that prefer Kubernetes standards.
People adopting the Kubernetes Multi-Cluster Services (MCS) API want Linkerd to "just work" with ServiceExport / ServiceImport as the common language for cross-cluster services.

How should the problem be solved?

Would you consider adding first-class support for the Kubernetes MCS API in Linkerd—specifically recognizing and following the specifications of ServiceExport and ServiceImport for cross-cluster service discovery and routing?
This would allow users to rely on a standard model across environments and tooling.

Any alternatives you've considered?

• Continue using Linkerd's existing multi-cluster approach and resources.
• Third-party controllers that translate between MCS resources and Linkerd concepts.

These work, but they don't provide a simple, standard, vendor-neutral contract like the MCS API.

How would users interact with this feature?

As an alternative to the current options of labeling Services with mirror.linkerd.io/federated=member or mirror.linkerd.io/exported=true, users could use ServiceExport to expose services to other clusters.
In the other cluster, ServiceImport would be automatically created and become available for use.

References

KEP-1645: Multi-Cluster Services API
Multicluster Services API Overview - SIG Multicluster
kubernetes-sigs/mcs-api

Multi-Cluster Services API (Beta) — Cilium
Multi-Cluster Services API - Istio

Would you like to work on this feature?

None

[adleong]

Hi @kahirokunn!

During the design of Linkerd's federated services, we looked closely at the MCS KEP as both a potential solution and for inspiration. Ultimately, we determined that implementing the MCS API would require implementing our own DNS or adding a dependency on an existing DNS solution.

Given Linkerd's design philosophy of having a minimal surface area, we decided that federated services was the best way to provide multi-cluster discovery without bringing DNS into the picture.

With that said, we're definitely keeping an eye on the MCS API as it matures and evolves. I'm also curious to hear more about the tools that interoperate with MCS so that we can have an accurate picture of the value vs implementation tradeoff.

[kahirokunn]

Hi @adleong !
Thanks for your response 🙏

During the design of Linkerd's federated services, we looked closely at the MCS KEP as both a potential solution and for inspiration.

I evaluated federated services and found them truly excellent.
First of all, thank you!
For that reason, I think it would be wonderful if federated services could be used via the MCS API.

Ultimately, we determined that implementing the MCS API would require implementing our own DNS or adding a dependency on an existing DNS solution.

CoreDNS now officially supports multicluster DNS. (It used to be an official plugin, but now—remarkably—it’s built into Core!)
coredns/coredns#7266
This was released in CoreDNS v1.12.2.
https://github.com/coredns/coredns/releases/tag/v1.12.2

Therefore, if we adhere to CoreDNS’s multicluster DNS specification, Linkerd would not need to maintain its own DNS implementation.
However, some environments still use older versions of CoreDNS; in those cases, I think it’s fine not to support multicluster DNS. This is because Linkerd multicluster can control which DNS suffix to use via the targetClusterDomain field on the Link resource. If CoreDNS v1.12.2 or newer is present, we can use clusterset.local; otherwise, we can use cluster.local.

apiVersion: multicluster.linkerd.io/v1alpha3
kind: Link
metadata:
  name: wl2
  namespace: linkerd-multicluster
spec:
  targetClusterDomain: clusterset.local # or cluster.local

For reference, I checked Kubernetes distributions that use CoreDNS v1.12.2 or later; it’s already being adopted, and it looks likely to become commonplace.

Product	First version determined to "default 1.12.2+"	Link
k0s	v1.32.6+k0s.0	https://github.com/k0sproject/k0s/releases/tag/v1.32.6%2Bk0s.0
k3s	v1.31.12+k3s1	https://github.com/k3s-io/k3s/releases/tag/v1.31.12%2Bk3s1
RKE2	v1.32.6+rke2r1	https://docs.rke2.io/release-notes/v1.32.X
EKS	v1.33	https://docs.aws.amazon.com/eks/latest/userguide/managing-coredns.html

With that said, we're definitely keeping an eye on the MCS API as it matures and evolves. I'm also curious to hear more about the tools that interoperate with MCS so that we can have an accurate picture of the value vs implementation tradeoff.

There is GEP-1748: Gateway API Interaction with Multi-Cluster Services.

This proposal/spec allows using ServiceImport with HTTPRoute.

kind: HTTPRoute
apiVersion: gateway.networking.k8s.io/v1beta1
metadata:
  name: store
spec:
  parentRefs:
  - name: external-http
  rules:
  - matches:
    - path:
        type: PathPrefix
        value: /west
    backendRefs:
    - group: multicluster.x-k8s.io
      kind: ServiceImport
      name: store-west
      port: 8080
  - matches:
    - path:
        type: PathPrefix
        value: /east
    backendRefs:
    - group: multicluster.x-k8s.io
      kind: ServiceImport
      name: store-east
      port: 8080
  - backendRefs:
    - group: multicluster.x-k8s.io
      kind: ServiceImport
      name: store
      port: 8080

This has already been implemented in the Kubernetes ecosystem—for example, Envoy Gateway, an official Envoy project implementing the Gateway API. This gateway is truly excellent, with an unmatched development pace and feature improvements among the OSS projects I’ve seen.
https://gateway.envoyproxy.io/docs/tasks/traffic/multicluster-service/

In that tutorial, Submariner and Envoy Gateway interoperate by using the MCS API.
As shown below, you can reference a ServiceImport from an HTTPRoute.
https://raw.githubusercontent.com/envoyproxy/gateway/latest/examples/kubernetes/multicluster-service.yaml

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: backend
  namespace: default
spec:
  parentRefs:
    - name: eg
  hostnames:
    - "www.example.com"
  rules:
    - backendRefs:
        - group: multicluster.x-k8s.io
          kind: ServiceImport
          name: backend-default-cluster2
          namespace: submariner-operator
          port: 3000
      matches:
        - path:
            type: PathPrefix
            value: /

Karmada also has the MCS API in view.
https://karmada.io/docs/userguide/service/multi-cluster-service/#prerequisites

As for MCS API integrations with other parts of the ecosystem, those are the main examples today. There are additional benefits as well; I think it’s quite powerful that we can operate through a well-known interface like the MCS API. For example, there are multi-cluster mesh implementations that leverage the MCS API interface:

Istio MCS API
Google Cloud MCS: General Availability
Submariner: 0.13.3
MCS controller for AWS Cloudmap
Antrea Multi-cluster
Cilium Cluster Mesh

What this means is that many users have already learned the concepts of the MCS API and already have the mental model of controlling multi-cluster services by using ServiceExport and ClusterProperty. In other words, at this point it has established a position as a standard.

If Linkerd were to support federated services via the MCS API interface, it would mean users could start using it without relearning. For example, to check multi-cluster connectivity, users wouldn’t have to learn provider-specific tools each time (like linkerd multicluster check); by looking at the status of ServiceExport, they can confirm whether the export succeeded, and by confirming the existence/status of ServiceImport, they can see whether multi-cluster configuration is correct and the service is usable—all without any provider-specific additional learning.

This enables situations like the following:
You can set up alerts in AlertManager against the status of ServiceExport and ServiceImport, so that an alert is immediately raised when an anomaly is detected.
You can build provider-agnostic or pluggable general-purpose dashboards that visualize ServiceExport, ServiceImport, and the corresponding resources. This also expands possibilities for tools like Argo CD.

These benefits are very similar to the reasons and advantages for NGINX, Envoy, Istio, etc., to support Ingress or the Gateway API rather than defining their own custom resources.

Linkerd is a truly outstanding open-source project, and federated services are an excellent implementation; if they can be governed via the MCS API custom resources defined by SIG Multicluster’s KEP, I’m confident this will further promote Linkerd’s adoption within the broader SIG Multicluster context.

[adleong]

Thanks @kahirokunn! This is super helpful and informative, especially the parts about core kubernetes support for MCS via CoreDNS and the integrations with the Gateway API. Adopting MCS would be a big architecture change for linkerd-multicluster so I'm not sure when we'll have the bandwidth to explore and evaluate this in detail. But it's great to have this on our radar and to know that it may be a viable option.

[kahirokunn]

Thanks so much, @adleong — really appreciate you considering this. It’s great to hear MCS is on your radar.
When it’s helpful, I’m happy to provide additional context or feedback. Please feel free to ping me anytime. Thanks again for the thoughtful discussion and for all your work on Linkerd!

[kahirokunn]

Kubernetes v1.35 will ship with CoreDNS v1.12.3 by default, which already includes built-in multicluster DNS (clusterset.local). kubernetes/kubernetes#132288

Also, I’ve personally verified this functionality in kubernetes-sigs/mcs-api#126 💪