What is Kubernetes multi-cluster?

What is multi-cluster Kubernetes?

A single Kubernetes cluster can be extremely flexible; it can serve a single developer or application, or you can use it with various techniques to create a multi-tenant Kubernetes environment. At some point, however, you will find that you need a Kubernetes multi-cluster environment.

Multi-cluster Kubernetes is exactly what it sounds like: it’s an environment in which you are using more than one Kubernetes cluster. These clusters may be on the same physical host, on different hosts in the same data center, or even in different clouds in different countries, for a multi-cloud environment.

What makes your environment a Kubernetes multi-cluster environment is not just the fact that more than one cloud is in use. Kubernetes multi-cluster is what happens when an organization takes steps to coordinate delivery and management of multiple Kubernetes environments with appropriate tools and processes.

In this article, we’ll discuss:

• Reasons you’d want to use a Kubernetes multi-cluster environment

• Technical aspects to consider when designing a Kubernetes multi-cluster architecture

• Working with multiple Kubernetes clusters

• Connecting to multiple Kubernetes clusters

Why Multi-Cluster?

Of course, with multiple Kubernetes clusters comes an increase in complexity, so why would you even want to take something like this on? Multi-cluster Kubernetes provides several advantages, including:

• Tenant isolation: Even if you have only one development team, you’re going to need multiple Kubernetes environments to accommodate development, staging, and production, and out of the box, Kubernetes is not a multi-tenant architecture. In some cases, you can achieve this isolation using namespaces within a single cluster, but the Kubernetes security model makes it difficult to isolate these environments from each other, and in any case, you still run into the “noisy neighbor” problem, where one runaway app can affect any environments and applications sharing that hardware. Kubernetes multi-cluster environments enable you to isolate users and/or projects by cluster, simplifying this process.

• Availability and performance: By architecting a Kubernetes multi-cluster environment, you provide the ability to move workloads between clusters, if necessary, which enables you to avoid issues if one cluster should become bogged down or even disappear entirely.

• Simplified management: Remember, just adding additional clusters doesn’t make it a multi-cluster environment. In fact, using multiple isolated clusters, independently managed, can be a management nightmare. Instead, multiple clusters managed from a central point provides consolidating logging and auditing, and helps to prevent runaway spending from “shadow IT.”

• Failover: Having a multi-cluster environment enables you to ensure that workloads don’t experience downtime due to a problem within a single cluster, as you can seamlessly transfer them to another cluster.

• Geographic specificity and Regulatory control: One of the defining characteristics of cloud computing is that you don’t necessarily know (or care) where your workloads are running. In some cases, however, this can be an issue. For example, some countries and regions require that data on their citizens not leave their borders; others have strict regulations on how that data must be treated. By using multiple clusters in multiple locations, you can control the geography of your applications without sacrificing the flexibility of cloud native applications.

• Scaling and bursting of applications: Multi-cluster Kubernetes enables you to provide the ability to scale beyond the limitations of a single cluster by moving parts of your application to additional clusters. This technique is most useful, of course, when utilized in a <a href=””>multi-cloud Kubernetes</a> environment.

• Distributed systems: For geographically distributed organizations, it’s not necessarily practical to run the entire enterprise on a single cluster. In these situations, it can become more feasible to have clusters in each location, but have them centrally managed.

• Edge computing and Internet of Things (IoT): Perhaps the ultimate embodiment of a distributed system is edge computing, in which computing resources are placed as close as possible to the data on which they operate. Edge computing typically involves local clusters that process data and send results to regional clusters, which themselves perform operations and send the results to a central cluster. In an IoT environment, those edge clusters may have “leaf nodes” that consist of small compute devices that act as nodes for the cluster.

What does a Kubernetes multi-cluster architecture look like?

Just as there are many different ways to build an application, there are also multiple ways to architect a multi-cluster Kubernetes environment. In general, they fall into two categories:

cluster-centric and application-centric.

How do I connect to multiple Kubernetes clusters?

With a cluster-central Kubernetes multi-cluster architecture, the focus is on making multiple clusters appear to behave as one, as is the case with Kubernetes federation. In this situation, applications may be unaware of where they are actually running. While this approach can simplify management in some ways (particularly for developers) it does introduce additional complexity for operators, as a fault in one cluster can affect other clusters, leading to overall issues.

With an application-centric approach, individual clusters remain independent (though they may be administered from a single dashboard) and applications themselves are designed to be cluster-independent, migrating as necessary. This migration between clusters may be handled by the application directly, or it may be the purview of a service mesh such as Istio or LinkerD, which sends requests to individual endpoints based on desired criteria.

Whichever approach you choose, it’s important to consider the networking implications. A complete discussion of Kubernetes networking is beyond the scope of this article, but it does provide a great deal of power and flexibility — which means it can be complex to administer.

How do I work with multiple Kubernetes clusters?

In order to configure multi-cluster Kubernetes, we need to look at the ways in which we normally access a single cluster. Typically, you access Kubernetes using a client such as kubectl, which takes its configuration from a KUBECONFIG file. This file typically contains the definition of your cluster, such as:

apiVersion: v1
kind: Config
preferences: {}

clusters:
- cluster:
   certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ...
   server: https://172.19.113.9:443
  name: gettingstarted
- cluster:
   certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ...
   server: https://172.19.218.42:443
  name: demo

contexts:
- context:
    cluster: gettingstarted
    user: nick
  name: nick@gettingstarted
- context:
    cluster: demo
    user: nick
  name: nick@demo
current-context: nick@gettingstarted

users:
- name: nick
  user:
   auth-provider:
     config:
       client-id: k8s
       id-token: eyJhbGciOiJSUzI1NiIsInR5cC...
       idp-certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS...
       idp-issuer-url: https://containerauth.int.mirantis.com/auth/realms/iam
       refresh-token: eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSld...
     name: oidc

In this file, we see two clusters, getting started and demo, and I’m accessing each of them from a single user account.

(So if I were to ask “How many clusters are in Kubernetes?”, the answer is “how many are in your KUBECONFIG?”)

To switch between these clusters, it’s most convenient to use contexts, because they include both the cluster and use information. You can set these contexts by either editing the file by hand, or by using the kubectl config command, as in:

kubectl config --kubeconfig=mykubeconfigfile set-context appcontext --cluster=gettingstarted --namespace=app1 --user=nick

This adds the new context to the KUBECONFIG, as in:

...
contexts:
- context:
    cluster: gettingstarted
    user: nick
  name: nick@gettingstarted
- context:
    cluster: demo
    user: nick
  name: nick@demo
- context:
    cluster: gettingstarted
    namespace: app1
    user: nick
  name: appcontext
current-context: nick@gettingstarted
...

Now to use this new context, we can switch to it using:

kubectl config set current-context appcontext

Now any commands we execute will run against that new context.