KQueen: The Open Source Multi-cloud Kubernetes Cluster Manager
Nick Chase - January 16, 2018
Kubernetes got a lot of traction in 2017, and I'm one of those people who believes that in 2 years the Kubernetes API could become the standard interface for consuming infrastructure. In other words, the same way that today we get an IP address and ssh key for our virtual machine on AWS, we might get a Kubernetes API endpoint and kubeconfig from our cluster. With the recent AWS announcement about EKS bringing this reality even closer, let me give you my perspective on k8s installation and trends that we see at Mirantis.
All of those vendor solutions focus more or less on “their way” of k8s cluster deployment, which usually means specific a deployment procedure of defined packages, binaries and artifacts. Moreover, while some of these installers are available as open source packages, they're not intended to be modified, and when delivered by a vendor, there's often no opportunity for customer-centric modification.
There are reasons, however, why this approach is not enough for the enterprise customer use case. Let me go through them.
Customization: Our real Kubernetes deployments and operation have demonstrated that we cannot just deploy a cluster on a custom OS with binaries. Enterprise customers have various security policies, configuration management databases, and specific tools, all of which are required to be installed on the OS. A very good example of this situation is one of a customer from the financial sector. The first time the started their golden OS image at AWS, it took 45 minutes to boot. This makes it impossible for some customers to run the native managed k8s offering at public cloud providers.
Multi-cloud: Most existing vendors don't solve the question of how to manage clusters in multiple regions, let alone at multiple providers. Enterprise customers want to run distributed workloads in private and public clouds. Even in the case of on-premise baremetal deployment, people don't a build single huge cluster for whole company. Instead, they separate resources based on QA/testing/production, application-specific, or team-specific clusters, which often causes complications with existing solutions. For example, OpenShift manages a single Kubernetes cluster instance. One of our customers wound up with an official design where they planned to run 5 independent OpenShift instances without central visibility or any way to manage deployment. Another good example is CoreOS Tectonic, which provides a great UI for RBAC management and cluster workload, but has the same problem -- it only manages a single cluster, and as I said, nobody stays with single cluster.
Moreover, all public cloud providers will eventually offer their own managed k8s offering, which will put various k8s SaaS providers out of business. After all, there is no point paying for managed k8s on AWS to a third-party company if AWS provides EKS.
To summarize, I don't currently see any existing Kubernetes tool that provides all of those features.
KQueen supplies the backend API for provider-agnostic cluster management. It enables access from the UI, CLI, or API, and manages provisioning of Kubernetes clusters. It uses the following workflow:
Existing tools
Recently I did some personal research, and I discovered the following numbers around the Kubernetes community:- ~22 k8s distribution support providers
- ~10 k8s community deployment tools
- ~20 CaaS vendors
All of those vendor solutions focus more or less on “their way” of k8s cluster deployment, which usually means specific a deployment procedure of defined packages, binaries and artifacts. Moreover, while some of these installers are available as open source packages, they're not intended to be modified, and when delivered by a vendor, there's often no opportunity for customer-centric modification.
There are reasons, however, why this approach is not enough for the enterprise customer use case. Let me go through them.
Customization: Our real Kubernetes deployments and operation have demonstrated that we cannot just deploy a cluster on a custom OS with binaries. Enterprise customers have various security policies, configuration management databases, and specific tools, all of which are required to be installed on the OS. A very good example of this situation is one of a customer from the financial sector. The first time the started their golden OS image at AWS, it took 45 minutes to boot. This makes it impossible for some customers to run the native managed k8s offering at public cloud providers.
Multi-cloud: Most existing vendors don't solve the question of how to manage clusters in multiple regions, let alone at multiple providers. Enterprise customers want to run distributed workloads in private and public clouds. Even in the case of on-premise baremetal deployment, people don't a build single huge cluster for whole company. Instead, they separate resources based on QA/testing/production, application-specific, or team-specific clusters, which often causes complications with existing solutions. For example, OpenShift manages a single Kubernetes cluster instance. One of our customers wound up with an official design where they planned to run 5 independent OpenShift instances without central visibility or any way to manage deployment. Another good example is CoreOS Tectonic, which provides a great UI for RBAC management and cluster workload, but has the same problem -- it only manages a single cluster, and as I said, nobody stays with single cluster.
most existing vendors do not solve the question of how to manage clusters in multiple locations"My k8s cluster is better than yours" syndrome: In the OpenStack world, where we originally came from, we're used to complexity. OpenStack was very complex and Mirantis was very successful, because we could install it the most quickly, easily, and correctly. Contrast this with the current Kubernetes world; with multiple vendors, it is very difficult to differentiate in k8s installation. My view is that k8s provisioning is commodity with very low added value, which makes k8s installation more as vendor checkbox feature, rather than decision making point or unique capability. At the moment, however, let me borrow my favourite statement from a Kubernetes community leader: “Yes, there are lot of k8s installers, but very few deploy k8s 100% correctly.”
Moreover, all public cloud providers will eventually offer their own managed k8s offering, which will put various k8s SaaS providers out of business. After all, there is no point paying for managed k8s on AWS to a third-party company if AWS provides EKS.
K8s provisioning is a commodity, with very low added value.Visibility & Audit: Lastly, but most importantly, deployment is just the beginning. Users need to have visibility, with information on what runs where in what setup. It's not just about central monitoring, logging and alerting; it's also about audit. Users need audit features such as "all docker images used in all k8s clusters" or "version of all k8s binaries". Today, if you do find such a tool, it usually has gaps at the multi-cluster level, can providing information only for single clusters.
To summarize, I don't currently see any existing Kubernetes tool that provides all of those features.
KQueen as Open Cluster Manager
Based on all of these points, we at Mirantis decided to build a provisioner-agnostic Kubernetes cluster manager to deploy, manage and operate various Kubernetes clusters on various public/private cloud providers. Internally, we have called the project KQueen and, and it follows several design principles:- Kubernetes as a Service environment deployment: Provide a multi-tenant self-service portal for k8s cluster provisioning.
- Operations: Focus on the audit, visibility, and security of Kubernetes clusters, in addition to actual operations.
- Update and Upgrade: Automate updating and upgrading of clusters through specific provisioners.
- Multi-Cloud Orchestration: Support the same abstraction layer for any public, private, or bare metal provider.
- Platform Agnostic Deployment (of any Kubernetes cluster): Enable provisioning of a Kubernetes cluster by various community installers/provisioners, including those with customizations, rather than a black box with a strict installation procedure.
- Open, Zero Lock-in Deployment: Provide a pure-play open source solution without any closed source.
- Easy integration: Provide a documented REST API for managing Kubernetes clusters and integrating this management interface into existing systems.
KQueen supplies the backend API for provider-agnostic cluster management. It enables access from the UI, CLI, or API, and manages provisioning of Kubernetes clusters. It uses the following workflow:
- Trigger deployment on the provisioner, enabling KQueen to use various provisioners (AKS, GKE, Jenkins) for Kubernetes clusters. For example, you can use the Jenkins provisioner to trigger installation of Kubernetes based on a particular job.
- The provisioner installs the Kubernetes cluster using the specific provider.
- The provisioner returns the Kubernetes kubeconfig and API endpoint. This config is stored in the KQueen backend (etcd).
- KQueen manages, operates, monitors, and audits the Kubernetes clusters. It reads all information from the API and displays it as a simple overview visualization. KQueen can also be extended by adding other audit components.
KQueen in action
The KQueen project can help define enterprise-scale kubernetes offerings across departments and give them freedom in specific customizations. If you'd like to see it in action, you can see a generic KQueen demo showing the architecture design and managing a cluster from a single place, as well as a demo based on Azure AKS demo. In addition, watch this space for a tutorial on how to set up and use KQueen for yourself. We'd love your feedback!Want to be a Kubernetes expert?
Check out our Kubernetes and Docker Bootcamps and related certification.
