How do you build 12-factor apps using Kubernetes?
Nick Chase - April 17, 2017
It's said that there are 12 factors that define a cloud-native application. It's also said that Kubernetes is designed for cloud native computing. So how do you create a 12-factor app using Kubernetes? Let's take a look at exactly what twelve factor apps are and how you can apply them to your own project in this handy Kubernetes 12 factor checklist.
Principle I. Codebase
Principle 1 of a 12 Factor App is "One codebase tracked in revision control, many deploys".
For Kubernetes applications, this principle is actually embedded in the nature of container orchestration itself. Typically, you create your code using a source control repository such as a git repo, then store specific versions of your images in the Docker Hub. When you define the containers to be orchestrated as part of a a Kubernetes Pod, Deployment, DaemonSet, you also specify a particular version of the image, as in:
Applications can also behave differently depending on the configuration information with which they run.
Making sure that an application's dependencies are satisfied is something that is practically assumed. For a 12 factor app, that includes not just making sure that the application-specific libraries are available, but also not counting on, say, shelling out to the operating system and assuming system libraries such as curl will be there. A 12 factor app must be self-contained.
That includes making sure that the application is isolated enough that it's not affected by conflicting libraries that might be installed on the host machine.
Fortunately, if an application does have any specific or unusual system requirements, both of these requirements are handily satisfied by containers; the container includes all of the dependencies on which the application relies, and also provides a reasonably isolated environment in which the container runs. (Contrary to popular belief, container environments are not completely isolated, but for most situations, they are Good Enough.)
For applications that are modularized and depend on other components, such as an HTTP service and a log fetcher, Kubernetes provides a way to combine all of these pieces into a single Pod, for an environment that encapsulates those pieces appropriately.
The idea behind this principle is that an application should be completely independent from its configuration. In other words, you should be able to move it to another environment without having to touch the source code.
Some developers achieve this goal by creating configuration files of some sort, specifying details such as directories, hostnames, and database credentials. This is an improvement, but it does carry the risk that someone will check a config file into the source control repository.
Instead, 12 factor apps store their configurations as environment variables; these are, as the manifesto says, "unlikely to be checked into the repository by accident", and they're operating system independent.
Kubernetes enables you to specify environment variables in manifests via the Downward API, but as these manifests themselves do get checked int source control, that's not a complete solution.
Instead, you can specify that environment variables should be populated by the content of a Kubernetes ConfigMap or Secret, which can be kept separate from the application. For example, you might define a Pod as:
Of course, there's still a risk of someone mis-handling the files used to create these objects, but it's them together and institute secure handling policies than it is to weed out dozens of config files scattered around a deployment.
What's more, there are those in the community that point out that even environment variables are not necessarily safe for their own reasons. For example, if an app crashes, it may save all of the environment variables to a log or even transmit them to another service. Diogo Mónica points to a tool called Keywhiz you can use with Kubernetes, creating secure secret storage.
In a 12 Factor app, any services that are not part of the core application, such as databases, external storage, or message queues, should be accessed as a service -- via an HTTP or similar request -- and specified in the configuration, so that the source of the service can be changed without affecting the core code of the application.
For example, if your application uses a message queuing system, you should be able to change from RabbitMQ to ZeroMQ (or ActiveMQ or even something else) without having to change anything but configuration information.
This requirement has two implications for a 12 factor Kubernetes application.
First, it means that you must think about how your applications access and send out information. For example, if you have a backing database, you wouldn't want to have a local Mysql instance, even if you're replicating it to other instances. Instead, you would want to have a separate container that handles database operations, and make those operations callable via an API. This way, if you needed to change to, say, PostgreSQL or a remotely hosted MySQL server, you could create a new container image, update the Pod definition, and restart the Pod (or more likely the Deployment or StatefulSet managing it).
Similarly, if you're storing credentials or address information in environment variables backed by a ConfigMap, you can change that information and replace the Pod.
Note that both of these examples assume that though you're not making any changes to the source code (or even the container image for the main application) you will need to replace the Pod; the ability to do this is actually another principle of a 12 Factor App.
These days it's hard to imagine a situation where this is not true, but a good 12 factor app example must have a separate build stage. In other words, you should be able to build or compile the code, then combine that with specific configuration information to create a specific release, then deliberately run the release.
Releases should be identifiable. You should be able to say, 'This deployment is running Release 1.14 of this application" or something similar, the same way we say we're running "the OpenStack Ocata release" or "Kubernetes 1.6". They should also be immutable; any changes should lead to a new release. If this sounds daunting, remember that when we say "application" we're no longer talking about large, monolithic releases. Instead, we're talking about very specific microservices, each of which has its own release, and which can bump releases without causing errors in consuming services.
All of this is so that when the app is running, that "run" process can be completely automated. Twelve factor apps need to be capable of running in an automated fashion because they need to be capable of restarting should there be a problem.
Translating this to the Kubernetes realm, we've already said that the application needs to be stored in source control, then built with all of its dependencies. That's your build process. We talked about separating out the configuration information, so that's what needs to be combined with the build to make a release. And the ability to automatically run the application -- or multiple copies of the application -- is precisely what Kubernetes constructs like Deployments, ReplicaSets, and DaemonSets do.
Stateless processes are a core idea behind cloud native applications. Every twelve-factor application needs to run in individual, share-nothing processes. That means that any time you need to persist information, it needs to be stored in a backing service such as a database.
If you're new to cloud application programming, this might be deceptively simple; many developers are used to "sticky" sessions, storing information in the session with the confidence that the next request will come to the same server. In a cloud application, however, you must never make that assumption.
Instead, if you're running a Kubernetes-based application that hosts multiple copies of a particular pod, you must assume that subsequent requests can go anywhere. To solve this problem, you will want to use some sort of backing volume or database for persistence.
One caveat to this principle is that Kubernetes StatefulSets can enable you to create Pods with stable network identities, so that you can, theoretically, direct requests to a particular pod. Technically speaking, if the process doesn't actually store state, and the pod can be deleted and recreated and still function properly, it satisfies this requirement -- but that's probably pushing it a bit.
In an environment where we're assuming that different functionalities are handled by different processes, it's easy to make the connection that these functionalities should be available via a protocol such as HTTP, so it's common for applications to be run behind web servers such as Apache or Tomcat. Twelve-factor apps, however, should not be dependent on an additional application in that way; remember, every function should be in its own process, isolated from everything else. Instead, the 12 Factor App manifesto recommends adding a web server library or something similar to the app itself, By using port binding, 12 factor app services can await requests on a defined port, whether it's using HTTP or another protocol.
In a Kubernetes-based application, this is done partly through the architecture of the application itself, and partly by making sure that the application has all of its dependencies as part of the creation of the base containers on which the application is built.
When you're writing a twelve-factor app, make sure that you provide a way for it to be scaled out, rather than scaled up. That means that in order to add more capacity, you should be able to add more instances rather than more memory or CPU to the machine on which the app is running. Note that this specifically means being able to start additional processes on additional machines, which is, fortunately, a key capability of Kubernetes.
It seems like this principle was tailor made for containers and Kubernetes-based applications. The idea that processes should be disposable means that at any time, an application can die and the user experience won't be affected, either because there are others to take its place, because it'll start right up again, or both.
Containers are built on this principle, of course, and Kubernetes structures that manage multiple instances and maintain a certain level of availability even in the face of problems, such as ReplicaSets, complete the picture.
This is another principle that seems like it should be obvious, but is deeper than most people think. On the surface level, it does mean that you should have identical development, staging, and production environments, inasmuch as that is possible. One way to accomplish this is through the use of Kubernetes namespaces, enabling you to (theoretically) run code on the same actual cluster against the same actual systems while still keeping environments separate. In some situations, you can also use tools such as Minikube or kubeadm-dind-cluster to create near-clones of production systems.
At a deeper level, though, as the Twelve-Factor App manifesto puts it, it's about three different types of "gaps":
Some of these factors are outside of the realm of Kubernetes, of course; the personnel gap is a cultural issue, for example. The time and tools gaps, however, can be helped in two ways.
For the time gap, Kubernetes-based applications are, of course, based on containers, which themselves are based on images that are stored in version-control systems, so they lend themselves to CI/CD. They can also be updated via rolling updates that can be rolled back in case of problems, so they're well-suited to this kind of environment.
As far as the tools gap is concerned, the architecture of Kubernetes-based applications make it easier to manage, both by making local dependencies simple to include in the various images, and by modularizing the application in such a way that external backing services can be standardized.
While most traditional applications store log information in a file, the Twelve-Factor app directs it, instead, to stdout as a stream of events; it's the execution environment that's responsible for collecting those events. That might be as simple as redirecting stdout to a file, but in most cases it involves using a log router such as Fluentd and saving the logs to Hadoop or a service such as Splunk.
In building a 12 factor app using Kubernetes, you have at least two choices for logging capture automation: Stackdriver Logging if you're using Google Cloud, and Elasticsearch if you're not. You can find more information on setting Kubernetes logging destinations here.
This principle involves separating admin tasks such as migrating a database or inspecting records from the rest of the application. Even though they're separate, however, they must still be run in the same environment and against the same base code and configuration as the application, and their code must be shipped alongside the application to prevent drift.
You can implement this a number of different ways in Kubernetes-based applications, depending on the size and scale of the application itself. For example, for small tasks, you might use kubectl exec to operate on a specific container, or you can use the Kubernetes Job to run a self-contained application. For more complicated tasks that involve orchestration of changes, however, you can also use Kubernetes Helm charts.
So we want to know: which of these best practices are the biggest struggle for you? Where do you think you need to work harder, and what would make it easier for you? Let us know in the comments.
Thanks to Jedrzej Nowak, Piotr Shamruk, Ivan Shvedunov, Tomasz Napierala and Lukasz Oles for reviewing this article!
Check out more Kubernetes resources.
What is a 12 factor application?
The Twelve Factor App is a Software as a Service (SaaS) design methodology created by Heroku. The idea is that in order to be really suited to SaaS and avoid problems with software erosion -- where over time an application that's not updated gets to be out of sync with the latest operating systems, security patches, and so on -- an app should follow these 12 principles:- CodebaseOne codebase tracked in revision control, many deploys
- DependenciesExplicitly declare and isolate dependencies
- ConfigStore config in the environment
- Backing servicesTreat backing services as attached resources
- Build, release, runStrictly separate build and run stages
- ProcessesExecute the app as one or more stateless processes
- Port binding
Export services via port binding - Concurrency
Scale out via the process model - DisposabilityMaximize robustness with fast startup and graceful shutdown
- Dev/prod parityKeep development, staging, and production as similar as possible
- LogsTreat logs as event streams
- Admin processes
Run admin/management tasks as one-off processes
Principle I. Codebase
Principle 1 of a 12 Factor App is "One codebase tracked in revision control, many deploys".
For Kubernetes applications, this principle is actually embedded in the nature of container orchestration itself. Typically, you create your code using a source control repository such as a git repo, then store specific versions of your images in the Docker Hub. When you define the containers to be orchestrated as part of a a Kubernetes Pod, Deployment, DaemonSet, you also specify a particular version of the image, as in:
...In this way, you might have multiple versions of your application running in different deployments.
spec: containers: - name: AcctApp image: acctApp:v3
...
Applications can also behave differently depending on the configuration information with which they run.
Principle II. Dependencies
Principle 2 of a 12 Factor App is "Explicitly declare and isolate dependencies".Making sure that an application's dependencies are satisfied is something that is practically assumed. For a 12 factor app, that includes not just making sure that the application-specific libraries are available, but also not counting on, say, shelling out to the operating system and assuming system libraries such as curl will be there. A 12 factor app must be self-contained.
That includes making sure that the application is isolated enough that it's not affected by conflicting libraries that might be installed on the host machine.
Fortunately, if an application does have any specific or unusual system requirements, both of these requirements are handily satisfied by containers; the container includes all of the dependencies on which the application relies, and also provides a reasonably isolated environment in which the container runs. (Contrary to popular belief, container environments are not completely isolated, but for most situations, they are Good Enough.)
For applications that are modularized and depend on other components, such as an HTTP service and a log fetcher, Kubernetes provides a way to combine all of these pieces into a single Pod, for an environment that encapsulates those pieces appropriately.
Principle III. Config
Principle 3 of a 12 Factor App is "Store config in the environment".The idea behind this principle is that an application should be completely independent from its configuration. In other words, you should be able to move it to another environment without having to touch the source code.
Some developers achieve this goal by creating configuration files of some sort, specifying details such as directories, hostnames, and database credentials. This is an improvement, but it does carry the risk that someone will check a config file into the source control repository.
Instead, 12 factor apps store their configurations as environment variables; these are, as the manifesto says, "unlikely to be checked into the repository by accident", and they're operating system independent.
Kubernetes enables you to specify environment variables in manifests via the Downward API, but as these manifests themselves do get checked int source control, that's not a complete solution.
Instead, you can specify that environment variables should be populated by the content of a Kubernetes ConfigMap or Secret, which can be kept separate from the application. For example, you might define a Pod as:
apiVersion: v1 kind: Pod metadata: name: secret-env-pod spec: containers: - name: mycontainer image: redis env: - name: SECRET_USERNAME valueFrom: secretKeyRef: name: mysecret key: username - name: SECRET_PASSWORD valueFrom: secretKeyRef: name: mysecret key: password - name: CONFIG_VERSION valueFrom: configMapKeyRef: name: redis-app-config key: version.idAs you can see, this Pod receives three environment variables, SECRET_USERNAME, SECRET_PASSWORD, and CONFIG_VERSION, the first two from from referenced Kubernetes Secrets, and the third from a Kubernetes ConfigMap. This enables you to keep them out of configuration files.
Of course, there's still a risk of someone mis-handling the files used to create these objects, but it's them together and institute secure handling policies than it is to weed out dozens of config files scattered around a deployment.
What's more, there are those in the community that point out that even environment variables are not necessarily safe for their own reasons. For example, if an app crashes, it may save all of the environment variables to a log or even transmit them to another service. Diogo Mónica points to a tool called Keywhiz you can use with Kubernetes, creating secure secret storage.
Principle IV. Backing services
Principle 4 of the 12 Factor App is "Treat backing services as attached resources".In a 12 Factor app, any services that are not part of the core application, such as databases, external storage, or message queues, should be accessed as a service -- via an HTTP or similar request -- and specified in the configuration, so that the source of the service can be changed without affecting the core code of the application.
For example, if your application uses a message queuing system, you should be able to change from RabbitMQ to ZeroMQ (or ActiveMQ or even something else) without having to change anything but configuration information.
This requirement has two implications for a 12 factor Kubernetes application.
First, it means that you must think about how your applications access and send out information. For example, if you have a backing database, you wouldn't want to have a local Mysql instance, even if you're replicating it to other instances. Instead, you would want to have a separate container that handles database operations, and make those operations callable via an API. This way, if you needed to change to, say, PostgreSQL or a remotely hosted MySQL server, you could create a new container image, update the Pod definition, and restart the Pod (or more likely the Deployment or StatefulSet managing it).
Similarly, if you're storing credentials or address information in environment variables backed by a ConfigMap, you can change that information and replace the Pod.
Note that both of these examples assume that though you're not making any changes to the source code (or even the container image for the main application) you will need to replace the Pod; the ability to do this is actually another principle of a 12 Factor App.
Principle V. Build, release, run
Principle 5 of the 12 Factor App is "Strictly separate build and run stages".These days it's hard to imagine a situation where this is not true, but a good 12 factor app example must have a separate build stage. In other words, you should be able to build or compile the code, then combine that with specific configuration information to create a specific release, then deliberately run the release.
Releases should be identifiable. You should be able to say, 'This deployment is running Release 1.14 of this application" or something similar, the same way we say we're running "the OpenStack Ocata release" or "Kubernetes 1.6". They should also be immutable; any changes should lead to a new release. If this sounds daunting, remember that when we say "application" we're no longer talking about large, monolithic releases. Instead, we're talking about very specific microservices, each of which has its own release, and which can bump releases without causing errors in consuming services.
All of this is so that when the app is running, that "run" process can be completely automated. Twelve factor apps need to be capable of running in an automated fashion because they need to be capable of restarting should there be a problem.
Translating this to the Kubernetes realm, we've already said that the application needs to be stored in source control, then built with all of its dependencies. That's your build process. We talked about separating out the configuration information, so that's what needs to be combined with the build to make a release. And the ability to automatically run the application -- or multiple copies of the application -- is precisely what Kubernetes constructs like Deployments, ReplicaSets, and DaemonSets do.
Principle VI. Processes
Principle 6 of the 12 Factor App is "Execute the app as one or more stateless processes".Stateless processes are a core idea behind cloud native applications. Every twelve-factor application needs to run in individual, share-nothing processes. That means that any time you need to persist information, it needs to be stored in a backing service such as a database.
If you're new to cloud application programming, this might be deceptively simple; many developers are used to "sticky" sessions, storing information in the session with the confidence that the next request will come to the same server. In a cloud application, however, you must never make that assumption.
Instead, if you're running a Kubernetes-based application that hosts multiple copies of a particular pod, you must assume that subsequent requests can go anywhere. To solve this problem, you will want to use some sort of backing volume or database for persistence.
One caveat to this principle is that Kubernetes StatefulSets can enable you to create Pods with stable network identities, so that you can, theoretically, direct requests to a particular pod. Technically speaking, if the process doesn't actually store state, and the pod can be deleted and recreated and still function properly, it satisfies this requirement -- but that's probably pushing it a bit.
Principle VII. Port binding
Principle 7 of the 12 Factor App is "Export services via port binding".In an environment where we're assuming that different functionalities are handled by different processes, it's easy to make the connection that these functionalities should be available via a protocol such as HTTP, so it's common for applications to be run behind web servers such as Apache or Tomcat. Twelve-factor apps, however, should not be dependent on an additional application in that way; remember, every function should be in its own process, isolated from everything else. Instead, the 12 Factor App manifesto recommends adding a web server library or something similar to the app itself, By using port binding, 12 factor app services can await requests on a defined port, whether it's using HTTP or another protocol.
In a Kubernetes-based application, this is done partly through the architecture of the application itself, and partly by making sure that the application has all of its dependencies as part of the creation of the base containers on which the application is built.
Principle VIII. Concurrency
Principle 8 of the 12 Factor App is to "Scale out via the process model".When you're writing a twelve-factor app, make sure that you provide a way for it to be scaled out, rather than scaled up. That means that in order to add more capacity, you should be able to add more instances rather than more memory or CPU to the machine on which the app is running. Note that this specifically means being able to start additional processes on additional machines, which is, fortunately, a key capability of Kubernetes.
Principle IX. Disposability
Principle 9 of the 12 Factor App is to "Maximize robustness with fast startup and graceful shutdown".It seems like this principle was tailor made for containers and Kubernetes-based applications. The idea that processes should be disposable means that at any time, an application can die and the user experience won't be affected, either because there are others to take its place, because it'll start right up again, or both.
Containers are built on this principle, of course, and Kubernetes structures that manage multiple instances and maintain a certain level of availability even in the face of problems, such as ReplicaSets, complete the picture.
Principle X. Dev/prod parity
Principle 10 of the 12 Factor App is "Keep development, staging, and production as similar as possible".This is another principle that seems like it should be obvious, but is deeper than most people think. On the surface level, it does mean that you should have identical development, staging, and production environments, inasmuch as that is possible. One way to accomplish this is through the use of Kubernetes namespaces, enabling you to (theoretically) run code on the same actual cluster against the same actual systems while still keeping environments separate. In some situations, you can also use tools such as Minikube or kubeadm-dind-cluster to create near-clones of production systems.
At a deeper level, though, as the Twelve-Factor App manifesto puts it, it's about three different types of "gaps":
- The time gap: A developer may work on code that takes days, weeks, or even months to go into production.
- The personnel gap: Developers write code, ops engineers deploy it.
- The tools gap: Developers may be using a stack like Nginx, SQLite, and OS X, while the production deploy uses Apache, MySQL, and Linux.
Some of these factors are outside of the realm of Kubernetes, of course; the personnel gap is a cultural issue, for example. The time and tools gaps, however, can be helped in two ways.
For the time gap, Kubernetes-based applications are, of course, based on containers, which themselves are based on images that are stored in version-control systems, so they lend themselves to CI/CD. They can also be updated via rolling updates that can be rolled back in case of problems, so they're well-suited to this kind of environment.
As far as the tools gap is concerned, the architecture of Kubernetes-based applications make it easier to manage, both by making local dependencies simple to include in the various images, and by modularizing the application in such a way that external backing services can be standardized.
Principle XI. Logs
Principle 11 of the 12 Factor App is to "Treat logs as event streams".While most traditional applications store log information in a file, the Twelve-Factor app directs it, instead, to stdout as a stream of events; it's the execution environment that's responsible for collecting those events. That might be as simple as redirecting stdout to a file, but in most cases it involves using a log router such as Fluentd and saving the logs to Hadoop or a service such as Splunk.
In building a 12 factor app using Kubernetes, you have at least two choices for logging capture automation: Stackdriver Logging if you're using Google Cloud, and Elasticsearch if you're not. You can find more information on setting Kubernetes logging destinations here.
Principle XII. Admin processes
Principle 12 of the 12 Factor App is "Run admin/management tasks as one-off processes".This principle involves separating admin tasks such as migrating a database or inspecting records from the rest of the application. Even though they're separate, however, they must still be run in the same environment and against the same base code and configuration as the application, and their code must be shipped alongside the application to prevent drift.
You can implement this a number of different ways in Kubernetes-based applications, depending on the size and scale of the application itself. For example, for small tasks, you might use kubectl exec to operate on a specific container, or you can use the Kubernetes Job to run a self-contained application. For more complicated tasks that involve orchestration of changes, however, you can also use Kubernetes Helm charts.
How many of these factors did you hit?
Unless you're still working on desktop applications, chances are you can feel good about hitting at least a few of these essential principles of a 12 factor apps using Kubernetes. But chances are you also found at least one or two you can probably work a little harder on.So we want to know: which of these best practices are the biggest struggle for you? Where do you think you need to work harder, and what would make it easier for you? Let us know in the comments.
Thanks to Jedrzej Nowak, Piotr Shamruk, Ivan Shvedunov, Tomasz Napierala and Lukasz Oles for reviewing this article!
Check out more Kubernetes resources.
