Mirantis OpenStack: real open community development
When you talk about a company "doing open source," people frequently assume that employees build the product in-house, then release the source code to the community. When it comes to Mirantis and OpenStack, however, the situation is just the opposite. We start with upstream OpenStack and build our product through a combination of testing, hardening patches, and additional open source features that make it easier to use without creating vendor lock-in.
Once you add a commercial distribution such as Mirantis OpenStack to the mix, things can seem complicated -- particularly given that every piece of OpenStack code we work on is open source. Let's take a look at how this process works so you can understand which parts of Mirantis' work are, and are not, open.
How Mirantis works in the community
Although it would be simple to assume that Mirantis builds code in-house and then sends the result into the community, that's not, in fact, how it works.
To succeed in open source, a company must first nurture the community on which it aims to build its business. For Mirantis, that's OpenStack. That means that when we work on new features, we do them in public, in the community. When we find bugs, we fix them in public, in the community. The goal is to make OpenStack as strong as possible, so that we can provide the best value for our customers using it.
Mirantis OpenStack has zero proprietary code in it.
Let me say that again.
Mirantis OpenStack has zero proprietary code in it.
Of course, that doesn't mean that Mirantis OpenStack is identical to the OpenStack trunk; far from it. When we package up Mirantis OpenStack, it includes the following:
The latest stable branch of OpenStack
Hardening packages that include fixes for issues that we discover in testing
Select additional bug fixes that may or may not have been merged/backported by the community yet
Let's dig down a little and get some more details.
Mirantis OpenStack components
When you download Mirantis OpenStack, you get OpenStack code with multiple components that work together for optimized functionality.
OpenStack packages: The Mirantis OpenStack distribution is based on OpenStack project packages from the latest stable project release; for example, Mirantis OpenStack 6.x release is based on the OpenStack Juno release.
Fuel: Fuel is a deployment and management tool for OpenStack, related community projects, and plugins, enabling provisioning, deployment, and lifecycle management.
Continuous Integration (CI) Infrastructure: Mirantis OpenStack includes components to implement and test your OpenStack environment, such as OpenStack unit tests, Fuel unit and system tests, and the Rally and Tempest projects.
Mirantis OpenStack Express service: Mirantis OpenStack Express enables you to build your hybrid OpenStack cloud and add compute and data capacity on demand. This service provides a self-service portal, hardware on demand, and, of course, documentation.
All of these components are based on work from the Mirantis engineering team as well as from the OpenStack community, and virtually all of it is open source, as you can see from Figure 1:
Figure 1. Mirantis OpenStack Components
In Figure 1, anything in green is open source, while items in blue are not. The latter include Mirantis OpenStack Express components and hardening patches for OpenStack packages.
While the Mirantis OpenStack Express self-service portal and hardware-on-demand service are essential to the functioning of Mirantis OpenStack Express, they aren't actually part of OpenStack; they just enable us to provide OpenStack as a service.
Hardening patches for OpenStack packages is where a lot of people get tripped up. The actual patches that we use for hardening OpenStack are closed, but all of the code provided in them is open. (We'll explain how that's possible--as well as why we do it this way--in a moment.)
Let's take a look at how all of this works in practice.
Fixing bugs
There are two ways we might discover a bug; we might discover it as part of our testing and development, or we might discover it while resolving a customer support request. In either case, the process is the same:
Check to see if a bugfix already exists on the master branch of the affected component. If so, we submit a backport of the fix to the corresponding stable release branches in our internal fuel-infra repo.
If there is no existing bugfix, the Mirantis OpenStack components team:
Reports the bug to the affected OpenStack component (if it has not already been reported).
Documents the relation of the upstream bug in a comment or description of the corresponding Mirantis OpenStack bug report on Launchpad (in the Mirantis OpenStack project).
Proposes a fix for the master branch of the public git repository on review.openstack.org, and gathers initial feedback from the community to make sure it's on track.
Ports the fix to the stable release branches in a private git repository on review.fuel-infra.org.
At this point, the bug fix becomes available to our customers while we wait for it to be accepted and merged by the OpenStack community.
We follow a similar process when we discover bugs in Linux packages such as KVM, libvirt, Open vSwitch, and even the Linux kernel, as well as for Python dependencies of OpenStack components, such as kombu, and tools and libraries required for Fuel, such as, MCollective, Puppet, and Cobbler. One significant distinction here, however, is that the git repository on review.fuel-infra.org is public, not private.
We track bugs in Fuel and its Python and Ruby dependencies in the Fuel project on Launchpad. Bugs in all other packages are tracked in the Mirantis OpenStack project.
Hardening patches for OpenStack packages
Mirantis stays as close as possible to upstream OpenStack code, and as I said, we commit 100% of our bug fixes and patches upstream. However, reviewing those patches and getting them included in a stable OpenStack release takes time, so Mirantis provides some bug fixes to you, our users, ahead of the upstream to allow you to take advantage of them immediately.
While the source code of all software packages in the Mirantis OpenStack distribution is open, the breakdown of the difference between the code shipped in Mirantis hardened packages and the corresponding public stable release of OpenStack into individual patches and their relation to bugs (and other reasoning behind specific code changes) constitutes the core of Mirantis intellectual property, and is closed.
So what does that mean? Well, consider the thousands of patches that are submitted to OpenStack each release cycle. Because we don't add proprietary extensions to Mirantis OpenStack, you know that every bit of code we ship is included in one of those patches. However, we don't necessarily include every single patch in Mirantis OpenStack.
The "hardening patches" are simply the log that shows which specific bug fixes we chose to apply, why, and in what order -- the record of how to get from the community stable branch to Mirantis OpenStack.
In other words, hardened packages are the log of our build of working OpenStack packages, a build that is designed for successful production deployment. Not all upstream patches are created with production deployment in mind; it's that selection and application process, established through testing and review of patches and bugs, that distinguishes Mirantis OpenStack from any arbitrary build or application of patches.
Which Mirantis OpenStack components are open
All other Mirantis OpenStack product components, including Mirantis patches for non-OpenStack packages, are available to the general public and, where applicable, released under open source licenses, with Apache License 2.0 preferred. Some examples include:
Source code and specs for Python libraries and Linux packages
Why do we do this? Because we recognize that open sourcing these components is a benefit to both the wider community and to the Mirantis product engineering team. For example, when Mirantis publishes the source code for Python libraries and Linux packages, users can build an ISO easily, without hacking to get the correct versions of third-party Python libraries. When users contribute bug fixes to the OpenStack master branch, the Mirantis OpenStack team can spend less time porting bug fixes across OpenStack releases. It also means our engineers don't have to spend time porting bug fixes between Mirantis OpenStack and the "vanilla" version.
The lifecycle of changes going into Mirantis OpenStack is summarized in Figure 2:
Figure 2. Mirantis OpenStack changes lifecycle
Once again, green boxes represent code repositories and issue trackers that are open to the community, blue boxes represent trackers and repositories that are only available to Mirantis employees.
In particular, we use closed git repositories to track the hardening patches for OpenStack components, and a closed support portal to protect confidential customer information.
While we keep the intermediate builds of Mirantis OpenStack private until we have a build that our Quality Assurance team has approved for general availability, we publish builds based on the same code with the Mirantis branding removed. This way, users can experiment with the features from the future releases of Mirantis OpenStack, and yet can count on our quality standards and our support when deploying production environments using officially branded Mirantis OpenStack release builds.
Handling security vulnerabilities
Mirantis follows the responsible disclosure rules for non-zero-day security vulnerabilities. That means that embargoed vulnerabilities are reported and fixed in private, and not published until the disclosure date coordinated with other software vendors. Security-related bugs in Fuel and Mirantis OpenStack projects on Launchpad are marked private until the fix is published via the Mirantis OpenStack Technical Bulletins Process. Security vulnerabilities in base operating system packages are resolved based on the corresponding Linux distribution's security announcements and security updates channels.
Package maintenance
Although base operating systems such as CentOS and Ubuntu include RPM and .deb packages for most OpenStack and Fuel dependencies, not all of these packages have the versions required by Mirantis OpenStack. To solve this problem, the Mirantis components team often builds RPM and .deb packages of specific versions using available sources.
This process is completely open. The teams responsible for specific packages for Fuel, Mirantis OpenStack components, and Mirantis OpenStack Linux produce both code and package specs for their packages. The Fuel OpenStack continuous integrations team creates all required git repos, gerrit, and branches.
Understanding the coordination of Mirantis OpenStack code
As you can see, Mirantis works within the community to make OpenStack as strong as possible, while still providing the best possible options and creating new solutions for our customers. Mirantis is committed to making our code completely open and uses multiple methods to get you the quickest updates while working within the upstream community.
