Cloud, Drivers and OpenStack DriverLog
OpenStack is about having the freedom to exploit new technology and solutions quickly. You put that in practice by evolving and optimizing your cloud’s performance, features and cost as fast as the cooperative efforts of hundreds of vendors and thousands of contributors can make possible. But doing this with confidence means making decisions about, and investments in hardware, software and systems — and that means having meaningful, detailed and trustworthy information about OpenStack drivers: the logical glue connecting third-party products with OpenStack components.
Let’s first understand what the problem is and why it matters. Then, in Part 2 below, we’ll look at how it works and how to use it.
Why DriverLog?
To proceed confidently with engineering, platform acquisition and scale-up, customers and the community need answers about drivers:
Are drivers available to make a specific distribution or version of OpenStack, or an OpenStack component (e.g., Neutron-network) work with a specific storage appliance or networking solution?
Are available drivers being reliably tested under continuous integration?
Who stands behind the assertion that a given driver will solve a specific integration problem?
There are various strategies out there for doing this work and providing these assurances. The classic approach is for one central party to ‘own’ and administer a testing, certification and trusted distribution process. Centralized solutions, however, imply some degree of centralized authority; they can undermine transparency. What’s almost as bad is that they tend to be slow: no single entity has the bandwidth, universal expertise or resources to keep driver testing on-pace with releases as project diversity and scale continue to grow.
OpenStack DriverLog
For this reason, we at Mirantis think that the community itself may be able to provide a solution to the problem of driver/version proliferation: one can be more-efficient, timely, transparent and trustworthy solution. DriverLog is an open source project that aims to provide:
- A single, consolidated source of truth on all third party OpenStack vendor drivers
- Access to instructions on how to configure any vendor's product with OpenStack.
- Information on whether a vendor's driver is embedded in the upstream codebase of an OpenStack release.
- Information on whether a vendor has deployed an automated test infrastructure that provides feedback on the vendor's driver passing or failing functional and integration tests.
- Quick shortcuts for OpenStack deployers to contact maintainers of vendor drivers in case they run into issues.
The DriverLog Dashboard(s)
The DriverLog back end is an open source project hosted on stackforge. Details on vendor drivers are aggregated in a single json file and anybody is free to use that data and expose it in any way via a web dashboard. At Mirantis we’ve built such a dashboard as a part of stackalytics.com. OpenStack foundation is doing the same to populate the driver section of OpenStack marketplaces. Driverlog dashboard implementation at stckalytics.com lets you filter drivers by relevant OpenStack release, project and vendor (e.g., Icehouse + Neutron (Networking) + Midokura) as well as search on keywords. Results can be sorted by column and display project, vendor, driver description, trunk compatibility data (e.g., Grizzly, Folsom, Havana, Icehouse), note whether the driver is CI tested, and identify the maintainer.
Simplicity is really the point of this front-end. Driver info needs to be searchable quickly and reliably, from any device — and there’s no need to restrict access to this information (the dashboard is read-only). The in-progress specification for DriverLog therefore includes an easy, open-web UI that can be used by anyone.
Who Benefits from DriverLog?
This initiative has three beneficiaries.
For vendors, DriverLog initiative removes some of the burden of asserting and promoting driver compatibility with different OpenStack releases by defining and aggregating one common and measurable version of Truth.
The community at large will find, in DriverLog, a level, open playing-field for maintaining information about vendor drivers in a centralized place.
Customers — probably the biggest beneficiaries — will gain from being able to quickly access accurate and up-to-date information about OpenStack drivers developed and maintained by a variety of vendors.
Ultimately, the result — a benefit for everyone — will be a more-reliable and more-open OpenStack.
How is DriverLog Maintained?
DriverLog integrates with OpenStack’s standard Gerrit revision-tracking workflow in a very simple way, by using Gerrit to manage insert/update/delete operations on the DriverLog “database” - a simple JSON file (default_data.json) containing driver information, maintainer contact info, supported releases and status on integration with OpenStack CI. The JSON file is patched by its maintainers as necessary. It’s committed, reviewed and approved using the standard OpenStack workflow, and updated automatically. The DriverLog back end connects the dots between the vendor’s external CI system and Gerrit/OpenStack Code Review, insuring that CI tested status is always kept updated in the JSON file.
Using static JSON here, instead of a conventional database, makes it easier to create alternative front-ends or integrate this valuable driver information with other tools. It also makes life easier for driver maintainers, who can commit changes directly to the JSON file by submitting driver listings and updating existing driver info (including the CI tested mark). New commits are reviewed and then merged into the live file by a DriverLog Core Engineer. Again, the system works just like all other OpenStack development, offering the same transparency and cooperative workflow. Since driverlog launch, over a dozen companies have helped contribute to the project in submitting updates and patches:
“CI tested” - Continuous Integration for OpenStack
To help establish a standard, the “CI tested mark” reflects that a driver has been produced under Continuous Integration with OpenStack, and has been subjected to automated unit-testing with Tempest, the OpenStack test-suite. Vendors are encouraged — (if they haven’t already done so) to set up an external OpenStack CI system, preferably with the Jenkins open source CI server. Documentation on setting up CI is already available in several documents as follows:
(Note that this is not entirely a vendor-driven exercise; if you consume OpenStack components, you can encourage your vendors to simplify your uptake of their latest and greatest by making this system part of your purchase/acceptance criteria.)
Once the CI system is set up, it can be configured to incorporate the driver in local builds and then submit these to automated unit-testing using the Tempest test suite. Depending on results, the vendor can then submit a patch on the DriverLog JSON datafile, noting the driver’s promotion (to “CI tested” status) and then keeping their CI/test process synchronized with the branch or branches they want to follow.
In the process of doing that merge, the results of testing performed by the vendor’s external CI system are stored in Gerrit by the DriverLog back-end, which runs a batch job every four hours which:
Identifies which Gerrit reviews relate to this driver (using the driver’s CI environment ID (ci_id).
Parses all merged patches from new to old.
Identifies when a given ci_id and branch appear together for the first time and stores the ID and branch in the JSON file, along with links to
test remarks stored in OpenStack Code Review and,
via the ‘patch comment’ field in Code Review, to full test results stored on the vendor’s external CI system.
This way, the DriverLog front-end can display current and accurate driver CI status. It also enables drill-downs, giving access to basic merge info and fine-grained vendor-hosted test results. The system seems admirably simple and harmonious with other contributor and engineer workflows. It offers reasonable objective oversight, both by community contributors and DriverLog Core Engineers. It’s timely and automated. And it provides visibility that should give customers good assurance of the validity of claims.
With this background in place, let’s now move to how this works.
Getting Started with DriverLog
In this first stage, Mirantis is coordinating the DriverLog project, working with PTLs and driver maintainers. We are all trying to make participation as frictionless as possible for vendors and other interested community members.
Some minimal constraints are needed, at this stage, to limit workloads. Since the largest number of third-party drivers are developed for Nova, Cinder, Neutron and Sahara, the first phase of DriverLog will focus on drivers for these components.
To populate the DriverLog database, information about specific drivers was gathered from project wiki pages:
Vendors involved have been extremely supportive in validating the correctness of source information.
Initially, most drivers will be listed without the CI tested mark, to be promoted to CI tested status in due course. To get promoted to CI tested vendors have to a) an external CI test harness, as described in instructions below; b) update DriverLog records
To simplify things, below is the really quick summary of the content from the 4 documents above.
How to add a driver and get a CI tested mark?
The DriverLog process makes it fairly simple for vendors to enter new drivers and update driver listings. The CI tested mark is then gained through successful automated test-suite execution in tandem with a vendor’s external continuous integration setup.
Including Drivers
A vendor can include its driver like this:
The vendor adds a new section to the JSON “database,” committing a patch. For example:
{
"project_id": "openstack/cinder",
"vendor": "Ceph", "name": "Ceph RADOS Block Device (RBD)", "description": "The Ceph RADOS Block Device driver allows to use Ceph RADOS block devices (RBD) for volumes.", "maintainers": [ { "name": "Josh Durgin", "email": "josh.durgin@inktank.com ", "launchpad_id": "jdurgin" }, { "name": "Mike Perez", "launchpad_id": "thingee" }, { "name": "Edward Hope-Morley", "launchpad_id": "edward-hope-morley" } ], "releases": ["Cactus", "Diablo", "Essex", "Folsom", "Grizzly", "Havana", "Icehouse"] },
The configuration file is automatically validated against the config file scheme.
An DriverLog Core Engineer receives a request to review a commit.
The engineer reviews and, if all is well, merges the update.
The DriverLog dashboard displays the new information.
CI tested Status
Vendors can set up a Jenkins CI server and test their drivers with Tempest in synch with target branches. Once Tempest has run, the vendor can update information about their driver in the DriverLog database and promote it to ‘CI tested,’ like this:
The vendor patches the DriverLog configuration file, changing the verification level on their driver to ‘external_ci_verification’ and including a ci_id:
{"project_id": "openstack/neutron","vendor": "Big Switch","name": "Big Switch Neutron ML2 Driver","description": "An ML2 Mechanism Driver to integrate ML2 deployments into Big Switch/Floodlight controlled networks. This driver only controls physical infrastructure so it is meant to be combined with the openvswitch mechanism driver.","maintainers": [{"name": "Kevin Benton","irc": "kevinbenton","email": "kevin.benton@bigswitch.com","launchpad_id": "kevinbenton" }],"wiki": "http://docs.openstack.org/trunk/config-reference/ ",content/networking-plugin-ml2_ bigswitch.html "ci": {"id": "bsn","success_pattern": "BigSwitch-ML2-Driver \\S+ : SUCCESS", "failure_pattern": "BigSwitch-ML2-Driver \\S+ : FAILURE" },"releases": ["Icehouse"]},
An DriverLog Core Engineer reviews, approves, and merges the update.
The actual results of driver testing are stored in Gerrit. The DriverLog back-end uses ci_id to determine which Gerrit reviews are related to this driver.
The DriverLog Dashboard displays the driver’s new status and provides a drill-down:
By clicking on [Verification Status] users can see an OpenStack Code Review page. The Review page contains the comment from CI with links to log files.
