Benchmarking OpenStack at megascale: How we tested Mirantis OpenStack at SoftLayer

It seems like one of the leading amateur sports in cloud is to ask if OpenStack is ready for Enterprise ‘prime time’ in production. Common water-cooler conversations ask about stability and performance of OpenStack-based clouds at scale. But what does ‘scale’ mean? What level of scale is relevant in the real world?

From our work with many enterprise-class OpenStack customers, we’ve observed that ‘large private cloud’ generally refers to something that runs into tens of thousands of VMs. Hundreds of users from hundreds of projects manipulate those VMs simultaneously. At this scale, I think it’s fair to say, a ‘private cloud’ essentially becomes a ‘virtual data center’.

A virtual data center has some meaningful advantages compared to a classic physical datacenter, not least in flexibility. The time between the customer's order and having the requested infrastructure available drops from days/hours to minutes/seconds. On the other hand, you need to be confident about your Infrastructure-as-a-Service platform in terms of stability under the load.

Scaling Mirantis OpenStack on Softlayer

We set out to validate desired behaviors in Mirantis OpenStack under loads you would expect to see in just such a virtual datacenter. Working with our friends at IBM Softlayer, we carved out an infrastructure of 350 physical machines in two locations. And for starters, we wanted to establish some baseline for an SLA for virtual data centers powered by Mirantis OpenStack.

Obviously, size matters. But there were other questions about an OpenStack virtual datacenter at Softlayer that we thought were important to answer right out of the gate. For example:

• Could the VDC could service hundreds of requests at the same time?

• Would the user experience suffer as more virtual servers are added in the virtual datacenter?

To find the answer to the first question, we’d need a series of tests with increasing numbers of requests issued simultaneously. Our target concurrency for this phase of benchmarks was 500. We approached the second question by refining it to gauge how long it takes to provision a server in our virtual datacenter, and, more importantly, how duration depends on the number of existing servers.

Based on expectations listed above, we designed first set of benchmarks. This first set is aimed to exercise the Mirantis OpenStack Compute infrastructure, the key component of the platform. The exercise targeted not only the Nova API and scheduler, but Compute agents on hypervisor servers as well.

Because we're benchmarking deployment time, measuring the scalability of the OpenStack Compute service doesn't require creating any load inside the VMs, so we can use an OpenStack flavor with a small footprint, a small image to boot into this flavor and no tasks executed inside the VM. For default flavors (or instance types) defined in OpenStack, the size of the instance does not have a significant effect on provisioning time. However, we’d need to test on real hardware to have a clear understanding of what the Nova Compute agent could handle under load.

Baseline SLA parameters we wanted to verify were:

• variance of virtual server instance boot time, including any dependencies between the time and the number of virtual instances in the cloud database;

• success rate of the boot process for our instances, in other words, how many requests out of 100 are expected to fail for one reason or another.

We set a target for this benchmark at 100 thousand virtual instances running in a single OpenStack cluster. With such a small footprint flavor, our experiment could be compared to a micro-site service for mobile applications. Another possible use case for this type of workload is a testing environment for a mobile application server with thousands of clients running on independent embedded systems. It was our hypothesis that we could figure this out with several hundred servers -- generously provisioned by Softlayer -- for a limited period of time as our test bed.

Setting up an OpenStack benchmarking environment

“How much hardware do you need?” is a question we often hear, and we often joke with customers that the answer is “More.” But it was clear that running a hundred thousand VMs required sufficient hardware firepower. IBM Softlayer has a pretty broad variety of configurations available, and they were as curious as we were to see what OpenStack could deliver at this level.

To calculate hardware resources needed, we first established a type of instance, or ‘flavor’, in OpenStack terms. Our flavor has 1 vCPU, 64MB of RAM and 2GB of disk, much smaller than a m1.micro AWS type, for example. This is just enough to boot a Cirros test image.

Using a very small instance type doesn't impact results of the test because for the default flavors (or instance types) defined in OpenStack, the size of the instance does not have a significant effect on provisioning time. This is because those flavors only define the number of CPUs, the amount of RAM, and the disk file size. Those resources are usually provisioned with effectively zero variance based on instance size, because unlike in an AWS instance, which also involves provisioning additional services such as EBS, OpenStack instances are self-contained; creating ephemeral storage is accomplished by essentially creating an empty file, for example. For this reason, we can use a very small instance type without skewing the test results.

Totals for resources are:

• 100,000 vCPUs,

• 6250 Gb of RAM,

• 200TB of disk space.

So we concluded that a good representative compute server configuration for this test would include 4 physical cores (+4 for HT), 16Gb of RAM and 250GB of disk space. Because we're not going to perform any actual load on the VMs, we can comfortably overcommit CPU as 1:32 (twice the default overcommit rate for OpenStack) and RAM as 1:1.5, which allows us density of 250 virtual servers per physical node. With these non-standard overcommit rates, we only needed approximately 400 hardware servers to hit the target of one hundred thousand VMs.

Knowing that our controllers would be handling the heaviest load, we spec’d servers for those with a more advanced configuration. Each controller had 128GB of RAM and SSD storage to accelerate the work of the OpenStack state database that runs on the controller.

One interesting note: due to differences in configuration between Compute and Controller nodes, they were provisioned in different SoftLayer datacenters with 40ms latency between them; it’s not unusual to expect that in a real cluster, you might have exactly this kind of distributed configuration. Fortunately, due to low latency between the Softlayer datacenters, no significant effect of this distributed architecture was noticed in the benchmark.

OpenStack details

We took release 4.0 of the Mirantis OpenStack product, which includes OpenStack 2013.2 (Havana) and deployed it to servers provided by SoftLayer. To do this, we had to tweak our Deployment Framework: because we could not boot off the Mirantis OpenStack ISO in SoftLayer, we just copied the needed code via rsync over the standard operating system image.  Once we'd deployed the cluster, we could use it for benchmarking.

Our configuration included only three OpenStack services: Compute (Nova), Image (Glance) and Identity (Keystone). This provides basic functionality to the Compute service, which is sufficient for our use cases (mobile platform testing).

For networking, we used the Nova Network service in FlatDHCP mode with the Multi-host feature enabled. It has total feature parity with Neutron in our simple use case (single private network, FlatDHCP network and no floating IPs).  (Nova Network was recently restored from deprecated status in OpenStack upstream.)

During the benchmark, we had to increase certain parameters for the default OpenStack configuration. The most important change is the size of the connection pool for the SQLAlchemy module (max_pool_size and max_overflow parameters in the nova.conf file). In all other aspects we leveraged the standard Mirantis OpenStack HA architecture, including synchronously replicated multi-master database (MySQL+Galera), software load balancer for API services (haproxy) and corosync/pacemaker for IP level HA.

Rally benchmark engine

To actually run the benchmark test, we needed a test engine; the logical choice was the Rally project for OpenStack. Rally automates test scenarios in OpenStack; i.e., it enables you to configure steps to be performed for each test run, specify the number of runs, and record the time to complete each request. With these metrics, we could calculate the parameters that needed verification: boot times variance and success rate.

We already had experience using Rally on SoftLayer, so that helped us a lot in this benchmark.

The benchmark configuration looks something like this:

{
    "execution": "continuous",
    "config": {
        "tenants": 500,
        "users_per_tenant": 1,
        "active_users": 250,
        "times": 25000
    },
    "args": {
        "image_id": "0aca7aff-b7d8-4176-a0b1-f498d9396c9f",
        "flavor_id": 1
    }
}