5 Steps to OpenStack success: Benchmarking Heat with Rally
Adoption of OpenStack products and technology is growing at a breathtaking pace. Every day new contributors join the OpenStack community. This results in continuous updates, in terms of both new features and performance improvements. If you’re reading this, you probably already know that OpenStack is really a huge ecosystem of cooperative services, which can make troubleshooting especially hard in the event of a system failure or performance and scaling issues. Rally is a tool designed to help address these issues.
For a full overview of Rally and how it works you can check out the Rally wiki page at https://wiki.openstack.org/wiki/Rally, or get the code itself from stackforge at https://github.com/stackforge/rally. In short, however, Rally is a tool for verifying and benchmarking OpenStack clouds. “Benchmarking” in case of Rally means: stress, scalability, performance and regression testing of OpenStack clouds.
In this post, we'll look at using Rally to benchmark OpenStack Heat. Heat is a service that orchestrates multiple composite cloud applications using templates, through both an OpenStack-native ReST API and a CloudFormation-compatible Query API. Heat's position as the base for all OpenStack PaaS solutions means that it is crucial to be 100% sure that this component works well under high load and at scale.
This tutorial covers Rally's basic concepts and explains how to use Rally in daily life, showing you how you can get a report that evaluates a Heat operation's performance under load in just five steps.
Step 1: Install Rally
Installing Rally itself is straightforward, and takes only two commands:
git clone https://github.com/stackforge/rally.git cd rally && sudo ./install_rally.sh
This results in a series of log entries that should look something like the following:
Figure 1. Sample installation log output.
Next you'll need to register your cloud.
Step 2. Register the OpenStack cloud with Rally
To register your OpenStack cloud, Rally will need your credentials. You can pass them using local environment variables, or via a specific file. Let's look at each of these options.
Using local environment variables
To use environment variables, set up the OpenStack environment variables by running these commands (substituting your own values, of course!):
export OS_USERNAME=”admin”
export OS_TENANT_NAME=”admin”
export OS_PASSWORD=”admin”
export OS_AUTH_URL=”http://your_cloud.url:5000/v2”
Next, register the OpenStack cloud specified by those variables with Rally. To do that, run following command:
rally deployment create --name test --fromenv
This command uses the existing environment variables to create a new deployment and updates the ~/.rally/openrc file with all environment variables. So if you need to use standard OpenStack python clients you can just “source” this file and use them:
source ~/.rally/openrc
nova list # Will return list of Active VMs
Figure 2. Sample deployment create output using local environment variables.
If, on the other hand, you want to create a deployment based on an OpenStack cluster other than the one specified by your current environment variables, you can use a json file as input.
Registering OpenStack cloud using input file
To register an arbitrary OpenStack cloud -- independant of your current environment variables --you can use an input file. To do that, create a json file that lists all of the variables in this format:
{
"type": "ExistingCloud",
"auth_url": "http://example.net:5000/v2.0/",
"admin": {
"username": "admin",
"password": "admin",
"tenant_name": "admin"
}
}Now you should just run the same rally deployment create command, specifying the file rather than running from the environment:
rally deployment create --name test --filename existing.json
You should see something similar to the following:
Figure 3. Sample deployment create output using the json configuration file.
Now that your deployment has been created, you can make sure everything is in place.
Step 3. Check availability of the registered OpenStack cloud
After registering your OpenStack cloud, the first thing to do it is to check to make sure everything's working properly. Otherwise, everything else you'll do will just be beating a dead horse. To check the most recent deployment, execute this command:
rally deployment check
If everything is working properly, you'll see output similar to the following:
Figure 4. Sample deployment check output.
As you can see, the status for each service is listed in the table. In this case, Rally is set up properly. If there's a problem, the deployment check will tell you what it is. For example, you might see a message such as:
Figure 5. Sample deployment check output with error.
In this case, the credentials of the registered cloud are incorrect. For historical reasons, Rally doesn't allow you to change the credentials of a registered cloud, so you’ll have to register the same cloud again, but this time with proper credentials. ;)
Step 4. Finally! Let’s start benchmarking!
Once you're satisfied that everything works, you're ready to benchmark your cloud. To examine your cloud, let’s run a Rally task. A Rally Task is a set of benchmarks that will be run against a cloud. Before launching the task, let’s take a look at the format of the task input file.
Rally is pretty flexible, and can test a lot of different scenarios, so sometimes the easiest way to run a benchmark is to copy its task from the sample repo in stackforge: https://github.com/stackforge/rally/tree/master/doc/samples/tasks/scenarios
All task input files have a similar structure:
---
ScenarioClass.scenario_method:
-
args:
...
runner:
...
context
...
sla:
…
ScenarioClass.scenario_method is the full name of the benchmark scenario.
Section "args" is used to pass the parameters to the benchmark scenario method.
(For example, what flavor and image to use for booting the VMs.)Section "runners" specifies the load type that should be generated.
(For example, "run this scenario every 1 second".)Section "context" defines the benchmark environment that should create Rally.
(For example, "create 10 users in 5 tenants and set up their quotas".)Section "sla" defines success criteria for the benchmark.
(For example, the maximum percentage of failures.)
Let's choose HeatStack.create_and_delete_stack benchmark, which you can find in the heat directory. It contains the following definition:
---
HeatStacks.create_and_delete_stack:
-
runner:
type: "constant"
times: 10
concurrency: 2
context:
users:
tenants: 2
users_per_tenant: 3
This benchmark scenario consists of a sequence of basic actions:
create stack - the action sends a request to create a stack and wait until this operation is accomplished.
delete stack - the action sends a request to delete a stack and wait unit this operation is fully accomplished.
This benchmark scenario will be run by 6 temporary created users from 2 tenants (2 tenants x 3 users_per_tenant) 10 times in total. At any given moment, only 2 users will be running this scenario simultaneously.
To run this benchmark, you can simply use:
rally task start create-and-delete-stack.yaml
A successful test result looks like this:
Figure 6. Sample output of create_and_delete_stack benchmark task.
As you can see, Rally's results contains a lot information about the performance of Heat -- not only the “duration” of a single user scenario, but also detailed information about every atomic action (such as create_stack and delete_stack). Also in table we have the "success" column, which enables you to determine whether this particular OpenStack deployment can handle the type of load specified in the task.
But those results are just simple a representation; it gets even better in step 5.
Step 5. Getting Graphics & Charts!
Benchmarking is great, but to be really useful we need to present results in easy understandable form, such as graphics and charts. Fortunately, with Rally you can generate pretty reports with a simple and single, but very powerful command:
rally task report <task_id> --out report.html
The following is an example of a Rally generated report:
