OpenStack Project Technical Lead Interview Series #2: Monty Taylor, OpenStack CI(Continuous Integration) Project

Here’s the second in our series of interviews with OpenStack Project Technical Leads on the Mirantis blog. Our goal is to educate the broader tech community and help people understand how they can contribute to and benefit from OpenStack. Naturally, these are the opinions of the interviewee, not of Mirantis. We’ve edited the interview for clarity and post length.

Our second interview is with Monty Tayor, OpenStack CI (Continuous Integration) Project Technical Lead.

Mirantis: Can you please introduce yourself?

Monty Taylor: I work for Hewlett Packard on OpenStack, I sit on the OpenStack Foundation Board as well as on the OpenStack Technical Committee, and I run the OpenStack CI and Infrastructure Project.

Q: What is your history with OpenStack?

A: I was at Rackspace in the team that originally got the OpenStack project up and going.  So I've been around since the beginning. You can still see my name attached to some of the Launchpad projects, because I helped set those up, and Launchpad doesn't let you change those things.

Q: Why do you engage with OpenStack?

A: I absolutely believe in the importance of having an open cloud, with everyone moving into managed services and cloud applications.

Q: What are your responsibilities as the Project Technical Lead?

A: The infrastructure team is a very self-driven group, a bunch of independent-minded folks. Rather than actually telling anyone on the team that they should do this or do that, we work collaboratively. In terms of leading, it's really more about being an ambassador, an outward interface for people. I talk with various companies who are giving us resources and I am a face as much as I can be for the effort.

Q: Can you explain what Continuous Integration (CI) is? And why does it matter for OpenStack?

A: We started the project with the idea of a gated trunk. Rather than having a human responsible for merging code into the development trunk, which is actually a very boring job, we wanted to have that done automatically.

One of the ways we do that is that we run tests on every change that somebody uploads, and then after it's been code reviewed, we test it again, because the velocity of OpenStack is so great that you're almost assured that the project you're trying to contribute to will have moved on since you wrote the code. So we want to make sure that nothing that goes in will break the trunk.

We have 800 developers actively working on OpenStack. If we let any of them break something, that has impact on the other 799 of them, and then that winds up being a waterfall effect, right? Everybody would be very frustrated, and nobody would be able to get anything done, because they'd be cleaning up behind other people.

So what we do matters to OpenStack because of quality reasons and the project’s velocity.

Q: What is genuinely unique about OpenStack CI? - compared to other CI systems, such as Jenkins or Buildbot?

A: We absolutely take away everybody's commit rights, no one can directly commit code to anything. That's a pretty harsh stance that we've taken. It's a line in the sand that we've drawn that I think is pretty unique.

Jenkins is one of the elements of the system that we use, but it doesn't really understand what we're trying to do here. So we've had to make additional tools around it to build out our CI system.

Jim Blair, who's one of the key folks on the infrastructure team wrote a tool called Zuul, which sort of runs all of our project gating now. And it ensures that we don't relax our stringent requirements that everything works. It keeps track of changes that come into OpenStack in a serial fashion, because that's the most correct way to do it. If you have 150 patches that you want to run, and each patch takes an hour to test, there aren't 150 hours in the day. Zuul is able to correctly test those things in a parallel manner.

With Zuul we are pushing the boundaries of how people contribute code to a project. HP is using Zuul internally now, Wikipedia has started using it, and we’re seeing more and more people picking it up. We're driving the industry in more ways than just making cloud software.

Q: What has the CI community accomplished so far?

A: There's four core infrastructure people who are running the entire project on their shoulders, and we're able to keep releasing things. That's spectacular.

All of the infrastructure that we run for the OpenStack project is self managed in public repositories. So anybody who wants to come up and contribute is able to do that. I think that's an accomplishment, and we've been able to draw on some good community participation.

Q: What capabilities will CI provide in the OpenStack Havana release?

A: Currently we are working on rolling out patches to Jenkins and Zuul to make things more scalable. We're injecting a technology called Gearman into the mix that will allow us to have multiple Jenkins masters, which is a something that nobody else has. That way, we don't have a bottleneck on the scalability of a single Jenkins master.

We've also started to get Logstash up and going, which is a central log aggregator. When something breaks, giving developers enough data to be able to debug what happened in the test has been a challenge.

Also, we're doing some work to collect test output in a streaming fashion, so that we can give feedback to developers more quickly. If one of the tests fails, for whatever reason, as soon as we know that, we can inform developers that the entire test run is going to fail. So they don't have to wait till the end of the hour-long test run to know what they have to debug.

A lot of our work is about getting more feedback to developers, so that we can continue to work in an agile manner without relaxing our stringent requirements on what it means to patch in OpenStack.

Q: You've talked about making sure that OpenStack tooling is democratized. What does that mean to you?

A: We have to ensure as much as possible there is not a single person with too much power or responsibility and that doesn’t apply only to tooling but to the entire OpenSack project. It's not purely out of fear of somebody becoming powerful in its own sake, as much as that a single person doesn't scale. If anybody can actually come in and fix a problem, then it still might be the same four people doing it, for the most part. But it means there's a safety valve for the project.

It also means that you don't have to convince somebody else that your problem needs to be prioritized over somebody else's, which ends in political games. You've got five different people who want to get something done, and one of them has more friends, so they convince people to work on that. If everybody has equal access, any person can just step up and do the work – without having to convince somebody else that the work is worthwhile.

More than 200 companies are involved in OpenStack. If someone wants to do more testing on Solaris for example, I want to make it possible for them to enable that testing, rather than them have to convince me that it's important, and then convince me that I should learn how to do that. That's not fair for them, and it's not fair for me.

Q: It seems like you've actually managed to hit a balance between being optimistic about the human condition and realistic at the same time.

A: Yeah. If I wasn't optimistic, I wouldn't want to be doing collaborative development with other people, because I wouldn't think that anybody had anything to offer. At the same time, you've also got to recognize that people are going to make mistakes. I know I make mistakes all the time, and I want something that's going to catch my mistakes for me, so that I can stay focused on the problem, and not to have worry about the fact that I forgot a semicolon in a syntax in a file somewhere. Computers can figure that stuff out for me, thank you.

Q: Speaking philosophically, what is your vision for what OpenStack could be?

A: The internet itself has done an excellent job at distributing applications that have standard interfaces. There are many different web servers out there, but they work the same from a consumer perspective. We don't have that same thing happening in the cloud space right now. Amazon, GoGrid, Rackspace, Cloud.com, all these very different clouds are coming up, and they all want to innovate, which is good.

There's an optimistic and a pessimistic side of this. The optimistic side says that they're all just solving a problem nobody else solved before. The pessimistic side says that they are single companies, and they want you to remain their customer - just as Prodigy and CompuServe did before the standardized web came about.

I'd love the open cloud to be based on OpenStack, and even further, like a guy on Facebook, a friend of mine just recently said: “Anybody who's using the word cloud to describe the distributed application that's just in their own data center is using the word incorrectly. They're actually talking about a cluster.” And I agree with that.  A cloud should be 50, 100, 1,000 different companies all running OpenStack. As a consumer I can just do stuff on that aggregate cloud, and not necessarily need to rewrite my application, or to even know in some cases that it's running on five different providers' networks.

Q: What would you like people to know about CI?

A: I'm still not sure that everybody knows that it's an open system, available to anybody to come in, that we're very friendly people, and we’d love more people to come and play with us.

Q: How can people specifically get started?

A: So the first couple of things, we've got the IRC channel #OpenStack-Intra and we keep a bug tracker on Launchpad. A great way to start is just sort of hanging out for a little bit and watching code reviews, putting in good comments or feedback, start to get a sense of what's a hot topic right now.

Q: Do you think OpenStack has changed open source in general? And if so, how?

A: We've got a lot of companies involved in OpenStack and we've managed to do a pretty good job of producing a lot of work in a context that is very corporate heavy. It seems like it's one of the first things that's operating that way. Linux obviously has a lot of corporations involved, but it took a while to get to that point. IBM and these guys didn’t jump on board immediately. It was a long road of people doing it in their off time.

Nobody works on OpenStack on their off time. It's completely a business-driven thing. At the same time, all these companies come together, and none of them has been a dominant player. As a CI guy, I think we're helping to push how people are thinking about how open source can be done in a rigorous and infrastructural-based testing environment, with our gated trunk and stuff like that. So I think these are all ways in which we're forging new ground both technologically and organizationally.

Q: Thank you very much, Monty!

A: You’re welcome.