OpenStack Project Technical Lead Interview Series #5: Julien Danjou, OpenStack Ceilometer Project
This post is the fifth of a continuing series of interviews with OpenStack Project Technical Leads on the Mirantis blog. Our goal is to educate the broader tech community and help people understand how they can contribute to and benefit from OpenStack. Naturally, these are the opinions of the interviewee, not of Mirantis.
Here the interview is with Julien Danjou, OpenStack Ceilometer Project Technical Lead.
Mirantis: Can you please introduce yourself?
Julien Danjou: I'm a free software hacker working freelance. I'm a contributor to a lot of FOSS projects, and besides a Debian developer for more than a decade.
Q: What is your history with OpenStack? Why do you engage?
A: I started working on OpenStack with eNovance, who hired me by the end of 2011 to build the first European cloud platform based on OpenStack. At that time I had no clue about what OpenStack was about, but working on a open source IaaS platform built in Python was really exciting, so I jumped in. Since then, the ecosystem built and the technical challenges we're tackling are still thrilling; lots of good reasons to continue!
Q: What is genuinely different about OpenStack compared to other open source projects you have been involved in so far?
A: I would say pragmatism and good technology choices, which is probably one of the qualities that makes an open source project successful.
Also, many of the open source projects I am or have been involved in have a proportion of both their users and contributors base in individuals, who contribute in their spare time as a hobby. In OpenStack, there's almost no user or contributor not being engaged as part as their company assignment. That makes the project a lot more dynamic, growing faster than many other ones, and gives it the ability to tackle ambitious goals in almost a snap.
Q: What are your responsibilities as the Ceilometer Project Technical Lead?
A: I'm to oversee the project and make sure we're following the direction we agreed upon during the last design summit in Portland. I also tend to see me as the glue between every participants of the project, since I've either a foot or an eye in every area of the project. Being a PTL is almost a full-time job, especially considering the amount of implemented blueprint we're targeting during this release.
Q: Can you explain Ceilometer’s role within OpenStack? Why does Ceilometer matter?
A: Most people building their IaaS platform want to bill someone for using its resources. Based on this first use case, we defined the Ceilometer role inside OpenStack as being the metering point of an OpenStack platform. The scope extended to a larger meter collection so we can address a lot of different use cases, from billing to alarm triggering.
Ceilometer’s goal is to count everything that happens in an OpenStack platform, so you can bill for it, or analyze it in any way.
Q: What is genuinely unique & disruptive about Ceilometer?
A: The ability to have an unique point of collection and interrogation about all your cloud metrics, and this for both operators and users, with the ability to trigger events and actions based on these metrics. This is something that will open a tremendous number of possible applications we haven’t seen yet.
We're also not limited to anything OpenStack specific. Almost every area of Ceilometer is extensible via plugin so you can plug your own metering collection system, publish your metrics to external system, or meter your PaaS platform with Ceilometer directly.
Q: What has the Ceilometer community accomplished so far?
A: We built the project from scratch, plugging it into every OpenStack component without being invasive. Once we did enough, we entered the incubation process, and then graduated as an integrated project. All of that in only one year. After only 6 months of development, we had OpenStack operators using Ceilometer to meter their platforms and bill or charge back for it, generating gigabytes of data metrics per day.
Q: Which capabilities will Ceilometer provide in the OpenStack Havana release?
A: I think we are one of the projects with the most blueprint implementation planned. Our big goal for Havana is the alarming feature, which will allow users and operators to trigger events based on metrics evaluation. Also, this will be a cornerstone for Heat, which will provide auto-scaling capability based on this Ceilometer feature and the associated data.
We're also planning and bringing new metering capabilities, like measuring network bandwidth in Quantum and schedule events in Nova, and enhancing our public API to allow for more fine-grained requests and statistics retrieval.
Q: Which vendors are mostly engaging and providing plug ins? Whom would you like to see contributing to Ceilometer?
A: For now, we don't have any external vendor distributing plugins for Ceilometer. Which is a good thing, because it means that all of our plugins are open source, and distributed with Ceilometer. They all have been contributed as part of the base code by the Ceilometer developers. However, we would be happy to see contribution around non-OpenStack systems that are used by such platforms, like Ceph for example.
Q: Does Ceilometer still have any “childhood ailments” - anything that needs to be fixed?
A: No, and that was actually one of the criteria to get out of the incubation process. You can’t graduate into OpenStack if you know that you are going to do the project all over again. So we may have diseases, but that would be grown-up bugs. :-)
Q: What are typical best practices?
A: The best practice for Ceilometer is deploying it with a fine-grained configuration that will match your needs in term of audit, billing and statistics. Ceilometer generates a huge amount of metering data, so you want to be sure you're generating just enough to fit your needs.
Q: What do you wish people knew about this project (hidden benefits/features)?
A: I think there's a lot of potential in the data Ceilometer is collecting. People usually just look at it as being usable for billing, but the value in it can be so much more, from capacity planning, to defining usage trends for your cloud platform. You can leverage your deployment by feeding this data into the right tools.
Q: Are there any common misconceptions about Ceilometer?
A: We had some misconceptions at the beginning of the project, where people expected Ceilometer to print PDF bills or to monitor their platform and call a sysadmin if something was down. It's now pretty clear that we're the OpenStack metric warehouse and that you can use it to build any application on top of it, including billing or active monitoring. But we don't aim to provide such features.
Q: What are the necessary prerequisites?
A: There's no particular prerequisites in term of hardware. For software, Ceilometer uses the same technologies (Python) as the rest of OpenStack. As for know-how, you definitely need to understand the global OpenStack architecture since you are going to plug Ceilometer into every component!
Q: What is the approximate investment to get Ceilometer up and running?
A: For a simple straightforward installation you probably need an OpenStack engineer for a couple of days to set things up correctly. The more your usage and load are going to increase, the more you might need to tune and enhance the Ceilometer deployment. This is why and how some of our current developers started to contribute, actually.
Q: What advice would you give to people / corporations facing a similar challenge as you had in in the beginning? ... when you have no clue about OpenStack and you have to jump in ...
A: It's definitely useful to be able to discuss OpenStack with other people or companies. I would definitely suggest getting close to the ones that can help you because they already know about OpenStack. There are so many things right now going on in this ecosystem, that it's sometimes hard to follow and get up-to-date information or documentation. So you need to connect.
Then, I guess it depends if you're used to being engaged into open source. As an individual, if you're already into free software communities, it's easy to join OpenStack because its functioning and values are almost the same as any open source project. So the rules apply: read, learn, listen, ask (smart) questions, and then start contributing with what you got.
If you're a company, you definitely need people knowing the open source dynamics and functioning to be effective. If your developers aren't able to contribute to OpenStack by socially engaging in a debate about some features they would like to get implemented or merged, you'll end up as a private vendor with a locally developed software: you won't do any OpenStack in the long run.
Q: Whom would you like to see contributing to Ceilometer?
A: We have a lot of different things into Ceilometer, from the data collection to the REST API providing the data back. So depending on which part somebody would like to contribute to, that could be somebody adding support for Ceilometer on their own "product" (e.g. a PaaS platform) or someone into data analysis helping us regarding the REST API.
Q: Which functionalities need to be enhanced or tested?
A: We clearly need more data consumers at this point. There's already a lot of data we're collecting and storing, but we don't have enough feedback on how people want to consume and query them, so our API tends not to be handy in some cases.
As for tests, there's a big lack in the storage drivers area. Our first and well supported one is MongoDB, but the SQLAlchemy and HBase storage drivers would need more testing, and probably fixes.
Q: How specifically can people get started?
A: I think that like most OpenStack project, the easiest to get something working and play with Ceilometer is to get your hand on a devstack installation. Ceilometer is fully integrated into devstack, so it's easily enabled and deployed along Nova, Glance, etc. Once you do that, you're able to toy with Ceilometer and can try to use or contribute to it in any possible way!
Q: Thank you very much, Julien.
A: You’re welcome!
