How to Take the CI/CD Plunge or How I Learned to Stop Worrying and Love OpenStack -- Q&A

On Tuesday, March 3, Christopher Aedo and I presented a webinar about Continuous Integration/Continuous Delivery (CI/CD) and how you can use it to create better software. We didn't have a chance to answer all of your questions during the webinar, so as promised, here are the rest of the questions and Christopher’s answers, plus the ones we did get to.

Question: Does this only work in an agile environment?
Answer: You could integrate automated testing and deployment with a waterfall process but you don't gain too much with that approach. You do at least gain from the automated testing and deployment, but if you're following a more rigid development process it's hard to see how your testing and deployment at a late stage will positively inform your development process

Question: How do I find the code?
Answer: You must first search within! Ask your heart and your spirit will guide you!  (That, and if you're looking for the OpenStack code, you can find it on Github...)

Question: What would that review system/process look like. You mention automated testing, but no human intervention?
Answer: There is human intervention in setting up the tests, and human intervention in code review. The Gerrit workflow used in OpenStack is a pretty good example of this, and you can find more information here: http://docs.openstack.org/infra/manual/developers.html

Question: We use SVN for source control, do you recommend switching to GIT? Why?
Answer: There's no reason to switch from SVN to Git if your team is comfortable with SVN. Some argue branching/merging is easier in Git, but generally it comes down to familiarity with the tool, and little is gained by switching tools if most of your users are using the existing tool well.

Question: Hi, the "Review Process" that may find missing tests, is it automated or manual? Other than running some coverage test, how can we find out some tests are missing and something not tested (without having test engineers review the tests manually and sign-off on test existence)?
Answer: This is where your QA team comes in. Presumably they've designed a thorough test matrix, and used that to write the actual tests that run in your pipeline. Developers checking in code need to write the unit tests (and usually functional tests as well), and THAT piece is very cultural - everyone has to be onboard with it (so nobody reviewing the code gives a +1 to code missing test coverage). Ultimately, most organizations rely on the QA team being the last manual gate here, making sure the tests cover everything they're supposed to.

Question: Having an understanding of basic CI/CD, is there any alternative to Jenkins to pick up new code, build and deployment. If yes, please throw a light on those projects.
Answer: You can find alternatives to Jenkins here: http://alternativeto.net/software/jenkins/. We're most familiar with Jenkins, so if you're looking for an alternative you'll probably want to try several before settling on something.

Question: Has there been a need to roll back changes? And if so, how would that be done?
Answer: Sure, everyone has had to back out a change sometime! CI/CD aside, it's just like any other coding situation in which you're using a version control system such as git or SVN. Sometimes it's easy - if it's an idempotent change that only touches code, you just revert that commit (go back to the commit just prior to the bad merge). On the other hand, if the changes included things such as database schema changes, the rollback procedure can get ugly.  In the end, though, rollbacks happen before the code gets integrated, so it's independent of the CI/CD process.

Question: How is Zuul compared to Go?
Answer: Go is a programming language, Zuul is an automated testing framework/project gating system (http://ci.openstack.org/zuul/) so we can't really compare them.

Question: Do you have any advice on how we can deploy on every Gerrit patchset?
Answer: You can use a post-commit hook, more details here: http://stackoverflow.com/questions/12794568/how-to-configure-git-post-commit-hook

Question: Gerrit vs. Github's code review process.... can you compare and contrast? Recommendation?
Answer: Personally I like Gerrit more for very large/complicated projects with a great many contributors. There's a pretty good comparison found here, also in favor of Gerrit: https://gist.github.com/mbbx6spp/70fd2d6bf113b87c2719. Ultimately, though, it's similar to Git vs. SVN, and a lot of it comes down to personal choice. (Similar to Emacs vs. VI, obviously the answer is (REDACTED)!)

Question: How have Docker and LXC changed the way people perform CI/CD?
Answer: Containers can provide a faster and lighter weight way to start environments from snapshots (primarily on the storage side) vs. bare metal or even virtual machines.  In many cases containers also help improve utilization of hardware by allowing you to run more containers on a given machine vs. virtual machines.

Question: What kind of tools fill in the area of OpenStack performance testing and integration testing?
Answer: Rally has gained a lot of momentum in this area and is probably the leading tool for this job. You can find more details here: https://wiki.openstack.org/wiki/Rally

Question: How does the term continuous delivery relate to continuous integration/continuous deployment? Is the latter the same?
Answer: Yes, continuous delivery is the same thing as continuous deployment, and the idea is, like we said, you check your code in, it passes all the tests and it doesn't just sit waiting for the big release date, it is just deployed as the last step in the process.

Question: So what is the difference between unit and functional testing?
Answer: So the difference really is the unit test is a much smaller one, and it is isolated to a very small component of code. An example of the unit test would be if you have a piece of code that is supposed to add one to the two numbers that were supplied to it, so the test will validate that that little snippet of code actually adds one to whatever number it's handed to.

Consider the unit test as something that tests the smallest possible part of your program, a functional test would validate functionality.  So to prove some specific functionality works, that test will look like a combination of multiple unit tests.  This answer from StackOverflow covers this topic really well.

Question: Can you elaborate on the code review process? Does it appear before or after check in? Who are the approvers? Does Q/A have any involvement?
Answer: Okay. So the code review process is flexible and is based on your organization in large part. Q/A normally would have involvement, although generally the Q/A team's involvement is much more on the testing side of things. So your Q/A team's job is to make sure that the tests cover what they expect to be covered, more so than expecting the Q/A team to review all the code.

So within OpenStack, anyone – literally anyone, can set up an account and start reviewing code. They can give a minus one if they think the code is bad; they can add comments to it, or they can give a plus one and say, “I think this code looks really good.” Then you've got the core OpenStack reviewers who have a lot of experience with code review and who have been officially voted for to act in that capacity. Individual reviewers carry less weight than the core reviewers, who are the final gate for the code.

Question: Do you recommend using OpenStack Solum?
Answer: The Solum project, in essence, is meant to automate the creation of a CI-CD pipeline on top of OpenStack. So the Solum project is meant to provide a framework for all of this stuff. And so I would say it's worth looking at. I don't know, because I can honestly say I haven't looked at Solum in the last two or three months, but I know that they have been working – been making a lot of progress since it was launched, I think, at the Hong Kong summit, or right around then. And the premise at least is fantastic; it would really give you an easy way to actually supply like a git URL, and let Solum automate a lot of your testing and that pipeline setup from there. So it's certainly worth looking at.

Question:  What I don't understand is how are you sure it will work on a customer's production environment when you, as the developer, do not have control over the "purity" of the production environment? Isn't it possible that that environment has some kind of library patch that could break things?
Answer: Absolutely true. There is a very clear limit to how much you can guarantee when you're doing like a project like OpenStack, right? It's super complicated, lots of different interrelated projects and all intended to be deployed on virtually any hardware you can imagine. So in that circumstance you can't be certain it will work in a customer's environment but you can be certain that if the customer's environment exactly matches your testing environment it will work. And so you can define that reference architecture and say if you buy hardware that matches these specifications and configure it in this specific way that matches our testing environment, it will work.

So if you scale that back down to kind of a smaller application or a smaller focus, you can create an application that doesn't rely on external – any external libraries and maybe has very minimal external requirements and reduces the scope of requirements for your application such that there aren't too many things a customer can do that would make it not work. For example, look at Apple and limited hardware. When they release their software, they know exactly what hardware it's going to be running on, and they can pretty much guarantee that it's going to work.

Question: Are you using dev ops personally and what is your opinion compared to CD? It seems to me that they work together, though, don't they?
Answer: Yeah, absolutely. Dev ops and continuous delivery go hand in hand. So we use it at Mirantis, and I'm assuming that's the question: if we use it personally? So I am now Mirantis, I'm happy to answer that. But yes, we absolutely do use dev ops. So in this particular case we have cross-functional teams. So we have a group of people at Mirantis that, in essence, owns the hardware that all of our testing and deployment happens on; that's our dev ops team. But that dev ops team, they're also regular engineers and they work closely with all of the software engineers and they work closely with the Q/A team to define the tests so that we kind of don't have those silos. Yeah, so I would say the two just kind of go hand in hand.

Question: So you’re basically doing continuous delivery on the scripts and such that make up your dev ops also? So they literally go hand-in-hand?
Answer: Yes.

Question: Is it possible to contribute to OpenStack using a tool different from Git?
Answer: No, it's not possible to contribute to OpenStack using a tool different from Git, in essence, because all of the code for OpenStack is in a Github repository. So in order to work with OpenStack, you've got to use the Get tools

Question: So is using Git, Garrett, Jenkins, Zuul, and other services supposed to be easy? Are there some details on how to get all of this working together?
Answer: I say all of this is easy, and certainly it's not that easy, but most developers don't have to set this all up if they're trying to contribute to OpenStack. If you're just trying to contribute to OpenStack all you need is git and git review and an account.

It's only if they're making their own environment that it gets complicated. And to that end, the OpenStack infrastructure team has documented how they have set up Zule and how third-party developers can create their own CI infrastructure that integrates directly with OpenStack. So you can have, for instance, Microsoft with the Hyper-V. Microsoft actually puts a lot of code back into OpenStack and makes sure that their Hyper V Hypervisor works with OpenStack, and one of the ways they do that is by creating and hosting a CI – a set of servers, a CI infrastructure that when code that touches the Hyper V stuff gets checked in, it triggers the testing back on the Windows hardware.

Summary

These are the answers to questions we were able to address in the webinar, along with those we did not have time to get to. The video is ready, along with slides.