The reality of DevOps: ECommerce on OpenStack using Mirantis Cloud Platform
Nick Chase - January 01, 2011
We can talk about the benefits of a DevOps environment all we want, but for the people who are directly involved, the reality is much more complicated.
The Internet Mall, Mall.cz, is an ecommerce site located in the Czech Republic, selling all manner of things, from children's toys to dog food to auto supplies, and pretty much everything in between. As such, the site is a complicated piece of machinery, an application called eShop consisting of dozens of microservices that all must work together flawlessly so the whole thing doesn't come crashing down.
It also had a significant price tag.
The price wasn't necessarily the main obstacle, however. "Our big problems were that with the VMware cluster, we can't effectively create and destroy resources as we need them, so it has the effect of slowing down development. Also, because it's a closed system, we can't make modifications the way we might like to."
What Internet Mall needed, IT managers realized, was an open source system that developers could use, but one they could also modify if necessary. This way, developers could create resources as they needed them.
This kind of self-service arrangement would enable developers to move at their own speed, without being constrained by operations and what they could get done at any given time -- and with a 10-1 developer-to-operator ratio, that was an important consideration, particularly after the company was acquired by Rockaway group in 2016, almost tripling the size of their infrastructure.
In the fast-moving world of ecommerce, they knew that If they couldn't find a way to solve this problem, competitors would be able to exploit their vulnerability, perhaps opening a gap in the market from which Internet Mall wouldn't be able to recover.
First, they tried Docker containers. After all, containers are convenient for developers to use, they're well-suited to microservices, the team was familiar with the purpose of containers, and let's face it: they're hot right now.
Unfortunately, containers turned out to be "just too big a leap from our current processes," Martin said. "Docker just didn't have enough of an ecosystem. We couldn't imagine implementing everything we needed on our own."
After checking out several options, Martin and his team decided they needed an OpenStack cloud. OpenStack provided the self-serve nature that was missing in their original VMware cluster, it was open source so they could make any modifications they needed, and the ecosystem was large enough to provide everything that they needed.
Local to Internet Mall was Mirantis Czech (then known as tcp cloud), which was providing MK, a very early version of what would become Mirantis Cloud Platform. MK would enable them to not only use OpenStack, but to customize it for what the needed. Mirantis worked with Internet Mall to work out the various terms and assumptions, and together they built a Proof of Concept just to make sure that this was the direction in which the company wanted to go.
With the new cloud in place, it was time to make use of it. This process, however, was simpler said than done.
The company was very open with their developers. "We told them, 'we're giving this to you, and we're here to help you if you need us. You can choose your own architecture; you're not dependent on ops, here are the benefits of various ways to do this, and so on.'"
Fortunately, the advantages, such as increased development velocity and greater control for developers proved to greatly outweigh the discomfort some developers had trying to shift to the DevOps model. For example, while developers may not be comfortable creating infrastructure at first, because the DevOps model treats infrastructure as code, it's easier for them to adjust to -- and easy to recover from should there be a problem.
After several months of running the PoC, Internet Mall decided to go ahead and build a production cloud and later they initiated the upgrade from MK to MCP. "We did experience the typical issues that go with a major upgrade," Martin says, "but there was nothing really critical, and Mirantis made sure that everything got fixed."
"Of the four of us, two were all in favor, and two are sure that the whole thing is going to come crashing down," Martin laughs. "We don't know for sure yet who's right, but so far there haven't been major problems."
It's a fundamentally different paradigm; these operators, who are used to controlling the infrastructure, see developers spinning up processes for development, test, and so on. There's a reduced dependency on operations from developers, and some ops feel like the devs are "stealing" their stuff and will misuse it, even if it's unintentional. The organization is also adjusting to this new chain of command, and who's responsible for what.
To get people on board, Martin says, they basically compared it to changes that have already happened with regard to old school processes and lines of control. First you cut out physical machines to get to virtual. Now, he explains, you're cutting the next line of control, which was the operations obstacle.
It's said that developers are in the business of promoting progress creating change, and operators are in the business of promoting stability by preventing change. In this respect, they work against each other, but now that the new cloud is in place and developers are using it -- and DevOps practices -- Internet Mall staff finds that they are working in the same direction, all looking at the big picture.
Even though there was originally some concern on the part of some of the operations team, the most important change Internet Mall has seen is still people-related. "The biggest benefit of this project," Martin says, "is cutting that dependency between developers and operations, making the developers more self-sustaining."
These days, eShop is one big microservices-based application with approximately 50 individual services. Of those 20 or so on OpenStack,10 or so on VMware only, and the rest run on both, with approximately 90% of all appropriate production eShop code running on OpenStack. Going forward, Internet Mall is adding more resources, and learning more about the platform. The plan is that once they've stabilized their current operations, they will look at moving everything over to the MCP cloud, with a focus on more containerized components to run on the Kubernetes portion that runs alongside OpenStack.
The Internet Mall, Mall.cz, is an ecommerce site located in the Czech Republic, selling all manner of things, from children's toys to dog food to auto supplies, and pretty much everything in between. As such, the site is a complicated piece of machinery, an application called eShop consisting of dozens of microservices that all must work together flawlessly so the whole thing doesn't come crashing down.
From VMware to open source
For years, Internet Mall's head of the Linux platform team, Martin Olexa, has been running the site mainly on a VMware cluster. This arrangement had all of the benefits of virtualization, of course, including the ability to make effective use of their hardware.It also had a significant price tag.
The price wasn't necessarily the main obstacle, however. "Our big problems were that with the VMware cluster, we can't effectively create and destroy resources as we need them, so it has the effect of slowing down development. Also, because it's a closed system, we can't make modifications the way we might like to."
What Internet Mall needed, IT managers realized, was an open source system that developers could use, but one they could also modify if necessary. This way, developers could create resources as they needed them.
This kind of self-service arrangement would enable developers to move at their own speed, without being constrained by operations and what they could get done at any given time -- and with a 10-1 developer-to-operator ratio, that was an important consideration, particularly after the company was acquired by Rockaway group in 2016, almost tripling the size of their infrastructure.
In the fast-moving world of ecommerce, they knew that If they couldn't find a way to solve this problem, competitors would be able to exploit their vulnerability, perhaps opening a gap in the market from which Internet Mall wouldn't be able to recover.
Docker or OpenStack?
To start, Martin and his team decided to experiment with several options.First, they tried Docker containers. After all, containers are convenient for developers to use, they're well-suited to microservices, the team was familiar with the purpose of containers, and let's face it: they're hot right now.
Unfortunately, containers turned out to be "just too big a leap from our current processes," Martin said. "Docker just didn't have enough of an ecosystem. We couldn't imagine implementing everything we needed on our own."
After checking out several options, Martin and his team decided they needed an OpenStack cloud. OpenStack provided the self-serve nature that was missing in their original VMware cluster, it was open source so they could make any modifications they needed, and the ecosystem was large enough to provide everything that they needed.
Moving forward with OpenStack
Now they needed an implementer.Local to Internet Mall was Mirantis Czech (then known as tcp cloud), which was providing MK, a very early version of what would become Mirantis Cloud Platform. MK would enable them to not only use OpenStack, but to customize it for what the needed. Mirantis worked with Internet Mall to work out the various terms and assumptions, and together they built a Proof of Concept just to make sure that this was the direction in which the company wanted to go.
With the new cloud in place, it was time to make use of it. This process, however, was simpler said than done.
The tough part of moving to DevOps
The hardest part, Martin says, was moving all of their development and operations to the "cloud way" of doing things. "All of our people have to change their mindset," he says. "We're moving tasks to developers that they wouldn't normally do, such as choosing technology. For example, developers now have to think about networking and so on, where they used to just rely on operations."The company was very open with their developers. "We told them, 'we're giving this to you, and we're here to help you if you need us. You can choose your own architecture; you're not dependent on ops, here are the benefits of various ways to do this, and so on.'"
Fortunately, the advantages, such as increased development velocity and greater control for developers proved to greatly outweigh the discomfort some developers had trying to shift to the DevOps model. For example, while developers may not be comfortable creating infrastructure at first, because the DevOps model treats infrastructure as code, it's easier for them to adjust to -- and easy to recover from should there be a problem.
After several months of running the PoC, Internet Mall decided to go ahead and build a production cloud and later they initiated the upgrade from MK to MCP. "We did experience the typical issues that go with a major upgrade," Martin says, "but there was nothing really critical, and Mirantis made sure that everything got fixed."
The people factor
That's not to say that all of Internet Mall's challenges are resolved. Despite having a developer staff of more than 40 people, the company has only 4 operators -- and not all of them were entirely on board with the change."Of the four of us, two were all in favor, and two are sure that the whole thing is going to come crashing down," Martin laughs. "We don't know for sure yet who's right, but so far there haven't been major problems."
It's a fundamentally different paradigm; these operators, who are used to controlling the infrastructure, see developers spinning up processes for development, test, and so on. There's a reduced dependency on operations from developers, and some ops feel like the devs are "stealing" their stuff and will misuse it, even if it's unintentional. The organization is also adjusting to this new chain of command, and who's responsible for what.
To get people on board, Martin says, they basically compared it to changes that have already happened with regard to old school processes and lines of control. First you cut out physical machines to get to virtual. Now, he explains, you're cutting the next line of control, which was the operations obstacle.
It's said that developers are in the business of promoting progress creating change, and operators are in the business of promoting stability by preventing change. In this respect, they work against each other, but now that the new cloud is in place and developers are using it -- and DevOps practices -- Internet Mall staff finds that they are working in the same direction, all looking at the big picture.
What comes next
Internet Mall has created a situation where they can now shift away from having a Single Point of Failure into more of a "cattle" situation, in which software runs using open source on commodity hardware. Nothing is vendor-dependent, and when components die they can simply be replaced -- whether they're hardware or software.Even though there was originally some concern on the part of some of the operations team, the most important change Internet Mall has seen is still people-related. "The biggest benefit of this project," Martin says, "is cutting that dependency between developers and operations, making the developers more self-sustaining."
These days, eShop is one big microservices-based application with approximately 50 individual services. Of those 20 or so on OpenStack,10 or so on VMware only, and the rest run on both, with approximately 90% of all appropriate production eShop code running on OpenStack. Going forward, Internet Mall is adding more resources, and learning more about the platform. The plan is that once they've stabilized their current operations, they will look at moving everything over to the MCP cloud, with a focus on more containerized components to run on the Kubernetes portion that runs alongside OpenStack.
