Open Source IoT Platform based on OpenStack and Kubernetes
Our blog post explains open source IoT platform introduced at OpenStack Summit Keynote at Austin in more detail. First we explain our approach and vision to IoT, technical overview and show two sample use cases.
Internet of Things (IoT) is “the next big thing” in cloud computing era. Leading industry vendors present their solution for IoT and interpret them to fit their business strategy. For this reason IoT is abused as a new buzzword for vendor proprietary business solutions. Term IoT can mean almost everything and it even less specific than cloud computing services. The Internet of Things revolves around increased machine-to-machine communication, it is built on networks of data-gathering sensors and actuators connected to cloud computing services that process all the information. It is going to make everything in our lives from streetlights to seaports “smart.”
At tcp cloud we look at IoT in different way than other vendors. In same way we deliver private cloud solutions to our customer, using existing open source projects and extending our cloud service approach to create universal IoT platform, that can handle multiple use cases. We have defined following requirements:
- Open source software
- Whole platform must be based on existing open source solutions and must not be developed by a single vendor. We wanted to make use of existing platforms: OpenStack, Kubernetes, Docker, OpenContrail, etc.
- HW and vendor independce
- No vendor lock-in on both software and hardware side. IoT gateway CPU must have either x86/64 or ARM architecture. We do not want to be locked to any vendor with expensive proprietary appliances.
- Interoperable
- IoT platform must be universal and usable for multiple use cases. For instance IoT gateways can be used in the street lamps for counting objects the same way as in the smart factory or industry 4.0 application.
Therefore we designed following high level architecture, which uses open source projects OpenStack, Kubernetes, OpenContrail and Docker.
- Sensors
- Any arbitrary sensor for reading environmental data (temperature, humidity, CO2 level), object counts, etc.
- IQRF Network
- IQRF is a robust wireless mesh technology operating on sub-gigahertz ISM bands. Easily integrated with sensors.
- Gateways
- Data from sensors is processed by IQRF coordinator in IoT gateways. These can be any x86/64 or ARM architecture, which run systems from prebuild images based on Debian with Docker, Kubernetes and OpenContrail vRouter.
- Internet
- Gateways are connected through any arbitrary Internet connection (GSM/LTE/WiFi), because SDN creates dynamically L3VPNs between Docker services and data center.
- Datacenter
- Datacenter contains OpenStack cloud and Kubernetes control plane. Both these orchestration solutions used OpenContrail as single SDN (Software Define Networking). This enable to manage whole platform from central point. We can remotely start any Docker container on location and create dynamical connection to services in OpenStack. OpenStack cloud contains data stores and big data processing services.
- Vizualizations / API Access
- All data is visualized in web portal and accessible through REST API service.
More technical details are covered in section Technology Overview. Let’s have a look at first two Use Cases, where we started prototyping the solution.
Smart City Prototype
The first use case is SmartCity project of a small city Pisek located in Czech Republic. The SmarCity concept and architecture will deploy over 3,000 endpoints and approximately 300 IoT gateways that run in high-vailability mode in Kubernetes driven containers. Part of the solution is an open data portal and data API available for third party companies that provides information about:
- Traffic flow, routing, parking
- Monitoring, management, energy saving
- E-commerce, marketing, tourist infromation
- Environmental analysis
- Lifestyle, social services, social networks
Target solution uses IoT gateways based on RaspberryPi 2 serving as IoT gateways. Data from gateways is stored in Graphite and processed by custom datamining applications and results are displayed in City citizens portal based on Leonardo CMS, which is web service allowing mixing complex visualizations with arbitrary content. This Open Data portal enables data access through dashboard visualizations or API.
The following screen shows sample output from crossroad Kollarova x Zizkova with Vehicle and Pedestrian passages for specific period.
You can read more about this project in “A step forward in making cities smarter” at SuperUser Magazine from Austin or following presentation from KubeCon 2016.
Smart Conference at OpenStack Summit Austin
To proof that our IoT platform is really independent on application environment, we took one IoT gateway (RaspberryPi 2) from the city project and put into Austin Convention Center during OpenStack Summit together with IQRF based mesh network connecting sensors that measure humidity, temperature and CO2 levels. This demonstrates ability that IoT gateway can manage or collect data from any technology like IQRF, Bluetooth, GPIO, and any other communication standard supported on Linux based platforms.
We deployed 20 sensors and 20 routers on 3 conference floors with a single active IoT gateway receiving data from entire IQRF mesh network and relaying it to dedicated time-series database, in this case Graphite. Collector is MQQT-Java bridge running inside docker container managed by Kubernetes. The most interesting is distance between docker container on Raspberry in conference and virtual machine running in Europe data center. Dynamic network overlay tunnels are provided by OpenContrail SDN. Further explanation is covered in section Technical overview.
The following picture shows single wireless IQRF mesh network during sensors and routers discovery. Zones 0 - 1 cover conference floor 1 and 2 - 4 floor 4.
IQRF is a wireless mesh technology operating on sub-gigahertz ISM bands. It provides very easy integration, product interoperability, robust mesh networking with maximum of 240 hops, range of hundreds of meters and ultra-low power operation.
The following screenshot shows real time CO2 values from different rooms on 2 floors. Historical graph shows values from Monday. You can easily recognize when the main keynote session started and when was the lunch period.
For Austin data collecting, following schema covers services
Technology Overview
This part further explains our technical concept for IoT platform. The IoT platform was created with general vision, which is to collect, manage and process data from thousands of endpoints securely and dynamically with centralized management.
Therefore architecture is divided into two main parts:
Datacenter
Datacenter is central point for management of entire IoT platform. There OpenStack IaaS cloud with virtual machines is running along cloud and SDN control planes. These machines contain time-series storages, data processing clusters, data API proxy access, visualization web services, etc.
Gateway
IoT Gateways are located at any target place like street lamps, factory machines, home appliances, etc. SDN provides transport layer connecting remote IoT gateways with cloud services. Gateways can be multiplatform, it is possible to mix x86/64 and ARM devices today. It is possible to host multiple sensor platforms for multiple customers on single gateway because of micro services segmentation (Docker containers) and Kubernetes multi-tenancy support. The platform is able to provide scalable multi-tenant space where applications and sensors are on same network regardless the distance.
The following schema shows datacenter layers and components on gateway side. The section Detail schema shows deeper information.
Detail Schema
Detail schema provides logical view on architecture side of whole IoT platform. Left side shows datacenter and right side gateway explained in the previous section.
As you can see below OpenStack is used as cloud for hosting all control services as well as all big data processing and frontend visualisation units. Kubernetes on gateways is used for micro segmentation of services necessary for multi-tenancy and security between different sensors. OpenContrail is used to connect both sides and provide network segmentation between Kubernetes PODs and OpenStack Project VMs.
As already mentioned segmentation is done by SDN overlays. The important is only IP connectivity between datacenter edge router and IoT gateway. Lowest layer is VPN between gateway gateway OS and Edge Router in DC. Next layer is SDN, where OpenContrail enables direct communication between VM (OpenStack Cloud) and Container(Gateway). This approach allows to choose from variety of sensors and actuators, privilege them and securely connect with processing applications inside of cloud.
Datacenter contains following services:
- Management Services
- HW cluster running VMs with all control services: OpenStack controller, OpenContrail controller (SDN), Kubernetes master, Salt Master.
- OpenStack Cloud
- OpenStack projects provides segmentation for different virtual machine services like databases (graphite, influxdb, openTSDB), big data processing (Hadoop) and data visualization (Grafana, LeonardoCMS). It runs on KVM hypervisors and uses OpenContrail neutron plugin for networking.
- Edge Routers
- OpenContrail creates iBGP peering with datacenter Edge routers, where propagates dynamically network routes from both OpenStack VMs and Kubernetes pods on IoT gateways. It creates standard L3VPN as MPLSoverGRE or MPLSoverUDP.
Remote gateways contain components:
- Kubernetes Minion
- Kubernetes minion communicates with Kubernetes master in datacenter and manages PODs by kubelet. Kubelet uses opencontrail plugin, which connects docker containers with vRouter agent.
- Kubernetes PODs
- Kubernetes PODs are single or multiple docker containers connected to vRouter. PODs are segmented by labels. This enable to start different application, which can read from different message bus as IQRF, Bluetooth or GPIO.
- Docker Containers
- Docker containers in Kubernetes PODs brigs great benefit of having any kind of operation system easily without any special installation. For instance IQRF uses simple java application with specific version, which can be delivered by container in several minutes and do not mismatch operation system of gateway itself.
Application View
The following schema provides explanation of application view. This shows that VMs inside of OpenStack cloud can reach Docker container in any geographic location in L2 or L3 private network thanks to OpenContrail overlay. Therefore application developers can use same tool as they uses in standard cloud. They can deploy VM application controller by HEAT and then Kubernetes services in containers on remote gateways by simple yaml files.
For instance we gather data from environmental sensors. Sensor is directly connected to container where data is processed in Docker container and send them to Graphite time series database. Because we want to present data graphically and realtime we use another VM with Leonardo CMS which reads from Graphite APIs and displays data on website. Accordingly to this we can create different projects on same principles in same cloud with multiple inputs and outputs.
Conclusion
We tried to briefly explain our vision and prototype deployments of tcp cloud IoT platform. Currently we are working on detail design for whole Smart City solution.
We got a great response from community after having life showcases at OpenStack Summit in Austin and KubeCon in London this year. Our concept seems to be accepted as a possible way of dealing with security, resiliance and performance concerns of IoT platforms and lot of technology partners want to join us in our effort and extend our IoT platform to their solution. We are now starting on concept Industry 4.0 to create first Smart Factory based on open source projects.
If you want to see IoT in action, you can register for OpenStack Day Prague where we show again live presentation with our industry partners.
If you are interested in any kind of cooperation let us know or follow us on twitter and check blogs.
Sources
- Building a SmartCity with IoT - keynote from OpenStack Summit Austin https://www.youtube.com/watch?v=Ym_CZ8-crD8
- SmartCity IoT on Kubernetes - KubeCon 2016 London https://www.youtube.com/watch?v=_L_pR6ZYwEs
- http://www.computerworld.com/article/3062900/cloud-computing/openstack-summit-checking-in-after-another-six-months.html#jump
- Creating a smart conference with OpenStack and containers - http://superuser.openstack.org/articles/creating-a-smart-conference-with-openstack-and-containers
- Output from Austin conference - http://austin.tcpcloud.eu
- Samples from Pisek cross streets - https://pisek.urbandata.cz
- Slides from Keynote - http://www.slideshare.net/JakubPavlik1/building-a-smartcity-with-iot-keynote-slides
