Kubernetes and OpenStack multi-cloud networking

OpenContrail overview

OpenContrail is open source SDN & NFV solution, with tight ties to OpenStack since Havana. It was one of the first production ready Neutron plugins along with Nicira (now VMware NSX-VH) and last summit’s survey showed it is the second most deployed solution after OpenVwitch and first of the Vendor based solutions. OpenContrail has integrations to OpenStack, VMware, Docker and Kubernetes.

Kubernetes network plugin kube-network-manager was under development since OpenStack summit at Vancouver last year and first announcement was released in end of year.

The kube-network-manager process uses the kubernetes controller framework to listen to changes in objects that are defined in the API and add annotations to some of these objects. Then it creates network solution for the application using the OpenContrail API that define objects such as virtual-networks, network interfaces and access control policies. More information is available at this blog

Architecture

We started testing with two independent Contrail deployments and then set up BGP federation. The reason for federation is keystone authentication of kube-network-manager. When contrail-neutron-plugin is enabled, contrail API uses keystone authentication and this feature is not yet implemented at kubernetes plugin. The Contrail federation is described in more later in this post.

The following schema shows high level architecture, where on left side is OpenStack cluster and Kubernetes cluster is on the right side. OpenStack and OpenContrail are deployed in fully High Available best practice design, which can be scaled up to hundreds of compute nodes.

The following figure shows federation of two Contrail clusters. In general, this feature enables Contrail controllers connection between different sites of a Multi-site DC without the need of a physical gateway. The control nodes at each site are peered with other sites using BGP. It is possible to stretch both L2 and L3 networks across multiple DCs this way.

This design is usually used for two independent OpenStack cloud or two OpenStack Region. All components of Contrail including vRouter are exactly the same. Kube-network-manager and neutron-contrail-plugin just translate API requests for different platforms. The core functionality of the networking solution remains unchanged. This brings not only robust networking engine, but analytics too.

Application Stack

Application deployment

This section describes workflow for application provisioning on OpenStack and Kubernetes.

OpenStack side

At the first step, we have launched Heat database stack on OpenStack. This created 3 VMs with PostgreSQL and database network. Database network is private tenant isolated network.

# nova list
+--------------------------------------+--------------+--------+------------+-------------+-----------------------+
| ID                                   | Name         | Status | Task State | Power State | Networks              |
+--------------------------------------+--------------+--------+------------+-------------+-----------------------+
| d02632b7-9ee8-486f-8222-b0cc1229143c | PostgreSQL-1 | ACTIVE | -          | Running     | leonardodb=10.0.100.3 |
| b5ff88f8-0b81-4427-a796-31f3577333b5 | PostgreSQL-2 | ACTIVE | -          | Running     | leonardodb=10.0.100.4 |
| 7681678e-6e75-49f7-a874-2b1bb0a120bd | PostgreSQL-3 | ACTIVE | -          | Running     | leonardodb=10.0.100.5 |
+--------------------------------------+--------------+--------+------------+-------------+-----------------------+

Kubernetes side

At kubernetes side we have to launch manifests with Leonardo and Nginx services. All of them can be displayed there.

In order for it to run successfully with networking isolation, look at the following sections.

• leonardo-rc.yaml - Replication Controller for Leonardo app with replicas 3 and virtual network leonardo

apiVersion: v1
kind: ReplicationController
...
  template:
metadata:
  labels:
    app: leonardo
    name: leonardo # label name defines and creates new virtual network in contrail
...

• leonardo-svc.yaml - leonardo service expose application pods with virtual IP from cluster network on port 8000.

apiVersion: v1
kind: Service
metadata:
  labels:
    name: ftleonardo
  name: ftleonardo
spec:
  ports:
    - port: 8000
  selector:
    name: leonardo # selector/name matches label/name in replication controller to receive traffic for this service
...

• nginx-rc.yaml - NGINX replication controller with 3 replicas and virtual network nginx and policy allowing traffic to leonardo-svc network. This sample does not use SSL.

apiVersion: v1
kind: ReplicationController
...
  template:
    metadata:
      labels:
        app: nginx
        uses: ftleonardo # uses creates policy to allow traffic between leonardo service and nginx pods.
        name: nginx # creates virtual network nginx with policy ftleonardo
...

• nginx-svc.yaml - creates service with cluster vip IP and public virtual IP to access application from Internet.

apiVersion: v1
kind: Service
metadata:
  name: nginx
  labels:
    app: nginx
    name: nginx
...
  selector:
    app: nginx # selector/name matches label/name in RC to receive traffic for the svc
  type: LoadBalancer # this creates new floating IPs from external virtual network and associate with VIP IP of the service.
...

Lets run all manifests by calling kubeclt

kubectl create -f /directory_with_manifests/

This creates following pods and services in Kubernetes.

# kubectl get pods
NAME             READY     STATUS    RESTARTS   AGE
leonardo-369ob   1/1       Running   0          35m
leonardo-3xmdt   1/1       Running   0          35m
leonardo-q9kt3   1/1       Running   0          35m
nginx-jaimw      1/1       Running   0          35m
nginx-ocnx2      1/1       Running   0          35m
nginx-ykje9      1/1       Running   0          35m

# kubectl get service
NAME         CLUSTER_IP      EXTERNAL_IP     PORT(S)    SELECTOR        AGE
ftleonardo   10.254.98.15    <none>          8000/TCP   name=leonardo   35m
kubernetes   10.254.0.1      <none>          443/TCP    <none>          35m
nginx        10.254.225.19   185.22.97.188   80/TCP     app=nginx       35m

Only Nginx service has public ip 185.22.97.188, which is floating ip configured as LoadBalancer. All traffic is now balanced by ECMP on Juniper MX.

To get cluster fully working, there must set routing between leonardo virtual network in Kubernetes Contrail and database virtual network in OpenStack Contrail. Go into both Contrail UI and set same Route Target for both networks. This can be automated too through contrail heat resources.

The following figure shows how should look final production application stack. At top there are 2 Juniper MXs with Public VRF, where are floating IPs propagated. The traffic is ballanced through ECMP to MPLSoverGRE tunnel to 3 nginx pods. Nginx proxies request to Leonardo application server, which stores sessions and content into PostgreSQL database cluster running at OpenStack VMs. Connection between PODs and VMs is direct without any routed central point. Juniper MXs are used only for outgoing connection to Internet. Thanks to storing application session into database (normally is memcached or redis), we do not need specific L7 load balancer and ECMP works without any problem.

# kubectl describe svc/nginx
Name:                   nginx
Namespace:              default
Labels:                 app=nginx,name=nginx
Selector:               app=nginx
Type:                   LoadBalancer
IP:                     10.254.225.19
LoadBalancer Ingress:   185.22.97.188
Port:                   http    80/TCP
NodePort:               http    30024/TCP
Endpoints:              10.150.255.243:80,10.150.255.248:80,10.150.255.250:80
Session Affinity:       None

# kubectl describe svc/ftleonardo
Name:                   ftleonardo
Namespace:              default
Labels:                 name=ftleonardo
Selector:               name=leonardo
Type:                   ClusterIP
IP:                     10.254.98.15
Port:                   <unnamed>       8000/TCP
Endpoints:              10.150.255.245:8000,10.150.255.247:8000,10.150.255.252:8000

185.22.97.188/32   @[BGP/170] 00:53:48, localpref 200, from 10.0.170.71
                      AS path: ?, validation-state: unverified
                    > via gr-0/0/0.32782, Push 20
                    [BGP/170] 00:53:31, localpref 200, from 10.0.170.71
                      AS path: ?, validation-state: unverified
                    > via gr-0/0/0.32778, Push 36
                    [BGP/170] 00:53:48, localpref 200, from 10.0.170.72
                      AS path: ?, validation-state: unverified
                    > via gr-0/0/0.32782, Push 20
                    [BGP/170] 00:53:31, localpref 200, from 10.0.170.72
                      AS path: ?, validation-state: unverified
                    > via gr-0/0/0.32778, Push 36
                   #[Multipath/255] 00:53:48, metric2 0
                    > via gr-0/0/0.32782, Push 20
                      via gr-0/0/0.32778, Push 36

Conclusion

We have proved that you can use single SDN solution for OpenStack, Kubernetes, Bare metal and VMware vCenter. The more important thing is that this use case can be actually used for production environments.

If you are more interested in this topic, you can vote for our session Multi-cloud Networking for OpenStack Summit at Austin.

Currently we are working on requirements for Kubernetes networking stacks and then provide detailed comparison between different Kubernetes network plugins like Weave, Calico, OpenVSwitch, Flannel and Contrail at scale of 250 bare metal servers.

We are also working on OpenStack Magnum with Kubernetes backend to bring developers self-service portal for simple testing and development. Then they will be able to prepare application manifests inside of OpenStack VMs, a then push changes of final production definitions into git and at the end use them at production.

Special thanks go to Pedro Marques from Juniper for his support and contribution during testing.

Jakub Pavlik, Marek Celoud & tcp cloud team