Deployment of Kubernetes with Bosh

The new deployment of Kubernetes, which is to simplify the administration and configuration of Kubernetes and Bosh in the long term, is currently a particularly intensive and controversial topic of discussion. Today we don’t want to miss to give you an insight into our opinion on PKS – of course not without raising some (important) open questions for us and certainly also for one or the other user.

Background::

• • To enable Pivotal to deploy native Docker containers, Pivotal has set up the Continer service PKS.

We want to emphasize one aspect right at the beginning:
PKS is not an alternative, but serves to complete the services around the Pivotal platforms. This aspect becomes clear once again when Pivotal PKS also describes as follows:

• Stateful, clustered workloads like Apache Spark and Elasticsearch
• Apps that are already containerized as Docker images (like from an ISV)
• Workloads that require access to infrastructure primitiv

What exactly does Cubernet deployment stand for now?

Similar to CF, an open source variant “Kubo” (Kubernetes/Bosh) is available for PKS and a Bosh-based Kubernetes deployment has recently been launched: https://github.com/cloudfoundry-incubator/kubo-deployment

On the way to the official release …

• At the moment this is still an Incubator project, which is still a long way from an official release.
• Nevertheless, it can already be described today as so stable that it can be used sensibly.
• The deployment itself is based on the https://github.com/cloudfoundry-incubator/kubo-release, which is currently available in version 0.11.0.

The following will show you how to deploy Kubo to bosh-lite and OpenStack. We want to show you how to activate a fully functional Cubernet cluster with a few little tricks.

Technical introduction

Which aspect is directly noticeable at the beginning of Kubo’s use? This is clearly the structure of the repository itself for deployment. Unfortunately, the authors have neglected one aspect from our point of view: they have not concentrated on using the strengths of the Bosh modular manifesto with Bosh 2. This is shown by the fact that the repository is full of shell scripts (January 2018) instead of modular Bosh deployment files.

The good news in this context: There is light at the end of the tunnel! Fortunately, some efforts have been made to migrate the repository in the necessary direction… of course we will keep you informed as soon as there is news here.
Furthermore, the current kubo-deployment plans to deploy a dedicated director, although the usefulness of this must be questioned.

And what should be emphasized with regard to deployment?

At the beginning it is particularly important to us to emphasize one thing: We will not use shell scripts. Instead, the result should be a deployment based on a 5 line shell command that looks like this:

bosh deploy -d kubo manifests/cfcr.yml 

-o manifests/ops-files/vm-types.yml 

-v master_vm_type=default 

-v worker_vm_type=default 

-o manifests/ops-files/replace-name.yml 

-v deployment_name=kubo 

-o manifests/ops-files/use-runtime-config-bosh-dns.yml 

-o manifests/ops-files/availability-zones.yml

Which “setup tool” is required for this? For a successful realization you should pay attention to these steps:

• an existing Bosh Director
  • two different vm_types in the Cloud Config
  • Stemcell 15
  • a network with 4 free IPs
• Bosh Director with credhub enabled `-o /workspace/releases/bosh-deployment/credhub.yml`
• Bosh Director with DNS enabled `-o /workspace/releases/bosh-deployment/local-dns.yml `
• Update runtime config: `bosh update-runtime-config /workspace/releases/bosh-deployment/runtime-configs/dns.yml`

Then you should check out the repository via git clone via git clone https://github.com/cloudfoundry-incubator/kubo-deployment
After the successful checkout it was necessary to add two files under manifests/ops-files:

• availability-zones.yml
• replace-name.yml

availability-zones.yml

- type: remove
path: /instance_groups/name=master/azs

- type: remove
path: /instance_groups/name=worker/azs

replace-name.yml

- type: replace
path: /name
value: ((deployment_name))

The former is only necessary if there are no availability zones (bosh-lite e.g.)

Replace name.yml is necessary to avoid that each deployment is called cfcr and therefore only a parallel deployment is possible.

Once this has been successfully added to the repository, it is then possible to deploy Kubo against an existing Director

Here we come to the open questions mentioned at the beginning:

• How can this be connected to Cloud Foundry? → Consulting evoila
• Where can this be deployed? → Managed Multi Cloud evoila
• How can this be offered to customers/users in a standardised way? Through our OSB Service Broker Framework → OSB Framework evoila
• The deployment does not include a load balancer, which we have solved with our OSB-HaProxy release → OSB Framework evoila
• The deployment does not include any monitoring, which we have solved with our OSB monitoring release → OSB Framework evoila