Automate jumpbox (jump-server) VM creation in vSphere 7 with Tanzu

Eldin Egrlic
17. October 2022
Reading time: 6 min
Automate jumpbox (jump-server) VM creation in vSphere 7 with Tanzu

What is VMWare’s vSphere with Tanzu?

VMWare’s vSphere with Tanzu is a cloud computing virtualization platform. The main feature of this new release is the possibility to turn your existing vSphere infrastructure into a Kubernetes platform within a few configuration steps. This allows you to keep your underlaying infrastructure as it is and build a production-ready Container platform on top of that, where you can host your containerized applications.

One of the pivotal points of vSphere with Tanzu is the Supervisor cluster. The supervisor cluster provides the management plane on which Tanzu Kubernetes clusters are built. This supervisor management cluster can host a large number of Tanzu Kubernetes clusters (TKC). TKCs are the clusters where applications get deployed in the end. These clusters act just like any other Kubernetes cluster – you can update, upgrade or hosts services and pods on it. You can also assign as many nodes as you need to those clusters. Those nodes are virtualized and created out of the resources we have inside our vSphere environment.

vSphere with Tanzu requires specific networking configuration to enable connectivity to the Supervisor Clusters, vSphere Namespaces, and all objects that run inside the namespaces, such as vSphere Pods, VMs (virtual machines), and Tanzu Kubernetes clusters. In a vSphere environment you can use NSX-T to provide connectivity to VMs and services; the same can be done in vSphere 7 with Tanzu. In this article, we will use NSX-T for the networking stack to provide connectivity to also TKC nodes and other objects in our Kubernetes environment.

Why we need a Jumpbox VM?

In vSphere with Tanzu, by default, you don’t have direct SSH access to Tanzu Kubernetes Cluster nodes deployed in a vSphere namespace. These cluster nodes are living in a special network called Namespace network which is configured in the Workload network of the Supervisor cluster. The Tanzu Kubernetes Cluster nodes will belong to a dedicated subnet, for example 172.16.0.0/16 and only VMs that are inside this network can communicate with each other.

Let’s say that your cluster site is named as Cluster-SiteA. You can check your network configuration in the vSphere UI: Menu → Inventory → Cluster-SiteA → Configure → Network (sidebar menu item) → Workload network.

You can make this network routable and set up firewalls so that you could actually reach out all the VMs running in the vSphere namespaces. However, the network is typically too big, and it is not a good idea to open the whole network to the outside. This network should remain an internal network – used only by the Supervisor cluster.

There are cases where you need to SSH into cluster nodes. For example, if something went wrong and we need to troubleshoot a worker node VM, you will need to SSH into it. However, having SSH access to the TKC nodes is not as straightforward as it seems. You don’t have direct access to TKC nodes since those are running in a different network – Namespace network. To SSH into a TKC node, you can SSH to the supervisor cluster and from there you could make SSH access to a TKC node. However, this approach is not recommended as it may cause security issues and you do not want to give your DevOps Teams permissions to SSH to the Supervisor Cluster. The Supervisor Cluster is part of your vSphere infrastructure and should remain transparent for your customers.

Depending on your environment, you might need to configure your firewall, too. Even more, it is difficult to automate tasks if you don’t have direct SSH access to a TKC node.

Fortunately, there is another way to get it done – accessing the cluster nodes using a private key directly from within the namespace network using a jump box VM managed by vSphere with Tanzu.

Automate VM creation with the VM Service

As you may already know, in Kubernetes you can deploy your applications and then expose them to the outside using a service, i.e., a LoadBalancer. In vSphere with Tanzu, you have to configure a proper network to handle Ingress traffic and then, when exposing a service to the outside using a LoadBalancer, an external IP from that network will be automatically assigned to the VM.

This is not only true for pods and deployments, but also for virtual machines self. In vSphere with Tanzu, indeed, you can deploy a virtual machine and expose it to the outside in the same way that you’d do with a normal Kubernetes resource. Doing this, you will be able to use your Kubernetes infrastructure to deploy, configure and expose a VM to the outside so that you can use it as a jump box to “jump” into the vSphere infrastructure. For this purpose, you must set up the VM Service.

By using VM service, you can create this jump box VM in a couple of minutes. Typically, you would need to create a virtual machine with Terraform or Ansible. The VM would be based on a local Linux Ubuntu image. You also would need to set up all the networking configuration around this VM manually. And what if our image is outdated? You would have to get the newest one first. The whole process is slow and complex.

With VM Service, you have a way to automate this process – and make it much simpler as well. VM Service allows you to use the latest image from the marketplace maintained by VMware and even better – you deploy the image within the Supervisor Cluster with a single YAML file.

 width=

In simple words, it is Kubernetes friendly, and it follows the Kubernetes standard. Furthermore, you don’t need to set up networking, because you can just refer to NSX-T, which will provide everything for you. Then, you finally expose your Jump box VM to a LoadBalancer Service which is already pre-configured – so you just get an external IP address that you can reach from outside.

To be able to do all of this, first you will create a jump box VM that connects to the workload network. This jump box will live in the same network range (the one you configured) as other cluster nodes. For this purpose, it is best to create a special vSphere namespace where your jump box will be deployed. Within vSphere with Tanzu, you can use VM service to deploy this jump box. The steps are explained below.

Pull images from VMware Cloud Marketplace

 width=

First, you need to get a Linux Ubuntu image that will serve as a base for our VM. Log in to the VMware Cloud Marketplace and search for “VM Service” in the search tab; select a VM Service. (i.e., VM Service Image for Ubuntu).

Click on Subscribe → follow the Settings and give a name to the Content Library. You can select Autoupdate option if you want future updates to be automatically provided.

When you are done, check your subscription. The status should be subscribed, and you should see the Subscription URL below on the page.

Download images to your Content Library

Log in to your vCenter. Go to Content Libraries and click on the Create button to create a new content subscription. Select the Subscribed content library option and paste in the URL that you received for the VM Service. You can select the Download content immediately option if you wish. In the next step select the storage where the downloaded content will be saved. If everything is configured properly, you should see there a Linux Ubuntu image, which you will reference later in the configuration YAML file. You will use this image to build your VM.

 width=

Configure the vSphere Namespace

Create a new vSphere namespace in which you want to deploy your jump box VM. In this example, I will name this namespace as ns-jumphost-01. Go to the namespace in the vSphere UI and set up the common configuration for the namespace as usual, like permissions, quotas and storage. Additionally, you will have to configure the VM Service Tab. For this, select manage VM classes and give access to the virtual machine classes that you want to be available in the namespace. Then, select manage content libraries and add the content library where the Ubuntu VM service is located. Now, we are ready to deploy our jump box VM.

 width=
 width=

Deploy your Jumpbox Virtual Machine

First, you need to prepare the YAML file as a template for the virtual machine, virtual machine service and config map.

As you want to set up a ready-to-use jump box VM, you will want to add some standard configuration to your template. This can be done using Cloud Config. You can create a cloud config file and deploy it as a config map in Kubernetes. Finally, you can pass the config map in the metadata section of the config map. Below is an example of Cloud Config file which can be used. If you want to add additional features in your Cloud Config file, you can use examples from the documentation here: https://cloudinit.readthedocs.io/en/latest/topics/examples.html
Of course, it is recommended to generate your own SSH key and replace it with the sample one. Then, you should encode the template below in Base64 format.

#cloud-config
ssh_pwauth: true # forbid password authentication
password: str0ng1P4ssw0rd
users:
  - name: jbx
    shell: /bin/bash
    gecos: VMware User
    sudo: ALL=(ALL) NOPASSWD:ALL
    lock_passwd: true
    ssh_authorized_keys: 
      - ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHFi7OizUGcpBNKOkX+A2rz+Y7lvBFlFJ+KBxXivRi8sFww+rgbgWBMzGe2Ojx+CcY93lGTtMaTGpWkIuEqJQeQ= jumpbox-user
package_update: true
package_upgrade: true
package_reboot_if_required: true
packages:
  - git
  - jq

After you Base64-encoded the template above, we need to update the Virtual Machine YAML template. Replace the encoded contents with the one in the template below, under the property named user-data. This YAML template contains definitions for the following resources: VirtualMachine, ConfigMap, VirtualMachineService. These resources will be deployed onto the Supervisor cluster. Of course, feel free to customize the template below according to your needs.

apiVersion: vmoperator.vmware.com/v1alpha1
kind: VirtualMachine
metadata:
  name: bastion
  namespace: ns-jumphost-01
  labels:
    app: bastion
spec:
  networkInterfaces:
  - networkType: nsx-t
  className: best-effort-small
  imageName: ubuntu-20-1633387172196
  powerState: poweredOn
  storageClass: tanzu
  vmMetadata:
    configMapName: ubuntu-cm-01
    transport: OvfEnv
---
apiVersion: v1
kind: ConfigMap
metadata:
    name: ubuntu-cm-01
    namespace: ns-jumphost-01
data:
  user-data: >-
    I2Nsb3VkLWNvbmZpZwpzc2hfcHdhdXRoOiB0cnVlICMgZm9yYmlkIHBhc3N3b3JkIGF1dGhlbnRpY2F0aW9uCnBhc3N3b3JkOiBzdHIwbmcxUDRzc3cwcmQKdXNlcnM6CiAgLSBuYW1lOiBqYngKICAgIHNoZWxsOiAvYmluL2Jhc2gKICAgIGdlY29zOiBWTXdhcmUgVXNlcgogICAgc3VkbzogQUxMPShBTEwpIE5PUEFTU1dEOkFMTAogICAgbG9ja19wYXNzd2Q6IHRydWUKICAgIHNzaF9hdXRob3JpemVkX2tleXM6IAogICAgICAtIGVjZHNhLXNoYTItbmlzdHAyNTYgQUFBQUUyVmpaSE5oTFhOb1lUSXRibWx6ZEhBeU5UWUFBQUFJYm1semRIQXlOVFlBQUFCQkJIRmk3T2l6VUdjcEJOS09rWCtBMnJ6K1k3bHZCRmxGSitLQnhYaXZSaThzRnd3K3JnYmdXQk16R2UyT2p4K0NjWTkzbEdUdE1hVEdwV2tJdUVxSlFlUT0ganVtcGJveC11c2VyCnBhY2thZ2VfdXBkYXRlOiB0cnVlCnBhY2thZ2VfdXBncmFkZTogdHJ1ZQpwYWNrYWdlX3JlYm9vdF9pZl9yZXF1aXJlZDogdHJ1ZQpwYWNrYWdlczoKICAtIGdpdAogIC0ganE=
---
apiVersion: vmoperator.vmware.com/v1alpha1
kind: VirtualMachineService
metadata:
  name: bastion
spec:
  selector:
    app: bastion
  type: LoadBalancer
  ports:
    - name: ssh
      port: 22
      protocol: TCP
      targetPort: 22

You can use kubectl apply -f vm-template.yaml in order to deploy these resources. If everything is deployed correctly through the CLI, you should see the virtual machine and the virtual machine service running in the ns-jumphost-01 namespace. Also, in the vSphere UI you can see that the virtual machine bastion is up and running. Additionally, you can see that your VM is based on the image ubuntu-20-1633387172196 which you pulled directly from the VMWare marketplace in the first step.

 width=
 width=

As you can see, in a couple of simple steps we created a jump box VM inside a Supervisor cluster namespace. Now, the only remaining step is to test if we can access the VM from the outside network. Since this VM is just like any other – we can use it to run scripts and apps according to our needs – safely and securely.