My previous blog article, is about the integration of NSX ALB for K8s as ingress controller based on AKO (AVI Kubernetes Operator). Both blog articles are focused on VMware Tanzu for vSphere, but the previous one was based on vSphere 7 and this one will be based on vSphere 8.
This blog article helps you to understand how to integrate the vSphere with Tanzu guest clusters with NSX, if you are using Antrea as CNI.
Additionally, you will learn what are the advantages of the Antrea integration for NSX.
For Tanzu you can do the deployment with the integration for vSphere networking including NSX ALB or the integration with NSX-T.
For the NSX-T integration the NSX-T basic LoadBalancer was used in all versions prior vCenter 8 update 2.
A dedicated blog article for this topic will be available soon.
For the Antrea integration with NSX it is obviously necessary to use the integration with NSX-T for your vSphere with Tanzu deployment. As default after you deployed vSphere with Tanzu based on the NSX-T integration, you already have an integration between the Tanzu supervisor cluster and NSX-T done by NCP (NSX Container Plugin). At this point NSX-T will be used by the supervisor cluster to create new T1 routers, segments and LoadBalancing services for newly created K8s Clusters or K8s services. The created K8s clusters which are running as Tanzu guest clusters do use Antrea as CNI, but do not have any integration with NSX-T so far.
The integration of Antrea with NSX-T will add the following benefits.
For the integration between Antrea and NSX-T you need a “Principal Identity User” within NSX-T. For the integration to work it is mandatory to assign the “Enterprise Admin” role for the “Principal Identity User”.
Any other roles available for NSX-T are not sufficient to get the integration with Antrea working.
This requirement might be a problem in some use cases if the provider for NSX-T is not the admin for the K8s Cluster which needs to be integrated, since the user will deliver full permissions.
A solution could be the new NSX-T feature “NSX projects” introduced in NSX 4.1, but I did not find any information this feature is currently supported for the integration between Antrea and NSX-T.
In this blog I will assume you already know how to create a TKG cluster with Antrea as CNI.
As preparation for the Antrea integration with NSX-T you need to download the “VMware Container Networking with Antrea, NSX Interworking Adapter Image and Deployment Manifests“, which is available in the VMware Customer Connect portal under “VMware Container Networking with Antrea”.
Inside the download package you will find the following files.
The file “bootstrap-config.yaml” needs to be changed based on your environment and the previously created Principal Identity User.
Within the file you just need to change the values for the following parameters.
For the TLS key and cert you need the content in one line base64 encoded, which can be done by the following commands.
<path to cert> | base64
<path to key> | base64
Afterwards you should check the images used in the files “interworking.yaml” and “deregisterjob.yaml”, since these repos might be unreachable.
To prevent issues with the availability of the images you should use the VMware public repo shown below.
image: projects.registry.vmware.com/antreainterworking/interworking-photon:<image version>
For version 0.11.0 the path to the imaged will be “projects.registry.vmware.com/antreainterworking/interworking-photon:0.11.0”.
Now you are ready for the deployment which can be done with the following two commands.
kubectl create clusterrolebinding privileged-role-binding --clusterrole=psp:vmware-system-privileged --group=system:authenticated
kubectl apply -f bootstrap-config.yaml -f interworking.yaml
The cluster role binding should be set to allow the imaged to be deployed with your Kubernetes cluster, the apply of the “boostrap-config.yaml” and the “interworking.yaml” will start the registration of your Antrea CNI with the NSX-T Manager.
After the deployment is started you can check the status with the following two commands.
k -n vmware-system-antrea get jobs
NAME COMPLETIONS DURATION AGE
register 1/1 90s 2m35s
k -n vmware-system-antrea get pods
NAME READY STATUS RESTARTS AGE
interworking-6edc37756a-ktzcd 4/4 Running 0 2m30s
register-gzn5s 0/1 Completed 0 2m30s
After the registration is successfully done, you can also check the status in the NSX-T Manager. In the screenshot below you can check the status of the relevant components “Antrea Controller”, “Management Plane Adapter” and “Central Control Plane Adapter”. This information is shown for each K8s Cluster which is integrated through the Antrea CNI and can be found under System → Fabric → Nodes → Container Clusters → Antrea.
In addition, you can see the different objects deployed in the K8s clusters as well as the K8s cluster itself and the used CNI. These information can be found under Inventory → Containers.
You are also able to see the existing K8s namespaces and in which K8s cluster they are deployed.
After the integration between Antrea and NSX-T is successfully done, you can create Distributed Firewall rules which will be realized as “Antrea Cluster Network Policies” in the K8s Cluster.
Rules created by the NSX-T Manager will be visible from the NSX-T Manager UI or API, and will be synced to the K8s cluster as “Antrea Cluster Network Policies”. It is also possible to create “Antrea Cluster Network Policies” from the K8s API, but these rules will not be visible within the Distributed Firewall.
This lack of overview is a disadvantage, but you are still able to see these rules by the Traceflow feature of NSX-T and can see by which “Antrea Cluster Network Policiy” a communication is either allowed or blocked.
Further the policies created by the NSX-T do have a higher priority than the policies created by the K8s API. That means you can define policies to ensure the guidelines within your environment which cannot be overwritten by the K8s users or admins.
As an example, I created a DFW policy for my K8s Cluster “k8s-cl-test1” and one DFW rule to decide if the communication between two pods will be allowed. In the screenshot below you can see a policy applied to the previously mentioned K8s cluster and a rule with the “frontend” Pod as source of the communication. This rule is also applied to the redis database Pod. Based on this rule an input “Antrea Cluster Network Policy” is created.
Note: If you want to create an output rule, you have to specify the Destination of the communication instead of the source. You cannot configure source and destination in one rule, since a “Antrea Cluster Network Policy” can be either an output or input rule. You can limit the input rule to a target Pod by defining the objects where this rule should be applied, as done in the example above.
Further I created “Antrea Cluster Network Policy” from the K8s API based on the following YAML file.
- action: Allow
- protocol: TCP
Based on the NSX Traceflow feature you can check which rule will be used to either allow or block the communication.
For the first test the DFW rule will be enabled and the policy created by the K8s API is also active, which can be checked with the command “kubectl get acnp -A” and shows the following output in our example.
kubectl get acnp -A
NAME TIER PRIORITY DESIRED NODES CURRENT NODES AGE
a9e8144d-b7f3-4e2e-9919-e6ee421c1551 nsx-category-application 1.0000000532907336 1 1 6m24s
acnp-with-stand-alone-selectors securityops 5 1 1 34d
The “Tier” shows if the rule is applied within one of the NSX DFW categories like “nsx-category-application” or in any other category of the defined by K8s like “securityops”.
If both policies are created the NSX DFW rule do have the higher priority, as shown in the following screenshot of the NSX Traceflow feature.
The screenshot below shows the packet flow and the “Network Policy ID” which was used to allow the communication.
Based on the “Network Policy ID“ you can find the details about the applied rule including the detailed configuration.
Now I disabled the DFW rule created in NSX and the communication will be filtered by the rule created through the K8s API, as shown below.
In the following screenshot you can see that the used rule is not part of the categories available in NSX.
From my point of view this is a very good way to secure the K8s workload, also within the K8s cluster. The most valuable feature is that you can see which rule is responsible to allow or block a specific communication and if this rule is either managed by NSX or K8s.
Further it is very important from security point of view to separate the responsibility for the enforcement points of the network and application security. To enforce this responsibility and prevent anyone to ignore these responsibilities it is also important to make sure no one of the foreign teams can change anything they are not allowed to change.
For example, it would be critical if a K8s Admin is able to change or overwrite a firewall rule which is managed by the overall security or network team and is based on a company regulation.
I am very happy that these problems are addressed, since “Antrea Cluster Network Policies” can be either created by the NSX Admin oder K8s Admin, but the K8s Admin is not able to overwrite or change the rules created by the NSX Admin. Further the rules created by the NSX Admin does have a higher priority.