NSX: Sharing the overlay transport VLAN between ESXi TEPs and Edge TEPs

Since NSX-T version 3.1.0 and higher, it is possible under certain circumstances to use the Edge TEP and the Host TEP network with the same VLAN. VMware has published a KB article (KB 83743) that defines the framework parameters for this.

Edge TEP and ESXi host TEP can be configured on the same VLAN in the following configurations:

1. edge VM TEP interface connected to a portgroup on an ESXi host not prepared for NSX
2. edge VM TEP interface connected to a portgroup on a switch not used by NSX, on an ESXi host prepared for NSX
3. edge VM TEP interface connected to a logical switch/segment on a vDS7 with NSX-T 3.1.0 or above and is on a host prepared for NSX
4. edge VM TEP interface connected to a logical switch/segment on a NVDS with NSX-T 3.1.0 or above and is on a host prepared for NSX

I am focusing on the third option here because this configuration is useful in a pure Layer 2 environment, and we can prepare all existing ESX hosts for NSX. Also, this configuration is especially useful for LAB environments.

Prerequisites

For this blog, I have recreated the design in my HomeLAB. This blog does not deal with the deployment of NSX and assumes that at least one NSX Manager and a functioning vSphere cluster are available. The screenshots are of the NSX version 4.1.2.1.0.

Environment:

• 2 ESXi servers with 2 pNICs
• 1 NSX Manager
• 2 Edge VMs
• VLAN 15 for TEP, VLAN 31 Uplink 1, VLAN 41 Uplink2
• Overlay transport zone, ESX VLAN transport zone, Edge VLAN transport zone – ESX Uplink Profile, Edge Uplink Profile
• 2 VLAN trunk segments for the fastpath interface of the Edge VM
• 2 VLAN uplink segments for the Edge VM uplinks (BGP)

Profiles

First, I created the uplink profiles for the ESX servers and the edge node. The Named Teaming Policies in NSX-T are used to send the correct transit traffic to the correct pNIC of the host. It is important to use a standby uplink for the host uplink profiles because in the event of a pNIC or link failure, this will prevent the overlay traffic of the edge node from falling into a black hole. Normally, this would happen via the settings of the distributed port group, but that cannot be used in this setup.

Next, I created the Edge Uplink profile. The Named Teaming Policy is also used here. It is important at this point that the edge node does not support standby links. The edge node is configured with a single N-VDS and uses both fastpath interfaces for TEP traffic. In my LAB, a default MTU of 1700 is used.

Transport Zones

Next, transport zones must be created. It is important to ensure that the VLAN transport zone for the edge nodes contains the correct uplink teaming policy names. The same applies to the VLAN transport zone of the ESX servers.

VLAN segments

Two VLAN segments, which allow the TEP VLAN and the uplink VLANs (Trunk VLAN Segments), are required. In addition, each segment must be assigned to the correct Named Teaming Policy and transport zone, so that in the event of a link failure traffic does not get lost. As we can not use VDS port groups in this setup, this must be regulated via the Named Teaming Policy of the VLAN segments. Our uplink segments are created as usual and are assigned to the corresponding transport zone and named teaming policy.

ESX Host configuration

The ESX host is configured for NSX via the transport node profile. The correct VLAN transport zone and the overlay transport zone must be selected here. The VLAN transport zone that is assigned to the ESX host, via the transport node profile, is important because it allows the Edge Node VMs to later be bound to the uplink vlan trunk segments.

Edge VM configuration

The last step is to configure the Edge VM. It is particularly important here that the uplinks are placed on the created VLAN trunk segments. Otherwise, TEP communication between the ESX servers and the Edge VM is not possible. For North/South communication, the Edge Node must have the correct VLAN transport zone. This transport zone is different from the VLAN transport zone of the ESX servers.

Conclusion

In the NSX Network Topology View, you can now see that the trunk segments are connected to the Edge VM.

The configuration does not differ significantly from the normal Multiple VTEP configuration. It allows the use of all ESX hosts for overlay networks in a 3-node setup. In addition, the TEP traffic between the ESX server and the edge node does not need to be routed, which makes this solution very appealing for LAB or PoC environments. Furthermore, it is fully supported by VMware and can be also used in pure Layer 2 environments, which makes it interesting for brownfield NSX installations.