Deploy NSX 4.x Edge nodes on unsupported vSphere

NSX 4.0 onwards only supports vSphere 7 and later. In my homelab I’m still running ESXi 6.7 which is joined to a vCenter 8 instance. It’s because the RAID controller in my host is unsupported in ESXi 7+ and no native drivers are available. Running an older ESXi version is fine for me since I do most testing on an nested vSphere 8 lab.

The issue

During the rebuild of my homelab I want to run NSX 4.0. The NSX Manager deploys and runs just fine as VM on ESXi 6.7. The Edge nodes are another thing. When deploying an Edge node on ESXi 6.7 U3 (latest) via NSX Manager it fails with the error:

Ovf deploy for vm <edge name> failed on vc <vcenter fqdn>: Line 375: Unsupported hardware family 'vmx-20 vmx-19 vmx-18 vmx-17'.

Performing a manual OVA deploy via vCenter doesn’t do the job either. That one fails with the error:

Issues detected with selected template. Details: - -1:-1:VALUE_ILLEGAL: No supported hardware versions among [vmx-20, vmx-19, vmx-18, vmx-17]; supported: [vmx-04, vmx-07, vmx-08, vmx-09, vmx-10, vmx-11, vmx-12, vmx-13, vmx-14, vmx-15].

When looking at the VMware KB article Virtual machine hardware versions (1003746), it shows that vmx-17 onwards is actually vSphere 7 and later. To get it running, I need the Edge node to run with vmx-15 VM hardware version, which represents ESXi 6.7 U2 or later. Let’s see what’s possible…

The fix

In a homelab (or dev / test usage) running Edge nodes on an unsupported vSphere version is just fine. So I’m not advocating running it in production this way. See it more as a last resort to get thing up and running.

Having said that as a disclaimer, the Edge node can run on a unsupported version just fine. The deploy method in that case is manual and not possible via NSX Manager.

To get the Edge running on ESXi 6.7 consists of a couple of steps:

1. Download the Edge OVA files
2. Unzip the OVA into the separate files (OVF, MF, Cert, VMDK and so on)
3. Edit the OVF file
4. Create ans modify SHA1 checksum in the MF file
5. Delete the certificate file
6. Create new OVA file

Download Edge OVA

Download the NSX Edge for VMware ESXi OVA file from the VMware Customer Connect site.

Extract Edge OVA file

OVA files are actually a container for all the files that make up the VM (OVF, MF, Cert, VMDK) in tar format. Therefore is can be extracted with standard CLI methods. Thanks to Saurabh Gupta mentioning this in his blog that OVA’s are ‘just’ tar files.

#Extract OVA file
Host:nsx-edge User$ tar -xvf nsx-edge-4.0.1.1.0.20598735.ova
x nsx-edge-4.0.1.1.0.20598735.ovf
x nsx-edge-4.0.1.1.0.20598735.mf
x nsx-edge-4.0.1.1.0.20598735.cert
x nsx-edge.vmdk

Edit OVF

Modify the version and description of the minimum supported ESXi version in the Edge OVF file. Thanks to Macky Ruiz mentioning this in his blog.

The sed command looks a bit odd because it’s run on MacOS. On Linux it would be ‘sed -i ‘s/<before>/<after>/’ <filename>‘

#Modify the values in the OVF file
Host:nsx-edge User$ sed -i "" "s/vmx-17/vmx-17 vmx-15/" nsx-edge-4.0.1.1.0.20598735.ovf
Host:nsx-edge User$ sed -i "" "s/7.0/6.7 U3/" nsx-edge-4.0.1.1.0.20598735.ovf
Host:nsx-edge User$ sed -i "" "s/version 17/version 15/" nsx-edge-4.0.1.1.0.20598735.ovf

#Check if the text is replaced in the OVF file
Host:nsx-edge User$ grep vmx-15 nsx-edge-4.0.1.1.0.20598735.ovf
        <vssd:VirtualSystemType>vmx-20 vmx-19 vmx-18 vmx-17 vmx-15</vssd:VirtualSystemType>
Host:nsx-edge User$ grep 'version 15' nsx-edge-4.0.1.1.0.20598735.ovf
* VM hardware version 15 or greater (vSphere 6.7 U3 or greater)
* VM hardware version 15 or greater (vSphere 6.7 U3 or greater)
* VM hardware version 15 or greater (vSphere 6.7 U3 or greater)
* VM hardware version 15 or greater (vSphere 6.7 U3 or greater)

If the bold marked output is there, you’re good to go.

Create and modify SHA1 checksum in the MF file

Since the OVF file is modified, a new checksum needs to be created. Else the OVA deploy in vCenter would still fail.

#Create a new SHA1 checksum
Host:nsx-edge User$ shasum nsx-edge-4.0.1.1.0.20598735.ovf
fc7712efb34ea6af7ea2b64f3a8b5b41fa28192e  nsx-edge-4.0.1.1.0.20598735.ovf

#Lookup the current SHA1 checksum in the MF file
Host:nsx-edge User$ cat nsx-edge-4.0.1.1.0.20598735.mf
SHA1(nsx-edge-4.0.1.1.0.20598735.ovf)= 9bdcd5ca1033637fe66418800772e727ea9f31c0
SHA1(nsx-edge.vmdk)= a3d5b88fa174965fd1ee9284c1b54e9177ac02eb

#Modify the checksum in the MF file
Host:nsx-edge User$ sed -i "" "s/9bdcd5ca1033637fe66418800772e727ea9f31c0/fc7712efb34ea6af7ea2b64f3a8b5b41fa28192e/" nsx-edge-4.0.1.1.0.20598735.mf

#Check if the new checksum is present in the MF file
Host:nsx-edge User$ cat nsx-edge-4.0.1.1.0.20598735.mf
SHA1(nsx-edge-4.0.1.1.0.20598735.ovf)= fc7712efb34ea6af7ea2b64f3a8b5b41fa28192e
SHA1(nsx-edge.vmdk)= a3d5b88fa174965fd1ee9284c1b54e9177ac02eb

Delete cert file

The cert file needs to be removed now since the cert file contains a checksum and certificate of the MF file. Since the MF file is changed and only VMware can create a new cert file, it needs to go.

#Delete the cert file
Host:nsx-edge User$ rm nsx-edge-4.0.1.1.0.20598735.cert

Create new OVA

For convenience let’s create a new ova file (which is actually a tar file). Else you need to select all 3 files during the vCenter ‘Deploy OVF Template‘ wizard instead of one.

#Delete old OVA file
Host:nsx-edge User$ rm nsx-edge-4.0.1.1.0.20598735.ova

#Create a new OVA file
Host:nsx-edge User$ tar -cvf nsx-edge-4.0.1.1.0.20598735.ova *
a nsx-edge-4.0.1.1.0.20598735.mf
a nsx-edge-4.0.1.1.0.20598735.ovf
a nsx-edge.vmdk

Create the Edge node VM

Now the new OVA is created, let’s check if the effort is worth is. In vCenter select ‘Deploy OVF Template‘ in the place you want to have the Edge VM in and see if it works.

If you get passed step 3 in the workflow you should be fine and can continue to configure the needed parameters and deploy the Edge node. Notice that the ‘Publisher‘ field is empty now, because the cert file was delete. Else it would show ‘VMware …‘.

Alternatively Edge nodes can also be deployed using OVFTool. The NSX-T 3.2 Documentation page describes that process: Install NSX Edge on ESXi Using the Command-Line OVF Tool.

Register the Edge node with NSX Manager

It could be that the Edge node needs to be registered to the NSX Manager, because it’s manually deployed. If that is the case follow the step according to the NSX documentation page: Join NSX Edge with the Management Plane.

What else could go wrong

Even when succeeded in deploying the Edge node VM on an unsupported vSphere version, things still could go wrong. That’s because Edge nodes since NSX-T 3.2 require huge page (1GB) support and secondly Receive Side Scaling (RSS) support for UDP traffic (Geneve overlay traffic).

Huge page support

When your ESXi host cannot Power On the Edge node (even on ESXi 8), the host CPU lacks huge page support. In that case it lacks support for the ‘PDPE1GB‘ CPU feature. Check the VMware KB article 87244 for that.

Huge page support is enabled for VM’s when the advanced VM parameter below is added. Which is done automatically for a new Edge deployment. The following advanced VM parameter enables it.

featMask.vm.cpuid.PDPE1GB = "Val:1"

If the CPU lacks support, it shows in vCenter:

When deployed on ESXi 7 and later via NSX Manager, it shows:

RSS for UDP support

When a new Edge node for NSX-T 3.1 and later is deployed, it requires Receive Side Scaling (RSS) support for UDP. This feature is available in vSphere 6.7 onwards. VMware advises to use vSphere 6.7 U3 for this feature. This is explained in the NSX-T Documentation page: Enhance NSX Edge Performance after ESXi Host Upgrade.

RSS support for UDP is enabled for VM’s when the advanced VM parameters below are added.

ethernet0.pnicFeatures = "4"
ethernet0.udpRSS = "1"
ethernet1.pnicFeatures = "4"
ethernet1.udpRSS = "1"
ethernet2.pnicFeatures = "4"
ethernet2.udpRSS = "1"
ethernet3.pnicFeatures = "4"
ethernet3.udpRSS = "1"

To conclude

For homelab or test / dev usage it could be beneficial to be able to deploy Edge nodes on unsupported vSphere versions. For my lab that’s the way forward since the hosts lacks driver support for certain hardware. This way you can still benefit from running the latest and greatest in your lab.

Have fun in the lab.

Cheers, Daniël

Useful links

Virtual machine hardware versions (1003746)

NSX-T Edge Node Upgrade from 3.1.3 to 3.2 fails with “NSX Edge configuration has failed. 1G hugepage support required (87244)

VMware Customer Central: NSX Edge for VMware ESXi

NSX documentation page: Join NSX Edge with the Management Plane

NSX-T Documentation page: Install NSX Edge on ESXi Using the Command-Line OVF Tool

NSX-T Documentation page: Enhance NSX Edge Performance after ESXi Host Upgrade