You may recognize this. You’ve updated your ESXi hosts via vSphere Lifecycle Manager (vLCM) or similar method, the host rebooted during remediation and… that’s it. Your host is not reachable anymore.

That happened to my lab server last week, while updating it to the latest ESXi 8.0 patch release. So what went wrong? In short, this happens when ESXi does not have inbox drivers for your NIC and the vLCM Image configuration does not has the async NIC driver added. When the hosts is rebooted during the remediation process the issue is obvious.

So how to install a new NIC driver if the hosts is not reachable over the network anymore and you don’t want to tinker with creating a custom ISO (with the NIC driver included) and update or even reinstall your hosts. The options are limited in this case, but there is a way.

The solution

Luckily ESXi is able to access FAT formatted USB Drives to fix this issue. Unfortunately some of the old methods do not work nowadays. For example VMware KB article (2065934) ‘Configuring a USB Flash drive to transfer files between Windows and an ESXi/ESX host‘ does not work anymore for ESXi 8. The same counts for this old post of William Lam called ‘How to Access USB Storage in ESXi Shell‘.

Fortunately I found a post on called ‘Accessing USB Flash Drive from VMWare ESXi‘ that was an excellent starting point. This contained all I needed to be able to copy the NIC driver from a USB drive to a datastore on the culprit ESXi host and fix the issue.

Step 1: Prepare the USB Drive

The general steps are to put the USB Drive in your computer, remove all data from the USB Drive, create a small partition and lastly, format it with the FAT filesystem.

The drive used in this post is a 32 GB Transcend JetFlash 780 USB 3.1 Gen 1. These durable MLC based USB Drives work excellent as ESXi boot drive or in this case, for recovery.

Prepare the drive in Windows

In this example I used ‘diskpart‘ in Windows 11 (22H2).


DISKPART> list disk

  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online         2794 GB      0 B        *
  Disk 1    Online          931 GB      0 B
  Disk 2    Online          931 GB   931 GB
  Disk 3    Online          931 GB      0 B
  Disk 4    Online          931 GB      0 B        *
  Disk 5    Online           28 GB    26 GB

DISKPART> select disk 5
(Disk 5 is the USB Drive I want to use)

Disk 5 is now the selected disk.

(Make sure to select the correct drive, since all data on the device will be deleted in this step!) 

DiskPart succeeded in cleaning the disk.

DISKPART> create partition primary size=4000
(Creates a small 4 GB primary partition for backwards compatibility reasons)

DiskPart succeeded in creating the specified partition.

DISKPART> format fs=fat quick label=drivers
(Formats the partition USB Drive with the FAT filesystem and labels the partition as 'drivers')

  100 percent completed



Step 2: Copy the ESXi NIC driver to the USB Drive

This step does not need explanation ;-).

In my examples I shortened the name of the ESXi driver file to “”. Normally an ESXi driver file would called something like “”

Step 3: Determine the right USB disk device in ESXi

In this step we’ll determine which USB disk connected to the ESXi hosts is the right one and how is named. On the ESXi hosts with the missing driver move to the ESXi Shell by pressing “ALT + F1” on the console and login.

If the login screen of the ESXi Shell is not available it’s probably not enabled. Enable the ESXi Shell by logging into the Direct Console User Interface (DCUI) by pressing “F2” and enable the “ESXi Shell” in the “Troubleshooting Options” section.

Open the ESXi Shell by pressing “ALT + F1” and login. Stop the USB Arbitrator service on the ESXi host to make USB devices locally available.

[user@host:~] /etc/init.d/usbarbitrator stop
watchdog-usbarbitrator[654321]: Terminating watchdog process with PID 123456
stopping usbarbitrator...
usbarbitrator stopped

Put the USB Drive into the ESXi host and determine which USB disk devices the ESXi hosts knows about.

[user@host:~] esxcli storage core device list | grep -i usb
   Display Name: Local USB Direct-Access (mpx.vmhba32:C0:T0:L0)
   Is USB: true
   Is USB: false
   Display Name: Local USB Direct-Access (mpx.vmhba34:C0:T0:L0)
   Is USB: true

In my case I have two USB disks attached to this hosts. One is used for booting ESXi, while the other has the missing driver. So which one has the driver file?

Based on the output above, both USB disks start with the ‘path based identifier‘ “mpx.vmhba3“. Let’s check the amount of partitions on both. The one with a single partition has the driver file, while the other has four partitions because it’s the ESXi boot device.

[user@host:~] ls /dev/disks | grep -i mpx.vmhba3

The lines with a number after the last colon (:) represent the partition number.

The output above shows that the USB disk device we’re looking has path identifier “mpx.vmhba34:C0:T0:L0“. This is the right one because it has a single partition, compared to “mpx.vmhba32:C0:T0:L0” which has four partitions.

Step 4: Copy the ESXi NIC driver file

In this step the driver file will be copied to a datastore on the ESXi hosts. If it’s shared datastore, the driver file could even be used on multiple hosts if the shared datastore is still accessible.

In early ESXi versions the USB disk device would show up as a datastore, but in recent versions that’s not the case anymore. In older blogs and KB articles it’s mentioned that the source partition could also be NTFS. NTFS partitions are not accessible since the ‘ntfscat‘ utility is not part of recent ESXi versions anymore.

Copy files from the USB drive must be done with the ‘mcopy‘ tool, since regular copy tools do not work. To copy a file to the host local datastore use the command below. Do not forget to add the partition number (:1) to the end of the USB disk device path identifier.

[[user@host:~] mcopy -i "/dev/disks/mpx.vmhba34:C0:T0:L0:1" ::/ /vmfs/volumes/<datastore>/
Not ok Unix file ==> good

I’ve tested with FAT and FAT32 formatted partitions on the source USB drive and both worked just fine.

Before moving on to the installation of the copied NIC driver file test the file.

[user@host:~] unzip -l /vmfs/volumes/<datastore>/
Length Date Time Name
--------- ---------- ----- ----
324 11-20-2022 22:27 index.xml
205 11-20-2022 22:27 vendor-index.xml
3033 11-20-2022 22:27
356356 11-20-2022 22:27 vib20/vmkusb-nic-fling/VMW_bootbank_vmkusb-nic-fling_1.11-1vmw.800.1.20.61054763.vib
-------- -------
359918 4 files

The NIC driver zip file seems okay and is ready to be installed on the ESXi host.

Step 5: Install the ESXi NIC driver

Check my earlier post ‘Manually Updating ESXi Hosts‘ for more details how to install or update ESXi updates, patches and drivers. The command to install the driver would look like the example below. To make sure the integrity drive file is correct a dry-run installation of the driver installation can be performed first. In that case the ‘–dry-run‘ option can be added after the ‘install‘ command.

[root@host:~] esxcli software vib install --depot=file:///vmfs/volumes/<datastore>/
Installation Result
   Message: Operation finished successfully.
   Reboot Required: true
   VIBs Installed: VMW_bootbank_vmkusb-nic-fling_1.11-1vmw.800.1.20.61054763
   VIBs Removed:
   VIBs Skipped:

After rebooting the network connectivity to the ESXi host should be fixed. If you’re not rebooting the ESXi host after the drive installation, don’t forget to start the usbarbitrator service again.

To conclude

Luckily I was able to revive my Lab host using initial help of the vCommunity. Previously, I did not know tools like ‘mcopy’ even existed in ESXi and that the ‘usbarbitrator’ service needed to be stopped when u USB Drive needs to be locally.

All of this shows the importance of having a Lab to tinker around, make mistakes and more important learn how to fix it.

Cheers, Daniël

Useful links

Manually Updating ESXi Hosts

Woshub: Accessing USB Flash Drive from VMWare ESXi

VMware Documentation: Path based identifier


Leave a Reply

Avatar placeholder

Your email address will not be published. Required fields are marked *