Troubleshoot iSCSI TCP connectivity from ESXi with netcat

The ESXi port scanner: how to use the nc command to check if TCP ports on the iSCSI target are reachable.

Many VMware administrators might not be aware of ESXi having a simple, but functional, port scanner utility built in. Netcat for ESXi can be used to verify access to layer four TCP ports in an easy way.

When troubleshooting or verifying IP storage it is often useful to be able to test the physical network connectivity between the ESXi hosts and the iSCSI SAN target. The ordinary vmkping utility is very good and helpful. However, vmkping only verifies a physical path and that the remote IP replies with ICMP. If doing a successful vmkping we have still no real verification if the remote IP listen to the iSCSI TCP port and if it accepts TCP sessions from our hosts.

In the ESXi shell, available through either the direct console or SSH, you could use the netcat utility, called nc, to test layer four connectivity.

Assume we have this situation: two VMkernel adapters attached to a iSCSI storage LAN. The iSCSI target has the IP address 192.168.213.100.

Start a SSH session to the ESXi host and use the command line:

nc -z IP-ADDRESS TCP-PORT

The -z means that we want to just establish the session through the TCP three way handshake and then shut the session down gracefully. Zero (for -z option) data is sent.

Since the TCP port for iSCSI target is by default 3260 a command line could look like:

nc -z 192.168.213.100 3260

As seen above we get a nice and clear confirmation of the TCP status of the iSCSI target.

The port scan component of nc does only a three way TCP handshake and then a four way proper finish of the TCP session. This means a minimum of resources and traffic is used.

We can also verify that each of our VMkernel IP addresses has access. By using the option -s you could also specify the outgoing interface.

nc -z -s SOURCE-VMKERNEL-IP TARGET-IP TCP-PORT

For example, to specify vmkernel interface 192.168.213.52 and test if has the ability to setup a TCP session with the iSCSI target 192.168.213.100 use:

nc -z -s 192.168.213.52 192.168.213.100 3260

Above we see that both VMkernel adapters have been tested and verified connectivity all the way up to layer four (TCP) on the iSCSI Target.

Finally, if we should indeed have connectivity errors at some point, or just waiting for the service to come up after a target reboot or similar, and you like to test the nc tool multiple times then the -w options could be useful. If the remote TCP port is not reachable the nc timeout is very long on ESXi. If we suspect the target being not reachable and want a quicker response then set the timeout value with -w.

nc -z -w TIME-OUT-IN-SECONDS IP-TARGET TCP-PORT

nc -z -w 1 192.168.213.101 3260

In this case: no news is bad news. We did not manage to set up a TCP session and might lack connectivity even at lower layers. Use vmkping to check layer three access. If also using jumbo frames on the storage network be sure to use the correct parameters or the test is useless.

With the netcat (nc) utility we have the possibility to verify connections up to layer four with TCP from ESXi Shell or SSH.