Useful commands for troubleshooting VMware HCX

Troubleshooting VMware HCX has been one of my main tasks on several customer engagements lately. I compiled a list of useful commands that help me troubleshoot HCX on a daily basis. Before delving into details about the actual commands, let’s have a look at the basics.

CCLI Basics

What is HCX CCLI? – The Central CLI (CCLI) on VMware HCX allows you to execute commands available centrally on the HCX Manager to view the run time state for HCX services. The Central CLI reduces troubleshooting time by providing centralized diagnostics. It also improves the security posture of the Service Mesh appliances by eliminating the need to run the SSH service on them. To use it, first you must activate the Central CLI in VMware HCX Manager. Make sure that you read the HCX documentation on VMware’s online documentation page in order to familiarize yourself with the CCLI.

Check network connectivity behind proxy server

This test uses the curl command to verify if a proxy server is forwarding requests from HCX Manager to the Internet.

1. Open the HCX Manager Console or login via SSH.
2. Execute the following command: curl -v -I -x 10.0.11.1:9090 -k https://connect.hcx.vmware.com

Troubleshooting Site Pairing

There is an issue with site pairing. It is unknown if there is a proxy or firewall between HCX Manager and the internet. All the necessary ports are open. You can confirm this by executing the following command on the HCX Manager:

1. Open the HCX Manager Console or login via SSH.
2. Run the following command: openssl s_client -connect connect.hcx.vmware.com:443 -showcerts -servername connect.hcx.vmware.com

The output should look like this:

If you see a certificate from anything other than connect.hcx.vmware.com then there is proxy between the HCX Manager and the internet.

Perform system health check

1. Open the HCX Manager Console or login via SSH.
2. Open CCLI.
3. Execute the following command: hc
4. For additional details, execute: hc -d

Execute Performance Test

Just like the title says, this command performs performance check between the HCX-IX appliances.

1. Open the HCX Manager Console or login via SSH.
2. Open CCLI and go to HCX-IX appliance.
3. Execulte the following command to perfom all performance checks: perftest all
4. For specific performance tests, execute perftest –help command to see all available options.

Network Connectivity between Interconnect Appliances

To verify connection between two HCX Interconnect appliances you can use either traceroute or tcpdump command.

 Traceroute Method

1. Open the HCX Manager Console or login via SSH.
2. Start CCLI and go to the HCX-IX or NE appliance.
3. Use ssh command to open ssh session to this particular appliance.
4. Execute the following command: traceroute -I -s <LOCAL_HCX-IX_IP> <REMOTE_HCX-IX_IP>
5. If you want to test with a specific network port, for Example 4500, execute: traceroute -U -p 4500 -s <LOCAL_HCX-IX_IP> <REMOTE_HCX-IX_IP>

Tcpdump Method

This command can be used to verify if network traffic is arriving on the HCX-IX or NE appliance on a port 4500 which is used for communication between the appliances.

1. Open the HCX Manager Console or login via SSH.
2. Start CCLI and go to the HCX-IX or NE appliance.
3. Use ssh command to open ssh session to this particular appliance.
4. Execute the following command: tcpdump -ni any port 4500 -c 10

HCX Log Locations

Additionally, you may want to check HCX logs to investigate further. The logs can be found in the following locations on the HCX Manager and the appliances.

HCX Manager

Go to /common/logs/admin folder

• Check app.log for application events
• Check web.log for web service events

HCX Interconnect Appliance

Go to /var/log/vmware folder

• See hbrsrv*.log for warnings or errors

Happy troubleshooting! 🙂

Cheers!

– Marek.Z