ESXi 6.5 – 503 Service Unavailable

Posted on by Jordansphere

Problem

There was a storage issue which caused one particular host not to recover correctly.  I was unable to login to the ESXi GUI and when I tried to restart the management services the GUI completely stopped functioning with the following error:

503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http16LocalServiceSpecE:0x03ff40e8] _serverNamespace = / action = Allow _port = 8309)

The majority of VMs were still working but 2 VMs were in read-only mode due to the storage issues.

I wasnt getting the correct responses from from esxcli commands

eg

# esxcli storage nmp device list

displayed 

"Connection failed"

 

Resolution

After fully resolving the storage issues and restarting hostd and vpxa services (along with the services.sh again) the GUI was still was giving the 503 errors.

After exhausting all options the last restort was power down any working VMs using the OS then reboot the host via the DCUI  with "force VM reboot" option enabled. This restarted all the VMs on the the other hosts using HA.