Scenario:  I needed to reboot a host but a single VM was not migrated when I called the Redeploy VMs command in vCloud Director. It was if VCD didn’t recognise the VM as vSphere didn’t even attempt to move it off the host.

First of all, here are the instruction to place an ESXi host managed by vCloud Director into maintenance mode.

 

1)  Log in to vCloud Director portal. Click Manage & Monitor then Hosts on the left hand side. Right click Disable Host.

 

 

2) Once complete, right click the the host again and select Redeploy All VMs

3) This should then migrate the VMs off the host then enter and exit maintenance mode

 

At this point I looked at vCenter and watched as the "ghost" VM wasnt even attempted for migration. After a while the operation to enter maintenance mode just timed out.

Looking in the VCD, the following error appeared:

com.vmware.ssdc.library.exceptions.VimTimedoutException: The vCloud Director operation has timed out.
vCenter Server task (moref: task-341231) failed in vCenter Server 'vcenter.jordansphere.local' (68207e7b-e232-4777-abf8-05525b07eefa).
    at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:518)
    at com.vmware.ssdc.library.vim.LmVim.Convert(LmVim.java:552)
    at com.vmware.vcloud.val.taskmanagement.TaskWaiterInvRecord.CheckForError(TaskWaiterInvRecord.java:196)
    at com.vmware.vcloud.val.internal.impl.VC20ManagedServer.enterMaintenanceMode(VC20ManagedServer.java:1102)
    at com.vmware.vcloud.val.internal.impl.VC20ManagedServer.evacuateVirtualMachines(VC20ManagedServer.java:1056)
    at com.vmware.ssdc.backend.CManagedServer.evacuateVirtualMachines(CManagedServer.java:2917)
    at com.vmware.vcloud.rcl.impl.RclServiceImpl.migratePoweredOnVirtualMachinesTask(RclServiceImpl.java:1214)
    at com.vmware.vcloud.rcl.impl.RclServiceImpl.executeTask(RclServiceImpl.java:2056)
    at com.vmware.vcloud.management.system.TaskServiceImpl$3.doInSecurityContext(TaskServiceImpl.java:1248)
com.vmware.vcloud.val.internal.impl.VC20ManagedServer.evacuateVirtualMachines(VC20ManagedServer.java:1056)
    at com.vmware.ssdc.backend.CManagedServer.evacuateVirtualMachines(CManagedServer.java:2917)
    at com.vmware.vcloud.rcl.impl.RclServiceImpl.migratePoweredOnVirtualMachinesTask(RclServiceImpl.java:1214)
    at com.vmware.vcloud.rcl.impl.RclServiceImpl.executeTask(RclServiceImpl.java:2056)
    at com.vmware.vcloud.management.system.TaskServiceImpl$3.doInSecurityContext(TaskServiceImpl.java:1248)
    at com.vmware.vcloud.management.system.TaskServiceImpl$3.doInSecurityContext(TaskServiceImpl.java:1243)
    at com.vmware.vcloud.management.system.TaskServiceImpl$SecurityContextTemplate.executeForOrgAndUser(TaskServiceImpl.java:1609)

 

Still no clues. I decided to investigate the VM itself. As it turns out the VM was (stuck) in the middle of a VMware Tools installation. I  tried to eject the media using VCD but that did not work.

The solution was to log into vCenter right click the VM -> End VMware Tools Install 

 

This then allowed me to redeploy the VM successfully via VCD.