Issue

We have a Veeam Repository Server  (6.5.0.106) located in Datacentre 1. We have a 1Gb link to Datacentre 2 which houses a Veeam Proxy server. The Veeam Repository Server calls all the jobs both locally in DC1 and in DC2 remotely. All backup data is stored locally – via pRDMs – to their DC (so there is no storage traffic going over the 1Gb link). Hosts involved are all ESXi 5.0 and the backup mode is "Virtual Appliance".

During some network maintenance we temporarily severed the 1GB link between the DCs. During that time the backups failed – as you'd expect. Since then the jobs have been failing with an array of error messages – which I have screenshot and logged below:
a) "Error: Client error: VDDK command cannot be completed in reasonable time. "
b) "Cannot use CBT: Soap fault. Detail: '', endpoint: ''"
c) "Cannot use CBT: Soap fault. Detail: '', endpoint: ''
Timed out waiting for all VDDK disks to close.
Error: Client error: Soap fault. Detail: '', endpoint: '' Unable to retrieve next block transmission command. Number of already processed blocks: [5064]. "

Here is an example of the error.

But the errors/warnings on VMs wouldn't be constant, so sometimes one VM would backup without error, one would backup with a CBT warning and another would fail.

 

I suspected it might not be using the correct Veeam Proxy so I forced the job to use the correct one (although it was set to Automatic it looked liked it was trying the DC1 Veeam proxy before DC2). The Veeam Proxy and Veeam Repository Server were both rebooted and size was recalculated.

Backup still failed

1) Veeam support recommended the following

Un-tick the "Use Multiple TCP/IP connections per job" which can be found in MENU -> Traffic Throttling.

Backup still failed

2) The Veeam support team thought it might have been the API timing out (as a lot of the VMs were failing after 20 mins) so we changed from [hotadd] to [nbd] mode.

To do this you select Backup Infrastructure -> Backup Proxies ->  Right click the required Veeam Proxy (172.17.19.5 in my case) -> Properties.  Then select the Transport Mode by clicking Choose and then the required mode. In my case I chose Network for the test.

Backup still failed

 

3) I then created a brand new job (although leaving out a quite large VM of about 1TB). It *almost* worked (see below):

Unfortunately it stopped at 85% of the last VM (the disk it failed on was 300MB)

Veeam Support suggested it was worth upgrading from 6.5.0.106 -> 6.5.0.144. I patched this and then tried a "retry" of the job. This time it completed successfully.

I wasn't certain it was completely working so the following evening I ran the job again (this time adding the large VM I had omitted previously).

 

Resolution

It appears a patch fixed the issue in the end.  Unfortunately it took 7 days to locate the problem so I guess I will start there in the future !!