Is it possible to make rsync VMware split .vmdk's aware?

Tue Mar 24 18:52:38 GMT 2009

Hi Matt,

I am working a couple of days now with monolithic files and I keep
hitting io time outs on large vmdk's while the line (IPSec tunnel) stays
online.

24-03-09 18:08:16 --> ERROR: /usr/local/bin/rsync --progress --stats
--compress --timeout=21600 --block-size=2048
--log-file=/vmware/scripts/dailyfullbackup/logs/20090323_23.45.01_dailyf
ullbackup.log-SteadySync-PVSBS2K3
/vmware/scripts/dailyfullbackup/localbackups/PVSBS2K3/20090323_23.45.01/
* 192.168.100.20::SteadySync/PNM-VMW01/PVSBS2K3/ got an error!
24-03-09 18:08:16 --> Adding SteadySync logfile for this VM:
2009/03/24 02:13:37 [20518] 
2009/03/24 02:13:37 [20518] building file list
2009/03/24 04:52:43 [20518] <f.sT.... PVSBS2K3-1.vmdk
2009/03/24 18:07:16 [20518] <f.sT.... PVSBS2K3-2.vmdk
2009/03/24 18:07:16 [20518] <f..T.... PVSBS2K3.vmx
2009/03/24 18:08:16 [20518] io timeout after 43567 seconds -- exiting
2009/03/24 18:08:16 [20518] rsync error: timeout in data send/receive
(code 30) at io.c(165) [sender=2.6.9]

I have already used an high timeout value. Could you please explain how
the timeout works and when it could happen? I have noticed that the
sender rsync process shows a larger filesize progress than actually
written on the receiver at that moment. This difference keeps getting
greater on the way to the end of the file.

PVSBS2K3-2.vmdk = 83Gb

I am using an older version as you see, this is because when I was
developing the backup, rsync version 3.0 was just released and I felt
better with the use of latest older version. Will 3.0.5 solve the
problem or handle large files better?

I'd like to hear from you and thanks in advance,

Kind regards,

Bas Bahlmann

Steady IT Systeembeheer
Hondsdrafweg 24  | 1508 BX Zaandam
t 075 - 789 06 40
f 075 - 890 16 18
i www.steadyit.nl

-----Oorspronkelijk bericht-----
Van: Matt McCutchen [mailto:matt at mattmccutchen.net] 
Verzonden: woensdag 18 maart 2009 16:52
Aan: Bas Bahlmann || Steady IT Systeembeheer
CC: rsync at lists.samba.org
Onderwerp: Re: Is it possible to make rsync VMware split .vmdk's aware?

On Wed, 2009-03-18 at 09:02 +0100, Bas Bahlmann || Steady IT
Systeembeheer wrote:
> -         I am making a local copy with vmware-vdiskmanager to an USB
> disk in the split "thin-disk" format of the vmdk's
> 
> -         Then I start rsync to our datacenter to replicate the split
> "thin-disk" vmdk's

> Sometimes, because of the defragment within the VM or Exchange, the
> disk layout changes so much that a split .vmdk file that was very
> little and now becomes filled with 2Gb data. As a result rsync has to
> transfer 2Gb of data for that .vmdk which takes a lot of time. In my
> opnion that's not nessesary because the data is probably available in
> another split .vmdk because it was moved across the virtual disk.

> Is it possible to make an option in Rsync which reads out the vmdk
> config file for the split disks so it can search for known data across
> all the split .vmdk files within one virtual disk?

Specific support for VMware split images would be way too
special-purpose to justify putting it in the main version of rsync.  A
more general option to make each file transfer use multiple basis files
(or even all the existing destination files) may be reasonable.  You
could join this enhancement request, which I might then reopen:

https://bugzilla.samba.org/show_bug.cgi?id=5954

But in this scenario, why are you using a split image?  Does it have
some advantage over a monolithic image (with which rsync would Just
Work)?

-- 
Matt