rsync --link-dest won't link even if existing file is out of date
kmk at sanitarium.net
Wed Apr 15 00:45:17 MDT 2015
-----BEGIN PGP SIGNED MESSAGE-----
On 04/14/2015 11:35 PM, Henri Shustak wrote:
>> Ill take a look but I imagine I cant backup the 80 Million files
>> I need to in under the 5 hours i have for nightly
>> maintenance/backups. Currently it's possible by recycling
I would expect that recycling directories actually makes this worse.
With an empty target directory you don't even need the overhead of
- --delete (not as bad as it used to be thanks to --delete-during but it
is still overhead). If your backup window is only 5 hours then that
leaves you with 19 hours a day to do other things on your backup
server(s) such as deleting off old backups. Get all those unlink()
calls out of your backup window. Bad enough you need to do 80 million
calls to stat().
> To cover that many files in that much time you will require a high
> speed system. Just another thought. Perhaps splitting the backup
> onto multiple backup servers / storage systems would reduce the
> backup time so that it fits into your window?
Agreed completely here. It is much easier to make more backup servers
than it is to make one big one that can handle the entire load. We
divide our backup load by server. IOW, each backup server has a list
of production servers that it backs up.
> Also, I strongly agree with the previous posts relating to file
> system snapshots. ZFS is just one file system which supports this
> kind of system.
I have also attempted to use btrfs in Linux for this. I even wrote up
a presentation for my local LUG about it:
Unfortunately there was nothing but grief. The btrfs just wasn't
stable enough and the btrfs-cleaner kernel thread drove performance
into the ground. We eventually had to abandon it in favor of ZFS on
As far as "fast box" goes we decided on 8GB of RAM for most of the
backup servers and essentially whatever CPU can handle that much RAM.
Most of them are older AMD Athlon 64 X2 desktops. We do have one
with a quad core CPU and 16GB of RAM. That is the only one running
ZFS de-duplication as that is the big RAM hog.
This email is protected by LBackup, an open source backup solution.
Kevin Korb Phone: (407) 252-6853
Systems Administrator Internet:
FutureQuest, Inc. Kevin at FutureQuest.net (work)
Orlando, Florida kmk at sanitarium.net (personal)
Web page: http://www.sanitarium.net/
PGP public key available on web site.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2
-----END PGP SIGNATURE-----
More information about the rsync