<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-GB link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>We are using rsync to maintain a backup of some relatively large file systems, around 3 million files, but in an effort to save space have decided that some files should not be synchronised. In order to do this we have created a filter file containing the list of files to exclude. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>This filter file is created on what is ultimately the destination end of the synchronisation so we first copy this file to the source end.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>rsync --stats --log-file=/tmp/rsync_mp3.log --archive --human-readable /mnt/backup/audio/mp3_64/.rsync-exclude 10.0.60.64:/mnt/encoded/audio/mp3_64/<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Then we perform the main synchronisation from source to destination<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>rsync --stats --log-file=/tmp/rsync_mp3.log --archive --human-readable --filter="merge,e /mnt/backup/audio/mp3_64/.rsync-exclude" 10.0.60.64:/mnt/encoded/audio/mp3_64/ /mnt/backup/audio/mp3_64/<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I have noticed that the .rsync-exclude file does not seem to be processed. I see no errors related to this in the output but believe it is because it contains over 120,000 entries and is maybe exceeding some internal 16bit limit in rsync of the number of entries that can be in a filter file. We have other file systems where the filter file has around 30,000 entries and this seems to be working fine. <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>I was wondering:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Does anyone knows if this is a hard internal limit that we have hit?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>If so, Is it possible to get around this limit by specifying more than one file, not per directory files, just multiple files?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>If not, I guess that moving to per directory filter files would be the answer, we have around 5,000 directories on the file system. The reason we chose a single file initially was that it is very quick to synchronise this one file. If I do use per directory files, is there a way to synchronise those files from destination to source quickly without checking all the other files on the destination end?<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>For instance running this takes an awfully long time as it searches through every dri & file for the .rsync-exclude files<o:p></o:p></p><p class=MsoNormal>rsync --stats --log-file=/tmp/rsync_mp3.log --archive --human-readable --dry-run --progress --filter="include, .rsync-exclude" --filter="include, */" --filter="exclude, *" /mnt/backup/audio/mp3_64-CLIP/ 10.0.60.64:/mnt/encoded/audio/mp3_clip/<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>This is the verison of rsync being used:<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>rsync version 3.0.6 protocol version 30<o:p></o:p></p><p class=MsoNormal>Copyright (C) 1996-2009 by Andrew Tridgell, Wayne Davison, and others.<o:p></o:p></p><p class=MsoNormal>Web site: http://rsync.samba.org/<o:p></o:p></p><p class=MsoNormal>Capabilities:<o:p></o:p></p><p class=MsoNormal> 64-bit files, 64-bit inums, 64-bit timestamps, 64-bit long ints,<o:p></o:p></p><p class=MsoNormal> socketpairs, hardlinks, symlinks, IPv6, batchfiles, inplace,<o:p></o:p></p><p class=MsoNormal> append, ACLs, xattrs, iconv, no symtimes<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Thanks in advance for any help.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Regards<o:p></o:p></p><p class=MsoNormal>Ben<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p></div></body></html>