Building hash table times for large files

Rob Bosch robbosch at msn.com
Fri Nov 2 17:30:24 GMT 2007


I'm running pre4 on a 77GB file.  It seems like the hash table is taking a
long time to be built.  I'm not sure what is involved in this step but as an
example the following is logged during a run:

send_files(11, priv1.edb) 
send_files mapped priv1.edb of size 79187419136 
calling match_sums priv1.edb 
<f.st...... priv1.edb 
hash search b=131072 len=79187419136 
built hash table for entries 0 - 52427 
built hash table for entries 52428 - 104855 
built hash table for entries 104856 - 157283 
built hash table for entries 157284 - 209711

The first two hash table entries (0 - 52427 and 52428 - 104855) took only a
few minutes.  The next two entries written to the log file have taken
several hours.  

Does this make sense?  Should the entries take so long to be built?

Rob





More information about the rsync mailing list