pauses sync'ing between tmpfs and disk on Linux 2.4.x

[Tim Moore]

I use rsync similarly.  In addition to he standard /tmp traffic, junkbuster 
has a 4MB  jarfile and 18MB log, ccache has 16 directories and 8000 files 
of which 3000 are > 10k.  Both are heavily used.

At boot time /tmp is populated from on disk storage:
/bin/nice -19 /usr/bin/rsync -aq --whole-file /usr/TMP /tmp &

At shutdown /tmpfs deltas are put back:
/usr/bin/rsync -aq --delete --no-whole-file \
--exclude "*-lock" --exclude "*-unix" --exclude "*=:0.0" /tmp /usr/TMP &

Here are some representative timings (start == boot, stop == shutdown):
[9:26] abit:/tmp > egrep '(stop|start)\) .*  tmp' /var/log/messages.3
Mar 19 20:53:55 abit rc.local: (stop) starting  tmp /tmp/ /usr/TMP
Mar 19 20:53:57 abit rc.local: (stop) finished  tmp /tmp/ /usr/TMP
Mar 20 09:31:18 abit rc.local: (start) starting  tmp /usr/TMP/ /tmp
Mar 20 09:31:55 abit rc.local: (start) finished  tmp /usr/TMP/ /tmp
Mar 20 21:07:37 abit rc.local: (stop) starting  tmp /tmp/ /usr/TMP
Mar 20 21:07:41 abit rc.local: (stop) finished  tmp /tmp/ /usr/TMP
Mar 21 20:33:21 abit rc.local: (start) starting  tmp /usr/TMP/ /tmp
Mar 21 20:34:17 abit rc.local: (start) finished  tmp /usr/TMP/ /tmp

There is never a hang or pause, in fact the process would be transparent if 
I didnt have xosview in a corner window.  The problem probably isn't the 
way rsync, /tmpfs or 2.4 work, so you may want to look elsewhere.

kernel: 2.4.30-rc1
cpu/mem: athlon XP 2800, 1GB PC2700
rsync: 2.6.3
disk: 3xRAID-5 (old 20GB PATA drives, about 50MB/s throughput)

[9:07] abit:/tmp > df /tmp
Filesystem           1K-blocks      Used Available Use% Mounted on
tmpfs                   460800    159492    301308  35% /tmp
[9:18] abit:/tmp > ls -la
total 7729
drwxrwxrwt  11 root   root       480 Mar 26 08:57 .
drwxr-xr-x  26 root   root      1024 Mar 26 08:51 ..
-r--r--r--   1 root   root        11 Mar 26 08:52 .X0-lock
drwxrwxrwt   2 root   root        60 Mar 26 08:52 .X11-unix
drwxrwxr-x  18 tim    tim     163900 Mar 23 23:34 .ccache
drwxrwxrwt   2 xfs    xfs         60 Mar 26 08:52 .font-unix
drwx------   3 tim    tim         60 Jan 11 23:35 .wine-tim
drwx------   2 root   root        40 Feb  6 00:44 .xf86config1053
drwx------   2 root   root        40 Feb  6 00:46 .xf86config1077
-rw-------   1 tim    tim          0 Feb  9 22:24 0vws1uvh.zip
-rw-------   1 tim    tim          0 Feb  9 22:22 56gtsq4w.zip
-rw-rw-r--   1 tim    tim     178108 Mar 15 21:35 SATA_HotPlug.pdf
-rw-rw-r--   1 tim    tim     234529 Mar 15 21:33 SATA_PCI_CardBusHost.pdf
-rw-rw-r--   1 tim    tim     175906 Mar 15 21:36 SATA_illus_guide.pdf
srwx------   1 tim    tim          0 Mar 26 08:52 afterstep-500.DISPLAY=:0.0
-rw-r--r--   1 root   root     16686 Jan 30 17:18 bonnie.log
-rw-r--r--   1 root   root     35644 Jan  8 18:08 build.log
drwxr-xr-x   2 tim    tim         40 Mar  7 21:05 hsperfdata_tim
drwxr-xr-x   2 nobody nobody      80 Dec 25 23:56 junkbuster
drwx------   2 tim    tim         60 Mar 16 22:46 mcop-tim
-rw-r--r--   1 root   root    485728 Feb 21 12:18 rsync.log
-rw-rw-r--   1 tim    tim    6693164 Mar 15 21:40 serialata10a.pdf
-rw-rw-r--   1 tim    tim      27475 Jan 31 17:07 smp.log
-rw-r--r--   1 root   root         0 Feb 12 15:19 tmp.dd

t.

Ray Van Dolson wrote:
> I've set up a 1GB tmpfs filesystem on a system with a single IDE disk and
> 2GB's of memory.  I'm storing a large amount of RRD files (~300MB) on the
> tmpfs filesystem to make their generation a bit speedier... this part works
> great.
> 
> However, I want to rsync these files over from time to time to a directory on
> the local filesystem (same physical server).  I'm using rsync 2.6.4pre3 and
> am hoping to understand a bit better what is happening.
> 
> When I run rsync -av /path/to/tmpfs /path/to/diskdir things move along pretty
> fast for while, then there's a big pause... sometimes for up to 30+ seconds
> where nothing seems to be happening, but all IO on the system ceases (can't do
> anything in another xterm).  Then rsync starts moving along again for a while,
> and then pauses again... after about three such pauses it finishes the entire
> rsync process.
> 
> I'm wondering what I can do to speed things up... perhaps whatever processes
> that write to the tmpfs filesystem are fighting with rsync... but doesn't
> rsync just need read access?
> 
> I've also tried with -W (copy whole file) and used a smaller -B value.
> Anyways, I'll explain a bit about the curren scenario:
> 
> 1. Daemon on system receives data from remote devices.
> 2. Daemon calls rrdtool to write a .rrd file to the tmpfs filesystem.
> 3. rsync runs periodically (every 5 minutes) to sync up the tmpfs filesystem
>    with a directory on the local filesystem.
> 
> The tmpfs filesystem is a directory with about 100 directories inside of it,
> each containing rrd files.  I'm considering using a script to just rsync one
> of these subdirectories at a time over a period of time to "distribute" the
> load.
> 
> The main issue is that while rsync is running and enters these random 30
> second "pauses", no IO can happen and things get really backed up on the
> system and it ends up being faster for me to just dump the tmpfs completely
> and go back to writing directly to disk as slow as that was...
> 
> Any suggestions?
> 

-- 
  | for direct mail add "private_" in front of user name