RAM speedup

[Rupert Gallagher]

‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐
On Sunday 28 June 2020 13:58, Matthias Schniedermeyer <ms at citd.de> wrote:

> On 27.06.2020 11:22, Rupert Gallagher via rsync wrote:
>
> > On Friday 26 June 2020 21:58, Rupert Gallagher via rsync rsync at lists.samba.org wrote:
> >
> > > Hello,
> > > As disks are slow and rsync reads and writes so much that for the bus this is the equivalent of context switching galore, would it be possible to use RAM as a buffer? Say, you have 10GB of spare RAM, rsync uses the bus to its peak for reading 10GB, then again for writing it down. This would be more efficient than lot of small read/write operations.
> > > Thank you
> >
> > Current task: rsync 752 GB
> > source disk
> > Writing speed: 77 MB/s
> > Reading speed: 97 MB/s
> > target disk
> > Writing speed: 117 MB/s
> > Reading speed: 99 MB/s
> > Actual time: 380 min (6.3 hours) to copy 648 GB
> > Actual speed: 28 MB/s (648/380 = 1.7 GB min =~ 1700MB min / 60 min = 28MB sec)
>
> Unfortunatly you left out every other detail.


> Complete rsync commandline?

/usr/local/bin/rsync --recursive --links --times --modify-window=1 --devices --specials --update --owner --group --perms --delete --delete-before --delete-excluded --exclude-from=/etc/excluded_from_backup.conf --numeric-ids --outbuf=Block --inplace --link-dest=/backup/latest/ /archive /backup


> What hardware? (From the numbers it is only clear that you seem to talk about HDDs.)
> What HDDs?

source:
ST2000NX0403  sata hdd
Writing speed      : 117 MB/s
Reading speed      : 99 MB/s

destination:
ST5000LM000-2AN1 sata hdd
Writing speed      : 74 MB/s
Reading speed      : 89 MB/s

> What computer? (Laptop? Desktop? Server? Raspberry Pi? Age?)

Supermicro A2SDi-4C-HLN4F, newish

> What "Buses"? ( a) Any modern "bus" is NOT saturated by those numbers. b) All modern "buses" (Except USB, to some degree) are P2P, you can't even connect 2 devices to the same bus. (Except USB, but there are usually several controllers so you don't have to use same bus).)

Supermicro CSE-M14TQC 4xSAS/SATA bay, connected with a CBL-SAST-0616(50cm) Mini-SAS HD to 4 SATA cable. The CSE receives the 4 sata cables, the mini-sas end is plugged on the main board.



> With or without networking involved?

no network involved

> What Filesystem? What mount-options?

FFS2

rw,nodev,nosuid,softdep,noatime

> AVG Filesize? Directory structure? Fragmentation?

mixed

> Are you:
>
> -   Copying 1 752GB File(*)

no

> -   752 1GB Files

no

> -   10.000.000 78KB Files inside a directory-Tree with 10.000.000 Million subdirectories

no

> -   100.000.000 7.8KB Files in 100.000.000 Million subdirectories.

no

>     Personally my usual copying-speed for the "from local XFS formated HDD to local XFS formated USB-3-HDD"-case is about 350GB/Hour (or about 100MB/s.), and that has been the case for years.

I shall be so lucky...


>     *:
>     Even copying 2 different 752GB files can be quite a different monster depending on file-fragmentation.
>     Copying a heavily fragmented file (from a HDD) is slow, even when it is a single big file.
>     HDDs are very "sensitive" to sub-optimal usage, every 'seek' kills of several ms where 0KB can be read/written.
>     The worst-case performance (or IOPS) of a HDD is counted in KB/s(!!), if you have to seek after every sector read or written.
>
>     --
>
>     Matthias
>