RAM speedup

[Matthias Schniedermeyer]

On 27.06.2020 11:22, Rupert Gallagher via rsync wrote:
> On Friday 26 June 2020 21:58, Rupert Gallagher via rsync <rsync at lists.samba.org> wrote:
> 
> > Hello,
> >
> > As disks are slow and rsync reads and writes so much that for the bus this is the equivalent of context switching galore, would it be possible to use RAM as a buffer? Say, you have 10GB of spare RAM, rsync uses the bus to its peak for reading 10GB, then again for writing it down. This would be more efficient than lot of small read/write operations.
> >
> > Thank you
> 
> 
> Current task: rsync 752 GB
> 
> source disk
> Writing speed: 77 MB/s
> Reading speed: 97 MB/s
> 
> target disk
> Writing speed: 117 MB/s
> Reading speed: 99 MB/s
> 
> Actual time: 380 min (6.3 hours) to copy 648 GB
> Actual speed: 28 MB/s (648/380 = 1.7 GB min =~ 1700MB min / 60 min = 28MB sec)

Unfortunatly you left out every other detail.
Complete rsync commandline?
What hardware? (From the numbers it is only clear that you seem to talk about HDDs.)
What HDDs?
What computer? (Laptop? Desktop? Server? Raspberry Pi? Age?)
What "Buses"? ( a) Any modern "bus" is NOT saturated by those numbers. b) All modern "buses" (Except USB, to some degree) are P2P, you can't even connect 2 devices to the same bus. (Except USB, but there are usually several controllers so you don't have to use same bus).)
With or without networking involved?

What Filesystem? What mount-options?
AVG Filesize? Directory structure? Fragmentation?

Are you:
- Copying 1 752GB File(*)
- 752 1GB Files
- 10.000.000 78KB Files inside a directory-Tree with 10.000.000 Million subdirectories
- 100.000.000 7.8KB Files in 100.000.000 Million subdirectories.

Personally my usual copying-speed for the "from local XFS formated HDD to local XFS formated USB-3-HDD"-case is about 350GB/Hour (or about 100MB/s.), and that has been the case for years.

*:
Even copying 2 different 752GB files can be quite a different monster depending on file-fragmentation.
Copying a heavily fragmented file (from a HDD) is slow, even when it is a single big file.
HDDs are very "sensitive" to sub-optimal usage, every 'seek' kills of several ms where 0KB can be read/written.
The worst-case performance (or IOPS) of a HDD is counted in KB/s(!!), if you have to seek after every sector read or written.


-- 

Matthias