Static server side listing

[Joachim Otahal (privat)]

Mark Constable schrieb:
> I've looking for a solution for this and no amount of googling has
> come up with anything.
>
> Is it possible to provide a static listing on a server, say every
> 24 hours, that a standard end-user rsync can pull and use?
>
> I have a lot of files to provide and the idea of every request
> dynamically providing a file list in real time is killing my
> server and is simply not needed. I am quite prepared to swap in
> (atomically) an alternate file tree every 24 hours as long as I
> can also provide a static file list. I know the files will not
> change for 24 hours and could easily handle a 10Kb to 100Kb static
> list being downloaded, plus the actual delta downloads, but adding 
> 100/sec listings of 20,000 files is a killer.

How many are requesting and how often, and does the server do more than 
providing files?
Taking 100 seconds to do a 20000 files list looks slow to me, more like 
time to upgrade RAM and/or HDD.
When it is OK to let the users have an 24h old filelist, is it at the 
same time OK if the user gets only up to 24h old files? If no then it 
can't be helped, if yes another solution would be to set up a seperate 
server for that high number of rsync requests which provides a 
once-per-day or once-per-hour rsynced copy of the main files. For that 
separate dedicated server a pre-generated list would be unneeded since 
the OS would cache the never-changing files.