Static server side listing
Joachim Otahal (privat)
Jou at gmx.net
Thu Dec 22 13:32:14 MST 2011
Mark Constable schrieb:
> I've looking for a solution for this and no amount of googling has
> come up with anything.
> Is it possible to provide a static listing on a server, say every
> 24 hours, that a standard end-user rsync can pull and use?
> I have a lot of files to provide and the idea of every request
> dynamically providing a file list in real time is killing my
> server and is simply not needed. I am quite prepared to swap in
> (atomically) an alternate file tree every 24 hours as long as I
> can also provide a static file list. I know the files will not
> change for 24 hours and could easily handle a 10Kb to 100Kb static
> list being downloaded, plus the actual delta downloads, but adding
> 100/sec listings of 20,000 files is a killer.
How many are requesting and how often, and does the server do more than
Taking 100 seconds to do a 20000 files list looks slow to me, more like
time to upgrade RAM and/or HDD.
When it is OK to let the users have an 24h old filelist, is it at the
same time OK if the user gets only up to 24h old files? If no then it
can't be helped, if yes another solution would be to set up a seperate
server for that high number of rsync requests which provides a
once-per-day or once-per-hour rsynced copy of the main files. For that
separate dedicated server a pre-generated list would be unneeded since
the OS would cache the never-changing files.
More information about the rsync