[Samba] WinNT offline file attribut

[John E. Malmberg]

Illtud Daniel wrote:
 > "John E. Malmberg" wrote:
 >
 >> On Thu, 25 Apr 2002, Illtud Daniel wrote:
 >
 >
 >> The question that none of the Windows based HSM vendors would give
 >> me an answer on was: Is there any way to make sure that a copy of
 >> all files shelved and unshelved exists on the storage robot, and
 >> how do I restore things when the real disk fails.  I would think
 >> that question should be easy to answer.
 >
 > Now I'm confused. It may be because that before coming to this
 > thread my HSM terminology was different to yours. I use 'migrated'
 > for when a file is written to tape (or optical, whatever). 'fetched'
 > for getting it back and writing it to the stub file, 'purged' for
 > removing the file from the extended volume and replacing it with a
 > stub file. You purge only migrated files (for obvious reasons), and
 > a file open on the stub will trigger a fetch.

Ok, your terms are more precise than mine.

 > To answer your questions with regard to OTG DX2000: How do you know
 > that a copy of all files shelved exists on the robot? Assuming you
 > mean 'shelved' to be 'migrated & purged' and 'unshelved' to mean
 > 'migrated but not purged', then you don't know, you trust. You can
 > run tape reports to list what's on each tape, but bugs
 > notwithstanding, if a file's been migrated, then it should be on the
 > tapes. DX can backup its internal database (stub file -> tape
 > location) to file which you can stick somewhere safe. In event of
 > distaster, you can rebuild the stubs by just restoring from this file.
 > You can read about this on OTG's website:
 > http://www.otg.com/KnowledgeBase/default.htm try 'dxdrivedump.exe' -
 > that knowledge base will give you a lot of info on how DX does stuff.

Yes, I would want to make sure that all files exist on a tape after they
have have been in existance for a period of time.

HSM software should be doing disk space management in the background, 
not necessarily shelving and purging files on demand.

HSM directories are by nature usually read only archives, so the HSM 
software should be able to migrate but not purge in the background so 
that if it needs free space in a hurry, it does not have to do a migrate 
  and purge to get it.

 >>
 >> I am sure that there is interest.  Because of the caching issue,
 >> unless you have enough files so that they all do not fit on the
 >> disk, you may not notice a performance problem.
 >
 > :)  We'll force them to tape, don't worry! The disk is 1.44TB, so yes,
 > we'll have a job filling it, but you can set policies to ensure that
 > migrating starts when your disk reaches 0.01% full, or to migrate as
 > soon as a file appears on the volume, and to force a purge (stubify)
 > of all files.

Yes you can force a purge, but if the HSM software is smart, it will do 
anything it can to avoid going to the robot to retreive a file, so even 
if it claims to have replaced a file with a stub, the file may still be 
resident on the disk.

So when a file is opened, the HSM software just swaps the stub with the 
actual file that is still on the disk becasue the migration process did 
not really delete the file.

So unless you cause the HSM software to overwrite the files that had 
been "purged", you will not see the performance impact.

Virtual memory works the same way.  When my process is over it's quota, 
some pages get moved from my address space and put on the free or 
modified lists to be used by other processes.

If my process accesses those pages, if the page is still on the free 
list or the modified list, I get the same page back instead of having to 
go to the disk and wait for it.

 > By the way, we're moving to ADICs AMASS because we've had problems
 > with DX2000 which have never been resolved (including loss of
 > hundreds of gigs of content, which isn't fun). An open HSM solution
 > would be a godsend.

Unfortunately the number of users that need an HSM system is small, and 
with the size of hard drives shrinking fast, getting smaller.

The cost of the robot hardware is the gating factor.

Now using a gzip archive instead of a robot, may make it more 
economically feasable.

-John
wb8tyw at qsl.network
Personal Opinion Only