[clug] Is a weekly RAID scrub too much?
Rodney Peters
rodneyp at iinet.net.au
Mon Feb 27 00:05:46 UTC 2017
On 26/02/17 23:11, Eyal Lebedinsky wrote:
> On 26/02/17 22:16, Paul Wayper wrote:
>> On 26/02/17 17:32, Eyal Lebedinsky wrote:
>>> On 26/02/17 17:15, Paul Wayper wrote:
>>>> On 24/02/17 22:35, Eyal Lebedinsky wrote:
>> [...]
>>>>> Is the scrub doing more harm than good by shortening the service
>>>>> life of the
>>>>> disk?
>>>>
>>>> So, what do you mean by "weekly RAID scrub"?
>>>>
>>>> I've never heard of anyone doing this on a RAID array.
>>>>
>>>> I've used 'scrub' on a disk or files to overwrite them with random
>>>> data to
>>>> avoid anyone using a scanning electron microscope on them to view
>>>> your data
>>>> again. With disk geometries, cylinder tolerances, and encoding
>>>> mechanisms
>>>> these days, even overwriting with zeros will render the data
>>>> inaccessible to
>>>> almost all attackers.
>>>
>>> I use 'shred' for this.
>>>
>>>> But I've never heard of 'RAID scrubbing'. What are you doing, Eyal?
>>>
>>> RAID scrub refers to, in linux software RAID, a 'check' request. It
>>> means that
>>> the full RAID is read and verified. For RAID6 it means P and Q are
>>> verified.
>>
>> I looked up the ever-helpful Arch Linux pages on RAID and LVM:
>>
>> https://wiki.archlinux.org/index.php/Software_RAID_and_LVM#Scrubbing
>>
>> That took me to Wikipedia's page on "Data Scrubbing":
>>
>> https://en.wikipedia.org/wiki/Data_scrubbing
>>
>> Which contained lots of hyperbolic, panic-inducing language like:
>>
>> """Data integrity is a high-priority concern in writing, reading,
>> storage,
>> transmission, or processing of the computer data in computer
>> operating systems
>> and in computer storage and data transmission systems. However, only
>> a few of
>> the currently existing and used file systems provide sufficient
>> protection
>> against data corruption."""
>>
>> To me this is totally overblown. In all the many years I've been
>> looking at
>> files, I don't think I've ever seen a file corrupted on a healthy
>> disk. I've
>
> I had the scrub discover problems a few times. I do not know the
> reason for those
> failures, there was no disk error logged and I was not aware of any
> event that
> could cause it.
>
> However, these problems pointed to files that were read happily but
> the data
> delivered is likely to be bad.
>
> [aside]
> We do not have a "safe read" option for raid6, which could check data
> as it is
> read and correct it if necessary (and possible). Furthermore, while
> raid6 can
> do a 'repair', which is like a 'check' but on detecting an error it
> recalculates
> the syndromes, this repair is not so good. raid6 has enough data to
> detect
> *which* sector is bad and rewrite it rather than assume the data is
> good and
> the syndromes are bad. Note: if a disk returns an actual i/o error
> then the
> kernel does repair the failed sectors on the fly.
>
> In short, when a 'check' error is detected I identify the affected
> files and
> deal with their content, restoring from backup if necessary.
> [/aside]
>
>> seen one IBM XT with dodgy memory that started corrupting everything
>> that was
>> written to disk - and funnily enough running full disk scans made the
>> problem
>> worse. In the case of RAID 5 and 6 you've already got full
>> redundancy should
>> one piece go missing - you can literally lose an entire disk and keep
>> going in
>> degraded mode, so why do you care if a block goes bad?
>>
>> My favourite bit was:
>>
>> """Due to the high integration density of contemporary computer
>> memory chips,
>> the individual memory cell structures became small enough to be
>> vulnerable to
>> cosmic rays and/or alpha particle emission. The errors caused by these
>> phenomena are called soft errors. This can be a problem for DRAM- and
>> SRAM-based memories."""
>>
>> So the solution to the possibility of one of the sectors going
>> silently bad -
>> despite all the systems in the drive itself to detect and correct
>> these errors
>> - is to read it into memory which can also be faulty? Tell me again
>> how this
>> is going to help? :-)
>>
>> Joking aside, I think maybe a scrub once a year might be a
>> possibility if
>> you're dealing with hardware you don't entirely trust. But I'd keep
>> it to a
>> minimum because, as you observe, drives do wear out with overuse, and
>> full
>> disk scans will do that.
>
> I doubt a yearly scrub will ever come out clean :-(
> But I probably need to reduce the frequency to monthly.
>
> Question: How often other people do a scrub?
>
>> It's actually one of the common failure modes in RAID arrays - you have
>> several near-failing drives but they're fine with the normal
>> workload. One
>> drive fails and you replace it, but now the RAID rebuild has to scan
>> every
>> single disk, so all the other drives near fail point are pushed over
>> the limit.
>
> True, but while seeing a second disk fail during a rebuild is scary,
> it is not
> fatal. In my experience disks fails slowly, a few bad sectors at a
> time, and I
> can deal with a few bad sectors while replacing a failing disk.
>
> [aside]
> I did not yet have a catastrophic failure in the 12 years of running
> software
> raid, but the last (current) set of disks is the worst. These are WD
> blacks and
> out of 7 disks six were replaced (some more than once) and the seventh
> had
> "events" suggesting it is next. This is after 3.5 years in the planned
> 5 years
> life of this array.
>
> BTW, these disks do not list a workload limit, and I assume they are
> designed
> for continuous use with a high load.
I have not used Blacks, but you might not have them in suitable
application. WD list Blacks in the personal storage category and not in
Business storage. The spec does put a limit of 300k load/unload cycles.
I recognise that this is an aside, but perhaps Red, that were available
3.5 yr ago or the newer Gold, would be a more appropriate choice for the
impending replacements.
> [/aside]
>
>> But by "common" here I mean "this almost never happens". I've heard
>> anecdotes, and I'm hoping some people on the list can share hilarious
>> stories
>> about drive and array failures :-)
>>
>> So, anyone running BTRFS? Can you grep your logs for "csum failed
>> ino" and
>> tell us all how often you see that error? BTRFS stores checksums on
>> metadata
>> and data (XFS stores checksums only on metadata), and verifies the
>> checksum on
>> read and re-reads data if it's not correct. If we could get some
>> kind of
>> statistical analysis of the number of reads and writes performed vs
>> the number
>> of actual checksum failures, we'd get an idea of how likely they are
>> in the
>> real world.
>>
Perhaps not quickly. I do run BTRFS, although on a systemd system and
might need a bit of reading to figure out what the equivalent grep
phrase would be.
>> Have fun,
>>
>> Paul
>
> cheers
>
Rod
More information about the linux
mailing list