[clug] Disk size

[Chris Smart]

On Tue, 28 Aug 2018, at 14:28, Eyal Lebedinsky via linux wrote:
> I thought we need some comic relief after the recent fun on the hill. 
> Not Linux specific but...
> 
> For a while now I noticed that recently acquired USB disks offer less 
> storage than advertised.
> 
> For example, a 32GB disk
> 	SanDisk Ultra USB 3.0 Flash Drive 64GB
> provides (as fdisk tells)
> 	Disk /dev/sdj: 57.9 GiB, 62109253632 bytes, 121307136 sectors
> 
> A smaller one
> 	SanDisk Ultra Fit USB 3.0 Flash Drive 32GB
> has
> 	Disk /dev/sdj: 28.7 GiB, 30752636928 bytes, 60063744 sectors
> 
> I just now saw a message (unrelated) on the linux-raid list mentioning:
> 	"I should note that for some reason the "32G" Optane only has 29.260 G
> 	bytes (27.3 GiB)"
> So SSDs are afflicted too.
> 
> What is this trend? The GB/Gib excuse ran out of steam? What is it now?
> 
> Will disk space collapse into a black hole and become a write-only medium?
> 

I expect the extra difference is probably due to over-provisioning, the amount of space reserved on an SSD for managing the writing of data (mostly required as the drive starts getting full). The amount of space will often be different for each vendor, but probably somewhere between 5-10%.

IIRC, an SSD cannot write new data unless it first goes and clears out the block. This is a slow operation and so SSDs instead write to new blocks that it knows are already empty and then later comes back and clears out (garbage collects) the unused blocks so that they can take new data straight away next time.

Writing to fresh blocks is fine while there's lots of spare space, but as an SSD gets full, the drive can't easily write to new blocks due to fragmentation (where some pages in a block are written), so it slows down a lot as it has to move data and clear out blocks first. The over-provisioning space works around this problem by just always making sure there is some free space (which the user cannot use).

Notably, some manufacturers like Samsung will allow you to change the size of this over-provision (with a Windows tool), and making it larger (say 25%) results in much greater performance as the drive gets full.

-c