[clug] Partitioning for fun and benefit

Thu Oct 17 12:27:18 UTC 2019

Thanks Steve and others for comments so far.

In my email, I was sharing my research into understanding the benefits of, and how, when and where to use 'partitioning' to improve performance, robustness, and security, for both linux servers and linux personal computers, particularly Debian installations. 

My Linux partitioning experiences have been with small business and personal systems (home servers and home personal computers), but I would not mind hearing about partitioning used in enterprise environments.

On Thursday, 17-10-2019 at 16:55 steve jenkin wrote:
> George,
> 
> Unix grew up needing a lot of ‘volumes’ mounted - because disks were small compared to storage requirements and were _slow_ - both rotational delay/ latency and head movement (even time to switch between disks).
> 
> There was a time before ’swap’ could be within a filesystem - you had to allocate a raw volume or three for ’swap space’.
> But now, don’t even need that.
> 
> These days, with MBR or GPT, you can throw everything into a single volume / partition and have it Just Work.
> Running MBR, especially with GRUB, you need a separate “/boot” partition AFAIK.
> 
> People who quote “Security” as a reason aren’t, to me, understanding the problem.
> It’s much, much easier to break into an overly complex system than a minimal, simple system - because People Make Mistakes - and to maintain security for a long delta-T, you’re allowed ZERO mistakes through the whole period.

I think my testing may have looked over the top, i.e. overly complex, as I merged all partitioning suggestions into one example for testing.  Typically different types of Linux servers have different vulnerabilities, for example web servers can set noexec on /tmp to help reduce the attach surface from web applications.

>From my reading (see the links I provided), adding a [small] level of Security is certainly one reason that is given. 

I saw a number of suggest separating root from data volumes that could consume all available space and breaking the system. For my simple KVM installations, I have learned to separate /var/lib/libvirt/images from /root as I get too enthusiastic in making VMs and use up all available space.

I believe some like to separate /var/log from /var/log/audit because when they use the same partition one method to break auditing recording a hackers actions is consume all available space by spamming into the syslog and other logs.

The web links I provided give various reasons for separating specific partitions.

> 
> The cyber-security problem is highly asymmetric:
> 	- attackers only have to find _one_ way in,
> 	- the defenders have to create and maintain 100% coverage, they need to block _all_ ways in.

Using partitioning, is just one way to reduce 'the ways in'. Not the only or best way by any means.

> 
> Anyone who assumes non physically isolated systems can contain a ‘determined attacker’ to a selected sub-set of the system (eg partitions) hasn’t thought it through. There’s even attacks to escape from VM’s - not sure how many.

It was explained to me that there is no "completely secure system", our goal is to make it more troublesome than it is worth for the hacker. Sadly some hackers are just looking for the challenge, the more challenging you make it, the more fun it is for them.

> If anyone gets super-user access, lots of ‘volumes’ are useless - they offer attackers more, not fewer, places to hide things and ways to break things.

If someone gets super-user access, true,  ‘volumes’ are useless. The point is to make it harder to obtain super-user access, which I understand why people recommend using nosuid on /tmp.

> 
> To see what works in ’secure storage’, research what the likes of Edward Snowden and Andy Greenberg do with their machines.
> I doubt they use an avalanche of partitions.

I would love to know how security aware people protect their systems.  I am starting my research with the very simple method of using partitions. I expect SELinux and AppArmor are more effective methods, but my current research is on the topic of partitioning.

> 
> I’ve not built a system with GPT & UEFI BIOS - I understand they don’t require a boot-loader (eg GRUB) because UEFI scans partitions.
> 
> Logical Volumes were initially created to lash multiple small disks together into One Big Volume, then disks got bigger than normal storage requirements and that use went away.
> 
> So LVM’s were co-opted to provide RAID levels and other stuff - which IMHO is conflating operations of two independent subsystems.
> LVM’s are now typically much smaller than a full disk - they get useful differently now.
> 
> Operationally, LVM gives two significant operational benefits:
>  - with dynamic resizing, don’t have to run out of space in a volume, increase as needed without affecting operations
>  - on-the-fly disk upgrade / replacement:
> 	- add new physical disks to the system (need hot-swap / removable drives)
> 	- create PV’s the same size as existing PV’s
> 	- mirror old and new PV’s, LVM initiates auto-sync
> 	- wait for sync to finish
> 	- take old PV’s off-line (confirm not in use)
> 	- removed unused disks, go to pub at 5PM with everyone else.
> 
> Since 2009 when we hit 4TB HDD’s, the Drive Doubling Time slowed to a crawl…
> There’s not that many people that need to lash together multiple 8TB drives - most systems could get away with RAID-1 for protection against single HDD failure/ fault. [but not controller, backplane, memory, data corruption, etc]

If my memory is correct, my first Hard drive was cutting edge, a 33MB voice coil when RRL'ed and in the Telephone exchange I worked in we had a 2MB HD that had a three foot diameter platter, and now look at Seagate's low end Ironwolf range of RAID capable disk drives. If I had the funds, I would try a 8 or 16 disk RAID 10 of 16TB drives just for the enjoyment of trying to fill up that space !
https://www.seagate.com/au/en/internal-hard-drives/hdd/ironwolf/
16 TB, 14 TB, 12 TB, 10 TB, 8 TB, 6 TB, 4 TB, 3 TB, 2 TB, 1 TB

BTY: I tend to use 2 and 3TB drives, no RAID, data being rsynced across multiple drives. I have found rebuilding RAIDs is so time consuming.

> 
> On top of this, we’re now got at least two levels of SSD:
>  - NVMe & SSD
> 
> and three or more types of HDD:
> 
>  - ‘fast’ and high duty cycle - ‘Enterprise’ disks - expensive and smaller
>  - slow, large, ’nearline’ - with med-high daily data transfer - things you can build a NAS from
>  - removable (USB) drives - cheap, low duty cycle, slow in every way.
> 
> There are candidates for ‘cold’ storage appearing now.

NVMe is a good step forward from previous SSD and HD for speed, however I was more looking to use partitioning for improving  robustness and security. That said, I often use SSD for root and HD for /home and swap. While SSD is faster for swap than HD, it can shorten the life of SSDs. I feel "more comfortable" using HD than SSD for /home simply because I don't trust that SSD drive will not fail sooner than a HD. I have an a number of SSDs fail, but I think in over 30 years, only one HD (if you read my previous emails asking about RAID, you might realise I have little RAID experience).  Should I be more trusting of today's consumer SSDs?

If I could make /root, /usr and /opt read only (except for when applying updates), then I would like to put them on a SSD. I have yet to learn how read-only is achieved, I read it requires several links from root to writable partitions?

> 
> ===========
> 
> I’m not sure what problem you’re trying to solve or what ‘Problem Space’ you’re trying to operate in.

Apologies, I could have made that clearer. This email was about "partitioning".

I am wanting to understand the benefits and, how, when and where to use 'partitioning' to improve performance, robustness, and security, for both linux servers and linux personal computers, particularly Debian installations. 

My Linux partitioning experiences have mainly been with small business and personal computers (i.e PCs).

> 
> ===========
> 
> If you have a ’need for speed’ on any type of equipment, Flash beats Rotating ‘Rust’ by such a large margin it’s ridiculous.
> Using tricks like “Short Stroking” and 3x or higher RAID-1 improves the access times of HDD’s - but now they even struggle against Flash doing what they do best - streaming data.
> 
> Notably, that Radio Astronomy data collection effort that yielded the computed image of a Gravitational Lens (IIRC - pls correct if wrong)
> they used removable RAID-5 ‘packs’ of Helium-filled 8TB or 10TB drives (sealed, worked at Altitude).
> It cost them a bomb buying the disks - if money was no object, they’d have had better results with SSD’s - at many times the price.
> [IIRC, they bought 1,000’s of HDD - large numbers]
> 
> If this is a home machine, for everyday Desktop use, you don’t even need a ‘rescue’ partition because a USB drive or Optical Disk lets you do that. But having "/boot” in a separate partition is still the preference for GRUB (with UEFI, don’t need it)
> 
> If this is a home server or experimental machine - some sensible dimensioning of DRAM, NVMe, SSD, HDD’s and USB drives will provide different speed/ cost tradeoffs suitable for your project and budget.
> 
> In that environment, you’d want to separate different bootable environments, have shareable partitions for common data and/or DB’s and possibly dedicate some space to VM images - especially sharing an SSD.
> 
> If this is a large SME server - needs careful planning and capacity forecasting on all sorts of components & subsystems.
> 
> But if you’re Amazon or Google, you don’t need “partitions" (ex-Googlers might wade in at this point)
> because you’re doing things very differently.
> 
> We’ve a whole range of people on the list who do ‘interesting’ jobs within computing.
> 
> If you were to ask some targeted questions of them, I’d expect you’d get useful and helpful responses.
> 
> "Simple is Good” is an Operations & Admin maxim - but only “Simple Enough, not too simple” - if you can avoid multiple partitions, that’s a big operational and recovery/ rebuild/ upgrade benefit.
> 
> From the Radio Astronomers data collection system:
> 	their “simplest” solution was a wee bit complicated and intricate for daily use in most cases.
> 
> HTH
> steve
> 
> > On 16 Oct 2019, at 12:15, George at Clug via linux <linux at lists.samba.org> wrote:
> > 
> > Partitioning for fun and benefit
> > 
> > Hi,
> > 
> > I would like to discuss and get your opinions about using multiple partitions when installing Linux.
> > 
> > I am not sure if it is even worth doing, though lots of security web pages advocate the practice (in various ways).
> > 
> > A site that had a good explanation of why ...
> > http://www.daniloaz.com/en/the-importance-of-properly-partitioning-a-disk-in-linux/
> > "I am a strong supporter of simplicity and the principle that less is more, but as far as security and performance of information systems is concerned, we must be able to strike a balance between keeping things simple and exposing ourselves as little as possible to potential threats while trying to obtain the maximum performance of all system elements involved."
> > 
> > The site that I found most useful... 
> > http://www.softpanorama.org/Commercial_linuxes/Security/protective_partitioning_of_the_system.shtml
> > "Note: Questionable recommendations. should be taken with the grain of salt."
> > 
> > I read that security that is controlled and/or circumvented by fstab/remounting is not that much security, but I guess any "security hardening" that does not stop your computer/server doing its job is a "good thing"?
> > 
> > For performance I would like to have root on a [consumer] SSD as read only (and any other partitions that I could), but I don't know how to do this, and it also creates more work for applying updates. The security benefit to effort might not be worth the effort. (yet many more hours of research, I guess)
> > 
> > Talking about "that does not stop your computer/server doing its job", well many web security sites suggests adding noexec to your /var partition, but I found that this broke Debian's package installation. I could run with this enabled, then remove it when applying updates, but that sounds like too much hard work.
> > 
> > Well whatever your personal thoughts are, you are welcome to comment on the results of my testing or question my sanity.
> > 
> > Notes:
> > 1) "/var/lib/mysql", "/var/www" are only for testing, and implementation depends on what your servers uses.
> > 2) I have tested upgrading from Debian Jessie, through to Buster, and installing qemu-kem, then Mate GUI and Virt-Manager. In practice, I usually make "/var/lib/libvirt/images" a separate partition/drive.
> > 3) Testing so far has been very limited, no postfix, dovecot/Courier IMAP, no chroot testing, no web site CGI/PHP, etc.
> > 4) No testing with docker or lcx.
> > 5) I was considering using a 8TB HD, I experimented with the below partition sizes which seem to suite the needs of my servers.
> > 6) After many configuration changes, and much retesting of partitioning, I may have got confused on some things.
> > 7) I discovered "Linux limits the partitions per drive to 15 partitions..."  - https://www.debian.org/releases/potato/i386/ch-partitioning.en.html
> > 8) Time might not be a finite resource, but it is a limited resource.
> > 
> > 50GB  /               ext4    noatime,errors=remount-ro
> > 1GB  /boot           ext4    noatime,nodev,nosuid,noexec        
> > 50GB  /home           ext4    noatime,nodev,nosuid,noexec 
> > 2GB  /opt            ext4    noatime,nodev,nosuid       
> > 2GB  /srv            ext4    noatime,nodev,nosuid,noexec  (I do not have Service Data to test with)    
> > 10GB  /tmp            ext4    relatime,nodev,nosuid        (I think noatime broke something, but relatime worked)
> > 20GB  /usr            ext4    noatime,nodev                (I increased /usr from inital 10GB to 20GB)
> > 2GB  /usr/local      ext4    noatime,nodev
> > 10GB  /var            ext4    noatime,nodev,nosuid         (Will nosuid break chrooted email systems ?) 
> > 2GB  /var/lib/mysql  ext4    noatime,nodev,nosuid,noexec  (Example only. Size of production databases?)
> > 2GB  /var/log        ext4    noatime,nodev,nosuid,noexec
> > 2GB  /var/log/audit  ext4    noatime,nodev,nosuid,noexec
> > 2GB  /var/tmp        ext4    relatime,nodev,nosuid
> > 2GB  /var/www        ext4    noatime,nodev,nosuid,noexec  (Example only. Size of production web files?)
> > 
> > 
> > ---------------------------------------------------------
> > 
> > # df -h
> > Filesystem      Size  Used Avail Use% Mounted on
> > udev            4.9G     0  4.9G   0% /dev
> > tmpfs          1000M  9.1M  991M   1% /run
> > /dev/vda2        46G  589M   43G   2% /
> > /dev/vda6       9.1G  4.0G  4.7G  47% /usr
> > tmpfs           4.9G     0  4.9G   0% /dev/shm
> > tmpfs           5.0M  4.0K  5.0M   1% /run/lock
> > tmpfs           4.9G     0  4.9G   0% /sys/fs/cgroup
> > /dev/vda8       9.1G   22M  8.6G   1% /tmp
> > /dev/vda4       1.9G  2.9M  1.7G   1% /opt
> > /dev/vda5       1.9G  2.9M  1.7G   1% /srv
> > /dev/vda9       9.1G  694M  7.9G   8% /var
> > /dev/vda10      1.9G  2.9M  1.7G   1% /var/lib/mysql
> > /dev/vda11      1.9G   37M  1.7G   3% /var/log
> > /dev/vda13      1.9G  2.9M  1.7G   1% /var/tmp
> > /dev/vda7       1.9G  2.9M  1.7G   1% /usr/local
> > /dev/vda15       46G  4.3G   40G  10% /home
> > /dev/vda1       922M   81M  778M  10% /boot
> > /dev/vda12      1.9G  2.9M  1.7G   1% /var/log/audit
> > /dev/vda14      1.9G  2.9M  1.7G   1% /var/www
> > tmpfs          1000M   20K 1000M   1% /run/user/1000
> > /dev/sr0        3.8G  3.8G     0 100% /media/cdrom0
> > ---------------------------------------------------------
> > # mount | grep "/dev/v"
> > /dev/vda2 on / type ext4 (rw,noatime,errors=remount-ro)
> > /dev/vda6 on /usr type ext4 (rw,nodev,noatime)
> > /dev/vda1 on /boot type ext4 (rw,nosuid,nodev,noexec,noatime)
> > /dev/vda15 on /home type ext4 (rw,nosuid,nodev,noexec,noatime)
> > /dev/vda9 on /var type ext4 (rw,nosuid,nodev,noatime)
> > /dev/vda14 on /var/www type ext4 (rw,nosuid,nodev,noexec,noatime)
> > /dev/vda7 on /usr/local type ext4 (rw,nodev,noatime)
> > /dev/vda8 on /tmp type ext4 (rw,nosuid,nodev,relatime)
> > /dev/vda5 on /srv type ext4 (rw,nosuid,nodev,noexec,noatime)
> > /dev/vda10 on /var/lib/mysql type ext4 (rw,nosuid,nodev,noexec,noatime)
> > /dev/vda11 on /var/log type ext4 (rw,nosuid,nodev,noexec,noatime)
> > /dev/vda13 on /var/tmp type ext4 (rw,nosuid,nodev,relatime)
> > /dev/vda12 on /var/log/audit type ext4 (rw,nosuid,nodev,noexec,noatime)
> > /dev/vda4 on /opt type ext4 (rw,nosuid,nodev,noatime)
> > ---------------------------------------------------------
> > 
> > 
> > 
> > Sites that I found useful... 
> > http://www.softpanorama.org/Commercial_linuxes/Security/protective_partitioning_of_the_system.shtml
> > "Note: Questionable recommendations. should be taken with the grain of salt."
> > 
> > https://www.cyberciti.biz/tips/linux-security.html
> > "21. Separate Disk Partitions For Linux System
> > Separation of the operating system files from user files may result into a better and secure system."
> > 
> > https://access.redhat.com/discussions/641923
> > "Our practice is to isolate /var/tmp & /var/log to independent LVs.
> > In our environment, we make separate logical volumes for /tmp/ /var/ /var/log and /var/log/audit .
> > We place /var/log/audit in it's own logical volume especially on tomcat or web servers.
> > Ideally you should be mounting /var with noexec and nosuid flags."
> > 
> > https://www.tecmint.com/linux-server-hardening-security-tips/
> > "2. Disk Partitions
> > Make sure you must have following separate partitions and sure that third party applications should be installed on separate file systems under /opt."
> > 
> > https://wiki.archlinux.org/index.php/Fstab#Field_definitions
> > "Note: noatime implies nodiratime. You do not need to specify both."
> > 
> > https://www.cloudberrylab.com/resources/blog/linux-server-hardening-guide/
> > "Partitioning"
> > 
> > https://buildmedia.readthedocs.org/media/pdf/simp/latest/simp.pdf
> > "Disk Partitioning"
> > 
> > https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/6/html/installation_guide/s2-diskpartrecommend-x86
> > https://docs.fedoraproject.org/en-US/Fedora/13/html/Installation_Guide/s2-diskpartrecommend-x86.html
> > 
> > https://tomthorp.me/blog/moving-docker-data-directory
> > 
> > https://www.thegeekstuff.com/2010/09/linux-file-system-structure/
> > 
> > https://access.redhat.com/discussions/641923
> > "There is a commonly held wisdom that /var should by default be seperated from the root partition (for example https://access.redhat.com/site/articles/10332)."
> > 
> > https://www.cyberciti.biz/tips/linux-security.html
> > "21. Separate Disk Partitions For Linux System"
> > 
> > https://wiki.debian.org/ReadonlyRoot
> > 
> > 
> > 
> > 
> > 
> > -- 
> > linux mailing list
> > linux at lists.samba.org
> > https://lists.samba.org/mailman/listinfo/linux
> 
> --
> Steve Jenkin, IT Systems and Design 
> 0412 786 915 (+61 412 786 915)
> PO Box 38, Kippax ACT 2615, AUSTRALIA
> 
> mailto:sjenkin at canb.auug.org.au http://members.tip.net.au/~sjenkin
> 
> 