[clug] Cloud servers, ZFS, SnapRAID+MergerFS (was Re: Topic for Feb meeting)

Tue Feb 21 06:50:13 UTC 2017

Wow. Lots of technical stuff there that has gone over my head. Hence, the ask for help in probably a more simple system. 

It all started when I received a Western Digital Technologies (WDT)  Mycloud, 4Tb hardware. I did manage to set it up but it didn't have a client for Linux, it only backed up media, it wasn't encrypted and it was supposed to send my data to WDT which I definitely don't want. I want my data in my hands. Anyway, I took it back to the shop and got my money back. I thought there must be something like this using Linux. Hence my post. 

John. 

On 21 February 2017 4:41:18 PM AEDT, Tony Lewis <tony at lewistribe.com> wrote:
>Define 'cloud'.  Do you mean compute+storage for home?  I'm at the 
>burn-in test stage for a new home server for just this. Indulge me...
>
>My requirement is for a robust NAS that can grow (lots of disk bays), 
>but could also have enough grunt for the home network, some general 
>web-ish services, and some VMs.
>
>I bought a Dell C6105 cloud server off eBay.  It is actually three 
>servers in one, and I believe is used by cloud providers to use or sell
>
>as cloud services.  In 2RU I got 144GB of RAM, 6 x 6-core Opteron 2419 
>processors, and 12 x 1TB 3.5" SATA drives.
>
>Each separate server has 1/3 of these, so 48GB RAM, 2x CPUs and 4x 
>drives, all x 3.
>
>Power is non-trivial.  There's a single PSU that powers the lot. Zero 
>cards on = 50W of idle power draw.  Turn one processor card on and it 
>takes a total of 190-250W depending on how hard the CPUs are pushed.  
>That's more than I would like, but probably less power than the 
>equivalent Xeon rack servers of their age.  Fully running, it's about 
>400W, but I don't anticipate running more than one server card 
>regularly, unless I need to heat the garage.
>
>Price was compelling.  The nearest I could price a new JBOD chassis
>plus 
>decent RAM, CPU, PSU, motherboard was north of $2500, and then disks on
>
>top of that.  I got the whole server including the disks, delivered
>from 
>the US, for about $1200.  That justifies extra power consumption for
>me.
>
>I will use Proxmox for virtualisation, giving me LXC and KVM.
>
>I was all set for using ZFS, either through a virtualised FreeNAS 
>instance, or natively on Debian, but am now seriously considering 
>SnapRAID + MergerFS.  See here: 
>https://www.linuxserver.io/2016/02/02/the-perfect-media-server-2016/ . 
>
>For home use, this seems like a good compromise.  For me this is where 
>it gets interesting.
>
>ZFS is enterprise-grade, but does not grow easily.  You can't add a 
>single drive and have it restripe like you can with MDADM.  To grow
>your 
>ZFS pool once you outsize it, you have two options: add a whole new
>vdev 
>(another bunch of disks, requiring their own bays etc), or you replace 
>each disk one by one, and once you have replaced them all, by ZFS magic
>
>you get the extra disk space.  I was looking at 8 disks in my RAID 
>array, and so when the time came to upgrade, I would have to buy a new 
>batch of 8 disks.  No point in anything less.
>
>But SnapRAID and MergerFS allow me to use mismatched drives, and to add
>
>and remove disks at will (with some caveats and careful handling).  The
>
>main downside I see is that when you restore, you lose file ownership 
>and permissions.  This is a grievous oversight IMO, but I should be
>able 
>to work around it by only really storing media files on there, where 
>permissions are less of an issue.
>
>I'll do some testing with SnapRAID+MergerFS and get a feel for it.
>
>Tony
>
>On 21-Feb-17 3:37 PM, jhock at iinet.net.au wrote:
>> Just for self interest, if anyone wanted to show me how to set up an
>open source cloud system, on a Linux system like Ubuntu, that allowed
>me to backup my entire file storage on my Android phone, wife's PC and
>my Xubuntu netbook, via my router's wifi, I would definitely like that
>to be a CLUG topic. But that's my self interest. :--)
>>
>> Oh, and the cloud could be a Raspberry Pi with an attached 2Tb hard
>disk plugged into the router to reduce the power supply, would also be
>perfect. :--)
>>
>> John.
>>
>> On 21 February 2017 3:17:53 PM AEDT, Steven Hanley <sjh at svana.org>
>wrote:
>>> On Sun, Feb 05, 2017 at 08:51:52AM +1100, Stephen Hocking wrote:
>>>> This part of the group would be, which reminds me...
>>>
>>> An oh look it is CLUG week, if you have a talk that does sound like
>a
>>> pretty
>>> cool topic for everyone to learn about.
>>>
>>> 	See You
>>> 	    Steve
>>>
>>>> On 5 February 2017 at 00:53, Brenton Ross <rossb at fwi.net.au> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> I have just finished putting together a presentation on testing
>for
>>> my
>>>>> VICI project.
>>>>>
>>>>> It covers a testing library for unit tests, asynchronous testing
>>> for
>>>>> threads and child processes, testing of modules and programs and
>>>>> simulating user interactions with GUI programs.
>>>>>
>>>>> Would the group be interested ?
>>>>>
>>>>> Brenton
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> linux mailing list
>>>>> linux at lists.samba.org
>>>>> https://lists.samba.org/mailman/listinfo/linux
>>>>>
>>>>
>>>>
>>>> -- 
>>>>
>>>>    "I and the public know
>>>>    what all schoolchildren learn
>>>>    Those to whom evil is done
>>>>    Do evil in return"		W.H. Auden, "September 1, 1939"
>>>> -- 
>>>> linux mailing list
>>>> linux at lists.samba.org
>>>> https://lists.samba.org/mailman/listinfo/linux
>>> -- 
>>> Steven Hanley sjh at svana.org http://svana.org/sjh/diary
>>> i can see her features begin to blur, as she pours herself
>>> into the mold he made for her, and for everything he does
>>> she has a way to rationalize
>>>    Fixing Her Hair - Imperfectly - Ani
>>>
>>> -- 
>>> linux mailing list
>>> linux at lists.samba.org
>>> https://lists.samba.org/mailman/listinfo/linux
>
>
>-- 
>linux mailing list
>linux at lists.samba.org
>https://lists.samba.org/mailman/listinfo/linux