[Samba] 1MB/s gigabit transfers on dell poweredge

[John H Terpstra - Samba Team]

godwin at acp-online.co.uk wrote:
> Ok I am dealing with two John's here :)
> 
> John Terpstra,
> 
> Thanks for your reply.
> 
> whoa ... 90 Mbytes/sec.... I would give an arm to get that kind of
> throughput. Right now I am getting a little over 1% of that throughput :-D
> 
> Can you tell me what nic's, switch and cabling are you using in your setup?

I gave you the NICs in the cTDB cluster (shown below).

My home network has the following NICs:

datastore:
01:00.0 Ethernet controller: Attansic Technology Corp. L1 Gigabit
Ethernet Adapter (rev b0)

Windows client:
Realtec RTL8168/8111 PCI-Express Gigabit Ethernet NIC (Built into ASUS
M2A-VM mobo)

Linux client:
02:00.0 Ethernet controller: Attansic Technology Corp. L1 Gigabit
Ethernet Adapter (rev b0)  (Built into ASUS P5QC mobo)

Cat5 cabling - home-grown.

Switch:
D-Link 1 Gigabit Model DGS-2208


> I am using lustre as the backend to samba CTDB. I dont know how much you
> know about lustre so I wont bore you with a sermon :-). (I am not a lustre
> expert myself - not yet ;-) )

I'm always ready to listen to an expert talk about something that
excites him/her.

> But my initial tests of lustre do give promising results. Using a native
> lustre client on linux, I can get high throughputs. The only limitation
> being the network interconnect on the client side. Using a 4 port PCI-E
> Intel EEPRO's bonded togeather should give me 30x4=120MB/s (in a worst
> case scenario considering inferior eqpt)

Using the bonded Intel quad-port NIC (Intel Corporation 82571EB Gigabit
Ethernet Controller) over two bonded ports I've seen rates of 90
MBytes/sec aggregate in each direction.  That is doing DRBD sync of two
block volumes each way.  i.e.:  2 x 4TB from machine A to B, and 2 x 4TB
from machine B to A - both running at the same time.

But I would not expect to see that high a throughput over a Samba
cluster.  Your mileage might vary!

> As I mentioned in my last post, The issue is whether samba ctdb can scale
> to that bandwidth. Right now on a standard samba install on debian etch I
> am trying to get more than 1.4 MB/s (NFS etc works fine at 50MB/s).

This is not a Samba problem.  You have a hardware issue of some sort.

> Its only when I resolve this can I look at ctdb with lustre.

I'd like to be kept updated on your progress with this.

> If I had the luxury of using native lustre client, I am sure of meeting or
> exceeding the 100MB/s objective. Unfortunately Mac's dont have a native
> lustre client yet. Of course the whole system will still have to be
> carefully handbuilt and tuned with each component whether its the
> nic,Motherboard,switch, cabling etc; all chosen to work optimally with
> each other.
> 
> I guess my current speed issue is related to the samba version. 

Don't guess - prove it. :-)  There have been way too many such
suspicions on this list.  It's time this got put to bed.

> I am using 3.0.24 that comes standard with debian etch. I need to upgrade. 
> I will be testing out various updated versions of Samba today.
> 
> I will keep the list posted with results of my samba-ctdb, lustre trials
> as and when they are conducted as well as my current samba speed issue.

Please email me (off-list) your smb.conf configuration, and your CTDB
config files.  I'd like to compare notes.

Cheers,
John T.

> Thanks,
> Godwin Monis
> 
> 
>> godwin at acp-online.co.uk wrote:
>>> John, thanks once again for the quick reply.
>>>
>> ... [snip]...
>>
>>> I am eager to understand how you are getting 50MB/s on samba transfers
>>> as
>>> considering the overheads added by the samba protocol, you should be
>>> getting 60-75 MB/s using scp/rsync/nfs.
>> My home network running Samba-3.3.1 over 1 Gigabit ethernet transfers up
>> to 90 Mbytes/sec.  The file system is ext3 on a Multidisk RAID5 array
>> that consists of 4x1TB Seagate SATA drives.
>>
>> When transferring lots of small files the rate drops dramatically.  When
>> I run rsync over the same link between the same systems the transfer
>> rate for large files hovers around 55 Mbytes/sec and drops to as low as
>> 1.5 Mbytes/sec when it hits directories with lots of small files (<100
>> Kbytes).
>>
>>> I am also working on a COTS storage solution using lustre and Samba
>>> CTDB.
>>> The aim is to provide 1000MB/s to clients (100MB/s to each client) so
>>> they
>> Have been working on two Samba-cTDB cluster installations. One of these
>> is based on RHEL and has Samba-cTDB on a front-end cluster that sits on
>> top of RHCS, over a GFS2 file system, over LVM, over iSCSI.  The
>> back-end consists of two systems that each have 32TB of data that is
>> mirrored using DRBD.  The DRBD nodes are exported as iSCSI targets.
>>
>> So far with 2 active front-end nodes (each has 8 CPU cores) and running
>> the NetBench workload using smbtorture, the highest peak I/O I have seen
>> is 58 MBytes/sec.  The iSCSI framework is using bonded multiple 1
>> Gigabit ethernet adaptors and the cluster front-end also uses multiple 1
>> Gigabit ethernet.
>>
>> I would love to find a way to get some more speed out of the cluster,
>> and hence if you can meet your 100 Mbtes/sec objective I'd love to know
>> how you did that!
>>
>> PS: Using Samba-3.3.1 with CTDB 1.0.70.
>>
>>> can edit video online. The solution also needs to be scalable in terms
>>> of
>>> IO and storage capacity and built out of open source components and COTS
>>> so there is no vendor lock in. Initial tests on lustre using standard
>>> dell
>>> desktop hw are very good. However I need samba ctdb to communicate with
>> The moment you introduce global file locking I believe you will see a
>> sharp decline in throughput.  Clusters are good for availability and
>> reliability, but throughput is a bit elusive.
>>
>>> the clients as they are Apple macs. I havent reached samba ctdb
>>> configuration yet. But the gigabit ethernet issue had me scared till I
>>> received your reply. Now I see a lot of hope ;-).
>>>
>>> Once again thanks for your help and do let me know if I can reciprocate
>>> your kindness.
>>>
>>> Cheers,
>>> Godwin Monis
>>>
>>>
>>>>> Now, if its not asking for too much, can you let me know
>>>>> 1. the network chipsets used on your server and client
>>>>>
>>>> Main servers
>>>> fileserv ~ # lspci | grep Giga
>>>> 02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
>>>> Gigabit Ethernet (rev 03)
>>>> 02:09.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
>>>> Gigabit Ethernet (rev 03)
>> NICs on the cluster servers I am working with are:
>>
>> nVidia Corporation MCP55 Ethernet - dual ports on mobos
>> Intel Corporation 82571EB Gigabit Ethernet Controller - quad port
>>
>>>> dev6 ~ # lspci | grep Giga
>>>> 02:09.0 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
>>>> Gigabit Ethernet (rev 03)
>>>> 02:09.1 Ethernet controller: Broadcom Corporation NetXtreme BCM5704
>>>> Gigabit Ethernet (rev 03)
>>>>
>>>> datastore0 ~ # lspci | grep net
>>>> 00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2)
>>>> 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2)
>>>>
>>>> datastore1 ~ # lspci | grep net
>>>> 00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2)
>>>> 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2)
>>>>
>>>> datastore2 ~ # lspci | grep net
>>>> 00:08.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2)
>>>> 00:09.0 Bridge: nVidia Corporation MCP55 Ethernet (rev a2)
>> I hope this is useful info for you. Let me know if I can assist you in
>> any way.
>>
>> Cheers,
>> John T.
>>
>>
> 
>