SMB direct or NFS

Myth

Member
Feb 27, 2018
148
7
18
Los Angeles
Wazzz up guys!!

So I have a linux server, an OPEN Drive to be exact and a iMac Pro with a 40GigE to thunderbolt 3 adapter.

The Linux server is connected via 4x 40GigE lag (LACP) to an arista switch. And the iMac is connected to the switch via a single 40GigE link.

So I'm probably gonna turn on jumbo frames for this setup to allow maximum performance. The reason why we need so much bandwidth is because they want to edit video RAW 5k 2 streams per system.

I don't know what flow control settings to use, I'll try to ask the vendor OpenDrive and see what they recommend. But a lot of these smaller server companies don't really know the tweaks of their network settings that well.

Then of course they are use either NFS or SMB to allow the iMac Pro's to connect to the storage. So it's a single 40GigE link. I don't know which protocol to use. I think the storage vendor may default to NFS, but I want to have a backup plan in-case this setup doesn't provide enough bandwidth.

At my old job everything was Windows. And if I was using SMB 3.0 direct I could have more than one network connection to the switch and windows would then utilize the extra link, as long as it was in the same sub-net. So if I was doing a data copy between two windows machines, and let's say the windows server was a 40GigE to the switch and the client windows machine was 2x 1GigE to the same switch, and each interface has it's own IP address in the same subnet, then SMB would spread out the data transfer, via an explorer copy I could get 2GigE throughput.

The editing system we are using is going to be Black Magic Resolve, so I don't know if they have the same SMB technology to spread the data transfer out. It's my understanding that this SMB direct is a windows thing.

So I'm thinking, Since we are using a Linux Server and an iMac client via 40GigE links, what protocol would be best? The OpenDrive seems to be a really fast NAS, since it uses NFS and SMB as it's network protocol. You guys have any ideas? Also, any experience with jumbo and flow control?
 

RageBone

Active Member
Jul 11, 2017
298
78
28
SMB spreading over multiple Nics is called SMB Multichannel and it is available in samba for a few versions already i think.
You need to also enable it on the server side though.

SMB with RDMA is a mystery to me currently. I have the feeling that it is also available in samba, but not openly stated as "RDMA or Direct" but under the same feature flags as multi channel with two additional settings for the amount of ques or so.
The only tools i know to diagnose if you have a working SMB Direct connection are only available in windows and in my experience it is very picky about what Nics you have and if they are capable of it. Had cx2s first and then reflashed hp cx3s and meh.

No clue about Mac OS or RDMA or NFS on it in that regard. The 40GbE Nics that i have seen for Mac were Mellanox Cx3s i think, so that could work if the driver is up to the task.
 

Myth

Member
Feb 27, 2018
148
7
18
Los Angeles
I've heard on the Linux Server side they are using SMB 3.1.1 and are now recommending SMB. I wonder if I added another 40GigE link to the client and set it to the same subnet, if the data transfer would "multi-thread" across 2x 40GigE ports. I know that making them a LAG would not improve throughput. And I'm not sure how much throughput I will get with Jumbo Frames. I think with standard frames we got around 750MBps on data trasfers.

On a windows box, if I turned on jumbo frames I would get around 1250MBps data transfer. But I've learned that jumbo frames do not have that much of an impact on Linux boxes as much as they do on Windows Boxes. Anyone have any experience with jumbo frames on Linux based smb servers?
 

RageBone

Active Member
Jul 11, 2017
298
78
28
SMB3_11 is the newest SMB version that i know off, i think its there since windows 10.
You have to enable multi channel on the samba server first before it can do that.
 

Myth

Member
Feb 27, 2018
148
7
18
Los Angeles
So It looks like I can enable SMB multichannel on the Linux server with a simple configuration line in the smb.conf file.

However I'm having a hard time finding out if Mac supports multichannel SMB. I know a windows client would. I guess I have to test it... ugh.