Which nic or switch for Windows Server and SMB direct/RDMA

[gea]

More than 10 years ago a said goodbye to Windows SMB for ZFS and Solaris SMB due a better stability and similar ACL options.

Now I am back to try Windows (free 2019 and newer ones), mainly because of SMB direct/RDMA with an absolute superiour performance, lowest latency and cpu load. ACL management based on fine granular ntfs permissions with inheritance in the filesystem and worldwide unique AD security references based on Windows SID without stupid SAMBA id mappings is another reason for me for Windows.

Which nics can you recommend (20G or faster, SMB direct/RDMA capable) to connect a Windows Server with Windows 10 or 11?
(with good driver stability and affordable prices)

Any remarks about a switch and connectivity
(nic-nic or over a switch)

 

[pimposh]

(with good driver stability and affordable prices)

Mellanox CX-4 seems to be in ~65Eur range which gives nice bang for buck ratio, aspm support, no transciever locks, wide availability, stable both Windows and Linux drivers.

 

[TRACKER]

I also would recommend Mellanox CX-4 series.

 

[gea]

Do you use nic-nic connectivity?
Are there affordable switches up from 25 Gb?

 

[TRACKER]

I use only nic-switch connections on my Mikrotik CRS504-4XQ, not exactly affordable but for brand new 100G switch - not bad at all.

 

[DavidWJohnston]

I mentioned RDMA in another recent thread: https://forums.servethehome.com/ind...etup-os-for-smb-share-only.45456/#post-440633

I run Mellanox CX4 50G and 100G, and a Celestica DX010 100G switch. With RoCE SMB-Direct I get around 4.5 GB/sec in a single file copy.

The Mellanox cards are affordable used, and solid with good drivers. The DX010 is a good homelab tinkering switch, but not best for production work. 100G switches generally use a lot of power, if you can get away with a copper DAC cable, that is the most power efficient.

 

[gea]

Summary from another discussion:

Mellanox CX4
Care about model numbers for (IB and) Ethernet support. We need Ethernet for SMB direct and drivers for Windows Server and Win10/11.
Crossflash to Mellanox firmware for other brands possible (if something does not work)

NVIDIA ConnectX-4 InfiniBand/Ethernet Adapter Cards User Manual

docs.nvidia.com

https://www.servethehome.com/mellan...5-and-connectx-6-ethernet-comparison-chart-1/

Mellanox CX5
more features, always IB and Ethernet support for original Mellanox (with firmware tool to switch mode)
Crossflash for other brands with Mellanox firmware often not possible (does not matter if it already works for Ethernet)

Performance
25-50G (3-6 GB/s) are fast enough for server - nic SMB direct connects without a switch, even for 4/8k video (much cheaper than 100G),
A server mainboard with 7 pci slots allows up to 5 nics (10 SMB direct clients ex Win11) + 2 remaining slots for SAS HBA/ NVMe/video adapter
Ip setup a little more complicated with several nics in the server but this is a one time setup
Care about a fast enough system (CPU, RAM, Pool layout ex multi mirrors of SAS hd/ssd or NVMe) for your use case

OS
SMB direct/RDMA needs a Windows Server as SMB server (up from a free 2019). For clients use Windows 11

Question: DAC cables (this is where I am unsure)?
There are copper cables up to 5m and active fiber cables up to 100m for easy nic-nic or nic switch connectivity
Anyone tried an active cable for 25G-100G with Ethernet like

https://www.fs.com/de-en/products/154830.html?attribute=37257&id=1744719

Anything forgotten?

 

[i386]

SMB direct/RDMA needs a Windows Server as SMB server (up from a free 2019).

Or a linux distro with kernel >6.6, currently only ubuntu 24.04 is a stable release with such a kernel version.

Question: DAC cables?
There are copper cables up to 5m and active fiber cables up to 100m for easy nic-nic or nic switch connectivity
Anyone tried an active cable for 25G-100G like
https://www.fs.com/de-en/products/154830.html?attribute=37257&id=1744719

what exactly is your question?
if you should use dac, aoc (active optical cable) or optical transceivers? this depends on the the distance you need to cover and budget. dac is cheap and covers short distances, aoc are more epensive and cover longer distances (up to 100m), optical transceivers are the most exepnsive option but can cover 100m to 80km+

 

[pimposh]

These MikroTik branded DACs - i use couple of them and so far had no issues with any of switches nor NICs.
Two versions are available- XS+DA0001 (1m) and XS+DA0003 (3m). Of course they might be not long enough for your needs.

 

[gea]

Or a linux distro with kernel >6.6, currently only ubuntu 24.04 is a stable release with such a kernel version.

what exactly is your question?
if you should use dac, aoc (active optical cable) or optical transceivers? this depends on the the distance you need to cover and budget. dac is cheap and covers short distances, aoc are more epensive and cover longer distances (up to 100m), optical transceivers are the most exepnsive option but can cover 100m to 80km+

Is SAMBA now capable or stable with SMB direct/RDMA like a Windows Server is for many years?
Do these aoc cables work well in the range 5m to 100m with nic to nic Mellanox CX4/5 and 25-100G to build a high performance workgroup for 4k/8k video editing?

 

[Gilbert]

Mellanox CX-4 seems to be in ~65Eur range which gives nice bang for buck ratio, aspm support, no transciever locks, wide availability, stable both Windows and Linux drivers.

I have both CX-4lx and t580 iwarp, in your opinion, which has better performance and easiest to implement in both Windows Server 2022 and Linux?

 

[gea]

It can be that Windows 11 Pro for Workstations is capable to act as SMB Direct server so you do not need a Windows Server OS for small workgroup with a high performance NAS but this requires a confirmation as some sources claims that Windows Server is mandatory.

 

[Gilbert]

It seems that Windows 11 Pro for Workstations is capable to act as SMB Direct server so you do not need a Windows Server OS for small workgroup with a high performance NAS.

And which flavor of Linux is the most compatible with SMB Direct, as I have both linux and windows guest(Proxmox) and a separate Server 2025 node as well as a synology for backup.

 

[gea]

I have not seen any report of a stable SMB direct implementation on Linux + SAMBA.
You are propably limited to SMB multichannel on Linux clients, not as superiour as RDMA but not so bad.

SMB direct at the moment seems to be a solution between Windows machines.
Correct me if you have other informations.

 

[Gilbert]

I have not seen any report of a stable SMB direct implementation on Linux + SAMBA.
You are propably limited to SMB multichannel on Linux clients, not as superiour as RDMA but not so bad.

SMB direct at the moment seems to be a solution between Windows machines.
Correct me if you have other informations.

From my research into the subject. ksmbd which is a linux version of smb rdma , is supposed to be compatible with SMB Direct, which has been taken off the experimental list and is now labeled stable, and I believe is part of the kernels on anything higher than 6.5 maybe? I have been looking for examples of implementation, as in respect to windows. From what I understand it is only available in the role of smb server. Which means windows would be the client. KSMBD - SMB3 Kernel Server — The Linux Kernel documentation

 

[gea]

ksmbd instead SAMBA seems the path for SMB direct on Linux at least for the server role.
Not sure if it is already a real alternative regarding security, stability and compatibility to SAMBA or a Windows SMB server.

 

[Gilbert]

ksmbd instead SAMBA seems the path for SMB direct on Linux at least for the server role.
Not sure if it is already a real alternative regarding security, stability and compatibility to SAMBA or a Windows SMB server.

I have installed it, and will see what it brings to the table, rather easy actually. On Debian(apt install ksmbd-tools) then follow the instructions for setting up ksmbd.conf and the man instructions. And this is for home use not production, so not really worried about maximum security. Stability we will see, and the link I posted specifically states it supports smb direct, which is windows rdma by definition. It turns off smb multichannel by default in the config file, which the author says will be improved in future updates. My understanding is Samba and Smb direct are two different things. One is a protocol the other is the transport. And windows server supports smb direct and samba/cifs.

 

[gea]

No, SMB is the protocol, SAMBA and ksmbd are two different SMB servers
but ksmbd tries to be compatible with SAMBA settings from smb.conf.
SMB multichannel seems currently not supported in ksmbd (what slows non RDMA clients)

A Windows SMB Server supports SMB multichannel, SMB via ip and SMB direct (SMB via RDMA)
CIFS is an outdated SMB variant

 

[BackupProphet]

Mellanox 4+ support lossy RoCE, so having lossless ethernet is not an absolute requirement, but still nice to have.
Chelsio use iWARP, but it is traffic over TCP, and not like RoCE which uses UDP.
I think Mellanox switches are great, especially the newer generation for 25/100G. Low power usage, and tuning for RoCE is as easy as writing #roce enable

 

[Gilbert]

No, SMB is the protocol, SAMBA and ksmbd are two different SMB servers
but ksmbd tries to be compatible with SAMBA settings from smb.conf.
SMB multichannel seems currently not supported in ksmbd (what slows non RDMA clients)

A Windows SMB Server supports SMB multichannel, SMB via ip and SMB direct (SMB via RDMA)
CIFS is an outdated SMB variant

Thanks for the info. As long as I can get the performance I want, I will be happy. I have the hardware, so we will see.