Which nic or switch for Windows Server and SMB direct/RDMA

[gea]

More than 10 years ago a said goodbye to Windows SMB for ZFS and Solaris SMB due a better stability and similar ACL options.

Now I am back to try Windows (free 2019 and newer ones), mainly because of SMB direct/RDMA with an absolute superiour performance, lowest latency and cpu load. ACL management based on fine granular ntfs permissions with inheritance in the filesystem and worldwide unique AD security references based on Windows SID without stupid SAMBA id mappings is another reason for me for Windows.

Which nics can you recommend (20G or faster, SMB direct/RDMA capable) to connect a Windows Server with Windows 10 or 11?
(with good driver stability and affordable prices)

Any remarks about a switch and connectivity
(nic-nic or over a switch)

 

[pimposh]

(with good driver stability and affordable prices)

Mellanox CX-4 seems to be in ~65Eur range which gives nice bang for buck ratio, aspm support, no transciever locks, wide availability, stable both Windows and Linux drivers.

 

[TRACKER]

I also would recommend Mellanox CX-4 series.

 

[gea]

Do you use nic-nic connectivity?
Are there affordable switches up from 25 Gb?

 

[TRACKER]

I use only nic-switch connections on my Mikrotik CRS504-4XQ, not exactly affordable but for brand new 100G switch - not bad at all.

 

[DavidWJohnston]

I mentioned RDMA in another recent thread: https://forums.servethehome.com/ind...etup-os-for-smb-share-only.45456/#post-440633

I run Mellanox CX4 50G and 100G, and a Celestica DX010 100G switch. With RoCE SMB-Direct I get around 4.5 GB/sec in a single file copy.

The Mellanox cards are affordable used, and solid with good drivers. The DX010 is a good homelab tinkering switch, but not best for production work. 100G switches generally use a lot of power, if you can get away with a copper DAC cable, that is the most power efficient.

 

[gea]

Summary from another discussion:

Mellanox CX4
Care about model numbers for (IB and) Ethernet support. We need Ethernet for SMB direct and drivers for Windows Server and Win10/11.
Crossflash to Mellanox firmware for other brands possible (if something does not work)

NVIDIA ConnectX-4 InfiniBand/Ethernet Adapter Cards User Manual

docs.nvidia.com

https://www.servethehome.com/mellan...5-and-connectx-6-ethernet-comparison-chart-1/

Mellanox CX5
more features, always IB and Ethernet support for original Mellanox (with firmware tool to switch mode)
Crossflash for other brands with Mellanox firmware often not possible (does not matter if it already works for Ethernet)

Performance
25-50G (3-6 GB/s) are fast enough for server - nic SMB direct connects without a switch, even for 4/8k video (much cheaper than 100G),
A server mainboard with 7 pci slots allows up to 5 nics (10 SMB direct clients ex Win11) + 2 remaining slots for SAS HBA/ NVMe/video adapter
Ip setup a little more complicated with several nics in the server but this is a one time setup
Care about a fast enough system (CPU, RAM, Pool layout ex multi mirrors of SAS hd/ssd or NVMe) for your use case

OS
SMB direct/RDMA needs a Windows Server as SMB server (up from a free 2019). For clients use Windows 11

Question: DAC cables (this is where I am unsure)?
There are copper cables up to 5m and active fiber cables up to 100m for easy nic-nic or nic switch connectivity
Anyone tried an active cable for 25G-100G with Ethernet like

https://www.fs.com/de-en/products/154830.html?attribute=37257&id=1744719

Anything forgotten?

 

[i386]

SMB direct/RDMA needs a Windows Server as SMB server (up from a free 2019).

Or a linux distro with kernel >6.6, currently only ubuntu 24.04 is a stable release with such a kernel version.

Question: DAC cables?
There are copper cables up to 5m and active fiber cables up to 100m for easy nic-nic or nic switch connectivity
Anyone tried an active cable for 25G-100G like
https://www.fs.com/de-en/products/154830.html?attribute=37257&id=1744719

what exactly is your question?
if you should use dac, aoc (active optical cable) or optical transceivers? this depends on the the distance you need to cover and budget. dac is cheap and covers short distances, aoc are more epensive and cover longer distances (up to 100m), optical transceivers are the most exepnsive option but can cover 100m to 80km+

 

[pimposh]

These MikroTik branded DACs - i use couple of them and so far had no issues with any of switches nor NICs.
Two versions are available- XS+DA0001 (1m) and XS+DA0003 (3m). Of course they might be not long enough for your needs.

 

[gea]

Or a linux distro with kernel >6.6, currently only ubuntu 24.04 is a stable release with such a kernel version.

what exactly is your question?
if you should use dac, aoc (active optical cable) or optical transceivers? this depends on the the distance you need to cover and budget. dac is cheap and covers short distances, aoc are more epensive and cover longer distances (up to 100m), optical transceivers are the most exepnsive option but can cover 100m to 80km+

Is SAMBA now capable or stable with SMB direct/RDMA like a Windows Server is for many years?
Do these aoc cables work well in the range 5m to 100m with nic to nic Mellanox CX4/5 and 25-100G to build a high performance workgroup for 4k/8k video editing?