While I am still waiting for a howto for a working Linux ksmbd server with SMB Direct/rdma and Windows clients, there are many infos for a troublefree working ultrafast Windows Server NAS setup with SMB Direct and up to 10Gbyte/s:
SMB Direct/rdma and Windows Server (cheap Windows 2019 Essentials is mostly ok with 25 users/ 50 devices included)
The key for ultrafast NAS transfers is SMB Direct/rdma. With 25G nics you can get a NAS performance similar to a local NVMe (3GB/s) with lowest latency and CPU load. With 100G you can even go up to 10 GB/s over SMB.
Especially for multi-user 4/8k video use cases such a performance is wanted for editing directly from NAS
Windows Server supports SMB Direct up from Server 2012 and is still the best option for SMB Direct as it is a „it just works“ solution. The only alternative is ksmbd on Linux but I have not seen a success story for SMB Direct and Windows clients up to now.
SMB Direct requirements
You need RDMA capable nics 25-100G that should support RoCE v2 (RDMA over Converged Ethernet)
Most common are Mellanox ConnectX 4-6 (single or dualport) or Intel E810 (up to 4x25G, or low profile)
You can use direct cabling
between a Windows Server NAS and Linux or Windows 11 Pro clients with DAC cables (passive up to 5m or active ones). This is the cheapest and most power efficient method as you do not need a 100G sitch. A power
efficient nic like a ConnectX- 4LX is under 10W with DAC cables. For normal communication you can use your 1-10G nic
connected to your current switch or router.
The alternative is a 100G switch.
A cheap 25W option with 4x100G is a Mikrotik CRS504-4XQ with newest firmware to support RoCEv2,
Switch setup
help.mikrotik.com
Example
Hab gerade bei mir zuhause die Verkabelung umgestellt und dann 2h Fiber debugged... Stellt sich raus: Der Zulieferer hat mir komplett falsche Kabel geschickt. Bestellt war OS2 MTP APC... Die gibt es eigentlich NUR in APC, ich hab die noch nie UPC gesehen (Multimode MTP ist dagegen immer UPC...
www.hardwareluxx.de
Linux client setup
Hab gerade bei mir zuhause die Verkabelung umgestellt und dann 2h Fiber debugged... Stellt sich raus: Der Zulieferer hat mir komplett falsche Kabel geschickt. Bestellt war OS2 MTP APC... Die gibt es eigentlich NUR in APC, ich hab die noch nie UPC gesehen (Multimode MTP ist dagegen immer UPC...
www.hardwareluxx.de
Windows Client setup
and infos about NVMe tuning for newest Windows 11 and Server 2025
Random 4K performance is looking good with a native NVMe implementation.
www.pcgamer.com