Search results

No my explorer caps at 2.8 on a 25gbe connection. Small files will run 2.8GB/s without problem. Files over 50GB the speed will slow down some but in general I can maintain over 2.4GB/s until my basic m.2 drives either run out of cache or start to warm up.

But your not reaching smb direct speeds with file explorer. Your only getting 1.9 GB/s which is like 15gbps on a 25gbps card so no you are not getting SMB direct speed it seems.

What windows version is your client? I'm using 24h2 pro for workstations as they say smb direct isn't supported on home or pro.

I've ran with and without multi channel on my setup and it doesn't change anything. As a matter of fact Windows has something called "Zero Touch ROCE". I can do a fresh install of Windows, change nothing but the IP of my Mellanox card and the MTU size to 9014 and get 23gbps on a 25gbps...

This is all just a test environment so not staying this way. I've heard of issues with refs but those are older threads and so far I have not seen any issues but only time will tell.

Hard to say, I am not 100% positive PFC makes much of a difference or not as I just enable it from the get go as Nvidia's documentation states PFC is needed for ROCE.

Lot's of good reading here if you want to look it over. https://dl.dell.com/manuals/all-products/esuprt_data_center_infra_int/esuprt_data_center_infra_network_adapters/mellanox-adapters_users-guide_en-us.pdf

Client adapter-Funny we are both "network 4", For PFC Data Center Bridging needs to be added to the installed features on the client machine. PS C:\Users\sean> Get-NetAdapterQos -Name "Ethernet 4" Name : Ethernet 4 Enabled : True Capabilities...

The fact that your 10GB file transfer speeds are what they should be but the 100+GB speeds are slower you may want to look into what the drive temps are doing during the file transfer. Mine seem to hit around 55c by the end of a 100GB transfer. You do have Data Center Bridging configured on the...

I tried for like an hour last night to upload some pictures before I gave up.

Windows File Explorer does seem to choke when doing these large files, I normally use Directory Opus anyway and this is what I am seeing. Server>Client Client>Server

I'll have to find something to download that is that big, Nothing I have is that big of a file.

I personally have not tried much larger than 10gb file size, I have been using windows explorer which seems to top out at 2.8 GB/s. If I do the same transfer but with directory opus it tops out at 3.1GB/s.

No I am saying the 40 gig file size used for the transfer. Maybe try a smaller size, like 10gb??

RDMA bypasses CPU limits, that's its sole purpose. One thing to check is on the server look at the event viewer and see if you have any pcie whea errors or errors pertaining to the nics or RDMA. A 40gig transfer could just be filling up the cache on the drives.

I'll post up this evening after work since I'm not at my PC right now. For the rdma stats open performance monitor, press the little plus button and scroll till you see RDMA, once you click on that right below it you will see your adapter listed, select it and click add. Once that is done you...

I'll post my setup more in detail tonight. I am running server 2025, nvmes are in a striped config. I do not use flow control on the nic as rdma works best with PFC which isn't exposed in the gui for the nics configuration. I posted a link further up with a link to setting up the nics and that...

No worries just making sure. I run an epyc 7402 but I have not tried zfs on windows as I get amazing bandwidth using refs. 3.1GB/s both ways no problem with CX4 lx 25gbe. 4.5GB/s both ways with CX4 40gbe cards. Like you said must just be something with zfs on windows that needs to mature. I...

Just asking but I'd assume you don't have something like the network card and the nvme's on all chipset lanes right.

I use 2 nvme's in RAID 0 for testing. Server and client both are NVME storage.