Help Needed with 10GbE speeds.

[atreides]

Greetings everyone, I am green when it comes to networking. My issue is that i cannot get speeds faster than 230MB/s.

I have a D-Link DGS-1510-28X with 4 10GbE Sfp+ ports. I am connecting 4 of my PCs of which i want to transfer very large files across each with a Mellanox Connectx-2 NIC.

I am not sure where to go from here. Clearly the 4 pcs are faster (but not getting speeds i should) than my other pcs on the network, which include all the family pcs on the same switch.

I am using ramdisk for the transfer.

Any help would be greatly appreciated.

cheers.

 

[i386]

Some information is missing so here are a few general questions:

What os do you use on the pcs/servers?
Firmware up to date?
Drivers up to date (the driver shipped with windows 10 doesn't enable all connectx2 features/settings)?
Jumbro frames enabled (pcs, switch)?
Send/receiver buffers set to 4k?

 

[atreides]

thanks so much for the reply, answers in the quote below.

Some information is missing so here are a few general questions:

What os do you use on the pcs/servers? > mostly win 10 pro pcs and 2 macs > server is win 12 r2

Firmware up to date? > yes

Drivers up to date (the driver shipped with windows 10 doesn't enable all connectx2 features/settings)? > going through update driver on the properties window. on the site, i was clueless as to what to download.

Jumbro frames enabled (pcs, switch)? > on the switch yes, when i tried this on the PCs i lost the network for some reason. i still want the pcs to have internet access through their gigabit link, however during tests, i had them disabled

Send/receiver buffers set to 4k? > i am not sure what this is or how to do it.

 

[atreides]

Some information is missing so here are a few general questions:

What os do you use on the pcs/servers?
Firmware up to date?
Drivers up to date (the driver shipped with windows 10 doesn't enable all connectx2 features/settings)?
Jumbro frames enabled (pcs, switch)?
Send/receiver buffers set to 4k?

More specifically, firmware on the switch is up to date.
Jumbo frames is enabled by port on the switch only.

 

[i386]

Firmware + firmwaretools
http://www.mellanox.com/page/firmware_table_ConnectX2EN
Drivers
http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_sw_drivers

 

[atreides]

Firmware + firmwaretools
http://www.mellanox.com/page/firmware_table_ConnectX2EN
Drivers
http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_sw_drivers

okay, updated... thanks for the links. now when i change the jumbo frame to 9014, both pcs now stay on the network so the drivers helped with that, but no real change in speed.

so now both nics and switch (on the ports only) are set to jumbo frames of 9014.

I do not understand the send receive buffer if you could explain please.

what else am i doing wrong. The nics are using the sfp+ ports on the switch and get internet from the gigabit network, should there be separation with vlans or something like that?

 

[atreides]

Firmware + firmwaretools
http://www.mellanox.com/page/firmware_table_ConnectX2EN
Drivers
http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_sw_drivers

i think i found the receive send buffers, on the nic configure > advanced tab.
receive buffer = 4096
send buffer = 2048 ~ what should the value be?

 

[i386]

what else am i doing wrong.

You probably nothing. I start to believe that your ramdisk might be cause for the low performance.

Instead of testing the performance with ramdisks can you try it with iperf or jperf and report the results?

Edit:

i think i found the receive send buffers, on the nic configure > advanced tab.
receive buffer = 4096
send buffer = 2048 ~ what should the value be?

The max value, 4096

 

[atreides]

okay, will do.

 

[MiniKnight]

How are you testing? That's normally the reason people can't get 10G speeds.

 

[atreides]

How are you testing? That's normally the reason people can't get 10G speeds.

thanks for the reply,

I am testing with ramdisk to ramdisk from softperfect.
I also used ntttcp and it got 218 MB/s where ramdisk got as high as 240 MB/s

However, now i am having issues pinging, when i changed the jumbo frames, for some reason, i cannot ping, i cannot use ntttcp or iperf, i get "connection refused" but ramdisk can still transfer, weird.

 

[atreides]

okay, rebooted and got iperf working, transfer is 2.38 GBystes and bandwidth is 2.04 Gbits/sec

 

[MiniKnight]

What command are you using? Are you single thread or multiple streams? You need multiple for 10G

 

[atreides]

What command are you using? Are you single thread or multiple streams? You need multiple for 10G

what command would that be "-P" and how many ?

 

[atreides]

i still did not use the multiple stream, but a test from the other 2 pcs gave a transfer of 4.24 GBytes and 3.62 Gbits/sec

 

[Schoondoggy]

i still did not use the multiple stream, but a test from the other 2 pcs gave a transfer of 4.24 GBytes and 3.62 Gbits/sec

Gbits or GBytes? General testing of 10GbE tends to show transfers rates as high as 1.2 GByte/s. The limiting factor in transferring large files tends to be storage I/O performance. How big are your files and what is the storage in your PC's, MAC and server?

Sent from my LG-H910 using Tapatalk

 

[atreides]

Gbits or GBytes? General testing of 10GbE tends to show transfers rates as high as 1.2 GByte/s. The limiting factor in transferring large files tends to be storage I/O performance. How big are your files and what is the storage in your PC's, MAC and server?

Sent from my LG-H910 using Tapatalk

4 Gig files is what i used to test over ram disk and i tested on iperf. Gbiits/s is what it said, however i am thinking cable or nic as one pc drops the network connection after waking from sleep and if i disable and renable the NIC it comes back on. The other 3 nics have not done that

Frustrating. I am using the generic fsp+ DAC passive from FS.com that are 3ft, i have one that is going to be 30 feet away but is not ready yet, so i cannot test as i am building the machine. BUT i am using an AOC cable for that distance.

 

[acquacow]

I hope you get this figured out, it's definitely frustrating to get it all working initially.

I'm setup with 10gig copper over cat5 here at home. Took a while to get it to play nicely with my gig-e network running in parallel (had to set interface metrics and play with some fire-walling to get certain services to only advertise on specific interfaces).

That said, I can push over 900MB/sec point to point now no prob. It's quite nice.

 

[Schoondoggy]

I know you had mentioned that you had been testing with a ramdisk, but if you convert the numbers you posted above into MB/s they would be 530MB/s and 452MB/s that looks like SATA III SSD transfer rates. What drives are in your PC and MAC?

Sent from my LG-H910 using Tapatalk

 

[atreides]

I know you had mentioned that you had been testing with a ramdisk, but if you convert the numbers you posted above into MB/s they would be 530MB/s and 452MB/s that looks like SATA III SSD transfer rates. What drives are in your PC and MAC?

Sent from my LG-H910 using Tapatalk

For now i am testing 3 computers before i add more to the switch.... Windows server 2012 r2 and 2 win 10 machines, each with a NVME PCIE m.2 drive