Hi Guys,
I just bought a few 40GigE Mellanox Connect x-3 cards and connected two storage servers over an active QOSFP+ optical cable. We use RAID arrays which are very fast. On I/O meter we get around 5100mbps across the servers over the 40gE connection.
As a side note we had to open "opensm" which starts a service to get the two servers to see each other, since we don't have a switch.
So the deal is, we want the fastest possible file transfer speeds. We work largely in the Media and Entertainment market, so people need to move data around fast!
We don't see much difference between 40GigE and 10GigE as far as file transfer is concerned. Like I need to move 100, 55gigabyte .mov files across the server in record time, but I max out at like 1.2GBps. That's using Teracopy with 9 threads. With windows explorer file transfer its around 800MBps.
These are the same speeds I get on my 10GigE connection. I believe it's because the 40GigE uses 4x10GigE and somehow combines those four threads. While a program like IO meter can access the threads, the file transfer process cannot.
Does anyone have any ideas on how to get these file transfer speeds up direct server to server? I've tried using Teracopy, Richcopy, and File Explorer simultaneously but they all seem to share a 1.2GBps pipe.
We use Windows Server 2016 direct connect by manually assigning IP addresses and then double clicking the "opensm" server which is inside the driver files. So that's how we connect them via 40GigE. And we can get like 48Gbps on speed tests, but no where near that while doing file transferring.
I'm already aware of the different type of files .dpx .mov .mxf and their transfer rate differences, so it's not the file type that is the problem, as I'm comparing the same types of files across a 10gE vs a 40gE and they are the same.
Please any advice would be appreciated.
-Myth
I just bought a few 40GigE Mellanox Connect x-3 cards and connected two storage servers over an active QOSFP+ optical cable. We use RAID arrays which are very fast. On I/O meter we get around 5100mbps across the servers over the 40gE connection.
As a side note we had to open "opensm" which starts a service to get the two servers to see each other, since we don't have a switch.
So the deal is, we want the fastest possible file transfer speeds. We work largely in the Media and Entertainment market, so people need to move data around fast!
We don't see much difference between 40GigE and 10GigE as far as file transfer is concerned. Like I need to move 100, 55gigabyte .mov files across the server in record time, but I max out at like 1.2GBps. That's using Teracopy with 9 threads. With windows explorer file transfer its around 800MBps.
These are the same speeds I get on my 10GigE connection. I believe it's because the 40GigE uses 4x10GigE and somehow combines those four threads. While a program like IO meter can access the threads, the file transfer process cannot.
Does anyone have any ideas on how to get these file transfer speeds up direct server to server? I've tried using Teracopy, Richcopy, and File Explorer simultaneously but they all seem to share a 1.2GBps pipe.
We use Windows Server 2016 direct connect by manually assigning IP addresses and then double clicking the "opensm" server which is inside the driver files. So that's how we connect them via 40GigE. And we can get like 48Gbps on speed tests, but no where near that while doing file transferring.
I'm already aware of the different type of files .dpx .mov .mxf and their transfer rate differences, so it's not the file type that is the problem, as I'm comparing the same types of files across a 10gE vs a 40gE and they are the same.
Please any advice would be appreciated.
-Myth
Last edited: