Hi All!
I'm new to the STH forums and wanted to get some guidance/help as to my issue.
I have two devices that have this card and the specs:
- Dell Poweredge R620 (2 * E5-2690 V2, 192GB of RAM, All 2.5" Samsung 850/860 EVO SSD's)
- Custom built ITX server (i7-8700k, 1 * 32gb DIMM, 1 * Samsung 960 EVO NVMe SSD, 1 * 500gb Samsung 860 EVO)
Both systems are using the same exact card Model (MCX353A-FCBT) with the same exact firmware (2.42.5000) and are on PCIe 3.0 x16 slots.
TL;DR - I've configured both systems with both Windows Server 2016 and CentOS 7 with the cards being tuned for single port traffic (Server 2016) and High Throughput (CentOS 7) and made sure that RoCE works for both OS's, but still dont know where the issues with transfer speeds lie for both OS's. Is it a hardware bottleneck on my ITX system or is it somewhere within the configuration of either system? I've posted some benchmarks below along with attached some outputs. I can provide links to guides I've used if necessary.
Detailed version
The purpose of the ITX system will eventually be a high-capacity backup server, but also be used as a regular system when needed. In terms of OS, I've configured Windows Server 2016 and CentOS 7 for both systems, with VMWare running as a secondary boot for the r620. I am also utilizing KVM on the centos 7 array for the server, as I eventually want to create a virtual environment and utilize 40gb speeds (or close to it).
In terms of benchmarking, everything that I have done testing-wise seems to point that I am getting close to 40gb speeds across both devices. Here's what I've done in terms of benchmark testing:
- ntttcp (Windows)
- ib_send_bw (Linux)
- Atto (Windows)
- iperf (Both OS)
However, when I try to do an actual file transfer between both systems (whether its Windows or Linux), I'm getting around 550MB/s, which is nowhere near the speeds that I should be getting (which is anything above 3.3GB/s). With all of the disk benchmarking done, I noticed that transferring from the ITX system to the R620 is fine, but going the reverse slows the speeds down significantly.
I've posted mostly all of the benchmarks in the Mellanox community page from a post I created a couple of weeks ago (with exception of the iperf results), but didn't get any response after a week.
I'm posting the screenshots and outputs of the testing here:
(Ignore the RAID 5 testing, as my arrays are all RAID 1 as of now on the server)
Note that when going from the server to the ITX system yields some bad results and that is where I think the issues are, but again am not sure.
What I also found interesting was that ntttcp (at least for me) did not like to perform multi-threaded tests and would just hang at the version. But doing a single threaded test will run with no issues, as seen on the ntttcp output file.
Lastly, iperf will come back with nearly 40Gbit speeds when doing the command (will post it if needed).
I've followed some guides to get RoCE working for both OS's on both systems and can paste the links if necessary, but it is where I am stuck at the moment.
Any help/advice/guidance is greatly appreciated.
Thank you.
I'm new to the STH forums and wanted to get some guidance/help as to my issue.
I have two devices that have this card and the specs:
- Dell Poweredge R620 (2 * E5-2690 V2, 192GB of RAM, All 2.5" Samsung 850/860 EVO SSD's)
- Custom built ITX server (i7-8700k, 1 * 32gb DIMM, 1 * Samsung 960 EVO NVMe SSD, 1 * 500gb Samsung 860 EVO)
Both systems are using the same exact card Model (MCX353A-FCBT) with the same exact firmware (2.42.5000) and are on PCIe 3.0 x16 slots.
TL;DR - I've configured both systems with both Windows Server 2016 and CentOS 7 with the cards being tuned for single port traffic (Server 2016) and High Throughput (CentOS 7) and made sure that RoCE works for both OS's, but still dont know where the issues with transfer speeds lie for both OS's. Is it a hardware bottleneck on my ITX system or is it somewhere within the configuration of either system? I've posted some benchmarks below along with attached some outputs. I can provide links to guides I've used if necessary.
Detailed version
The purpose of the ITX system will eventually be a high-capacity backup server, but also be used as a regular system when needed. In terms of OS, I've configured Windows Server 2016 and CentOS 7 for both systems, with VMWare running as a secondary boot for the r620. I am also utilizing KVM on the centos 7 array for the server, as I eventually want to create a virtual environment and utilize 40gb speeds (or close to it).
In terms of benchmarking, everything that I have done testing-wise seems to point that I am getting close to 40gb speeds across both devices. Here's what I've done in terms of benchmark testing:
- ntttcp (Windows)
- ib_send_bw (Linux)
- Atto (Windows)
- iperf (Both OS)
However, when I try to do an actual file transfer between both systems (whether its Windows or Linux), I'm getting around 550MB/s, which is nowhere near the speeds that I should be getting (which is anything above 3.3GB/s). With all of the disk benchmarking done, I noticed that transferring from the ITX system to the R620 is fine, but going the reverse slows the speeds down significantly.
I've posted mostly all of the benchmarks in the Mellanox community page from a post I created a couple of weeks ago (with exception of the iperf results), but didn't get any response after a week.
I'm posting the screenshots and outputs of the testing here:
(Ignore the RAID 5 testing, as my arrays are all RAID 1 as of now on the server)
Note that when going from the server to the ITX system yields some bad results and that is where I think the issues are, but again am not sure.
What I also found interesting was that ntttcp (at least for me) did not like to perform multi-threaded tests and would just hang at the version. But doing a single threaded test will run with no issues, as seen on the ntttcp output file.
Lastly, iperf will come back with nearly 40Gbit speeds when doing the command (will post it if needed).
I've followed some guides to get RoCE working for both OS's on both systems and can paste the links if necessary, but it is where I am stuck at the moment.
Any help/advice/guidance is greatly appreciated.
Thank you.
Attachments
-
2.1 KB Views: 4
-
1.2 KB Views: 2