WS2012r2, Hyper-V, SMB 3.0 and RDMA

[istav555]

Hi all,

I’ve been exploring SMB 3.0 and RDMA and have a few simple questions that someone with relevant knowledge should answer (I believe) very easily.

The goal is to set up two Hyper-V servers and a file server (all WS2012r2) with SMB 3.0 shares where my VHDXs will be stored. No clustering or anything fancy. I just want to have 4 gigabit NICs from each server going to an unmanaged gbit switch (4x NICs per Hyper-V host and also 4 NICs from the switch to the SMB server).

Then, I set up VMs using VHDX disks on the file server. Each VM will have its own VHDX, no VHDX sharing at all.

My questions, which I wasn’t able to answer by googling or searching the forum, are:

1. I understand that NIC teaming, LAGs, or anything else is NOT required. SMB 3.0 works as is, no special config necessary (and that’s why I’d prefer to save some money and go with an unmanaged switch). Is that right?

2. Do I use separate IPs for every NIC on each server? In general, do the NICs require any kind of special set up?

3. Do I need a “special” router that “supports” RDMA? I came upon the “RoCE” term but I wasn’t able to find this out for sure.

4. Any recommendations on the NICs? Brand/model?

5. Will I actually be able to have near DAS performance using RDMA?

Thanks!

PS. This thread is similar but not 100% and I didn't want to possibly hijack it...

 

[markpower28]

1. I understand that NIC teaming, LAGs, or anything else is NOT required. SMB 3.0 works as is, no special config necessary (and that’s why I’d prefer to save some money and go with an unmanaged switch). Is that right?
Yes. that's what SMB 3 muti-channel for.

2. Do I use separate IPs for every NIC on each server? In general, do the NICs require any kind of special set up?
Yes. Just make sure only one NIC has default gateway.

3. Do I need a “special” router that “supports” RDMA? I came upon the “RoCE” term but I wasn’t able to find this out for sure.
RDMA is a hardware feature in the NIC/HCA. Mellanox brand NIC does have this feature.

4. Any recommendations on the NICs? Brand/model?
For SMB 3.0 Any NIC should be fine. For SMB 3.0 with RDMA you will be better off with NIC has RDMA feature build-in

5. Will I actually be able to have near DAS performance using RDMA?
There are lots of reference in the forum and the answer is yes. 1 GB NIC will not able to do that, but a NIC/HCA with 40GB and above will do.

When people mentioned about SMB 3.o with great performance that's only half of the story. The other half is SMB direct which requires NIC/HCA support RDMA function.
Improve Performance of a File Server with SMB Direct

 

[britinpdx]

See here for a good overview of SMB3

The feature that you are trying to implement is SMB Multichannel, which is enabled by default. I think that the GBe NIC has to support RSS in order for SMB Multichannel to work. No special setup is necessary, no need to setup NIC teaming or LAGG.

It's pretty much plug and play. Just setup each NIC for DHCP and let your router or DHCP server hand out addresses.

In terms of NIC's, my preference is Intel all the time. The Intel Quad VT is inexpensive these days, and are RSS capable. (I believe these are 82575 based devices). The 82574 (very common on many Supermicro motherboards) are also RSS capable, but I ran into issues with a 82579LM on a Supermicro X9SCM-F that didn't want to play nice. (Per ARK, that SKU is supposed to be RSS capable, but it may have been a driver issue, I didn't spend much time debugging)

In 2012R2, open up a Powershell window and run the commandlet "Get-SmbClientNetworkInterface". That will report out if the NIC is RSS capable.

 

[istav555]

Thanks guys, I really appreciate you taking the time to help out.

I need some time to check the links you provided and also some of the keywords you mentioned.

I'll be back

 

[cesmith9999]

RDMA and ROCE only happen on 10GB or getter networks.

Chris

 

[PnoT]

You can get into a 2 or 3 computer setup with some ConnectX-2 cards for under $100 without needing any switches or complicated setup. If you're wanting to play with RDMA that's a cheap way to start.

 

[istav555]

Once again, thank you all for the replies.

Indeed the only way to have RDMA is to go 10GB or faster. Even then, not all 10GB NICs support RDMA.

And, after some digging through in the forum I found out that the cheapest way to get to 10GB+ is infiniband. So, a couple questions:

As PnoT suggested (thanks!), I could use direct pc-to-pc ConnectX-2 cards. Is there anything special that I should be aware of in this case? Apart from the fact that the cards should list RDMA in their specs.

The other option is to go all in and get a proper switch. Again through digging up in the forum I found some threads with references to Mellanox routers. Here's one.

The problem is that the cheapest one I can find in eBay is this one. And I certainly do not need 36 ports...

Any suggestions on some other router?

I'm a complete newbie on this, so I'd like to keep things as simple as possible. For example I'd like to avoid having to flash NICs with custom ROMs, get into trouble with licenses on the router, get confused with port interfaces, etc. I could spend up to $1000 for an 8-port set up right now. Maybe a little more... If anyone has something that he would like to sell and that works out of the box (router & 3 NICs) just let me know.

 

[britinpdx]

With Infiniband there a few different generations, connectors/cables and speeds that you can deal with. The most common today are:

SDR (Single Data Rate) is 2.5Gbps
DDR (Dual Data Rate) is 5 Gbps
QDR (Quad Date Rate is 10Gbps
FDR (Fourteen Data Rate) is 14 Gbps

Links can be bonded in 1x, 4x, 8x and 10x. (4x bonding is the most common, so 4xSDR bonded links give you an effective 10Gbps "pipe" for SDR, 20Gbps for DDR, 40Gbps for QDR and 56Gbps for FDR)

SDR and DDR typically use CX4 connectors (usually copper), QDR and FDR use QSFP connectors (can be copper or fiber).

Infiniband needs a way to route data between ports and that is the job of the Subnet Manager (SM). It can either be built into the firmware on the Switch, or run as a service on an OS.

Mellanox have done a great job of designing their silicon to support a variety of protocols. Their Virtual Protocol Interconnect (VPI) technology actually supports both Infiniband and Ethernet. Each Connect-x IC can support up to 2 ports.

Mellanox therefore offer Infiniband/VPI cards that support both Infiniband and Ethernet, and have QSFP connectors, and also Ethernet cards that have SFP+ connectors. Not only that, but with firmware, they can "throttle" the performance of their silicon. (This allows the marketing team to take a good, better, best approach and sell a Connectx-2 range of cards, single port or dual port, with Infiniband speed tiered as DDR or QDR). It's all in the firmware ....

BTW, the Connectx-2 family has now moved to EOL, so don't expect future firmware/software support.

There are a ton of different connectx cards available, and I posted the Mellanox "decoder ring" in a different thread to assist with determining what is what. right down to bracket heights.

Also be aware that lots of companies OEM the Mellanox Infiniband equipment, so you will see Dell, HP, Sun etc OEM variants also. They typically use their own part numbers, so are not quite as easy to track, but usually you can figure out what they are. They are very likely to have their own firmware, or more specifically their own identifiers (PSID), that make firmware upgrades a little more challenging.

I could use direct pc-to-pc ConnectX-2 cards. Is there anything special that I should be aware of in this case?

Not really, just be sure to look for RevB or later. For a single port VPI card capable of QDR, look for MHQH19x card (where x, the rev of the card, is B or later), and for a dual port card look for the MHQH29x variants. Then pick up a QSFP to QSFP cable and you will be set hardware wise.

Any suggestions on some other router?

First of all, you are looking for an Infiniband Switch. Typically they are listed by their speed (SDR, DDR, QDR etc) and the faster they are the more you will pay, and the noisier they get !!
Infiniband switches and cards will negotiate speed, so it is possible to have a SDR switch and a QDR card work happily, as long as you pay attention to cables. The SDR switch will have CX4 connectors, and the QDR card will have QSFP connectors, so you will need a CX4 to QSFP cable to connect them.

You can find older Infiniband switches for next to nothing. Here's an older Flextronics switch (an OEM for the Mellanox) that's under $70 shipped to your door. I've never used that switch so I can't personally recommend it, it's unsupported and who knows what firmware it has, but none the less, that switch supports 8 ports of 10Gbps connectivity. "But it's only SDR" I hear you say .... yes, it's only SDR but try finding an 8 port 10Gbe switch for $70 shipped, and this guy has about the same performance, if you are willing to do some extra work. (Infiniband is not quite as plug and play as Ethernet, but once you wrap your arms around it, it's not that bad)

The QDR switches are coming down in price quite nicely, and you can probably pick up a Sun, HP OEM for under $400.

I'm a complete newbie on this, so I'd like to keep things as simple as possible. For example I'd like to avoid having to flash NICs with custom ROMs

Well, with Infiniband you are going to have to roll up your sleeves, dig in and get over the initial hurdles, then you will probably find it plain sailing. There's always a chance that you will have to upgrade firmware on cards or even switches, and the Mellanox tools for this are command line based. There's a little RTFM but it's not so difficult, and Google is your friend.

Given that this thread started as a 2012/SMB thread I presume that Windows is your target OS. It's pretty easy and there are lots of resources available for guidance (Jose Baretto's Blog is my goto resource)
This is the specific link that has the Server 2012 information on RDMA and Infiniband.

The most simple setup is 2 servers, one Connectx-2 Infiniband card in each server, one QSFP-QSFP cable, and OpenSM running on one of the servers. Note that your Mellanox NICs must have firmware version 2.9.8350 or higher to work with SMB. You should be able to pickup 2x VPI cards (single or dual port) and a cable easily for under $200. If you don't want to deal with eBay, I can probably hook you up with a couple of cards and a cable ready to rock'n'roll.

 

[Nader]

Hi Everyone, hope everything is going well on your side of the fence. I am new here and in need of help getting RDMA to work on my little storage/SAN project on Windows Server 2016.


So I have 2 PCs directly attached (no switch involved)
I bought 2 of these HP/Mellanox Infiniband QSFP to QSFP cables on Ebay
I bought 2 of these Mellanox MHQH29C-XTR cards on Ebay
I have updated the firmware to 2.9.8350
My client is Windows 10 and my server is windows 2016
OpenSM running as a service
Both ports are up, I can access my storage server but the speed is disappointing.

Max transfer rate I am getting is 240MBps using a ramdisk. I have used similar blog posts to try to get RDMA to work but I am failing miserably. I have used OFED version 4.6 and 5.25, disable Eset on both machines and I still got the same results.

I use perfmon to monitor RDMA traffic on both machines and all the counters stay at zero. I also use iperf and got similar results. I am barely getting 2Gbps out of the 32Gbps that these cards are rated at.

I am new to this InfiniBand stuff but I never thought it would be so easy to fail at this.

If someone can shed some light, any suggestions or help will be greatly appreciated.

Thanks.

 

[i386]

240 mbyte/s is slow for a ramdisk, even without rdma.

About rdma & windows: the windows client os don't support rdma. Several blogs reported that the technical preview of windows 10 supported rdma, but it was later removed from the rtm version.

 

[Nader]

Thanks for letting me know. I stumble on a blog talking about RDMA in Hyper-V and how it is not supported. I am guessing because I use that NIC in Hyper-V switch it disable RDMA. So on the weekend, I am building a separate machine to test this.

Thanks.

 

[Cutha]

I have updated the firmware to 2.9.8350

I use perfmon to monitor RDMA traffic on both machines and all the counters stay at zero. I also use iperf and got similar results. I am barely getting 2Gbps out of the 32Gbps that these cards are rated at.

Hi Nader,

I think you need firmware 2.10.720 to get RDMA working. I went down the same path as you and it was a bit of painful learning experience. I got most of my cards flashed up to 2.10.720.

I started here (post #3) on my journey to get RDMA working. Without this forum I would be lost.

Using a RAM Disk on Windows Server as a host and direct connecting an i5 (4 threads) client Windows Server these were my results:


Command Line: diskspd.exe -c500M -d10 -r -w0 -t4 -o8 -b512K -Sh -L \\192.168.10.1\d$\testfile.dat

Input parameters:

timespan: 1
-------------
duration: 10s
warm up time: 5s
cool down time: 0s
measuring latency
random seed: 0
path: '\\192.168.10.1\d$\testfile.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing read test
block size: 524288
using random I/O (alignment: 524288)
number of outstanding I/O operations: 8
thread stride size: 0
threads per file: 4
using I/O Completion Ports
IO priority: normal



Results for timespan 1:
*******************************************************************************

actual test time: 10.01s
thread count: 4
proc count: 4

CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 18.73%| 1.09%| 17.63%| 81.30%
1| 14.67%| 0.47%| 14.20%| 85.36%
2| 22.63%| 0.94%| 21.69%| 77.40%
3| 13.11%| 0.62%| 12.48%| 86.92%
-------------------------------------------
avg.| 17.28%| 0.78%| 16.50%| 82.75%

Total IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 8023179264 | 15303 | 764.19 | 1528.38 | 5.232 | 0.111 | \\192.168.10.1\d$\testfile.dat (500MB)
1 | 8021082112 | 15299 | 763.99 | 1527.98 | 5.233 | 0.110 | \\192.168.10.1\d$\testfile.dat (500MB)
2 | 8024227840 | 15305 | 764.29 | 1528.58 | 5.232 | 0.077 | \\192.168.10.1\d$\testfile.dat (500MB)
3 | 8014790656 | 15287 | 763.39 | 1526.78 | 5.238 | 0.241 | \\192.168.10.1\d$\testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total: 32083279872 | 61194 | 3055.86 | 6111.71 | 5.234 | 0.149

Read IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 8023179264 | 15303 | 764.19 | 1528.38 | 5.232 | 0.111 | \\192.168.10.1\d$\testfile.dat (500MB)
1 | 8021082112 | 15299 | 763.99 | 1527.98 | 5.233 | 0.110 | \\192.168.10.1\d$\testfile.dat (500MB)
2 | 8024227840 | 15305 | 764.29 | 1528.58 | 5.232 | 0.077 | \\192.168.10.1\d$\testfile.dat (500MB)
3 | 8014790656 | 15287 | 763.39 | 1526.78 | 5.238 | 0.241 | \\192.168.10.1\d$\testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total: 32083279872 | 61194 | 3055.86 | 6111.71 | 5.234 | 0.149

Write IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | \\192.168.10.1\d$\testfile.dat (500MB)
1 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | \\192.168.10.1\d$\testfile.dat (500MB)
2 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | \\192.168.10.1\d$\testfile.dat (500MB)
3 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | \\192.168.10.1\d$\testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00 | 0.000 | N/A


%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | 3.778 | N/A | 3.778
25th | 5.233 | N/A | 5.233
50th | 5.234 | N/A | 5.234
75th | 5.235 | N/A | 5.235
90th | 5.237 | N/A | 5.237
95th | 5.240 | N/A | 5.240
99th | 5.304 | N/A | 5.304
3-nines | 6.361 | N/A | 6.361
4-nines | 14.206 | N/A | 14.206
5-nines | 17.738 | N/A | 17.738
6-nines | 17.738 | N/A | 17.738
7-nines | 17.738 | N/A | 17.738
8-nines | 17.738 | N/A | 17.738
9-nines | 17.738 | N/A | 17.738
max | 17.738 | N/A | 17.738

Using a RAM Disk on Windows Server as a host and direct connecting an i5 (4 threads) client Windows 10 these were my results:


Command Line: diskspd.exe -c500M -d10 -r -w0 -t4 -o8 -b512K -Sh -L \\192.168.10.1\d$\testfile.dat

Input parameters:

timespan: 1
-------------
duration: 10s
warm up time: 5s
cool down time: 0s
measuring latency
random seed: 0
path: '\\192.168.10.1\d$\testfile.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing read test
block size: 524288
using random I/O (alignment: 524288)
number of outstanding I/O operations: 8
thread stride size: 0
threads per file: 4
using I/O Completion Ports
IO priority: normal



Results for timespan 1:
*******************************************************************************

actual test time: 10.00s
thread count: 4
proc count: 4

CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 56.25%| 0.78%| 55.47%| 43.75%
1| 87.03%| 0.94%| 86.09%| 12.97%
2| 19.53%| 1.09%| 18.44%| 80.47%
3| 53.91%| 0.78%| 53.12%| 46.09%
-------------------------------------------
avg.| 54.18%| 0.90%| 53.28%| 45.82%

Total IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 7438073856 | 14187 | 709.35 | 1418.69 | 5.636 | 7.666 | \\192.168.10.1\d$\testfile.dat (500MB)
1 | 6994526208 | 13341 | 667.05 | 1334.09 | 5.994 | 6.878 | \\192.168.10.1\d$\testfile.dat (500MB)
2 | 7246184448 | 13821 | 691.05 | 1382.09 | 5.786 | 10.406 | \\192.168.10.1\d$\testfile.dat (500MB)
3 | 7378829312 | 14074 | 703.70 | 1407.39 | 5.680 | 8.506 | \\192.168.10.1\d$\testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total: 29057613824 | 55423 | 2771.13 | 5542.27 | 5.771 | 8.475

Read IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 7438073856 | 14187 | 709.35 | 1418.69 | 5.636 | 7.666 | \\192.168.10.1\d$\testfile.dat (500MB)
1 | 6994526208 | 13341 | 667.05 | 1334.09 | 5.994 | 6.878 | \\192.168.10.1\d$\testfile.dat (500MB)
2 | 7246184448 | 13821 | 691.05 | 1382.09 | 5.786 | 10.406 | \\192.168.10.1\d$\testfile.dat (500MB)
3 | 7378829312 | 14074 | 703.70 | 1407.39 | 5.680 | 8.506 | \\192.168.10.1\d$\testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total: 29057613824 | 55423 | 2771.13 | 5542.27 | 5.771 | 8.475

Write IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | \\192.168.10.1\d$\testfile.dat (500MB)
1 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | \\192.168.10.1\d$\testfile.dat (500MB)
2 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | \\192.168.10.1\d$\testfile.dat (500MB)
3 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | \\192.168.10.1\d$\testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00 | 0.000 | N/A


%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | 0.901 | N/A | 0.901
25th | 3.130 | N/A | 3.130
50th | 4.795 | N/A | 4.795
75th | 5.759 | N/A | 5.759
90th | 6.261 | N/A | 6.261
95th | 19.356 | N/A | 19.356
99th | 29.903 | N/A | 29.903
3-nines | 39.786 | N/A | 39.786
4-nines | 316.039 | N/A | 316.039
5-nines | 317.736 | N/A | 317.736
6-nines | 317.736 | N/A | 317.736
7-nines | 317.736 | N/A | 317.736
8-nines | 317.736 | N/A | 317.736
9-nines | 317.736 | N/A | 317.736
max | 317.736 | N/A | 317.736



And run directly against the RAM disk (no network) these are the results:

Command Line: diskspd.exe -c500M -d10 -r -w0 -t12 -o8 -b512K -Sh -L D:testfile.dat

Input parameters:

timespan: 1
-------------
duration: 10s
warm up time: 5s
cool down time: 0s
measuring latency
random seed: 0
path: 'D:testfile.dat'
think time: 0ms
burst size: 0
software cache disabled
hardware write cache disabled, writethrough on
performing read test
block size: 524288
using random I/O (alignment: 524288)
number of outstanding I/O operations: 8
thread stride size: 0
threads per file: 12
using I/O Completion Ports
IO priority: normal



Results for timespan 1:
*******************************************************************************

actual test time: 10.11s
thread count: 12
proc count: 12

CPU | Usage | User | Kernel | Idle
-------------------------------------------
0| 122.09%| 0.93%| 121.16%| 0.00%
1| 122.09%| 1.39%| 120.70%| 0.00%
2| 121.94%| 0.15%| 121.78%| 0.00%
3| 122.09%| 1.08%| 121.01%| 0.00%
4| 122.09%| 2.32%| 119.77%| 0.00%
5| 122.09%| 0.93%| 121.16%| 0.00%
6| 122.09%| 0.31%| 121.78%| 0.00%
7| 122.09%| 1.85%| 120.24%| 0.00%
8| 124.10%| 0.46%| 123.64%| 0.00%
9| 124.10%| 0.77%| 123.33%| 0.00%
10| 123.79%| 2.32%| 121.47%| 0.00%
11| 124.10%| 0.62%| 123.48%| 0.00%
-------------------------------------------
avg.| 122.72%| 1.09%| 121.63%| 0.00%

Total IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 13314818048 | 25396 | 1255.96 | 2511.92 | 2.785 | 0.371 | D:testfile.dat (500MB)
1 | 13975945216 | 26657 | 1318.32 | 2636.65 | 2.653 | 0.614 | D:testfile.dat (500MB)
2 | 13277593600 | 25325 | 1252.45 | 2504.90 | 2.793 | 0.545 | D:testfile.dat (500MB)
3 | 13324255232 | 25414 | 1256.85 | 2513.70 | 2.783 | 0.374 | D:testfile.dat (500MB)
4 | 13126074368 | 25036 | 1238.16 | 2476.31 | 2.825 | 1.894 | D:testfile.dat (500MB)
5 | 13490978816 | 25732 | 1272.58 | 2545.15 | 2.749 | 0.328 | D:testfile.dat (500MB)
6 | 13439074304 | 25633 | 1267.68 | 2535.36 | 2.759 | 0.394 | D:testfile.dat (500MB)
7 | 13405519872 | 25569 | 1264.52 | 2529.03 | 2.766 | 0.271 | D:testfile.dat (500MB)
8 | 13555990528 | 25856 | 1278.71 | 2557.42 | 2.735 | 0.447 | D:testfile.dat (500MB)
9 | 13412335616 | 25582 | 1265.16 | 2530.32 | 2.765 | 0.798 | D:testfile.dat (500MB)
10 | 13061062656 | 24912 | 1232.02 | 2464.05 | 2.839 | 1.597 | D:testfile.dat (500MB)
11 | 13472104448 | 25696 | 1270.80 | 2541.59 | 2.752 | 0.587 | D:testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total: 160855752704 | 306808 | 15173.20 | 30346.41 | 2.766 | 0.842

Read IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 13314818048 | 25396 | 1255.96 | 2511.92 | 2.785 | 0.371 | D:testfile.dat (500MB)
1 | 13975945216 | 26657 | 1318.32 | 2636.65 | 2.653 | 0.614 | D:testfile.dat (500MB)
2 | 13277593600 | 25325 | 1252.45 | 2504.90 | 2.793 | 0.545 | D:testfile.dat (500MB)
3 | 13324255232 | 25414 | 1256.85 | 2513.70 | 2.783 | 0.374 | D:testfile.dat (500MB)
4 | 13126074368 | 25036 | 1238.16 | 2476.31 | 2.825 | 1.894 | D:testfile.dat (500MB)
5 | 13490978816 | 25732 | 1272.58 | 2545.15 | 2.749 | 0.328 | D:testfile.dat (500MB)
6 | 13439074304 | 25633 | 1267.68 | 2535.36 | 2.759 | 0.394 | D:testfile.dat (500MB)
7 | 13405519872 | 25569 | 1264.52 | 2529.03 | 2.766 | 0.271 | D:testfile.dat (500MB)
8 | 13555990528 | 25856 | 1278.71 | 2557.42 | 2.735 | 0.447 | D:testfile.dat (500MB)
9 | 13412335616 | 25582 | 1265.16 | 2530.32 | 2.765 | 0.798 | D:testfile.dat (500MB)
10 | 13061062656 | 24912 | 1232.02 | 2464.05 | 2.839 | 1.597 | D:testfile.dat (500MB)
11 | 13472104448 | 25696 | 1270.80 | 2541.59 | 2.752 | 0.587 | D:testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total: 160855752704 | 306808 | 15173.20 | 30346.41 | 2.766 | 0.842

Write IO
thread | bytes | I/Os | MB/s | I/O per s | AvgLat | LatStdDev | file
-----------------------------------------------------------------------------------------------------
0 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | D:testfile.dat (500MB)
1 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | D:testfile.dat (500MB)
2 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | D:testfile.dat (500MB)
3 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | D:testfile.dat (500MB)
4 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | D:testfile.dat (500MB)
5 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | D:testfile.dat (500MB)
6 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | D:testfile.dat (500MB)
7 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | D:testfile.dat (500MB)
8 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | D:testfile.dat (500MB)
9 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | D:testfile.dat (500MB)
10 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | D:testfile.dat (500MB)
11 | 0 | 0 | 0.00 | 0.00 | 0.000 | N/A | D:testfile.dat (500MB)
-----------------------------------------------------------------------------------------------------
total: 0 | 0 | 0.00 | 0.00 | 0.000 | N/A


%-ile | Read (ms) | Write (ms) | Total (ms)
----------------------------------------------
min | 1.615 | N/A | 1.615
25th | 2.684 | N/A | 2.684
50th | 2.748 | N/A | 2.748
75th | 2.805 | N/A | 2.805
90th | 2.865 | N/A | 2.865
95th | 2.916 | N/A | 2.916
99th | 3.082 | N/A | 3.082
3-nines | 15.619 | N/A | 15.619
4-nines | 33.859 | N/A | 33.859
5-nines | 102.033 | N/A | 102.033
6-nines | 102.160 | N/A | 102.160
7-nines | 102.160 | N/A | 102.160
8-nines | 102.160 | N/A | 102.160
9-nines | 102.160 | N/A | 102.160
max | 102.160 | N/A | 102.160

I hope this post was of some use to you. I always worry about offering advice on here because I feel like I know nothing compared to most people on the forum.