F2uSoe: 9 disk NVMe results 19GB/S - IOPS --- 1.5mil (only)

[Naeblis]

CPUs were upgraded to 26733 and memory is now 8 x 32 LRDIMM

Well 4 more drives showed up today so i am up to 9 NVMe Drives (1.2 x 7 + 400GB x 2) they are housed in the two Intel Cages + 1 PCIe.

1st thoughts, Crystal Disk Mark < Coronal Jessep voice> "You can't handle the Truth!" < /Coronal Jessep voice> CDM only reports a power level of OVER 9000! Cant crack 10. So i don't think it was designed to go to deal with numbers this high.


So we switched to ATTO and capped out at 19GB/s that breaks down to 2.1 GB/s from each drive. slighly slower than the reads on the 400Gb versions. (7 x 1.2 would only give me 17.5). so no breaking 20GB/s today. I have three more 1.2 incoming.



also the CPU usage on ATTO was much less then what CDM uses



also of note as soon as we hit go on ATTO the fans on the server went into Turbo mode and have not shut up even after all the testing is done.


Ill have IOPs in a few minutes

Thom

 

[Biren78]

Hot! IOmeter? and which drivers?

 

[Naeblis]

IOmeter is in progress right now. and using the Intel NVMe Drivers

 

[Naeblis]

Ok now i am scratching my head. I am only able to get 1.5mil iops. Crazy Talk, I know right.

3 workers per drive


it did not get better with 4 workers per drive, sure did make my CPU% go up though

 

[Patrick]

18363E3 I like that you have so many digits that you need the E.

 

[Chuntzu]

Since I spent all of yesterday benchmarking as well I feel your pain when the numbers don't line up to expectations. Diskspd is what I use then verify on iometer. With regards to iometer the I set up enough workers to match the number of cores including hyperthreading and set up a deep queue depth like 32 and move up to 128. When running the random io tests you should be redlining the cpu with the workers. If you over saturate the cpu with workers ie have use 50 workers when my dual 2680v2 have 40 threads for example then the number tend to be much lower since the cpu is spending time managing threads instead of io. Last thing I bring up because I am a bone head and did this when learning how to use io meter but you have to manually assign workers to drive, can't just go to the all managers and select all drive and they evenly distribute. Have to click on a manager and then select a drive. Seriously drove me crazy when I first started using it since I didn't check this and the numbers weren't matching what I expected them to be.

 

[Naeblis]

Since I spent all of yesterday benchmarking as well I feel your pain when the numbers don't line up to expectations. Diskspd is what I use then verify on iometer. With regards to iometer the I set up enough workers to match the number of cores including hyperthreading and set up a deep queue depth like 32 and move up to 128. When running the random io tests you should be redlining the cpu with the workers. If you over saturate the cpu with workers ie have use 50 workers when my dual 2680v2 have 40 threads for example then the number tend to be much lower since the cpu is spending time managing threads instead of io. Last thing I bring up because I am a bone head and did this when learning how to use io meter but you have to manually assign workers to drive, can't just go to the all managers and select all drive and they evenly distribute. Have to click on a manager and then select a drive. Seriously drove me crazy when I first started using it since I didn't check this and the numbers weren't matching what I expected them to be.

in my case when i used 4 workers per drive (36 out of 48 cores) the IOPs dropped by 100k compared to 3 workers per drive (27 out of 48 cores).

the same sort of problem also presents itself with CDM. my optimal threads is 18 out of 48 cores. when i get the 12 drives i can try and even 2 3 and 4 and compare the results.

However in the End. We have very little control on threads and queue depth being presented to the storage so this is just an exercise in knowing <Scotty voice> "Jim she just can't take anymore" </Scotty voice>.

My original plan was to have all 1.2 TB drives in a tier 2 storage raid (mirror, 4.8 Tb of NVMe + 3 TB of SSDs). Based on this testing I have decided to break up my drives like this.

1. VHDs for OS & Programs drive = 400gb + 400GB mirror + 3tb of SSDs in a 2 tier group.
   1. 2.2 GB/s sec & 400k+ IOPs
2. Data 1 Drive = 1.2.+ 1.2 (mirror)
   1. 2.6 GB/s sec & 400k+ IOPs
3. Data 2 Drive = 1.2.+ 1.2 (mirror)
   1. 2.6 GB/s sec & 400k+ IOPs
4. Data 3 Drive = 1.2.+ 1.2 (mirror)
   1. 2.6 GB/s sec & 400k+ IOPs
5. Temps Drive Drive = 1.2.+ 1.2 (mirror)
   1. 2.6 GB/s sec & 400k+ IOPs
6. Logs Drive Drive = 1.2 (no mirror)
   1. 2.6 GB/s sec & 400k+ IOPs

This will give the SAN the ability to divide the IO and focus it where it is needed most, SQL baby!

well this is the plan. ill let you all know if it works.

 

[T_Minus]

I cannot wait until we can do some "real world" SQL tests

PS: If you have CDM + NVME CPU usage screenshots share those I'm trying to find mine, but I think it's worthwhile showing how much CPU is used for the IOPs.

 

[Naeblis]

I cannot wait until we can do some "real world" SQL tests

PS: If you have CDM + NVME CPU usage screenshots share those I'm trying to find mine, but I think it's worthwhile showing how much CPU is used for the IOPs.

Last night we took down the config that we used to get 1.5 mil IOPS. we have decided to focus this week on remote speeed. Why limit all those beautiful IOPS to just one server? Let them come out and play.

With the arrival of :

• 36 port Mellanox SX6036 switch FDR14 (yea we beat our head on the wall with that FDR10 one)
• BLUE tabled cables
• FDR cards
• Enough LSI3008 cards to get 1 node of our SOFS cluster up
• 6 Dual Port backplanes

it was time to use the NVMe drives as the baseline play with the network settings on SOFS node 1 cluster. This way we could see how much performance we could get off that baby and do some serious abusing of 32 SAS SSDs drives.

The NVMe SAN is hosting all the VMs at the moment so i did not want to risk messing something up settings there. Once we find the best settings we will take them to the NVMe SAN and post results.

Just as a note, CDM is unreliable over 5GB/s. additionally it can only assign workers to one drive. I got the best IOPS results using IOmeter and dividing up the drives.

 

[T_Minus]

Ooooh exciting Sounds like it's coming along nicely! With the holidays behind us time to push forward.

 

[Naeblis]

Whatcha know, I did have a raid 0 set up with 8 Drives.

TLDR here is the write CPU usage at different stages,



With CDM I have not seen a difference in sequential performance by using more than 1 thread. Thus CPU 0 is maxed
Reads



With the 4K and CDM we found the sweet spot at 18 threads. your guess is as good as mine as to why.

one thing i did notice is a minor amount of bleed over on 8 CPUs


back to 1 CPU for the last two read tests





writes
i think there is slightly lower CPU usage, stupidly i did not include the total % in my screen grab







final numbers
-----------------------------------------------------------------------
CrystalDiskMark 5.0.2 x64 (C) 2007-2015 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 9038.627 MB/s
Sequential Write (Q= 32,T= 1) : 7027.882 MB/s
Random Read 4KiB (Q= 32,T=18) : 1965.272 MB/s [479802.7 IOPS]
Random Write 4KiB (Q= 32,T=18) : 1783.748 MB/s [435485.4 IOPS]
Sequential Read (T= 1) : 6760.855 MB/s
Sequential Write (T= 1) : 6135.785 MB/s
Random Read 4KiB (Q= 1,T= 1) : 38.129 MB/s [ 9308.8 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 218.307 MB/s [ 53297.6 IOPS]

Test : 4096 MiB [N: 10.0% (50.1/503.9 GiB)] (x4) [Interval=5 sec]
Date : 2016/01/01 19:33:36
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

 

[Naeblis]

just went back and looked at the total CPU%. for some reason the M drive gives better sqeuentials. I have to ask Alex what the difference is.



perfmon sure does not show 25%. set scale to 30 max to better see the numbers

-----------------------------------------------------------------------
CrystalDiskMark 5.0.2 x64 (C) 2007-2015 hiyohiyo
Crystal Dew World : Crystal Dew World
-----------------------------------------------------------------------
* MB/s = 1,000,000 bytes/s [SATA/600 = 600,000,000 bytes/s]
* KB = 1000 bytes, KiB = 1024 bytes

Sequential Read (Q= 32,T= 1) : 9777.682 MB/s
Sequential Write (Q= 32,T= 1) : 9216.003 MB/s
Random Read 4KiB (Q= 32,T=18) : 1936.543 MB/s [472788.8 IOPS]
Random Write 4KiB (Q= 32,T=18) : 1780.390 MB/s [434665.5 IOPS]
Sequential Read (T= 1) : 7053.335 MB/s
Sequential Write (T= 1) : 6896.085 MB/s
Random Read 4KiB (Q= 1,T= 1) : 38.021 MB/s [ 9282.5 IOPS]
Random Write 4KiB (Q= 1,T= 1) : 215.454 MB/s [ 52601.1 IOPS]

Test : 4096 MiB [M: 16.7% (100.2/599.9 GiB)] (x3) [Interval=5 sec]
Date : 2016/01/01 20:20:27
OS : Windows Server 2012 R2 Datacenter (Full installation) [6.3 Build 9600] (x64)

 

[Quasduco]

Your task manager shows 2.67GHz, and that chip turbos to 3.2. I bet perfmon is scaling against the max 3.2, whereas task manager is scaling against current frequency.

 

[alex1002]

Can we see some pics of the hardware?

 

[Naeblis]

Can we see some pics of the hardware?

There are see real in these threads

Search Results for Query: F2uSoe | ServeTheHome and ServeThe.Biz Forums

 

[Chuntzu]

Couple of quick things, first being awesome numbers! Second if you are using newer CDM it's using diskspd from Microsoft so why not just use that? It's extremely easy to use and very powerful. It is a direct replacement for sqlio. It also has the nice feature of being able to use remote shares as targets (long day and forgot this term) ie \\server01\share1\testfile.dat, \\server02\share01\testfile.dat.

With the comment about the task manager showing lower speeds on the cpu, make sure to set power management within the os to high performance. Then do the same thing in the bios so that they system doesn't try and down clocking the cpu. I figured this piece of information out the hard way.

Keep up the testing, hope to contribute more as soon as all my parts arrive!