Adventures with vSphere 6.5 and Raid-0

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Matt Lund

Member
Oct 20, 2016
48
8
8
48
This is kind of a build log with some mistakes. I enjoy hearing from others on this kind of thing so thought I'd record my mistakes!

Recently got one of the Supermicro Xeon-D setups, the one with the D-1567 which is 12-core 2.1 to 2.7 GHz. One of the first things I needed it for was an application testing setup where data is pulled from azure blob storage and loaded into a sql server. I wrote the app to do this and need to regularly run it so I can experience any problems myself. It's a bit I/O intensive - reads about 600GB of json documents and ends up with a 1.5 TB SQL Server database and it likes lots of cores.

So I didn't have much time to set up my server and get familiar with it. I knew I wanted a Raid-0 with three S3700 800GB drives to be the data drive for the SQL database. Raid-0 is perfect in this case because the data is easy re-creatable in the case of a failure, this is only a test system, and any throughput bonuses from Raid-0 are welcome.

I considered very briefly installing Windows Server 2016 directly to the hardware but I'll use this hardware for other things as well so what I really wanted was vSphere with a Windows Server 2016 VM that, when used, would take most of the 128GB of RAM I put in the server as well as most of the cores.

So, I set up the Raid-0 with the built-in Intel RST. I've done that before on a workstation and thought it'd be fine. Turns out that vSphere didn't see the resulting 2.4TB drive, it just saw the three individual drives. I knew RST isn't robust but I thought that for a Raid 0 that wouldn't be a big deal. Quick google searching told me that RST doesn't actually present the logical drive to the operating system. Rather, it counts on the operating system to have drivers needed to treat it like one logical volume. This explains why this has worked on a workstation with windows but not from vSphere.

So I then tried to see whether vSphere has any poor man's software raid that would give me a Raid-0 logical volume. A bit of panic set in because I need to do my first run of the app tonight.

So attempt 2 was to create a data store from one of the 800GB drives then add the second and third. In my mind this is what Raid 0 is anyway, right? Perhaps not...

I proceeded to create a VM and carved out the first 100GB of this datastore's space for the Windows OS. Already at this point there were red flags. Windows Server 2016 took like 1-2 hours to install. Then when it finally finished I attempt to run Crystal DiskMark on the C: drive and 10 minutes later it was still working. I then carved out 2TB of the datastore and mounted it as a separate disk for the VM and tried DiskMark there. Same thing. Just odd. So perhaps this should have worked better but I didn't have much time to troubleshoot it. Why did I install the OS into the same datastore that would be the SQL storage - probably another odd choice.

I then considered whether the 9207-i that I bought to play with might help here. Problem is it's in IT mode. I've never owned one of these before (it arrived about the same time as the server) but I was able to follow some instructions for switching from IR to IT mode but in my case did the reverse. I was a bit intimidated given that I was already feeling rushed but this worked fine. The article used a virtual disk which I had a bit of trouble with so I did thumb drive instead but other than that it worked out fine.

With the 9207-i now in IR mode (does that make it a 9217?) I was able to create a proper RAID 0 volume. Awesome. Booted into vSphere and removed the old datastores. I also connected a Samsung 850 Evo 1TB (meh) and a 8TB Red (needed room for DB backups). All three separate datastores.

I then created the VM on the Samsung datastore. Windows installed in minutes. Much better. Next I attached a persisted independent disk to the VM with about 2TB of the Raid-0 volume. And again with the 8TB drive.

Ran Crystal against the C: drive, D: drive (the Raid 0), and the E: drive (the 8TB Red). The D: drive was the one I really care about. The sequential is pretty close to what I saw with the same three disks in a Raid-0 via RST on a workstation. The 4K is a worse, which I sort of expected. Now I gotta wait 2 hours while I restore a DB via 1 gigabit from a NAS. Then it's time to start doing runs of the application. Having said that, I'm really not sure how much my workload wants high 4K performance versus sequential. So we'll see.

Had the Raid 0 not worked out with the 9207 in IR mode, I think I could have just created 3 SQL Server mdf files to the three separate drives. But I really wanted the Raid 0 approach.

The one other learning from this was that I had some problems with the IPMI interface to the Supermicro. This was mostly when working with RST which turned out to be a dumb approach to take anyway. But what was frustrating was then from within the pre-boot RST utility (accessed via IPMI) most of my keyboard clicks were being delayed 10-20 seconds and some of them lost. This made something like even typing in a volume name rather difficult. FWIW, I was using the HTML5 interface. I may play more with the Java interface in the future but was hoping the HTML5 would work out well.
 

Matt Lund

Member
Oct 20, 2016
48
8
8
48
Here are the differences in the performance of the Raid0 with three S3700 800GB drives. Basically the sequential is fine within the virtualization but the 4k suffers.

Workstation / RST Raid 0 / no virtualization (Windows 10 Pro)

Read
Q32T1 - 1394.0
4K Q32T1 - 432.8
Seq - 1151.0
4K - 35.2

Write
Q32T1 - 1326.0
4K - 398.9
Q32T1 - 1089.0
Seq 4K - 90.3

Supermicro D1567 / 9207 IR Raid 0 / vSphere / Windows Server 2016 VM

Read
Q32T1 - 1351.0
4K Q32T1 - 405
Seq - 810.9
4K - 12.78

Write
Q32T1 - 1268.0
4K - 411.3
Q32T1 - 902.1
Seq 4K - 24.11
 

Matt Lund

Member
Oct 20, 2016
48
8
8
48
This business with inter-machine 1 gigabit is just not gonna cut it. Part of my excitement with the D-1567 is moving to 10 gigabit. Can't wait to end up with a 10-gigabit setup between workstation, vSphere box, and a FreeNAS box. Moving a 1TB backup file is like 2 hours at 1 gigabit speeds!
 
  • Like
Reactions: T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,625
2,043
113
Appreciate the log and look forward to the build :) sounds awesome and my next couple builds will be for mysql testing/comparisons.
I'll be interested if you re-test with a SAS3 HBA and see if you get anymore performance in raid0 with those 3, you just might but not sure with only 3.
 
  • Like
Reactions: Matt Lund

Matt Lund

Member
Oct 20, 2016
48
8
8
48
I wouldn't think SAS3 versus SAS2 would affect 4k performance, would it? I'd like to know if so though I think I'd have trouble justifying buying another controller :)
 

Matt Lund

Member
Oct 20, 2016
48
8
8
48
The variant I might try is passing the 9207 (albeit in IR mode) through vSphere to the VM. I wonder if the 4k performance would come back to native speeds at that point. The downside in the case of a build like this is that I only get *one* PCI-E expansion port. So if I pass the controller through to a VM, that's gonna be the only VM that gets it.