iSCSI + MPIO + ESXi6

[BSDguy]

Sorry if this is a long post but I could use some help please

I just passed my VCP6-DCV exam and as a result of that I purchased two of the following servers:

Supermicro 5028D-TN4T

Each server has 128GB of RAM and I also bought a Cisco SG300-28 managed gigiabit switch. I am using my Netgate SG-2440 as the firewall/router with pfsense. I already had a custom built server based on a Supermicro X10SL7-F motherbaord that I will be using for the SAN using Starwinds Virtual SAN as the iSCSI target.

When I studied for my VCP6 exam I did everything nested in VMware Workstation 12.5 so this is my "real" first vSphere 6 cluster setup with shared storage. I've setup my SAN as follows:

• Windows Server 2016 RTM with the MPIO role installed (and I enbaled MPIO for iSCSI devices)
  
• 32GB of RAM
• For now, two Samsung SM863 480GB SATA SSD drives to be used as datastores (no RAID)
• Two Samsung Pro 840 128GB drives mirrored for boot volume (not used for VMs)
  
• BIOS, IPMI and LSI firmware is all on the latest as of today
• Installed Starwinds Virtual SAN 8.0.9996.0
• Installed a quad HP NC364T gigabit network card to be used for iSCSI traffic

On the ESXi side I have ESXi6 on the latest build as of today and also installed a quad low profile HP NC364T network card which will be used for iSCSI traffic only. On the Cisco switch 12 of the ports are in a VLAN for iSCSI storage traffic only.

In vCenter I created 4 vmkernels with the following IP addresses: 192.168.60.6 to .9 (all on one subnet) and am using "network port binding" for the software iSCSI initiator (all the vmks are showing as compliant). On the two datastores (each datastore is backed by an SM863 SSD) I have enabled Round Robin. I also have a dedicated virtual switch with 4 port groups (one for each ISCSI vmk) and have each port group to use one of the NICs on the quad NIC card.

On the local SAN side I have benchamrked the disks and always get 400-500MB/s speeds using ATTO/CrystalMark.

After creating a test VM on one of the datastores I get about 450MB/s when I run CrystalMark and about 300-350MB/s when running ATTO but when I copy a largeish file (3GB zip file) from one folder to another WITHIN the VM I get sub 100MB/s speed. Is this normal or acceptable?

Since I am new to iSCSI, MPIO and shared storage I guess what I am wondering is, what sort of speeds should I see considering I am using SSD storage only with iSCSI/MPIO? I think MPIO is working as I can see traffic going across all four vmks when I run esxtop on the ESXi6 server. I'm just baffled why a benchmark tool show 300-450MB/s speeds in a VM but when I do a real world file copy it never goes above 100MB/s? I only have two VMs currently, vCenter and my test Windows VM so there is no load on this system. Oh, I also changed the default Round Robin IOPS value from 1000 to 1.

So what I'm trying to get at is, how do I know my underlying iSCSI storage (with MPIO and Round Robin) is performing correctly before putting 20 or more VMs on it? I can't help but feel something isn't quite right with the performance. Storage vMotion is painfully slow (about 80MB/s) but I think Starwind are bringing out a new version next week to resolve this.

In Starwind Virtual SAN I have set each disk device to have 4GB write back cache.

I'm hoping someone can say, haha you missed this or that. Please help I just "assumed" that having 4Gb/s of bandwidth across the four 1Gb NICs using iSCSI and MPIO would have given me at least 300MB/s speeds on an SSD backed datastore. I know theres overhead etc so I'll never get the full 4Gb/s (or 500MB/s) speeds but I assumed 300MB/s would be realistic?

 

[azev]

for ISCSI multipath to work you need to setup the VMK interfaces in different network and different vlan.
for example :

vmk1: 192.168.60.6
vmk2: 192.168.61.6
vmk3: 192.168.62.6
vmk4: 192.168.63.6

you have to do the same for the targets, so that you have 1 target per subnet/vlan.

 

[BSDguy]

Thanks for the reply!

I thought you *had* to use a aingle subnet for network port binding to work (and be compliant)?

 

[wildchild]

Dont forget to setup proper path settings and max ios per path.


Verstuurd vanaf mijn SM-G850F met Tapatalk

 

[BSDguy]

Dont forget to setup proper path settings and max ios per path.


Verstuurd vanaf mijn SM-G850F met Tapatalk

I set each datastore to use an IOPS value of 1 for RR, is this what you are referring to?

What do you mean by: setup proper path settings?

 

[azev]

single subnet is for multinic vmotion, but you have to put one interface in standby mode.

If you have a single vswitch with 4 nic attached it, you will have to setup each of your vmk to only use a single nic and put the other into unused.
Do this so that each vmk is using only one of the nic and you will be able to bind them to the software iscsi.

 

[wildchild]

I have made a pretty good write, including cisco ios settings , which will provide you with ready commands. Seek and you shall find

Verstuurd vanaf mijn SM-G850F met Tapatalk

 

[TuxDude]

for ISCSI multipath to work you need to setup the VMK interfaces in different network and different vlan.
for example :

vmk1: 192.168.60.6
vmk2: 192.168.61.6
vmk3: 192.168.62.6
vmk4: 192.168.63.6

you have to do the same for the targets, so that you have 1 target per subnet/vlan.

That is actually not required - it is possible to have iSCSI MPIO working properly with multiple vmk interfaces all within a single VLAN/subnet. It is at times required, when you are dealing with an iSCSI target that only has a single "floating" IP that can move around but you still want all of your vmk-nics's in active use - eg. an HP StoreVirtual array.

 

[BSDguy]

single subnet is for multinic vmotion, but you have to put one interface in standby mode.

If you have a single vswitch with 4 nic attached it, you will have to setup each of your vmk to only use a single nic and put the other into unused.
Do this so that each vmk is using only one of the nic and you will be able to bind them to the software iscsi.

I have 4 port groups on the virtual swotch used for iSCSI and each one only has one active NIC. The rest are unused.

 

[BSDguy]

I have made a pretty good write, including cisco ios settings , which will provide you with ready commands. Seek and you shall find

Verstuurd vanaf mijn SM-G850F met Tapatalk

What Cisco IOS settings are you referring to? I'm not using jumbo frames but is there anything else that needs to be changed on the switch side of things for iSCSI?

 

[BSDguy]

That is actually not required - it is possible to have iSCSI MPIO working properly with multiple vmk interfaces all within a single VLAN/subnet. It is at times required, when you are dealing with an iSCSI target that only has a single "floating" IP that can move around but you still want all of your vmk-nics's in active use - eg. an HP StoreVirtual array.

Everything I have read so far has been about using a single subnet so that iSCSI network port binding can work.

My iSCSI target has 4 IP addresses (one on each 1Gb NIC interface) so there is no "floating" IP.

So I'm a bit unsure where to go from here? Single subnet? Multiple subnets? Have I misconfigured something?

 

[whitey]

You caught all the big ones I saw or am aware of 'high level'. I know I have tested single subnet w/ port binding iSCSI setup and had no issues. Then again I'm 10G and a single conn is not what I am fighting with.

10G is cheapish these days...haha just sayin' SG500 10G may be right at home for you. You'll never look back. Also M$/Starwinds is NOT my forte' so take any of this with a grain of salt but your vSphere settings look accurate coming from a VCAP holder, I did turbo read the thread though :-D.

 

[BSDguy]

You caught all the big ones I saw or am aware of 'high level'. I know I have tested single subnet w/ port binding iSCSI setup and had no issues. Then again I'm 10G and a single conn is not what I am fighting with.

10G is cheapish these days...haha just sayin' SG500 10G may be right at home for you. You'll never look back. Also M$/Starwinds is NOT my forte' so take any of this with a grain of salt but your vSphere settings look accurate coming from a VCAP holder, I did turbo read the thread though :-D.

Thanks for the post!

This is the thing, I'm not sure if I have a problem?! Since this is my first shared storage setup I'm hoping someone can say: Yeah, looks good to me or sounds about right. Or, woooaaa you did what? ;-)

I do plan on going to 10gb next year but am waiting for the switches to become quieter. The Cisco SG300-28 is passively cooled so it's silent which means its great for my lab in the lounge!

I don't have a single connection, I have four 1Gb NICs in the ESXi server and the SAN (each server has a PCIe quad card). So in theory I should have up to 4Gb/s of bandwdith for iSCSI (ie: 500MB/s).

I just want to make sure my expectations are correct and that the underlying speed of the storage is ok before building my entire infrastructure on top of it. I can't help but think that if I move files around within a VM I should be getting 300-450MB/s but I'm not. I'm getting sub 120MB/s speed with a single VM on the one datastore.

 

[TuxDude]

This is the thing, I'm not sure if I have a problem?! Since this is my first shared storage setup I'm hoping someone can say: Yeah, looks good to me or sounds about right. Or, woooaaa you did what? ;-)

Well, there vSphere side of things sounds correct to me. The target side gives me one of those "woooaaa you did what? ;-)" kind of moments, but mostly because imho windows has no purpose except maybe the OS for a gaming machine, and even then preferably not. So I have no idea if your starwind setup is correct (why are you even using a 3rd party iSCSI target when its built into modern windows - or preferably switch to a linux or bsd based iscsi target), or if windows is even capable of working in a single-subnet configuration like that. It's quite possible that with multiple nics all in a single subnet that windows will decide to always send traffic out the wrong port or some other stupid thing.

I just want to make sure my expectations are correct and that the underlying speed of the storage is ok before building my entire infrastructure on top of it. I can't help but think that if I move files around within a VM I should be getting 300-450MB/s but I'm not. I'm getting sub 120MB/s speed with a single VM on the one datastore.

Your expectations are correct - using MPIO you will be able to use the full bandwidth of all of your NICs even with just a single iSCSI session from a single vSphere host - other network bonding technologies (eg. LACP) would limit you to the bandwidth of a single NIC in such a config. Of course that is assuming that whatever is on both ends can keep up with the transfer - you should be fine with SSDs being much faster than 1G nics, but spinning disks could end up limiting you to far less than 1gbps of throughput.

 

[BSDguy]

Well, there vSphere side of things sounds correct to me. The target side gives me one of those "woooaaa you did what? ;-)" kind of moments, but mostly because imho windows has no purpose except maybe the OS for a gaming machine, and even then preferably not. So I have no idea if your starwind setup is correct (why are you even using a 3rd party iSCSI target when its built into modern windows - or preferably switch to a linux or bsd based iscsi target), or if windows is even capable of working in a single-subnet configuration like that. It's quite possible that with multiple nics all in a single subnet that windows will decide to always send traffic out the wrong port or some other stupid thing.

Heh, well let me explain ;-) I considered FreeNAS but decided against it (and this is coming from a huge FreeBSD fan...I ran a FreeBSD email server for years) as I read quite a bit about fragmentation in ZFS and that you could only use half your storage space in ZFS when running an iSCSI target. The iSCSI target built into Windows is ummm rubbish to be blunt. Maybe Server 2016 is different but up until 2012R2 it doesn't support hardware acceleration (VAAI) for vSphere and I think it doesn't support caching. Maybe you're right and Windows is doing something stupid with the 4 NICs but I'm not sure. I have posted something on the Starwinds forum but no-one has commented yet ;-(

Your expectations are correct - using MPIO you will be able to use the full bandwidth of all of your NICs even with just a single iSCSI session from a single vSphere host - other network bonding technologies (eg. LACP) would limit you to the bandwidth of a single NIC in such a config. Of course that is assuming that whatever is on both ends can keep up with the transfer - you should be fine with SSDs being much faster than 1G nics, but spinning disks could end up limiting you to far less than 1gbps of throughput.

Thanks for confirming that because I was beginning to doubt myself. I think all the hardware can keep up with a 500MB/s transer (switch, NICs, servers, SSDs etc).

Having said all that, any ideas for what I should be using for my iSCSI shared storage with 2 ESXi 6 hosts? I chose Starwind because I have an NFR license, it supports VAAI, MPIO and Round Robin and does caching.

I'm open to suggestions. I just want to maximise the performance on the storage side of things as this will be the foundation of my lab. I don't mind being told I chose the wrong product for the job ;-)

 

[TuxDude]

Windows is ummm rubbish to be blunt.

Fixed that for you.

Having said all that, any ideas for what I should be using for my iSCSI shared storage with 2 ESXi 6 hosts? I chose Starwind because I have an NFR license, it supports VAAI, MPIO and Round Robin and does caching.

Off the top of my head, I know that the LIO-iSCSI target for linux also supports MPIO and VAAI (doing round-robin is a feature of the client, only requiring that the server can properly handle multiple active paths under MPIO), and will also allow you to try different transport protocols if you like (it will do iSCSI, iSER, FC, FCoE, SRP, etc. if you have the hardware for it). It's all built into RHEL7 (or CentOS 7 or other clones if you prefer) and not too hard to set up - if you can manage a bsd mail server I'm sure you can figure it out.

As for caching in whatever target you use - be careful with that. They should handle read-cache fine, but unless they're doing it with a small NVMe drive or something be VERY careful with write-cache. Assuming you aren't running nv-dimms anything caching writes is asking for data loss/corruption in the event of a crash/power-outage. Yes, you can configure many of them to do that if you are willing to accept the risk (including lio), but make sure you understand the risks before you go down that road.

 

[NetWise]

Coming in late, skimmed what I could, and some responses:

* The single subnet with 4 VMK's is more than fine. Various vendors are going to have different best practices, but it is doable. The main reason you don't do this is if the array doesn't know how to handle paths (get rid of it) or if you want 100% control over which subnet traverses which switches. If your switching does stacking or has a good inter-switch links, it doesn't matter.

* Windows Software ISCSI is a "you can use it", but has never been supported with vSphere, and has always been a little problematic. Totally not on the HCL, though the Microsoft StorSimple 8100 is.

* This gives me some heartburn:
After creating a test VM on one of the datastores I get about 450MB/s when I run CrystalMark and about 300-350MB/s when running ATTO but when I copy a largeish file (3GB zip file) from one folder to another WITHIN the VM I get sub 100MB/s speed. Is this normal or acceptable?
Synthetic benchmarks are multi-threaded with high queue depth. Windows file copy is NOT. You will NOT get published speeds at maximum theoretical test only, when doing a single file, file copy. The file has to speed up, start using bigger packets, read from one drive, fill memory, etc. A much better test of this would be to fire off a dozen copies and watch all 4 of your NIC's for total usage, and measure your aggregate performance.

* Its not clear if Windows 2016 ISCSI/MPIO is installed as WELL as StarWind. Definitely use StarWind, if you have it.

* I'm surprised the Cisco SG300-28 is working for ISCSI at all. It may be fanless and low powered, but with small packet buffers, it's probably not going to deal with ISCSI loads very well. More often that not, the ISCSI issues I've seen aren't mis-configuration, but mis-utilizing a low-end or access switch, in an ISCSI/high-packet/high-buffer role. I suspect under heavy load, you'll see this fall down on you.

Overall, the big takeway is stop trying to use a single threaded file copy to diagnose MPIO. It's worthless. If your benchmarks are showing it's pushing 4x1GbE, then it's nearly certainly fine.

 

[BSDguy]

Off the top of my head, I know that the LIO-iSCSI target for linux also supports MPIO and VAAI (doing round-robin is a feature of the client, only requiring that the server can properly handle multiple active paths under MPIO), and will also allow you to try different transport protocols if you like (it will do iSCSI, iSER, FC, FCoE, SRP, etc. if you have the hardware for it). It's all built into RHEL7 (or CentOS 7 or other clones if you prefer) and not too hard to set up - if you can manage a bsd mail server I'm sure you can figure it out.

As for caching in whatever target you use - be careful with that. They should handle read-cache fine, but unless they're doing it with a small NVMe drive or something be VERY careful with write-cache. Assuming you aren't running nv-dimms anything caching writes is asking for data loss/corruption in the event of a crash/power-outage. Yes, you can configure many of them to do that if you are willing to accept the risk (including lio), but make sure you understand the risks before you go down that road.

Thanks for the post. I have never come across LIO-iSCSI so I'll look into it. I like the fact that it supports FC...nice! I was quite keen on picking up some 4G FC kit off eBay before going the iSCSI route. I was going to have dual port FC cards in the ESXi hosts and then a quad port FC card in the SAN.

The caching I was referring to was in RAM. I do have a UPS for my lab.

* The single subnet with 4 VMK's is more than fine. Various vendors are going to have different best practices, but it is doable. The main reason you don't do this is if the array doesn't know how to handle paths (get rid of it) or if you want 100% control over which subnet traverses which switches. If your switching does stacking or has a good inter-switch links, it doesn't matter.

Excellent, so I got this right!

* Windows Software ISCSI is a "you can use it", but has never been supported with vSphere, and has always been a little problematic. Totally not on the HCL, though the Microsoft StorSimple 8100 is.

I want to avoid using the MS iSCSI target at all costs.

* This gives me some heartburn:
After creating a test VM on one of the datastores I get about 450MB/s when I run CrystalMark and about 300-350MB/s when running ATTO but when I copy a largeish file (3GB zip file) from one folder to another WITHIN the VM I get sub 100MB/s speed. Is this normal or acceptable?
Synthetic benchmarks are multi-threaded with high queue depth. Windows file copy is NOT. You will NOT get published speeds at maximum theoretical test only, when doing a single file, file copy. The file has to speed up, start using bigger packets, read from one drive, fill memory, etc. A much better test of this would be to fire off a dozen copies and watch all 4 of your NIC's for total usage, and measure your aggregate performance.

So I tried to do 9 file copies all at the same time (about 3GB each in size) using the Windows file copy and was averaging about 80MB/s. esxtop across all four NICs was showing about 600-800mb/s (which is poor considering I have 4gb of bandwidth!). This just seems bad...

* Its not clear if Windows 2016 ISCSI/MPIO is installed as WELL as StarWind. Definitely use StarWind, if you have it.

You install the MPIO role in Windows and then install Starwind Virtual SAN. The MS iSCSI target is NOT installed. After installing the MPIO role I ran the MPIO utilty to enable MPIO for iSCSI devices.

* I'm surprised the Cisco SG300-28 is working for ISCSI at all. It may be fanless and low powered, but with small packet buffers, it's probably not going to deal with ISCSI loads very well. More often that not, the ISCSI issues I've seen aren't mis-configuration, but mis-utilizing a low-end or access switch, in an ISCSI/high-packet/high-buffer role. I suspect under heavy load, you'll see this fall down on you.

Remember, I am running a home lab so it has to be practical. I hear what you saying but surely this switch can handle the load of two ESXi servers and one SAN with me being the only person using it?? I spent ages looking for a switch I could use at home and read good reviews of other people using it in a lab setting so you've got me worried about this now! I know there are better options but they are just too noisy for home use. I just find it hard to believe that a Cisco switch can't handle 500MB/s of throughput? Have I screwed up on the switch I bought? ;-(

Appreciate the time you took to comment, thank you!

 

[NetWise]

So I tried to do 9 file copies all at the same time (about 3GB each in size) using the Windows file copy and was averaging about 80MB/s. esxtop across all four NICs was showing about 600-800mb/s (which is poor considering I have 4gb of bandwidth!). This just seems bad...

Are you copying from one drive on one datastore to another? Are you using PVSCSI for your disk controller or LSI SAS?

You install the MPIO role in Windows and then install Starwind Virtual SAN. The MS iSCSI target is NOT installed. After installing the MPIO role I ran the MPIO utilty to enable MPIO for iSCSI devices.

That rings a bell now. Just wanted to ensure you weren't trying both.

Remember, I am running a home lab so it has to be practical. I hear what you saying but surely this switch can handle the load of two ESXi servers and one SAN with me being the only person using it?? I spent ages looking for a switch I could use at home and read good reviews of other people using it in a lab setting so you've got me worried about this now! I know there are better options but they are just too noisy for home use. I just find it hard to believe that a Cisco switch can't handle 500MB/s of throughput? Have I screwed up on the switch I bought? ;-(

"it has to be practical". You can't lab a full size pickup with a wheelbarrow Yes, noise and heat are a concern, but that practicality goes two ways.

As for "hard to believe that a Cisco switch" can't handle something - there is a reason they have different product lines. A Core switch doesn't do the same as an Access switch. The SG/SF line of switches is really not "Cisco" as much as "Now Cisco, made by Linksys". It doesn't run IOS, doesn't have a console port, etc. It's a decent access switch. However, 500MB/sec of throughput that can be lossy and arrive in any order, and comes in bursts is not quite the same as the access patterns on ISCSI.

All in all, I think what you're seeing is expected. If your benchmarks run okay, and don't stumble or fall, the switches and ISCSI configuration are likely just fine. If they weren't, the multi-threaded, best case scenario, we're going to pound on it as hard as possible testing wouldn't be great.

I just ran a test on our 10GbE Nimble array in my VDI session copying a 500MB file from one folder to another - and get about 80MB/sec. The point isn't how fast ONE can go, it's the fact that 100 users can get this speed all day long that counts. You probably need to stop fretting so much, and actually go and USE it and see how it behaves under load from multiple VM's.

 

[TuxDude]

The caching I was referring to was in RAM. I do have a UPS for my lab.

A UPS does not protect you against software crashing - if you are using windows as a target I would expect that to happen on a regular basis. But its your lab / your data - you decide how much risk you are willing to accept in exchange for some extra performance.

You install the MPIO role in Windows and then install Starwind Virtual SAN. The MS iSCSI target is NOT installed. After installing the MPIO role I ran the MPIO utilty to enable MPIO for iSCSI devices.

Is that a requirement of Starwind or something? I wouldn't expect you to need the MPIO role installed at all - that is really the MPIO client and shouldn't be involved at all, unless the windows box is also connecting to the iSCSI target for some reason.