iSCSI + MPIO + ESXi6

[NetWise]

Absolutely everything @TuxDude said!!!


Sent from my iPhone using Tapatalk

 

[BSDguy]

Hmmm infiniband looks like an interesting option with amazing bandwidth but after reading up a bit on it it sounds like it could get complicated with subnet managers?

So I'm back to looking at 10Gb NICs:

One each of these for the ESXi hosts:

Mellanox MNPH28B-XTC ConnectX Network Adapter 10Gbe PCI-E x8 NIC | eBay

Two of these in the SAN:

Mellanox MNPH29C-XTR ConnectX-2 2x Port 10GbE SFP+ Network Adapter High Profile | eBay

With 4 of these cables:

Dell 10GbE 2M SFP+ DAC Cable 1JTTN 01JTTN | eBay

Will these cables work? There seem to be so many options for cables.

I was having a look on the Mellanox drivers website:

http://www.mellanox.com/page/software_overview_eth

http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_driver

ESXi 6 is supported but Windows Server 2016 has beta drivers...this could be a problem?

My second quad 1Gb NIC arrived today so that will be a cool test to do when I have both ESXi hosts hooked up to the iSCSI storage!

 

[TuxDude]

Hmmm infiniband looks like an interesting option with amazing bandwidth but after reading up a bit on it it sounds like it could get complicated with subnet managers?

And even more complicated when you start to think about what protocol(s) to run on top of infiniband - whether you go IPoIB and use standard iSCSI on that, or let the storage say on more native IB and use SRP or something. I believe there's also a FCoIB standard out there though I might be wrong. I have a few boxes with onboard IB adapters at home, but don't have a switch so my IB experience is near-zero.

Will these cables work? There seem to be so many options for cables.

Not sure on that answer - but that is an important question. SFP+ compatibility (cables or transceivers) can be very picky in some products.

ESXi 6 is supported but Windows Server 2016 has beta drivers...this could be a problem?

Considering how new Server 2016 is, if they've already released a beta driver I would consider that a good thing. That indicates to me that they at least plan to put out a driver for the new OS even though those cards are very old.

 

[BSDguy]

Will these cables work? There seem to be so many options for cables.

Not sure on that answer - but that is an important question. SFP+ compatibility (cables or transceivers) can be very picky in some products.

Yeah, so I have read. I've never used or worked with SFP+ so wanna make sure before purchasing.

ESXi 6 is supported but Windows Server 2016 has beta drivers...this could be a problem?

Considering how new Server 2016 is, if they've already released a beta driver I would consider that a good thing. That indicates to me that they at least plan to put out a driver for the new OS even though those cards are very old.

I don't know how long it will be before Mellanox will release an RTM driver but it may be worth waiting. Can't say I want to run my iSCSI storage with a beta driver!

I didn't realise these cards were "old", is there a newer version worth considering rather, albeit at a higher cost I assume?

 

[Marsh]

Mellanox already have a NON-BETA driver for Windows 2016, you have to ignore the beta page, go directly to the download page.

http://www.mellanox.com/page/products_dyn?product_family=32&mtag=windows_driver

The cards that you linked is not IB card, it is EN ethernet card. You will not be using IB protocol.

With the latest firmware, even the older ConnectX-2 cards would do RDMA over ROCE on top of Windows 2016.
I setup a low power 2 nodes Windows 2016 HA Storage Space Direct cluster last week with a pair of the older Mellanox ConnectX-2 EN card as cluster interconnect network. The cheap Mellanox cards works wonder, last year, the dual port cards were going for around $25, cheaper than a quad port 1G network card.

What happen to the HP card that you linked yesterday?
If you paying more than $45 per cards, then look at the SUN Oracle CX-3 cards?
The SUN Oracle CX-3 card is VPI cards, you will have choice between IB or EN per port. Then you could run IB on one port, EN on the other port.

 

[BSDguy]

Mellanox already have a NON-BETA driver for Windows 2016, you have to ignore the beta page, go directly to the download page.

Thank you, I didn't see this. I just saw beta driver and assumed they are still working on it.

What happen to the HP card that you linked yesterday?
If you paying more than $45 per cards, then look at the SUN Oracle CX-3 cards?
The SUN Oracle CX-3 card is VPI cards, you will have choice between IB or EN per port. Then you could run IB on one port, EN on the other port.

Here they are:

HP Mellanox ConnectX 10Gb PCI-e G2 Dual Port HCA 518001-001 516937-B21 | eBay

HP 516937-B21 Dual-port 10Gb Ethernet Card 518001-001 / Mellanox ConnectX-2 SFP+ | eBay

I have added so many 10Gb cards to my watchlist in eBay so it's getting tricky keeping track of all these options! I doubt I'll get a setup for $25 per card considering I need to get them shipped over here.

Ok, SUN Oracle CX-3 cards, tell me more ;-) Firstly, what is VPI? And IB or EN per port, sorry what is this? Infiniband and Ethernet per port? Sorry if I sound stupid. I searched for "SUN Oracle CX-3" on eBay but came back empty handed, do you have a specific model number so I can research this?

 

[BSDguy]

So after all the fiddling with ESXi and storage I rebuilt my ESXi host today and installed my second quad 1Gb NIC. I've got an odd issue I didn't have before, I have iSCSI configured on a single subnet as before with port binding but for some reason it looks like all traffic is being sent through one vmk by default. If I look at the routing table I can see that all traffic for the iSCSI subnet going down vmk1.

Considering I have 4 NICs in the ESXi host directly connected to the SAN, how can I create a static route on the ESXi host that says:

If traffic is to go to 192.168.60.6 then use vmk1
If traffic is to go to 192.168.60.7 then use vmk2
etc..

If I ping 192.168.60.7 from the ESXi server it times out but if I tell it to use a specfic vmk the ping works:

vmkping -d -s 8972 192.168.60.7-I vmk3

Also, ESXi boot times are terrible while iSCSI times out.

Do I need static routes or is it better to just use multiple subnets with no port binding?

 

[Marsh]

@BSDguy

I'll let Wikipedia explain, they do a much better job than me.
RDMA over Converged Ethernet - Wikipedia
RDMA over Converged Ethernet - Wikipedia
VPI Virtual Protocol Interconnect
http://www.mellanox.com/pdf/prod_architecture/Virtual_Protocol_Interconnect_VPI.pdf
Ebay search :Oracle 7046442 , 40GBe for less than $100 USD.
Change a config file , flash the firmware could get you to 56GBe. The card also works in IB mode.
Sun Oracle 40Gb/Sec Dual Port QDR Infiniband PCIe HCA Adapter M3 Card 7046442 | eBay

One more thing. I installed 2 of the Mellanox CX-2 cards, load up Windows 2016, installed Mellanox dirver. No tweaking, just run NTttcp , I was getting line speed 10Gb/s out of the box.

 

[BSDguy]

@BSDguy

I'll let Wikipedia explain, they do a much better job than me.
RDMA over Converged Ethernet - Wikipedia
RDMA over Converged Ethernet - Wikipedia
VPI Virtual Protocol Interconnect
http://www.mellanox.com/pdf/prod_architecture/Virtual_Protocol_Interconnect_VPI.pdf
Ebay search :Oracle 7046442 , 40GBe for less than $100 USD.
Change a config file , flash the firmware could get you to 56GBe. The card also works in IB mode.
Sun Oracle 40Gb/Sec Dual Port QDR Infiniband PCIe HCA Adapter M3 Card 7046442 | eBay

One more thing. I installed 2 of the Mellanox CX-2 cards, load up Windows 2016, installed Mellanox dirver. No tweaking, just run NTttcp , I was getting line speed 10Gb/s out of the box.

Thanks for the post, appreciate it. I'm a bit confused by the Sun Oracle 40Gb/Sec cards. These are infiniband cards right? So after reading the PDF/links you posted, does this mean they can operate in Ethernet mode OR infiniband mode (where a subnet manager is needed)?

I'm obvioulsy missing something but if I bought 4 of these cards (one for each ESXi host and two for the SAN) could I direct connect these with the correct cables and get 40Gb/s "Ethernet" speeds? What cables would I need?

On eBay I can buy each Sun Oracle 40Gb/S card for £45 shipped (no cables though) so the thought of these is tempting but if it's too complicated (ie: subnet manager needed?) then I think 10Gb SFP+ may be a better option? Ironically the 10Gb SFP+ option will cost more than the 40Gb Sun Oracle card, go figure.

Thats some good speeds you got with the Mellanox cards!

Setup some VMs this weekend on the iSCSI storage and so far so good. I have 4 VMs setup and the performance seems fine so far. The issues I was having earlier had to do with using dynamic discovery on the software iSCSI initiator on the ESXi host. I changed that to static discovery and things went much smoother from there. I've got jumbo frames enabled for the storage and am now using virtual distributed switches which are awesome.

 

[BSDguy]

Hi All

Been a while since my last post but I have been spending some serious time using my new lab! Am running 25 VMs so far and only using about 88GB of RAM out of the 256GB (although 5 of the VMs are temporary until I migrate my email to the new VMs).

Anyway things have been running great using iSCSI and have been using a quad 1Gb NIC in each host running crossover connections to the SAN (whic has 8 x 1Gb ports in it).

Having said all that I want to remove one of the quad 1Gb NICs in the SAN as I want to plug in a M2 adapter so I can use an M2 NVMe PCIe SSD drive for a datastore in vSphere using iSCSI. The problem though is my SAN only has two PCIe slots:

PCI Express 2.0 x4 (Max speed: 2000MB/s)
PCI Express 3.0 x8 (Max speed: 8000MB/s)

So I was thinking of replacing the dual quad 1Gb NICs in the SAN with a single dual port card (10Gb Ethernet or 40Gb IB) and then using the M2 adapter card in the other slot. Obviously I would have to replace the quad 1Gb NICs in my two ESXi hosts as well.

The problem is I just can't make up my mind whether I should go with 10Gb Ethernet NICs or 40Gb InfiniBand cards? I've done some reading on InfiniBand but I think I have overwhelemed myself with all the terms that get thrown around like RDMA, iSER, IPoIB, Subnet Manager etc etc.

I basically want to direct connect (no switch for now) my two ESXi hosts to the SAN and want a minimum of 10Gb per host to the SAN. If I go with Infiniband can I do this successfully without a switch using ESXi 6 and Windows Server 2016 with Starwind Virtual SAN? Can this combination work and will RDMA work for maximum performance? If that is not an option then does than mean I have to use IPoIB with iSCSI? Is this slow(er)? Does iSCSI work over RDMA?

I really want to take my storage network to the next level as I am running 6 SSDs in the SAN and want to add one or two M2 NVMe PCIe SSD drives so want to install a really good network for storage. I plan on adding a third ESXi host but this is 6 - 9 months into the future so would need to consider a switch (10Gb Ethernet or IB) at that stage.

Please help me decide here as I am battling to make a decision...problably due to my limited understanding of IB and I haven't used 10Gb yet.

Thank you.

 

[OBasel]

just passed my VCP6-DCV exam and as a result of that I purchased two of the following servers:

Congratulations

 

[BSDguy]

Congratulations

Thanks, I worked seriously hard to pass it considering the long hours I have been working! Thanks again!

 

[BSDguy]

I'm considering the Ubiquiti EdgeSwitch ES-16-XG for use in my virtualisation environment. It will be used for iSCSI only storage traffic with 2 ESXi hosts and my SAN (with a third host added later this year).

Considering the issues I had with the Cisco switches performance with iSCSI (mentioned earlier in this thread), has anyone used the Ubiquiti EdgeSwitch ES-16-XG switch for iSCSI traffic? Does it perform well?

 

[wildchild]

I'm considering the Ubiquiti EdgeSwitch ES-16-XG for use in my virtualisation environment. It will be used for iSCSI only storage traffic with 2 ESXi hosts and my SAN (with a third host added later this year).

Considering the issues I had with the Cisco switches performance with iSCSI (mentioned earlier in this thread), has anyone used the Ubiquiti EdgeSwitch ES-16-XG switch for iSCSI traffic? Does it perform well?

Yes and yes
It is a bit picky when it comes to gbics or dacs though

 

[BSDguy]

Yes and yes
It is a bit picky when it comes to gbics or dacs though

Ok, so yes you've used it for iSCSI and yes it performs well...great!

What kind of speeds/performance/IOPS do you get when using this switch for your storage with iSCSI?

Any recommendations for the 10Gb SFP+ NICs and cables I should get?

Been looking at Intel and Mellanox but the options are mind boggling for someone like me who has yet to use 10Gb yet!

 

[wildchild]

Theres a nice topic on the unifi-16-xg.
I believe i posted my test results there.
The unifi and edgeswitch share hardware bases so should be the same test result.

I have some cisco 2m dacs that work fine, but i think i got lucky.
I believe for now the safest bet would be the ubiquiti gbics.
Check the compatibilty topic for other options

 

[BSDguy]

Thanks but after reading the whole thread did I miss your post about stress testing this switch with iSCSI traffic? I did see:

On to putting iscsi hurt on my switches

Was there some further testing you did?

 

[BSDguy]

So after much research I've decided to put Infiniband to the side (for now!). I was looking ahead to when I grow my cluster to 3 or more nodes and didn't want to buy a (noisy) Infiniband switch. I'm still confused my all the buzzwords with Infiniband and trying to figure out which OS supports RDMA, iSER etc etc.

To upgrade my storage network I decided to use 10Gb with SFP+ interfaces and chose the following hardware:

• Mellanox ConnectX 2 10GbE MNPA19-XTR Low Profile (for the ESXi hosts)
• Mellanox ConnectX 10GbE MNPH29C-XTR High Profile (for the SAN)
  
• Two Cisco SFP-H10GB-CU3M Three 3 Meter 10GbE Twinax Cable (for direct connect between hosts and SAN)

This will be my first 10Gb setup at home and my first ever use of 10Gb so I'm excited

I'll be replacing my 4 x quad 1Gb NICs (16 ports in total!) used for iSCSI storage with just 4 x 10Gb ports...amazing!

The plan is to later expand this with a 10Gb switch like the Ubiquiti EdgeSwitch ES-16-XG but only if the firmware matures and fixes issues with 10Gb and the Supermicro 5028D-TN4T servers I have...

 

[wildchild]

Thanks but after reading the whole thread did I miss your post about stress testing this switch with iSCSI traffic? I did see:

On to putting iscsi hurt on my switches

Was there some further testing you did?

ah sorry , i thought i had posted them here...
Sorry for later response.. been working on a silly vmware template issue today ( see other topic)

[ ID] Interval           Transfer     Bandwidth       Retr  Cwnd
[  4]   0.00-1.00   sec  1.09 GBytes  9.40 Gbits/sec  564    614 KBytes   
[  4]   1.00-2.00   sec  1.09 GBytes  9.38 Gbits/sec  507    624 KBytes   
[  4]   2.00-3.00   sec  1.09 GBytes  9.38 Gbits/sec  507    607 KBytes   
[  4]   3.00-4.00   sec  1.09 GBytes  9.39 Gbits/sec  507    601 KBytes   
[  4]   4.00-5.00   sec  1.09 GBytes  9.38 Gbits/sec  508    594 KBytes   
[  4]   5.00-6.00   sec  1.09 GBytes  9.39 Gbits/sec  504    445 KBytes   
[  4]   6.00-7.00   sec  1.09 GBytes  9.38 Gbits/sec  505    472 KBytes   
[  4]   7.00-8.00   sec  1.09 GBytes  9.38 Gbits/sec  461    590 KBytes   
[  4]   8.00-9.00   sec  1.09 GBytes  9.38 Gbits/sec  506    605 KBytes   
[  4]   9.00-10.00  sec  1.09 GBytes  9.38 Gbits/sec  507    626 KBytes   
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec  5076             sender
[  4]   0.00-10.00  sec  10.9 GBytes  9.38 Gbits/sec                  receiver

so this is using intel x520DA-2's single path
one connected to VMWare 6 u2 -> cisco 2m DAC -> unfi -> cisco 2m DAC ->Omnios

 

[BSDguy]

Thanks very much, thats really helpful!