ZFS Advice for new Setup

[humbleThC]

RE: 4096 Byte MTU max of the 4036.
Have you compared IOPs and latency when doing transfers to & from high performing SSD or NVME pools and the affects the MTU limitation may have vs direct connect with the higher value?

I havent yet.

Few things I was talking with a friend about potentially testing 'for fun'

Tear down one of my ESX servers, and install Linux on it directly.
Then i could test Linux>Linux and see how that compares to the Linux>ESX SRP benchmarks.
Then rebuild Linux back to OmniOS, retest against Linux & ESX initiators.
Then rebuild to Solaris, retest against Linux & ESX.
Finally go back to Windows 2016 one last time, retest against Linux & ESX.

I'll very likely not end up with any Linux as a initiator myself, but if theory proves to be correct on the OFED, and the VMWare drivers being weak/slow/untunable, then i'll find it.
- And if that is the case, i'll keep the Linux initiator host around to help finish my 'NAS back-end bake-off'.
- Then, i can finalize on my NAS solution, and put Linux host back to ESX6 cluster.

Somewhere in between there, i need to test the direct-connect cable between two ports of Linux>Linux and Linux>ESX and play with just the MTU 4096 -vs- 65520 on CX2 hardware.
(which also means swapping back'n'forth between hardware & software subnet mgrs, should be fun!)​

 

[humbleThC]

Unless MPOGR has done most of those tests and can save me the leg work with 'the gist'

 

[T_Minus]

@humbleThC & @mpogr Sent you guys a PM, I think I may be able to help speed things along.

 

[mpogr]

Sorry guys, can't help much in the way of testing. Very busy at work, and my home system is in a stable working state, so don't want to mess around much with it...

 

[humbleThC]

I'm much closer to being able to 'publish' a guide specific to CentOS/RedHat (latest) + Mellanox OFED (latest) + SCST (latest) w/ SRP Targets w/ ZoL Volumes for virtual SRP disks to ESX6 (working). Just need to have mpogr proof check it, and I might run it through both OS variants tomorrow to look for missing steps / additional info I could provide.

 

[humbleThC]

Either one of you gonna publish a definitive 'using SRP storage in vSphere' guide? Not talking a 'hand holding' soup-to-nuts work instruction but maybe a primer w/ golden nuggets of knowledge/lessons learned/config files/solid links to articles that cover each component of config would sure be nice. I have a config almost there (vSphere w/ 1.8.2.5 SRP vib's, CentOS7 ZoL latest from openzfs, OFED latest, missing SCST) but need to bone up on SCST apparently. Oh and I got sidetracked w/ another project.

I see a scst.conf file attached so there's part of the puzzle, any other critical config files? Remind me @humbleThC what IB switch are you using? Looks like CX2 cards for sure.

Take a peek @ NAS - VMware SRP Guide - Wiki³

I need to fix the section on IB Partitiosn (step 8) because i vastly misunderstood that step (obvious when you reed it) i.e. I have a switch, so the partitions.conf file for me on the hosts is unused/not-required.

Besides that i'm getting closer

 

[whitey]

Aww hell yea!!! Thanks, just freed up some HW to take on a test run of this soon. Will proof it and let ya know of hangups or if it's silky smooth.

 

[humbleThC]

Cleaned it up a little, particularly the IB Partitions section (step 8).

 

[Gabriel Mateiciuc]

Hi, nice effort to put the jigsaw pieces together.
I could add, form my experiences that :
1. When you'll try some CX3's you'll notice quite some speed bump.
2. On the VMware side, things are less nice :
- you go with 1.8.2.5 - SRP and no IPoIB (so no VM networking there) - that has the advantage of working on the 6.5
- you go with 1.8.3.0 beta - and you get all the bells and whistles that IB has (bar SR-IOV) but you'll have to stick with 6.0
Notice no ETH options, because, yes, I'm one of those that has a IB only SAN fabric (no ETH) in my lab.
Right now I'm in the process of getting my main storage from Omnios to Ubuntu 16.04 - a bit stuck at building the ib_srpt module. Probably being bold and going with the 4.0 MOFED doesn't help me in that direction.
For another less busy period, I'll restart some work on building the drivers for VMware.

 

[mpogr]

Hi, nice effort to put the jigsaw pieces together.
I could add, form my experiences that :
1. When you'll try some CX3's you'll notice quite some speed bump.
2. On the VMware side, things are less nice :
- you go with 1.8.2.5 - SRP and no IPoIB (so no VM networking there) - that has the advantage of working on the 6.5
- you go with 1.8.3.0 beta - and you get all the bells and whistles that IB has (bar SR-IOV) but you'll have to stick with 6.0
Notice no ETH options, because, yes, I'm one of those that has a IB only SAN fabric (no ETH) in my lab.
Right now I'm in the process of getting my main storage from Omnios to Ubuntu 16.04 - a bit stuck at building the ib_srpt module. Probably being bold and going with the 4.0 MOFED doesn't help me in that direction.
For another less busy period, I'll restart some work on building the drivers for VMware.

Can you, please, share how you succeeded to make the 1.8.2.5 drivers work under ESXi 6.5? For the life of me, I couldn't make them work no matter what I tried...

Sent from my SM-G920I using Tapatalk

 

[mpogr]

Hi, nice effort to put the jigsaw pieces together.
I could add, form my experiences that :
1. When you'll try some CX3's you'll notice quite some speed bump.
2. On the VMware side, things are less nice :
- you go with 1.8.2.5 - SRP and no IPoIB (so no VM networking there) - that has the advantage of working on the 6.5
- you go with 1.8.3.0 beta - and you get all the bells and whistles that IB has (bar SR-IOV) but you'll have to stick with 6.0
Notice no ETH options, because, yes, I'm one of those that has a IB only SAN fabric (no ETH) in my lab.
Right now I'm in the process of getting my main storage from Omnios to Ubuntu 16.04 - a bit stuck at building the ib_srpt module. Probably being bold and going with the 4.0 MOFED doesn't help me in that direction.
For another less busy period, I'll restart some work on building the drivers for VMware.

Re building SCST with MOFED 4.0, you have to use the trunk, the 3.2.x branch no longer compiles with it.

Sent from my SM-G920I using Tapatalk

 

[Gabriel Mateiciuc]

Hello,
For esxi6.5 - I've used this :
MLNX-OFED-ESX-1.8.2.5-10EM-600.0.0.2494585
Standard "IB fabric" procedure - disable/remove the nmlx drivers (inbox) then install 1.8.2.5
I only set the params for mlx4_core for mtu and msix.
It seems to work but right now I'm fiddling with SCST. I've managed to get it to build (the tricky part was the ib_srpt driver - which has a nice redhat friendly Makefile - especially the MOFED paths/detection) but on a C7.3 with ml 4.9 kernel with MOFED 4.0
Does anyone has hit this problem with esxi when marking a lun as not ssd in vsphere makes it go away and never return ?

 

[mpogr]

Hello,
For esxi6.5 - I've used this :
MLNX-OFED-ESX-1.8.2.5-10EM-600.0.0.2494585
Standard "IB fabric" procedure - disable/remove the nmlx drivers (inbox) then install 1.8.2.5
I only set the params for mlx4_core for mtu and msix.

I have tried this about a month ago for quite significant amount of time with no luck at all. Haven't got the SRP adapters showing up in ESXi. The log inspection revealed the drivers failed loading. This has been confirmed by Mellanox themselves: vmlkilunx based drivers are no longer supported in ESXi 6.5, that's the reason this won't work. Are you sure you actually got this working?

It seems to work but right now I'm fiddling with SCST. I've managed to get it to build (the tricky part was the ib_srpt driver - which has a nice redhat friendly Makefile - especially the MOFED paths/detection) but on a C7.3 with ml 4.9 kernel with MOFED 4.0

I've refreshed my SCST setup on CentOS 7.3 after updating to the latest kernel and MOFED 4.0. I had to move to the SCST trunk in order to get it compile. The 3.2.x branch I've been using with MOFED 3.4 is no longer compiling.

Does anyone has hit this problem with esxi when marking a lun as not ssd in vsphere makes it go away and never return ?

This is probably the symptom of 1.8.2.5 drivers actually not working on ESXi 6.5, as they should. Please, read here for further info on the subject.

 

[Gabriel Mateiciuc]

Ok, so far, to sum it up :

C7 - kernel ml 4.9.x + mofed 4.0 + scst trunk (3.3.x) on top of zfs 0.7.0-rc3 - functional
Gentoo (yeah, i got fed up and went back to the roots - gentoo-sources + scst trunk + ofed (somehow in-tree) + zfs latest - functional.

Not ubuntu per se - but the entire clusterfuck brought by the kmod/dkms - which is cool when it works, but a pain to work around when it doesn't

Esxi :
tha stable 6.0 - this is actually weird as 1.8.3.0 seems to start acting up at times.
uname -a :
VMkernel ... 6.0.0 #1 SMP Release build-4600944 Nov 3 2016 22:17:36 x86_64 x86_64 x86_64 ESXi

esxcli software vib list | grep -i mel
nmst 3.8.0.56-1OEM.600.0.0.2295424 MEL PartnerSupported 2017-01-19
mft 3.7.1.3-0 Mellanox PartnerSupported 2017-01-19
net-ib-addr 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
net-ib-cm 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
net-ib-core 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
net-ib-ipoib 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
net-ib-mad 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
net-ib-sa 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
net-ib-umad 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
net-mlx4-core 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
net-mlx4-ib 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
net-mst 3.7.1.3-1OEM.550.0.0.1331820 Mellanox PartnerSupported 2017-01-19
net-rdma-cm 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
scsi-ib-iser 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
scsi-ib-srp 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18

esxcfg-scsidevs -a :

vmhba_mlx4_0.1.1mlx4_core link-n/a gsan.8100000000000000xxxxxxxxxxxxx (0000:03:00.0) Mellanox Technologies MT27500 Family [ConnectX-3]
vmhba196608ib_iser online iqn...:xx:xx Mellanox iSCSI over RDMA (iSER) Adapter

esxcfg-nics -l :

vmnic_ib0 0000:03:00.0 ib_ipoib Up 56252Mbps Full xx:.. 4092 Mellanox Technologies MT27500 Family [ConnectX-3]

Esxi - 6.5 - mind you, the 1.8.2.5 is some new release - 15.03.2016 if I'm not mistaking - that has the key element - built against the new vmkapi.

uname -a :
VMkernel .... 6.5.0 #1 SMP Release build-4887370 Jan 5 2017 19:17:59 x86_64 x86_64 x86_64 ESXi

esxcli software vib list | grep -i mel
net-ib-addr 1.9.10.6-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
net-ib-cm 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
net-ib-core 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
net-ib-ipoib 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
net-ib-mad 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
net-ib-sa 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
net-ib-umad 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
net-memtrack 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
net-mlx-compat 2.4.0.0-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
net-mlx4-core 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
net-mlx4-en 1.9.10.6-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
net-mlx4-ib 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
net-rdma-cm 1.9.10.6-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
scsi-ib-iser 1.9.10.6-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
scsi-ib-srp 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04

looks like I was lazy to remove some inbox/leftovers - these other versions are not used anyway, as we all know.

esxcfg-scsidevs -a :
...
vmhba33 mlx4_core link-n/a gsan.810000000000000010e0000xxxxxxx (0000:04:00.0) Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s]
vmhba34 mlx4_core link-n/a gsan.810000000000000010e0000xxxxxxx (0000:04:00.0) Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s]

esxcfg-nics -l :
...
vmnic1000202 0000:04:00.0 ib_ipoib Up 40000Mbps Full xx:.. 4092 Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s]
vmnic2 0000:04:00.0 ib_ipoib Up 40000Mbps Full xx:.. 4092 Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s]

So far the performance seems to be there - I don't have some powerful machines as targets to test as I'm moving data around at the moment.
Besides that, I've been running Omnios until recently on my main storage (Not working with CX3 or nvme above 1.1 made me try Linux for that too) so a compare would be a bit stretched.

Let me know what else I could help with.

@mpogr - It's funny, I've just realized you're the one with the thread on the mlnx community site . I've stumbled upon it a lot of times and, for the record, I fully agree - No point in restating the facts. Sometimes they seem idiots - you get ipoib and srp but no iser ... which you say as a company is the future ... why ?!
The situation described there is exactly what prompted me to look into building esxi driver for IB support. Let's say I haven't gotten too far but it's time (and nerve) consuming.

 

[mpogr]

Very interesting re 1.8.2.5 drivers on ESXi 6.5. Looks like you've been using the updated build (4887370), I tried only the original one a couple of months back and couldn't get it to work at all. Maybe it's time to give it another try. Pity it's such a painful exercise to go back to 6.0 if things go astray with 6.5...

As for the discussion on the MLNX Communities forums, I think it's pretty clear by now: people like us (using MLNX hardware to build high-speed custom storage solutions for ESXi on the cheap) are not of any interest to MLNX as a company. They know nobody in the enterprise will bother with this and companies will keep spending zillions on specialised solutions from EMC/HPE/NetApp. Their target market is HPC, so they focus on supporting Linux as a platform. That's also why, when it comes to ESXi, they spend most efforts on enabling SR-IOV to serve Linux guests.

Considering this, it would be very nice of you if you could indeed undertake on developing the ESXi drivers with IB support. Unfortunately, I have neither time nor knowledge to contribute to this effort. Please, update the community here if you make any progress!

 

[Gabriel Mateiciuc]

Very interesting re 1.8.2.5 drivers on ESXi 6.5. Looks like you've been using the updated build (4887370), I tried only the original one a couple of months back and couldn't get it to work at all. Maybe it's time to give it another try. Pity it's such a painful exercise to go back to 6.0 if things go astray with 6.5...

It was the same for the initial 6.5 build. 1.8.2.5 worked (those released last year). Yeah, going back to 6.0 ... hurts a bit.
I've learnt my lessons with 6.0 about keeping it updated. When it was launched, i've had the same urge to go back. That was until the updates started rolling and things got where they were supposed to.

As for the discussion on the MLNX Communities forums, I think it's pretty clear by now: people like us (using MLNX hardware to build high-speed custom storage solutions for ESXi on the cheap) are not of any interest to MLNX as a company. They know nobody in the enterprise will bother with this and companies will keep spending zillions on specialised solutions from EMC/HPE/NetApp. Their target market is HPC, so they focus on supporting Linux as a platform. That's also why, when it comes to ESXi, they spend most efforts on enabling SR-IOV to serve Linux guests.

+1 here, I have customers that would be the right target for MLX. Funny thing, they stick to spending a lot of zeroes on FC tech and even more on the overrated Intel adapters and then get a heart attack when they find out that a small new feature doubles the spending.
I wouldn't agree for the cheap part. If you build your lab/infra and add up ...
It's cheap when you compare to the big names. Research is $$$ expensive.

Considering this, it would be very nice of you if you could indeed undertake on developing the ESXi drivers with IB support. Unfortunately, I have neither time nor knowledge to contribute to this effort. Please, update the community here if you make any progress!

Well, time is what we all have little. I'll post any updates worth mentioning.

 

[mpogr]

I
I wouldn't agree for the cheap part. If you build your lab/infra and add up ...
It's cheap when you compare to the big names. Research is $$$ expensive.

You can't even compare the cost of the "proper" storage with what we're discussing here. CX-3 adapters are below 100 USD per piece now. An 18 port switch can be purchased for less than 300.
But it's definitely not plug-and-play, that's for sure... and no support provided other than here

 

[humbleThC]

we are all the support one should need

 

[humbleThC]

Is there any reason to consider upgrading from CX2's and a 4036E to CX3's and w/e the cheap 18 port switch model# is?
(Keeping in mind.. I'm limited to PCI 2.0 x8 slots)

If the argument is better driver/firmware support, and VMWare 6.5 support, that's a pretty good argument.

 

[_alex]

Hi, not sure if cx3 will work at all in pcie 2.0 x8.
Tried with cx354a some days ago in an older 1366 system with no luck. Card is good and works in 2011-3 with pcie 3.0.