1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

ZFS Advice for new Setup

Discussion in 'Linux Admins, Storage and Virtualization' started by humbleThC, Jan 2, 2017.

  1. humbleThC

    humbleThC Member

    Joined:
    Nov 7, 2016
    Messages:
    99
    Likes Received:
    7
    I havent yet.

    Few things I was talking with a friend about potentially testing 'for fun'

    Tear down one of my ESX servers, and install Linux on it directly.
    Then i could test Linux>Linux and see how that compares to the Linux>ESX SRP benchmarks.
    Then rebuild Linux back to OmniOS, retest against Linux & ESX initiators.
    Then rebuild to Solaris, retest against Linux & ESX.
    Finally go back to Windows 2016 one last time, retest against Linux & ESX.

    I'll very likely not end up with any Linux as a initiator myself, but if theory proves to be correct on the OFED, and the VMWare drivers being weak/slow/untunable, then i'll find it.
    - And if that is the case, i'll keep the Linux initiator host around to help finish my 'NAS back-end bake-off'.
    - Then, i can finalize on my NAS solution, and put Linux host back to ESX6 cluster.

    Somewhere in between there, i need to test the direct-connect cable between two ports of Linux>Linux and Linux>ESX and play with just the MTU 4096 -vs- 65520 on CX2 hardware.
    (which also means swapping back'n'forth between hardware & software subnet mgrs, should be fun!)​
     
    #141
    Last edited: Jan 23, 2017
  2. humbleThC

    humbleThC Member

    Joined:
    Nov 7, 2016
    Messages:
    99
    Likes Received:
    7
    Unless MPOGR has done most of those tests and can save me the leg work with 'the gist' :)
     
    #142
  3. T_Minus

    T_Minus Moderator

    Joined:
    Feb 15, 2015
    Messages:
    6,078
    Likes Received:
    1,210
    @humbleThC & @mpogr Sent you guys a PM, I think I may be able to help speed things along.
     
    #143
  4. mpogr

    mpogr Member

    Joined:
    Jul 14, 2016
    Messages:
    84
    Likes Received:
    38
    Sorry guys, can't help much in the way of testing. Very busy at work, and my home system is in a stable working state, so don't want to mess around much with it...
     
    #144
  5. humbleThC

    humbleThC Member

    Joined:
    Nov 7, 2016
    Messages:
    99
    Likes Received:
    7
    I'm much closer to being able to 'publish' a guide specific to CentOS/RedHat (latest) + Mellanox OFED (latest) + SCST (latest) w/ SRP Targets w/ ZoL Volumes for virtual SRP disks to ESX6 (working). Just need to have mpogr proof check it, and I might run it through both OS variants tomorrow to look for missing steps / additional info I could provide.
     
    #145
    Last edited: Jan 23, 2017
  6. humbleThC

    humbleThC Member

    Joined:
    Nov 7, 2016
    Messages:
    99
    Likes Received:
    7
    Take a peek @ NAS - VMware SRP Guide - Wiki³

    I need to fix the section on IB Partitiosn (step 8) because i vastly misunderstood that step (obvious when you reed it) i.e. I have a switch, so the partitions.conf file for me on the hosts is unused/not-required.

    Besides that i'm getting closer
     
    #146
    mpogr and whitey like this.
  7. whitey

    whitey Moderator

    Joined:
    Jun 30, 2014
    Messages:
    2,501
    Likes Received:
    765
    Aww hell yea!!! Thanks, just freed up some HW to take on a test run of this soon. Will proof it and let ya know of hangups or if it's silky smooth.
     
    #147
  8. humbleThC

    humbleThC Member

    Joined:
    Nov 7, 2016
    Messages:
    99
    Likes Received:
    7
    Cleaned it up a little, particularly the IB Partitions section (step 8).
     
    #148
    Gabriel Mateiciuc likes this.
  9. Gabriel Mateiciuc

    Gabriel Mateiciuc New Member

    Joined:
    Apr 21, 2016
    Messages:
    25
    Likes Received:
    10
    Hi, nice effort to put the jigsaw pieces together.
    I could add, form my experiences that :
    1. When you'll try some CX3's you'll notice quite some speed bump.
    2. On the VMware side, things are less nice :
    - you go with 1.8.2.5 - SRP and no IPoIB (so no VM networking there) - that has the advantage of working on the 6.5
    - you go with 1.8.3.0 beta - and you get all the bells and whistles that IB has (bar SR-IOV) but you'll have to stick with 6.0
    Notice no ETH options, because, yes, I'm one of those that has a IB only SAN fabric (no ETH) in my lab.
    Right now I'm in the process of getting my main storage from Omnios to Ubuntu 16.04 - a bit stuck at building the ib_srpt module. Probably being bold and going with the 4.0 MOFED doesn't help me in that direction.
    For another less busy period, I'll restart some work on building the drivers for VMware.
     
    #149
  10. mpogr

    mpogr Member

    Joined:
    Jul 14, 2016
    Messages:
    84
    Likes Received:
    38
    Can you, please, share how you succeeded to make the 1.8.2.5 drivers work under ESXi 6.5? For the life of me, I couldn't make them work no matter what I tried...

    Sent from my SM-G920I using Tapatalk
     
    #150
  11. mpogr

    mpogr Member

    Joined:
    Jul 14, 2016
    Messages:
    84
    Likes Received:
    38
    Re building SCST with MOFED 4.0, you have to use the trunk, the 3.2.x branch no longer compiles with it.

    Sent from my SM-G920I using Tapatalk
     
    #151
  12. Gabriel Mateiciuc

    Gabriel Mateiciuc New Member

    Joined:
    Apr 21, 2016
    Messages:
    25
    Likes Received:
    10
    Hello,
    For esxi6.5 - I've used this :
    MLNX-OFED-ESX-1.8.2.5-10EM-600.0.0.2494585
    Standard "IB fabric" procedure - disable/remove the nmlx drivers (inbox) then install 1.8.2.5
    I only set the params for mlx4_core for mtu and msix.
    It seems to work but right now I'm fiddling with SCST. I've managed to get it to build (the tricky part was the ib_srpt driver - which has a nice redhat friendly Makefile - especially the MOFED paths/detection) but on a C7.3 with ml 4.9 kernel with MOFED 4.0
    Does anyone has hit this problem with esxi when marking a lun as not ssd in vsphere makes it go away and never return ?
     
    #152
  13. mpogr

    mpogr Member

    Joined:
    Jul 14, 2016
    Messages:
    84
    Likes Received:
    38
    I have tried this about a month ago for quite significant amount of time with no luck at all. Haven't got the SRP adapters showing up in ESXi. The log inspection revealed the drivers failed loading. This has been confirmed by Mellanox themselves: vmlkilunx based drivers are no longer supported in ESXi 6.5, that's the reason this won't work. Are you sure you actually got this working?
    I've refreshed my SCST setup on CentOS 7.3 after updating to the latest kernel and MOFED 4.0. I had to move to the SCST trunk in order to get it compile. The 3.2.x branch I've been using with MOFED 3.4 is no longer compiling.
    This is probably the symptom of 1.8.2.5 drivers actually not working on ESXi 6.5, as they should. Please, read here for further info on the subject.
     
    #153
  14. Gabriel Mateiciuc

    Gabriel Mateiciuc New Member

    Joined:
    Apr 21, 2016
    Messages:
    25
    Likes Received:
    10
    Ok, so far, to sum it up :

    C7 - kernel ml 4.9.x + mofed 4.0 + scst trunk (3.3.x) on top of zfs 0.7.0-rc3 - functional
    Gentoo (yeah, i got fed up and went back to the roots :) - gentoo-sources + scst trunk + ofed (somehow in-tree) + zfs latest - functional.

    Not ubuntu per se - but the entire clusterfuck brought by the kmod/dkms - which is cool when it works, but a pain to work around when it doesn't

    Esxi :
    tha stable 6.0 - this is actually weird as 1.8.3.0 seems to start acting up at times.
    uname -a :
    VMkernel ... 6.0.0 #1 SMP Release build-4600944 Nov 3 2016 22:17:36 x86_64 x86_64 x86_64 ESXi

    esxcli software vib list | grep -i mel
    nmst 3.8.0.56-1OEM.600.0.0.2295424 MEL PartnerSupported 2017-01-19
    mft 3.7.1.3-0 Mellanox PartnerSupported 2017-01-19
    net-ib-addr 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
    net-ib-cm 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
    net-ib-core 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
    net-ib-ipoib 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
    net-ib-mad 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
    net-ib-sa 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
    net-ib-umad 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
    net-mlx4-core 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
    net-mlx4-ib 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
    net-mst 3.7.1.3-1OEM.550.0.0.1331820 Mellanox PartnerSupported 2017-01-19
    net-rdma-cm 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
    scsi-ib-iser 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18
    scsi-ib-srp 1.8.3.0-1OEM.500.0.0.472560 Mellanox PartnerSupported 2017-01-18

    esxcfg-scsidevs -a :

    vmhba_mlx4_0.1.1mlx4_core link-n/a gsan.8100000000000000xxxxxxxxxxxxx (0000:03:00.0) Mellanox Technologies MT27500 Family [ConnectX-3]
    vmhba196608ib_iser online iqn...:xx:xx Mellanox iSCSI over RDMA (iSER) Adapter

    esxcfg-nics -l :

    vmnic_ib0 0000:03:00.0 ib_ipoib Up 56252Mbps Full xx:.. 4092 Mellanox Technologies MT27500 Family [ConnectX-3]

    Esxi - 6.5 - mind you, the 1.8.2.5 is some new release - 15.03.2016 if I'm not mistaking - that has the key element - built against the new vmkapi.

    uname -a :
    VMkernel .... 6.5.0 #1 SMP Release build-4887370 Jan 5 2017 19:17:59 x86_64 x86_64 x86_64 ESXi

    esxcli software vib list | grep -i mel
    net-ib-addr 1.9.10.6-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    net-ib-cm 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    net-ib-core 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    net-ib-ipoib 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    net-ib-mad 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    net-ib-sa 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    net-ib-umad 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    net-memtrack 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    net-mlx-compat 2.4.0.0-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    net-mlx4-core 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    net-mlx4-en 1.9.10.6-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    net-mlx4-ib 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    net-rdma-cm 1.9.10.6-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    scsi-ib-iser 1.9.10.6-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04
    scsi-ib-srp 1.8.2.5-1OEM.600.0.0.2494585 MEL PartnerSupported 2016-12-04

    looks like I was lazy to remove some inbox/leftovers :) - these other versions are not used anyway, as we all know.

    esxcfg-scsidevs -a :
    ...
    vmhba33 mlx4_core link-n/a gsan.810000000000000010e0000xxxxxxx (0000:04:00.0) Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s]
    vmhba34 mlx4_core link-n/a gsan.810000000000000010e0000xxxxxxx (0000:04:00.0) Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s]

    esxcfg-nics -l :
    ...
    vmnic1000202 0000:04:00.0 ib_ipoib Up 40000Mbps Full xx:.. 4092 Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s]
    vmnic2 0000:04:00.0 ib_ipoib Up 40000Mbps Full xx:.. 4092 Mellanox Technologies MT26428 [ConnectX VPI - 10GigE / IB QDR, PCIe 2.0 5GT/s]

    So far the performance seems to be there - I don't have some powerful machines as targets to test as I'm moving data around at the moment.
    Besides that, I've been running Omnios until recently on my main storage (Not working with CX3 or nvme above 1.1 made me try Linux for that too) so a compare would be a bit stretched.

    Let me know what else I could help with.

    @mpogr - It's funny, I've just realized you're the one with the thread on the mlnx community site :). I've stumbled upon it a lot of times and, for the record, I fully agree - No point in restating the facts. Sometimes they seem idiots - you get ipoib and srp but no iser ... which you say as a company is the future ... why ?!
    The situation described there is exactly what prompted me to look into building esxi driver for IB support. Let's say I haven't gotten too far but it's time (and nerve) consuming.
     
    #154
    mpogr and T_Minus like this.
  15. mpogr

    mpogr Member

    Joined:
    Jul 14, 2016
    Messages:
    84
    Likes Received:
    38
    Very interesting re 1.8.2.5 drivers on ESXi 6.5. Looks like you've been using the updated build (4887370), I tried only the original one a couple of months back and couldn't get it to work at all. Maybe it's time to give it another try. Pity it's such a painful exercise to go back to 6.0 if things go astray with 6.5...

    As for the discussion on the MLNX Communities forums, I think it's pretty clear by now: people like us (using MLNX hardware to build high-speed custom storage solutions for ESXi on the cheap) are not of any interest to MLNX as a company. They know nobody in the enterprise will bother with this and companies will keep spending zillions on specialised solutions from EMC/HPE/NetApp. Their target market is HPC, so they focus on supporting Linux as a platform. That's also why, when it comes to ESXi, they spend most efforts on enabling SR-IOV to serve Linux guests.

    Considering this, it would be very nice of you if you could indeed undertake on developing the ESXi drivers with IB support. Unfortunately, I have neither time nor knowledge to contribute to this effort. Please, update the community here if you make any progress!
     
    #155
  16. Gabriel Mateiciuc

    Gabriel Mateiciuc New Member

    Joined:
    Apr 21, 2016
    Messages:
    25
    Likes Received:
    10
    It was the same for the initial 6.5 build. 1.8.2.5 worked (those released last year). Yeah, going back to 6.0 ... hurts a bit.
    I've learnt my lessons with 6.0 about keeping it updated. When it was launched, i've had the same urge to go back. That was until the updates started rolling and things got where they were supposed to.

    +1 here, I have customers that would be the right target for MLX. Funny thing, they stick to spending a lot of zeroes on FC tech and even more on the overrated Intel adapters and then get a heart attack when they find out that a small new feature doubles the spending.
    I wouldn't agree for the cheap part. If you build your lab/infra and add up ... :)
    It's cheap when you compare to the big names. Research is $$$ expensive.

    Well, time is what we all have little. I'll post any updates worth mentioning.
     
    #156
    mpogr likes this.
  17. mpogr

    mpogr Member

    Joined:
    Jul 14, 2016
    Messages:
    84
    Likes Received:
    38
    You can't even compare the cost of the "proper" storage with what we're discussing here. CX-3 adapters are below 100 USD per piece now. An 18 port switch can be purchased for less than 300.
    But it's definitely not plug-and-play, that's for sure... and no support provided other than here :)
     
    #157
  18. humbleThC

    humbleThC Member

    Joined:
    Nov 7, 2016
    Messages:
    99
    Likes Received:
    7
    we are all the support one should need :)
     
    #158
  19. humbleThC

    humbleThC Member

    Joined:
    Nov 7, 2016
    Messages:
    99
    Likes Received:
    7
    Is there any reason to consider upgrading from CX2's and a 4036E to CX3's and w/e the cheap 18 port switch model# is?
    (Keeping in mind.. I'm limited to PCI 2.0 x8 slots)

    If the argument is better driver/firmware support, and VMWare 6.5 support, that's a pretty good argument.
     
    #159
  20. _alex

    _alex Active Member

    Joined:
    Jan 28, 2016
    Messages:
    689
    Likes Received:
    63
    Hi, not sure if cx3 will work at all in pcie 2.0 x8.
    Tried with cx354a some days ago in an older 1366 system with no luck. Card is good and works in 2011-3 with pcie 3.0.
     
    #160
Similar Threads: Advice Setup
Forum Title Date
Linux Admins, Storage and Virtualization New home server setup need advice. Jan 30, 2016
Linux Admins, Storage and Virtualization Switching from Windows based Storage/Transcode Server to Ubuntu Advice Nov 21, 2017
Linux Admins, Storage and Virtualization ZFS on Linux w/ KVM .....hard drive utilization advice. May 16, 2017
Linux Admins, Storage and Virtualization Advice on ZFS/FreeNAS for 40TB Feb 1, 2017
Linux Admins, Storage and Virtualization Need NAS Advice - Is there anything better than Synology? Jun 10, 2013

Share This Page