Beware of EMC switches sold as Mellanox SX6XXX on eBay

Discussion in 'Networking' started by mpogr, Aug 2, 2016.

  1. Poco

    Poco New Member

    Joined:
    Nov 16, 2018
    Messages:
    3
    Likes Received:
    0
    So, my replacement EMC SX6005 switch came in..
    I thought something was wrong with my switch since my QDR Amphenol fiber cable didn't pick up a link to the switch with the last one, so that's why a returned it.

    However I'm curious if anyone knows some cheap DAC cables which work with this switch and CX-2 40Gbps cards.

    I kind of just bought this switch imagining it would be essentially just like my Voltaire ISR9024D-M I bought a long time ago..
    I wish I read this forum post about the EMC switches in short.

    Thanks in advance.
     
    #201
  2. cecil1783

    cecil1783 New Member

    Joined:
    Aug 9, 2017
    Messages:
    7
    Likes Received:
    7
    I'm not sure what you consider cheap, but I've been getting new Mellanox FDR DACs (1.5m and 2m) for around $35 on ebay, I'm sure FDR-10 cables would be less.

    Are you running a subnet manager on an infiniband connected machine? The SX6005 is unmanaged, it's just a dumb collection of copper until something tells it how the network works (opensm). You won't see a connection on any port, from any machine, until you have a subnet manager running.


    aa
     
    #202
    Poco likes this.
  3. Poco

    Poco New Member

    Joined:
    Nov 16, 2018
    Messages:
    3
    Likes Received:
    0
    I am running OpenSM from OFED, yes.
    The cards are using the mlx4_core and mlx4_ib modules, not Ethernet (mlx4_en).

    I'm just guessing it's my cables at this point.
    They work in point-to-point and have a 40Gbps link, but nada on the end of the switch oddly..

    I'll look around for some of those FDR cables though.
    Also; is it just safest to buy the official Mellanox cables I'm assuming?
     
    #203
  4. cecil1783

    cecil1783 New Member

    Joined:
    Aug 9, 2017
    Messages:
    7
    Likes Received:
    7
    I really can't say if any other cables will work or not. I believe others here have successfully used non-Mellanox cables, but can't say for sure which ones. A lot of people seem to be having good luck with cables from fs.com. Hopefully someone else can chime in with the cables they've had the best luck with, if not Mellanox.


    aa
     
    #204
    Poco likes this.
  5. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    109
    Likes Received:
    68
    A remark on IPoIB vs EN mode: everything else being equal (FDR and 56GbpE), you get SIGNIFICANTLY less network throughput in IPoIB mode comparing to native EN. And when I say significantly, I mean 1/2 the throughput as the best case scenario.

    That said, as long as storage speed is the main concern, you get the best throughput/latency with IB over SRP. However, with SRP being now deprecated in virtually every environment (except Linux on both ends), you have to move to iSER, which requires IP as the underlying layer, which means you pretty much have to use the EN mode with flow control enabled in order to achieve speeds that get close to the theoretical 56Gbps limit.
     
    #205
    Poco and Rand__ like this.
  6. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,816
    Likes Received:
    380
    is there an overview which describes the available options nowadays with platforms and protocols (and potentially recommendations) ?

    Would still be looking for the fastest option for ESX and nfs on any ZFS capable system (Solaris>FreeBSD>Linux in order of preference)...
     
    #206
  7. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    109
    Likes Received:
    68
    I'm not aware of any overviews, but, AFAIK, if you want to stick with the recent versions of ESXi (6.5 and above), your only option is Linux with ZoL and iSER (I use SCST implementation, but the built in should work as well).

    Edit: and yes, it is slower than pre-6.5 SRP had been...
     
    #207
  8. Terry Wallace

    Terry Wallace Member

    Joined:
    Aug 13, 2018
    Messages:
    30
    Likes Received:
    7
    I’ve been using gtek cables from amazon and I have 2 Mellanox switches 3 quantas and all my nic’s are Mellanox ConnectX-3 flashed over from hp.
    Never had a cable problem. I buy the cheap cables listed as open switch.
     
    #208
    Poco likes this.
  9. trippehh

    trippehh New Member

    Joined:
    Oct 29, 2015
    Messages:
    16
    Likes Received:
    3
    Anyone managed to get IPv6 routing on Ethernet going on these post-conversion?

    The switch ASIC seems capable, the OS is, but the CLI is hiding all IPv6 options except for the ones used by the management network interface.

    Same with both eth-single-switch and vpi-single-switch profiles.

    "ipv6 enable" on the root node, gone
    ipv6 options for the VRFs gone
    ipv6 options for the interfaces gone

     
    #209
  10. petreza

    petreza New Member

    Joined:
    Dec 28, 2017
    Messages:
    6
    Likes Received:
    0
    PCIe x8 2.0 should do 4GB/s one way. That's 32Gbps minus overhead. Why is autodestruct getting only 14Gbps (or 12.7Gbps)? Is it actually a limit imposed by the processor - X56xx?

    I have two PowerEdge R810 with Xeons X7560 and three ConnectX-3 Pro EN - MCX314A-BCCT - which I cannot test right now. Would I be able to get close to the theoretical limit of the PCIe x8 2.0 with this setup or am I also limited? I ask because I was planning to get a cheap, EMC OSed Mellanox MSX6012 switch and go through the pain of converting it to Mellanox OS, but if my speed is limited to approx. 10GbE, then there is no point.
    (As you could probably tell I quite new to all this.)

    Thanks for your help.
     
    #210
    Last edited: Dec 24, 2018
  11. petreza

    petreza New Member

    Joined:
    Dec 28, 2017
    Messages:
    6
    Likes Received:
    0
    (newbie question)
    Can a switch like Mellanox SX6012 have some of its ports operate in Ethernet mode while others in Infiniband? If so, is there any interference or do the two modes function as if there are two switches in one chassis?
     
    #211
  12. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,494
    Likes Received:
    338
    With the vpi profile the switch will still work like a single switch:
    SX6036_1.JPG
    SX6036_2.jpg
     
    #212
  13. petreza

    petreza New Member

    Joined:
    Dec 28, 2017
    Messages:
    6
    Likes Received:
    0
    Thank you i386!
    On a side note, can MCX314A-BCCT be crossflashed to MCX354A-FCCT?
    Thanks!
     
    #213
    Last edited: Dec 25, 2018
  14. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,816
    Likes Received:
    380
    Good to know, thanks.
    O/c the 314 is a x8 card and the 354 would be a x16 card, so its not a 'full' cross switch, but very good to know its working firmware side.
    Still looking to try that with my CX5-EN ;)
     
    #214
  15. petreza

    petreza New Member

    Joined:
    Dec 28, 2017
    Messages:
    6
    Likes Received:
    0
    I forgot to put ? at the end (fixed now) - I am asking not reporting.
    Sorry
     
    #215
  16. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,816
    Likes Received:
    380
    Ah. Usually it worked on CX3s, also with EN to VPI iirc, but have not tried it. Probably might work inside the same family (x8 to x8, x16 to x16) but not sure on LX/EN to VPI.
    Always hoped so but in reality not even LX/EN's are so cheap that it makes sense to get them over VPIs
     
    #216
  17. mpogr

    mpogr Active Member

    Joined:
    Jul 14, 2016
    Messages:
    109
    Likes Received:
    68
    Guys, is there any reasonable (price-wise) substitute to CX-3 cards available now in the wild? My CX-3s are fine, but they don't perform with RDMA-over-IP as well as newer cards would, impacting iSER performance comparing to what SRP used to provide... Has to be at least 56Gbps in EN mode though...
     
    #217
  18. Rand__

    Rand__ Well-Known Member

    Joined:
    Mar 6, 2014
    Messages:
    2,816
    Likes Received:
    380
    No.
    CX4/5 are expensive most of the time, although you can get deals for CX4 around $150/200. You'll only get 40Gbps on them though on a FDR switch, for 56 you need CX5's. Both can do 100Gbs o/c if you get the right model and switch
     
    #218
  19. oxynazin

    oxynazin New Member

    Joined:
    Dec 10, 2018
    Messages:
    7
    Likes Received:
    1
    Hi. Thanks for the guide. Still waiting for the switches (will arrive in a month or two, may be even three).

    Did you do benchmarks for iSER vs SRP? How much faster SRP in your case? I found this article: iSCSI vs iSER vs SRP on Ethernet & InfiniBand
    It seems that iSER on Ethernet is not so bad. And they used ConnectX-3 cards (MCX354A-FCBT).

    BTW what is RDMA-over-IP? I can find some articles about "RDMA-over-IP", but no protocol definition or standard, and the first link is about RoCE.
     
    #219
  20. i386

    i386 Well-Known Member

    Joined:
    Mar 18, 2016
    Messages:
    1,494
    Likes Received:
    338
    He probably means RDMA over Converged Ethernet ("RoCE")
     
    #220
Similar Threads: Beware switches
Forum Title Date
Networking Beware - Emulex 10GbE Virtual Fabric Adapter II x8 PCI-E slot Jun 22, 2014
Networking 25gbe switches, anyone with a list and pricing? Dec 17, 2018
Networking Any reason not to buy higher port switches? Nov 26, 2018
Networking Quiet-speed ... switches with a few QSFP ports with DIY Noctua replacement fans? Nov 19, 2018
Networking Do 1Gb switches matter? Nov 12, 2018

Share This Page