HP Brand Mellanox CX455 IB/Ethernet 100G Single Port QSFP28 and Switch compatibility

[dadscorp]

Hi Folks,
I am new to the forum.
I have bought some HP Branded CX455 IB/Ethernet 100g Single port QSFP28 Cards. and 7 Mtr QSFP28 Active Fiber cables. these work great with the IB switch I have.

Now i have a requirement for running VMWare and after breaking my head for over 2 weeks I learned that VSphere 7 has no support for the Mellanox CX455 in IB mode and it only supports ethernet mode at 100G. Now I plan to run a Virtual Lab for which I need VMware and since i have NVME U2 Drives - 4-5 per server. I want to be able to run these at the best speed possible. I got myself a SN2100 but it refused to work and the seller is not responding as it too time to get the stuff shipped to my country.
I first shipped to my virtual address in USA and from there i used the USA2me and got it and it does not work DAAM. I spent over 500-600 USD on shipping and Customs alone over and above the USD1000 on the SN2100 and it does not work.

I have seen the below options.
EDGE-CORE-7512-32X-100GBE
EDGE-CORE-7412-32X-100GBE
EDGE-CORE-7712-32X-100GBE
Celestica Seastone DX010 32-Port 100G QSFP28
Arista DCS-7060CX-32X - Now this says QSFP100 - not sure if its the same as QSFP28.

I just want to run VSphere + VSAN 7

Can somebody guide me in the right direction.

Thanks in advance

 

[whbeers]

I just added a 7060CX to my homelab and couldn't be happier - QSFP100 is Arista's name for QSFP28, amusingly an attempt to reduce confusion:

What is the difference between QSFP28 and 100G QSFP? They are the same. The “QSFP” form factor was originally defined for <10G speeds. When it was adopted for 40G, the name became QSFP+ to denote the higher aggregate performance. The same “QSFP” form factor was later adopted for 100G but the electrical interface had to be upgraded to handle 25Gbps/lane. The electrical interface for 100G can handle up to 28Gbps, hence the engineering and industry name is QSFP28. Arista refers to the 100G form factor as “100G QSFP” to avoid any confusion

(from https://www.arista.com/assets/data/pdf/Arista100G_TC_QA.pdf)


note that you should also be able to run these cards with 40G transceivers if you want to save a bit on the switch.

lastly, you may have already figured this out, but I wrestled with the HP firmware for a bit trying to get it to negotiate different rates / use an fs.com DAC/transceiver for a while before flashing the stock firmware (mstflint -d [pci device address] -i firmware.bin -allow_psid_change burn), after which everything I threw at it worked magically.


(and for the benefit of anyone else coming across this thread wondering how to switch these cards to ethernet mode: mstconfig -d [pci device address] s LINK_TYPE_P1=2)

 

[Rand__]

So have you given up on the SN2100? Is it DOA? or whats the problem with it?

Any of the options above should work in ETH mode (likely with SoNIC), but I'd concentrating on fixing your existing one first

 

[dadscorp]

I have the switch but its not giving any link on connecting it.
even tried connecting the console cable. nothing comes on the console port too.
so basically the switch is DOA
any ideas if i can get the spare motherboard for it and replace it or something like that. its a SN2100 after all.

 

[Rand__]

Any LEDs lighting up at all when you power it on?
Tried various console settings?

 

[dadscorp]

Yes. when we power on. the LED for power / psu etc. come up.

The switch will initialize right. while booting the right bank of 8 ports out of 16 do blink for a second so i guess the left bank is either dead or administratively shutdown.

the management port ethernet does not pick any IP and nothing showing up on the DHCP server as well. so I guess thats down too. and I did try different settings on the console cable. this cable is working with another IB Switch so that console cable works.

 

[Rand__]

Hm that sounds perfectly fine. If its a HPE SN2100 (similar to the deal i posted a few weeks back) then its normal that only 8 ports are up too.

Was it a new or used switch? Best bet would be to play some more with console cable...

 

[dadscorp]

Thanks Rand... sounds encouraging. I have never used any 100GBE switch. But If i have to connect to my IB Switch i put the cards in IB Mode and in Debian i start the OpenSM and infiniband-diags packages and load the modules required - modprobe mlx5_en mlx5_core ib_umad ib_uverbs ib_ipoib and bingo the cards work and ibstatus shows a link and the switch and the card LED do light up as well. if i assign a IP to the ib0 interface I can ping too. all this working perfectly in Linux. even the console cable works with the HyperTerminal in Linux with my IB switch so that's ok too.

put the SN2100 and nothing is working...of course I need to put the card into ethernet mode I do it from the bios. but no luck
I have tried different ports on the switch, tried different cables, cards all in vain. I have only one cable that's the standard Cisco Console cable. Do you think its compatible with the SN2100 or the SN2100 console cable is different ?
I tried with both 9600 and 115200 baud rate, 1 stop bit, rest none... no luck.. any ideas

 

[Rand__]

Hm, sounds ok




Again, new or used switch? Anything the seller reports? Did it work for them?

 

[dadscorp]

Thanks for the above info Rand.. appreciate how you are trying to help me out.

This is a old switch, and the seller went offline after a few days and is not replying and my period of returns is out too so a bad buy i guess.
Happens at times when you order of ebay.

but since there is no reply on the console port and no link what so ever i am just almost giving up on this.

Getting a EDGE-CORE 7512-32X 100GBE - hopefully it should work with the cards i have in VMWare. that's about all I need -- Throughput . I dont need any of the advanced management functions of the switch .

 

[Rand__]

can you get link on any of the active 100g ports? at 40g maybe?

I have heard not so good things about that particular seller, but have no experience with them myself, just make sure to get one with OS (and verify its actually running with Sonic so you're not hit by license fees)

 

[dadscorp]

can you get link on any of the active 100g ports? at 40g maybe? -- No nothing

Oh daam !!! one more bad seller. don't tell me. it said 1718 rating with 99.3% Positive feedback so I assumed him to be geniune.

lets hope I get a working switch. He is sending me one with ONIE installed on it.

Let it come lets see if it works and if I can get Sonic working on it.

 

[Rand__]

Ok, good luck with it
For the SN2100 - if oyu had a second you could try cloning the msata drive, but with a single one thats difficult.
Have you opened it up to see if there is any damage?
You getting your cash back (ie paypal insured)?

 

[dadscorp]

Nope... all the dates have gone passed by. i tried everything before posting here this is more than 3-4 months old story. so no Paypal insurance now.

Yes I did open up the switch. cannot find any apparent damage and I paid USD 2000 to that Idiot for it. a real cheat.

Just keeping my fingers crossed hope I get a working switch this time. will post my results after I get mine

 

[Rand__]

Ouch

And yes please, let us know how it goes with the new one...

 

[alex.forencich]

I got a celestica seastone DX010 switch a while back for a very good price and got sonic up and running on it. However, the sonic interface is terrible and I have not been able to set it up to work as an ethernet switch - I cannot ping through it, even after reconfiguring the ports for L2 mode. Sonic also does not support PTP on that switch, despite the switch ASIC supporting PTP, and that ended up basically being a deal-breaker. So I ended up getting an Arista 7060CX, and that works great. Tested with ConnectX-4, ConnectX-5, and corundum. The 7060 is also quite a bit shorter than the seastone.

 

[alex.forencich]

One thing to note about console cables: there is no single standard for RS232 over RJ-45 so they can have all sorts of oddball pinouts, are you sure you are connected to it correctly?

 

[klui]

Lots of vendors use Cisco's rollover standard: Juniper, Brocade, Arista, Celestica.... can all use the same Cisco DB9-to-RJ45. The only thing you need to worry about is if there is a DB9-to-DB9 in between and it's not a Cisco do you have to try 2 different ones. Most cheap cables aren't labeled so you would have to make sure you use a null modem type.

I remember it was either Nortel or Extreme that needed a serial DB9-to-DB9 instead of null modem.

 

[alex.forencich]

Good to know that Arista and Celestica use the same pinout, but what about Mellanox?

 

[klui]

Mellanox uses the same Cisco console cable, too.