Mellanox SX6036 Compatibility

[uncensoredtr]

I am planning to buy a SX6036 but have some questions.
Found some cheap Juniper QFX-QSFP-40G-SR4 optics but couldn't find any info about compatibility with the switch.
Using Mellanox MAM1Q00A-QSA anyone had any problems with sfp+ compatibility? For example using Mikrotik S+RJ10 (2.5/5/10G) (for Wireless AP)
Found some cheap HP JG327A 3M 40G DAC cables since HP used Mellanox stuff, I believe they should be compatible.

My Other Switches:
Mikrotik CRS309-8S+IN
Mikrotik CSS326-24G-2S+RM
TP-Link TL-SG3428XMP
APs:
2x TP-Link Omada EAP-660 HD

What transcievers or dacs has been tested with mellanox cards(CX354) and switches(SX6036)?

 

[naptastic]

Some things you should know first:
1. SR4 and LR4 transceivers use MPO-12 cable (SR "short range" multimode, LR "long range" single-mode) and MPO cables are really, really expensive.
2. There's probably no way to directly connect an SX6036 to any of your other switches.
3. The only real compatibility list is what Mellanox publishes, and everything on that list... is made by Mellanox.
4. Beware EMC-branded switches.

I have a set of 40gbE SMF transceivers from eBay that were $10 each. I can't even tell what brand they are from looking at them. They work fine in the CX354A cards, either as Ethernet or FDR10 InfiniBand. However, my ConnectIB cards and EMC-branded SX6005 switch will not even recognize that they are plugged in.

The HP cables you mention are only 40gbit, and cost more than the Mellanox-branded 56gbit cables on eBay, at least right now. The price curves on QSFP* cables is WEIRD. I got four 2m cables for $20 each, and two 65m cables (216 feet!) for $50 each, all Mellanox, all brand-new in sealed packaging. Every other distance was significantly more expensive.

Actual advice: have a plan in case you find out that things aren't compatible. Return, re-sell, come back here and see if we can figure something out together.

 

[NablaSquaredG]

LR4 transceivers use MPO-12 cable

No, they don't. LR4 = Single Mode

2. There's probably no way to directly connect an SX6036 to any of your other switches.

Why not? There are at least 3 ways

1. Breakout Direct Attach Cable (QSFP to 4x SFP+) will work
2. QSA Adapter + some normal optic will work (beware of LR 10GBase optics, they might not work. But I had good results with SR optics)
3. 40GBase-SR optic + MTP to 4x LC breakout + 10G-SR optic in the other switch

What transcievers or dacs has been tested with mellanox cards(CX354) and switches(SX6036)?

Mellanox cards aren't generally picky (except if you want 56GBit or Infiniband)

The switch has some vendor locks in place, you can remove them with

fae cable-stamping-unlock fdr10
fae cable-stamping-unlock 40g_lr4

Be aware that officially, the SX6036 only supports 40g_lr4 optics in a couple of ports (I'm not sure, but I think it's 1, 3, 33, 35)

 

[uncensoredtr]

First off thank you for replying and trying to help out.
I have to mention some things about my situation here.
I live in Turkey therefore shopping off ebay means big shipping and import taxes for me.
This changes a lot what i can find locally and what i can buy globally. (You can think like Brazil)
I will give some examples about some of the items later on so you can better understand the situation.

1. SR4 and LR4 transceivers use MPO-12 cable (SR "short range" multimode, LR "long range" single-mode) and MPO cables are really, really expensive.

I am aware of this and locally found dirt cheap Panduit 30m MPO-12 cables and bought 6 of them for like 15$ each.

4. Beware EMC-branded switches.

AFAIK you can tell them apart by front color right? Here are the pictures from the seller. They ask for 400$ each (I think it would be better if they were asking for 300$ but fair enough for not waiting shipping and tax process) and they won't sell unless i buy both. (This means i have to sell the other which i am not a fan of)

2. There's probably no way to directly connect an SX6036 to any of your other switches.

As mentioned by NablaSquaredG there are ways to get them connected.
I will go for his 3rd option there because they won't be in the same rack.
I will use DACs for servers on the same rack.

3. 40GBase-SR optic + MTP to 4x LC breakout + 10G-SR optic in the other switch

The switch has some vendor locks in place, you can remove them with

fae cable-stamping-unlock fdr10
fae cable-stamping-unlock 40g_lr4

I was not aware of this thanks for the tip. Do i need to use this commands after reboot or it is set it and forget it type of situation?

Be aware that officially, the SX6036 only supports 40g_lr4 optics in a couple of ports (I'm not sure, but I think it's 1, 3, 33, 35)

I don't plan to use LR optics but i know about this situation.


About the pricing the stuff I ve found:

Found some cheap Juniper QFX-QSFP-40G-SR4 optics but couldn't find any info about compatibility with the switch.

I found them for 40$ each from a datacenter in my city which is clearing out 40g stuff.
On ebay cheapest deal i can trust is Cisco QSFP-40G-SR4 for 18$ each plus 30$ shipping (+1$ each extra item) plus 20-25% import tax.
For 4 items it will be in the 130$ range to buy from ebay. Local one will be 160$ and maybe i can get a small discount.

Found some cheap HP JG327A 3M 40G DAC cables since HP used Mellanox stuff, I believe they should be compatible.

Asking price of the DACs I mentioned are 53$ each. There is a chance to get a small discount as well.
HiFiber branded 3m DACs on Amazon US costs 30$ plus 27$ shipping and import fees so 57$.
On ebay 2m DACs i can find are 40$ range and 3m DACs are 50$ range shipping included but without import fees.

 

[uncensoredtr]

I have been able to get info on the HP JG327A DACs for CX3 NICs.
They are in the compatibility list and same goes for Cisco SR4 optics as well.
(Source: Mellanox ConnectX-3 Firmware Release Notes Rev 2.42.5000)
I think they should have been using their switches to validate the cables with their cards.
I am pretty confident that these DACs will work. Juniper optics are still a mystery tho.
Since the datacenter is in my city, I think it might be possible to talk to them for a test or return if they don't work.
To be able to negotiate on this tho I have to buy the switches first.
Get them prepared (licence, firmware updates etc) using DACs and take one with me to datacenter if needed.
Do I need IB cable as well? Can I use DAC or management port?

BTW I don't know if they will be handy or not but I got Lab grade power supply, 16ch Logic Analyzer, TL866II EEPROM programmer, 100MHz Oscilloscope etc.

 

[uncensoredtr]

After seeing Beware of EMC switches sold as Mellanox SX6XXX on eBay thread, I was thinking a lot of people had these gear and I can learn from their experiences here.

 

[P0rt4lN3T]

No, they don't. LR4 = Single Mode


Why not? There are at least 3 ways

1. Breakout Direct Attach Cable (QSFP to 4x SFP+) will work
2. QSA Adapter + some normal optic will work (beware of LR 10GBase optics, they might not work. But I had good results with SR optics)
3. 40GBase-SR optic + MTP to 4x LC breakout + 10G-SR optic in the other switch



Mellanox cards aren't generally picky (except if you want 56GBit or Infiniband)

The switch has some vendor locks in place, you can remove them with

fae cable-stamping-unlock fdr10
fae cable-stamping-unlock 40g_lr4

Be aware that officially, the SX6036 only supports 40g_lr4 optics in a couple of ports (I'm not sure, but I think it's 1, 3, 33, 35)

Hi guys so in theory which 40Gbps GBICs we should use to plug a couple of devices in our rack with about 3 to 5mts distance mac.. i don´t plan to run copper dac cables.. i would preffer fiber solution.