Enable 56GbE on HP-649281-B21 / Mellanox 354 with direct connect?

[BackupProphet]

According to this information it should be possible. Has anyone tried and confirmed this? Or should I just enable infiniband instead?

 

[Freebsd1976]

i tried 56gbe and 56gb ib , after crossflash 354a-qcbt to fcbt
,in windows it show 56gb both eth and ib, i think your nic will also support both 56gb eth and ib

 

[i386]

It works with the proper cables, but these cables are rare & more expensive (at least when you look at >1m)

 

[Freebsd1976]

It works with the proper cables, but these cables are rare & more expensive (at least when you look at >1m)

i buy 56gb sac from taobao in china , 1m cost 10$ and 2m cost 15-20$

 

[arglebargle]

You need an FDR cable to do 56Gb, I picked up an FDR DAC on eBay for ~$25 (I wanted it fast, otherwise I'd have gone to taobao.)
IBM 1m Mellanox QSFP Passive Copper FDR14 InfiniBand Cable 00W0049 Server ZZ 883436241496 | eBay

I actually wonder if there's anything special about these or if they're just a bit flipped in the cable's eeprom. I know @fohdeesha managed to reprogram a screwed up monoprice breakout DAC using I2C on a Brocade switch, I wonder if there's a viable DAC out there that'll do FDR with a little I2C hacking.

edit: I'm doing 56Gb between two of these HP cards right now, FYI.

 

[fossxplorer]

Will 56GbE work with a back to back link?I.e without switch, which is mentioned as a requirement.

 

[arglebargle]

Will 56GbE work with a back to back link?I.e without switch, which is mentioned as a requirement.

Yes, I'm running 56Gb FDR IB between a windows box and linux.

root@meta ~]# ibstat
CA 'mlx4_0'
       CA type: MT4099
       Number of ports: 2
       Firmware version: 2.42.5000
       Hardware version: 1
       Node GUID: 0x0002c9030039af50
       System image GUID: 0x0002c9030039af53
       Port 1:
               State: Active
               Physical state: LinkUp
               Rate: 40
               Base lid: 0
               LMC: 0
               SM lid: 0
               Capability mask: 0x04010000
               Port GUID: 0x0202c9fffe39af50
               Link layer: Ethernet
       Port 2:
               State: Active
               Physical state: LinkUp
               Rate: 56
               Base lid: 1
               LMC: 0
               SM lid: 1
               Capability mask: 0x02514868
               Port GUID: 0x0002c9030039af52
               Link layer: InfiniBand

 

[fossxplorer]

But Ethernet link is 40Gb, not 56Gb though.

 

[arglebargle]

But Ethernet link is 40Gb, not 56Gb though.

I'm not using an FDR cable for that link.

 

[xephael]

I'm not using an FDR cable for that link.

Is 56GbE restricted to IB, or does it work on Ethernet with FDR14 as well?

 

[i386]

56GBe works with fdr14 cables

 

[nikey22]

Is there any problem running optical transceivers and OM4 in a peer-to-peer connection for 40Gbe traditional-ethernet? I'm sure this will eliminate IB connectivity. But still, I wonder if it's possible.

 

[vanfawx]

IB should work so long as one node runs a subnet manager. I don't see why direct attach optical wouldn't work either.