Movin' on up...to 40GbE

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
W

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
Well after firmware snafu on my HP branded MLX HCA's (wrong vendor cross-flash on PSID apparently, was not 40G eth capable), I re-flashed to HP provided latest firmware, re-gained full functionality w/ 40GbE (hell even 56GbE capable), updated junos to latest recommended, updated jweb application for full feature set web access (even though I typically cli config junos) and my MLX QSFP fiber cables even worked no issues.

It was a journey no doubt!

Image candy for 40G triumph!

ex4300-1.jpg ex4300-2.jpg ex4300-3.jpg

EDIT: Stupid 5M fiber DAC's, they are nice but I need about 1M :-D
 
Last edited:
  • Like
Reactions: Patrick, pyro_ and i386
W

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
iperf stats using jperf on two ubuntu 16.04 boxes, single vcpu, may try dual vcpu to see if it yields any more juice under the hood. Two VM's on different ESXi hypervisors, jumbo end-to-end, trying to tap 40G, 25Gbps, not too shabby! :-D

upload_2017-4-10_19-20-25.png
 
  • Like
Reactions: T_Minus
W

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
So I have 1 port free on the EX4300 and I have dual port HCA's w/ one port free on each server. If someone would be so kind as to validate that I could use these to come off that last free 40G port and into each server at 10G that would be most appreciated.

Juniper Networks QFX-QSFP-DACBO-1M 40G QSFP+ Breakout Cable | FS.COM

I already have the MLX MAM1Q00A-QSA converters to go on the HCA side of the house so I'd image it's possible if I understand this correctly.

TIA, whitey

EDIT: These look VERY interesting if I could trust that they would work. I KNOW the fs.com one's will work as I have never got a bad cable from them but ohh how I hate copper/DAC's (weight/tension/flexibility/cable mgmt-wise).

Avago AFBR-7IER03Z 40G QSFP+to 4x10G SFP+ Fibre Optic Cable3M 9ft | eBay
 
Last edited:
P

pyro_

Active Member
Oct 4, 2013
747
165
43
Make sure that the ports are not in vc mode but in Ethernet and access modes and that the correct vlan has been assigned as no vlan is assigned by default

Also on the cx3 cards ensure that they are set to be in Ethernet not IB. I had the same issues at first juniper does no vender lock their switches so I would take a guess that that is not the issue


Sent from my iPhone using Tapatalk
 
W

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
This is what ya want/validated to work for me at least Juniper EX4300 QSFP ports to CX3 cards at 40GbE.

New Mellanox HP 498386-B23 3M 4x DDR/QDR QSFP IB Fiber Optic Cable | eBay

EDIT: This is what @pyro_ is referring to, this will take all 4 ports and set them to host connect mode.

request virtual-chassis vc-port delete pic-slot 1 port 0
request virtual-chassis vc-port delete pic-slot 1 port 1
request virtual-chassis vc-port delete pic-slot 1 port 2
request virtual-chassis vc-port delete pic-slot 1 port 3
commit
 
A

adgenet

Member
Apr 12, 2016
40
17
8
I have a quick question for you 40GbE gurus out there.
I just picked up two connectx-3 cards and wanted to link the two up directly about 15M apart (in a separate room)
Are those MPO cables that seem to cost a gazillion dollars per meter (combined with surprisingly cheap QSFP+ transceivers) my only option? I saw a cisco transceiver for 40G over a single LC fiber but it was prohibitively expensive, the opposite problem of former... Active optical 40GbE cables designed for QSFP+ and 40GbE also seem sort of expensive.

Everybody I've talked to say that the old "regular qsfp" (not QSFP+) gear designed for 40G infiniband will not work for 40GbE (active cables included).

I asked mellanox support regarding one active "regular qsfp" "infiniband" cable I saw on ebay:
Thank you for contacting Mellanox Technical Support.
Please note that MC2206310-015 is an Infiniband only cable (Mellanox® active fiber cable, IB QDR/FDR10, 40Gb/s, QSFP, 15m) . And cannot be used in ethernet.

If you would like to receive recommendation for Ethernet cables we can redirect this case to our sales department which can provide you with best cables for your needs
Please let us know if there is anything else we can assist with.
We are looking forward to hearing from you.
Click to expand...
Are they telling the truth or do they just want me to buy more expensive cables? How is it that 40Gb Ethernet and infiniband would require different cables despite having similar transfer rates?
What do you smart folk here think I should do? Give up on 40GbE and run sfp+ adapters over LC fiber at 10Gb?
 
W

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
W

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
To clean it up a bit, this on each end probably (switch and MLX CX3, Juniper coded on switch side if you really wanna get particular but i think Juniper is less-concerned/picky abt optics so the generics 'should work')

40GBASE-SR4 QSFP+ 850nm 150m MTP/MPO Transceiver | FS.COM ($49)

10m (33ft) MTP Female to MTP Female 12 Fibers OM4 50/125 Multimode Trunk Cable, Polarity B, Elite, LSZH Bunch | FS.COM (custom 15M $85)

Not bad $134 out the door w/ free ship

EDIT: Not 100% sure if MLX cares abt optics but if you do need to you can maybe use these on the MLX QSFP HCA ports.

Mellanox MC2210411-SR4 40GBASE-SR4 QSFP+ Transceiver | FS.COM

If someone knows off hand that'd be great otherwise reach out to FS rep and validate.
 
I

i386

Well-Known Member
Mar 18, 2016
4,241
1,546
113
34
Germany
adgenet said:
Are they telling the truth or do they just want me to buy more expensive cables?
Click to expand...
bs. I'm using 2 brocade "fiber channel" cables with qsfp connectors between two connect-x3 and it works flawless. Running ib-diag (a tool from mellanox) and it shows support for up to qdr infiniband support:
Code: 
Vendor: BROCADE
OUI: 0x51e
PN: 58-0000033-01
SN: PAE116113000492
Rev: A
Length: 1 m
Type: Copper cable- unequalized
SupportedSpeed: SDR/DDR/QDR
And from wikipedia : "QSFP+ is an evolution of QSFP to support four 10 Gbit/sec channels carrying 10 Gigabit Ethernet, 10GFC FiberChannel, or QDR InfiniBand.[4] The 4 channels can also be combined into a single 40 Gigabit Ethernet link."
QSFP - Wikipedia

Edit: I should read more carefully.
I think the connectx-3 (and newer) don't work very well with 3rd party active cables & transceivers for 40gbe/infiniband. I have tested so far fdr finisar (all the same model) and various cisco 40gbe transceivers + fibers. No links between the hosts (host<->host).

With the qsfp+ to sfp+ adapter (Using a 40GbE (QSFP+) NIC with a 10GbE Switch (SFP+)) I could get all transceivers for 10gbe to work, no matter if they were mellanox, finisar or cisco branded.
 
Last edited:
A

adgenet

Member
Apr 12, 2016
40
17
8
Thanks for the advice.
I guess I'll need to think about it some more before pulling the trigger on cables and/or transceivers.
 
W

whitey

Moderator
Jun 30, 2014
2,766
868
113
41
adgenet said:
Thanks for the advice.
I guess I'll need to think about it some more before pulling the trigger on cables and/or transceivers.
Click to expand...
Down from thousands or several hundred prior to $58 I'd say get one of those I mentioned above from that ebay seller and test...pretty dead set certain you will be happily surprised w/ 40GbE speeds :-D
 
  • Like
Reactions: T_Minus
You must log in or register to reply here.
Top Bottom