Mellanox Connectx-3 link down ESXI with Voltaire 4036

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
J

justinm001

New Member
Apr 11, 2018
19
1
3
38
hey all. I'm very new to Mellanox and IB

I picked up a few MCX354A-QCBT and a Voltaire 4036 with cables. I updated the voltaire and some of the MCX354A-QCBT to the MCX354A-FCBT firmware (for 40GBE).
On Windows server 2016 I show the link as up and connected but not pulling DHCP (have 1GB ethernet connected to Voltaire)
Voltaire able to ping and pulling IP
On ESXI I show the ConnectX-3 cards but link down, I tried using latest drivers from VMware's site.

Shouldn't it just connect and work or am I missing something. Anyone know why the link shows up in Windows but not ESXI? Also why can't I get any ethernet traffic through? Is there something I need to do to switch from fabric to ethernet?
 
D

darkconz

Member
Jun 6, 2013
193
15
18
It seems like you are running in IB mode? Not too sure how to set the cards to IB in ESXi..


Sent from my iPhone using Tapatalk
 
J

justinm001

New Member
Apr 11, 2018
19
1
3
38
Its a 4036E which I believe is Ethernet mode, or IB but not sure. I'm not sure how to switch modes or if there's a way.
 
J

justinm001

New Member
Apr 11, 2018
19
1
3
38
this must be where i'm confused. I'm just trying to get esxi vsan on faster networking. Can I not use it for ethernet? if not can i use IB for vsan traffic? Ideally i'd like to use both ports, one for vsan and other for all else.

Also I'm unable to get the sfp+ ports to work yet, think my Cisco sfp-10g-sr sfp+ transceivers aren't compatible with the 4036. But it should work with the 1GB right, or is that just for management?
 
I

i386

Well-Known Member
Mar 18, 2016
4,245
1,546
113
34
Germany
J

justinm001

New Member
Apr 11, 2018
19
1
3
38
I actually just found those posts and working through it.

I'm very new at infiniband, so I cant get ethernet traffic through the IB ports? Why do they have sfp+ ethernet if i can get traffic routed to it? I'm not a network guy so its a bit over my head.

also do you know what sfp+ transceiver might work?
 
I

i386

Well-Known Member
Mar 18, 2016
4,245
1,546
113
34
Germany
justinm001 said:
I'm very new at infiniband, so I cant get ethernet traffic through the IB ports?
Click to expand...
Correct

SFP+ 4036e <---> ethernet nic, switch or router
QSFP+ 4036e <---> infiniband nic or switch

If the 4036 receives a datagram on the qsfp ports and the target is on the ethernet network it will process the datagram and send it over the ethernet ports. Same with ethernet frames: they arrive at the sfp ports, the switch processes the frame, creates an infiniband datagram and sends it to the target qsfp port.


justinm001 said:
also do you know what sfp+ transceiver might work?
Click to expand...
On the mellanox nics you should be able to use any optical transceiver/cable you can find.
On the switch side it's not that simple and you should use mellanox coded transceivers*.

*I have an sx6036 and it works fine with mellanox coded transceivers from fiberstore, but refuses to work with cisco transceivers.
 
J

justinm001

New Member
Apr 11, 2018
19
1
3
38
I think I understand. But i still shouldn't be getting link down in ESXI right? I think thats my main problem.
Cables and everything look good and on a windows machine we're getting link just no traffic (which makes sense because nothing else on it). We tried direct attach to other card and to itself but no link or lights. Also tried another Voltaire switch with older firmware.

MLNX-OFED-ESX-2.4.0.0-10EM-600.0.0.2494585.zip driver doesn't show cards in network adapter
MLNX-NATIVE-ESX-ConnectX-3_3.16.11.6-10EM-650.0.0-offline_bundle-7347702.zip shows cards but link down
is there another driver I should be using?
 
J

justinm001

New Member
Apr 11, 2018
19
1
3
38
yes, below is the settings, maybe something's off in there.

subnet manager info is:
sweep_interval: 15
max_wire_smps: 16
lmc: 0
max_op_vls: 5
transaction_timeout: 150
head_of_queue_lifetime: 16
leaf_head_of_queue_lifetime: 16
packet_life_time: 18
sminfo_polling_timeout: 5000
polling_retry_number: 12
reassign_lids: disable
babbling_port_policy: disable
routing_engine_names: minhop
log_flags: 7
force_link_speed: 0
polling_rate: 30
mode: enable
state: master
sm_priority: 4
 
J

justinm001

New Member
Apr 11, 2018
19
1
3
38
If I plug it directly into another server it will work, so it seems like its the voltaire switch. Is it a setting or do I need to find a ethernet one
 
J

justinm001

New Member
Apr 11, 2018
19
1
3
38
Should I just get a sx6036 or sx1012? Will that provide what I'm looking for? Does it provide Ethernet. I can just put one of these cards in my router and I believe it's supported.
 
T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,641
2,058
113
Did you change to IB?

Ethernet on QSFP will not be possible.
 
T_Minus

T_Minus

Build. Break. Fix. Repeat
Feb 15, 2015
7,641
2,058
113
justinm001 said:
UMMMMM Not sure how to do that. I assumed they were IB originally
Click to expand...
I thought you said you were trying to get Ethernet to work out of those ports on your systems?
Have you re-configured them is what I'm asking, not the switch.

Now that you know ethernet does not work on the IB Ports -- Just making sure you didn't have any specific configs, NICs added, etc for IB in your systems that you forgot ot change back.
 
J

justinm001

New Member
Apr 11, 2018
19
1
3
38
Can anyone verify that MLNX-OFED-ESX-2.4.0.0-10EM-600.0.0.2494585.zip will even work with ESXI6.5? It seems like its 6.0 only. No matter what I do I can't get the link up. tried various firmware and drives and installed/uninstalled all kinds of stuff. Maybe Connectx3 IB doesn't work with ESXI6.5. Or somethings up with the 4036e subnet manager
 
J

justinm001

New Member
Apr 11, 2018
19
1
3
38
Looks like I was finally able to get them up using the 1.8 driver and restarting the subnet manager on the 4036e a couple times.
 
You must log in or register to reply here.
Top Bottom