40Gb Ethernet switch for Mellanox ConnectX-3 CX354A cards

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
M

Meng Wang

New Member
Dec 19, 2018
9
0
1
Hello,
I recently found Mellanox ConnectX-3 cards can work on both of ib and ethernet mode. I configured one port as ethernet mode and connect to an Arista 40Gb switch with QSFP cable. From the host side, a new ethernet link is detected by system and can be configured with ip address. But the switch side said the interface is down even though it could detect the transceiver insert. The switch side info is shown below. Does it due to Arista switch cannot support Mellanox VPI card? If so, is there any recommended 40Gb ethernet switch can work with this Mellanox card?

Switch side info:
Ethernet21/1 is down, line protocol is down (notconnect)
Hardware is Ethernet, address is 444c.a80c.9192 (bia 444c.a80c.9192)
Ethernet MTU 9214 bytes , BW 40000000 kbit
Full-duplex, 40Gb/s, auto negotiation: off, uni-link: unknown
Down 25 minutes, 30 seconds
1 link status changes since last clear
Last clearing of "show interface" counters never
5 minutes input rate 0 bps (0.0% with framing overhead), 0 packets/sec
5 minutes output rate 0 bps (0.0% with framing overhead), 0 packets/sec
0 packets input, 0 bytes
Received 0 broadcasts, 0 multicast
0 runts, 0 giants
0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards
0 PAUSE input
0 packets output, 0 bytes
Sent 0 broadcasts, 0 multicast
0 output errors, 0 collisions
0 late collision, 0 deferred, 0 output discards
0 PAUSE output
 
T

tsteine

Active Member
May 15, 2019
171
83
28
I can confirm Mellanox ConnectX-3 cards work just fine with Arista switches, would be weird if they didn't.

I have CX354A cards connected to a DCS-7050Q-16 working just fine.

I have found that the switch is somewhat finnicky about auto negotiating though.

For running config, I need this set for the particular interface.
interface Ethernet5/1
speed forced 40gfull


Code: 
Ethernet5/1 is up, line protocol is up (connected)
  Hardware is Ethernet, address is 001c.737a.11ce (bia 001c.737a.11ce)
  Ethernet MTU 9214 bytes , BW 40000000 kbit
  Full-duplex, 40Gb/s, auto negotiation: off, uni-link: n/a
  Up 6 days, 15 hours, 48 minutes, 4 seconds
  Loopback Mode : None
  59 link status changes since last clear
  Last clearing of "show interface" counters never
  5 minutes input rate 371 kbps (0.0% with framing overhead), 60 packets/sec
  5 minutes output rate 431 kbps (0.0% with framing overhead), 70 packets/sec
     4927076693 packets input, 7246210623813 bytes
     Received 143634 broadcasts, 1772 multicast
     0 runts, 0 giants
     0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards
     0 PAUSE input
     4860036131 packets output, 6970179125149 bytes
     Sent 2269961 broadcasts, 1627207 multicast
     0 output errors, 0 collisions
     0 late collision, 0 deferred, 48658 output discards
     0 PAUSE output
 
M

Meng Wang

New Member
Dec 19, 2018
9
0
1
tsteine said:
I can confirm Mellanox ConnectX-3 cards work just fine with Arista switches, would be weird if they didn't.

I have CX354A cards connected to a DCS-7050Q-16 working just fine.

I have found that the switch is somewhat finnicky about auto negotiating though.

For running config, I need this set for the particular interface.
interface Ethernet5/1
speed forced 40gfull


Code: 
Ethernet5/1 is up, line protocol is up (connected)
  Hardware is Ethernet, address is 001c.737a.11ce (bia 001c.737a.11ce)
  Ethernet MTU 9214 bytes , BW 40000000 kbit
  Full-duplex, 40Gb/s, auto negotiation: off, uni-link: n/a
  Up 6 days, 15 hours, 48 minutes, 4 seconds
  Loopback Mode : None
  59 link status changes since last clear
  Last clearing of "show interface" counters never
  5 minutes input rate 371 kbps (0.0% with framing overhead), 60 packets/sec
  5 minutes output rate 431 kbps (0.0% with framing overhead), 70 packets/sec
     4927076693 packets input, 7246210623813 bytes
     Received 143634 broadcasts, 1772 multicast
     0 runts, 0 giants
     0 input errors, 0 CRC, 0 alignment, 0 symbol, 0 input discards
     0 PAUSE input
     4860036131 packets output, 6970179125149 bytes
     Sent 2269961 broadcasts, 1627207 multicast
     0 output errors, 0 collisions
     0 late collision, 0 deferred, 48658 output discards
     0 PAUSE output
Click to expand...
Thanks for the confirmation. I connect with cables used to work good on infiniband switch, because I see both of infiniband and Arista ethernet switch require QSFP cables. Is it a cable issue? Does 40g ethernet switch require a special QSFP?
 
T

tsteine

Active Member
May 15, 2019
171
83
28
It has worked fine with Mellanox Coded transceivers and intel/mellanox coded QSFP+ DACs for me so far.

Depending on your arista device, you may need to create an enable3px file in the /mnt/flash folder (just the file, doesn't need any text content or anything)
I believe there are other commands to enable third party transceivers, but I don't know which model of switch you have.


Code: 
[admin@localhost flash]$ ls -la
total 1169944
drwxrwx--- 6 root eosadmin      4096 Apr 24 21:35 .
drwxr-xr-x 3 root root           100 Apr  7 09:46 ..
-r-xr-x--- 1 root eosadmin 598992604 May 16  2019 .boot-image.swi
drwxrwx--- 2 root eosadmin      4096 Jan  2  2019 .extensions
-rwxrwx--- 1 root eosadmin 598992604 May 16  2019 EOS-4.18.11M.swi
-rwxrwx--- 1 root eosadmin        28 May 16  2019 boot-config
-rwxrwx--- 1 root eosadmin      2494 May 16  2019 cf_16052019
drwxrwx--- 3 root eosadmin      4096 Feb 25 00:33 debug
-rwxrwx--- 1 root eosadmin         0 Jan  7  2019 enable3px
drwxrwx--- 3 root eosadmin      4096 Apr  7 09:48 persist
drwxrwx--- 3 root eosadmin      4096 Jan  2  2019 schedule
-rwxrwx--- 1 root eosadmin      3495 Apr 24 21:35 startup-config
-rwxrwx--- 1 root eosadmin         0 Jan  2  2019 zerotouch-config
[admin@localhost flash]$
 
M

Meng Wang

New Member
Dec 19, 2018
9
0
1
tsteine said:
It has worked fine with Mellanox Coded transceivers and intel/mellanox coded QSFP+ DACs for me so far.

Depending on your arista device, you may need to create an enable3px file in the /mnt/flash folder (just the file, doesn't need any text content or anything)
I believe there are other commands to enable third party transceivers, but I don't know which model of switch you have.


Code: 
[admin@localhost flash]$ ls -la
total 1169944
drwxrwx--- 6 root eosadmin      4096 Apr 24 21:35 .
drwxr-xr-x 3 root root           100 Apr  7 09:46 ..
-r-xr-x--- 1 root eosadmin 598992604 May 16  2019 .boot-image.swi
drwxrwx--- 2 root eosadmin      4096 Jan  2  2019 .extensions
-rwxrwx--- 1 root eosadmin 598992604 May 16  2019 EOS-4.18.11M.swi
-rwxrwx--- 1 root eosadmin        28 May 16  2019 boot-config
-rwxrwx--- 1 root eosadmin      2494 May 16  2019 cf_16052019
drwxrwx--- 3 root eosadmin      4096 Feb 25 00:33 debug
-rwxrwx--- 1 root eosadmin         0 Jan  7  2019 enable3px
drwxrwx--- 3 root eosadmin      4096 Apr  7 09:48 persist
drwxrwx--- 3 root eosadmin      4096 Jan  2  2019 schedule
-rwxrwx--- 1 root eosadmin      3495 Apr 24 21:35 startup-config
-rwxrwx--- 1 root eosadmin         0 Jan  2  2019 zerotouch-config
[admin@localhost flash]$
Click to expand...
Thanks. Our switch is Arista DCS-7050QX-32-F. Let us try enabling 3rd part transceivers.
 
L

Labs

Member
Mar 21, 2019
88
16
8
tsteine said:
It has worked fine with Mellanox Coded transceivers and intel/mellanox coded QSFP+ DACs for me so far.

Depending on your arista device, you may need to create an enable3px file in the /mnt/flash folder (just the file, doesn't need any text content or anything)
I believe there are other commands to enable third party transceivers, but I don't know which model of switch you have.


Code: 
[admin@localhost flash]$ ls -la
total 1169944
drwxrwx--- 6 root eosadmin      4096 Apr 24 21:35 .
drwxr-xr-x 3 root root           100 Apr  7 09:46 ..
-r-xr-x--- 1 root eosadmin 598992604 May 16  2019 .boot-image.swi
drwxrwx--- 2 root eosadmin      4096 Jan  2  2019 .extensions
-rwxrwx--- 1 root eosadmin 598992604 May 16  2019 EOS-4.18.11M.swi
-rwxrwx--- 1 root eosadmin        28 May 16  2019 boot-config
-rwxrwx--- 1 root eosadmin      2494 May 16  2019 cf_16052019
drwxrwx--- 3 root eosadmin      4096 Feb 25 00:33 debug
-rwxrwx--- 1 root eosadmin         0 Jan  7  2019 enable3px
drwxrwx--- 3 root eosadmin      4096 Apr  7 09:48 persist
drwxrwx--- 3 root eosadmin      4096 Jan  2  2019 schedule
-rwxrwx--- 1 root eosadmin      3495 Apr 24 21:35 startup-config
-rwxrwx--- 1 root eosadmin         0 Jan  2  2019 zerotouch-config
[admin@localhost flash]$
Click to expand...
New models are using a different method from what is read here on the forum: service unsupported-transceiver x
It is not clear how you get the code... maybe someone can share more details.
 
T

tsteine

Active Member
May 15, 2019
171
83
28
the code is given to you by Arista support apparently, to tie the use of unsupported transceivers to an organization.

the syntax is
service unsupported-transceiver <stringname> <8-digit hex key>

It is not clear to me whether this is required to be valid, or if you can simply fill in something random,
 
L

Labs

Member
Mar 21, 2019
88
16
8
thanks for the info. I didn't know this. I don't have any Arista switch to test it...
 
You must log in or register to reply here.
Top Bottom