This was originally posted in the mega-thread (https://forums.servethehome.com/ind...powerful-10gbe-40gbe-switching.21107/page-300) if you are finding this from an internet search, great info there, but no responses to this issue/questions.
Questions:
Should I give up trying to get a Brocade 1020 NIC working?
Do I need to pickup something like a MCX354A-QCBT Mellanox CX354A ConnectX-3 VPI QDR IB 10GbE Dual-Port QSFP NIC? If so, why do some of the MCX345A-QCBT list themselves as 10Gb/40Gb, some as 10Gb only, and some as 40Gb only?
My situation:
I now have two ICX-6610, one of the Dell Networking Cable QSFP+ 40GBE - 4X SFP+ 10GBE 3M 27GG5 cables. All of the licenses are good and both units are online. The Dell cable works fine between the two ICX6610, but I'm having problems getting it working with the " Cavium QLogic 1010/1020/1007/1741 10Gbps CNA (rev 01)" installed in my Dell R910.
The card is visible (lspci):
05:00.2 Ethernet controller: Cavium QLogic 1010/1020/1007/1741 10Gbps CNA (rev 01)
05:00.3 Ethernet controller: Cavium QLogic 1010/1020/1007/1741 10Gbps CNA (rev 01)
But the links always show as down both on the server:
6: ens5f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
link/ether 8c:7c:ff:20:c0:32 brd ff:ff:ff:ff:ff:ff
altname enp5s0f2
7: ens5f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
link/ether 8c:7c:ff:20:c0:33 brd ff:ff:ff:ff:ff:ff
altname enp5s0f3
and on the switch:
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/2/7 Down None None None None No 1 0 cc4e.2484.0568
1/2/8 Down None None None None No 1 0 cc4e.2484.0568
1/2/9 Down None None None None No 1 0 cc4e.2484.0568
(the one connected between the switches at the moment is 1/2/10:
SSH@racksw1>sh int br e 1/2/10
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/2/10 Up Forward Full 10G None No 1 0 cc4e.2484.0568
)
The Dell breakout DAC
SSH@racksw1>sh media e 1/2/7
Port 1/2/7:Type : 40GBASE-Passive Copper
Vendor Name: Amphenol Serial Num: CN027GG554Q4JWR Revision: C
All of the QSFP ports are removed from stacking and stacking is disabled:
SSH@racksw1>sh conf
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disable
stack mac cc4e.2484.0568
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
hostname racksw1
ip dhcp-client disable
ip dns server-address 192.168.1.1
ip route 0.0.0.0/0 192.168.1.1
!
no telnet server
username root password .....
no snmp-server
!
!
clock timezone gmt GMT-07
!
!
ntp
disable serve
server 216.239.35.0
server 216.239.35.4
!
web-management https
hitless-failover enable
!
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/5
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
speed-duplex 10G-full
!
interface ve 1
ip address 192.168.1.58 255.255.255.0
!
!
end
I picked up the Brocade 1020 NIC because I read in a forum post that they just work. I have since found another post that describes them as finicky. Not sure where to go from here. Do I need to pickup something like a MCX354A-QCBT Mellanox CX354A ConnectX-3 VPI QDR IB 10GbE Dual-Port QSFP NIC? If so, why do some of the MCX345A-QCBT list themselves as 10Gb/40Gb, some as 10Gb only, and some as 40Gb only?
Questions:
Should I give up trying to get a Brocade 1020 NIC working?
Do I need to pickup something like a MCX354A-QCBT Mellanox CX354A ConnectX-3 VPI QDR IB 10GbE Dual-Port QSFP NIC? If so, why do some of the MCX345A-QCBT list themselves as 10Gb/40Gb, some as 10Gb only, and some as 40Gb only?
My situation:
I now have two ICX-6610, one of the Dell Networking Cable QSFP+ 40GBE - 4X SFP+ 10GBE 3M 27GG5 cables. All of the licenses are good and both units are online. The Dell cable works fine between the two ICX6610, but I'm having problems getting it working with the " Cavium QLogic 1010/1020/1007/1741 10Gbps CNA (rev 01)" installed in my Dell R910.
The card is visible (lspci):
05:00.2 Ethernet controller: Cavium QLogic 1010/1020/1007/1741 10Gbps CNA (rev 01)
05:00.3 Ethernet controller: Cavium QLogic 1010/1020/1007/1741 10Gbps CNA (rev 01)
But the links always show as down both on the server:
6: ens5f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
link/ether 8c:7c:ff:20:c0:32 brd ff:ff:ff:ff:ff:ff
altname enp5s0f2
7: ens5f3: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN mode DEFAULT group default qlen 1000
link/ether 8c:7c:ff:20:c0:33 brd ff:ff:ff:ff:ff:ff
altname enp5s0f3
and on the switch:
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/2/7 Down None None None None No 1 0 cc4e.2484.0568
1/2/8 Down None None None None No 1 0 cc4e.2484.0568
1/2/9 Down None None None None No 1 0 cc4e.2484.0568
(the one connected between the switches at the moment is 1/2/10:
SSH@racksw1>sh int br e 1/2/10
Port Link State Dupl Speed Trunk Tag Pvid Pri MAC Name
1/2/10 Up Forward Full 10G None No 1 0 cc4e.2484.0568
)
The Dell breakout DAC
SSH@racksw1>sh media e 1/2/7
Port 1/2/7:Type : 40GBASE-Passive Copper
Vendor Name: Amphenol Serial Num: CN027GG554Q4JWR Revision: C
All of the QSFP ports are removed from stacking and stacking is disabled:
SSH@racksw1>sh conf
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30tT7f3
!
stack unit 1
module 1 icx6610-48p-poe-port-management-module
module 2 icx6610-qsfp-10-port-160g-module
module 3 icx6610-8-port-10g-dual-mode-module
stack disable
stack mac cc4e.2484.0568
!
!
vlan 1 name DEFAULT-VLAN by port
router-interface ve 1
!
!
aaa authentication web-server default local
aaa authentication enable default local
aaa authentication login default local
hostname racksw1
ip dhcp-client disable
ip dns server-address 192.168.1.1
ip route 0.0.0.0/0 192.168.1.1
!
no telnet server
username root password .....
no snmp-server
!
!
clock timezone gmt GMT-07
!
!
ntp
disable serve
server 216.239.35.0
server 216.239.35.4
!
web-management https
hitless-failover enable
!
!
interface ethernet 1/3/1
speed-duplex 10G-full
!
interface ethernet 1/3/2
speed-duplex 10G-full
!
interface ethernet 1/3/3
speed-duplex 10G-full
!
interface ethernet 1/3/4
speed-duplex 10G-full
!
interface ethernet 1/3/5
speed-duplex 10G-full
!
interface ethernet 1/3/6
speed-duplex 10G-full
!
interface ethernet 1/3/7
speed-duplex 10G-full
!
interface ethernet 1/3/8
speed-duplex 10G-full
!
interface ve 1
ip address 192.168.1.58 255.255.255.0
!
!
end
I picked up the Brocade 1020 NIC because I read in a forum post that they just work. I have since found another post that describes them as finicky. Not sure where to go from here. Do I need to pickup something like a MCX354A-QCBT Mellanox CX354A ConnectX-3 VPI QDR IB 10GbE Dual-Port QSFP NIC? If so, why do some of the MCX345A-QCBT list themselves as 10Gb/40Gb, some as 10Gb only, and some as 40Gb only?
