T580-SO-CR; GS7200; Fiberstore 4xSFP:1xQSFP (Generic)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

AJXCR

Active Member
Jan 20, 2017
565
96
28
35
I've been around and around with attempting to establish a connection between two Chelsio T580-SO-CR cards and a Gnodal GS7200 switch. Because this is my first attempt with 10G/40G/fiber, I'm sure that I'm overlooking a minor detail. Unfortunately, however, to date I've been unable to identify exactly what that detail is.

Hardware:
GS7200
2x Chelsio T580-SO-CR
WS2016 DC Server
FreeNAS Server
2x Fiberstore.com 4x SFP:1x QSFP cables (generic)

On the switch I've:
Updated the firmware to version 2.6.4 per Gnodal's instructions
Configured the default IP and subnet
Configured the mgmt port IP and subnet
Enabled the ports to be used
Set the system clock
Created a port-channel/link aggregation for the two groups of 4 ports

On the cards I've:
Installed the latest version of Unified Wire from the Chelsio Website
Configured the IP address to be on the same subnet as the switch
Set the VLAN Identifier to match that of the switch

Observations:
I get absolutely nothing out of the FreeNAS machine really haven't even attacked that yet. Everything below describes interaction with the Windows machine.
Prior to installation of Unified Wire, I get green led indicators on the switch for each connected port and "show interfaces status" reports:
"Status= Connected; Duplex= Full; Speed= 10 Gbps; Negotiation= No-Negotiation"

Post installation of UW, I have no lit port indicators on the back of the switch and status= not connected.

"show interfaces description": Status= Up; Protocol= down
"show interfaces transceiver": Status= present; Name= Fiberstore; Part Number= QSDP-4SFP10G-DAC

"show etherchannel summary":
Port-channel Module Admin Status is enabled
Port-channel Module Oper Status is enabled
Port-channel System Identifier is **:**:**:**:**:**

Flags:
D - down P - in port-channel
I - stand-alone H - Hot-standby (LACP only)
U - in-use

Number of channel-groups in use: 1
Number of aggregators: 1

Group Port-channel Protocol Ports
-------------------------------------------------------------------------
1 Po1(D) LACP Eth0/1(D),Eth0/2(D),Eth0/3(D),Eth0/4(D)

"GNOS# show interfaces port-channel 1":

po1 up, line protocol is down (not connect)
Bridge Port Type: Customer Bridge Port

Hardware Address is **:**:**:**:**:**
MTU 1500 bytes

"GNOS# show interfaces etherchannel":


Port : Eth0/1
-------------

Port State = Down, Not in Bundle
Channel Group : 1
Mode : Active
Port-channel = Null
Pseudo port-channel = Po1
LACP port-priority = 128
LACP Wait-time = 2 secs
LACP Port Identifier = 1
LACP Activity : Active
LACP Timeout : Long

Aggregation State : Aggregation, Defaulted

Port : Eth0/2
-------------

Port State = Down, Not in Bundle
Channel Group : 1
Mode : Active
Port-channel = Null
Pseudo port-channel = Po1
LACP port-priority = 128
LACP Wait-time = 2 secs
LACP Port Identifier = 2
LACP Activity : Active
LACP Timeout : Long

Aggregation State : Aggregation, Defaulted

Port : Eth0/3
-------------

Port State = Down, Not in Bundle
Channel Group : 1
Mode : Active
Port-channel = Null
Pseudo port-channel = Po1
LACP port-priority = 128
LACP Wait-time = 2 secs
LACP Port Identifier = 3
LACP Activity : Active
LACP Timeout : Long

Aggregation State : Aggregation, Defaulted

Port : Eth0/4
-------------

Port State = Down, Not in Bundle
Channel Group : 1
Mode : Active
Port-channel = Null
Pseudo port-channel = Po1
LACP port-priority = 128
LACP Wait-time = 2 secs
LACP Port Identifier = 4
LACP Activity : Active
LACP Timeout : Long

Aggregation State : Aggregation, Defaulted

LACP Port Admin Oper Port Port
Port State Priority Key Key Number State
---------------------------------------------------------------------
Eth0/1 Down 128 1 1 0x1 0xa2
Eth0/2 Down 128 1 1 0x2 0xa2
Eth0/3 Down 128 1 1 0x3 0xa2
Eth0/4 Down 128 1 1 0x4 0xa2

Port-channel : Po1
-------------------

Number of Ports = 4
HotStandBy port = null
Port state = Port-channel Ag-Not-Inuse
Protocol = LACP
Aggregator-MAC **:**:**:**:**:**
Maximum number of Ports = 8
 

i386

Well-Known Member
Mar 18, 2016
4,220
1,540
113
34
Germany
2x Fiberstore.com 4x SFP:1x QSFP cables (generic)
You mean this is a fanout cable with qsfp on one end and four sfp+ "plugs" on the other end?

If yes these cables don't work with network adapters, only swithes with qsfp ports and the support for it (qsfp plug goes to switch and sfp goes to the nodes/hosts)
 

AJXCR

Active Member
Jan 20, 2017
565
96
28
35
You mean this is a fanout cable with qsfp on one end and four sfp+ "plugs" on the other end?

If yes these cables don't work with network adapters, only swithes with qsfp ports and the support for it (qsfp plug goes to switch and sfp goes to the nodes/hosts)

Are you 100% confident in this? Chelsio actually has the option to configure the T580 QSFP port as 4x10SFP ports and sells a spider cable... How else would you use the QSFP to 4x SFP option?
 

abq

Active Member
May 23, 2015
675
204
43
Are you 100% confident in this? Chelsio actually has the option to configure the T580 QSFP port as 4x10SFP ports and sells a spider cable... How else would you use the QSFP to 4x SFP option?
Did you ever get the 40gb qdr card to fan out to 4x separate 10gb sfp+ ports? ...this would be an awesome discovery for cheap home switch!
 

AJXCR

Active Member
Jan 20, 2017
565
96
28
35
Did you ever get the 40gb qdr card to fan out to 4x separate 10gb sfp+ ports? ...this would be an awesome discovery for cheap home switch!
I honestly haven't spent a minute of time on it since then.. Both Chelsio's documentation and the hardware they sell suggest it should work. I think I may have been going about it wrong though.. the card should probably be set to operate in 4 x 10G ports mode via the powershell script they provide, and then the 10G links should be aggregated on both sides. I was trying to leave the card in 40G mode and only aggregate on the switch side.
 

abq

Active Member
May 23, 2015
675
204
43
I honestly haven't spent a minute of time on it since then.. Both Chelsio's documentation and the hardware they sell suggest it should work. I think I may have been going about it wrong though.. the card should probably be set to operate in 4 x 10G ports mode via the powershell script they provide, and then the 10G links should be aggregated on both sides. I was trying to leave the card in 40G mode and only aggregate on the switch side.
Thank You, I do not think I have found anyone who actually got switch mode to work.
 

AJXCR

Active Member
Jan 20, 2017
565
96
28
35
Just a quick update on this..

After an immense amount of time and frustration trying to get this setup to work I finally threw up my hands and ordered the following items from Fiberstore:

2x 3m (10ft) MTP Female to 4 LC UPC Duplex 8 Fibers OM3 50/125 Multimode Harness Cable, Polarity B, Elite, LSZH Bunch #41460

2x Customized 40GBASE-SR4 QSFP+ 850nm 150m MTP/MPO Transceiver for MMF Compatible Brands: Generic #48558

8x Customized 10GBASE-SR SFP+ 850nm 300m DOM Transceiver Compatible Brands: Mellanox #50000

Upon receiving the items and installing the first cable, I'm immediately showing "connected" on RHEL 7.3 :)
 

Jerry Renwick

Active Member
Aug 7, 2014
200
36
28
43
Congrats on the new setup! Some brand switches really have the compatibility issue with DAC cables.
 

nephri

Active Member
Sep 23, 2015
541
106
43
45
Paris, France
Glad it works now.

Do you think it was transceivers or cables issues (or both) ?

I'm really waiting the EU shop of FS.com, i have some bought to do :p
 

AJXCR

Active Member
Jan 20, 2017
565
96
28
35
Well, I was highly convinced that it was the transceiver on the card side as, unlike the switch, it would not recognize the transceiver at all.. Based on what seems to now be working, however, I'm not so sure. I'll be ordering more cables for various purposes and may try a few different combinations.