Help understanding 10G Compatability, did I buy the wrong thing?

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
S

surfrock66

New Member
Nov 20, 2023
9
1
3
I've been on a recent journey to upgrade my home network a bit, hoping to get 10G fiber between my NAS and my hypervisors. I thought I was being careful when choosing parts for compatibility, but I've just finished wiring up 6 links and 0 are showing as up on either end, making me think I have some fundamental misunderstanding which needs to be addressed.

I got a Brocade ICX6610 as my switch, which supports 8 10G SFP+ slots in the front.

I purchased 16 SFP's, both of which are Brocade 10Gb 57-0000075-01:

www.ebay.com

Brocade 10Gb 57-0000075-01 10G-SFP-SR 10Gbase-sr/sw SFP+ Fiber Optic Transceiver | eBay

· Hot-pluggable SFP+ footprint. Such as SFP/ SFP+/SFP28/ QSFP/ QSFP28/ CFP/ HBA / Network Card/ Raid Card/ Expander Card/ AOC/ DAC/ Raspberry Pi etc. All the products manufactured strictly carry out international and national relevant technical standards.
www.ebay.com

Lastly, I purchased 3 QLogic Broadcom Dual Port 10GB SFP+ Network Adapter Card PCIe BCM957810A1006G:

www.ebay.com

QLogic Broadcom Dual Port 10GB SFP+ Network Adapter Card PCIe BCM957810A1006G | eBay

We stand behind our products. The listed prices do not apply.
www.ebay.com

I do get an optical light out of the SFP on the brocade switch, and out the fiber cable at the other end of it, but I'm not sure if I'm seeing that at the other end. In my proxmox hosts and truenas server, I see the 10G nics, but the links are showing down (they're down on brocade as well).

On the proxmox hosts, if I do dmidecode I'm not seeing SFP info, and ethtool isn't helping either. I also tried unclipping and swapping the fiber cables in case they were crossed, no dice. My theory at this point is the PCI cards are NOT as compatible as I thought they were with the SFPs. I'm over my head on determining compatability, I bought these a long time ago and don't remember where I got the info that they would work, but any guidance would be appreciated. If it's indeed the PCI cards, are there any recommendations on what I should go with?

If it's useful, Proxmox does see the 10G card, I just don't think it sees the SFP's. I can see all 6 SFPs on the brocade if I do "show media" on the ports.

1711774546140.png
 
C

Chriggel

Member
Mar 30, 2024
74
33
18
Not sure if I correctly got what you said. Do you have light on the SFPs in the switch AND on the SFPs in the NICs as well? If you don't, well, it will obviously never work and show link down on both ends.

What you're seeing there is how the PCIe slot enumeration works, ethernet, PCIe domain 6, slot 0, function 0 and 1. This has nothing to do with the SFPs, at least not directly. I'm not familiar with Proxmox, it says No under Active and Autostart, I guess this is a switch and you can turn it on so that it says Yes instead?
 
S

surfrock66

New Member
Nov 20, 2023
9
1
3
I need to get into an awkward spot to look into the sfp's on the hosts, so I'm not sure, I'll be checking again tonight. For sure the ones in the switch have light. I am just wondering if there's a way to confirm comparability between the HBA card and those SFP's, I thought I had confirmed it but I think I'm too confused to trust my research.
 
C

Chriggel

Member
Mar 30, 2024
74
33
18
I tried to look that up and verify it for you, but with all the acquisitions that took place, parts of Brocade's network business went to QLogic and then Brocade as a whole went to Broadcom, but QLogic was acquired by Cavium and then Marvell, I'm not so sure anymore where you could look up compatibility for legacy hardware.

The Broadcom 57810 used to be the QLogic 57810, I think, and it's fairly old by now, it could have gotten compatibility with Brocade SFPs after the Brocade network acquisition by QLogic, but I don't know. I think it existed before that time.

Since you already have them anyway, it's probably for the best to ignore that for now, until you could find out if there's light on the SFPs in the NICs.
 
B

blunden

Active Member
Nov 29, 2019
492
155
43
surfrock66 said:
I need to get into an awkward spot to look into the sfp's on the hosts, so I'm not sure, I'll be checking again tonight. For sure the ones in the switch have light. I am just wondering if there's a way to confirm comparability between the HBA card and those SFP's, I thought I had confirmed it but I think I'm too confused to trust my research.
Click to expand...
Don't look into the light in the fiber as I'm pretty sure it can damage your eyes, at least the light from some types of transceivers.
 
S

surfrock66

New Member
Nov 20, 2023
9
1
3
2 different ports connecting 2 different devices, I have a light in the switch, but not from the other end. Looks like the cards and SFP's aren't compatible? I'm happy to buy new cards, but at this point I'd have no idea how to make sure I'm getting one to match the SFP's.
 

Attachments

T

Tech Junky

Active Member
Oct 26, 2023
370
124
43
Here's a silly thought that might apply... When adding thunderbolt to a PC you have to enable it in the uefi. Shouldn't really apply to a nic but, stranger things have happened.
 
T

TRACKER

Active Member
Jan 14, 2019
187
57
28
  • Like
Reactions: nexox
C

Chriggel

Member
Mar 30, 2024
74
33
18
surfrock66 said:
I am having trouble searching for some sort of support table of cards/SFP's, is that a thing? Or what kind of terms/acronyms am I looking to match up?
Click to expand...
It would be a thing, but like I've mentioned before, several acquisitions in the industry made it particularly difficult. Information gets lost along the way. It's a bit too late now, but it was probably not the best idea to get Brocade SFPs for the NICs. You don't need the same SFPs on both ends, so it would've been better to get other NICs and/or other SFPs for those.

Back to your screenshot though, it says the NIC isn't active. Is that normal in Proxmox and is it reasonable to expect light on a port that's disabled? I don't know anything about about Proxmox, but disabling the interfaces in the OS could also disable the SFP.
 
  • Like
Reactions: ColdCanuck
andrea87

andrea87

Member
Oct 15, 2022
63
86
18
36
North-east Italy
I would like to point out something very simple that you might have not considered. How is the cable arranged? Have you used the fiber optic cable as is or swapped the pair on one side? You might be bringing the "transmit" side of one sfp into the other.
 
S

surfrock66

New Member
Nov 20, 2023
9
1
3
andrea87 said:
I would like to point out something very simple that you might have not considered. How is the cable arranged? Have you used the fiber optic cable as is or swapped the pair on one side? You might be bringing the "transmit" side of one sfp into the other.
Click to expand...
I tried that with no luck, also no light seems to be coming through the sfp on the nic side, where it is on the switch side.

Chriggel said:
It would be a thing, but like I've mentioned before, several acquisitions in the industry made it particularly difficult. Information gets lost along the way. It's a bit too late now, but it was probably not the best idea to get Brocade SFPs for the NICs. You don't need the same SFPs on both ends, so it would've been better to get other NICs and/or other SFPs for those.

Back to your screenshot though, it says the NIC isn't active. Is that normal in Proxmox and is it reasonable to expect light on a port that's disabled? I don't know anything about about Proxmox, but disabling the interfaces in the OS could also disable the SFP.
Click to expand...
Not sure since this is the first time I've dealt with 10G nics on proxmox, but i'm seeing the same thing on TrueNAS Scale. I know there's things like dmidecode and ethtool that should show me SFP info and those are not, making me think the nic can't really read the SFP. I dunno, I'm a bit over my head here.

It seems I should either try to get some alternate SFP's, or Brocade NICS, and it's probably easier to go the SFP's then (Since I can hot swap them without turning the systems off)? A cursory ebay search for qlogic or broadcom SFP's didn't turn any up, is there something specific I should look for to make sure it's compatible with that Qlogic nic?
 
S

surfrock66

New Member
Nov 20, 2023
9
1
3
So I see this card, which is the same model as the one I got, but it comes with 2 SFP's:

www.ebay.com

Broadcom Dual Port 10GB SFP+ PCIe Card w/Two SFP Transceivers, BCM957810A1006G | eBay

TESTED, WORKS FINE.
www.ebay.com

Then that matches these SFP's which I think would work:

www.ebay.com

Finisar FTLX8571D3BCL SFP+SR/SW 10Gb/s 850nm Multimode SFP+ Transceiver Grade A | eBay

Original Finisar FTLX8571D3BCL 10GB SFP+ Optic Transceiver. Why Choose Us! We are here 24/7, should you need any help.
www.ebay.com

For $8 each, think it's worth a shot?
 
C

Chriggel

Member
Mar 30, 2024
74
33
18
Finisar is the OEM that was used by many server/storage/network vendors around the time when this was recent hardware. They're basically the generic SFPs of the time, working in many systems/components that weren't specifically coded to only accept certain SFPs.

Your combination of NICs and SFPs not working is either due to the NICs being coded to only accept certain SFPs, or the SFPs being specifically coded to only work in certain devices that can deal with it, like your switch. That NIC being sold with these Finisar SFPs suggests that it's either open for most SFPs or it's specifically made to accept those, so it should work if the seller really tested this combination and found that it works.

But in the absence of proper documention (or someone who still remembers which SFPs were shipped with these NICs when they were new), it's still a bit of a gamble, but probably worth trying.
 
  • Like
Reactions: surfrock66
You must log in or register to reply here.
Top Bottom