Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[AgentXXL]

Thanks! I've ordered a CRS305 and one of the Mikrotik S+RJ10 SFP+ modules just in case my Wiitek doesn't work. I'll hopefully have success later this week when they arrive.

The Mikrotik CRS305 and S+RJ10 SFP+ module arrived and have been implemented. The S+RJ10 module and my existing Wiitek modules all seem to report the correct speeds but I'm going to use the S+RJ10 on my Aquantia 5Gbps NIC as it's the one causing me the most trouble. Still waiting for Asus to supply a working firmware update tool for the onboard NIC.

There's a firmware download from station-drivers.com that supposedly has a modified xml file. That file holds the system IDs that are valid for the updater tool to work. The modded file was supposed to work for onboard NICs on Asus motherboards, but apparently not for all Asus boards. Mine's a Prime x299 Deluxe II and it won't upgrade the NIC firmware even though it detects it and sees that the firmware is out of date.

But even as it stands, the Mikrotik is allowing me to get up to 3.5Gbps speeds in both directions vs the sub-1Gbps speeds in one direction when both systems were connected directly to the ICX6610 SFP+ ports. The speed will hopefully improve further once the NIC firmware is updated. I have my unRAID servers both connected to the Mikrotik and then a DAC cable to the ICX6610.

I'm still learning about all the features of the CRS305 - I'm running it with the SwitchOS firmware instead of RouterOS as my pfSense system handles my routing requirements. I'm trying to figure out if I should use port isolation or VLANs to segregate the 1Gbps management port from the 4 x SFP+ ports. Any suggestions?

Thanks again for the assistance provided so far!

 

[richtj99]

First - Thank you both so much for the help!

No, that wouldn't be a 'best practice' at all; you don't want to create VEs except in places where you need them. And yes, if there is no VE, then that VLAN is 'layer 2 only' as far as the ICX device is concerned, it cannot do any layer 3 work (routing) in that VLAN.

Thats good to know - no VE = Layer 2
VE = Possible layer 3 if configured?

Your first steps would be to identify the VLANs that you want the ICX #1 to be able to route; in each of those VLANs, create a VE in ICX #1. On ICX #1, add a default route to the 'upstream' router (whatever you are using) so that it can route traffic that is *not* destined for those VLANs to something else which can handle it.

So I only have two Vlans that are using 10gb - Vlan 10 & Vlan 20.

Vlan 10 is a "no internet" wan (with certain firewall exceptions) and it is also restricted to Vlan 10.
Vlan 20 has full access to Vlan 10, Wan, and all other Vlans.

As the firewall (sonicwall) is restricting traffic from Vlan 10 to everywhere, would the routing still work with firewall rules?

Or does the switch ignore firewall rules as it is doing the routing before we get to the firewall?

Next step would be to reconfigure at least two hosts to use the VE addresses as their 'default router' or 'default gateway' instead of the upstream router's addresses. With that done, those hosts will send cross-VLAN traffic to ICX #1 for routing, instead of the upstream router. If ICX #1 can route the traffic directly it will, if it cannot it will send the traffic to the upstream router.

That might actually answer my above firewall rule question - at least partially.

So Vlan 10 (VE 192.168.1.251 - Brocade switch (#1) gets set as the gateway within the Vlan (can be done on the router itself or use a windows dhcp, or doesnt matter?) If 192.168.1.251 is unable to route the traffic, it sends it forward to the sonicwall?

As far as restricting access to the management interfaces through the VEs, that can be be done other ways, including access-groups and probably other methods. If *any* IP address on an ICX is reachable from a host, even if it doesn't go through a VE on the same VLAN as the host, then the management interfaces are reachable, so just avoiding creation of a VE in that VLAN won't be sufficient to block that type of access.

I have one VE setup on switch (#1), on Vlan 20 - anyone on Vlan 20 can ping the switch or access it. I also have an IP helper setup for DHCP on some vlan (including Vlan 10). Firewall rules block Vlan 10 to Vlan 20 traffic, though Vlan 20 does give IP's - could someone on Vlan 10 access the Vlan 20 VE management?

I added a VE on Vlan 10 specifically for LibreNMS to access but Im not sure how to enable SNMP on Vlan 10 without enabling management.

Thanks,
Rich

 

[richtj99]

Again thank you for taking the time to spoon feed me. I think I am slowly getting it.

Following kpfleming's post, here's a basic L3 config I mocked up on a 7150:

Spoiler

ICX7150-C12 Router#sh run
Current configuration:
!
ver 08.0.95eT213
!
stack unit 1
  module 1 icx7150-c12-poe-port-management-module
  module 2 icx7150-2-copper-port-2g-module
  module 3 icx7150-2-sfp-plus-port-20g-module
  stack-port 1/3/1
  stack-port 1/3/2
!
global-stp
vlan 1 name DEFAULT-VLAN by port
spanning-tree
!
vlan 11 by port
tagged ethe 1/1/1
untagged ethe 1/1/11
router-interface ve 11
!                                                           
vlan 12 by port
tagged ethe 1/1/1
untagged ethe 1/1/3
router-interface ve 12
!
vlan 20 by port
tagged ethe 1/1/1
router-interface ve 20
!
ip dhcp-client disable
ip route 0.0.0.0/0 172.16.21.2
!

!                                                           
interface ve 11
ip address 10.100.11.1 255.255.255.0
!
interface ve 12
ip address 10.100.12.1 255.255.255.0
ip helper-address 1 10.100.11.2
!
interface ve 20
ip address 172.16.21.1 255.255.255.0
end

In the example, VLAN/VE 20 is the transit to the router; the router needs to have reverse routes on it for each subnet that the switch is handling.

So if Brocade #1 is what connects to your router, you configure every VLAN and VE there and each subnet uses Brocade #1 VE IPs as gateway, along with the default route to the firewall. Your downstream switches just have the required VLANs configured and you trunk them back to #1. The port to your firewall, if physical, would be an untagged port in the transit VLAN.

So I need to add routes on the sonicwall for the two Vlans (10 & 20) which are 10gb -or do I need to add routes for every vlan (1gb also)? I think I can specify gateways on each Vlan on the sonicwall - point those to the Brocade (#1)?

It sounds like all other switches below (#1) can be left alone assuming no VE interface?

The ip helper-address statement is because I run a single DHCP server for all pools.

I am using the IP helper address statement only on my sonicwall for a few specific vlans - Should I also add that statement into the Brocade?
Or
Does that need to be added to the brocade as the sonicwall is no longer the gateway therefore devices on the network need to look at the brocade for the IP helper address?

For management on downstream switches, define a VE for the VLAN that normally does management, it will make writing ACLs easier to only worry about one interface in cases where switches don't.

Then as kpfleming said, you would use ACLs to prevent VLAN 40 members from hitting the management interfaces, assuming that VLAN 40 needs routing as well.

I would like to learn more about how ACL's work on the brocades. I will start to google that now!

Would the ACL's replace firewall rules if the switch is doing the routing?

If I currently dont allow Vlan 10 to Vlan 20 traffic via firewall, do I need to make that rule on the switch vs the firewall?

Thanks,
Rich

 

[LodeRunner]

No problem on the basic config; everyone has to start somewhere and I had that laying around from some other L3 related posts.

Routing: correct. If you use subnets in the same RFC1918 space, then you could do one route entry on the SonicWALL, for example:
SonicWALL IP: 172.16.24.1/30
ICX Router IP: 172.16.24.2/30
VLAN 10: 192.168.10.0/24
VLAN 20: 192.168.20.0/24

Then the reverse route from SW to ICX would be 192.168.0.0/16 (the entire RFC1918 block) via 172.16.24.2.
Or you write two separate routes: 192.168.10.0/24 via 172.16.24.2 and 192.168.20.0/24 via 172.16.24.2

For sanity, I always place my transit VLANs in an entirely separate RFC1918 space than the LANs being served.

DHCP: I don't know how well a SonicWALL handles being a DHCP server for things that are not in the subnet of one of its physical or sub (VLAN) interfaces. I know pfSense and opnSense do not do it properly (you can't configure a DHCP pool for a subnet that's not on one of its interfaces). I run Windows servers with Active Directory and AD integrated DHCP and DNS. Windows DHCP is standard compliant and will happily serve address pools that are not an interface local subnet. Pretty sure ISC DHCP on Linux does the same, but I've never used or configured it.

Because the ICX will be making all routing decisions, then each VLAN that needs DHCP service that is not the VLAN the DHCP server is in need an ip helper-address statement pointing to the IP of the DHCP server.

Firewall rules:
The SonicWALL would be handling all WAN > LAN and LAN > WAN traffic, so you still do rules there for that; and you just make address objects as per usual to attach to the policies.

ACLs on the switch in your case would primarily be for inter-VLAN traffic, though you could use them to drop certain traffic before even forwarding it to the SonicWALL. Rules on the SonicWALL would not be able to touch traffic between VLANs as the switch is making those routing decisions and the SW never sees the traffic to even process. So traffic between VLAN 10 and 20 in this setup would be controlled only by switch ACLs.

 

[ixen]

Hi,
Today I've updated my 7250 from 8.0.80e to 8.0.95f. After the update everything seems to be working properly, except for the connections on 10Gbit ports using Mikrotik's S+RJ10. Servers on those ports reports link going up, down, up, down several times per second.
It was working properly before the upgrade. Anyone encountered similar issue?

Just before ending up with 8.0.95f I tried upgrading to 9.0.10a, but there were too many suspicious boot warnings about missing files, problems with bringing some bridge up and some issues with tun device - hence the 8.0.95f.

 

[kpfleming]

Note that it is not strictly necessary to use IP helpers. It's completely acceptable for *both* the ICX and the SonicWALL to have interfaces/addresses in each and every VLAN. The *clients* decide which of them will be used for routing, not the routers.

So for example you can have the SonicWALL have an IP address/interface on each VLAN and use that for providing DHCP service to that VLAN, but the DHCP server would be configured to provide the IP address of the *ICX* to clients that should use the ICX for routing. The routing and DHCP functions are completely distinct, even when they live on the same box, and they can be used quite flexibly, if the DHCP server's configuration mechanism will allow it.

You're trading one type of complexity (IP helpers) for another, so it's up to you to decide which you prefer.

 

[bitbckt]

Today I've updated my 7250 from 8.0.80e to 8.0.95f. After the update everything seems to be working properly, except for the connections on 10Gbit ports using Mikrotik's S+RJ10. Servers on those ports reports link going up, down, up, down several times per second.

I had the same issue on 8.0.95 and switched transceivers to some FS.com units. That both resolved the problem and lowered temps a bit.

 

[deeceesth]

No traffic is flowing. I have two WAN interfaces, one is dedicated for my homelab stuff and one is for general users. When I power cycle one of the modems only the WAN that I didn't touch stays up.

After reboot, does traffic still flow to the internet without trying to force a DHCP refresh? Most ISP leases are long enough to cover a CPE restart, so pfSense won't bother refreshing a lease that's more than 50% of the lease time left; it's following RFC.

 

[crackelf]

mellanoxeseses (and most NICs) don't support qsfp breakout, it's a single connection of either 40gbe or 10gbe. If you want to drop it to a 10gbE SFP connection, search ebay for qsfp > sfp adapter and stick that in the NIC - Mellanox MAM1Q00A-QSA 655874-B21 40G QSFP+ To 10G SFP+ Network Cable Adapter | eBay

Fantastic thank you for the resources and info.

I'll move this over to this thread if needed, but I just got these ConnectX-4 Lx in the mail, and as you mentioned

They work perfectly and don't need a driver download from mellanox, debian has and will have the mlx kernel driver included in it like, forever probably. Removing that would be like removing the Intel ixgb driver

edit: I moved this over to another thread
quick answer for anyone looking: echo 8 > /sys/devices/pci0000\:00/0000\:00\:1d.0/0000\:03\:00.0/sriov_numvfs

 

[simbo]

Got a small problem where I can't get the 10GB breakout interfaces to come up. I'm trying to brake out the 40GB to 10GB LCs using some FS.com gear. I've followed the setup guide for removing the ports from the stack.

To get the 10G LCs on the front of the rack, I've purchased the following:
- QSFP-SR4-40G Brocade 40G-QSFP-SR4 Compatible 40GBASE-SR4 QSFP+ 850nm 150m DOM MTP/MPO MMF Optical Transceiver Module
- 12FMTPOM4 1m (3ft) MTP®-12 (Female) to MTP®-12 (Female) OM4 Multimode Elite Trunk Cable, 12 Fibers, Type B, Plenum (OFNP), Magenta
- FHD-1MTP4LCDOM4 FHD MTP®-8 Cassette, 8 Fibers OM4 Multimode, Universal Polarity, MTP® to 4 x LC Duplex (Aqua), 0.35dB max

I'm using FS.com 10GB 850nm SR transceiver's on the other end with out any problems over OM4 cables.

SSH@sw-core(config)#show media validation


Port       Supported Vendor               Type
----------------------------------------------------------------------
1/2/2      Yes       BROCADE              40G QSFP-SR4
1/2/3      Yes       BROCADE              40G QSFP-SR4
1/2/4      Yes       BROCADE              40G QSFP-SR4
1/2/5      Yes       BROCADE              40G QSFP-SR4
1/2/7      Yes       BROCADE              40G QSFP-SR4
1/2/8      Yes       BROCADE              40G QSFP-SR4
1/2/9      Yes       BROCADE              40G QSFP-SR4
1/2/10     Yes       BROCADE              40G QSFP-SR4
1/3/3      Yes       FS                    Type  : 10GE  Passive Twinax  1m (SFP +) (Not supported)
1/3/4      Yes       FS                    Type  : 10GE  Passive Twinax  1m (SFP +) (Not supported)
1/3/5      Yes       FS                    Type  : 10GE  Passive Twinax  5m (SFP +) (Not supported)
1/3/6      Yes       FS                    Type  : 10GE  Passive Twinax  5m (SFP +) (Not supported)
1/3/7      Yes       OEM                   Type  : 10GE SR 300m (SFP +)

1/3/7 is an optic chucked into the front port used to test the LC cables. The other 1/3's are LAGs go to other switchgear.

SSH@sw-core(config)#show conf
!
Startup-config data location is flash memory
!
Startup configuration:
!
ver 08.0.30uT7f3
!
stack unit 1
  module 1 icx6610-48p-poe-port-management-module
  module 2 icx6610-qsfp-10-port-160g-module
  module 3 icx6610-8-port-10g-dual-mode-module
stack disable

FWIW, I set the 1/2/2 to 1/2/5 and 1/2/7 to 1/2/10 to speed-duplex 10G-full also left blank.

Here's the interface summary

1/2/1      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/2/2      Down    None    None None  None  No  1    0   748e.f8fe.c148
1/2/3      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/2/4      Down    None    None None  None  No  1    0   748e.f8fe.c148
1/2/5      Down    None    None None  None  No  1    0   748e.f8fe.c148
1/2/6      Down    None    None None  None  No  1    0   748e.f8fe.c148
1/2/7      Down    None    None None  None  Yes 1    0   748e.f8fe.c148  SW-XXX1
1/2/8      Down    None    None None  None  Yes 1    0   748e.f8fe.c148  SW-XXX2
1/2/9      Down    None    None None  None  Yes 1    0   748e.f8fe.c148  SW-XXX3
1/2/10     Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/3/1      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/3/2      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/3/3      Up      Forward Full 10G   3     Yes 1    0   748e.f8fe.c148  SW-DXX-1
1/3/4      Up      Forward Full 10G   3     Yes 1    0   748e.f8fe.c148  SW-DXX-2
1/3/5      Up      Forward Full 10G   2     Yes 1    0   748e.f8fe.c148  SW-RXX-1
1/3/6      Up      Forward Full 10G   2     Yes 1    0   748e.f8fe.c148  SW-RXX-2
1/3/7      Down    None    None None  None  Yes 1    0   748e.f8fe.c148
1/3/8      Down    None    None None  None  Yes 1    0   748e.f8fe.c148

Here's the interface that has a plugged in LC cable:

SSH@sw-core(config)#show int ethe 1/2/7
  10GigabitEthernet 1/2/7 is down, line protocol is down
  Port down for 2 hour(s) 25 minute(s) 13 second(s)
  Hardware is   10GigabitEthernet , address is 748e.f8fe.c148 (bia 748e.f8fe.c17f)
  Configured speed 10Gbit, actual unknown, configured duplex fdx, actual unknown
  Configured mdi mode AUTO, actual unknown
  Member of 11 L2 VLANs, port is dual mode in Vlan 1, port state is BLOCKING
  BPDU guard is Disabled, ROOT protect is Disabled, Designated protect is Disabled
  Link Error Dampening is Disabled
  STP configured to ON, priority is level0, mac-learning is enabled
  Openflow is Disabled, Openflow Hybrid mode is Disabled,  Flow Control is config enabled, oper enabled, negotiation disabled
  Mirror disabled, Monitor disabled
  Mac-notification is disabled
  Not member of any active trunks
  Not member of any configured trunks
  Port name is SW-XXX1
  MTU 10200 bytes, encapsulation ethernet
  300 second input rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  300 second output rate: 0 bits/sec, 0 packets/sec, 0.00% utilization
  54035 packets input, 15271970 bytes, 0 no buffer
  Received 2649 broadcasts, 3205 multicasts, 48181 unicasts
  0 input errors, 0 CRC, 0 frame, 0 ignored
  0 runts, 0 giants
  129068 packets output, 41755954 bytes, 0 underruns
  Transmitted 11005 broadcasts, 55975 multicasts, 62088 unicasts
  0 output errors, 0 collisions
  Relay Agent Information option: Disabled

Egress queues:
Queue counters    Queued packets    Dropped Packets
    0              115303                   0
    1                   0                   0
    2                   0                   0
    3                   0                   0
    4                   0                   0
    5                   1                   0
    6                   0                   0

SSH@sw-core>show ver
  Copyright (c) 1996-2016 Brocade Communications Systems, Inc. All rights reserved.
    UNIT 1: compiled on Apr 23 2020 at 13:17:12 labeled as FCXR08030u
                (10545591 bytes) from Primary FCXR08030u.bin
        SW: Version 08.0.30uT7f3
  Boot-Monitor Image size = 370695, Version:10.1.00T7f5 (grz10100)
  HW: Stackable ICX6610-48-HPOE
==========================================================================
UNIT 1: SL 1: ICX6610-48P POE 48-port Management Module
         Serial  #: 2ax5o2jk68e
         License: ICX6610_ADV_ROUTER_SOFT_PACKAGE   (LID: H4CKTH3PLN8)
         P-ENGINE  0: type E02B, rev 01
         P-ENGINE  1: type E02B, rev 01
==========================================================================
UNIT 1: SL 2: ICX6610-QSFP 10-port 160G Module
==========================================================================
UNIT 1: SL 3: ICX6610-8-port Dual Mode(SFP/SFP+) Module
==========================================================================
  800 MHz Power PC processor 8544E (version 0021/0023) 400 MHz bus
65536 KB flash memory
  512 MB DRAM
STACKID 1  system uptime is 5 day(s) 1 hour(s) 38 minute(s) 27 second(s)
The system started at 12:06:39 GMT+10 Sun Feb 20 2022

The system : started=cold start

I've double checked that the cables are plugged in. I have 2 x of the above kit and I get the same behaviour on each 40GQSFP, MTP cable and breakout cassette.

Any ideas on what to try next?

 

[fohdeesha]

The IP doesn't get lost by pfsense while in use. I was just doing some robustness testing of my setup and noticed when I power cycle my modem I can't get pfsense to get a new lease no matter what I do. I have to do that synchronized power cycle and renew in order for it to work.
The modem is just a straight modem and I get a public IPv4, not funky DMZ or anything. I get a new IP once in a while but not every time. its just standard residential cable.

If I use one of the onboard intel NICs on my pfsense box I don't see any of these issues.

It's most likely a bug (or just...a not implemented feature) in your modem's firmware that's making it not send out a dhcp force renew broadcast (FORCERENEW ) on fresh boot, which gives clients new fresh leases even when the end client doesn't explicitly ask for one.

The reason you don't run into the bug/issue when the modem is directly connected to your pfsense box is because when you reboot the modem and its directly connected to your pfsense box, pfsense sees the physical link completely drop and come back up - this will obviously force a dhcp release, then a renew request on pfsense's behalf when the link is back up - it doesn't need the dhcp server side to broadcast a force renew.

when a switch is in between the two and you reboot the modem, the pfsense box doesn't see any link flaps, and will gladly hold onto the DHCP lease it was given earlier until either the lease timer expires, it sees a FORCERENEW broadcast, or you manually bounce the lease. A nicely coded modem immediately sends a "dhcp force renew" on fresh boot, so any clients on its connected l2 with old slowly expiring leases will immediately drop them and grab a new lease. I had an old docsis modem that didn't have forcerenew implemented, and yes it was annoying. my current DOCSIS modem (CM600) has it implemented and works great running through an ICX then to a virtual pfsense instance when rebooted, as an example

 

[AgentXXL]

Another question for the network gurus here... probably more of a Mikrotik question but since it's connected to the ICX6610 I'm posting it in this thread.

Installing the Mikrotik CRS305 as a 'rate converter' for the onboard Aquantic 5Gbps NIC in one of my systems has provided better transfer speeds, although it's still not as good as I expect. That's likely due to the older firmware on the onboard NIC - I'm still waiting on Asus to provide a working firmware updater.

Alas I'm seeing frequent 'disconnects' from the system with the 5Gbps NIC. I notice it when watching content via Plex and randomly playback will halt. I sometimes get the familiar 'buffering' message from the Plex client about my network not being fast enough, other times it just sits there. If I try to ping the media server, it won't respond. Going to the management page for the Mikrotik, it shows the link as up, but I can't seem to figure out why it stops responding.

My solution has been to reboot the Mikrotik, which in SwOS takes less than 20 seconds. After the reboot completes, the system in question 'reconnects' automatically and Plex playback resumes. I've also had this happen during file transfers between systems. With those I usually have to restart the transfer from where it left off.

While my speeds were much worse when directly connected to the ICX6610, I wasn't seeing these disconnects. I went with the Mikrotik due to the 6610 not supporting 2.5 and 5Gbps NBase-T with most of the common SFP+ to RJ45 modules. I purchased a Mikrotik S+RJ10 module with the CRS305 as recommended, and it shows as a 5Gbps link, but so did my Wiitek modules when I tried them.

If anyone has any suggestions on how to troubleshoot this, or know if there are options I can configure on the Mikrotik to try and mitigate this issue, please let me know. Thanks!

 

[crackelf]

Does anyone know if you can put 1x40G modules in all three of the module slots on the 7450? I'm not seeing an option to set the front module slot to 1x40G qsfp, only 4x10g sfp. Tagging @LodeRunner but any 7450 owners might have ran into this already too.

 

[fohdeesha]

Does anyone know if you can put 1x40G modules in all three of the module slots on the 7450? I'm not seeing an option to set the front module slot to 1x40G qsfp, only 4x10g sfp. Tagging @LodeRunner but any 7450 owners might have ran into this already too.

on the 24 port chassis yes on the 48 port chassis no. 7450 big dumb doodoo switch with 9 million dollar 4x10g modules

 

[crackelf]

on the 24 port chassis yes on the 48 port chassis no. 7450 big dumb doodoo switch with 9 million dollar 4x10g modules

Biggest face palm ever. Why?! I...nevermind. Thank you hahaha I guess I'm on the hunt for a 24p variant of this bad boy. Really liking this switch otherwise though.

 

[frogtech]

@fohdeesha

looks like the latest firmware for the fastiron on ICX platforms has a completely revamped GUI, version 9.0.10a

i know in your informational posts you commended the ICX7250 platform because it used an honor based licensing system, but i was never sure if that was only because it was on v8080 firmware (which i assume translates to 8.0.80 in ruckus firmware notation)

was it ever planned by ruckus/brocade to phase out the honor based licensing for these switches?

i'm aware you may not be the biggest fan of gui for switches but it will be nice to integrate these with my ruckus r500 which have a similar interface aesthetic

 

[manutech]

Only if you can find the specific SFP+ module that's known to work, specifically the Supermicro AOM-AQS-107-B0C2-CX. Be careful - there are many knock-offs that claim to be 100% compatible but all that I've found all use the Marvell chip vs the Supermicro which uses the Aquantia AQS-107. It appears to have a larger buffer which some report works properly with switches like the 6610 that don't specifically allow any SFP+ ports to link at 2.5 or 5Gbps. See this post for more details:

https://forums.servethehome.com/index.php?threads/gbase-t-transceivers-performance-with-nbase-t-marvell-88x3300-v-s-aquantia-aqs-107.30004/

Just confirm that I bought a Supermicro Aquantia AQS-107-B0C2-CX 10G on ebay for $125 , and it's working great at 2.5Gbps from my main PC with a ASUS TUF X570-Pro motherboard, so far iPerf got 2.2 and 2.3 Gbps sustained,

since that was the last one from that vendor, now in the hunt for another one at a reasonable price

Thanks again for the reply

 

[Serhan]

Is there any way to find out the manifacturing date of an ICX7450 by running a cli command or opening the case?

 

[ixen]

I had the same issue on 8.0.95 and switched transceivers to some FS.com units. That both resolved the problem and lowered temps a bit.

Thanks for confirming the issue.
I've finally decided to downgrade fw to 8.0.90mc where it's still working.

 

[deeceesth]

This is great info. Thanks for the insight. I had a feeling that pfsense not detecting a down link had something to do with it.

I’m going to play around with my setup a bit more with this knowledge in mind.

It's most likely a bug (or just...a not implemented feature) in your modem's firmware that's making it not send out a dhcp force renew broadcast (FORCERENEW ) on fresh boot, which gives clients new fresh leases even when the end client doesn't explicitly ask for one.

The reason you don't run into the bug/issue when the modem is directly connected to your pfsense box is because when you reboot the modem and its directly connected to your pfsense box, pfsense sees the physical link completely drop and come back up - this will obviously force a dhcp release, then a renew request on pfsense's behalf when the link is back up - it doesn't need the dhcp server side to broadcast a force renew.

when a switch is in between the two and you reboot the modem, the pfsense box doesn't see any link flaps, and will gladly hold onto the DHCP lease it was given earlier until either the lease timer expires, it sees a FORCERENEW broadcast, or you manually bounce the lease. A nicely coded modem immediately sends a "dhcp force renew" on fresh boot, so any clients on its connected l2 with old slowly expiring leases will immediately drop them and grab a new lease. I had an old docsis modem that didn't have forcerenew implemented, and yes it was annoying. my current DOCSIS modem (CM600) has it implemented and works great running through an ICX then to a virtual pfsense instance when rebooted, as an example