Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[imeguras]

RoCE "technically" requires switches to support PFC (priority flow control) to run over them, and the ICX line does not support pfc as they are the "campus" product line. For RoCE feature support you'd need the data center line, which is the VDX series (which I really don't like).

PFC support on the switch ensures the Ethernet transport layer is lossless - eg no dropped frames, which is how RoCE guaruntees it's speed/very low latency. That said, RoCE has its own error correction and retransmission faculties, it will run just fine over a "lossy" network (eg a switch without PFC support), it won't corrupt your data or anything malicious like that. It will just impact performance.

On a non-oversubbed switch with only a few clients, how often will you get dropped frames and therefore how much will it kill RoCE performance? I haven't the slightest clue, I've been wondering that myself. I've seen tests showing plain RoCE (eg the type intended for lossless networks only) working plenty fast without PFC in smaller networks, but I've also heard from engineers "don't ever do that, it'll be slower than non-rdma protocols" so I really have no clue.

Maybe someone here with an existing pfc-enabled switch and RoCE clients and some spare time can run a quick benchmark, then disable PFC on the switch and run it again. I honestly can't even begin to guess the performance impact, I would assume the biggest impact would be to latency, and you might get dropped frames that need to be recovered (costly time wise) just from one nic outpacing the switch or vice versa. What I do know is many people will tell you "just don't do it without PFC support"

Is this outdated? https://www.commscope.com/globalassets/digizuite/61731-ds-icx-7450.pdf supposedly the 7450(among others) supports PFC
Would ECN be able to be implemented with openFlow?

EDIT: (although the vdx stuff is still great insight), i would just like to be able to hack something with the "hardware flexibility" of the ICX lineup

 

[Mechotronic]

Noob question, but can I use one of the SFP ports on my 6450 as a media convertor and fence it to one port? I have it on switch firmware fyi. I am about to get rid of my fiber gateway, and was thinking I could do that so I could run copper to my OPNsense box wan-in. Would it work, and would it be performant? I have 1 gig wan so the copper port speed isn't a limit for me. I can always get a dedicated convertor, but had the idea.

 

[blunden]

Noob question, but can I use one of the SFP ports on my 6450 as a media convertor and fence it to one port? I have it on switch firmware fyi. I am about to get rid of my fiber gateway, and was thinking I could do that so I could run copper to my OPNsense box wan-in. Would it work, and would it be performant? I have 1 gig wan so the copper port speed isn't a limit for me. I can always get a dedicated convertor, but had the idea.

I don't see why not. A media converter is essentially just a switch with two ports of different media types as far as I know. Put it on its own vlan though.

Performance should be line rate.

 

[Mechotronic]

I don't see why not. A media converter is essentially just a switch with two ports of different media types as far as I know. Put it on its own vlan though.

Performance should be line rate.

That's what I was thinking. Thanks.

 

[fohdeesha]

Noob question, but can I use one of the SFP ports on my 6450 as a media convertor and fence it to one port? I have it on switch firmware fyi. I am about to get rid of my fiber gateway, and was thinking I could do that so I could run copper to my OPNsense box wan-in. Would it work, and would it be performant? I have 1 gig wan so the copper port speed isn't a limit for me. I can always get a dedicated convertor, but had the idea.

If you're one of the rare people who has dedicated Ethernet Internet service and not gpon/xgpon. If you do which is most likely, your "fiber gateway" from your ISP is doing a lot more than a media converter, and it's not your typical optic. It will not work just plugging it into a regular 1g sfp and may knock ~50 other users offline

 

[blunden]

If you're one of the rare people who has dedicated Ethernet Internet service and not gpon/xgpon. If you do which is most likely, your "fiber gateway" from your ISP is doing a lot more than a media converter, and it's not your typical optic. It will not work just plugging it into a regular 1g sfp and may knock ~50 other users offline

AON isn't that uncommon for residential customers outside the US.

 

[kevindd992002]

I've been using my ICX6450-48P (got from ebay) for over a year now. Recently, I've had problem with two ports in a span of just a few months and they have similar issues. They can supply PoE power to my PoE cameras but there is link established. Transferring the cameras to different ports fixes the issue so I know the issue is in the switchports themselves. What do I need to check? Maybe dirty pin contacts or something?

 

[chlastakov]

I noticed that with the monoprice DAC breakout, it was showing up as absolutely nothing with the "show media" command. Even though it doesn't affect the behavior of the links, I figured something was up with the I2C EEPROM in the DAC cable (this is where vendor/serial/cable type info is stored for optics and DACs).

You can see the show media command can't find any vendor or cable type info for the monoprice DAC, so it shows EMPTY for the 4 breakout slots:

telnet@ICX3#show media | include 1/2/[2-5]
Port 1/2/2:  Type : EMPTY
Port 1/2/3:  Type : EMPTY
Port 1/2/4:  Type : EMPTY
Port 1/2/5:  Type : EMPTY

So I dropped down to the debug console (over serial, press ctrl+y, let go, then press m, then enter) to use some I2C read and write commands.

First, let's read the EEPROM from a proper official brocade DAC cable:

OS>i2c read 3f 0 256
0000: 0d 00 06 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0010: 00 00 00 00 00 00 01 81-00 00 00 00 00 00 00 00 ................
0020: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0030: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0080: 0d 00 23 08 00 00 00 00-00 00 00 00 67 00 00 00 ..#.........g...
0090: 00 00 01 a0 42 52 4f 43-41 44 45 20 20 20 20 20 ....BROCADE
00a0: 20 20 20 20 07 00 05 1e-35 38 2d 30 30 30 30 30     ....58-00000
00b0: 33 33 2d 30 31 20 20 20-41 20 04 05 00 00 46 08 33-01   A ....F.
00c0: 00 00 00 00 50 41 45 31-31 34 31 30 33 30 30 30 ....PAE114103000
00d0: 38 37 36 20 31 34 30 33-30 35 20 20 00 00 00 c2 876 140305  ....
00e0: 31 31 31 30 34 30 39 30-37 31 20 20 20 20 20 20 1110409071
00f0: 20 31 20 20 20 20 20 20-20 20 00 00 00 00 00 00  1        ......

You can see the first half is mostly empty, and the vendor information does not begin until the second half of the EEPROM, where you get the byte specifying the cable type, the vendor string, a serial number, etc. This is as it should be.

Now let's read the Monoprice DAC that doesn't show up properly:

OS>i2c read 41 0 256
0000: 0c 00 21 00 00 00 00 00-04 80 00 06 67 00 00 00 ..!.........g...
0010: 00 00 01 a0 4d 4f 4e 4f-50 52 49 43 45 20 04 20 ....MONOPRICE .
0020: 31 32 30 36 39 20 2e 2e-2e 2e 2e 2e 2e 2e 2e 2e 12069 ..........
0030: 77 77 77 2e 6d 6f 6e 6f-70 72 69 63 65 2e 63 6f www.monoprice.co
0040: 6d f4 31 32 30 36 39 31-34 31 30 30 30 39 36 43 m.1206914100096C
0050: 48 49 4e 41 20 20 20 00-00 00 f4 ff ff ff ff ff HINA   .........
0060: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0070: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0080: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
0090: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00a0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00b0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00c0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00d0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00e0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................
00f0: ff ff ff ff ff ff ff ff-ff ff ff ff ff ff ff ff ................

You can see they must have been in a rush programming these, as they stuck all the vendor and serial information in the incorrect section. No wonder it shows up as empty/unknown media type! Upon closer inspection, they programmed it like an SFP+ module. This first half of the EEPROM should be populated with these values like this in an SFP+, but for a QSFP+ module, it should be in the second half.

So I used a series of i2c write commands to copy byte for byte the contents of the official brocade DAC, to the monoprice DAC, overwriting the incorrectly placed vendor info with properly placed brocade info.

The monoprice DAC EEPROM contents now looks like this:

OS>i2c read 41 0 256
0000: 0d 00 06 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0010: 00 00 00 00 00 00 01 81-00 00 00 00 00 00 00 00 ................
0020: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0030: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0080: 0d 00 23 08 00 00 00 00-00 00 00 00 67 00 00 00 ..#.........g...
0090: 00 00 01 a0 42 52 4f 43-41 44 45 20 20 20 20 20 ....BROCADE
00a0: 20 20 20 20 07 00 05 1e-35 38 2d 30 30 30 30 30     ....58-00000
00b0: 33 33 2d 30 31 20 20 20-41 20 04 05 00 00 46 08 33-01   A ....F.
00c0: 00 00 00 00 50 41 45 31-31 34 31 30 33 30 30 30 ....PAE114103000
00d0: 38 37 36 20 31 34 30 33-30 35 20 20 00 00 00 c2 876 140305  ....
00e0: 31 31 31 30 34 30 39 30-37 31 20 20 20 20 20 20 1110409071
00f0: 20 31 20 20 20 20 20 20-20 20 00 00 00 00 00 00  1        ......

Look familiar? Our monoprice DAC now has identical EEPROM contents, compared to our actual official brocade DAC.

And as expected, after unplugging and replugging the cable, the switch now ID's it as a DAC, and thinks it's an official brocade part:

telnet@ICX3#show media | include 1/2/[2-5]
Port 1/2/2:  Type : 40GE-Passive Copper
Port 1/2/3:  Type : 40GE-Passive Copper
Port 1/2/4:  Type : 40GE-Passive Copper
Port 1/2/5:  Type : 40GE-Passive Copper

##even more details - still the monoprice DAC
telnet@ICX3#show media e 1/2/2
Port   1/2/2:Type  : 40GBASE-Passive Copper
Vendor Name: BROCADE          Serial Num: PAE114103000876 Revision: A

I wouldn't think this has any effect on functionality, as I said the links worked perfectly fine beforehand. Re-coding it into a brocade part like this was only possible because the monoprice DAC did not have the EEPROM write protect bit set. This exact same procedure can be used to re-code optics to different vendors to bypass vendor locks, but many non-OEM optics have a bit set that prevents EEPROM writing without special unlock strings

A little more information:

The SFP+ EEPROM data location layout, you can see the identity and vendor etc strings are at the beginning (like our QSFP+ monoprice incorrectly had) - http://files.siemon.com/sis/application-guide/sfp-plus-to-sfp-plus_eeprom_summary.pdf

The QSFP+ EEPROM layout, you can see that the vendor and cable type etc now does not start until around the middle. This is where the ICX was trying to read to figure out the cable type, and of course on the monoprice there was nothing there - http://files.siemon.com/sis/application-guide/qsfpplus_to_qsfpplus_eeprom_summary.pdf

Upon closer inspection, this could have potentially caused it refusing to link up, as it could not even read the ident bit to figure out what kind of cable it was. For QSFP modules, it reads byte number 128 (first byte on line 80 in the dump above). This byte is set to "0d" hex to tell the host it's a QSFP+ device. On the monoprice dac, when it read that byte, it was completely empty. You can see in the brocade dump, that bit is correctly set to 0d.

here's a table of possible values for that byte:

http://imgur.com/2Id6xp9

You can see that if we set that byte to 03 instead, we could make the switch think it's an SFP module. It would probably not like this, given it's a QSFP+ slot (or on second thought, it would just assume it's an SFP+ optic in an SFP+ to QSFP+ adapter)

And upon even further digging, the EEPROM also contains signal attenuation values for the given cable at both 2.5ghz and 5ghz (bytes 186 and 187 respectively), and the host (the switch) uses these values to set up it's transmitter and receiver EQ appropriately. Without it being able to read these on the stock monoprice DAC, the transmitter and receiver probably defaulted to "fallback" values, which could have also resulted in the not quite working links.

Again, copying over the brocade DAC contents should fix this, as it contains these values (4db and 5db, respectively). I'd imagine the attenuation values there are very close to what the monoprice DAC is supposed to have, given they are both the same length, and both 30AWG twinax. Given how much stuff the monoprices come with wrong from factory, I'm surprised they ever worked, especially with cisco equipment (which is what they're advertised for). Thankfully it's not write-locked, so all this is easily fixed with a few commands

I wanted to try your approach, but it gets over my head. I bought some DAC Cable NetApp 112-00176 0.5m QSFP to QSFP External SAS Cable and it's working as 40G in Brocade, but I't like to reprogram them to work with Arista too.

Here's EEPROM content:

0000: 0c 00 02 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0010: 00 00 00 00 00 00 01 81-00 00 00 00 00 00 00 00 ................
0020: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0030: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0040: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0050: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0060: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0070: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
0080: 0c 00 21 00 00 00 04 41-80 80 d5 00 00 00 00 00 ..!....A........
0090: 00 00 01 80 4d 6f 6c 65-78 20 49 6e 63 2e 20 20 ....Molex Inc.
00a0: 20 20 20 20 00 00 09 3a-31 31 32 2d 30 30 31 37     ...:112-0017
00b0: 36 20 20 20 20 20 20 20-41 30 00 00 00 00 46 8e 6       A0....F.
00c0: 00 00 00 00 32 30 36 32-32 30 31 35 39 20 20 20 ....206220159  
00d0: 20 20 20 20 31 32 30 33-30 32 20 20 00 00 00 13     120302  ....
00e0: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................
00f0: 00 00 00 00 00 00 00 00-00 00 00 00 00 00 00 00 ................

I didn't wanted to just overwrite it with your Brocade code. Tried to understand it and make only little modifications. First of all, I don't understand why it's working as 40G when bit 128 is 0c which indicates QSFP, not QSFP+. Also, will it work if I change just Vendor name, or other data needs to match and I can brake something?

 

[puffer]

I have a 7250-48p working well, I'd like to get rid of my Fios "router" which sits between the ONT and the 7250. However I have two OpenBSD firewalls with CARP which require a static IP, so just removing it and having the firewall pickup the DHCP on the external interface isn't an option. Currently the Fios router has a static IP of 172.16.0.1 which is both FWs default gateway and I have a vlan (10) where the two ports from the FWs and the Fios router all sit untagged.

From reading the various docs and googling I have pieces of a plan I just don't know how to put it all together, and since removing the Fios router requires a call to "customer support" I'd prefer if I was as reasonably certain I can be that it'll work before doing that.

My thinking was something along these lines:

Create a VLAN and VE (4095) where I enable the dhcp client
Create a VE in vlan 10 that currently has the two CARP connections and set the IP to 172.16.0.1

Somehow route from vlan 10 to vlan 4095. This is where things get fuzzy. Is it as simple as adding a route from 172.16.0.1 to the VE which has the dhcp address from Fios? Or do I need to create VRFs for this to work?

I have create the following config so far where:

1/1/46 is current Fios router
1/1/48 is where I'll plug in the ONT
1/1/6 and 1/1/38 are the FWs

vlan 10 name internet by port
untagged ethe 1/1/6 ethe 1/1/38 ethe 1/1/46
spanning-tree 802-1w

vlan 4095 name ont by port
untagged ethe 1/1/48
router-interface ve 4095
spanning-tree 802-1w

ip dhcp-client ve 4095

Now if my "is it this easy" piece from above works I'd assume that would look something like this:

vlan 10
router-interface ve 10

interface ve 10
ip address 172.16.0.1 255.255.255.0

ip route 172.16.0.1 255.255.255.0 ve 4095

However I suspect more is required, but other than it probably involves VRF I've not figured it out. Anyone done something like or have any hints on how to go about it?

 

[SeRiusMe]

Hello! After finding this great article, some months ago I bought a ICX6610 for replacing a tp-link in my homelab. I've been reading all the resources in this thread and getting used to the interface. So far I've configured my vlans, linked to the old switch and been testing it with two cameras.
I'm currently at page 390 of this thread, but I would like to have it configured.
Can anyone help me with the configuration? I'm still not understanding the uplink and acl for routing.

I have a dedicated server with opnsense and I have it currently configured as in stick linked to the tp-link via SFP+. The router does all inter-vlan routing. I bought this new switch so I could offload inter-vlan routing as I was experiencing some slowliness.

There are 4-5 vlans, being: default/sys, trusted lan, iot, work/guest and recently nvr.
In essence:
SYS: Equipment lands. The only accessing this vlan it's me with my computer.
LAN: Personal devices. ATM it has access to all other.
IOT: Generally speaking does not have access to other vlans, but port rules apply as I have the NAS on LAN.
WORK: Completely isolated, still has access to resources in my computer (sharing a folder)
NVR: Does not have access to other vlans. The nvr server is currently on IOT with access to NVR and may be moved to NVR but then It should have incoming access for gui and NAS backup resources.

I have some Unify APs serving vlans so connected to trunk ports.

I would want to configure the new switch so it does inter-vlan routing, but still I have some "complex" rules in my firewall.
How would be a configuration for inter-vlan routing with a router-in-stick?
Would be possible an hybrid routing that allows me to do the most of routing at the switch and still having some working rules at the router?

Thank you very much.

 

[fohdeesha]

AON isn't that uncommon for residential customers outside the US.

indeed, but I am american so I must assume everyone and everything else is american too. it's our national hobby

 

[kapone]

Hello! After finding this great article, some months ago I bought a ICX6610 for replacing a tp-link in my homelab. I've been reading all the resources in this thread and getting used to the interface. So far I've configured my vlans, linked to the old switch and been testing it with two cameras.
I'm currently at page 390 of this thread, but I would like to have it configured.
Can anyone help me with the configuration? I'm still not understanding the uplink and acl for routing.

I have a dedicated server with opnsense and I have it currently configured as in stick linked to the tp-link via SFP+. The router does all inter-vlan routing. I bought this new switch so I could offload inter-vlan routing as I was experiencing some slowliness.

There are 4-5 vlans, being: default/sys, trusted lan, iot, work/guest and recently nvr.
In essence:
SYS: Equipment lands. The only accessing this vlan it's me with my computer.
LAN: Personal devices. ATM it has access to all other.
IOT: Generally speaking does not have access to other vlans, but port rules apply as I have the NAS on LAN.
WORK: Completely isolated, still has access to resources in my computer (sharing a folder)
NVR: Does not have access to other vlans. The nvr server is currently on IOT with access to NVR and may be moved to NVR but then It should have incoming access for gui and NAS backup resources.

I have some Unify APs serving vlans so connected to trunk ports.

I would want to configure the new switch so it does inter-vlan routing, but still I have some "complex" rules in my firewall.
How would be a configuration for inter-vlan routing with a router-in-stick?
Would be possible an hybrid routing that allows me to do the most of routing at the switch and still having some working rules at the router?

Thank you very much.

1. Start reading at the beginning of this thread.
2. Continue reading till the end.
3. Go back and do it at least twice more.

If you still haven't found your answers, I'll be happy to help you.

 

[kapone]

Currently the Fios router has a static IP of 172.16.0.1

That makes no sense. That's not a public IP. Your Fios router is going to get a public IP from the ONT.

which is both FWs default gateway and I have a vlan (10) where the two ports from the FWs and the Fios router all sit untagged.

Might need more details, because I don't understand this topology. Are you saying that both your firewalls are on the LAN side of the router, and that's their config?

removing the Fios router requires a call to "customer support"

It doesn't. Simply unplug it and plug in your own router. If you're extra scared, release the WAN IP on the FIOS router before unplugging.

Create a VLAN and VE (4095) where I enable the dhcp client

er...you mean you want the switch to get an IP directly from the ONT?? That'd be a public IP.

Create a VE in vlan 10 that currently has the two CARP connections and set the IP to 172.16.0.1

hmm...

Somehow route from vlan 10 to vlan 4095.

And now we get to the root of the problem. That's not quite the way to setup a transit network. You can have your WAN terminate at the switch (in a VLAN with no VE) and your router connected to that VLAN as well (your router's WAN interface will end up getting the public IP) as the WAN, and then do all the downstream config on the LAN interface(s) on the router.

Your problem is that you're trying to do HA firewalls, and it's not that they require static IPs, they require TWO public IPs. Now, there's ways to get around it, but none of the ways really work well and will end up being a maintenance nightmare. I suggest dropping the HA requirement, and simply keep the two firewalls as hot/cold backups.

 

[blunden]

indeed, but I am american so I must assume everyone and everything else is american too. it's our national hobby

That checks out!

It was good thing to mention the limitation with PON though.

 

[SeRiusMe]

1. Start reading at the beginning of this thread.
2. Continue reading till the end.
3. Go back and do it at least twice more.

If you still haven't found your answers, I'll be happy to help you.

Are you kidding, aren't you?

I'm still at page 390. In what I've read, I've seen that a similar case has already appeared. But it wasn't clear to me.
I plan on continuing reading until this present page this evening. Please, also take into account that not being a native english speaker, it's harder to me.
As I bought a 6450 just today, I have also planned to start again, but this time paying more attention to the 6450s specifics.

In the meantime... there's nothing you can tell me?
C'mon, I already know to create intefaces, vlans, acls... it is working with only one uplink.
Please pleeease..

 

[kapone]

Are you kidding, aren't you?

I'm still at page 390. In what I've read, I've seen that a similar case has already appeared. But it wasn't clear to me.
I plan on continuing reading until this present page this evening. Please, also take into account that not being a native english speaker, it's harder to me.
As I bought a 6450 just today, I have also planned to start again, but this time paying more attention to the 6450s specifics.

In the meantime... there's nothing you can tell me?
C'mon, I already know to create intefaces, vlans, acls... it is working with only one uplink.
Please pleeease..

https://forums.servethehome.com/index.php?threads/layer-3-switch-w-pfsense.23236/post-216621

 

[puffer]

That makes no sense. That's not a public IP. Your Fios router is going to get a public IP from the ONT.

I should have been more clear. The Fios router gets a public IP on the WAN side from the ONT, the LAN side is configured with a static IP which faces the internal network. The LAN side has 172.16.0.1, and the firewalls have 172.16.0.2 and 172.16.0.3, then there is a floating CARP interface which has 172.16.0.4.

Might need more details, because I don't understand this topology. Are you saying that both your firewalls are on the LAN side of the router, and that's their config?

Yes, basically it looks like this (hopefully this comes out)

                                                                     (1/1/6)  --> (172.16.0.2) FW 1
[ONT] -> (public IP) [FIOS Router](172.16.0.1) -> (1/1/46) [ICX 7250]             (172.16.0.4) carp0 
                                                                     (1/1/38) --> (172.16.0.2) FW 2

Further the FWs really do most of the routing, prior to the ICX I had an L2 switch with VLANs, FWs took care of everything and switch provided a way to plug things in and give POE to an AP. My hope with replacing with the L3 capable ICX was that I could consolidate the Fios router (for lack of better term) into the ICX while keeping everything else mostly as it is.

It doesn't. Simply unplug it and plug in your own router. If you're extra scared, release the WAN IP on the FIOS router before unplugging.

noted.

er...you mean you want the switch to get an IP directly from the ONT?? That'd be a public IP.

yes, just as the FIOS router today gets the public IP, I want the ICX to pick that up on a single interface which is only accessible to the firewalls, and then route traffic from the 172.16.0.0/24 network out to the public IP. I should note that the 172.16.0.0/24 network is only used on those dedicated "external" FW interfaces and the Fios LAN side router. Clients inside the network have no access to this and use a different IP schema.

And now we get to the root of the problem. That's not quite the way to setup a transit network. You can have your WAN terminate at the switch (in a VLAN with no VE) and your router connected to that VLAN as well (your router's WAN interface will end up getting the public IP) as the WAN, and then do all the downstream config on the LAN interface(s) on the router.

Perhaps I'm misunderstanding (likely), but this sounds like the solution if I had a single FW which would pick up the IP from the ONT.

Your problem is that you're trying to do HA firewalls, and it's not that they require static IPs, they require TWO public IPs. Now, there's ways to get around it, but none of the ways really work well and will end up being a maintenance nightmare. I suggest dropping the HA requirement, and simply keep the two firewalls as hot/cold backups.

The way it's currently setup has worked well for many years, but it requires some sort of router device infront of the HA firewalls to pick up the address. I can always literally leave it as it is and it works, but the goal was to get rid of the Fios router.

 

[kapone]

The way it's currently setup has worked well for many years, but it requires some sort of router device infront of the HA firewalls to pick up the address. I can always literally leave it as it is and it works, but the goal was to get rid of the Fios router.

That was my point. You still have a SPOF with the FIOS router (and you're double NATed). The fact that you have dual firewalls behind it does not improve the resiliency. True HA would require two public IPs, one for each firewall.

 

[puffer]

That was my point. You still have a SPOF with the FIOS router (and you're double NATed). The fact that you have dual firewalls behind it does not improve the resiliency. True HA would require two public IPs, one for each firewall.

As you rightly point out the current setup has many downsides, eventually it will be replaced with something better. However in the meantime I was hoping for a solution which while it may be sub-optimal wouldn't necessarily be any worse than it currently is, and I'd be rid of the fios router.

 

[kapone]

As you rightly point out the current setup has many downsides, eventually it will be replaced with something better. However in the meantime I was hoping for a solution which while it may be sub-optimal wouldn't necessarily be any worse than it currently is, and I'd be rid of the fios router.

With a single public IP, and a Layer 3 switch...and the need to do inter-vlan routing on the switch...and HA firewalls... You kinda don't have too many choices. You can remove the FIOS router, but you'll still have "some" router that is single, and then double NAT to your HA config.

That's why I suggested a hot/cold firewall config. That gets you to a TRANSIT model, where the WAN from the ONT is terminated directly on the switch (in a VLAN, no VE), both hot/cold firewalls have their WAN interfaces connected to that VLAN, DHCP active for WAN, but only one firewall active at a time.