ConnectX-3 /w QSFP to 4x SFP+ b/o Cables as 4 Router IPs to Streamline 10Gb Network?

[klui]

Sometimes new AOCs go on sale on eBay for a pretty good price. Not rock bottom like $3 for your DACs.

 

[coolelectricity]

It doesn't do anything of those except DHCP, VLANs, routing (and PBRs, VRFs etc...).

My .02 You're making this more complex than it needs to be. There's a few principles that keep me sane.

1. Never virtualize your firewall, primary storage or standalone router (not the L3 routing we're talking here).
2. You can go wild with VLANs, but keep asking yourself this. Do I really need to segment these devices off?
3. Use Figma/Mural/Miro/Visio whatever, and keep a current network diagram handy. Keyword: current, i.e you make changes to your network, you update your diagram.
4. Script/Automate as much as you can.
5. Look at 1-4 again.

I'm totally overthinking it, but just from the shell shock of the beginning stages of learning the nuance and ins and outs of new gear / systems. It's part of my process:

Discover the thing >
Research the thing >
Think enough has been learned to be knowledgeable >
Place the orders for the gear > Unbox and connect >
Discover you're basically "Jon Snow" >
Keep throwing good money after bad trying to make the original misconceptions work, like a square peg in a round hole >
Admit defeat >
Finally get it right ~

I'm still admittedly in a Jon Snow phase
*For anyone that comes across this and doesn't know the admittedly dated Pop Culture reference, this gif gives enough of the idea to follow:

- That is one AWESOME looking rack. I'll assume your pushing some pretty impressive applications with a need for that kind of bandwidth and compute!

- I'm also all about that "cheaper for the same performance" life. If it works, it works. And leaves more funds for other performance improvements (or... vacations? pay raises?) My setup is WAY more modest than that, but it wouldn't even be what it is if I were always buying "the ideal" or "top of the line" gear, so I totally get it. Hence the whole threa starter - the cheapness of older Mellanox Gear, and now exploring the cheap 40Gbe Brocade stuff. A lot of times we could just wait a few years and pick it all up at a (dramatic) discount, too. Unless your pushing some kind of Global AI or NASDAQ-listed business assets, I've found that's the way to go (especially when its "your money").
-

That's not to say using more expensive (and hopefully ideal) gear is the wrong approach, but that timing the "buy" is important, too.

I feel like I messed up when I got my first pair of switches with single 10Gbe SFP+ ports. I didn't think I could get a low-cost NIC to 10Gbe SPF+ ports, so I shopped for 2 transceivers instead. It all worked out in the end because I decided to just just a long CAT7 to link 2 parts of the house together and the longest DAC I've ever found under $100 was about 25 ft? I think they go up to slightly longer, but then it starts getting more equal in expense over $100, anyway. Inventory and spares are nice for some things, I guess.

Case and point, I looked high and low for a used/refurb Rack of some kind to start being more legit than "curb candy furniture / cabinets with server chassis mounted to it" and 2-3+ Generation old Compute and Networking, but theres a time and place for everything. I have a 1 Generation old "mini server" that has RIDICULOUS performance but the form factor limits its capacity. Picking and choosing what I actually need the highest performance for in my App Stack and general homelab workload is key to balance out unwasted capacity and optimal performance. But everything eventually evolves, too. Hence the considerations for reduced power consumption, heat/noise reduction and interoperability of "legacy" gear - The reality is most things aren't a "start from scratch", but a slow (ish) evolution. Like my learning curve for 10+Gbe networking!

As for the management side of the house, yes trying to organize everything across a mix of Physical Hardware is most ideal, but I haven't found an affordable 2.5+ Gbe Router so I'm planning and experimenting with a somewhat simple vRouter to drive a segment across the 2.5G+ Switches on the network, buuuuuuut..... also while learning/studying/working with other things like some new tool/cloud platform/ hypervisor/programming language/Security+/CKAD/etc etc al in tandem. So.. I have learned enough to know that keeping it all straight in my recovering coder brain gets REALLY hard since I'm often just like:

Step 1 of the Slow descent into madness: "oh, I need that port on that subnet? - Done. Oh, I need access to that SR? that DB? done."

Step 2: - Oh I need that protocol to be allowed between those two segments at 8:03 am for 2 hours and 36 minutes every 3 days, except Sundays? Done.

Step C; but... wait... what happened now? why did everything break...? What was I doing? How did I do that?

Part (00000100) { #Where am I? What causes the Sun to flare? What is the point of existence?... What have I done...} ;

Tombstone quote: It was working before...

Usually during the whole "skill and career growth" phases of my life, where things start to overlap a bit more often and lightbulb moments start coming faster, it starts to hit me that "There's a lot to this I just don't know... *yet". I'm having that moment again.

The Network Diagrams are a great suggest. I've done a few segments before, application / cloud side, but general rule of "K.I.S.S." goes out the window in the playground phases - I've spent a few weekends trying to figure out why my network segments disconnect every 10 minutes (*shakes fist in the air at Cisco*...) and reconfiguring 5 physical routers to work together again after doing firmware updates to patch the latest CVE. "Tech Debt" is REAL. And it sneaks up on you, every time. I think its showing that I still think it can be mostly avoided with proper research and upfront knowledge, but deep down I know you're right. The only real solution is "learn from your mistakes, keep it simple (,stupid...), and get your hands dirty by *actually* doing it. I'm just trying to keep my blast radius small, and my ramp up time short. Tapping Expert/ "Sr level" folks is as good as it gets (why I'm even here, ranting like a madman on *too much coffee*).

Trying to soak up some of sweet sweet sanity vibes, I'll break it down the way you suggested:

1. Never virtualize your firewall, primary storage or standalone router (not the L3 routing we're talking here).
- I generally don't do a ton of FWing or Static Routing for internal networks/segments, just generally "block it all" at the "tap" (re: modem/Public Internet/ L3) and make a few exceptions if I have a need or desire (going out of town, hosting a service to support a cloud app without paying $1000s for a month of testing something, etc). - So this is a check in the Yes box (1 of 5 is a good start, right?)
But in some cases there might be a need to do something funky like "multi-hop configs" with Static Routes for recovering to a backedup snapshot/golden image if I fubar something so bad I need to start over(this is often when I am experimenting with new topologies or "big tools" like K8s, Observability or Security-related tooling). - The general idea being "break stuff and learn"- but... also NOT breaking my "primary segments" in that process (ever again...), hence the favoring of virtualizing the experimental configs on vRouters/vSwitches.

That said, are you recommending using only one single router with enough of a feature set to slice and dice up your segments - Like Ubiquity's EdgeOS of some kind of 3rd Party/Aftermarket/*"WRT" (assuming non-Enterpri$e Router$ for above said budget reasons)? Or is it the security implications of "soft-Routers" having too many points of entry? Maybe the reliability factors involved where "if any one thing goes wrong, it all breaks" that come with hosting a router in a VM? This is a question I've often pondered but I'm not exactly "Mr Popular" since nerding out in my tech cave is how I've spent a lot of weekends, and even when I am talking about this stuff with anyone that's knowledgeable about it, we don't usually discuss the nuances of multi-router / split trunking switches, etc etc. It'd be nice though! I'm into it.

2. You can go wild with VLANs, but keep asking yourself this. Do I really need to segment these devices off?

- In my limited experience with VLANs, the answer is NO.
ALWAYS NO.
But... there in lies my lack of confidence that I understand it all "well enough", too.
I've found VLANs are fine for generic Internet Access to "workstations" used by the "normies" of Marketing/Sales, etc. And I understand the use cases from VLANs and how they can help secure a network. But I've certainly gotten WAY more creative in some experimentation in my own homelab scenarios (I keep it REAL basic and easy to understand in a "workplace scenario", for all the reasons you imply, and likely understand VERY well).
I've admittedly been trying to experiment with VLANs to secure Applications and Network Services, which I've found to be a tragic mistake. I assume its because I just don't know how to properly manage a good "VLAN plan" in a way that I can "hack and track" as the application evolves and the needs for Apps grow and change. I wrestled with the idea that its just my limited knowledge of "advanced networking at scale", but then, when I just segment my CIDRs for the Application stuff without using VLANs, it works pretty well. I've done some trickery with Cloud ACLs/NACLs that work for those "edge cases", too. The obvious "risk" here, is that if I ever encountered an "inside risk", like a Developer/QA person with access to those segments, then they might do something bonkers and whacky "in code" that "unblocks" them from something they "THINK" they need access to... and that has downstream consequences (best case) or could be outright devious and try to hoover up internal/customer/partner/vendor data (worst case).
Do these "experiments and exercises" are typically more about how to not only make applications and dependent services work together in a secure, "least privileged" way, but to also add a few additional layers of observability and understanding in how to identify the differences between good intentions gone wrong (best case) and straight up malice (worst case).

3. Use Figma/Mural/Miro/Visio whatever, and keep a current network diagram handy. Keyword: current, i.e you make changes to your network, you update your diagram.
- I'm either N00b-ish or old school, using Diagrams.net / Draw.io and LucidCharts thus far - And I've done as much as I can to get away from Closed Source tools (sometimes they are just better though) so if Visio were already available in work like, maybe? I'm just not sure I ever want to build another GANTT Chart ever again... so I wouldn't opt for that for anyting "personal". Same for Figma "nowadays" (sadly).

Miro and/or Brainboard looks like they might be solid Freemium choices over totally old school, though - I happened upon this corporate-y but useful-for-starter-research comparison list, too. I definitely need to be better about this adn it'd be good practice to map out the more technical aspects of my homelab segments, which would build better habits of keeping the similar docs straight where "wages are involved".

But yea, to your point (again) I need to settle on the design/topology. There in lies (the biggest) part of my problem, for sure. The homelab network is constantly in flux, adding, removing and changing systems/nodes, and then also adding, tweaking... and now removing/replacing gear - it needs to stop. And it will, now. Since that "initial n00bish phase" is (mostly) over. I learned a lot though. Like "pick a trunk switch and stick with it, or you're going to have a REAL bad time".

The Brocade learning curve and "hands on experience" nuances will come next, but at least I already know what I'm going to do with it. If the reduction trend continues, I will likely just offload the Brocade in favor of something 100Gbe with less power consumption in a few years, too, and start the cycle all over. We shall see! But that said, I still often question if my actual design choices make the most sense for my mixed bag of "where I am now", to where I'm going in the short term, to what I'm thinking for the end goal.
Ideally?...

*********
Long Term - I'd like to set up a segment with super locked down access to some "cloud networks and services" on my Bare Metals and/or Guest VMs in the homelab to keep my cloud bills as low as possible

Mid Term - a segment for secure backups, ISOs and the like, then a segment for testing/development

Short Term - a segment for all things "local services" and automation (re Locally hosted DNS, NFS/SMB/SCP/SFTP etc, Maybe Ceph? CICD, Network Services, Load Balancers, blah blah; But in a "nested Segment" Databases and "Cold Storage".

Currently: A "Management Plane" - a segment for all things configuration and storage. A path to access any upstream device configs, services and firewalls that effect the rest of the network reaching public internet or "the segments I actually want to talk to each other (or... not)".

Previously: A STABLE, rock-solid segment as the "personal segment" that is for all other things basic, like web surfing, streaming, mobile device wifi, etc. (I think I've sorted this bit out, finally. But it feels like it took longer than it should have.)

Somewhere in the middle of all that might be cool to have a VPN or two, as well, for the things I might want to access externally for some reason or another (and an easy and non disruptive on/off toggle for all that).

That's GREATLY simplified from what I DID have/try, but perhaps its still a bit too ambitious. I'll find out sooner than later, once these missing pieces start showing up in boxes. And perhaps even more obviously I'm not going to broadcast the exact details in a public setting, but this is a high-level gist for "Separation of Concerns". These nuances are where I've made some terrible miscalculations on before, so in my 2nd round of "Enterprise Grade Networking gear" I'm hoping to sort it out faster / cleaner - Less "stuff" to plug in, WAY less complexity, more performance, WAY more stability.

So...
It might be worth asking:
How many "Bare Metal" / Physical Routers do you suggest for such a set up?
I'm trying to get it all down to One Physical Router "at the tap", and drive everything through 1-to-3 vRouters that slice everything up into the segments I've described via their own firewalls / routing configurations, so I'm not having unintended consequences for other parts of the network.
But based on what you've said and implied, maybe thats not a great approach? In my mind, managing each segment as if they were with its own network would allow me make settings like "only only access on port 443 to the Load Balancer" a lot less complicated, but its not exactly "basic" either.
Do you keep a "playground segment" separate? or do you use muliple physical routers?
*************

4. Script/Automate as much as you can.

I try to do this with each "service machine" (usually VMs, but sometimes Bare Metals, too) because I TOTALLY agree. But...
- I haven't mastered a flexible enough use case for anything other than Bash (or Powershell/BAT's - but I'm trying to get off most M$ stuff anyway) so if you have any solid suggests for any Python, maybe Ansible-family (AWX/RunDeck/etc) tools with generic templates or maybe something Terraform/OpenTofu I'd love to learn about it!

Right now I'm doing it all "custom" but its been tedious and slow.

I default to a lot of VM guest backups and dd images, most days. I'd like to get better at automating the homelab stuff, even if its just for restoring backups, but mostly for purposes like deploying, say, a private DNS or NFS server in a specific segment with a configuration template I can make a few changes to, like we do "at work". The "nice thing" about "at work" though, is there are usually some extremely predictable gaurd rails implied by the tools used throughout the workday. In the home lab, though, the sky is the limit, and things get REAL messy. Hence why we often get those yummy exceptions from those departments like those you're likely calling the shots for, boss man.

Maybe there are some "less extensive" tools of the trade than full blown large scaling tools like TF and Ansible I'm not aware of for more basic use cases though? I've been loving how good Xen is at making "instant copies" of guest VMs, so I have a "palette of templates" to draw from that makes it fast and easy to blow away something I've fubarred and start over, or simply make slight tweaks to different purposes. This is a far cry from "automating a stack" though. AWX and RunDeck are pretty heavy handed, and Terraform isn't much easier to maintain, hence why the templates make it "faster easier" to spin up and experiment with, given I already have a few OSes and initial setups that I'm familiar with and like (Network Manager, versus . But maybe there's a better way I'm not aware of?

I know we are straying off the originally explicit topic of Networking with 40Gbe, but that all that rambling above is where the Mellanox VPI/VDI stuff starts coming it (how they are used in the network, anyway).


5. Look at 1-4 again.

Stellar list. Your insights are deeply appreciated!

 

[kapone]

Wow...I'm not sure I could have written that much....even in a few days...

That said, are you recommending using only one single router with enough of a feature set to slice and dice up your segments - Like Ubiquity's EdgeOS of some kind of 3rd Party/Aftermarket/*"WRT" (assuming non-Enterpri$e Router$ for above said budget reasons)? Or is it the security implications of "soft-Routers" having too many points of entry? Maybe the reliability factors involved where "if any one thing goes wrong, it all breaks" that come with hosting a router in a VM? This is a question I've often pondered but I'm not exactly "Mr Popular" since nerding out in my tech cave is how I've spent a lot of weekends, and even when I am talking about this stuff with anyone that's knowledgeable about it, we don't usually discuss the nuances of multi-router / split trunking switches, etc etc. It'd be nice though! I'm into it.

I must be missing something obvious here. What's dictating your need for multiple routers? In my mind, the switch is the router, and it has everything you need to go wild. The firewall on the other hand, is for external network access (usually...).

So, your router can have as many segments as you like, on your firewall you define what segment/host/CIDR/whatever can/cannot access that external network.

Unless we're talking geographically dispersed networks with BGP/ASNs etc etc...which I don't think we are.

Here's my "sane" approach for a homelab situation. Two (or more) physical switches...with a bare metal firewall with enough ports to support these switches and at least one leftover port for "oh shit" access.

Home is one switch..connected to the firewall...connected to the external network. Don't **** with this.
Lab is on anything but the above switch...connected to the firewall...connected to the external network. **** this up all you want.

 

[i386]

The 40g DAC cables are thick...and heavy...

You are not going to like 100GBE and faster DAC cables

 

[coolelectricity]

Wow...I'm not sure I could have written that much....even in a few days...


I must be missing something obvious here. What's dictating your need for multiple routers? In my mind, the switch is the router, and it has everything you need to go wild. The firewall on the other hand, is for external network access (usually...).

So, your router can have as many segments as you like, on your firewall you define what segment/host/CIDR/whatever can/cannot access that external network.

Unless we're talking geographically dispersed networks with BGP/ASNs etc etc...which I don't think we are.

Here's my "sane" approach for a homelab situation. Two (or more) physical switches...with a bare metal firewall with enough ports to support these switches and at least one leftover port for "oh shit" access.

Home is one switch..connected to the firewall...connected to the external network. Don't **** with this.
Lab is on anything but the above switch...connected to the firewall...connected to the external network. **** this up all you want.

Coffee Brain types fast

Ok yes I see now I my lack of specifics

I'm calling a "router" in the wall of strugglebus-induced text on the last page(s?) "anything that can route the traffice for one or more CIDRs".

So in my soon-to-be-old configuration, I was using:

- 1 "main router" that connects "at the tap (the modem that provides internet)"

So I have that "primary router" that is physical.

Downstream from there, I have:

- A switch that gives me more ports to decide what else I want to add to the network, like more routers (wired and/or wifi) and switches (unmanged, managed, 100Mbe, 1Gbe, 2.5Gbe, 10Gbe, ...40Gbe? [soon hope. Shipping feels like forever here...])


But then I have a frankly bonkers amount of additional physical switches that's become a rats nest of experimentation

I also have 3-4 other physical routers that are both Wired-only and WiFi; Just for examples of the use cases:

- 1 physical router does VPN,
- 1 does old WPA2 WiFi

So each Physical Router (obviously) has its own firewall/port-forwarding/ subnet/CIDRs and also "services" like VPN, DNS, etc etc, so let me address that:

- I don't use most of those "secondary router"'s features, they are kind of "one trick ponies" where I'm using one or two features from them that I liked during my "learning processes".... but....

I got caught up in experimenting with which 3rd party/aftermarket firmwares I wanted to use.
After testing and trying a TON of different things out, learning A TON (often about things I never intended to).

As a result, I know now that:

I'm a huge fan of DD-WRT and OpenWRT, each with their own strengths and weaknesses.

The troubles with these, though is they both do not always support every Router Brand nor every model within the same brands
- Regarding DD-WRT:
There are nuances to the implementation based on the features of the CHIPSETS, not the brand or model/series, so its not necessarily on DD-WRT for basically having something like 4 to 8+ different "UI Templates" with similar look and feel, but sometimes totally different implementations of the same "service" or features, due to how the development often "branched off" to focus on changes/enhancements/improvements for specific chipsets. When I look at it through that "as a Developer" lens, I "get it" now. But... it also creates somewhat of a mess when I've made certain logical assumptions that just don't pan out to be aligned with the realities of these nuances... But it all has taught me a lot of networking, security, even OSS in general + Linux and Linux networking. I eventually started just doing some things directly in an SSH shell while logged into the router, since sometimes there were missing GUI elements/forms for certain things that didn't exist in every implementation/chipset-version.

It super annoying as sometimes I would rather use OpenWRT features/configurations/add-on services on the WiFi router that supports WPA3, but... I can't, because there isn't a mod/installer for my model - same is also true where DD-WRT tends to not support "wired-only routers", and a limited number of chipsets that are typically older (grumble-grumble-something-something AX Wifi, can't work, blah blah, I forget the specifics)

-Regarding OpenWRT:
What's FANTASTIC about OpenWRT, is its "modular add-ons" system, and its "ease of version update-ability".
I find it to be both a blessing and a curse with the add-ons being so "crowded" with low quality or stale "service modules". I'll be a little theroetical and hyperbolic here, so take it with a few grains of salt. But, say... you can't necessarily "just pick a Wiregaurd module", because of the 10+ you could select from, there might only be 10 that are actively maintained, and neither of them support the latest version of OpenWRT you are running on, or they are for a version that your device doesn't support, etc etc. It's extremely confusing, partially due to just how easy it is to FUBAR your configuration with a few wrong modules/addons. But, that said, once you find that sweet sweet combination of the working configuration with the right services you want and only install add-on you know to work after having to do 2353.2 different 30-30-30 resets for that last 2 years, it is TRIUMPHANT feels. 1000% - But even though DD-WRT might be kind of janky, old-looking and have its own issues, at least its rather curated with maybe only one or two features (that most people might actually want to use) that "aren't quite right". So I see the upsides and downsides to both.

With all THAT said?

pfSense and/or OPNSense does everything I need, and then some:
- Configuring isolated CIDRs on each individual physical port of a router is a CUSSWORTHY NIGHTMARE on DD-WRT
-- especially considering that DD-WRT gets really janky after upgrading to a later version of the firmware, which gets release something 1 to 3 times a week, in some cases. So its recommended that you "30-30-30" your device after every firmware upgrade. And for many reason (maybe this has changed for some models?) using the "backup and restore settings" feature doesn't always work, or results in janky operation. As with DD-WRT, so to might be with OpenWRT with "having to frequently reset", as sometimes I've noticed I might do something like install and then remove an add-on/package and sh!t's broke.

So despite the OpenWRT and DD-WRT being vastly different, sometimes even between "chipsets", there are (were?) situations where "I liked the simplicity/feature/function on one more than the other", when I was first "getting the hang of them" and finding their quirks and limitations.

But now, a few years later? I've realized I am "forced to make a choice" each time I want to update/upgrade DD-WRT and/or tweak/change something that isn't quite working... or... maybe its "possible there is a bug because I didn't 30-30-30 first" or... whatever I goofed up, I can't track down... or.... any number of other possibilities, really...

so...

the choice is to:
Either fight with it to try to fix it, or just save the brain power and time and factory reset.

The same is more or less true of OpenWRT, as well, except you can *usually (...) update to a newer minor version a little "smoother" with (fairly) more confidence. But still, if something gets janky... and you're reasonably certain you've got everything properly configured... and "it" STILL isn't working... or you totally lose functionality you don't think you SHOULD have lost during the changes... just reseting with a "scorched earth 30-30-30" is sometimes just faster and easier and less brainpower ASSUMING you can @!#$%ing remember the correct order of operations to get back to where you were before you borkt it all up (or the bug happened)...

But with OPNSense/pfSense?
It's a breeze configuing multiple "routers" with multiple CIDRs even - or calling them by perhaps a more proper name of "Gateways"

So, I have known for awhile that I want to ditch a lot of this physical router madness and try to use mostly virtualized Routers with a limited number of physical switches to "bridge to physical rooms".

It was cool at first, playing around with the different possibilities with *WRT software, but... I'm TOTALLY over it, now.

But I still want to have (at least) the 3 separated "segments":
1 - "Public" - for WiFi that's "weak" (WPA2) / basic internet usage / Guest Networks / etc
2 - "Private" - (and this is where it gets potentially slicey/dicey) For Data Storage, "Software Development", Load Balancers, Proxies, Network "Services" (NFS/SMB/other-file-servers, Web Servers, Private Git Repos, "Security Appliances" [VMs and/or Physical] and so on...)
3 - "Secured" / Management Plane - Physical and Virtual Machines that can access "all the things", "Local" Databases, maybe some other "select" services/servers/tooling like self-hosted NextCloud/OwnCloud, "private search" (re: SearX and the like), whatever else I will probably either be the only user of (lol, but still)

I have considered also having some kind of a "segment in between" for something like Cloudflare ZeroTrust (or a similar concept using crafty VPN tricks), too, but... baby steps.

So my currently way of thinking is something like this:

In the "1-public segment": Most things go here. This is where stuff like phones/laptops, etc can use a VPN if they need access to stuff in the other "segments" - "This segment" doesn't need to have more than one "link" to 10G, since I'm not expecting on having a 10Gbe Modem connection at a reasonably affordable price anytime in the near future (sadly). - 1 DHCP Server here makes sense.

in "2-Private" - At least 1 separate Hypervisor here, to keep it "clean", possibly 2 - This segment is where I want to have "the most network capacity and performance" for somewhat obvious reasons - maybe three or less total physical machines, depending on how much RAM/Disk is needed for the "Developery stuff" - maybe a separate "hot storage" machine for backups/apt/yum/dnf-cacher and similar? Haven't settled on all that yet ( probably add as needed, but want to leave things open for it) - 3 or less physical devices, ideally - This is the segment where the Brocade will likely "live closest to", and connect the other "segments" - Maybe 1 other DHCP Server here, that doesn't broadcast in the "1-Public" segment

in "3-Secured" - 3 or less Physical PC's here? 1 separate Hypervisor here, to keep it "clean", for Databases and "other stuff" that's best left secured; and then probably a "cold storage" machine that doesn't stay powered up 24/7 - Probably no DHCP here, but instead potentially a VLAN "someday", if I get ambitious enough to want to make a guide / practice.

I keep toggling back and forth for how to organize and/or divide everything out, but that above is basically a generic concept that I feel like "makes sense" - if I go with the "Securely and Remotely accessible segment" idea, I might do another "segment" - but I feel like 3 would be better/cleaner and I could finally get out of "hardware mode".

It also helps to type it all out and make it more solid and "real" in my mind, since I'm basically trying to go from 6+ "network segments" to the 3 or 4 I've outlines here, and up my network performance and reduce the complexity by having 1 or 2 physical routers pushing 2 or 3 physical switches, and virtualizing 3 or less routers (IpFire/OPNsense/pfSense/SophosXG/vSwitch whatever else I want to play with and learn), as "playgrounds", where if I jack something up, it's no big deal and doesn't goof up the rest of the topology (ever again...), I can just blow it all away and re-think things.

But I'm pretty sure I've covered all the bases I'll need / want to cover since I'm committed to reducing "the collection of chaos" that's accumulated over the years.

- If higher ISP speeds than 2.5Gbe ever become reasonable around here, it'll be easy to just swap out the "Edge Router" and keep a consistent topology (finally...)

- So since this I'm planning on "living with it" for a long while with the "brand decision", I want to make sure I'm not just going down more chaotic rabbit holes. Like I'm already seeing where some of my DACs have reviews that aren't jiving with Brocade, which kind sucks, but also not, since the QSFP-to-4x SFP+ cables I have seem compatible, so that should get me started once the switch arrives. It'll probably take a few days, maybe a week, to play with things and start cutting out a LOT of this crazy. I don't have 20 machines, but am somehow filling most of an old 24 port Cisco 1Gbe rn, lol. It's mostly all the LACP I'm doing, but still... I looking forward to simplifying AND gain performance. It was fun and cool learning and getting stuff to work I've never played with before, but TOTALLY OVER IT when it comes to troubleshooting when something gets weird (which I now know is usually Cisco's "features", for better or worse).

I'm here, hoping someone tears it apart, telling me what I'm not thinking about, or a better way to go about it all. I'll most likely go through a few iterations, but instead of dramatically changing topology constantly, I'll probably just play with the virtualized segments, once I have the final piece in place - which is the Brocade (until I switch to Mellanox 100Gbe in a few years or so, too )

 

[coolelectricity]

The Brocade has arrived!!!

I'm still "getting the hang of it" though, but its pretty sweet thus far! I didn't think things through too hard when I saw one pop up on the radar in great shape, I pretty much hit the "buy" button pretty quickly. I'm not at all regretting it, but I didn't get a model with 1Gbe RJ45 ports, so now I'm trying to hunt down a decent deal on something like a "10 pack" or RJ45 1Gbe/1.25Gbe transceivers that will work with Brocade and Cisco-compatible gear. I crimp my own Cat5/5A/6/6A's for the "long runs", but having a few 1/1.25Gbe DACs might not be a terrible idea either. I've never messed with the "Dual Fiber" / FC's before, but I did end up with 2x FC Transceivers in a "bundle" that I bought to get the QSFP to SFP+ adapters, so I'm not opposed to those, just a bit apprehensive because from what I understand, they can be fairly fragile (compared to DACs and CAT5/6/7/8).

Does anyone have a good source for a pile of 1Gbe / 1.25Gbe connectors and/or cables? I've scouted the usual places and it looks like picken' are slim for "bulk packs" right now (at a decent price). I'm thinking I probably only need 10 or less RJ45 transceivers, or some combination of connectors (Not picky about the 1Gbe stuff, since I'm trying to keep 1Gbe to a minimum now that I have better/faster gear).

 

[kapone]

I didn't get a model with 1Gbe RJ45 ports

What?? I thought you mentioned you got a 6610? Wait...you didn't by chance get the 6610-24F, did you?

 

[kapone]

1Gbe / 1.25Gbe

The Brocade does only 1/10/40gb on different ports.

 

[coolelectricity]

The Brocade does only 1/10/40gb on different ports.

Ahhhh yea my bad, I was looking at all the transceiver options and they show sometimes as "1.25Gbe" - which I assume means "hey if you have a 10Gbe SFP port you want to work 1/8th speed, you can do that!" (but why tho?) - yea you are right, the 24-ports are just 1Gbe

 

[coolelectricity]

What?? I thought you mentioned you got a 6610? Wait...you didn't by chance get the 6610-24F, did you?

I totally did:

License: ICX6610_ADV_ROUTER_SOFT_PACKAGE
...
HW: Stackable ICX6610-24F
==========================================================================
UNIT 1: SL 1: ICX6610-24F 24-port Management Module



Is that good or bad?...

 

[kapone]

hey if you have a 10Gbe SFP port you want to work 1/8th speed, you can do that!

You have a looong journey ahead of you...

 

[kapone]

I totally did:

HW: Stackable ICX6610-24F
==========================================================================
UNIT 1: SL 1: ICX6610-24F 24-port Management Module



Is that good or bad?...

Out of every 6610 model...you had to get the one that is all SFP... Maybe that's why it was so cheap...?

 

[coolelectricity]

Out of every 6610 model...you had to get the one that is all SFP... Maybe that's why it was so cheap...?

I'm thinking yes.

I'm not exactly unhappy with it, but once I "sort it all out" I think I'll only need maybe 6 to 10 transceivers. I have a few I can "borrow"/swap from other devices, but they are playing key roles in my current-but-soon-updated configuration. And I'd rather "go cheap" with some used one's and keep a few spares around, but I'm probably going to wait it out for either a fleaBay Lot, someone saying "psssst, I got you...", or another cheap source to buy from.

I've never messed with "Fiber stuff" before, but it looks like costs might be reasonable for bulk pack Txncvrs and shorter runs of cable bundles, so.. maybe? I'm reluctant on the Fiber, since I might change my mind of how to set everything up and don't want to buy a bunch of different lengths that won't get used. I crimp my own CAT5/6 so I'm still leaning towards "fiberless", but I did get a pair of LC's in a "bundle pack", so I will likely make that one exception.

But super open to suggestions for best practices / budget.


I'll Probably end up with a mixed bag, but I have enough to replace around 75% of my old Routers and switches now ("big" Cisco's and the unmanaged stuff). I just want to make sure I'm not going to end up in a situation where I moved everything around and have parts of my network down for DAYS or WEEKS because I forgot something stupid or my configurations can't/won't work like I thought they would.



Also...

You have a looong journey ahead of you...

now... I'm all like... What *ARE* the "1.25Gbe Transceivers" for?

Some kind of overhead "protection" or something? (as in: Why are they labeled 1.25Gb and not "just 1Gb"?) The "overhead factor" is what I always thought/assumed, as I've briefly read up about, but I've only had to use a handful of Transceivers over the years, until I started getting into 10+Gbe recently.

 

[klui]

You can get SFP to RJ45 pretty cheaply on eBay. Since they're 1 Gb, no need to worry about power draw.

I just saw a lot of seven Arista modules for US$35. And a lot of 18 Junipers for US$40.

 

[coolelectricity]

You can get SFP to RJ45 pretty cheaply on eBay. Since they're 1 Gb, no need to worry about power draw.

I just saw a lot of seven Arista modules for US$35. And a lot of 18 Junipers for US$40.

Even before I read this, I spotted the Aristas that I think you mention, but did not see the Junipers!
That deal is right up my alley!
I hunted it down with your clue of "lot of 18", Thanks man!

 

[coolelectricity]

Alright, so... the 10Gbe/40Gbe ensemble is now complete (for now?)
- I'm curious how well I did on budgeting this all out, though. I'm thinking I could have done a little better at lower costs, but I don't feel like I got totally ripped off, either.

I saw a lot of 48x 1Gbe Port units floating around on the fleaBay, some were in ROUGH shape. Issues like what looked like borkt ports, stuck transceivers, and stuff like missing/dead power supplies, etc. So while I'm not usually one to care so much about "how will it look in the IT closet?" I still wanted a unit that was in good working condition that I wouldn't have to turn into MORE projects hunting down parts and/or doing mods for.

After getting CRUSHED with high electricity costs over the last year or so (big reason why I *FINALLY* starting simplifying/reducing/optimizing my lab) I was also considerate of power consumption as I anticipate I'll keep whatever gear I got for 40Gbe for a long time (unless I fall off the wagon and relapse on this homelab addiction again )

So here is how it turned out:
- Brocade ICX6610-24"F" (all SFP+ ports...)
--- "ICX6610_ADV_ROUTER_SOFT_PACKAGE" (Wire Speed Routing capabilities)
--- "Fully Licensed" for all ports (and I assume all other features/protocols, too, but I'm not sure just yet - so far so good, though)
- 2x 40Gbe QSFP-to-QSFP DACs
- 2x 40Gbe QSFP-to-4x 10Gbe SFP+ Breakout DACs
- 12+ 1Gbe SFP+ -to-RJ45 transceivers
- *I won't count the total spend as having included any 10Gbe DACs or 10Gbe Transceivers since I already had those for a few other reasons, but for anyone just starting out with 10Gbe in a home lab, its NOT a negligible cost...

All in, for the ICX6610 I ended up with, plus "accessories", I'm at just over $200 USD.
(Maybe add about ~$50 for 2x 10/25/40Gbe capable NICs and QSFP-to-SFP+ adapter/transceivers, but those aren't inherently "exclusive" for use with the ICX6610 as I initially bought them for a different purpose anyway, since I thought it was SUCH a good deal!)

I know could have saved 10-20% (ish) in "upfront costs" by going with a more "beat up" 48p model with RJ45s built-in, and fohdeesha's guides make the licensing somewhat of a non issue (in 2025 and beyond - since the device/series/model has been EoL for a few years now)
- The lower power consuming 24 port models are pretty scarce right now (2025Q1: I assume most have been bought up by those in this forum! )
- The "F" model with all SFP+ ports is lower power consumption, though the "extra spend" for the (used) 1Gbe transceivers makes it roughly a 2 year "break even" cost for the power savings, but I'm thinking having 12/12 SFP+/RJ45 ports (and optionality) will inspire me to keep it longer than 2 years and/or increase/maintain the resale value (But that assumes price stability, which... isn't exactly a thing lately..).

I'm checking in with others on the forum that might spot anything I'm not factoring in for total costs, or to get low-key mocked and laughed at for spending too much. (Everyone has a different way of communicating and sharing information, right? )