need recommendation on 10/40/100gbe lower power with full feature networking

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

twinkletoes

New Member
Jan 8, 2024
11
1
1
i have been doing this stuff for some 33 years, and i am a little out of touch with the newer stuff..i saw some Arista recommendations lately..never used it before.

i buy everything on ebay and have forever. i stopped using cisco in 2005 when the 7206 and 7505 proved to be extremely unreliable, the cisco IOS at the time was full of bugs that i couldn't get them to fix (with a support contract) and to top it off, my 7505 cards were dying at an incredibly fast rate.. i went to juniper m20 which was amazing, extremely reliable, and i never needed a bit of support that i couldn't figure out by reading forums and juniper.net documentation. heck, most of the m20 hardware from 2005 still works perfectly fine today, probably ALL of it, minus some flash or hard disks (and maybe an FPC or two), it's just too slow and power hungry to be practical! i wanted to make an m20 shrine in my garage...not having juniper support meant some toil that shouldn't have been necessary, but oh well...

these days i am doing a lot of MPLS IP VPN, RSVP, vlan-ccc (subset of VPLS) and also some full route table routers. juniper is a huge power hog, my mx480s are using 1400 watts with mx-scbe, 4 MIC/MIC3 cards with around 32 10GbE, 4 40GbE and 2 100GbE. my mx240s are using 750w with two MICs with a total of 20 10GbE and 20 1GbE. that's a lot of power for something that can probably be done in much less these days. it pains me to say since i just upgraded to scb-e and newer MICs away from DPCs as they are finally cheap enough to be worth looking at. i also use a lot of SRX550 and MX80 to supplement the bigger MXes...

juniper has a big advantage, even with the power usage, in that the hardware is usually solid. as long as power works, it works, except perhaps if the room overheats past 130 or so, then you might have some failures, that's the only situation i've ever encountered where that was a problem in the MX240/480 chassis. actually the m20 even runs in that condition, i've never found a practical way to make an m20 with power stop running. i've had a few MX80 failures, those are slightly less reliable.

i do use openbsd in a 4 box 10gbe pfsync cluster as a NAT appliance. i have considered using BSD or linux, but junos is so extremely reliable, the hardware hot swappable, that i am really asking for a big iron recommendation with a similar level of testing and hardware capability (not mikrotik) that might use less power. maybe cisco should be back in the playbook these days? (and i already use mikrotik in smaller applications, but i would consider it to be a boutique solution like a bsd or linux box, solid for a specific application but not well tested against the variety of uses that the MX platform handles effortlessly)

i recently bought a bunch of bargain bin ACX2100/2200/4000 routers for tight spaces with shitty cooling, which use the same broadcom chipset running junos and hopefully i can use them to get 10GbE MPLS with RSVP, plus irb and both vlan-ccc and MPLS IP VPN. they'll do everything i need minus a full route table. my SRX550s are running in "packet mode", they are octeon chipset boxes that run junos. the SRX hardware is reliable enough that i would keep using them if they would run a modern junos, but they don't and i don't want to fight it. they are a little buggy if you try to do certain things, like run the 10gbe ports (i have nothing but problems trying to use the 10gbe ports.) the broadcom based ACX2k/4k is an inferior platform because you can't use the juniper lo0 firewall to protect its management interfaces. the SRX550s are just not well supported enough, i need 10gbe more than i need the lo0 firewall.

one thing all of the SRX/ACX products do is use less power! and the ACXes can fit in a very small space. what would you do?
 

zunder1990

Active Member
Nov 15, 2012
212
72
28
For work I use a arista DCS-7280QRA-C36S-M-R for bgp routing.

12x 100gb ports
24x 40gb ports
with 11 ports connected it is using 297watts according to the build in power status.
 

NablaSquaredG

Layer 1 Magician
Aug 17, 2020
1,355
827
113
I would‘ve recommended Mellanox SN2700, but you seem to do more carrier / enterprise networking / routing than datacenter and need much more sophisticated features than the standard EVPN-VXLAN / BGP
 

twinkletoes

New Member
Jan 8, 2024
11
1
1
this stuff isn't really sophisticated, at least not the way i'm actually using it. the big thing a juniper MX with MPC line cards provides is line-rate with small packets. which isn't necessarily that important, i can probably do the actual traffic levels i have with all low power "merchant" chipsets, not juniper trio or whatever is in the MPC3s. the juniper is just the functional network plumbing. i need 100/40/10GbE ports on the head-end, and 10 and 1GbE ports at remote sites. most everything up to 10Gbps is a radio link, with a few fiber links here and there, and it would be even nicer if i could script updating radio link speeds into RSVP bandwidth settings without some crazy juniper RSVP speed provisioning server running seventeen instances of java inside of four nested virtual machines to update one database value based on one snmp query. i haven't really tried yet.

i've never considered evpn-vxlan but i am thinking it is worth a look. from what i can gather, it's not entirely different from using vlan-ccc, but I am layering vlan-ccc (layer 2 direct) and also MPLS IP VPN (layer 3 VRFs) on top of RSVP which controls placement of mpls tunnels on router to router links. i rely on RSVP so that the tunnel souce router can make rough choices of which links to use between routers. like the dude said, it's the rug that really ties the room together. i have different speed links between routers that i need to make a rough attempt to not overrun where possible!

i use multiple tunnels (and hashing to keep traffic to/from a host pair on the same tunnel) with RSVP to help route for me. RSVP is nowhere near perfect or exact, in general i still need to keep making links fatter and more badass. if i have way too much traffic, RSVP will just fill up every link with tunnels and the whole network will turn to molasses. thing is, with RSVP I can create big ass rings where traffic can come in from either direction, and everything just kinda works. set it and forget it.

keeping in line with the eBay purchasing system, i recently tried flashing some of those eBay CIS0000000008 100GbE cards to MT_00000000437 and it was quite easy with a jumper wire in JP7 and "flint." thank you to @bitbckt for that tid bit. i wanted to start testing libreqos.io as a rate-limiting appliance with some H12SSL-i and some EPYC 7232 and so on. dude this stuff is so fast and amazing now i'm a bit dumbfounded. it's clear the capability is there with this modern hardware, but the software isn't quite good enough, handles edge cases properly, whatever, so i'm reluctant to try and use open source in a core network role where i need a proven appliance that can run forever in a closet (and has an instruction manual since i am not the only dude who will be touching it hopefully.) using linux and bsd as appliances hanging off an MX seems to work, but even that has limitations since i am the warranty. OTOH, piles of spare juniper gear and copies of 'juniper.conf' are pretty solid.

oh, and i like to put the H12SSL inside of CSE-523L-505B but those aren't really available anymore except as overpriced old stock on eBay.
 
Last edited: