48 port 10Gb SFP+ switch recommendations - iSCSI / vSphere

[sko]

(pfsense/opnsense do not have switches)

Why should they sell hardware? Both are software projects on top of a (heavily butchered) FreeBSD.

Don't know what this has to do with OPs question, but for routers on general-purpose-hardware I only use OpenBSD. But if I'd had to choose between those, I'd definately go with OPNSense.


To steer back from the mikrotik fanboy-trolling to OPs question:

I totally forgot about FS.com. They have various port counts available (12,20,24,32,48) and very decent prices. Haven't worked much with them though, but a friend that works at a local ISP swears by them. The cli is said to be pretty close to IOS and they have been rock-solid workhorses for them. IIRC they exclusively use FS.com switches in their outdoor/curb distributor cabinets, so they seem to be very temperature-tolerant.
Personally I've only used/configured a handful of FS access points for home/soho use at some friends/relatives. I've never heard of any problems from anyone and the one I've temporarily used at home (AP1167C) was also pretty much unobtrusive - i.e. after the initial setup 'it just worked'™ for the ~1.5 years I've used it.

 

[uberguru]

Name me a cisco or juniper router that has 100G and 40G ports
And that can do L3 hardware offloading, and in a 1U
And for the price of $2795.00, under $3k
Let me know when you are ready, I will wait

https://mikrotik.com/product/ccr2216_1g_12xs_2xq

You think they made that router for home labs?!?
I rest my case right here

 

[uberguru]

Don't know what this has to do with OPs question, but for routers on general-purpose-hardware I only use OpenBSD. But if I'd had to choose between those, I'd definately go with OPNSense.

Mikrotik and opnsense(with logo looking like a middle schooler made it) are not even close
Mikrotik routerOS is way more robust than the basic OPNsense, again with branding of a middle scholer
So this is how i know you are the one trolling

 

[i386]

Name me a cisco or juniper router that has 100G and 40G ports
And that can do L3 hardware offloading, and in a 1U
And for the price of $2795.00, under $3k

Arista 7050/7060
Mellanox 6012/6018/6036

 

[uberguru]

I'm hoping to upgrade to some all flash storage, 10Gb iSCSI (currently multiple 1Gb links with MPIO), and replace the R720CD's with an R740XD2. I plan to ditch the 1Gb switches, just run copper GBICs for ILO connections etc. I've hot a pair of HA Sophos XG firewalls which support 10Gb over SFP+, and there is little east/west traffic, so L2 is fine in this environment.
budget - Maybe $2-3k? cheaper is better, but quality is best.

Get out of the indoctrinated mindset
I recommend this mikrotik switch https://mikrotik.com/product/crs326_24s_2q_rm

For $499(brand new!!!), you can't go wrong, for full layer 3 switch with dual power supplies
If you need support, here is paid support

https://mikrotik.com/consultants

So there is no such things as no support from mikrotik
That is a myth, most people can't connect dots
so again ==> https://mikrotik.com/consultants

Good luck!!!

 

[uberguru]

Arista 7050/7060
Mellanox 6012/6018/6036

Are those routers?!? I did not say switch
I said router, that can also do L3 hardware offloading aka full switching
Please read my statements, i know what am saying


I said the following

- A rack 1U router​

- Can do L3 hardware offloading​

- Is under $3k brand new​

- Has up to or more than 2 x 100G and 12 x 40G ports​

- And that is as power efficient at max power of 128Watts!!!​

- Also might i add in compact design, without the bulkiness at just 14.5 inches deep!!!​

And why running away from cisco and juniper?
They are no longer equipped for this task?
Lets stick to cisco and juniper please, the enterprise juggernauts

 

[tjk]

So i have a question for you guys
@sko @i386 @tjk

mikrotik vs pfsense/opnsense for router(pfsense/opnsense do not have switches)
which one will you go with?

I use pfsense with commercial support on our own hardware.

 

[tsteine]

@uberguru
What scares me about that mikrotik router is that this is the picture they've provided for the routing performance, where they've benchmarked 1518 byte packets to show line rate.

These are not particularly impressive performance statistics for a 100gbe router.

My own setup with an intel Xeon 4 core cpu and a mellanox dual port 100gbe adapter running FD.io(vpp) on 2 cores exceeded the packet processing capability presented in this image (16.25 mpps vs 32.94 mpps). see here. It's even cheaper than the mikrotik router to boot.

 

[uberguru]

@uberguru
What scares me about that mikrotik router is that this is the picture they've provided for the routing performance, where they've benchmarked 1518 byte packets to show line rate.

These are not particularly impressive performance statistics for a 100gbe router.

My own setup with an intel Xeon 4 core cpu and a mellanox dual port 100gbe adapter running FD.io(vpp) on 2 cores exceeded the packet processing capability presented in this image (16.25 mpps vs 32.94 mpps). see here. It's even cheaper than the mikrotik router to boot.

Yeah may not be impressive yet
but they mentioned the RouterOS is not taking full advantage of the hardware of that router yet
Future releases of RouterOS will improve on those performance

 

[Rttg]

Future releases of RouterOS will improve on those performance

Beware the risk of vaporware…

 

[uberguru]

Beware the risk of vaporware…

This is not vaporware, more like honesty plus reality

 

[uberguru]

Name me a cisco or juniper router that has 100G and 40G ports
And that can do L3 hardware offloading, and in a 1U

like => MikroTik


Match the following specs

- A rack 1U router​

- Can do L3 hardware offloading​

- Is under $3k brand new​

- Has up to or more than 2 x 100G and 12 x 40G ports​

- And that is as power efficient at max power of 128Watts!!!​

- Also might i add in compact design, without the bulkiness at just 14.5 inches deep!!!​

Lets stick to cisco and juniper, the 2 kings of all "Enterprise" networking devices

Anyone else up for the task?

 

[zunder1990]

Hello! I am looking for some recommendations on cost effective 10Gb switching with decent parts availability. This is for a small internal use vSphere cluster, hoping to save some money buying used + spares instead of new. I'm presently using Juniper EX4200's in a stack, but want to move to 10Gb. important features would be power supply redundancy and ability to do MLAG/VPC across a virtual chassis. I've got 3 Dell R640 hosts running esxi 7.0u2, A few Dell Equallogic iSCSI SANs, and a few R720XD's running Veeam for backups. I tried a pair of unifi XG16's, but I'm unhappy with them, some weird software bugs where they're flap between an adopted/adopting state.
I'm hoping to upgrade to some all flash storage, 10Gb iSCSI (currently multiple 1Gb links with MPIO), and replace the R720CD's with an R740XD2. I plan to ditch the 1Gb switches, just run copper GBICs for ILO connections etc. I've hot a pair of HA Sophos XG firewalls which support 10Gb over SFP+, and there is little east/west traffic, so L2 is fine in this environment.
budget - Maybe $2-3k? cheaper is better, but quality is best.

Maybe a pair of Cisco Nexus 3K's? Cisco Nexus 3064-X, 3064-T, and 3064-32T Switches Data Sheet - Cisco

We use a ton of Nexus3000 C3064PQ in our MDU sites has core switches. It has 48x10gb and 4x40gb. You can get them on ebay for ~$1500.

 

[fohdeesha]

Anyone else up for the task?

Nobody can meet that task, because as you point out yourself, the mikrotiks are absurdly cheap brand new - do think this is done by magic? it's done by cutting massive costs in development and production - the firmware dev and QA team is 1/10th the size of the teams at places like arista or juniper - this is why they're the only switch manufacturer who have had their switches turned into botnets *three* times now. Not to mention the hilarious amount of weekend-ruining bugs (I remember being able to get one model in a reboot cycle a few years back with a single poorly crafted SNMPwalk). The power supplies are cheaper OEMs with cheap components, there's poorer (or even sometimes none) ESD protection on ports, non-ECC memory for the management plane, no hardware watchdog CPLD infrastructure on the boards, I could go on and on. Yes, they're cheap. That's kind of our whole point. Someone else said it best, they're cute little switches for lab setups

 

[tjk]

Nobody can meet that task, because as you point out yourself, the mikrotiks are absurdly cheap brand new - do think this is done by magic? it's done by cutting massive costs in development and production - the firmware dev and QA team is 1/10th the size of the teams at places like arista or juniper - this is why they're the only switch manufacturer who have had their switches turned into botnets *three* times now. Not to mention the hilarious amount of weekend-ruining bugs (I remember being able to get one model in a reboot cycle a few years back with a single poorly crafted SNMPwalk). The power supplies are cheaper OEMs with cheap components, there's poorer (or even sometimes none) ESD protection on ports, non-ECC memory for the management plane, no hardware watchdog CPLD infrastructure on the boards, I could go on and on. Yes, they're cheap. That's kind of our whole point. Someone else said it best, they're cute little switches for lab setups

but but but they are cheap! they must be enterprise class and reliable due to how cheap they are.

heck, so cheap buy a few extra cold spares, so when they break just go to the DC and swap in a new one.

 

[oneplane]

I sometimes don't buy MikroTik just to piss off fanboys, and because I want a different color interface and metal paint job, and Microtick doesn't have that. And then I take to the internets to let everyone know that I didn't buy it.

Back to the question at hand: MLAG or chassis aggregation so far has been mostly a fabric ASIC feature which then gets enabled with the ASIC driver in the controller firmware; so in general, switches that support it (as in, technically support it, licensing hell is a different arena) would be BMS-style hardware, except for the few odd companies that do their own ASIC (i.e. Cisco) where they can spin MLAG-capable hardware without adding a management controller to drive it.

Most of the 40G switches have support for it but I imagine that you'll want something to use existing copper connections or existing SFPs with? MLAG is mostly useful when you don't have enough ports, but considering your setup sounds like not a high quantity of devices, a 48 port 40Gbps switch might do the trick. Maybe two if you want redundancy (and in such a scenario you might not want MLAG anyway).

Upgrading the NICs to 40G isn't all that expensive if you go with the used pars strategy, and as a bonus, a single high speed link means every connection benefits from that speed, not just multipath.

but but but they are cheap! they must be enterprise class and reliable due to how cheap they are.

heck, so cheap buy a few extra cold spares, so when they break just go to the DC and swap in a new one.

Haha, this exact scenario is what we tend to do in west EU SMEs, instead of spending 2K on pretend-enterprise, we don't even go power user, but straight to the ODM in china and just get 3 boxes for 1k,1 hot spare, 1 cold spare and run OpnSense and CARP on them. If you're gonna go commodity, might as well go full commodity. You don't need the support or features from enterprise, but a consumer-grade CPE doesn't cut the mustard either.

 

[uberguru]

but but but they are cheap! they must be enterprise class and reliable due to how cheap they are.

heck, so cheap buy a few extra cold spares, so when they break just go to the DC and swap in a new one.

Cool will go mikrotik any day, like i keep saying best analogy for you guys is toyota vs better yet rolls royce
Yup toyota uses cheap hardware and it is crap and built with cutting corners
yeah the flood light not bright enough, might I add it doesnt have an umbrella or wine cooler
Sure keep buying used 10-15 year old gears because of enterprise tag and running outdated software versions
And keep deceiving yourselves you got the best setup that is never get exploited or never have any problem
Like the saying goes, the cisco and juniper keep running forever, with zero issues ever

bravo to you all, bravo

 

[tjk]

Cool will go mikrotik any day, like i keep saying best analogy for you guys is toyota vs better yet rolls royce
Yup toyota uses cheap hardware and it is crap and built with cutting corners
yeah the flood light not bright enough, might I add it doesnt have an umbrella or wine cooler
Sure keep buying used 10-15 year old gears because of enterprise tag and running outdated software versions
And keep deceiving yourselves you got the best setup that is never get exploited or never have any problem
Like the saying goes, the cisco and juniper keep running forever, with zero issues ever

bravo to you all, bravo

I don't think anyone is upset if you decide to use Mikrotik, have at it. Just make sure you understand the risk associated with it. If uptime and support aren't the most important aspects of your deployment, hell go buy and deploy as many Mikrotiks as you want.

if you need support when (not if) something goes wrong, or you put customers or data at risk when something goes wrong, that's on you. A lot of the folks on here have many years of networking experience and have deployed many different vendors equipment. I used Tik's in prod many years ago, it bit me in the ass. I'd rather buy used or EOL'd Cisco/Arista/BCom and deploy those, which have worked well for ME over the years, and buying used allows me to have cold spares on hand - without breaking the bank as you keep alluding to.

No one is telling you to go out and drop $50K on a new Arista.

Also, FYI I have a couple of the 10G Tik switches sitting in my home lab, including the radio shack quality power bricks they ship with em.

 

[Bergo]

I sometimes don't buy MikroTik just to piss off fanboys, and because I want a different color interface and metal paint job, and Microtick doesn't have that. And then I take to the internets to let everyone know that I didn't buy it.

Back to the question at hand: MLAG or chassis aggregation so far has been mostly a fabric ASIC feature which then gets enabled with the ASIC driver in the controller firmware; so in general, switches that support it (as in, technically support it, licensing hell is a different arena) would be BMS-style hardware, except for the few odd companies that do their own ASIC (i.e. Cisco) where they can spin MLAG-capable hardware without adding a management controller to drive it.

Most of the 40G switches have support for it but I imagine that you'll want something to use existing copper connections or existing SFPs with? MLAG is mostly useful when you don't have enough ports, but considering your setup sounds like not a high quantity of devices, a 48 port 40Gbps switch might do the trick. Maybe two if you want redundancy (and in such a scenario you might not want MLAG anyway).

Upgrading the NICs to 40G isn't all that expensive if you go with the used pars strategy, and as a bonus, a single high speed link means every connection benefits from that speed, not just multipath.




Haha, this exact scenario is what we tend to do in west EU SMEs, instead of spending 2K on pretend-enterprise, we don't even go power user, but straight to the ODM in china and just get 3 boxes for 1k,1 hot spare, 1 cold spare and run OpnSense and CARP on them. If you're gonna go commodity, might as well go full commodity. You don't need the support or features from enterprise, but a consumer-grade CPE doesn't cut the mustard either.

Planning on sticking to 10Gb, but using SFP+ DAC's, and some copper GBICs to ILO interfaces (or a separate OOB mgmt switch)

 

[uberguru]

I am going with mikrotik not just because of price only
They could charge 2x their current price and will still go with them
heck might even do for 3x current price

I go with them, one because of the license, simple and great
it is like an open-source license by a company that you know will be around for a while
lifetime license not some soon to hunt you EOL license from the God-like Enterprise brands

Second their software is easy to use. Sure am not a network engineer by day
So i need an OS that is easy to use and not overly complicated
Imagine how long it took for the supposed Enterprise with all the several number of the best network engineers could not even bring out a great GUI to manage devices. And when they finally did, it was some crappy GUI. I used juniper jweb with the SRX240, it was the crappiest thing. This was when i started to get the red-pill dose of the Enterprise illusion. Trust me i work for an enterprise company and it is a shit show with waste of money and many projects getting de-prioritized
Even got entry level engineers making decisions on products, you guys have no idea
You have been hooked on this Enterprise thing and it is tough to get you out of it

Third they are innovative, again get me a router that has these features like this one https://mikrotik.com/product/ccr2216_1g_12xs_2xq
No one else makes those, none, in a 1U with those kind of ports
It will take the Enterprise like a blade looking box to do what that thing does in a 14.5 inch deep 1U device
Can i add more power efficient than your bloatted Enterpirse brands too?
Yup with the radioshack power, i mean if it works who cares if its radioshack
No wonder they the Enterprise folks overcharge you guys to the bank to make you happy
You guys love to be lied to at a cost, and they got your bank accounts to deal with it

Oh i forgot, more compact design. not the overly deep boxes that are like a blade server long
acting like it is performing some alien-like features
this thing is just switching and routing, why the giant deep boxes?

and on and on and on, sure keep deploying your Enterprise and be looking for the engineers that need to be certified to manage them
I will keep my toyota, oops mikrotik and continue to smile at the indoctrination

Mikrotik has improved over last few years, you guys haven't caught on and that is fine
You can't waste time trying to wake up people pretending to be asleep