48 port 10Gb SFP+ switch recommendations - iSCSI / vSphere

[Bergo]

Hello! I am looking for some recommendations on cost effective 10Gb switching with decent parts availability. This is for a small internal use vSphere cluster, hoping to save some money buying used + spares instead of new. I'm presently using Juniper EX4200's in a stack, but want to move to 10Gb. important features would be power supply redundancy and ability to do MLAG/VPC across a virtual chassis. I've got 3 Dell R640 hosts running esxi 7.0u2, A few Dell Equallogic iSCSI SANs, and a few R720XD's running Veeam for backups. I tried a pair of unifi XG16's, but I'm unhappy with them, some weird software bugs where they're flap between an adopted/adopting state.
I'm hoping to upgrade to some all flash storage, 10Gb iSCSI (currently multiple 1Gb links with MPIO), and replace the R720CD's with an R740XD2. I plan to ditch the 1Gb switches, just run copper GBICs for ILO connections etc. I've hot a pair of HA Sophos XG firewalls which support 10Gb over SFP+, and there is little east/west traffic, so L2 is fine in this environment.
budget - Maybe $2-3k? cheaper is better, but quality is best.

Maybe a pair of Cisco Nexus 3K's? Cisco Nexus 3064-X, 3064-T, and 3064-32T Switches Data Sheet - Cisco

 

[altmind]

well, even 3064-x is somewhat old and is missing a whole lot of tunneling and storage-net optimization(roce) options.

considering this budget, are you looking for used enterprise hardware?

you can be sure you will be recommended eol brocade switches on sth.

 

[Bergo]

well, even 3064-x is somewhat old and is missing a whole lot of tunneling and storage-net optimization(roce) options.

considering this budget, are you looking for used enterprise hardware?

you can be sure you will be recommended eol brocade switches on sth.

Well, if I could get something that wasn't EOL that met my requirements that would be ideal - depending on cost I'd consider throwing support on whatever I bought. I just don't want to put down $10K+, but was disappointed with these $1000 Unifi switches. some some happy medium between the two ideally.

 

[sko]

How about the Nexus 5600 series? (e.g. 5672UP)

They will reach "end of maintenance" in may and will still receive vuln/security updates until 2024 [1]. They can be found for ~1500EUR or even less here in europe/germany, so prices in the US should be considerably lower.

You can pair the 5k with some nexus 2k fabric extenders [2] which are available dirt-cheap as used/refurbished units, so this would fit perfectly in your budget.


[1] End-of-Sale and End-of-Life Announcement for the Cisco Nexus 5500, 5600 and 6000 NX-OS 7.3 all versions
[2] Cisco Nexus 2000 Series Fabric Extenders Data Sheet

 

[uberguru]

Why cisco? why going with over priced gear?
What happened to stuffs like this? https://mikrotik.com/product/crs326_24s_2q_rm

At $499, you can get x2 of these and be super redundant
You can even do x4 of these depending if you really need that many ports

it is like trying to buy a car and requesting a bugatti
what happened to a toyota? they perform same functions
and the toyota does it even more efficiently cost aside

 

[tjk]

The words enterprise switch gear and Mikrotik don't belong in the same thread. Sure, if he was building a lab setup, or needed out of band ports, then maybe.

 

[uberguru]

The words enterprise switch gear and Mikrotik don't belong in the same thread. Sure, if he was building a lab setup, or needed out of band ports, then maybe.

You must be completely hooked on the enterprise buzz words
You tell me what the cisco switch can do that the mikrotik can't
Enterprise is just tag name for high price tag and over paying
Many times they are not worth much close to the price tags

I can bet my money you have never used mikrotik or even know much about it besides just knowing how to spell it
One of the best router OS out there, very easy to manage without the over complicated Enterprise junks many people are hooked on, blindly

 

[tjk]

I actually use Mikrotik for our OOB stuff and have been using them for 15+ years.

You are correct, if you need cheap layer 2 and don't need to worry about picking up the phone and getting help when something is wrong or chasing down a problem, then have at it, buy Mikrotik.

Forget about all the Mikrotik recent exploit/botnet stuff, here is an example of people having problems going on almost 2 years and Mikrotik not even responding about it:

https://forums.servethehome.com/index.php?threads/the-infamous-mikrotik-crs354-48p-4s-2q-thread.35958/#post-331335

CRS354-48P-4S+2Q+ traffic problem on ports 1 to 8 - MikroTik

forum.mikrotik.com

 

[Joshh]

You must be completely hooked on the enterprise buzz words
You tell me what the cisco switch can do that the mikrotik can't
Enterprise is just tag name for high price tag and over paying
Many times they are not worth much close to the price tags

I can bet my money you have never used mikrotik or even know much about it besides just knowing how to spell it
One of the best router OS out there, very easy to manage without the over complicated Enterprise junks many people are hooked on, blindly

This has to be one of the most interesting takes I have seen on this forum. Even Mikrotik does not agree with you.

 

[Bergo]

Why cisco? why going with over priced gear?
What happened to stuffs like this? https://mikrotik.com/product/crs326_24s_2q_rm

At $499, you can get x2 of these and be super redundant
You can even do x4 of these depending if you really need that many ports

it is like trying to buy a car and requesting a bugatti
what happened to a toyota? they perform same functions
and the toyota does it even more efficiently cost aside

dual hot swapable power supplies and MLAG/VPC is why I was looking for enterprise gear, but it does appear that if I run those with a router OS instead of the switch OS they maybe support VPC/MLAG?

 

[uberguru]

I actually use Mikrotik for our OOB stuff and have been using them for 15+ years.

You are correct, if you need cheap layer 2 and don't need to worry about picking up the phone and getting help when something is wrong or chasing down a problem, then have at it, buy Mikrotik.

Forget about all the Mikrotik recent exploit/botnet stuff, here is an example of people having problems going on almost 2 years and Mikrotik not even responding about it:

https://forums.servethehome.com/index.php?threads/the-infamous-mikrotik-crs354-48p-4s-2q-thread.35958/#post-331335

CRS354-48P-4S+2Q+ traffic problem on ports 1 to 8 - MikroTik

forum.mikrotik.com

Cisco has never been exploited?

Yes mikrotik has like zero support, but we are talking about a switch here
What support really? configuration support? Well that is on you there
And no the link i sent you is not a layer 2 switch, that is a full fledge layer 3 switch

If there is issue with a switch, then go buy another one or return for a new one
Their prices are too cheap to be trying to report issues with none working ports

Enterprise or not, everything can be exploited
So please spare me with that

 

[uberguru]

dual hot swapable power supplies and MLAG/VPC is why I was looking for enterprise gear, but it does appear that if I run those with a router OS instead of the switch OS they maybe support VPC/MLAG?

The switch i sent you has dual power supply
It is a layer 3 switch, full layer 3 switch

 

[uberguru]

This has to be one of the most interesting takes I have seen on this forum. Even Mikrotik does not agree with you.

Does not agree with me in what area(s)?

 

[uberguru]

Am not surprised at your comments, i understand what indoctrination is
When you have been told cisco or juniper or nothing since growing up
I get it man, we have all seen what indoctrination does especially in last few years in the world

It is like getting surprised when someone with bugatti is wondering if you would get to your destination safely in a so cheap toyota
So very much expected

Most of you folks can not even afford these so called "Enterprise" gears
They are enterprise for a reason, meant for companies to purchase
Most you guys probably talking about your experience with company gears not yours
Enterprise is merely a business to business price tag, so prices are meant to be way over priced

 

[sko]

mikrotik hardware might be "ok" but their firmware is a dumpster fire and deserves to die in hell.

If you treat their switches like dumb, unmanaged layer 2, they (mostly) work, but essentially they are crap.

I've dealt with a few routers and switches from mikrotik and you are _CONSTANTLY_ working around bugs, limitations and their incredibly brain-dead logic for (virtual) interfaces or their firewall rules. Their CLI is also a bad joke if you've worked with proper ones.

For non-critical tasks they are an ok-ish, cheap option if you are willing to deal with their crappy software.
BUT - with mikrotik everytime something in your network doesn't work or behaves oddly, you have to rule in yet another bug in their joke of a firmware, and more often than not this is the case. This is just completely unacceptable for network gear and ESPECIALLY because they seem to ignore most of those bugs as you might find by searching in some notworking-related forums (e.g. this one) where people are complaining about severeal (serious) issues for years without anything being fixed or even acknowledged as a bug/fault.
I still have a CRS305 paperweight at home that is practically unusable because even under light load it will just go quiet on all ports for several seconds multiple times a day. I could live with the mediocre performance (because it was cheap), but not with this annoyingly flaky behaviour. Their 'support' is a joke and this problem has come up more than often enough even on their own forums but is also ignored.


TL;DR: Yes, mikrotik is cheap. But that's it and it shows.
If you fix stuff for a hobby, don't have to rely on it and just want "moh' cheap" - buy mikrotik.
If you however just want something that works, keeps working and has proper and thorough documentation and support, then use proper gear from cisco, juniper etc.. brocade might also be an option, but thanks to the frequent mergers and acquisitions most of their documentation seems to be lost in the void.

 

[i386]

It is like getting surprised when someone with bugatti is wondering if you would get to your destination safely in a so cheap toyota
So very much expected

A better comparsion would be a truck (enterprise) vs. the cheap toyota (consumer/prosumer)...
One is made for "harsh" conditions and the other is made for working reliably and transporting goods

 

[sko]

A better comparsion would be a truck (enterprise) vs. the cheap toyota (consumer/prosumer)...
One is made for "harsh" conditions and the other is made for working reliably and transporting goods

The fitting analogue for mikrotik would be something like a Lancia Beta...

 

[uberguru]

So i have a question for you guys
@sko @i386 @tjk

mikrotik vs pfsense/opnsense for router(pfsense/opnsense do not have switches)
which one will you go with?

 

[uberguru]

A better comparsion would be a truck (enterprise) vs. the cheap toyota (consumer/prosumer)...
One is made for "harsh" conditions and the other is made for working reliably and transporting goods

toyota make trucks
please do apple to apple comparisons
better yet rolls royce vs toyota? if that is easier for you

 

[uberguru]

Hello,
We are a consulting company from Turkey. We have customers from very different sectors, banks, global electronics manufacturers, universities, ISPs, etc...

We use MikroTik routers with all of them. We use it mainly as firewall and hotspot. We use CAPsMAN as WiFi controller.
In a university, we deployed a hotspot for 4000 online users. So, yes we use it in enterprise environments.

Osman Kazdal

source: Mikrotik in enterprise company - MikroTik