Looking for 2.5g, and 10g switch, need help with ideas.

[Greg_E]

I am looking for a new switch for my lab. I currently have a Mikrotik CRS309 with eight SFP+ in bridge to get out to my router, one of those ports goes down to an old Cisco 2960s for extra "client" connections. I want/need to start moving to 2.5gbe because I'm probably going to mini size my lab, but not sure. So I'm currently looking for a switch that has around 20+ 2.5 ports, and 10+ SFP+ type ports (40gbe breakout OK). I'd like some POE+ as well, but might need to use a second switch for power.

I was looking at the Mikrotik CRS326-4C+20G+2Q+RM but cost is more than my lab can currently support. Any used stuff out there that comes close?

Current lab:

1 DL360e gen8 on multimode 10gbe - NAS
3 DL360p gen 8 on multimode mode 10gbe - XCP-NG (dual capable sfp+)
1 DIY NAS on 10gbe copper (dual capable 2.5gbe)
2 T740 on 10gbe DAC - vmware vsphere (dual capable sfp+ and single 2.5gbe capable)

Probably adding another T740 to the mix soon.

My goal is eventually replace the DL360g8 with more modern mini PC because I don't really need that much compute power in my lab. New mini PC may or may not support 10gbe, but would be at least 2.5gbe. My T740 are already fit with 2.5gbe from the a+e slot and would add to the third if I buy one. The DL360e may be retired as soon as I prove out the mini DIY NAS (thread in these forums).

 

[i386]

I don't think there are many switches with more than 8sfp+ ports AND >16 1/2.5gbe ports.
The closest i can think of IS a ruckus icx 7150-48zp with 48* rj45 ports and 8 Sfp+ ports. But it's very rare and therefore expensive (1k+ € in the EU)

 

[Greg_E]

I started looking at a Mikrotik 20 sfp+ & 2 qsfp switch, but adding a ton of expensive multi-gig copper modules might not make this work. Only plus see is I could slowly work towards the copper ports as needed. Still thinking about this option. The easily found modules are $45usd for 1/2.5/5/10 gbps speeds. I've found a few off brand that list as 1/2.5 for $25usd

I know FS has at least one switch, but pricing will be up around $2500+usd. I haven't looked to see if I can find something like this used yet.

I'm thinking a single switch is not going to work, I'm going to need to have multiple switches. This might break down into VM network and storage/management network on separate switches, really the correct way to do this.

Also, not opposed to used equipment. Like most of us, our personal labs are made of used stuff because it's what we can afford. My XCP-NG servers don't even have UEFI.

 

[ccie4526]

Cisco C3850-24XU... some of these have been on the bay for under $300. 24 ports of 1/2.5/5/10G copper, and if you're lucky, maybe a 4 or 8 SFP+ module as well. And PoE+/Cisco UPOE. Larger 1U switch that takes dual power and does make a bit of noise, but really gets the job done. I've got one in my home office rack.

 

[Greg_E]

Thanks, I'll look into the Cisco option. Found a nice FS switch, but not home lab pricing https://www.fs.com/products/206655.html

 

[Greg_E]

That NM-8-10 is a pricey option. I might have to figure out how to make an NM-4-10 work with a few copper adapters in the servers.

What OS does this Cisco switch run? I'm somewhat familiar with IOS 15 but nothing newer.

 

[ccie4526]

That NM-8-10 is a pricey option. I might have to figure out how to make an NM-4-10 work with a few copper adapters in the servers.

What OS does this Cisco switch run? I'm somewhat familiar with IOS 15 but nothing newer.

You don't want to go past 16.6.latest as they require smart licensing on firmware on that platform thereafter (eg 16.9 -> 16.12). Fortunately for me, the same IOS commands that worked when I first started with IOS 9.x are still working on the latest 17.x releases.

ETA: Wow, for what the prices for the NM-8-10G are running, you'd do better getting a C3850-12XS and the 50cm stacking cables, and just stack them together. I may or may not have done something EXACTLY like that....

Switch Ports Model              SW Version        SW Image              Mode
------ ----- -----              ----------        ----------            ----
*    1 16    WS-C3850-12XS      16.06.10          CAT3K_CAA-UNIVERSALK9 INSTALL
     2 38    WS-C3850-24XU      16.06.10          CAT3K_CAA-UNIVERSALK9 INSTALL

3850#sh inv
NAME: "c38xx Stack", DESCR: "c38xx Stack"
PID: WS-C3850-12XS-E   , VID: V02  , SN: NOPE
NAME: "Switch 1", DESCR: "WS-C3850-12XS-E"
PID: WS-C3850-12XS-E   , VID: V02  , SN: NOPE
NAME: "StackPort1/1", DESCR: "StackPort1/1"
PID: STACK-T1-50CM     , VID: V01  , SN: NOPE
NAME: "StackPort1/2", DESCR: "StackPort1/2"
PID: STACK-T1-50CM     , VID: V01  , SN: NOPE
NAME: "Switch 1 - Power Supply A", DESCR: "Switch 1 - Power Supply A"
PID: PWR-C1-350WAC     , VID: V02  , SN: NOPE
NAME: "Switch 1 - Power Supply B", DESCR: "Switch 1 - Power Supply B"
PID: PWR-C1-350WAC     , VID: V02  , SN: NOPE
NAME: "Switch 2", DESCR: "WS-C3850-24XU-E"
PID: WS-C3850-24XU-E   , VID: V03  , SN: NOPE
NAME: "StackPort2/1", DESCR: "StackPort2/1"
PID: STACK-T1-50CM     , VID: V01  , SN: NOPE
NAME: "StackPort2/2", DESCR: "StackPort2/2"
PID: STACK-T1-50CM     , VID: V01  , SN: NOPE
NAME: "Switch 2 - Power Supply A", DESCR: "Switch 2 - Power Supply A"
PID: C3KX-PWR-1100WAC  , VID: V01  , SN: NOPE
NAME: "Switch 2 - Power Supply B", DESCR: "Switch 2 - Power Supply B"
PID: PWR-C1-1100WAC    , VID: V01  , SN: NOPE
NAME: "Switch 2 FRU Uplink Module 1", DESCR: "2x1G 2x10G Uplink Module"
PID: C3850-NM-2-10G    , VID: V01  , SN: NOPE

 

[Greg_E]

Yeah, stacking might be the way to go. I'll have to look up the 12xs prices. The problem with a lot of these eBay sellers is that they can't be bothered to show you the firmware now. Gone are the days when these people cared enough to get you working stuff for your lab.

The 24 sfp+ Mikrotik is around $550, so that's kind of my maximum right now. Not going to rush into this. If I need to I can get the stacking cables for my 2960s and gain 2 more sfp+ by stacking another switch I have here. Could really go 2 more with a third 2960s if needed. Wouldn't give me the 2.5gbps but that's not yet a need.

 

[Greg_E]

OK, I went ahead and bought the MIKROTIK CRS326-24S+2Q+RM because it fits better right now. Got a discount offer from the seller and went ahead and bought it. It is just easier than messing with the Cisco stuff right now, even though I know it can't sustain bidirectional speeds for all ports at 100%, it is rated for non-blocking at half speed and the way I'll normally use it, this should be plenty fast enough, especially if I don't end up using the 40gbps ports for a while, which I probably won't need unless I expand a lot.

 

[bugacha]

OK, I went ahead and bought the MIKROTIK CRS326-24S+2Q+RM because it fits better right now. Got a discount offer from the seller and went ahead and bought it. It is just easier than messing with the Cisco stuff right now, even though I know it can't sustain bidirectional speeds for all ports at 100%, it is rated for non-blocking at half speed and the way I'll normally use it, this should be plenty fast enough, especially if I don't end up using the 40gbps ports for a while, which I probably won't need unless I expand a lot.

How about this ?

 

[Greg_E]

I need at least ten 10gbps ports right now, and most of those are SFP+ based (only 1 is base-T). The Mikrotik stuff is a known quantity, just like Cisco is a known quantity. I probably should have bought a Cisco because overall the performance would have been higher, but I don't think I need that level in my lab. I probably should have looked for used Extreme 5420 switches so I could mirror what I have for work and practice what could be dangerous things in the lab with the same equipment that is used in production.

All that said, I do like the web GUI for the Mikrotik stuff, I think the Netgear AV line is the next step up in web GUI. And people hate both of these. If you need simple work done, that Netgear AV line is just nice, they preset most of the things you would need. Yes kind of magic wand and you don't really know all the policies behind things, but sometimes it's nice to just click and button and "POOF", results.

 

[Greg_E]

The switch is here and seems to work. Waiting for a QSFP breakout cable so I only need to shut things down once to make some changes. Going to put both NICs from each of my ESXi servers on this cable because they sit next to each other and will be an easy way to get all 4 in the breakout used in one area. the third will need to be DAC cables pulled from the first two servers.