Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[itronin]

Right - that was the implied problem encountered - so what is the solution where I have a workstation on a port that I need to belong to multiple VLANs (feel like I'm missing something obvious).

Is there a reason you are not routing your VLANs within the 6450?

itr

 

[svtkobra7]

The solution is obvious - you need a NIC that supports VLANs if the workstation needs to be on multiple VLANs.

The only other alternative is multiple NICs, each on a port with a different untagged VLAN.

I do have two attached so definitely an idea! I'm surprised that all I see when looking at the adapter for Packet Priority & VLAN = VLAN enabled etc (no ability to enter a VLAN) ... maybe it has to be done via powershell or something? I can google it. I remember INTELProSet to have been much more "robust" when installed previously some time ago.

Is there a reason you are not routing your VLANs within the 6450?

itr

If I understand your question correctly, there is nothing to route atm as everything is on 10.0.0.0/24 (except virtual ESXi networking). When I get there I ultimately plan to segment "LAN" (i.e. today) from server networking and will route at that point. My knowledge in this genre is quite thin as I didn't set up my first vlan until I got the 6450 a number of months ago so everything takes me quite some time initially.

 

[Blue)(Fusion]

@itronin had a valid point. Just route all the VLANs at the switch.

Each VLAN will need a ve with an address/subnet. Change all your hosts to use the ve addresses as their gateway. With no ACLs, all traffic is routed now at the switch between VLANs so you won't lose access - provided DHCP is configured correctly upon changeover.

Ideally, you'll keep using your ve addresses as gateways anyway. Any inter-VLAN firewalling you want to do can be done with ACLs. I've only started using this method about a month ago and it works great. Far superior to pfSense, in terms of throughput (although not stateful, and requires a little more thought/trial-error).

 

[svtkobra7]

@itronin had a valid point. Just route all the VLANs at the switch.

Each VLAN will need a ve with an address/subnet. Change all your hosts to use the ve addresses as their gateway. With no ACLs, all traffic is routed now at the switch between VLANs so you won't lose access - provided DHCP is configured correctly upon changeover.

Ideally, you'll keep using your ve addresses as gateways anyway. Any inter-VLAN firewalling you want to do can be done with ACLs. I've only started using this method about a month ago and it works great. Far superior to pfSense, in terms of throughput (although not stateful, and requires a little more thought/trial-error).

Thanks @itronin and of course @Blue)(Fusion

Q: Is this suggestion for implementation today or forward looking? I suppose I don't understand quite yet as I haven't spent the time to sit down and learn all of this. And just seeking clarity in regards to your much appreciated guidance! Note: I've snapped current networking for ESXi-02 and placed it in the below spoiler as reference for my lack of understanding. Please kindly excuse / understand my lack of being able to speak intelligently on topic at all.

Assumption is that everything works today as I moved the router interface off the default vlan and because my network is flat, but once I create additional subnets, those ves are needed so everything gets routed appropriately?

Drawing on my very limited exposure, your comments wouldn't apply to today's in progress net where ESXi is routing, right? Example: I have a storage network set up on 10.0.30.0/19 for NFS41 and ESXi appears to be handling that routing just fine.

Since you brought up pfSense, one of the items I know I will need to find a solution for is pfSense not assigning DHCP to subnets other than the one it is connected to. How did you go about handling that?

Thanks again - this is one area where I would never shy away from advice!

Spoiler: ESXi-02 Networking, std switches

NB:

• FreeNAS is only connected to the pfsense LAN port group as a "hack" to get iocage / vnet networking working correctly. mtu = 9000 across the board. Not setting mtu = 9000 on that interface in FreeNAS allows me to successfully bridge virtual networking and provide DHCP to iocage.
  
• You don't actually see the pfSense VM there as it is on ESXi-01. I haven't been able to figure out how to get pfSense to work in HA, yet, so I just vMotion it if I need to take down ESXI-01.
  
• Very much a work in progress (as noted prior), but my storage network and vMotion network work. Sadly that is a big win for me.
• Noted prior but once I've built all of this out in standard switches in ESXi-01 & ESXi-02, I'm going to migrate the MLNX vSwitches (2 x 2 x 10 Gb) to a distributed switch (and deploy LACP) and likely leave the INTL vswitches (2 x 2 x 10 Gb) as standard, which are used for only pfSense today. Also, I'll probably stick an extra management interface on the INTL switch in case I have an issue with dvs.

 

[itronin]

I'd say implementation for today. The further down the path you go the more work you will create for yourself making changes to get things working. I don't mean that as a criticism in that you are doing things poorly but you may have some issues that you've designed yourself into (probably unintentionally)... then again maybe your design choices were intentional..

so if you want to be able to access IP's uniformly across your subnets via routing you are probably going to have make some adjustments to how you have it set up. As @Blue)(Fusion pointed out you can use ACL's to lock/limit access. I'd put that one as down the road because its easier to troubleshoot getting everything working first then troubleshoot locking things down. I sense from your design choices that you do ultimately want to lock down/limit access (for realsy, learning exercise or cause you can... etc.)

for example:
>Example: I have a storage network set up on 10.0.30.0/19 for NFS41 and ESXi appears to be handling that routing just fine.
^^^^^^^^^
I could be misreading your spoiler with my old eyes but the screenshot doesn't show a /19. When you say routing just fine... are you accessing
10.0.30.0/27 from other subnets - so you have set up ve's?

VLAN 30 is actually 10.0.30.0/27 <-- 30 addresses
VLAN 100 10.0.0.0/16 <- a lot of addresses.
VLAN 100 overlays VLAN 30

You may experience challenges at some point getting hosts in VLAN 100 to successfully talk to hosts in VLAN 30.

That may not be important but as you build this out if you have not already thought about monitoring - you will. You could get around that by
tossing your monitoring in a NON 10 private space.

Working with supernetting and subneting can be exciting and you can do and learn cool stuff with routing protocols.

But I sense that you are learning some of this as you go. You may want to consider downsizing things a bit early on and then move out to supernetting down the road. Maybe think in terms of /24's..You are working with Net 10 you still have LOTS of subnets to play with. Keep sizing consistent among your subnets for now too pare it down into things that are < /24 later to see how that works...

From the address space sizes it also sounds like you will be playing with lots of VM's... when I think of a home lab with lots of VM's I think of 50-100 or so. that fits nicely in a /24 (class c space)...

Why is this relevant. Think of your infrastructure as a house. if you have issues in your foundation then those translate into issues with the frame. going back and fixing the foundation can be a challenge.

My comments are not particularly relevant to cool Brocade switch thingies @svtkobra7 so if you'd like to continue this discussion maybe it should be moved into a thread of its own?

 

[svtkobra7]

My comments are not particularly relevant to cool Brocade switch thingies @svtkobra7 so if you'd like to continue this discussion maybe it should be moved into a thread of its own?

• Thank you very much for the thorough reply!
  
• After I've replied to each of your comments, I arrived at your suggestion for more appropriate placement, which I completely agree with and would love to take you up on the offer!
  
• Is VMware Networking more VMware, VirtualBox, Citrix or Networking ???

 

[itronin]

I'd say Networking based on where you are at. its really multi-discipline between Networking and VMware eventually. Perhaps you can entreat a mod to move it as they see appropriate? Or you can pick one or the other start a new thread and we can all pick up the discussion there.

 

[Blue)(Fusion]

Hmmm, it appears this Brocade thread is ALL over the place anyway, so I'll continue to respond in turn here. I hope a mod can start splicing this thread up into various new threads and keep the Brocade ICX-specific stuff in here.

Anyway, @svtkobra7 , your setup isn't far off from mine.

pfSense is on WAN firewall, DNS, DHCP, RADIUS, and NTP duties. So it can offer DHCP leases properly to each VLAN/subnet and respond to other requests on each VLAN/subnet I set up an interface on almost all of my VLANs. One VLAN is strictly for jumbo-frame Gluster traffic between a handful of hosts which have static IPs - it is strictly internal and needs no DNS, DHCP, etc. as each of the hosts are also members of other VLANs. pfSense is connected with a LACP LAGG of 3x Gigabit connections to the ICX.

The ICX is on router and VLAN "firewall" duty. I am using ACLs to restrict and pin-hole access between VLANs. Each VLAN (save the ve 4 for Gluster traffic) has a virtual ethernet interface ("ve") with an IP address. I just use /24s for each VLAN.

interface ve 2
 acl-logging
 ip access-group 102 in
 ip address 10.1.2.1 255.255.255.0
!
interface ve 3
 acl-logging
 ip access-group 103 in
 ip address 10.1.3.1 255.255.255.0
!
interface ve 5
 acl-logging
 ip access-group 105 in
 ip address 10.1.1.1 255.255.255.0
!
interface ve 6
 acl-logging                                                   
 ip access-group 106 in
 ip address 10.1.6.1 255.255.255.0
!
interface ve 10
 acl-logging
 ip address 10.1.10.1 255.255.255.0
!
interface ve 20
 acl-logging
 ip access-group 120 in
 ip address 10.1.20.1 255.255.255.0

If I were to remove the ip access-group lines from each of the above and all of my connected devices use the above IP respective of its associated VLAN as the gateway, each host can access all other hosts on every other VLAN. Basically, no firewall between VLANs.

FWIW, my VLAN setup currently:
1 - default unused
2 - VoIP phones
3 - IoT (cameras, firestick, smart TVs, smart thermostat)
4 - Storage
5 - Management
6 - Servers
10 - My desktops and laptop
20 - Guest (cell phones, guest devices, roommate's stuff, etc).

I have two Proxmox nodes which have all of the above listed VLANs trunked as well. This allows me to put any guest on any VLAN - even multiple virtual NICs for multiple VLANs.

All ports on my switch are set to be untagged VLAN 20 "guests" with only a few exceptions (cameras and IP phones). Everything else that I want on seperate VLANs are set locally at the device. Since I use lmost exclusively Linux, here's what I do:

ip link set dev enp3s0 up
ip link add link enp3s0 name enp3s0.4 type vlan id 4
ip link add link enp3s0 name enp3s0.10 type vlan id 10

ip link set dev enp3s0 mtu 9000
ip link set dev enp3s0.4 mtu 9000
ip addr add 10.1.4.10/24 dev enp3s0.4
ip link set enp3s0.4 up

This is an example on my desktop to be part of my "trusted" VLAN10 and "storage" VLAN 4.

I also use extensive ACLs for intra-VLAN traffic management. For example, VLAN3 (IoT) is not allowed to talk to any other VLAN unless it's a tcp connection to port 22, 80, or 443 that was initiated from VLAN 10 and has very specific allowances for a few devices to access the WAN.

 

[zxv]

Do ACLs on VLAN routing have any significant impact on latency or bandwidth?

 

[Blue)(Fusion]

Do ACLs on VLAN routing have any significant impact on latency or bandwidth?

Nope. Not on the ICX series, atleast.

 

[arglebargle]

Not sure if it was mentioned on this thread or not. I'm looking to connect two of these boxes via SFP to Ethernet 10GBaseT modules in order to utilize Cat-6 already in the house. Can anyone recommend some decent bang-for-the-buck transceivers?

These seem to be the bargain go-to at the moment: MikroTik

People are reporting varying levels of success, they seem to work well over moderate distances but get extremely hot with longer runs. Another user posted a couple of days ago reporting that his hit ~80C over a 125ft run, though to be fair they're only rated for use up to 30m / 98ft @10Gb.

https://forums.servethehome.com/index.php?threads/problems-with-mikrotik-s-rj10.24173/

FS.com also sells one, but I don't know much about it: FS SFP-10G-T 10GBASE-T SFP+ Copper RJ45 Transceiver Module

I don't know too much about actual performance for either module, I pulled fiber for my longer runs. Based on what I'm reading though you might want to take the time to fish fiber, unless your runs are short it sounds like there are significant drawbacks to the 10Gb modules. If your run is long and you've got enough copper in the walls I'd LACP 2-4 links at 5Gb and see how the modules do.

Sorry everyone, I know this is pretty off-topic for the Brocade thread. @BThunderW - You might want to make a separate thread, you'll probably get more info that way.

 

[itronin]

Hmmm, it appears this Brocade thread is ALL over the place anyway, so I'll continue to respond in turn here. I hope a mod can start splicing this thread up into various new threads and keep the Brocade ICX-specific stuff in here.

Yes hidden gems and treasures. I concur with what @Blue)(Fusion is recommending on ACL's.

@svtkobra7 Unless you're in a hurry I'd pause and take a step back for a moment. Based on what you've posted you have at least 1 underlying IP challenge. I also do disagree with using gi-normous subnets. (that's probably not fatal but...)... However that is up to you.

Warning may be TLDR.

<#Disclaimer>
What I've written here is an opinion and is not necessarily the right or best way for you - that's up to you to decide. Just offering ideas, food for thought etc.
</Disclaimer>

This is all probably overkill and may be extreme - but a stripped down version of this thought process is still IMO prudent. KISS is important too, over engineering / designing can also lead to challenges - but there's a balance here. If you think about it ahead of time and plan and maintain simplicity - adding functional areas, services, networks later etc is a lot easier. Not unlike an addition to a house if the house was designed for additions in the first place.


so:

Plan - you've already done good work mapping out storage, mgmt planes, and the like.

What services are "mission critical" in your environment? Don't forget WAF, SAF, PAF, RAF... Nothing like a good Saturday evening home engineering session that is met with a sudden silence and then yells from other rooms when Netflix no longer work-ie instead of Wookie-ing.

Does this network support household mission critical services?
If no - continue on to the networking section.

If yes - think carefully about whether to segregate those services entirely or build them into this infrastructure and protect them.
Here's a starter list and is by no means complete - really just something to get the noodle churning.

In no particular order:

LAN Services
DHCP (@Blue)(Fusion uses Pfsense, there are other options: switch based? dedicated server based with switch helper? etc.
DNS (are you running internal DNS?, 1 zone, many zones? if many think about how you will map zones to subnets)
Syslog
Internet (well depends on DHCP, DNS, transit, NAT etc. etc.) ?
Internal Backup of desktops laptops and the like?
Peer to Peer access and/or storage?
Shared Central Storage? (what forms? NFS, CIFS, Bonjour)
Monitoring of your infrastructure?
Printers?
VoIP?
IoT things (Ring, Nest, SCADA, 3d printers, Cameras)
WiFi (multiple SSID's mapping into the above?)
MFG ecosystems (for example, Apple)
Internet to your network VPN?
Classes of Users
Pinholing services (please don't)
Technology test beds (are you testing new (to you and/or industry) technologies) and if so who/what needs access to it?

Are you going to fulfill any of these services via VM's? One service per VM?

Once you've got your beginning set of services start thinking about the network and inter-relationship between the services

Networking

Consider whether you want to segregate any of the following (I think you do based on your previous posts):
IPMI Network? (mgmt plane)
vmware mgmt network? (mgmt plane)
storage
NFS 4.1
vSAN (are you going to play with this? do you want it on a separate network? plan now if you do - implement later if you want)
CIFS?
IOT (or cameras, or NEST, or RING etc.)
WAN
LAN <-> Internet
VPN <-> Internet <-> Your Network
Are you going to be doing IPv6? If so how and where.

Virtualization:
vMotion or other vendor equiv.?
vSAN or other vendor equiv. ?
mgmt planes?
prviate within virtual host network?

Physical interconnects, meta layers and guest system presentation.
only going to say this once for your VM hosts: trunk and tag, trunk and tag, trunk and tag
Even with with things like FreeNAS: consider trunk and tag.
leverage meta layers within your solutions to minimize impact of lower level changes.
Link AGG / connection protection - Build your network first and it may be simpler / easier to get working with single connections and add link agg or the like later.

IP-Wise (only talking about ipv4 here - ipv6 whole 'nother discussion)

Think about the subnet ranges you are going to use:
Is 10/8 right? 172.16/16? 172.27/16? 192.168/16? (please don't use 192.168.1-.3) You have a ton of options and the right one for you may be shaped by your answers to some of the previous questions.

Do you need to worry about IP subnet conflicts from outside coming in (VPN)?
Do you need to worry about IP subnet conflicts from inside going out to other networks (VPN)?
What classes of systems/clients need access to what?

You've already picked what I consider a nice convention, if possible use subnet numbering to match VLAN ID (your storage network for example)
Are there any other conventions to use? This can lead to limiting the number of VLANs but unless you are doing boundary condition testing 2 - 254 vlans may be sufficient. You can do more and still match vlan ID to subnet by extending into the next octet of the subnet.

What size subnets do you need? Do you want to add the complexity and brain cycles thinking about supernetting and/or classless subnetting while troubleshooting?

Access Control
@Blue)(Fusion has excellent points.

How do the various points that you map out fit into the capabilities of the ICX 6610 - what can you take advantage of and what do you need to learn to do so?

Last questions:
There's a bit of focus in your posts about virtualization, how many of your critical services are depending on it and if that part of your infrastructure fails, how fast can you implement a fall back or work-around and what is that plan?
Do any of the design/implementation decisions you are making create additional complexity for that plan? If so should that complexity be simplified?

trade-offs there are always trade-offs.

itr

 

[itronin]

Possible 2post RM kit solution for ICX6610

I don't know if these are sturdy enough to support a 6610 in a 2 post rack but at $26.00 I'm going to give it a shot to see if it works. At a minimum it will keep my switches from sliding back on the shelf they are resting on.

Brocade ICX6610-48P-PE Switch Rack Mount RMK Kit Corners Brackets | eBay

itr

I received my brackets. I purchased two sets but what I received was either 7 of one type and 1 of the other the 7 are semicircle brackets (most holes). the screw holes line up front or back on all the brackets though. the majority of my brackets have the extra holes whether front or back. I only need 4 brackets for my two switches so I will likely have a extra set in the not too distant future. I am guessing these brackets are used on multiple products based on the extra holes.

Does anyone know the proper length of the screws to mount the brackets to the switch's sides and I'll ask : any idea what the thread is? I'm disinclined to haul a 6610 down to the local ACE Hardware but may have to do it.

thanks,

itr

 

[Dave Corder]

Does anyone know the proper length of the screws to mount the brackets to the switch's sides and I'll ask : any idea what the thread is? I'm disinclined to haul a 6610 down to the local ACE Hardware but may have to do it.

I did that a couple weeks ago, but I neglected to note what the threads are. As I recall, there are two different sizes of threaded holes on the sides and I bought several screws for each size. I received my brackets yesterday, but I won't have time for a while to pull my 6610 out to check the alignment or sizes. If I have time this weekend, I may take the screws back and check them in the little thread-checker thingie, but no promises.

 

[arglebargle]

I received my brackets. I purchased two sets but what I received was either 7 of one type and 1 of the other the 7 are semicircle brackets (most holes). the screw holes line up front or back on all the brackets though. the majority of my brackets have the extra holes whether front or back. I only need 4 brackets for my two switches so I will likely have a extra set in the not too distant future. I am guessing these brackets are used on multiple products based on the extra holes.

Does anyone know the proper length of the screws to mount the brackets to the switch's sides and I'll ask : any idea what the thread is? I'm disinclined to haul a 6610 down to the local ACE Hardware but may have to do it.

thanks,

itr

I'd be more than happy to take those off your hands, I could definitely use rear brackets for my switches. Let me pop a couple of screws off one of mine and measure them for you.

edit: Oops, I can give you length but I don't have any way to measure thread pitch.

edit2: They're 5/16" (8mm) long and the over-diameter on the threads is roughly 4mm, I don't have any way to measure pitch.

edit3: All screw holes on mine are identical.

 

[itronin]

I did that a couple weeks ago, but I neglected to note what the threads are. As I recall, there are two different sizes of threaded holes on the sides and I bought several screws for each size. I received my brackets yesterday, but I won't have time for a while to pull my 6610 out to check the alignment or sizes. If I have time this weekend, I may take the screws back and check them in the little thread-checker thingie, but no promises.

@Dave Corder
take a close look at the brackets you received. do you see any way to mount them to the rear? At the back of my ICX 6610 I only have 3 holes all along the center line. they don't line up with any brackets that I received to secure more than a single hole at the rear. Front holes line up to 6 screw holes, the three in a triangle towards the center and two center line and one towards the bottom.

For me I only cared about the front brackets and for what I paid vs what a ICX6610-RMK costs(if you can find one) I am not complaining. Just making observations.

EDIT:
hmmm... Found the hardware installation guide. I did not see a mention of rear brackets for the two post installation. So dunno about those. I did find the BOM for kit. ....

Snip of the pertinent page is attached for anyone else that needs it. Ears are attached with:

Screw, 8-32 x 3/8 in., panhead Phillips

EDIT:

8-32 x 3/8 fit perfectly.

 

[fohdeesha]

Do ACLs on VLAN routing have any significant impact on latency or bandwidth?

zero impact, if they did it definitely would not be a layer 3 switch

 

[arglebargle]

So it looks like those are definitely two-post front brackets, the four-post mounting kit looks like this:

 

[Dave Corder]

@Dave Corder
take a close look at the brackets you received. do you see any way to mount them to the rear? At the back of my ICX 6610 I only have 3 holes all along the center line. they don't line up with any brackets that I received to secure more than a single hole at the rear. Front holes line up to 6 screw holes, the three in a triangle towards the center and two center line and one towards the bottom.

For me I only cared about the front brackets and for what I paid vs what a ICX6610-RMK costs(if you can find one) I am not complaining. Just making observations.

EDIT:
hmmm... Found the hardware installation guide. I did not see a mention of rear brackets for the two post installation. So dunno about those. I did find the BOM for kit. ....

Snip of the pertinent page is attached for anyone else that needs it. Ears are attached with:

Screw, 8-32 x 3/8 in., panhead Phillips

EDIT:

8-32 x 3/8 fit perfectly.

Any sag using those just two brackets?

 

[Churchill]

I got my switch up and going, firmware updated, it's not so bad as I thought. definately need some mounting brackets for this monster. Can anyone recommend me a good rack mount kit for this unit?

Will this one work?
Lenovo Fixed Rack Mount 1U and 2U for Four-Post Racks Brocade 00YHY936 ZZ 883436349819 | eBay