Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[ljvb]

We're posting on STH and have 40gbE switches in our closets, I think we crossed the overkill line a ways back

I'm pretty sure I went well past overkill a long time ago

I tried to post pictures.. but they are showing up broken.. but it was a picture of 1 DL180 (now a DL380G8), 1 Sipermicro 15 drive chassis, 1 HP C3000 blade chassis with 2 blades, each with 2 16 core Xeon E5-xxxx ad 64GB ram...

Edit: pictures worked finally

 

[tommybackeast]

so the POE power supplies are actually two power supplies in one. Both the PoE and non-PoE power supplies have a ~250w 12V supply in them, that's identical between them. This is what actually powers everything in the switch. But the PoE PSU has an additional ~750w 54V supply in it, and the PoE daughterboard takes power from this rail and injects it via the PoE PHY components onto PoE enabled ports.

So with a POE PSU in a switch that doesn't even have a POE daughterboard, the 54V rail will not even be touched/used. The PoE commands are there, but it's smart enough to give you an error if you try them on a non-poe chassis, or a poe chassis with non-poe supplies. The normal 12V rail that will be used will be coming from the same ~250w supply as a non-poe PSU

you are speaking of the 6610 in the above paragraphs, yes?

 

[tommybackeast]

nope should work fine. just keep in mind you don't need to reverse airflow unless your servers (heat generators) actually extend all the way to the rear posts of the rack (so the heat is exhausting right into the front face of the 6450). If they're short servers that end in the middle of the rack, and you reverse airflow of the 6450 and rear mount it, it's now going to be taking in air from it's rear, which is towards the middle of the rack where the short servers are exhausting - obviously not preferred compared to just leaving it stock and having it pull in fresher air from the very back of the rack

As always, thank you for your experience. It is a 12U Server Rack Cabinet and enclosed (XrackPro2) so air vents incoming on the bottom; and fans on the rear door. There will be a large 30"? deep Dell Server near the bottom; and a 26"? Synology NAS above it. (call it middle of rack). Hot air from the Dell Server and Synology will rise; and if I reversed the Switch fans, the switch will be sucking in hot air. (not good). So shall mount Switch on top, in rear, but not reverse the switch fans. I picked up a cheap temperature probe for my new/old Lantronix Network PDU, so can monitor temps with that. / Thanks for making me think about the hot air flow within the rack, for I was not.

 

[tommybackeast]

I'm pretty sure I went well past overkill a long time ago

I tried to post pictures.. but they are showing up broken.. but it was a picture of 1 DL180 (now a DL380G8), 1 Sipermicro 15 drive chassis, 1 HP C3000 blade chassis with 2 blades, each with 2 16 core Xeon E5-xxxx ad 64GB ram...

on top pix, what is that small tiny LED device on the bottom? with the four arrows ?

 

[ljvb]

on top pix, what is that small tiny LED device on the bottom? with the four arrows ?

That is just the led display on the C3000 blade chassis. Pain in the but to use, but needed to configured initial IP address of the management module.

 

[PGlover]

Over the last 2 weeks, I have had a lot of Wi-Fi connection issues with my UniFi AP-AC-Pro APs connected to the ICX6610 in a stacked configuration. I have 3 UniFi APs.

Does the UniFi APs play nice with the ICX6610? What APs are other folks using?

 

[fohdeesha]

works perfectly, UAP AC PRO here to a stack of 2x 6610's. see my note I tagged you in earlier about the built in dhcp server if you're still using it

 

[PGlover]

works perfectly, UAP AC PRO here to a stack of 2x 6610's. see my note I tagged you in earlier about the built in dhcp server if you're still using it

I am still using the built in dhcp server. The SSID that is using the DHCP server on the ICX6610 is the one having a problem. The other SSID (for Guest Access) is using the DHCP server from pfSense and I am not having a problem at all with that SSID.

I need to find your post on this topic.

 

[PGlover]

Gave the built in DHCP server a go to see if I would run into your issue, and I did indeed with a couple IOT type stuff. After a ton of debugging, it turned out to be the fact the FastIron DHCP server is not set as authoritative, and some DHCP stack implementations (like Roku's and Sonos) do not like this, and will ignore it.

I went ahead and just spun up a really lightweight debian VM running isc-dhcp-server, it has it's own transit vlan to the switch, and the switch is set to relay all DHCP requests to it.

Gives me WAY more control than the built in dhcp server in FastIron or pfsense, and with glass-isc-dhcp you get really nice visualizations and lookup of all leases that you wouldn't get otherwise: Akkadius/glass-isc-dhcp

let me know if you want some help setting it up as well as my isc-dhcp config, it's as simple as just apt install isc-dhcp-server then copy in a config, and it'll serve DHCP requests to multiple routed vlans on your switch (just need an ip-helper statement under each VE on the switch)

Great information.. I may try using pfSense as my DHCP server for my HOME VLAN. Need to figure out what is required to set this up...

 

[kapone]

Great information.. I may try using pfSense as my DHCP server for my HOME VLAN. Need to figure out what is required to set this up...

For pfSense to be the DHCP server for "anything", it needs to be an interface in pfSense.

The problem is, if you're using a layer 3 switch then your VLANs are defined at the switch level and you want the switch to do the routing, not pfSense.

And if you define the interface at the pfSense level, your layer 3 switch essentially becomes layer 2 for that VLAN. Not ideal.

And pfSense DHCP is not "VLAN aware", so you can't use the "ip-helper" command at the switch to point it to the pfSense IP. Not ideal.

The logical solution is to use a dedicated DHCP server (that is VLAN aware), that is outside pfSense and the switch, and then use the ip-helper commands to point to it.

 

[sean]

I have pfSense running DHCP on a VLAN using a tagged interface. Checking FreeBSD does show the VLAN interface.

 

[PGlover]

I have pfSense running DHCP on a VLAN using a tagged interface. Checking FreeBSD does show the VLAN interface.

That is how I am handling my Guest Wireless on VLAN 30 with pfSense running DHCP on that VLAN. So, I would need to pass VLAN 1 (Home LAN) on the trunk port going to pfSense. Within, pfSense, I would need to create another subinterface on the same port that has VLAN 2000 (Transit Network) and VLAN 30 (Guest Wireless).

I have several WIN Server 2012 and 2016 VMs. I am working if it is easier to setup Microsoft DHCP services on WIN 2012 or 2016 and then have my ICX6610 point to it?

 

[tommybackeast]

I do not understand everything on there, but compliments to how well it is done and the time it took to make it.

 

[kapone]

I am working if it is easier to setup Microsoft DHCP services on WIN 2012 or 2016 and then have my ICX6610 point to it?

That's what I use. Dead simple.

Create scopes for each VLAN in DHCP services, give them IP ranges and you're done. If you also use DNS on Windows, the DNS registration is automatic (assuming you set a few checkboxes).

 

[PGlover]

That's what I use. Dead simple.

Create scopes for each VLAN in DHCP services, give them IP ranges and you're done. If you also use DNS on Windows, the DNS registration is automatic (assuming you set a few checkboxes).

Can you share the ICX6610 commands to setup the IP helper to point to the DHCP server.

 

[kapone]

Can you share the ICX6610 commands to setup the IP helper to point to the DHCP server.

ip helper-address n xxx.xxx.xxx.xxx

- where n is a number starting from 1. If this is the only "helper", use 1.
- xxx.... is the IP address of the DHCP server.

You would enter this command while in the "interface" for the selected VLAN.

 

[PGlover]

Just need to figure out how to undo all these commands...

ip dhcp-server enable
ip dhcp-server pool 192.168.1.0/24
network 192.168.1.0 255.255.255.0
excluded-address 192.168.1.1 192.168.1.99
dhcp-default-router 192.168.1.2
dns-server 75.75.75.75 75.75.76.76 8.8.8.8 8.8.4.4
exit
ip dhcp-server pool 192.168.1.0/24
deploy

 

[fohdeesha]

it's extremely complex and may take hours so buckle up

no ip dhcp-server enable
no ip dhcp-server pool 192.168.1.0/24

 

[ljvb]

I had an issue using the helper command on the lag, till I realized I needed to issue the command on the VE of each of the associated vlans.

I recommend, if you don't need active directory, or any of the other services MS Server provides, to install a basic installation of linux (ubuntu is probably the best documented online outside of RedHat), and isc-dhcp-server. Really easy to configure, and much lighter on system resources, I run mine in a VM. I posted my configurations below.

My dhcp servers IP is 192.168.101.2

apt install isc-dhcp-server

/etc/default/isc-dhcp-server

INTERFACESv4="ens192"

/etc/dhcp/dhcpd.conf
Replicate each VLAN section for each subnet you want to provide dhcp for. You only need the top section once, although you can use each of those directives within the subnet ranges if you want something other than the defaults you set.

option domain-name "$yourdomainhere";
option domain-name-servers 192.168.16.1, 1.1.1.1;
default-lease-time 3600;
max-lease-time 7200;
authoritative;

# VLAN 15, VE Int 15
subnet 192.168.15.0 netmask 255.255.255.0 {
        option routers          192.168.15.1;
        option subnet-mask      255.255.255.0;
        range   192.168.15.50 192.168.15.100;
}

If you want to do static address mapping, For the vlan 101 section, I left out the "range" directive, only providing the default router IP and netmask.

subnet 192.168.101.0 netmask 255.255.255.0 {
        option routers          192.168.101.1;
        option subnet-mask      255.255.255.0;
}
host plex {
        hardware ethernet 00:50:56:9f:ba:42;
        fixed-address 192.168.101.20;
        option host-name "plex";
}

 

[kapone]

While nothing wrong with using Linux (it is most certainly one of the choices), Windows DNS and DHCP work perfectly fine, even if you don't need AD/Domain.

An excellent article on that is: https://blogs.msmvps.com/acefekay/2013/06/12/dns-dynamic-updates-in-a-workgroup/