Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

[fohdeesha]

i have upgraded my C12P to SPR09000ufi. It takes a while to startup after reload. i did check terminal and i get these errors:

Creating TUN device
Creating the first Tap interface
ERROR: Failure in gw add
Starting TPM Infra
Group tss and User tss created
hostname set to Ruckus-ICX
could not read config file /etc/radsecproxy.conf
No such file or directory
Starting TCSD Daemon in 'fi' namespace


Any ideas how to fix this?

Don't run beta firmware

 

[Vesalius]

i have upgraded my C12P to SPR09000ufi. It takes a while to startup after reload. i did check terminal and i get these errors:

Creating TUN device
Creating the first Tap interface
ERROR: Failure in gw add
Starting TPM Infra
Group tss and User tss created
hostname set to Ruckus-ICX
could not read config file /etc/radsecproxy.conf
No such file or directory
Starting TCSD Daemon in 'fi' namespace


Any ideas how to fix this?

You will need to go back to 8.0.95 for most of that to go away. It’s what recommended by ruckus (and more importantly @fohdeesha ). Otherwise disregard those boot messages and things will likely work fine, at least they have for me on the icx7150 I’m trialing 9000 on. If sticking with 9000 upgrade to 9000a as it does fix some issues, but most of those boot up messages on the console will persist.

 

[Originalus]

Thank you guys. Did not thought it is so early beta.
I will stick with 9000 series though.

 

[Cncjerry]

For the 4x10 port on the back of the icx6610, I understand how the DAC cable works. I then read how you can use a QSFP and MPO cable to a patch panel. But the linked patch panel is 24 ports. Does it have some signal splitting in it? The MPO Is carrying 4 logical 16Gb down clocked to 10Gb. Does the panel then split them out? I saw how the linked patch patch panel has 4 ports on the back. Is it expecting 6 links on these ports? Sorry if this is an obvious question.

I can't believe how cheap and fast this stuff is. My mac mini M1 with the Sonnet 10Gb link displaying sheet music from a 24 core server using SAS SSD raid threw me off because the pages turned so fast I didnt realize they changed.

Thanks

Jerry

 

[dos]

was hoping someone could chime in on the 6610 4x10 breakout ports on that back. does anyone know if they can be used as a single 40gb port without needing to be a breakout?

 

[MonumentNetworking]

To be honest this is probably because you haven't read through the thread. Its in here numerous times sir. The what's possible and how to do it is in this thread. There's a search function if you want faster responses to pre-answered questions.

 

[Blue)(Fusion]

To be honest this is probably because you haven't read through the thread. Its in here numerous times sir. The what's possible and how to do it is in this thread. There's a search function if you want faster responses to pre-answered questions.

You're not wrong, but it also doesn't hurt to give a direct answer anyway.

was hoping someone could chime in on the 6610 4x10 breakout ports on that back. does anyone know if they can be used as a single 40gb port without needing to be a breakout?

No.

 

[dos]

To be honest this is probably because you haven't read through the thread. Its in here numerous times sir. The what's possible and how to do it is in this thread. There's a search function if you want faster responses to pre-answered questions.

keep in mind there are over 300+ pages to go through. shortly after i posted that i did find an answer. however i'm not sure if that's based on actual first hand experience or just "well the doc says so...". typically anything that is a 40gb "bundle" usually defaults as 40gb and then gets configured as a 4x10gb links. thanks for the reply non the less. i'm still digging through the thread but loads of useful info to say the least.

 

[dos]

You're not wrong, but it also doesn't hurt to give a direct answer anyway.


No.

thanks for that.
for the record i also agree it's important to go through information available to you and learn as much as you can, but at the same time the whole point of information like this is to make it available to others to prevent them from having to unnecessarily reinvent the wheel.
that said, definitely appreciate the hard work and info from everyone.

 

[fohdeesha]

was hoping someone could chime in on the 6610 4x10 breakout ports on that back. does anyone know if they can be used as a single 40gb port without needing to be a breakout?

first page: https://forums.servethehome.com/ind...rful-10gbe-40gbe-switching.21107/#post-196457

also in the guide: FCX / ICX6610 - Fohdeesha Docs

 

[i386]

People seem to forget that this is a forum and not a support chat/channel

 

[fohdeesha]

updated the intermediate fastiron image used for getting the icx7xxx series up to UFI images from 8090m to 8090mc (8090m had a defect according to ruckus, and was pulled and replaced by 8090mc) - the defect only affected the layer2 image which the guide does not use, and on top of that the guide only uses this image as a stepping stone to 8095, but I went ahead and updated it anyway for completeness update intermediate version from 8090m to 8090mc · Fohdeesha/lab-docu@6656f3d

 

[anomaly]

For 6450 switches, ACL sequence numbers aren't supported (seems that requires >8.0.60), what is the best way to add new entries to an ACL in that case?

 

[CIR-Engineering]

Could someone please tell me, are the fans and power supplies the same for the 6610, 6650, and 7450? I may need to try some mixing and matching.

Thanks!
craigr

 

[Blue)(Fusion]

For 6450 switches, ACL sequence numbers aren't supported (seems that requires >8.0.60), what is the best way to add new entries to an ACL in that case?

Use extended ACLs.

ip access-list extended aclname
remark DENY ALL OTHER INTER-VLAN TRAFFIC                         
deny ip any 10.0.0.0 0.255.255.255 log 
remark ALLOW REMAINING TRAFFIC
permit ip any any
enable-accounting
exit

When it's time to alter something, copy the current ACL to a text editor, make your changes, and add a line to the top to "no" the access-list which will delete the entire thing.

no ip access-list extended aclname
ip access-list extended aclname
remark ALLOW ESTABLISHED TCP TRAFFIC
permit tcp any any established
remark ALLOW SOME INTER-VLAN TRAFFIC
permit tcp any 10.0.0.0/8 eq ssl
remark DENY ALL OTHER INTER-VLAN TRAFFIC                         
deny ip any 10.0.0.0 0.255.255.255 log 
remark ALLOW REMAINING TRAFFIC
permit ip any any
enable-accounting
exit

 

[anomaly]

Use extended ACLs.

When it's time to alter something, copy the current ACL to a text editor, make your changes, and add a line to the top to "no" the access-list which will delete the entire thing.

OK, is this compatible with the SSH access list as well or it only accepts standard ones?:

access-list 1 remark SSH-Protection
access-list 1 permit XXX 0.0.0.255
access-list 1 permit host HHH
access-list 1 permit host DDD
access-list 1 permit ZZZ 0.0.0.255
access-list 1 deny any log

That is what my current list looks like.

 

[Blue)(Fusion]

OK, is this compatible with the SSH access list as well or it only accepts standard ones?:

access-list 1 remark SSH-Protection
access-list 1 permit XXX 0.0.0.255
access-list 1 permit host HHH
access-list 1 permit host DDD
access-list 1 permit ZZZ 0.0.0.255
access-list 1 deny any log

That is what my current list looks like.

Yes,

It would end up being:

ip access-list extended SSH-Protection
  permit XXX 0.0.0.255
  permit host HHH
  permit host DDD
  permit ZZZ 0.0.0.255
  deny any log
  enable-accounting
exit

Bind it to an interface almost the same as before, except now it's a name.

int ve 1234
  ip access-group SSH-Protection in
exit

 

[juju]

I am looking to setup a transit from my 7250 to my proxmox server - for dns and dhcp servers there. Finding it hard to wrap my head around how to set it up on both ends.

I have setup a lag on the 7250 - 1/1/2 and 1/1/4 . This is connected to an lacp interface on the proxmox box. Now not sure how to configure the lag interface on the 7250 and also on the proxmox side ( more of the proxmox question so not relevant here but hoping someone has done it )

 

[Blue)(Fusion]

I use openvswitch on Proxmox and it works great. Use OpenVSwitch to bond the interfaces (LACP LAG), create an openvswitch interface on that bond with the IP address of the Proxmox server and default route as appropriate, and then when you configure the vNICs on your VMs, you can simply have it tag a VLAN as desired.

Here's my /etc/network/interfaces from one of my Promox servers:

auto lo
iface lo inet loopback

auto eno1
iface eno1 inet manual

auto eno2
iface eno2 inet manual

iface eno3 inet manual

iface eno4 inet manual

auto enp6s0
iface enp6s0 inet manual

auto enp6s0d1
iface enp6s0d1 inet manual

auto pve
iface pve inet static
address 10.23.70.11/24
gateway 10.23.70.1
ovs_type OVSIntPort
ovs_bridge vmbr0
ovs_options tag=2370

iface pve inet6 static
address 2603:xxxx:xxxx:2570::11/64

auto pvebackup
iface pvebackup inet static
address 10.23.75.11/24
ovs_type OVSIntPort
ovs_bridge vmbr1

auto bond0
iface bond0 inet manual
ovs_bonds enp6s0 enp6s0d1
ovs_type OVSBond
ovs_bridge vmbr0
ovs_options other_config:lacp-time=fast lacp=active bond_mode=balance-tcp

auto bond1
iface bond1 inet manual
ovs_bonds eno1 eno2
ovs_type OVSBond
ovs_bridge vmbr1
ovs_options bond_mode=active-backup

auto vmbr0
iface vmbr0 inet manual
ovs_type OVSBridge
ovs_ports bond0 pve

auto vmbr1
iface vmbr1 inet manual
ovs_type OVSBridge
ovs_ports bond1 pvebackup

EDIT to add:

In my case, the Proxmox management IP addresses are on interface "pve". The pve interface is not tagging VLAN traffic (although each VM vNIC is tagged as needed in their settings). Therefore, all VLANs are tagged on this LAG but the management IP network, which is dual-mode (ICX6xxx series term, not sure what it is in ICX7xxx).

 

[juju]

I use openvswitch on Proxmox and it works great. Use OpenVSwitch to bond the interfaces (LACP LAG), create an openvswitch interface on that bond with the IP address of the Proxmox server and default route as appropriate, and then when you configure the vNICs on your VMs, you can simply have it tag a VLAN as desired.

I did this but using a linux bridge. I am now able to ping the proxmox vm created for the dhcp server from the switch, but cant ping the switch from the proxmox server though they are directly connected via their LACP interfaces.