Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Spindle2274

New Member
Dec 1, 2023
4
0
1
For the life of me, I cannot ssh into the 6610, even after following all instructions multiple times.

My RSA key is 2048 bits.

Even if I enable password auth, it says my password is wrong when it isn't.

Any help would be greatly appreciated:

Here is my ssh config (all combos of commenting out lines have been tried):


Code:
StrictHostKeyChecking no
Host 192.168.2.2 (my switch)
#  PubkeyAcceptedKeyTypes +ssh-rsa
  HostKeyAlgorithms +ssh-rsa
  IdentitiesOnly yes
  KexAlgorithms +diffie-hellman-group1-sha1
  IdentityFile ~/.ssh/id_rsa_legacy
#  Ciphers aes256-ctr
Here are my verbose ssh logs from the connection attempt:

Code:
OpenSSH_8.9p1 Ubuntu-3ubuntu0.4, OpenSSL 3.0.2 15 Mar 2022
debug1: Reading configuration data /home/serveradmin/.ssh/config
debug1: /home/serveradmin/.ssh/config line 2: Applying options for 192.168.2.2
debug3: kex names ok: [diffie-hellman-group1-sha1]
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.2.2 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/serveradmin/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/serveradmin/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to 192.168.2.2 [192.168.2.2] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: Connection established.
debug1: identity file /home/serveradmin/.ssh/id_rsa_legacy type 0
debug1: identity file /home/serveradmin/.ssh/id_rsa_legacy-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.9p1 Ubuntu-3ubuntu0.4
debug1: Remote protocol version 2.0, remote software version RomSShell_5.40
debug1: compat_banner: no match: RomSShell_5.40
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 192.168.2.2:22 as 'root'
debug3: record_hostkey: found key type RSA in file /home/serveradmin/.ssh/known_hosts:138
debug3: load_hostkeys_file: loaded 1 keys from 192.168.2.2
debug1: load_hostkeys: fopen /home/serveradmin/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,sntrup761x25519-sha512@openssh.com,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ssh-ed25519@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com
debug2: ciphers ctos: aes256-ctr
debug2: ciphers stoc: aes256-ctr
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa
debug2: ciphers ctos: aes256-cbc,aes192-cbc,aes128-cbc,aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: aes256-cbc,aes192-cbc,aes128-cbc,aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha1
debug2: MACs stoc: hmac-sha1
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: diffie-hellman-group1-sha1
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes256-ctr MAC: hmac-sha1 compression: none
debug1: kex: client->server cipher: aes256-ctr MAC: hmac-sha1 compression: none
debug2: bits set: 518/1024
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: SSH2_MSG_KEX_ECDH_REPLY received
debug1: Server host key: ssh-rsa SHA256:hDr/FVms/sPE/zGI2X835eGo5Uo845zv8mM4PN7qnBA
debug3: record_hostkey: found key type RSA in file /home/serveradmin/.ssh/known_hosts:138
debug3: load_hostkeys_file: loaded 1 keys from 192.168.2.2
debug1: load_hostkeys: fopen /home/serveradmin/.ssh/known_hosts2: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts: No such file or directory
debug1: load_hostkeys: fopen /etc/ssh/ssh_known_hosts2: No such file or directory
debug1: Host '192.168.2.2' is known and matches the RSA host key.
debug1: Found key in /home/serveradmin/.ssh/known_hosts:138
debug2: bits set: 507/1024
debug3: send packet: type 21
debug2: ssh_set_newkeys: mode 1
debug1: rekey out after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: ssh_set_newkeys: mode 0
debug1: rekey in after 4294967296 blocks
debug1: Will attempt key: /home/serveradmin/.ssh/id_rsa_legacy RSA SHA256:UGYFYar2T1b6PgREJa+bk+X93bspta6XEsQlqbIM568 explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/serveradmin/.ssh/id_rsa_legacy RSA SHA256:UGYFYar2T1b6PgREJa+bk+X93bspta6XEsQlqbIM568 explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /home/serveradmin/.ssh/id_rsa_legacy RSA SHA256:UGYFYar2T1b6PgREJa+bk+X93bspta6XEsQlqbIM568 explicit
debug3: sign_and_send_pubkey: using publickey with RSA SHA256:UGYFYar2T1b6PgREJa+bk+X93bspta6XEsQlqbIM568
debug3: sign_and_send_pubkey: signing using ssh-rsa SHA256:UGYFYar2T1b6PgREJa+bk+X93bspta6XEsQlqbIM568
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
root@192.168.2.2: Permission denied (publickey).
 

ManoftheSea

Member
Apr 18, 2023
41
16
8
We're not so different, you and I.

Code:
vlan 10 name wired by port
 untagged ethe 1/1/43 
 router-interface ve 10
!
vlan 100 name Comcast by port
 untagged ethe 1/1/45 to 1/1/48 
!
vlan 101 name Wifi-Users by port
 tagged ethe 1/1/39 
 router-interface ve 101
!
vlan 200 name DMZ by port
 untagged ethe 1/1/31 ethe 1/2/4 
 router-interface ve 200
!
vlan 201 name Printers by port
 tagged ethe 1/1/39 
 router-interface ve 201
!
vlan 400 name Wifi-Guest by port
 tagged ethe 1/1/39 
 untagged ethe 1/1/37 
 router-interface ve 400
!
vlan 1000 name Internal by port
 tagged ethe 1/1/39 
 untagged ethe 1/1/38 ethe 1/1/40 
 router-interface ve 1000
!
vlan 1500 name DEFAULT-VLAN by port
Code:
ip dhcp-server pool guests
 dhcp-default-router 172.20.20.1 
 dns-server 8.8.8.8 1.1.1.1 
 excluded-address 172.20.20.1 172.20.20.99
 lease 0 1 0
 network 172.20.20.0 255.255.255.0
 static-mac-ip-mapping 172.20.20.100 0011.2233.4455
 deploy
!
!
ip dhcp-server pool internal
 dhcp-default-router 10.255.253.1 
 dns-server 8.8.8.8 1.1.1.1 
 excluded-address 10.255.253.1 10.255.253.99
 lease 0 1 0
 network 10.255.253.0 255.255.255.0
 static-mac-ip-mapping 10.255.253.100 0011.2233.4455
 static-mac-ip-mapping 10.255.253.251 0011.2233.4455
 static-mac-ip-mapping 10.255.253.253 0011.2233.4455
 deploy
!
!
ip dhcp-server pool printers
 dhcp-default-router 192.168.201.1 
 excluded-address 192.168.201.1 192.168.201.2
 excluded-address 192.168.201.4 192.168.201.255
 lease 1 0 0
 network 192.168.201.0 255.255.255.0
 static-mac-ip-mapping 192.168.201.3 0011.2233.4455
 deploy
!
!
ip dhcp-server pool users
 dhcp-default-router 192.168.101.1 
 dns-server 192.168.200.5 
 excluded-address 192.168.101.1 192.168.101.99
 lease 0 1 0
 network 192.168.101.0 255.255.255.0
 deploy
I don't see any reason your devices should be having an issue here. Any chance you're making use of ACLs? Here's my "guest network" ACL:
Code:
interface ve 400
 ip access-group 104 in 
 ip address 172.20.20.1 255.255.255.0
!
access-list 104 deny ip any 192.168.0.0 0.0.255.255 
access-list 104 permit ip any 172.20.20.0 0.0.0.255 
access-list 104 permit udp any eq bootpc any eq bootps
access-list 104 deny ip any 172.16.0.0 0.15.255.255 
access-list 104 deny ip any 10.0.0.0 0.255.255.255 
access-list 104 permit icmp any 192.168.0.0 0.0.255.255 echo-reply 
access-list 104 deny icmp any 192.168.0.0 0.0.255.255 
access-list 104 permit icmp any 172.20.20.0 0.0.0.255 
access-list 104 deny icmp any 172.16.0.0 0.15.255.255 
access-list 104 deny icmp any 10.0.0.0 0.255.255.255 
access-list 104 permit ip any any 
access-list 104 permit icmp any any
In particular, DHCP's DISCOVER is from 0.0.0.0:bootpc to 255.255.255.255:bootps, which is why the extra UDP rule has to be included.

To investigate your problem more, I'd suggest you mirror the traffic on that VE to somewhere that you can do a package capture or tcpdump. Your bindings say that the router thinks the address is assigned - why doesn't the NVR know?
 

ManoftheSea

Member
Apr 18, 2023
41
16
8
Code:
OpenSSH_8.9p1 Ubuntu-3ubuntu0.4, OpenSSL 3.0.2 15 Mar 2022
...
debug1: Authentications that can continue: publickey
...
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
...
debug1: Server accepts key: /home/serveradmin/.ssh/id_rsa_legacy RSA SHA256:UGYFYar2T1b6PgREJa+bk+X93bspta6XEsQlqbIM568 explicit
debug3: sign_and_send_pubkey: using publickey with RSA SHA256:UGYFYar2T1b6PgREJa+bk+X93bspta6XEsQlqbIM568
debug3: sign_and_send_pubkey: signing using ssh-rsa SHA256:UGYFYar2T1b6PgREJa+bk+X93bspta6XEsQlqbIM568
...
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
root@192.168.2.2: Permission denied (publickey).
This looks like it's not asking for your password at all. Does the ICX6610 allow publickey auth, anyway?
EDIT: I found documentation on it:
"copy tftp flash 10.168.1.234 pkeys.txt ssh-pub-key-file"
EDIT 2:
"enable" and "configure terminal", then "ip ssh pub-key-file tftp 192.168.2.138 public-key.txt"
as well as enabling pubkey in the "ip ssh" options. It looks like our 08.0.30 devices only allow RSA and DSA.


My ssh_config for challenge-response (username/password):
Code:
Host ICX6450*
  User root
  KexAlgorithms=+diffie-hellman-group1-sha1
  HostKeyAlgorithms=+ssh-rsa
  PreferredAuthentications keyboard-interactive
 
Last edited:

Spindle2274

New Member
Dec 1, 2023
4
0
1
"copy tftp flash 10.168.1.234 pkeys.txt ssh-pub-key-file"
Slight typo in that command, here is what I have: ip ssh pub-key-file tftp 192.168.2.138 public-key.txt

In my case, it says the password is wrong everytime I enter it, even after copying your ssh config.
 

ManoftheSea

Member
Apr 18, 2023
41
16
8
Yep, I was looking at a 09.0.0 document. Here's the 08.0.60 (which is too new, but seems to mostly work)

For others following along, the document says only RSA and DSA. So the key is generated with
"ssh-keygen -t rsa -b 2048 -f output-filename"
and the ssh-pubkey-file for the router is generated with
"ssh-keygen -e -f id_rsa_keyname"

Should result in something like
Code:
---- BEGIN SSH2 PUBLIC KEY ----
Comment: "2048-bit RSA, converted by ManoftheSea@myhost from OpenSSH"
AAAAB3NzaC1yc2EAAAADAQABAAABAQCvCeoQ7nLkwnpSt41S3dnGpB/p5cO6dV1m22lzW8
01NWi2Z4AZZ9LrwV3Cixyfcy4VwPyKP2UjNE/qXP5qLTexCEbTPKyC1wgX2lGieqApQ5Qq
+RvQMS+fGvviwrx0XQyYG5o/IqZwvKhNyht1rAMMHBn9phBvyvj+bkqNj125LU4E/lnf39
RuNOY/H59Q9GPiy72YvYB/Fcrujnd2RssiijE6UEjDTzRAMZCeEVi+qJUhlSrUWpVwykWu
L1Lj06O6/xu1blHNYkH5iHYvMEt5Rw5vP1UA0OXkszRZR3eO+0bzfaRpLVgLveZ/x5RtQA
ipBezCNtncltK2q92Ibkbr
---- END SSH2 PUBLIC KEY ----
Spindle, the verbose logs you posted don't show an attempt to use a password at all. To eliminate assumptions, you're typing the root password for the ICX device, not the password for your public key, right?

Furthermore, after uploading the key above, I did this, and have working pubkey auth:
Code:
Host ICX6450*
  User root
  IdentityFile=~/.ssh/id_rsa_ICX6450
  KexAlgorithms=+diffie-hellman-group1-sha1
  PubkeyAcceptedKeyTypes=+ssh-rsa
  HostKeyAlgorithms=+ssh-rsa
  #PreferredAuthentications keyboard-interactive
  PreferredAuthentications publickey
Additional, my debug log:
Code:
...
debug1: Offering public key: /home/ManoftheSea/.ssh/id_rsa_ICX6450 RSA SHA256:su29sKf2JaYJlwmymkInj837y4vWDlrRTS4aqQS9BQQ explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /home/derek/.ssh/id_rsa_ICX6450 RSA SHA256:su29sKf2JaYJlwmymkInj837y4vWDlrRTS4aqQS9BQQ explicit
debug3: sign_and_send_pubkey: using publickey with RSA SHA256:su29sKf2JaYJlwmymkInj837y4vWDlrRTS4aqQS9BQQ
debug3: sign_and_send_pubkey: signing using ssh-rsa SHA256:su29sKf2JaYJlwmymkInj837y4vWDlrRTS4aqQS9BQQ
debug3: send packet: type 50
debug3: receive packet: type 52
Authenticated to fe80::1%wlp166s0 ([fe80::1%wlp166s0]:22) using "publickey".
debug1: channel 0: new session [client-session] (inactive timeout: 0)
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Entering interactive session.
...
 
  • Like
Reactions: jode

Spindle2274

New Member
Dec 1, 2023
4
0
1
Thanks for the suggestions.

My key:

Code:
---- BEGIN SSH2 PUBLIC KEY ----
Comment: "2048-bit RSA, converted by ****"
AAAAB3NzaC1yc2EAAAADAQABAAABAQChRXoZFYD+ayUdc8jKwKiL6ECNdC0vCky1CSKW0l
GUUZJIt/J/r14rSBXcefBz5P9Bxu5zK4Isf2D0nlQg+lLjO60ytOhATu/aCw1vR0Nu4r4o
wPeNjbTc5af8O7HtkrT+H/efKO1LOeWFQR7xyaFRy+XGa8dg9QAJXnQrS92MKe0o6kodpi
3PWaU5pfHd2LQQaG165tQLwOr2hQ4vArP4YbIgpRXSM6R+ZLIRSnEEDsDk4hBmsZeSfEPg
Zu7yn+biaKTiYeTl9AyXoUO7bP4AQ6ZH+iV68v78tDFt9jpcldSmHbUiGwCbviUsoucufv
Bdp8w5Xce2ebFquZCXf2MN
---- END SSH2 PUBLIC KEY ----
~/.ssh/config:


Code:
StrictHostKeyChecking no
Host 192.168.2.2
  User root
  IdentityFile=~/.ssh/id_rsa_legacy
  KexAlgorithms=+diffie-hellman-group1-sha1
  PubkeyAcceptedKeyTypes=+ssh-rsa
  HostKeyAlgorithms=+ssh-rsa
  #PreferredAuthentications keyboard-interactive
  PreferredAuthentications publickey
root@192.168.2.2: Permission denied (publickey)

:(
 

ManoftheSea

Member
Apr 18, 2023
41
16
8
Well, I can only help you eliminate assumptions...
Does "show ip client-pub-key" agree with your key? does "show ip ssh config" agree that Public-key is an authentication method? Does "ssh -vvv switch-host" show that you're not offering multiple keys before the one you want?
 

Vesalius

Active Member
Nov 25, 2019
261
202
43
@fohdeesha is there a way to make thr dhcp server in a brocade icx6450 authoritative? I setup some pools and some of my devices cannot get an ip from the pool but others can. I don't have the luxury of setting up ISC dhcpd on another device.
Check out the post from @fohdeesha back in 2018 in regards to the FastIron 08.0.30 dhcp server implementation.

Gave the built in DHCP server a go to see if I would run into your issue, and I did indeed with a couple IOT type stuff. After a ton of debugging, it turned out to be the fact the FastIron DHCP server is not set as authoritative, and some DHCP stack implementations (like Roku's and Sonos) do not like this, and will ignore it.
 
  • Like
Reactions: bwahaha

Spindle2274

New Member
Dec 1, 2023
4
0
1
Well, I can only help you eliminate assumptions...
Does "show ip client-pub-key" agree with your key? does "show ip ssh config" agree that Public-key is an authentication method? Does "ssh -vvv switch-host" show that you're not offering multiple keys before the one you want?
Thank you, I appreciate any help I can get. Still no cigar, unfortunately.

show ip client-pub-key shows my public key that I am using (RSA 2048).


ssh -vvv switch-host only shows the intended key being offered.

I verfied file permissions. Re-uploaded the key. Tried from another machine. I even created a docker container based on Ubuntu 16.04 with OpenSSL 7 to see if it had something to do with my modern client, same result.


Code:
show ip ssh config
SSH server                 : Enabled
SSH port                   : tcp\22
Host Key                   :  RSA 2048
Encryption                 : aes256-cbc, aes192-cbc, aes128-cbc, aes256-ctr, aes192-ctr, aes128-ctr
Permit empty password      : No
Authentication methods     : Public-key
Authentication retries     : 3
Login timeout (seconds)    : 120
Idle timeout (minutes)     : 0
Strict management VRF      : Disabled
SCP                        : Enabled
SSH IPv4 clients           : All
SSH IPv6 clients           : All
SSH IPv4 access-group      :
SSH IPv6 access-group      :
SSH Client Keys            :
I have already wasted a day on this! Hopefully I can get some relief soon.

EDIT: I was finally able to connect by trying a 1024 bit key! Not ideal, obviously, but I am willing to accept the risk if I specify the allowed hosts manually.
 
Last edited:

kevindd992002

Member
Oct 4, 2021
122
6
18
We're not so different, you and I.

Code:
vlan 10 name wired by port
untagged ethe 1/1/43
router-interface ve 10
!
vlan 100 name Comcast by port
untagged ethe 1/1/45 to 1/1/48
!
vlan 101 name Wifi-Users by port
tagged ethe 1/1/39
router-interface ve 101
!
vlan 200 name DMZ by port
untagged ethe 1/1/31 ethe 1/2/4
router-interface ve 200
!
vlan 201 name Printers by port
tagged ethe 1/1/39
router-interface ve 201
!
vlan 400 name Wifi-Guest by port
tagged ethe 1/1/39
untagged ethe 1/1/37
router-interface ve 400
!
vlan 1000 name Internal by port
tagged ethe 1/1/39
untagged ethe 1/1/38 ethe 1/1/40
router-interface ve 1000
!
vlan 1500 name DEFAULT-VLAN by port
Code:
ip dhcp-server pool guests
dhcp-default-router 172.20.20.1
dns-server 8.8.8.8 1.1.1.1
excluded-address 172.20.20.1 172.20.20.99
lease 0 1 0
network 172.20.20.0 255.255.255.0
static-mac-ip-mapping 172.20.20.100 0011.2233.4455
deploy
!
!
ip dhcp-server pool internal
dhcp-default-router 10.255.253.1
dns-server 8.8.8.8 1.1.1.1
excluded-address 10.255.253.1 10.255.253.99
lease 0 1 0
network 10.255.253.0 255.255.255.0
static-mac-ip-mapping 10.255.253.100 0011.2233.4455
static-mac-ip-mapping 10.255.253.251 0011.2233.4455
static-mac-ip-mapping 10.255.253.253 0011.2233.4455
deploy
!
!
ip dhcp-server pool printers
dhcp-default-router 192.168.201.1
excluded-address 192.168.201.1 192.168.201.2
excluded-address 192.168.201.4 192.168.201.255
lease 1 0 0
network 192.168.201.0 255.255.255.0
static-mac-ip-mapping 192.168.201.3 0011.2233.4455
deploy
!
!
ip dhcp-server pool users
dhcp-default-router 192.168.101.1
dns-server 192.168.200.5
excluded-address 192.168.101.1 192.168.101.99
lease 0 1 0
network 192.168.101.0 255.255.255.0
deploy
I don't see any reason your devices should be having an issue here. Any chance you're making use of ACLs? Here's my "guest network" ACL:
Code:
interface ve 400
ip access-group 104 in
ip address 172.20.20.1 255.255.255.0
!
access-list 104 deny ip any 192.168.0.0 0.0.255.255
access-list 104 permit ip any 172.20.20.0 0.0.0.255
access-list 104 permit udp any eq bootpc any eq bootps
access-list 104 deny ip any 172.16.0.0 0.15.255.255
access-list 104 deny ip any 10.0.0.0 0.255.255.255
access-list 104 permit icmp any 192.168.0.0 0.0.255.255 echo-reply
access-list 104 deny icmp any 192.168.0.0 0.0.255.255
access-list 104 permit icmp any 172.20.20.0 0.0.0.255
access-list 104 deny icmp any 172.16.0.0 0.15.255.255
access-list 104 deny icmp any 10.0.0.0 0.255.255.255
access-list 104 permit ip any any
access-list 104 permit icmp any any
In particular, DHCP's DISCOVER is from 0.0.0.0:bootpc to 255.255.255.255:bootps, which is why the extra UDP rule has to be included.

To investigate your problem more, I'd suggest you mirror the traffic on that VE to somewhere that you can do a package capture or tcpdump. Your bindings say that the router thinks the address is assigned - why doesn't the NVR know?
No ACL's at all because I'm not yet at that part of the config. I'll have those later.

Yeah, that's what's weird. The switch knows it assigned the IP but the nvr doesn't. When I plug in a windows device on the same port, no issues. When I plug in the nvr to another netowrk in another house with a pfsense dhcp server, no issues. So it's got to be th combiantion of this switch and nvr, no?

I'll see how I can mirror the traffic of a ve to do a packet dump. But what ie the disadvantage of using a non-authoritative dhcp server?
 

kevindd992002

Member
Oct 4, 2021
122
6
18
Check out the post from @fohdeesha back in 2018 in regards to the FastIron 08.0.30 dhcp server implementation.



Well that's exactly the post that made me ask my initial dhcp question in the first place. But why does it not assign the IP? I thought having a non-authoritative dhcp is only bad when you multiple dhcp servers in a subnet or something, since it doesn't reply to NACK packets?
 

kevindd992002

Member
Oct 4, 2021
122
6
18
Apparently, the NVR not accepting the lease.
I guess the better question is what is the use case of a non-authoritative dhcp server?

I don't to have this same problem in the future for other devices. It's bizarre that an enterprise L3 switch has this type of problem.
 

notdeadyet

New Member
Jul 14, 2022
3
0
1
Multicast config is all default no changes. Once I upgraded the firmware it worked. What switch do you have? Does it have the latest firmware?
I have an ICX-6450P on the latest router firmware. Everything works except for multicast across VLANs. I can share my config if that could help with troubleshooting. I've tried everything, even running a VM with avahi on all VLANs and it still doesn't work.
 

rnavarro

Active Member
Feb 14, 2013
197
40
28
Alright, I've got a weird one and it's driving me absolutely bonkers. This is ALL in the context of IPv6

Problem


I'm having an issue with wireless IPv6 connectivity to the internet. When a wireless client roams from the initial association WAP (access point) it loses IPv6 connectivity to the internet.

If I re-associate (reset wifi on the device) after roaming IPv6 comes back for the new WAP, roaming again will break connectivity to the internet.

Environment

Router

I have an OpnSense router with a direct "transit" connection down to a pair of stacked ICX6610's. I turned off all dynamic routing protocols and I'm using only static routing on the ICX's and the opnSense.

Switches
The ICX 6610's are running the latest firmware:
Copyright (c) Ruckus Networks, Inc. All rights reserved.
UNIT 1: compiled on Aug 8 2023 at 23:06:54 labeled as SPR08095m
(33554432 bytes) from Primary SPR08095m.bin (UFI)
SW: Version 08.0.95mT213
Compressed Primary Boot Code size = 786944, Version:10.1.26T215 (spz10126)
Compiled on Tue Nov 29 04:43:15 2022
UNIT 2: compiled on Aug 8 2023 at 23:06:54 labeled as SPR08095m
(33554432 bytes) from Primary SPR08095m.bin (UFI)
SW: Version 08.0.95mT213
Compressed Primary Boot Code size = 786944, Version:10.1.26T215 (spz10126)

HW: Stackable ICX7250-24-HPOE


The ICX's are running as L3 gateways for their respective VLANs, each vlan is using SLAAC for IPv6 address allocation.

The ICX routing table has the upstream opnSense router as it's default route.

Access Points
I trunk 3 VLANs to three different Aruba InstantOn AP22 access points, the 3 access points are split accross multiple switch ports in the stack.

Debugging Attempts
Removed all dynamic routing protocols and moved to static routes

*after* roaming to a different access point
Ping from OpnSense to wireless device - fails
Ping from hardwired device to wireless device in the same VLAN (ie no routing) - succeeds
Ping from hardwired device to wireless device in a different VLAN (ie routed through 6610) - succeeds

Here is a tcpdump from the perspective of the OpnSense router:

OpnSense: 2001:db8:2d00:24f0::1
Wireless Client: 2001:db8:2d00:2405:207a:305e:39d9:b385

Associated with WAP1
19:57:46.434682 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 994, length 16
19:57:47.427977 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 995, length 16
19:57:47.434530 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 995, length 16
19:57:48.430550 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 996, length 16
19:57:48.438575 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 996, length 16
19:57:49.433206 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 997, length 16
19:57:49.499337 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 997, length 16

Roam to WAP2
19:57:50.437464 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 998, length 16
19:57:51.438580 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 999, length 16
19:57:52.440642 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1000, length 16
19:57:53.441971 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1001, length 16
19:57:54.443574 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1002, length 16
19:57:55.444589 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1003, length 16
19:57:56.448761 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1004, length 16
19:57:57.450438 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1005, length 16
19:57:58.452506 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1006, length 16
19:57:59.457450 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1007, length 16
19:58:00.458815 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1008, length 16
19:58:01.463074 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1009, length 16
19:58:02.464345 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1010, length 16
19:58:03.465588 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1011, length 16
19:58:04.467961 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1012, length 16
19:58:05.468944 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1013, length 16
19:58:06.470789 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1014, length 16
19:58:07.478637 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1015, length 16
19:58:08.479740 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1016, length 16
19:58:09.484910 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1017, length 16
19:58:10.492031 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1018, length 16
19:58:11.494773 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1019, length 16
19:58:12.502221 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1020, length 16
19:58:13.508098 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1021, length 16
19:58:14.514172 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1022, length 16
19:58:15.516477 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1023, length 16
19:58:16.519838 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1024, length 16
19:58:17.521306 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1025, length 16
19:58:18.523018 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1026, length 16
19:58:19.529879 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1027, length 16
19:58:20.532896 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1028, length 16
19:58:21.534109 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1029, length 16
19:58:22.535387 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1030, length 16
19:58:23.536953 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1031, length 16
19:58:24.537997 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1032, length 16
19:58:25.541886 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1033, length 16
19:58:26.546915 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1034, length 16
19:58:27.555055 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1035, length 16
19:58:28.563192 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1036, length 16
19:58:29.567090 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1037, length 16
19:58:30.568384 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1038, length 16
19:58:31.583843 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1039, length 16
19:58:32.595609 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1040, length 16
19:58:33.604105 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1041, length 16
19:58:34.610626 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1042, length 16
19:58:35.618694 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1043, length 16
19:58:36.622297 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1044, length 16
19:58:37.627183 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1045, length 16
19:58:38.629207 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1046, length 16
19:58:39.630363 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1047, length 16
19:58:40.633129 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1048, length 16
19:58:41.634566 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1049, length 16
19:58:42.636545 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1050, length 16
19:58:43.637942 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1051, length 16
19:58:44.638553 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1052, length 16
19:58:45.641411 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1053, length 16
19:58:46.642567 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1054, length 16
19:58:47.643843 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1055, length 16
19:58:48.645237 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1056, length 16
19:58:49.646044 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1057, length 16
19:58:50.649867 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1058, length 16
19:58:51.650661 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1059, length 16
19:58:52.651403 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1060, length 16
19:58:53.652821 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1061, length 16
19:58:54.653917 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1062, length 16
19:58:55.657144 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1063, length 16

Return to WAP1
19:58:55.665094 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 1063, length 16
19:58:56.657742 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1064, length 16
19:58:56.665141 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 1064, length 16
19:58:57.659884 IP6 2001:db8:2d00:24f0::1 > 2001:db8:2d00:2405:207a:305e:39d9:b385: ICMP6, echo request, seq 1065, length 16
19:58:57.664363 IP6 2001:db8:2d00:2405:207a:305e:39d9:b385 > 2001:db8:2d00:24f0::1: ICMP6, echo reply, seq 1065, length 16

During that entire test, pings from a device on the same VLAN and another device in a different VLAN experienced zero packet loss.

P.S. I'm seeing these issues ONLY with my wireless clients, my hard wired IPv6 client works great.
 

jode

Member
Jul 27, 2021
53
45
18
I wonder if you might be able to use ssh and an expect-script to send the commands to the switch
Took a little while, but this works.

Code:
#!/usr/bin/expect -f
## Sypnosis: run_script.exp <ICX_IP> <ICX_SCRIPT>
## Run script <ICX_SCRIPT> line by line on ICX <ICX_IP>. Connection via SSH certificate assumed.

set ICX_IP [lindex $argv 0]
set ICX_SCRIPT [lindex $argv 1]

set timeout 15
spawn ssh $ICX_IP
match_max 100000
expect -exact "SSH@ICX6610>"
send -- "enable\r"
expect "SSH@ICX6610#"
send -- "skip\r"
expect "SSH@ICX6610#"

set infile [open "$ICX_SCRIPT"]
while { [gets $infile line] >= 0 } {
  send "$line\r"
  expect -re "SSH@ICX6610(?:>|.*#)"
}

send_user "\r"
In the next step I'll try to create scripted tests.
 

rnavarro

Active Member
Feb 14, 2013
197
40
28
Relating to the above, I found another interesting detail.

I setup port mirroring on the ICX's to take a look at the traffic out of the WAP ports.

It looks like the ping from the OpnSense router is being routed through the switch, however it's going out of the wrong port.

The echo request seems to be stuck on the initial WAP port.

When doing a mirror of the roamed WAP port, I don't see the echo request.

What's more interesting is that the switch definitely updates it's internal neighbor cache:

show ipv6 neighbor 2001:db8:2d00:2405:207a:305e:39d9:b385
IPv6 Address LinkLayer-Addr State Age Port vlan IsR
2001:db8:2d00:2405:207a:305e:39d9:b385 ac3e.b17a.0038 STALE 6333 e 1/1/12 5 0

1/1/12 is the port of the roamed to WAP

It seems like there is a mismatch when routing traffic, as it doesn't seem to get the memo that this address is now on a different port.
 

rnavarro

Active Member
Feb 14, 2013
197
40
28
One more discovery for the night.

If I clear the ipv6 neighbor cache on the router, things start working even after a roam:

clear ipv6 neighbor 2001:db8:2d00:2405:207a:305e:39d9:b385

pings immediately start working again
 
  • Like
Reactions: fohdeesha

fohdeesha

Kaini Industries
Nov 20, 2016
2,835
3,275
113
33
fohdeesha.com
One more discovery for the night.

If I clear the ipv6 neighbor cache on the router, things start working even after a roam:

clear ipv6 neighbor 2001:db8:2d00:2405:207a:305e:39d9:b385

pings immediately start working again
ya v8030 no handle ipv6 neighbor enmtrties moving to different ports sdo good need to clear neihgbors
 

rnavarro

Active Member
Feb 14, 2013
197
40
28
ya v8030 no handle ipv6 neighbor enmtrties moving to different ports sdo good need to clear neihgbors
Booo these 7250's were perfect with POE and SFP+ port count.

I think I'm going to try try the 9.x branch for the hell of it to see what it does.

I guess worst case I'll just go back to terminating all my vlans on the OpnSense, but I hate hate hate taking down the whole network when it does updates.

We'll see.