If that’s a linux based TFTP server, check syslog. It may be attempting to read a no -existent path.
What was the full command you tried to execute on the switch?
Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)
- Thread starter fohdeesha
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
What was the full command you tried to execute on the switch?
Full command was
copy tftp flash 192.168.78.10 grz10100.bin boot. I don't use tftp much, I'll give the logs a check. I appreciate the input, not sure if I'm missing something setting up the tftp server.From my reading of the command/security reference, I think you could do this selectively too. Have an ACL for all DNS traffic (by port), and set the next-hop to your firewall instance. Then your firewall can do whatever NATing you need. This is probably something I'm going to do as well as I like to run my own DNS server so that I can have a hostname block list.
For those interested, not all tftp-hpa config files are formatted as described in the Brocade overview of the Fodeesha docs. Depending on implementation the following may be necessary:
Code:
# /etc/conf.d/tftpd
TFTPD_ARGS="--user=nobody --secure /srv/tftp/"I did not care for the noise that the Sunon maglev fans made in my 6450. I went with a couple of Delta FFB0412HN-5T20 fans that were quieter than stock but still moved plenty of air. If those aren't readily available, you can check out some of the fans linked in this post.
I'm still troubleshooting my poor speed between 5Gbe and 10Gbe NICs on my unRAID servers. iperf3 is showing a LOT of retransmits, regardless of which system is chosen as the server or client. I've made so many settings adjustments on my ICX6610 that I really don't know the current state of it. Which method is the one that will NOT erase my licenses:
factory set-default
reset
or
erase startup-config
reload
TIA!
factory set-default
reset
or
erase startup-config
reload
TIA!
Per this thread (https://forums.servethehome.com/ind...-t-marvell-88x3300-v-s-aquantia-aqs-107.30004), the Marvell-based NBase-T SFP+ adapters like the Wiitek aren't all that great in systems that don't natively support 2.5Gbps/5Gbps. Only the (expensive) Aquantia seems to handle those speeds well.
Interesting... some mildly conflicting info here on STH. Easy to happen with forums/communities like this. I based my purchase of the Wiitek on a review of it here on STH: https://www.servethehome.com/wiitek-sfp-10g-t-review-10gbase-t-adapter/
Of course their testing wasn't on an ICX6610, so it's possible that the Wiitek just doesn't work well with this switch. I've disabled the onboard 10Gbe Marvell/Aquantia NIC on the z690 system and added a Intel x520 PCIe x8 NIC connected via DAC cable directly to the switch. Alas I'm still seeing far too many re-transmits whether it acts as the iperf server or client.
I may try the same on the x299 system but I only have an older HPE/Qlogix dual SFP NIC in my spare parts. I had trouble with them overheating and dropping off the network so I won't trust it for long term use.
I'd still like to factory reset the ICX6610 just to make sure it's not one or more settings that I tried that's causing some of my issues. I just don't know which method to use to ensure I don't erase the licenses. I know I can just re-install the licenses from Fohdeesha but I was also planning to try and revert back to the licenses it came with.
It came with these but they're now invalid as I used the ones provided by Fohdeesha:
1 Node Lock ICX6610-PREM-LIC-SW
2 Node Lock ICX6610-10G-LIC-POD
From what I've read, the switch will have the same features whether I use the ones it came with or the ones provided by Fohdeesha, even though they are 3 licenses instead of the 2 it came with. I'd just prefer not to lose the licenses it came with and I don't want to spend the 24 hours downloading the full 64MB flash.
Oh, and it's even stranger as Marvell bought Aquantia, so the onboard NICs actually get drivers from the Marvell site. Alas running unRAID I'm not too sure I want to try installing the new Linux drivers that were released yesterday.
I was able to revert to the original licenses that came with the 6610 and do the factory reset. Alas my speeds between the 5Gbe and 10Gbe systems are now even slower, and retransmits are still quite large.
It's entirely possible I did something wrong. I followed your guide this 2nd time other than reflashing the firmware and installing the licenses you kindly provide. I didn't see any need to reflash with the same versions of firmware as my initial flash worked - the R, S and POE firmwares are all updated to the version provided in the zip file.The licenses mine came with are:
ICX6610-PREM-LIC
ICX6610-10G-LIC-POD (enabling all 8 front SFP+ ports)
I also had issues with the 2 SFP+ ports in that they wouldn't auto-negotiate. I had to go in and configure them as 10G full duplex before they came online - I didn't have to do this the first time I setup the switch, or I don't remember doing it. Perhaps that means the ICX-MACSEC-LIC isn't included in the ICX6610-PREM-LIC license? Even so, my understanding of MACSec is that it's only to enable point to point security at layer 2. Even if the PREM license doesn't include MACSec, I can't see why my speeds got even slower or why I had to manually set them to 10G full duplex.
In any case I'm going to try it again but I'm going to change things around a bit. On my new z690 system I'll pull the Intel x520 out and re-enable the onboard Marvell/Aquantia 10Gbe NIC. I'll use a CAT7 cable to connect the RJ45 from that system to one of my Wiitek modules and insert into one of the 10G enabled front SFP+ ports. I'll then install the Intel x520 NIC in my other system and disable the onboard 5Gbe NIC. The Intel x520 will be connected to the switch using a DAC cable. Then both systems will have 10G ports and hopefully the Wiitek module will perform better.
I did look into the Aquantia AQS-107 based modules and found some on eBay for $50 US each, but not sure if they are the ones that have the Aquantia chips or the Marvell chips with the smaller buffer. I see them at retailers for much more - $180 - $210 US a piece. It's also possible that the eBay ones are knock-offs as a few of the ads add the word 'compatible' after the AOM-AQS-107-B0C2-CX part number.
Anyhow, just taking a dinner break and then I'll go retry it again. Wish me luck!
Last edited:
A couple more questions that have probably been asked and answered numerous times, but I'm finding it quite difficult to search this long thread for some definitive answers. Is it still recommended to put the 10G capable ports into their own VLAN? I'm a rookie at VLAN configuration so if I did that, I suspect I'll need to setup some bridging or routing rules so that the 10G devices can see the 1G and 100M devices and vice versa. What do I need to configure so that they can use jumbo frames and/or a larger MTU? It looks like enabling jumbo frames affects all ports on the switch so I'm thinking that's going to create issues for slower connections.
I can set the MTU on my unRAID systems, but when I try to set the MTU of a specific port on the switch, it errors out. Is setting MTU a per-port option? Is it and/or enabling jumbo frames even recommended anymore? I've read lots of conflicting suggestions, some stating that you have to use a large MTU to gain the best speeds, but others stating not to enable jumbo frames and just leave the MTU at the default of 1500. I know that certain devices won't like large MTU packets that will likely lead to fragmentation. So what's the best way to configure things?
Again, TIA!
I can set the MTU on my unRAID systems, but when I try to set the MTU of a specific port on the switch, it errors out. Is setting MTU a per-port option? Is it and/or enabling jumbo frames even recommended anymore? I've read lots of conflicting suggestions, some stating that you have to use a large MTU to gain the best speeds, but others stating not to enable jumbo frames and just leave the MTU at the default of 1500. I know that certain devices won't like large MTU packets that will likely lead to fragmentation. So what's the best way to configure things?
Again, TIA!
- Do you have a router that is routing already?
- If no external router, then using the switch as a router, does have some limitations in what routing it does.
- Setting jumbo still allows you on a per-port or VLAN basis the MTU
- I would segment storage from other VLANs i.e. internal network on one VLAN storage another
- I would then run a per VLAN MTU, Set the VLAN for normal communications under a 1500 MTU, storage under the max MTU and configure your ports on the storage to the max MTU your nics will support.
- You would need to understand Brocade's network and routing to get all of that to work without some other routing device to do it for you.
- Deciding whether or not to do jumbo is really dependent on what type of storage you are doing. Small files, large files etc. Smaller files may be best with standard mtu, I mean like 1k files versus larger megabyte files.
1. Yes, I have a physical pfSense system operating as my router, connected to my ISP via DOCSIS 3.1 modem in bridge mode.
2. N/A
3. When I tried to assign a new MTU value to a specific port, it always gave an error that MTU is an unrecognized command. I was able to assign a MTU to a VLAN, but right now I only have the one VLAN containing all ports. The larger MTU dropped my overall network speed so I changed it back to the default of 1500 for now.
4. I've seen that recommendation before but I'm not quite sure what it means... my unRAID servers act as more than just storage. Docker containers and VMs run on my unRAID systems. Right now I've left the z690 system with its 10Gbe Aquantia NIC attached to one of the front 10G ports but the x299 system with its 5Gbe NIC is currently plugged into a 1Gbps port. That has eliminated all of the retransmits that iperf3 reported when it was connected to a 10G port via the Wiitek SFP+ module, but of course that limits my max speed to 1Gbps.
5. My research has led me to the same conclusion so I'm still planning to create a new VLAN for all of the 10G ports. My main concern is how to let other devices access that VLAN and vice versa. I assume that's why you asked about a router - I suspected I would have to implement some routing on it to allow the VLAN to be seen by other devices. For example, my Nvidia Shield needs to be able to contact Plex on the main media unRAID system.
6. Any sites or tips you can provide to illustrate what kind of routing rules I need to configure, presumably on pfSense?
7. Most of the files on my unRAID systems are large - I only rip full remuxes from my physical discs. I do also use the second server for backups of other systems in my LAN. The backups contain lots of smaller files like documents and pictures - I suspect a large MTU will result in a lot of wasted overhead in packet transmission. The reason I want more LAN speed between the two servers is this: I rip and edit my full remuxes on the 2nd system. When I'm satisfied with the file, it then gets moved into one of the media folders (Movies, TV, Music, etc). Its this transfer between the two systems that I'm trying to improve. I've read good and bad about enabling jumbo frames but right now my feeling is to not enable it.
Note that my plan to put an Intel x520 10Gbe NIC into the x299 system wasn't implemented. Unfortunately I'm using all of the x8 slots that my motherboard supports. The x299 system does have Thunderbolt 3 ports on it so I was contemplating buying a TB3 10Gbe NIC. Alas that's an expensive option, even more than trying to find a known good SFP+ to RJ45 module with the Aquantia AQS-107 chip. There appear to be a LOT of 3rd party knock-offs of the Aquantia module - most of the ads I've seen state that they are 'compatible' but so was my Wiitek.
Thanks for the response and suggestions!
You might be misunderstanding something: the MTU is the *maximum* packet size, it's not the required size of all packets. Increasing the MTU can't cause transfers of small files to be slowed down. The network stack will only send large packets when it has enough data to fill them.
Very likely... I'm retired now for health reasons but my career had me often working with our IT teams. I had never really wanted 10Gbe (or better) until recently but I had heard about jumbo frames and MTU. Whenever we needed faster transfers it was up to our network gurus to do the changes. Now I understand why they griped so much when I asked them to increase our throughput.
how do deduce if you should increase MTU over the default:
year = > 2005? then > no
hyperscalars and HPC need not apply
year = > 2005? then > no
hyperscalars and HPC need not apply
there is no autonegotiation for SFP+ / 10gbase-sr. the ports by default are locked and configured to 1gbps. to run them at 10gbe, you must load a valid license, then go in and set them to 10gbps instead, which is covered in the guide: ICX6610 Licensing - Fohdeesha DocsI was able to revert to the original licenses that came with the 6610 and do the factory reset. Alas my speeds between the 5Gbe and 10Gbe systems are now even slower, and retransmits are still quite large.
The licenses mine came with are:
ICX6610-PREM-LIC
ICX6610-10G-LIC-POD (enabling all 8 front SFP+ ports)
I also had issues with the 2 SFP+ ports in that they wouldn't auto-negotiate. I had to go in and configure them as 10G full duplex before they came online - I didn't have to do this the first time I setup the switch, or I don't remember doing it. Perhaps that means the ICX-MACSEC-LIC isn't included in the ICX6610-PREM-LIC license? Even so, my understanding of MACSec is that it's only to enable point to point security at layer 2. Even if the PREM license doesn't include MACSec, I can't see why my speeds got even slower or why I had to manually set them to 10G full duplex.
In any case I'm going to try it again but I'm going to change things around a bit. On my new z690 system I'll pull the Intel x520 out and re-enable the onboard Marvell/Aquantia 10Gbe NIC. I'll use a CAT7 cable to connect the RJ45 from that system to one of my Wiitek modules and insert into one of the 10G enabled front SFP+ ports. I'll then install the Intel x520 NIC in my other system and disable the onboard 5Gbe NIC. The Intel x520 will be connected to the switch using a DAC cable. Then both systems will have 10G ports and hopefully the Wiitek module will perform better.
I did look into the Aquantia AQS-107 based modules and found some on eBay for $50 US each, but not sure if they are the ones that have the Aquantia chips or the Marvell chips with the smaller buffer. I see them at retailers for much more - $180 - $210 US a piece. It's also possible that the eBay ones are knock-offs as a few of the ads add the word 'compatible' after the AOM-AQS-107-B0C2-CX part number.
Anyhow, just taking a dinner break and then I'll go retry it again. Wish me luck!
My 6610 came with the license to enable them and I guess the seller likely set them to that before shipping to me. Regardless, my revert attempt was a wild grasp at air, i.e. a waste of my time. I have since reloaded the licenses you provide and followed the 3 guides. Alas I still get far too many retransmits with the Wiitek handling the rate drop from 10Gbps to 5Gbps.
My unRAID server reports the connection as 5Gbps so the Wiitek is at least partially working. I'm looking for one of the Supermicro units with the AQS-107 chip and will likely buy it when I come across a supplier who has it in stock and will ship to Canada. It will be expensive... too bad the unRAID kernel doesn't support Thunderbolt 3 10Gbe NICs as they aren't as high priced.
Thanks again for all the information you've gathered and shared via your excellent site.
When I spoke about using your storage under a different VLAN, I should have stated that is if you had other devices that would be dedicated to talking to that storage. For your normal access to the storage server, you would use a different VLAN to communicate to it.
I tend to forget that most people do not run as a complicated setup as I do, where my servers that do a bunch of tasks like virtualization on top of the storage use the higher MTU, whereas the clients on the virtualization do not see the underlying storage because it has been virtualized. I then expose the storage virtually for desktops and servers to make use of it.
So for most using a higher MTU would not make sense to do if you are exposing your unRaid storage server directly to the client machines. If you were creating a cephs storage cluster or something like that you would want to use higher MTUs. A virtualization storage system that communicated between machines would only be via one VLAN for storage and another VLAN for external access like the Internet or management.
If you are on a flat network and will be accessing that server directly, you would never set that system up for a large MTU. Keep it at 1500.
One example for the use of higher MTU would be, something like, you have another system set up to do rsync of your unRaid box for backup to another, you could set a VLAN up for just that alone to increase throughput.
Never have used unRaid before, but if it supported a redundant system to backup to, then that would be a case as well. Your VLAN for backups or syncing would make use of it.
It is common in the industry to use a separate VLAN for backups, syncing, and other high throughput services that would never be connected via a client-side network.
Based on what you have said so far, you can set the MTU on the switch as high as you want, but set all of your interfaces at the correct MTU for use if the unRaid box needs Internet and client access.