Drag to reposition cover

Brocade ICX Series (cheap & powerful 10gbE/40gbE switching)

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

raab

New Member
Aug 24, 2019
12
3
3
Just as well I have another one coming as I wasn't sure if this one was going to turn up
 

Wolfstar

Active Member
Nov 28, 2015
159
83
28
48
Why on earth should a link light come on if a serial converter is connected to a eth port? Hmmm... that doesn't sound right to me at all (but I don't want to try with my adapter, sorry :p)...
Because a link light will come on with a loopback, which a serial adapter will certainly be able to provide. Problem is serial isn't really pulling/using power to the same level Ethernet is, so yeah, that could've fused the thing.

It works both ways too; I've personally seen a console port on a 6610-48 that very likely had an ethernet run plugged into it because the port was dead. Switch worked fine and the guy I sold it to was able to get a serial connection rigged up (I didn't have the knowledge for it) and confirm it was working and able to use it.
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
Because a link light will come on with a loopback, which a serial adapter will certainly be able to provide. Problem is serial isn't really pulling/using power to the same level Ethernet is, so yeah, that could've fused the thing.
Err... nope... or at least, I don't think so... because the ethernet link will come up if the board detects a signal on the cable, but looking at the pinout for the Serial-RJ45 the TX and RX are on pin 3 and 6, or where the green/green-stripe pair would be in T568B... and they are isolated from each other (the TX switch side is terminated on the base of a mosfet, either in the serial bridge chip or in a buffer, in both case it is seen as a capacitance to ground, to me it seems strange that a ethernet PHY could detect it as a remote peer...).
Pins 4 and 5 are shorted to ground (blue pair), pins 1 and 2 are RTS and DTR, both an output from the switch, so they couldn't in any way generate the ethernet carrier signal... strange as hell...

It works both ways too; I've personally seen a console port on a 6610-48 that very likely had an ethernet run plugged into it because the port was dead. Switch worked fine and the guy I sold it to was able to get a serial connection rigged up (I didn't have the knowledge for it) and confirm it was working and able to use it.
Very well possible; as stated, both signal pairs of a 100Base-T will hit both output pins of the serial port of the switch, or one input pin ad one output pin, but on the ethernet signalling the voltage is applied only between the wires of every single pair; there is no voltage between each pair and "ground" (in fact, unless they were using shielded wires, there's no common groud between two devices connected by an ethernet cable!).
 

Wolfstar

Active Member
Nov 28, 2015
159
83
28
48
All of which bets are off if you have a fried serial controller creating the loop. I've actually seen the link light thing stuck on with a management port that was accidentally plugged into on a 3750X, so I assume it's something getting borked along the way.
 
  • Like
Reactions: fohdeesha

raab

New Member
Aug 24, 2019
12
3
3
The serial cable still gets detected in windows if that's any consolation

I'll have to wait for the new cable to arrive because I get no output on either switches management port
 

raab

New Member
Aug 24, 2019
12
3
3
And one of them has a faulty PoE board, great :)

Code:
PoE: Stack unit 1 PS 1, Internal Power supply  with 370000 mwatts capacity is up
PoE Info: Adding new 54V capacity of 370000 mW, total capacity is 370000, total free capacity is 370000
PoE Info: PoE module 1 of Unit 1 on ports 1/1/1 to 1/1/24 detected. Initializing....
PoE Event Trace Log Buffer for 2000 log entries allocated
PoE Event Trace Logging enabled...
PoE Error: Device 0 failed to start on PoE module.
PoE Error: Device 1 failed to start on PoE module.
Resetting module in slot 1 again to recover from dev fault
PoE Info: Hard Resetting in slot 1....
PoE Info: Resetting module in slot 1....completed.
PoE Error: Device 0 failed to start on PoE module.
PoE Error: Device 1 failed to start on PoE module.
 

Churchill

Admiral
Jan 6, 2016
838
213
43
Is there anyway I can find out why my speeds are not 40GB or so? For some unholy reason my rsync copies are hovering around 45MB/s which is abysmal when I should be getting a helluva lot more. Disks are Western Digital Reds, nothing else is going on between the arrays the disks are reading/writing from. CPU is low, memory is low, networking is barely a blip in speed. I cannot push the 40GB speed that I was hoping for.
 

Wolfstar

Active Member
Nov 28, 2015
159
83
28
48
Is there anyway I can find out why my speeds are not 40GB or so? For some unholy reason my rsync copies are hovering around 45MB/s which is abysmal when I should be getting a helluva lot more. Disks are Western Digital Reds, nothing else is going on between the arrays the disks are reading/writing from. CPU is low, memory is low, networking is barely a blip in speed. I cannot push the 40GB speed that I was hoping for.
So, what's the array? What speed are you getting writing a large file to the destination? Speed reading from a large file from the source? Have you tried running iperf between the two endpoints and seen what sort of speeds you're getting? Do you have any other connections that might be getting used (like an old gigabit link that you're keeping for redundancy)?

If you're not seeing errors on the output of "show interface ethe 1/2/X" and the log isn't showing anything either, it's unlikely to be a basic switching issue, but we'd need to know a lot more about source and destination before we could hazard a guess on anything else.
 

Churchill

Admiral
Jan 6, 2016
838
213
43
So, what's the array? What speed are you getting writing a large file to the destination? Speed reading from a large file from the source? Have you tried running iperf between the two endpoints and seen what sort of speeds you're getting? Do you have any other connections that might be getting used (like an old gigabit link that you're keeping for redundancy)?
Array is a pair of Supermicro 2U servers with dual 6 core/12 thread Xeon 2600, RAM is 128GB in each. Both are controlled by SAS controllers and the disks are SATA.

I ran iperf between the two and only when I was pushing 10+ threads was I getting anywhere near 30GB/s speed between the two.

If you're not seeing errors on the output of "show interface ethe 1/2/X" and the log isn't showing anything either, it's unlikely to be a basic switching issue, but we'd need to know a lot more about source and destination before we could hazard a guess on anything else.
Both running 40GB hp cards reflashed to mellanox stock in ethernet mode. Jumbo frames is turned on for both.

I'm shocked I'm not getting at least 100MB/s from these transfers as the disks can easily sustain 150-200MB/s in transfer speeds.
 

Wolfstar

Active Member
Nov 28, 2015
159
83
28
48
Array is a pair of Supermicro 2U servers with dual 6 core/12 thread Xeon 2600, RAM is 128GB in each. Both are controlled by SAS controllers and the disks are SATA.

I ran iperf between the two and only when I was pushing 10+ threads was I getting anywhere near 30GB/s speed between the two.
Alright, so even if there was something wrong with the disks, there's still a problem with the network throughput.

Both running 40GB hp cards reflashed to mellanox stock in ethernet mode. Jumbo frames is turned on for both.

I'm shocked I'm not getting at least 100MB/s from these transfers as the disks can easily sustain 150-200MB/s in transfer speeds.
Are both devices in the same VLAN? If not, where's the routing being done, on the switch or elsewhere? If you've got spare cards/10G ports, can you pull the 40G links and test over the 10G and see if you get better throughput, the same, or worse? (Either iperf or file transfer, preferably both.)

Are we sure the NICs are running a full PCIe x8 electrical slot, and is it PCIe 2.0 or 3.0? (I think it has to be 3.0, but I have a vague recollection of motherboard PCIe generation being controlled by the CPU generation as well - just don't know if that was these CPUs or the E3s.) Why I ask is that PCIe 2.0 x8 tops out at 4GB/sec, which is 32 gigabit - suspiciously close your max transfer speed with multiple iperf threads.
 

fohdeesha

Kaini Industries
Nov 20, 2016
2,728
3,078
113
33
fohdeesha.com
30gbps iperf is good. you're not going to do much better than that without newer CPUs and some kernel tweaking. 45MB/s file shares sounds like NFS with sync=enabled or something
 

Churchill

Admiral
Jan 6, 2016
838
213
43
30gbps iperf is good. you're not going to do much better than that without newer CPUs and some kernel tweaking. 45MB/s file shares sounds like NFS with sync=enabled or something

I am doing an rsync between two systems to replicate the data.

These are both 2620 Xeon V2 so relatively new. I'd love to know what kind of kernel tweaking i can do.

The 40GB ports are all in their own VLAN seperated out from any network and traffic. fully isolated. I have 10GB ports I can hookup as well.
 

infoMatt

Active Member
Apr 16, 2019
222
100
43
I am doing an rsync between two systems to replicate the data.

These are both 2620 Xeon V2 so relatively new. I'd love to know what kind of kernel tweaking i can do.

The 40GB ports are all in their own VLAN seperated out from any network and traffic. fully isolated. I have 10GB ports I can hookup as well.
You could test if the bottleneck is in the switch making a direct connection between the two systems... if both NICs are at 40Gbe...
 

Whitecitadel

New Member
Feb 13, 2018
3
0
1
104
Looks like the guide needs an update as 6650 is no longer anywhere close to $300!

Is there anything between 6610 and 6650 with more QSFP ports than 4?

Churchill - try more iPerf threads to saturate the link?
 
Last edited: