[Mikrotik CRS226] LACP is killing the CPU

[voodooFX]

I have bought the CRS226 and all was good until I had to configure the link aggregation with my servers.
I have two esxi hosts and one (physical) freenas host.

On vmware I'm using the simple "Originating Virtual Port ID" algorithm and all is working fine, since there is no switch specific configuration needed.
On the freeNAS I used LACP with my Cisco SG300, so I configured the same on the CRS226 like this

1. removed the (two) ports from the bridge ("bridge1")
2. created new bonding (802.3ad) with the two ports
3. created new trunk with the two ports
4. created a new interface (bridge -> ports). Using 'bridge1' as bridge

After that, things are working but very badly: 45MB/s on a single stream (or 22MB/s on two streams) and the CRS226 CPU used at 100%.

Googling a bit looks like this is kind of a known issue for Mikrotik (since years ) so my question is: what can I use for freeNAS? I just would like to have the nics balanced across multiple streams, not looking for magic (200MB/s on a single stream etc. )

Suggestions?


P.S. with no aggregation the switch is doing 110MB/s (single scp stream), so I have no CPU/cable/other problems

 

[j_h_o]

You can't LACP on that switch in hardware. The LACP bridge you set up requires CPU involvement, which limits the speed.

So... don't do that. Use the SFP+ (10gbps) port to your server(s) instead?

 

[ttabbal]

Google says that it has 2 10Gbe ports... Could you run the server to one of those?

 

[voodooFX]

I j_h_o,

I'm already using 10G connectivity from the freeNAS server but directly attached to the (two) esxi hosts. (iSCSI)
The ports I would like to bond are used for management and NFS shares.

Btw if I can not use LACP it's OK, but what I can use then? I mean there must be an usable bonding algorithm on this thing

 

[voodooFX]

I found this: Manual:CRS examples - MikroTik Wiki

It should allow for static LACP bonding, but I'm getting the following error

[admin@MikroTik] /interface ethernet> set ether16 master-port=ether1-master
[admin@MikroTik] /interface ethernet> set ether24 master-port=ether1-master
[admin@MikroTik] > /interface ethernet switch trunk
[admin@MikroTik] /interface ethernet switch trunk> add name=trunk1 member-ports=ether16,ether24
[admin@MikroTik] > /interface bonding
[admin@MikroTik] /interface bonding> add name=bonding1 slaves=ether16,ether24 mode=balance-xor transmit-hash-policy=layer-2-and-3 link-monitoring=mii mii-interval=100ms
failure: ether16 already in ether1-master

I guess that the "ether2" in the guide is the "master-port" (ether1-master on my switch) otherwise it's unclear why I should use it as master.
Btw bonding is not possible with already assigned ports, as reported in the error.

 

[Pradawny]

I j_h_o,

I'm already using 10G connectivity from the freeNAS server but directly attached to the (two) esxi hosts. (iSCSI)
The ports I would like to bond are used for management and NFS shares.

Btw if I can not use LACP it's OK, but what I can use then? I mean there must be an usable bonding algorithm on this thing

CRS226 supports static link aggregation -> please check example @wiki

 

[voodooFX]

heheh already found, please have a look at the master port related error I'm getting

 

[Kybber]

As the example states, the bonding command is for RouterOS on the other end

 

[voodooFX]

As the example states, the bonding command is for RouterOS on the other end

Hi Kybber,

I understand that in the example ether2 is connected to the router
My router is also connected to ether2, which is slave to ether1 (ether1-master)

That said is still do not understand the mikrotik master-slave logic (I also had a look here but still..)
Do I need another unused port to be set as master for this bond? Looks like a waste

 

[Kybber]

As I understand the example, the last command should /not/ be issued on your CRS226, but on the other end. In other words: Your trunk was already completed after you ran "add name=trunk1 member-ports=ether16,ether24"

Never done this myself, though, so can't really help you out any further.

 

[voodooFX]

You are right!
So based on the guide, on the switch you just need to assign a master port to the ports to be aggregated and create a trunk with them.
I already did this but it's not working, of course...

I wonder if this "static LACP" is compatible with the freeNAS LACP setting...

 

[Kybber]

Do the other ports on the CRS that should communicate with the trunk also have ether1-master as their master port? On this device you can divide the available ports into a bunch of separate "virtual" switches by assigning a number of master ports and attaching other ports as slaves. All ports on the same switch need to share master port.

 

[voodooFX]

Do the other ports on the CRS that should communicate with the trunk also have ether1-master as their master port? On this device you can divide the available ports into a bunch of separate "virtual" switches by assigning a number of master ports and attaching other ports as slaves. All ports on the same switch need to share master port.

this is the port overview.
all the ports have the same master (and the only one defined)

ether16 and ether24 are the ports I would like to bond

Flags: X - disabled, R - running, S - slave
 #    NAME                                 MTU MAC-ADDRESS       ARP             MASTER-PORT                              SWITCH
 0 RS ether1-master                       1500 6C:3B:6B:7C:3B:0C enabled         none                                     switch1
 1 RS ether2                              1500 6C:3B:6B:7C:3B:0D enabled         ether1-master                            switch1
 2  S ether3                              1500 6C:3B:6B:7C:3B:0E enabled         ether1-master                            switch1
 3 RS ether4                              1500 6C:3B:6B:7C:3B:0F enabled         ether1-master                            switch1
 4  S ether5                              1500 6C:3B:6B:7C:3B:10 enabled         ether1-master                            switch1
 5  S ether6                              1500 6C:3B:6B:7C:3B:11 enabled         ether1-master                            switch1
 6  S ether7                              1500 6C:3B:6B:7C:3B:12 enabled         ether1-master                            switch1
 7 RS ether8                              1500 6C:3B:6B:7C:3B:13 enabled         ether1-master                            switch1
 8  S ether9                              1500 6C:3B:6B:7C:3B:14 enabled         ether1-master                            switch1
 9  S ether10                             1500 6C:3B:6B:7C:3B:15 enabled         ether1-master                            switch1
10  S ether11                             1500 6C:3B:6B:7C:3B:16 enabled         ether1-master                            switch1
11 RS ether12                             1500 6C:3B:6B:7C:3B:17 enabled         ether1-master                            switch1
12  S ether13                             1500 6C:3B:6B:7C:3B:18 enabled         ether1-master                            switch1
13 RS ether14                             1500 6C:3B:6B:7C:3B:19 enabled         ether1-master                            switch1
14  S ether15                             1500 6C:3B:6B:7C:3B:1A enabled         ether1-master                            switch1
15 RS ether16                             1500 6C:3B:6B:7C:3B:1B enabled         ether1-master                            switch1
16  S ether17                             1500 6C:3B:6B:7C:3B:1C enabled         ether1-master                            switch1
17  S ether18                             1500 6C:3B:6B:7C:3B:1D enabled         ether1-master                            switch1
18  S ether19                             1500 6C:3B:6B:7C:3B:1E enabled         ether1-master                            switch1
19 RS ether20                             1500 6C:3B:6B:7C:3B:1F enabled         ether1-master                            switch1
20  S ether21                             1500 6C:3B:6B:7C:3B:20 enabled         ether1-master                            switch1
21 RS ether22                             1500 6C:3B:6B:7C:3B:21 enabled         ether1-master                            switch1
22  S ether23                             1500 6C:3B:6B:7C:3B:22 enabled         ether1-master                            switch1
23 RS ether24                             1500 6C:3B:6B:7C:3B:23 enabled         ether1-master                            switch1
24  S sfp-sfpplus1                        1500 6C:3B:6B:7C:3B:24 enabled         ether1-master                            switch1
25  S sfpplus2                            1500 6C:3B:6B:7C:3B:25 enabled         ether1-master                            switch1

 

[voodooFX]

I managed to get a working situation with a port trunk group on the CRS226 and a link aggregation on freeNAS (type: Load Balance)

performances

scp from host1 = if0 = 50MB/s - if1 = 0MB/s (freeNAS) <---------------------- why only 50?
scp from host1+host2 = if0 = 50MB/s - if1 = 50MB/s (freeNAS) <------------ at least it's balancing
scp from host1+host2+host3 = if0 = 50MB/s - if1 = 80MB/s (freeNAS) <---- gain on the third stream is very limited

CSR CPU = 8-12%

so performance are somehow limited and I do not understand why, since the CPU is not overloaded.
I'm going almost out of ideas but I would feel very defeated if I will have to use freeNAS in Active/Backup configuration and I think also mikrotik should...

 

[Pradawny]

You need to run parallel connections to get the benefits from the port trunking each port needs at least one connection to aggregate the bandwidth

 

[voodooFX]

You need to run parallel connections to get the benefits from the port trunking each port needs at least one connection to aggregate the bandwidth

yes this is what's happening, but I see no reason why a single connection shouldn't reach 100MB/s