Switching hardware question

[BradleyD]

I have some knowledge of switching and routing having used dd-wrt and now openwrt for years and also having set up vlans. My question is about switching speed to another device on the same vlan.


Assume you have 2 computers that are able to send data at the max throughput of 10gbe (newer faster cpu, etc.) with higher end nics like intel, chelsio, etc. that have added capabilities.

Transfers could be over rj45 cat6a to same or over rj45 cat 6a to sfp+ or reverse.

If using a line rate 10gbe switch (and for this question, not considering mtu, encryption, etc.), it would be expected that throughput should be close to max through the switch with each end being on the same vlan.

Take something like a supermicro a2sdi-tp8f for example. It has 2 10gbe rj45 and 2 sfp+. Would the same equipment described above have a similar throughput near max like the switch with the supermicro (Atom C3858) device running something like pfsense (device not running a bunch of other thigs like IDS, VMs, etc.)?

The reason for asking is that the supermicro is also running the router (but should not need router or layer 3 like functions because each end in this example is on the same vlan) whereas the first example would seemingly do all the work in the switch since the connections are on the same vlan with another device runnning as router which I assume means that the data would never travel to said router.

In other words, can the switching for the same vlan in a strong enough device acting also as the router be as fast that of a discrete switch?

 

[scline]

Hrmm let me try to unpack this question.

Scenario 1:


Computer A to Computer B if doing basic Layer2 switching would have all its traffic processed by the switches ASIC, therefore you should get full bandwidth on the ports (10Gbps full duplex). The limiting factor here is going to be the computer processing power to push/receive this data and whatever protocols/overhead associated with the traffic. MTU should not make much of a difference for the switch side of things.

Scenario 2:


You using a server/computer as a switch. Normal x86 processors do not perform this type of task very well. If it's powerful then yes, you can get the full 10Gbps throughput between Computer A and Computer B - just at a higher CPU cost. The reason here is computer and server processors are not great and processing network traffic. Layer 2 in particular, also known as bridging interfaces, seems to do worse then Routing in my experience and generally is avoided. Higher MTU will help in this case though since there will be fewer packets for the processor to compute against.

TLDR with all features turned off, C3858 likely will be able to hit 10Gbps routing (or near it), I doubt it will be as higher or hit 10Gbps when switching. No way will it be able to hit the throughput of a dedicated switch. L3 or L2, computer CPU's just are not tailored to the task.

 

[dandanio]

"Can it?" Sure, it can be. "Is it?" Well, not always, or: it is not that obvious. As you know, switching over a multi port network adapter is not as obvious due to many conditions: PCIe speed, network buffers, cpu speed, OS tcp/ip stack and finally: STP. So, can it be done? Sure. and then some. Is that a2sdi-tp8f capable of doing it? Probably, but maybe not all the time on all ports? I do not know the architecture of the mobo to answer you authoritatively.

 

[BradleyD]

I have moved toward using an A2SDi-H-TF which is a 3758 board from supermicro with 2 10gb rj45 ports on it.

This is for a pfsense box although if Netgate ends up developing the SCLR os from the TSNR, that might be an option and would probably make my questions here not very relevant.

There would be a 10gb switch involved. However, my questioning originally was about using the atom 3858 mentioned in my first comment for switching. Again, that has changed with the atom 3758, but I have a question about switching on one port from the supermicro board.

First, of the 2 built in 10gb ports on SM board, one would be to the 10gb switch. The other should work for 2.5gb to the 2.5 gb cable modem (they are coming out now / soon). I have read on Netgate forums that the bandwidth should work at that rate with the current pfsense (although pfsense 2.5 could have an issue for a while) since the built in 2 port 10gbe on the board has x710 mac and I believe x550 phy if I recall correctly from intel info. Also, the info from Intel indicates that the 3758 will do 2.5 and 10gb along with 1gb, but it does not list 5 gb for those ports.

I might eventually want to add a higher end commercial type wifi 6E wap when they come out that can connect at 5gb and so I would need another port (also need a wifi client adapter that will do 6E). Ruckus already has one wap that will do 5gb on the rj45 port (wifi 6 not 6E) and with wifi 6E, I am seeing that connection speeds (not throughput) are advertised up to around 11,000mbps. Asus has a 6E router coming out fairly soon for instance. I would look to add a more commercial type wap like the ruckus, etc. as mentioned previously.

An x710 could be added (perhaps an x550 would work, but I have seen nothing on Netgate, Intel, anywhere else about x550s working at 5gb). The x710 would be 8 lanes of nic in a 4 lane pcie slot which the SM has, but I have read that the x710 2 port card should fit in the open ended slot and have also read that intel higher lane cards have worked in that type situation. Additionally, 4 lanes is enough for 2 ports of 10gbe based on pci 3 even if I decided to use the second port on the card say for 2.5gb to the modem as opposed to one on the mother board. Note: the 2 port x710 does not need 8 lanes for pcie 3, but I think they made both 2 port and 4 port cards the same lane size as a cost deal because the 4 port card does need more than 4 pcie 3 lanes.

So my question is a bit different now. I will say that I can connect a wap from another location with a switch that will do 5gb, but if I decided to have that x710 card involved, it would be easier to connect the wap to that card as it would be located very close to the router and so, less cabling.

There would be at least 2 SSIDs for the wap. One would be on a secondary vlan so it will have to be routed anyway. The other would be on the "main" vlan and from what I understand would need to be switched as far as I can tell. That being said, if I put that main SSID on a different vlan, I guess it too would have to be routed anyway, but would like to keep that SSID on the "main" vlan (just by choice).

So here is the question...

Based on the first response to my original post, I know it can route, but could that board with an x710 card which will do 2.5, 5, and 10gb switch up to 5gb basically when it needs to do so (for the wifi which might get pretty busy at times including large file transfers)?

Notes:

If a wap has a 10gb nic instead of or along with nbase-t, I would have an available 10gb port on the swich which makes the x710 unecessary.

I have looked at devices like the Mikrotik which can have rj45 10 gb modules that will do 2.5 gb at least and I suppose 5gb also, but not interested. I have no problem with those devices. It's just a choice.

I am not worried too much about cost and so adding a card is not a big deal. I'm not wealthy, but it can be afforded.