1U Supermicro Server 6x 10GBE RJ45 X10SLH-LN6TF LGA 1150 H3 X10SLH-N6-ST031

EasyRhino

Active Member
Aug 6, 2019
127
54
28
I wasted close to 3-4 days trying to make this motherboard work as a router + switch through windows/plex.
I haven't really used windows bridging, when I have messed with it it hasn't even worked.

However you can use linux or BSD (pfsense/opnsense) bridging. And it's... okay. not as good as a real switch but ok. Depending on CPU you should be able to get between like 1gig and 4+ gig speed.
 
  • Like
Reactions: Bert

EasyRhino

Active Member
Aug 6, 2019
127
54
28
I've done a fair bit of passthrough with Proxmox. (Started about 6 years ago. Variable success and problems.) Try this guide and if doesn't work post your issues. See if it's the board or the process.
Ultimate Beginners Guide to GPU Passthrough
so to follow up on my proxmox gpu passthrough attempts: I give up.

If I followed the reddi guide closely, and had to make two additional changes:
* had to patch and extract the old GT610 bios to support UEFU per here:
Forum - AMD and Nvidia GOP update (No requests, DIY) (win-raid.com)
* set machine type to pc-q35-3.1 (regular q35 had problems passing through the ethernet adapters)

And then I actually got the video out to display, and could even plug in a usb keyboard/mouse, but if I tried to reboot the windows VM is would hang the entire host. For debugging I tried moving the card to a different slot, and using legacy BIOS mode in proxmox, and neither fixed it.

So I went back to bare metal opnsense and now it's just a more overpowered router.

really bummed though since this box lives under my TV and it would be nice to have HDMI out.
 
  • Like
Reactions: Bert

lemkeant

New Member
Jan 19, 2021
2
0
1
I was able to pass through a "pair" (eno0 and eno1) of ethernet adapters to opnsense no problem. I don't remember if I Attempted to pass through one single jack or not.
Thanks. Im not sure, as anything I try to do seems to pass everything? I'm curious if it's because they're all under the same IOMMU group? Were you able to separate them? 2021_01_24_00_00_27_Proxmox_Proxmox_Virtual_Environment.png
 

Bert

Active Member
Mar 31, 2018
370
149
43
42
VyOS Switch config

show interface

configure
delete interfaces eth eth1 address
delete interfaces eth eth1 description
delete interfaces eth eth2 address
delete interfaces eth eth2 description
commit


Creating the bridge

set interfaces bridge br0

# set interfaces bridge br0 address 10.10.10.245/24

set interfaces bridge br0 address 10.222.222.1/24
set interfaces bridge br0 description 'bridge 1'

compare
commit
save
exit

Assign interfaces to the bridge
configure

set interfaces ethernet eth1 bridge-group bridge br0
set interfaces ethernet eth2 bridge-group bridge br0
set interfaces eth eth1 description 'br0 10gb member'
set interfaces eth eth2 description 'br0 10gb member'

set interfaces eth eth2 mtu 9000
compare
commit
save

May I learn what this config achieves? Does it create a bridge so we can set vyos to use the bridge instead of a single interface in its router functionality (nat. dhcp etc) or does it make the vyos work like a switch instead of router?
 

Bert

Active Member
Mar 31, 2018
370
149
43
42
I haven't really used windows bridging, when I have messed with it it hasn't even worked.

However you can use linux or BSD (pfsense/opnsense) bridging. And it's... okay. not as good as a real switch but ok. Depending on CPU you should be able to get between like 1gig and 4+ gig speed.
Thanks for the info! Yes windows bridge is a joke. I was not able to figure out how to set up a bridge with PfSense and I read PfSense performance is also dismal so I am giving up on PfSense.

I believe Marsh reported speeds close to 10Gb with vyos so I am now trying to make VYOS + HyperV work. I was able to set up VYOS as the router mode now if I can figure out how to configure VYOS with 5 port as a switch for the LAN and one port for the WAN. If anyone has already figured that, it would be great if I can take a peek. I am using vyos 1.1.8 as I couldn't figure out how to configure vyos 1.3.
 

Detonyle

Member
Mar 11, 2019
110
18
18
Does someone in Europe got some of those card.... shipping fees from ebay seller is sooo huge :/
 

dreamkass

New Member
Aug 14, 2012
23
5
3
Thanks for the info! Yes windows bridge is a joke. I was not able to figure out how to set up a bridge with PfSense and I read PfSense performance is also dismal so I am giving up on PfSense.

I believe Marsh reported speeds close to 10Gb with vyos so I am now trying to make VYOS + HyperV work. I was able to set up VYOS as the router mode now if I can figure out how to configure VYOS with 5 port as a switch for the LAN and one port for the WAN. If anyone has already figured that, it would be great if I can take a peek. I am using vyos 1.1.8 as I couldn't figure out how to configure vyos 1.3.
Running ESXi 7.0 U1 between 2 VM with a VyOS Router between them and x540-AT2 10GB nics, my servers are also the same as this 1U, running sys-6028u-tr4+, might pick one of theses U1 for FW/Routing purpose also running ESXi.

[root@iperf02 ~]# iperf3 -c x.x.x.x
Connecting to host 10.0.1.185, port 5201
[ 4] local x.x.x.x port 52064 connected to x.x.x.x port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.01 GBytes 8.70 Gbits/sec 147 789 KBytes
[ 4] 1.00-2.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.50 MBytes
[ 4] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.97 MBytes
[ 4] 3.00-4.00 sec 1.09 GBytes 9.41 Gbits/sec 0 2.36 MBytes
[ 4] 4.00-5.00 sec 1.09 GBytes 9.41 Gbits/sec 474 1.57 MBytes
[ 4] 5.00-6.00 sec 1.05 GBytes 9.04 Gbits/sec 777 1.23 MBytes
[ 4] 6.00-7.00 sec 1.10 GBytes 9.42 Gbits/sec 204 953 KBytes
[ 4] 7.00-8.00 sec 1.07 GBytes 9.22 Gbits/sec 0 1.57 MBytes
[ 4] 8.00-9.00 sec 1.09 GBytes 9.39 Gbits/sec 0 2.02 MBytes
[ 4] 9.00-10.00 sec 1.09 GBytes 9.35 Gbits/sec 1356 1.44 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 10.8 GBytes 9.28 Gbits/sec 2958 sender
[ 4] 0.00-10.00 sec 10.8 GBytes 9.27 Gbits/sec receiver

iperf Done.
 
  • Like
Reactions: Marsh and Bert

Bert

Active Member
Mar 31, 2018
370
149
43
42
Running ESXi 7.0 U1 between 2 VM with a VyOS Router between them and x540-AT2 10GB nics, my servers are also the same as this 1U, running sys-6028u-tr4+, might pick one of theses U1 for FW/Routing purpose also running ESXi.

[root@iperf02 ~]# iperf3 -c x.x.x.x
Connecting to host 10.0.1.185, port 5201
[ 4] local x.x.x.x port 52064 connected to x.x.x.x port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 1.01 GBytes 8.70 Gbits/sec 147 789 KBytes
[ 4] 1.00-2.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.50 MBytes
[ 4] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.97 MBytes
[ 4] 3.00-4.00 sec 1.09 GBytes 9.41 Gbits/sec 0 2.36 MBytes
[ 4] 4.00-5.00 sec 1.09 GBytes 9.41 Gbits/sec 474 1.57 MBytes
[ 4] 5.00-6.00 sec 1.05 GBytes 9.04 Gbits/sec 777 1.23 MBytes
[ 4] 6.00-7.00 sec 1.10 GBytes 9.42 Gbits/sec 204 953 KBytes
[ 4] 7.00-8.00 sec 1.07 GBytes 9.22 Gbits/sec 0 1.57 MBytes
[ 4] 8.00-9.00 sec 1.09 GBytes 9.39 Gbits/sec 0 2.02 MBytes
[ 4] 9.00-10.00 sec 1.09 GBytes 9.35 Gbits/sec 1356 1.44 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 10.8 GBytes 9.28 Gbits/sec 2958 sender
[ 4] 0.00-10.00 sec 10.8 GBytes 9.27 Gbits/sec receiver

iperf Done.
This is awesome but I am still hoping to get the configuration how to configure the bridges. Is this Vyos 1.1.8 or 1.3?



@Marsh , perhaps you can share your config?

I also want to ask which one would be better:

1. Use VYOS as full switch and then use pfsense and/or Windows RRAS as the router
2. Use VYOS as switch + router and then use nothing else.

The second option seems scary as it seems like using VYOS as home router could be quite a challenge as it may be missing many firewall/security features.
 
Last edited:

Marsh

Moderator
May 12, 2013
2,471
1,293
113
My test bed hardware ( CPU RAM ) and software config is in post #151
result in post #161 and vyos config is in #162

vyos version is 1.2.x

If you have questions, I am happy to answer.
 
  • Like
Reactions: Bert

Marsh

Moderator
May 12, 2013
2,471
1,293
113
The second option seems scary as it seems like using VYOS as home router could be quite a challenge as it may be missing many firewall/security features.

At one time , many years ago, vyos and Ubiquiti firewall Router shared similar code base.
vYos is a very capable firewall router, it is intimating to configure the firewall rules properly.
There are many examples of firewall / router if you search it.

One function is missing , DNS and DHCP tight integration.

I run vyos virtual router in all my internal network.
 
  • Like
Reactions: Bert

Bert

Active Member
Mar 31, 2018
370
149
43
42
Thank you very much, @Marsh! Your help will be greatly appreciated.

I tried to use the configuration at post but I am not able to get the router functionality working with the bridge (I decided to use VYOS as the router to keep things simple).

I was able to make the VYOS working as router before adding the switch configuration. I am running VYOS on VM but my devices are physical, hence all hypernet switches are external. I attached my script/configuration. I am not sure why setting the interface on bridge does not let devices connected to the ports taking address and connectivity. Other than one port, none of the ports on the switch is giving IP to the connected devices. Eth1 is the only one giving IP but routing is still not working hence there is no internet connectivity. Same VYOS set up was working fine before setting up the bridge.
 

Attachments

Last edited:

Bert

Active Member
Mar 31, 2018
370
149
43
42
I followed the instructions given here:
.

Pretty much my conf is exactly same other than the management port for me to ssh. I am pretty sure Hyper-V is configured correctly as it worked when I was running VYOS as router. Now, I expect to see the computer connected to one of the ports have received IP from the home router and indicate internet connectivity but none of them is happening. Am I missing something obvious with VYOS conf?
 

Attachments

Zombielinux

Member
Jun 14, 2019
30
6
8
Followup questions after reading this thread:

1) Can the individual pairs of nics be turned off (maybe via powertop?) I really only need one or two (for now)

2) What chassis is this? From seeing individual pictures, it looks like a CSE815. If that's the case, I'd be interested to see if an optical drive kit would fit in it.

I am looking to make a low power/maintenance SAS to iSCSI target server, and if this works, it would fit the bill.
 

EasyRhino

Active Member
Aug 6, 2019
127
54
28
from my old post #183, you can disable NICs (pairs) with jumpers. When I disabled all 3 it only reduced power consumption from idle 60W to 50W.