ROMED8-2T LAN Ports Not Working

[area51]

Hi all. I'm completely new to server builds and just built my first PC. It's based on the ROMED8-2T with an Epyc 7302P. 128 GB RAM (32x4). I have a 2TB SSD. 2x1TB M2 SSDs. 12TBx6 HDDs that will ultimately be in a raidz2 config. Noctua cooler. Corsair 750 power supply. Housing is Fractal Design Define 7.

The good news is that the CPU posted correctly the first time. All the drives and memory show up fine. I am able to connect to the admin web over the home LAN using the IPMI port. I was able to install Proxmox. However, I am unable to reach Proxmox web GUI over IPMI.

More importantly, I tried to instead use the the 2 10Gb LAN ports to connect and I'm not getting any DHCP assigned IPs. Also, when I plug in my ethernet cable into either of the 10Gb LAN ports, I don't see any status lights at all. On the other hand, plugging into the IPMI LAN port gives me status lights on the IPMI jack.

I have no idea what I'm doing wrong here. Am I even able to use those 10Gb ports within Proxmox? Is there a BIOS setting that I'm missing? Defective MB?

In summary: 1) how come I get a DHCP address via the IPMI and can reach the management web gui but can't reach the Proxmox gui and 2) how come I get no status lights or IP address when plugging into either of the 10Gb LAN ports?

 

[altmind]

X550-AT2 is very reliable and should be supported by any linux distro. have you tried restoring your bios settings to factory?

also, do you see the nics in ifconfig/ethtool?

 

[slidermike]

Area, a couple of things and in no specific order.
#1 "no lights" means disabled port: bad cable or not seeing a link from the other end aka the switch. check mobo bios for the nic's settings. Make sure they are enabled and you will likely want them set to dhcp.
#2 these older 10gb nic only do 1gb/10gb and to my knowledge you cannot set the rate on the board. Mine will not connect to my off-brand poe 2.5gb unmanaged switch no matter what I do. So using a quality switch couple be helpful once you get past the "no lights".
#3 Once you validate the nic is enabled and dhcp working, you should see it in the bios (takes a reboot after save/reboot if you changed the settings) as online as well as the lights on the port. Then you can boot to proxmox.
#4 if still no connection but you have link lights then from proxmox command you have to do some tweaking. See what interfaces are there and find the one that matches the Mac address of the 10gb port. It should have a name, possibly two. Examp: eto4 or ep07f01 with the command 'ip a' minus the quotes. Once you have identified and written down the interface name the you want to verify that interface is a member of the br0.
You will have to google how to edit but you can look at the configuration here by this command 'cat /etc/network/interfaces'
You will want the 10gb interface name to have an entry near the top and also in the br0 section which tells proxmox to add that phy port to the bridge which is what you connect to.
It may be easier to just reinstall proxmox so it does this but then you do not learn how this work

Screen shots of your ethernet bios settings/status would be helpful as well as the output of the two above proxmox commands. (once you get to the point where you get nic lights. Otherwise no use looking at proxmox. You have to get the layer 1 working first.

 

[i386]

is that a used board?
did you (or the previous owner) update firmware on the nics with firmware from intel?

I remember a thread here about bricked intel nics on an asrock mainboard after a firmware update.

 

[area51]

Hi all. Thank you for all your advice. I actually have figured most of it out. I think the main problem was that I didn't put the metal I/O faceplate on properly. Again, being a total novice, I didn't realize the little prongs had to touch the ports a certain way. After adjusting the plate, I get the small LED light activity on all the LAN ports.

I was actually able to get Proxmox up and running on one of the 10Gb LAN ports getting an assigned IP from my router. One of the issues was that the IPMI and eno1 ports were bonded by default and I could only get Proxmox to work by unbonding the two ports in the BIOS. The third eno2 port doesn't appear in the BIOS (see below).

i386: it's a brand new board with a used Epyc 7302. I have not touched the firmware at all so far.

There are still some problems. I can't get the second 10Gb port to show up in the BIOS at all. It appears inactive. However, when I plug in an ethernet cable there is a blinking orange-ish light which is identical to the light on the the other 10Gb LAN port that does show up in the BIOS. That second port also shows up as the eno2 interface in Proxmox but it's listed as "inactive."

The other problem is that I was getting a lot of disconnections in Proxmox. I think I have an idea what the problem might be: the ASRock IPMI/BMC has a web gui interface on port 443 for that same IP. I wonder if there is some conflict between the Proxmox management port and the IPMI management port (not sure why that would be as they're both https but one is port 443 and the other is 8006 by default.)

Some data dumps:

-----

cat /etc/network/interfaces
auto lo
iface lo inet loopback

iface eno1 inet manual

auto vmbr0
iface vmbr0 inet static
address 192.168.1.209/24
gateway 192.168.1.1
bridge-ports eno1
bridge-stp off
bridge-fd 0

iface enx5*********** inet manual

iface eno2 inet manual

source /etc/network/interfaces.d/*

-----

ethtool eno1
Settings for eno1:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
2500baseT/Full
5000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
MDI-X: Unknown
Supports Wake-on: umbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link

Link detected: yes
—

ethtool eno2
Settings for eno2:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
2500baseT/Full
5000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Unknown! (255)
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
MDI-X: Unknown
Supports Wake-on: umbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: no

—

ethtool enx5***********
Settings for enx5***********:
Supported ports: [ ]
Supported link modes: Not reported
Supported pause frame use: No
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: Not reported
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: Unknown!
Duplex: Half
Auto-negotiation: off
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
MDI-X: Unknown
Current message level: 0x00000007 (7)
drv probe link

Link detected: no