Also for the record Proxmox 9 changed all of the interface naming which can cause some confusion. It's simplified overall but trying to figure out what's what on a box like this can be a PITA. Essentially they have the os rename them to nic0, nic1 etc. When you do an IP link the altnames shown no longer reference the PCI bus ID's (as in enp0s2 etc) instead they seem to only reference the mac address (as in enxf80f6fed0812)
$127 Cisco ENCS5412/K9 Xeon-D 1557 (12 core), 32G ram
- Thread starter RobstarUSA
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I was not able to get 9.2 installed on this box and I was going to do an upgrade to check if the TUI i made still works in configuring the switch. I am still on bios version 2.5 and CIMC 3.2 (or something really early). I actually haven't been able to upgrade the bios or CIMC at all through the current CIMC. The 9.2 installer would not launch and it gives the following errors: "invalid magic number" and "you need to load the kernel first". I've had this issue before on older machines and it's related to CSM support but nothing I do on this box seems to work.
I'm glad my investigation was useful, impressive work!
I just completed the LAG implementation on the TUI -- also it now can self update to the latest release on github.
TE1/2 are actually the X710 backplane so messing with them is not a good idea. TE3/4 are most likely for the NIM but I don't have any on hand to test and most likely if you install a NIM they will show up differently and TE3/4 is like TE1/2 -- backplane traffic. TUI has been updated to correct that along with correct labeling to match the front panel labels.
I don't have an Cisco account. Could you DM me a link to the ISO? i want to check it out to see if they added new features to the NFVIS switch management. Also, the TUI should not be impacted by Proxmox 9.
It helped a lot. It's the main backbone to all of this and playing around led to getting PoE management to work.
Current state of the TUI
Here are features from NFVIS that could be ported over to the TUI for completeness. Let me know if you guys need them and I can move ahead.
| Implemented | Where |
|---|---|
| Port enable/disable | Ports view, space |
| VLAN create/delete | VLANs view, n / d |
| VLAN port membership | saved and replayed — no UI editor yet |
| Link aggregation | Ports view, g |
| PoE on/off | PoE view, space |
| MAC table, counters | read-only views |
Here are features from NFVIS that could be ported over to the TUI for completeness. Let me know if you guys need them and I can move ahead.
| Features | wcd tables that exist |
|---|---|
| Spanning tree (STP/RSTP/MSTP) | STP, RSTP, MSTP* |
| QoS — class/policy maps, policers, queueing | ClassMapList, PolicyMapList, AggregatePolicerList, CoS*, DSCP* |
| ACLs | ACLList, ACEList, ACLBindingList |
| 802.1X + RADIUS | Standard_802_1x*, RadiusServerList |
| IGMP/MLD snooping, multicast filtering | IGMPMLD*, MulticastGlobalSetting |
| Storm control | StormControlTable |
| LLDP / CDP | LLDP*, CDPInterfaceList |
| L3 — ARP, static routes, default gateway | ARPList, IPv4RouteList, IPv4GatewayList |
| Port mirroring (SPAN) | SpanDestinationTable |
| Private VLANs | PrivateVLAN* |
| Static MAC entries, aging | ForwardingStaticTable |
| LACP tuning — system/port priority, timeout | LACPGlobalSetting, LACPPortList |
Well I definitely could have used storm control and spanning tree lol. I made a dumb move and added both 10G interfaces to the same vmbr. Let's just say I spent all night trying to get my host to stop crashing on bootup. But I am running fine now and my 3 cisco 3800 wireless access points are connected to the switch module with no issues. Also I'm running bios version 4.00. I'd have to dig around again but the bios firmware is located somewhere on the ISO but it's inside of a package file. once you manage to extract it you can upgrade via CIMC (you can also find the CIMC firmware) I haven't dug around in the 4.15.6 ISO yet but I'm kind of scared now that y'all are saying that the newer versions disable boot options. For the record there is also a 26.1.1 software version which made some serious changes to the NFVIS GUI and made it much more responsive... but I still like proxmox WAY better. And if I could get the NIM module I have (It's a single 1G port) working I'd be ecstatic.
FYSA:
Proxmox Version: 9.2.5
BIOS Version: ENCS54_4.00 (Build Date: 04/04/2024)
CIMC Version: 3.2(14.19)
I have a fresh box on my desk I'm probably going to install 4.15.6 on anyway so if the bios doesn't lock it all down I might upgrade my home box.
As for the NIM I don't think it's tied to the switch at all. I don't see it as a PCI device ID for it or I would try passing it through to an 8000v and see if it populated but under normal conditions when you enable NGIO on an 8000v VM running on genuine NFVIS the NIM module appears as if it was inserted in the router (similar to how it would show up on a normal router chassis) they usually show up as G0/2/0 & G0/2/1 and the router appears to have full control over the hardware. I wonder if Te3/4 are just dead ports. (could be an interesting upgrade if the pins actually appear somewhere inside the box and it's just as easy as soldering a port to them).
FYSA:
Proxmox Version: 9.2.5
BIOS Version: ENCS54_4.00 (Build Date: 04/04/2024)
CIMC Version: 3.2(14.19)
I have a fresh box on my desk I'm probably going to install 4.15.6 on anyway so if the bios doesn't lock it all down I might upgrade my home box.
As for the NIM I don't think it's tied to the switch at all. I don't see it as a PCI device ID for it or I would try passing it through to an 8000v and see if it populated but under normal conditions when you enable NGIO on an 8000v VM running on genuine NFVIS the NIM module appears as if it was inserted in the router (similar to how it would show up on a normal router chassis) they usually show up as G0/2/0 & G0/2/1 and the router appears to have full control over the hardware. I wonder if Te3/4 are just dead ports. (could be an interesting upgrade if the pins actually appear somewhere inside the box and it's just as easy as soldering a port to them).
How did you manage the passthrough? I've got a 1GE NIM but I don't see it enumerated on the PCI bus anywhere.
I did some digging around and the NIMs are gated by CIMC. I found this out after trying to see if there's a workaround to get a NIM-SSD from the ISR 4000 to work.. The only exposure is likely TE3/4 which is for the NIM connection (backplane) that we can turn on/off via the switch API just like the TE1/2 (backplane between the Marvel switch (GE1/0-GE1/7) and the x710 which is the controller for those switch ports).
Last edited:
v.0.1.0 as been released: GitHub - foureight84/encs5400-almalinux
What has not been tested
Nothing here is known broken — it is unmeasured, and the missing piece is hardware rather than code. Grouped by what you would need:
Still not implemented. Three of these are blocked on missing information:
| Implemented | Where | Tested |
|---|---|---|
| Port enable/disable | Ports view, space | data plane |
| Port description, speed, duplex, flow control | Ports view, ENTER | data plane (speed + duplex forced on a live link) |
| VLAN create/delete/rename | VLANs view, n / d / N | on hardware |
| VLAN port membership | VLANs view, ENTER — access, trunk or general | data plane (isolation measured) |
| PoE on/off | PoE view, space | data plane (two PDs, 7.5 W) |
| PoE per-port power limit | PoE view, ENTER | data plane (enforced at classification) |
| Port mirroring (local SPAN) | TAB → mirror | data plane (copies counted) |
| Static MAC entries, aging | TAB → staticmac | on hardware |
| Storm control | TAB → storm | on hardware — rate limiting not measured |
| LLDP / CDP advertisement | TAB → lldp | on hardware — no neighbour table exists |
| LACP tuning — system/port priority, timeout | TAB → lacp | on hardware |
| MAC ACLs — rules and port bindings | TAB → acl | on hardware — filtering not measured |
| Spanning tree (STP/RSTP) | TAB → stp | on hardware per port — see the warning below |
| Link aggregation | Ports view, g | on hardware (0.0.4) — negotiation untested |
| MAC table, counters | read-only views, f flush, z zero | on hardware reads; flush/zero writes only |
| QoS — mode, trust, port CoS, CoS→queue, policers | TAB → qos | writes only |
| 802.1X + RADIUS | TAB → dot1x, radius | writes only |
| IGMP/MLD snooping | TAB → igmp | writes only |
| Private VLANs | TAB → pvlan | writes only |
| L3 — static routes, gateway, static ARP | TAB → l3 | writes only |
What has not been tested
Nothing here is known broken — it is unmeasured, and the missing piece is hardware rather than code. Grouped by what you would need:
| Needs | Untested |
|---|---|
| A second switch | LAG/LACP negotiation (including the auto vs on black-hole trap), STP loop breaking, root guard, BPDU guard |
| A traffic generator | Storm-control rate limiting, QoS marking and queueing, policers |
| Endpoints with IP addresses | ACL filtering, IGMP snooping, private VLAN isolation, L3 routing and static ARP |
| Nothing — just never run | MAC flush, counter clear, MSTP, DSCP maps, per-port shaping |
Still not implemented. Three of these are blocked on missing information:
| Area | Why |
|---|---|
| IPv4 ACL rules | switch-confd only ever built MAC ACLs. The element names inside <IPv4Parameters> appear nowhere in the datamined source, and a guessed rule is one the switch accepts and never matches. MAC ACLs are complete. |
| LLDP neighbour table | Does not exist. confd touches only LLDPGlobalSetting and LLDPInterfaceList, so there is nothing that answers "what is plugged into GE1/3". LLDP can be enabled and timed; it cannot be read back. |
| Remote SPAN | Needs a reflector port and a remote VLAN. Local SPAN works. |
| MSTP instances | The client can write region, revision, instance priorities and instance→VLAN maps; there is no view. MSTP on an 8-port edge switch with one region is not worth the screen — use encs-switch-api. |
| DSCP mutation/remark, per-port shaping, class/policy maps | The client can write all of them; there is no view. Reach them with encs-switch-api, or see docs/CONFIG.md. |
| Port security | InterfaceSecurityTable templates exist, but the mode and violation enums were never pinned down. |
Last edited:
Ok so the newer BIOS didn't remove the ability to get into the configuration. They just changed the way you need to do it. The CIMC has an option under computer -> BIOS to Enter the bios settings on the next boot. It exposes the ability to hit F2/ESC to enter it again. Also most of the settings you need to change are there in the CIMC anyway. Also 4.03 is the Latest BIOS from the 4.15.6 ISO. You can find the bin file under /packages/nfvis-fwupdate-<version>.x86.rpm on any of the NFVIS ISOs the files are under /opt/fwupdate inside the RPM. You'll want to grab the ENCS54_BIOS_X.XX.SPA and the CIMC_X.X.X.X.bin file. I still see the BIOS file in the latest versions (4.18.4 and 26.1.2) but it looks like the CIMC file is missing from some of the newer versions.
Last edited:
That is awesome! Really good to know.
Also, I noticed that mine always says "unknown" for the Overall Storage Status. I still can't figure out why that is.
It's because you don't have the optional RAID controller installed. My personal box says the same thing but all of our work systems came with the RAID card and show a status.
I've been trying to find that raid controller but I never see it on ebay. It adds SAS support too?
Yes it adds SAS support I've looked for them too even by part number and never found anything except out of stock listings for $600-8K on third party websites.
Also in case anyone was wondering the module that sits overtop of the switch chip is actually a PVDM DSP module for VOIP. Whether or not it has any use at all under Linux I have no idea or if it's actually in all of them or just all the ones I've messed with.
Also in case anyone was wondering the module that sits overtop of the switch chip is actually a PVDM DSP module for VOIP. Whether or not it has any use at all under Linux I have no idea or if it's actually in all of them or just all the ones I've messed with.
Version 0.2.0 released: Releases · foureight84/encs5400-almalinux
Biggest changes in this update is the install script now prompts whether you want to create a
Installer script will also add notes to the network bridges and interfaces so you know what they are and don't accidentally delete them.
Biggest changes in this update is the install script now prompts whether you want to create a
swbr0 switch on proxmox for VMs instead of using vmbr0 which is currently shared with almalinux. swbr0 is supposed to mimic NFVIS' lan-net which is isolated from management console. You can also create and tear down swbr0 in the TUI as well in case you said no the first time. The difference between lan-net and swbr0 is that NFVIS defaults it to GE1/0 (first port on the switch) where you can specify which port.Installer script will also add notes to the network bridges and interfaces so you know what they are and don't accidentally delete them.
Last edited:
The need for NVIS (Cisco's hypervisor) is no longer needed in order to control the switch ports. I created a small almalinux image that contains all of the needed kernel modules and software to control the switch along with a TUI. Reloading it with Proxmox won't kill the hardware features. It should work with ESXi but I haven't had a chance to test it yet.