$127 Cisco ENCS5412/K9 Xeon-D 1557 (12 core), 32G ram

[turbo]

For me, OOB management with remote console (CIMC), total core count, and VM-VM networking performance. I have about 20 VMs on mine with maybe 12 running at any one time.
I migrated a bunch of separate physical systems into VMs on mine and improved total performance and lowered power draw substantially. Since they are all on one box now the external network limits aren’t really an issue for me. With the 8 switch ports I can connect my high 2 usage clients and my WAPs directly to it.

 

[Navy_BOFH]

Same. I loaded Proxmox onto mine with Home Assistant in a VM with a handful of LXCs for other home tasks. Having IMPI was important since once all this is racked up it will be in a closet where I don't want to drag equipment to when I need to maintain/troubleshoot. My "other server" will be moved to an HPE EL1000 so I can get IPMI in something small/quiet with iGPU power.

 

[foureight84]

Well. Interesting.

So the CIMC chassis inventory still thinks this:
Memory
DIMMList:
id: 1
Name: DIMM_A
Capacity: 32768 MB
Speed: 2400
MemoryType: DDR4
MemoryTypeDetail: Synchronous
BankLocator: NODE 1
Manufacturer: Micron
SerialNumber: ##########
AssetTag: DIMM_A_AssetTag
PartNumber: 36ASF4G72PZ-2G3B1
Visibility: Yes
Operability: Operable
DataWidth: 72 bits

id: 2
Name: DIMM_B
Capacity: Not Installed
Speed: NA
MemoryType: NA
MemoryTypeDetail: NA
BankLocator: NA
Manufacturer: NA
SerialNumber: NA
AssetTag: NA
PartNumber: NA
Visibility: NA
Operability: NA
DataWidth: NA

But I finally was able to get the system to boot on a pair of 8GB DDR4 2133P DIMMs, and BIOS (2.6) reports them both.

So as best I can tell, the unit was failing POST with both of the 32GB DIMMs installed, and would only boot with the original 32GB DIMM. Thus, indicators point to a bad DIMM.

Sucks that Cisco have this box limited to 32GB DIMMs, I've got a shedload of 64GB DDR4 ECC DIMMs.... and yes, I tried to boot with one of those installed too... same problem, won't complete post.

Oh well.

Thanks @devz3ro for bailing me out on the BIOS thing.

Wait, so max is only 32GB? or 32GB per slot?

 

[ccie4526]

Wait, so max is only 32GB? or 32GB per slot?

So the CIMC won't update until the compute component is fully booted. My unit shows dual 32GB DIMMs for the total of 64GB. The unit does NOT support any form of 64GB DIMM.

Edit to add image.

 

[foureight84]

So the CIMC won't update until the compute component is fully booted. My unit shows dual 32GB DIMMs for the total of 64GB. The unit does NOT support any form of 64GB DIMM.

That's so unfortunate.

 

[ccie4526]

That's so unfortunate.

I agree, I have a large amount of 64GB DDR4 pulls from HPE servers and I was hoping to use two of them in this box, but nooooooooooooo.

 

[foureight84]

I agree, I have a large amount of 64GB DDR4 pulls from HPE servers and I was hoping to use two of them in this box, but nooooooooooooo.

I noticed that you had hardware locking issues after upgrading NFVIS. What version are you currently on?

EDIT: nvm. I missed the reply where you said it was due to a bad dimm stick and not hardware lockdown by cimc or bios update.

 

[Navy_BOFH]

What is the best way to update the BIOS/CIMC once on Proxmox? I have one online as a dedicated LXC/Docker node and now I realized the firmware is decently out of date and I am locked out of the CIMC.

 

[Navy_BOFH]

New thing I am experiencing and wondering if anyone else has seen the same:

I installed Proxmox 8.0.2 on both ENCS5412s I received. Both within a varying amount of time and a reboot or other event will cause port enp2s0f0 to drop down to 10Mbit/Full from 1000/Full. Both servers rebooting, reinitializing the interface, or forcing speed/duplex and disabling POE on my UDM Pro SE doesn't change it. If I move the vmbr0 interface to enp2s0f1 - everything returns to normal and I get gigabit from the server. I believe on my other ENCS I tried moving the bridge back to the original port and it dropped back down to 10/Full.

Has anyone else experienced this with Proxmox, or am I seeing an odd hardware failure in both servers now?

 

[peramus]

I had some issues and had to play around with the BIOS/CIMC settings for the shared vs dedicated port. I don't think proxmox likes it being setup for shared. But as far as the two RJ45/SFP ports I had no issues. Could just be a questionable network cable or take a look at the pins in the port on the ENCS itself make sure you don't have any bent ones.

 

[yeyus]

New thing I am experiencing and wondering if anyone else has seen the same:

I installed Proxmox 8.0.2 on both ENCS5412s I received. Both within a varying amount of time and a reboot or other event will cause port enp2s0f0 to drop down to 10Mbit/Full from 1000/Full. Both servers rebooting, reinitializing the interface, or forcing speed/duplex and disabling POE on my UDM Pro SE doesn't change it. If I move the vmbr0 interface to enp2s0f1 - everything returns to normal and I get gigabit from the server. I believe on my other ENCS I tried moving the bridge back to the original port and it dropped back down to 10/Full.

Has anyone else experienced this with Proxmox, or am I seeing an odd hardware failure in both servers now?

I experienced a similar issue when I enabled my CIMC port to share GE0/0. Once I reverted CIMC to use its dedicated port and unplugged the server for a bit it fixed the situation.

 

[Navy_BOFH]

I experienced a similar issue when I enabled my CIMC port to share GE0/0. Once I reverted CIMC to use its dedicated port and unplugged the server for a bit it fixed the situation.

That... would 100% explain that... lesson learned! I still need to figure out CIMC upgrade/reset so I can even use it - I rushed into installing Proxmox and then realized my CIMC was not reset.

 

[DRPNETOP]

I have purchased a used 5412/k9 but it seems to not have a bios installed. I can connect to CIMC both CLI and GUI but there is no inventory listed. It shows the BMC version but no BIOS version. With no access to a bios file, it is a paperweight. Just my luck.

 

[turbo]

just FYI I ended up getting an NIM-LTEA-EA module to test, but it does not appear to get powered up (under ESXi) or appear on any bus. I suspect there is some OOB mechanism to power it on. I assume it would work with NFVIS so I plan to do some investigation but I was curious if anybody has tried any other NIMs in these units?

 

[airsky98]

anyone have the latest package for firmware, nfvis, cmic, etc

please PM me

 

[jolly99]

Looking for ISRv Software. Please PM me.

 

[peramus]

Looking for ISRv Software. Please PM me.

I'm pretty sure you can straight up download the ISRv software from cisco's website just by signing up. BUT it won't do you a whole lot of good because without a license it's limited to 1MB of throughput.
That's one of the main reasons why these systems are available for so cheap. In order to make them work as intended you will have to spend thousands of dollars in licensing fees. They do support the latest CSR8000v platform as well but similarly that image is VERY easy to find but requires a license for any usable throughput outside a basic lab environment.

 

[airsky98]

still unable to find latest firmware package. anyone can share?

 

[peramus]

I know this thread is a couple months old but I'm still running this setup and making changes and additions. I've recently started messing around with a CAT8KV vm and was wondering if anyone has tried doing a passthrough of the switch module to a cisco router VM to see if it will detect and manage the marvel switch module that is presumably the same thing as a NIM-ES2-8?

 

[turbo]

Mine is still running great as well. I did try messing with passthrough but the switch module did not get activated, and the C8000V did not even seem to work with sr-iov on the XL710 virtual interfaces. The XML interface they use to manage the switch seems pretty specific to NFVIS.