$127 Cisco ENCS5412/K9 Xeon-D 1557 (12 core), 32G ram

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

bsmith7392

New Member
Nov 13, 2023
8
6
3
I am working on a simple ISO that can bootstrap the switch module, but in the meantime you can run NFVIS in a VM and pass through the “Marvell Ethernet Controller” should be PCI device 0e but I have seen it show up as 0b after messing around with various settings in the BIOS.

truenas core and ZFS does work both with RDM and passing through the Lynx Point SATA controller through ESXi so I see no reason that truenas scale wouldn’t work on the bare metal though I haven’t tried it.
Gotcha, I'm actually running TrueNAS Scale baremetal on one of these with no issues, running two Micron 1100 2TB SATA SSD's in a RAIDZ1. Things seem stable.
 

yeyus

New Member
May 8, 2021
14
14
3
After that I booted with rd.break to change the root password and the admin user password. The process that responds on port 22 is some sort of cisco proprietary SSHD but the real SSHD is listening on port 22222 at least in my ancient version of NFVIS. Need to disable the firewall to get to it though.

...

The Python code for configuring the switch is all in /opt/switch-confd
Were you able to get dropped into a shell through 22222? cisco documentation states it is an SCP only port
 

turbo

New Member
Mar 17, 2022
26
22
3
Were you able to get dropped into a shell through 22222? cisco documentation states it is an SCP only port
I'm not running NFVIS anymore, so this is from memory, I may have had to change the shell to /bin/bash and there is a process that runs that tries to "fix" the admin and root accounts, so I may also have made /etc/passwd and /etc/shadow immutable (chattr +i) in order to get the shell working
 

bsmith7392

New Member
Nov 13, 2023
8
6
3
I found the easiest way to get into NFVIS was to use the CIMC to mount a Linux Live ISO, boot from that, mount the internal SSD and remove the user.cfg as devz3ro pointed out.

After that I booted with rd.break to change the root password and the admin user password. The process that responds on port 22 is some sort of cisco proprietary SSHD but the real SSHD is listening on port 22222 at least in my ancient version of NFVIS. Need to disable the firewall to get to it though.

Maybe they made it better in the later versions, but mine was just a lousy wrapper for qemu that wouldn't even let you power off a VM if it was in the "wrong" state, like "starting up" :rolleyes:

I added 1x MTA36ASF4G72PZ-2G3B1 to expand to 64GB, and these UCS-SD120GBKS4-EV work for the drive carriers. I'm not using the S3510s in the ENCS, I swapped with some drives I had laying around (3.8TB PM863, 1.9TB HK4R) and they work fine.

The Python code for configuring the switch is all in /opt/switch-confd

Fans are controlled by the CIMC CPU I believe, and they are the same volume with ESXi as they are with NFVIS, very reasonable for a 1U but I'm not sure I'd want it in the same room with me. If you disconnect 1 fan though the thing goes full blast and that's extremely loud. In normal operation it's about the same level as my 1U 48P gig switch.
Out of curiosity, have you been able to find any rack ears (rack mount kit) for these? I was looking around and these are the closest i could find that seem to match the ear layout for these - For Cisco ACS-4430-RM-19 ISR 4430 4431 4330 19 inch Router Ears Rack Mount Kit | eBay
 
Last edited:

turbo

New Member
Mar 17, 2022
26
22
3
mine came with rack ears, I have other Cisco gear I could check Monday if they match up with the ASR or switch patterns. the unit can also be racked from the opposite side in case anybody cares- useful for me as I have mine hanging vertically
 
Last edited:

turbo

New Member
Mar 17, 2022
26
22
3
I really hate how Cisco has a million different rack ear configurations for their 1U devices. :mad: I must have 5 different sets, all different, and none match the ENCS ears.

There is a P/N though 700-45575-01 and some sites seem to list both this P/N and ACS-4430-RM-19. They do look identical.
 

bsmith7392

New Member
Nov 13, 2023
8
6
3
I really hate how Cisco has a million different rack ear configurations for their 1U devices. :mad: I must have 5 different sets, all different, and none match the ENCS ears.

There is a P/N though 700-45575-01 and some sites seem to list both this P/N and ACS-4430-RM-19. They do look identical.
Thanks! I did order the ones for the acs-4430-rm-19. Figured it was worth the $10 risk. I'll report back if they do indeed fit.
 

Fritz

Well-Known Member
Apr 6, 2015
3,391
1,397
113
70
Years ago I ordered a box full of rack ears and none of them were labeled. I've wasted more time digging through this box than a care to mention. :p
 
  • Haha
Reactions: sic0048

Navy_BOFH

Active Member
Aug 2, 2013
184
75
28
I am working on a simple ISO that can bootstrap the switch module, but in the meantime you can run NFVIS in a VM and pass through the “Marvell Ethernet Controller” should be PCI device 0e but I have seen it show up as 0b after messing around with various settings in the BIOS.

truenas core and ZFS does work both with RDM and passing through the Lynx Point SATA controller through ESXi so I see no reason that truenas scale wouldn’t work on the bare metal though I haven’t tried it.
Just curious how this ended up for you! I am closing on a new home soon and have been thinking of adding the ENCS into my rack as a small "home server" and leveraging the POE portion for cameras. My thought is to run Proxmox with the switch passed through to a VM with Scrypted/Frigate/etc, and running Home Assistant and a few other bits and passing the USB port over if I need to use an external dongle for Z Wave or similar.

I haven't had much time to play with these yet so I have been on pins and needles to see what the community has come up with!
 

ccie4526

Member
Jan 25, 2021
98
68
18
So after cussing my way through NFVIS 4.13 for the past few weeks, I have finally given up. I can get the switch communicating with devices externally, but I can't get the virtual machines to communicate to anything on the switch, configuring either through GUI or through CLI.

In the process of installing the ESXI 7.0.3u3-Cisco image. I'll figure out how to make this work without the onboard switch.
 

devz3ro

New Member
Nov 16, 2019
22
2
3
So after cussing my way through NFVIS 4.13 for the past few weeks, I have finally given up. I can get the switch communicating with devices externally, but I can't get the virtual machines to communicate to anything on the switch, configuring either through GUI or through CLI.

In the process of installing the ESXI 7.0.3u3-Cisco image. I'll figure out how to make this work without the onboard switch.
Yeah, I gave up on NFVIS about 15 minutes after using the latest version. It still feels like a hand full of people created it in their garage over a couple weekends while intoxicated. Not saying I could develop something better myself, it just doesn't feel "enterprise".

I've been using ESXi 8.0u2 with the cisco customized image and it's been working great. Just needed to do a few tweaks like enable SHA-256 TPM 2.0, enable speed stepping, and change cstate to 2 in the bios. Since I already have a poe switch I wasn't hell-bent on getting the encs internal poe switch working.

If you need (2) more ethernet ports, go on eBay and search for "MA-SFP-1GB-TX" they are around $20 each (I got mine for $12/each buying in bulk). Since the SFP ports work, this will give you 2 more copper ports. "MA-SFP-1GB-SX" is the model if you need fiber. I bought 2 of these for each encs I have and they work great, giving a total of 3 copper ports for ESXi and 1 copper port for the CIMC.





 
  • Like
Reactions: Fritz

ccie4526

Member
Jan 25, 2021
98
68
18
If you need (2) more ethernet ports, go on eBay and search for "MA-SFP-1GB-TX" they are around $20 each (I got mine for $12/each buying in bulk). Since the SFP ports work, this will give you 2 more copper ports. "MA-SFP-1GB-SX" is the model if you need fiber. I bought 2 of these for each encs I have and they work great, giving a total of 3 copper ports for ESXi and 1 copper port for the CIMC.
Fortunately I have no shortage of Cisco SFPs, 1G copper and fiber, 10G copper and fiber, etc.... and yeah I've got SFP0 connected to my home switch currently.

I'm about to try something really strange... adding the SFP1 port as a "uplink" in ESXI to the "internal" vSwitch, then running a short jumper from the CIMC port to the SFP... and see if I can access the CIMC from a VM on the "internal" vswitch.

I have a firewall VM that connects "internal" to "external", and "external" is where the internet connection resides... and that has an IPSec connection to my home firewall with routing across that virtual tunnel interface.

The ultimate plan is trying to drop this 1U box in a co-lo for cheap, let it be the hub for all my IPSec tunnels, and keep all the management connections secure. And yes, if that FW VM goes sideways, I lose access to the whole box, but the co-lo would be abount an hour drive away... an annoyance but not an inconvenience.
 

ccie4526

Member
Jan 25, 2021
98
68
18
BTW, do any of you guys know about the internal port on these? I'm interested in putting a raid card inside it if it's available to buy relatively cheap on somewhere like eBay. A card with mini sas would be ideal.
I've been trying to find one myself, to no avail. I've got a couple Intel S3710 800GB SSDs in the drive slots, and I'd really like to mirror them.
 

devz3ro

New Member
Nov 16, 2019
22
2
3
If you read further down on this page it mentions a possible raid card:

  • RAID Card: The RAID card improves the performance of the hard drive. The RAID card is installed in the Internal Service Processor (ISP) module of the motherboard.
 

devz3ro

New Member
Nov 16, 2019
22
2
3
I haven't been able to find any cisco "ISP" raid cards though. Does anyone have a cisco rep contact they could ask more info regarding this module?
 

ccie4526

Member
Jan 25, 2021
98
68
18
Ok, now I'm really puzzled. I picked up a second DIMM of part number 36ASF4G72PZ-2G3B1, but my system does not like it. At all. CIMC inventory won't detect it, and system just won't boot.

So I went through the usual troubleshooting steps, and this just really blew my mind.... it's like the existing item is now "hard-programmed" to live in the one DIMM slot it was originally found in.... the system won't even detect a different DIMM even installed.

So now I'm wondering if that 4.13 NFVIS and the corresponding CIMC and BIOS updates (that lock you out of BIOS now) have effectively borked the ability to change/add hardware.

If anyone has the older CIMC and/or BIOS images, please drop me a DM. I was about to go drop this unit in a co-lo once I did the memory upgrade, and now I'm kinda stuck.... although I could deploy with 32GB....
 

devz3ro

New Member
Nov 16, 2019
22
2
3
Ok, now I'm really puzzled. I picked up a second DIMM of part number 36ASF4G72PZ-2G3B1, but my system does not like it. At all. CIMC inventory won't detect it, and system just won't boot.

So I went through the usual troubleshooting steps, and this just really blew my mind.... it's like the existing item is now "hard-programmed" to live in the one DIMM slot it was originally found in.... the system won't even detect a different DIMM even installed.

So now I'm wondering if that 4.13 NFVIS and the corresponding CIMC and BIOS updates (that lock you out of BIOS now) have effectively borked the ability to change/add hardware.

If anyone has the older CIMC and/or BIOS images, please drop me a DM. I was about to go drop this unit in a co-lo once I did the memory upgrade, and now I'm kinda stuck.... although I could deploy with 32GB....
I added:

ENCS54_BIOS_2.6.SPA
ENCS54_BIOS_3.06.SPA

To the place where we shared files with each other before.

If you need to get into the bios after upgrading to the latest bios here are the steps that let me in:

1. Flashed (via cmic + tftp) the ENCS54_BIOS_2.6.SPA after installing NFVIS 4.13, then went into the bios and set the password.
2. Power cycled (pulled power, counted to 20, plugged back in - important, else the bios password won't save once you upgrade back) the encs.
3. Went into the bios and changed something (anything) and saved and exited.
4. Flashed (via cmic + tftp) the ENCS54_BIOS_3.06.SPA bios back.
5. Power cycled (pulled power, counted to 20, plugged back in) then verified I could access the updated bios with the password I set.