$127 Cisco ENCS5412/K9 Xeon-D 1557 (12 core), 32G ram
- Thread starter RobstarUSA
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Well. Interesting.
So the CIMC chassis inventory still thinks this:
Memory
DIMMList:
id: 1
Name: DIMM_A
Capacity: 32768 MB
Speed: 2400
MemoryType: DDR4
MemoryTypeDetail: Synchronous
BankLocator: NODE 1
Manufacturer: Micron
SerialNumber: ##########
AssetTag: DIMM_A_AssetTag
PartNumber: 36ASF4G72PZ-2G3B1
Visibility: Yes
Operability: Operable
DataWidth: 72 bits
id: 2
Name: DIMM_B
Capacity: Not Installed
Speed: NA
MemoryType: NA
MemoryTypeDetail: NA
BankLocator: NA
Manufacturer: NA
SerialNumber: NA
AssetTag: NA
PartNumber: NA
Visibility: NA
Operability: NA
DataWidth: NA
But I finally was able to get the system to boot on a pair of 8GB DDR4 2133P DIMMs, and BIOS (2.6) reports them both.
So as best I can tell, the unit was failing POST with both of the 32GB DIMMs installed, and would only boot with the original 32GB DIMM. Thus, indicators point to a bad DIMM.
Sucks that Cisco have this box limited to 32GB DIMMs, I've got a shedload of 64GB DDR4 ECC DIMMs.... and yes, I tried to boot with one of those installed too... same problem, won't complete post.
Oh well.
Thanks @devz3ro for bailing me out on the BIOS thing.
So the CIMC chassis inventory still thinks this:
Memory
DIMMList:
id: 1
Name: DIMM_A
Capacity: 32768 MB
Speed: 2400
MemoryType: DDR4
MemoryTypeDetail: Synchronous
BankLocator: NODE 1
Manufacturer: Micron
SerialNumber: ##########
AssetTag: DIMM_A_AssetTag
PartNumber: 36ASF4G72PZ-2G3B1
Visibility: Yes
Operability: Operable
DataWidth: 72 bits
id: 2
Name: DIMM_B
Capacity: Not Installed
Speed: NA
MemoryType: NA
MemoryTypeDetail: NA
BankLocator: NA
Manufacturer: NA
SerialNumber: NA
AssetTag: NA
PartNumber: NA
Visibility: NA
Operability: NA
DataWidth: NA
But I finally was able to get the system to boot on a pair of 8GB DDR4 2133P DIMMs, and BIOS (2.6) reports them both.
So as best I can tell, the unit was failing POST with both of the 32GB DIMMs installed, and would only boot with the original 32GB DIMM. Thus, indicators point to a bad DIMM.
Sucks that Cisco have this box limited to 32GB DIMMs, I've got a shedload of 64GB DDR4 ECC DIMMs.... and yes, I tried to boot with one of those installed too... same problem, won't complete post.
Oh well.
Thanks @devz3ro for bailing me out on the BIOS thing.
Same seller has 6 more available at $127 for the next three days...
Cisco ENCS5412/K9 Compute System 5412 Virtualization Appliance -Xeon D-1557 32GB | eBay
Cisco ENCS5412/K9 Compute System 5412 Virtualization Appliance -Xeon D-1557 32GB | eBay
Found this regarding the raid card. Pictures are small, but they show the before and after of the installation.
" 4. Align the RAID Controller card with the contact-PCIe pins and screw holes."
" 4. Align the RAID Controller card with the contact-PCIe pins and screw holes."
Install and Troubleshoot Cisco ENCS/c8300 RAID/JBOD Configurations
This document describes the JBOD and RAID methods for managing multiple hard drives on the ENCS and c8300 UCPE 1N8 devices.
www.cisco.com
Some reverse engineering info about the switch:
It seems the switch ASIC is a Marvell Prestera (98DX3***), it has its own ARM CPU that is PCIe mapped, the linux driver as @turbo said some time ago is
1. probe PCIe device
2. Find which
3. write
4. send an IRQ to wake up CPU.
5. write switch_firmware.bin some kind of linux image for u-boot
6. Wait for the kernel to boot
7. Wait for the main app, called ROS to boot
8. setup a new network
It seems the switch ASIC is a Marvell Prestera (98DX3***), it has its own ARM CPU that is PCIe mapped, the linux driver as @turbo said some time ago is
mv_pciboot will probe and map the Switch ASIC register into its BAR and expose a block device /dev/servicecpu. The swtich boot process involves /opt/switch-confd/remote_boot_app communicating with /dev/servicecpu and following a precise sequence governed by a state machine.1. probe PCIe device
2. Find which
/sys/bus/pci/devices/%s/config belongs to the switch and getting the BAR mapping3. write
booton.bin into the device and wait for a proper status change. (booton.bin is an ARMv7 bootloader, probably uboot-marvell)4. send an IRQ to wake up CPU.
5. write switch_firmware.bin some kind of linux image for u-boot
6. Wait for the kernel to boot
7. Wait for the main app, called ROS to boot
8. setup a new network
ifconfig int-LAN.2363 169.254.1.1 netmask 255.255.0.0 up so XML configuration payloads can be sent there.Another quick update about auto media sensing this time.
I just installed proxmox in one of the units I have, install was uneventful using MGMT interface (enp15s0), then I dived into the dual personality ports (GE0-0/enp2s0f0 and GE0-1/enp2s0f1)
Upon boot I checked the
To double check I also dumped the ethernet adapter EEPROM using
So next I proceeded to patch the GE0-0 RJ45 port to my switch and to my surprise it wasn't detected matching what other members have observed in this thread, the kernel didn't log anything when connecting the port. Then it occurred to me, in NFVIS all interfaces are set to administrative UP while proxmox will only up the interfaces that are configured so a quick
Success! For completeness I connected a random Mikrotik SFP that I have on GE0-1/enp2s0f1 and tried to dump its data:
unfortunately my SFP even though it is read by the card slot doesn't seem to work with either NFVIS or proxmox, and I don't have any other SFPs to try over here so I might just park this investigation for the time being. There might be some vendor locking associated but in that case I suspect it is baked into the EEPROM somehow See this thread
I just installed proxmox in one of the units I have, install was uneventful using MGMT interface (enp15s0), then I dived into the dual personality ports (GE0-0/enp2s0f0 and GE0-1/enp2s0f1)
Upon boot I checked the
igb module output and observed that MAS (media auto sense) was set up correctly and detected by the kernel.
Code:
[ 1.871281] igb: Intel(R) Gigabit Ethernet Network Driver
[ 1.871287] igb: Copyright (c) 2007-2014 Intel Corporation.
[ 2.570673] igb 0000:02:00.0 eth2: MAS: Enabling Media Autosense for port 0 <== **** THIS ****
[ 2.570743] igb 0000:02:00.0: added PHC on eth2
[ 2.570755] igb 0000:02:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 2.570757] igb 0000:02:00.0: eth2: (PCIe:5.0Gb/s:Width x4) 2c:4f:52:xx:xx:xx
[ 2.570832] igb 0000:02:00.0: eth2: PBA No: 106700-000
[ 2.570834] igb 0000:02:00.0: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[ 2.935230] igb 0000:02:00.1 eth4: MAS: Enabling Media Autosense for port 1 <== **** THIS ****
[ 2.935303] igb 0000:02:00.1: added PHC on eth4
[ 2.935316] igb 0000:02:00.1: Intel(R) Gigabit Ethernet Network Connection
[ 2.935317] igb 0000:02:00.1: eth4: (PCIe:5.0Gb/s:Width x4) 2c:4f:52:xx:xx:xx
[ 2.935392] igb 0000:02:00.1: eth4: PBA No: 106700-000
[ 2.935394] igb 0000:02:00.1: Using MSI-X interrupts. 8 rx queue(s), 8 tx queue(s)
[ 2.964070] igb 0000:0f:00.0: added PHC on eth5
[ 2.964097] igb 0000:0f:00.0: Intel(R) Gigabit Ethernet Network Connection
[ 2.964099] igb 0000:0f:00.0: eth5: (PCIe:2.5Gb/s:Width x1) 2c:4f:52:xx:xx:xx
[ 2.964143] igb 0000:0f:00.0: eth5: PBA No: 000300-000
[ 2.964145] igb 0000:0f:00.0: Using MSI-X interrupts. 4 rx queue(s), 4 tx queue(s)
[ 3.310372] igb 0000:02:00.0 enp2s0f0: renamed from eth2
[ 3.458619] igb 0000:02:00.1 enp2s0f1: renamed from eth4
...ethtool -e enp2s0f0 and checked that word 3 of it, byte offset 0x7, matched the proper media auto sense enable bits. [i350 Datasheet] section 6.4.1 up to this point everything pointed that the adapter should work fine.So next I proceeded to patch the GE0-0 RJ45 port to my switch and to my surprise it wasn't detected matching what other members have observed in this thread, the kernel didn't log anything when connecting the port. Then it occurred to me, in NFVIS all interfaces are set to administrative UP while proxmox will only up the interfaces that are configured so a quick
ip link set dev enp2s0f0 up and...
Code:
[ 2134.333530] igb 0000:02:00.0 enp2s0f0: MAS: changing media to copper
[ 2134.333538] igb 0000:02:00.0 enp2s0f0: Reset adapter
[ 2137.965854] igb 0000:02:00.0 enp2s0f0: igb: enp2s0f0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX
Code:
# ethtool enp2s0f0
Settings for enp2s0f0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
MDI-X: on (auto)
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
Code:
# ethtool -m enp2s0f1
Identifier : 0x03 (SFP)
Extended identifier : 0x04 (GBIC/SFP defined by 2-wire interface ID)
Connector : 0x07 (LC)
Transceiver codes : 0x00 0x00 0x00 0x08 0x00 0x00 0x00 0x00 0x00
Transceiver type : Ethernet: 1000BASE-T
Encoding : 0x01 (8B/10B)
BR, Nominal : 1300MBd
Rate identifier : 0x00 (unspecified)
Length (SMF,km) : 0km
Length (SMF) : 0m
Length (50um) : 0m
Length (62.5um) : 0m
Length (Copper) : 100m
Length (OM3) : 0m
Laser wavelength : 0nm
Vendor name : Mikrotik
Vendor OUI : 20:20:20
Vendor PN : S-RJ01
Vendor rev : 1.0
...Thanks for confirming that MAS works with the Linux igb driver, I started digging into it but the ESXi driver does not seem to have any support for this and it was easier to use a copper GBIC than continue messing with it. I used the cheapest SFP I could get quickly (specifically this one) and it works fine in ESXi. I tried running
ethtool -m to see what gets reported but it comes back with netlink error: I/O error on my system (alpine linux)Were you able to install Proxmox to the M.2? Also did you have any issues accessing the 2 SATA 2.5" drives? I work with these boxes on a regular basis at work and love the hardware but HATE the NFVIS software. Would love to be able to turn one into a beefy little Proxmox machine with tons of ports.
Also for anyone wondering the ENCS 5400 series does NOT in any way support ANY 10G interfaces (as confirmed by several cisco sales reps and engineers). I'm curious though about what modules proxmox or vmware actually detect since the NFVIS just feeds these through to the Router VM (presumably as a PCI passthrough).
Here's the list of supported NIM Modules for those who care. The 4G LTE modules can be found on ebay for less than $20 and would make an awesome backup WAN circuit.
Table 4. Supported NIMs
| NIM | Product Module | Minimum Software |
|---|---|---|
| LTE | NIM-4G-LTE-VZ | NFVIS 3.6.1 ISRv 16.6.1 |
| NIM-4G-LTE-ST | ||
| NIM-4G-LTE-NA | ||
| NIM-4G-LTE-GA | ||
| NIM-4G-LTE-LA | ||
| NIM-LTEA-EA | ||
| NIM-LTEA-LA | ||
| T1/E1 Data | NIM-1MFT-T1/E1 | NFVIS 3.6.1 ISRv 16.6.1 |
| NIM-2MFT-T1/E1 | ||
| NIM-4MFT-T1/E1 | ||
| NIM-8MFT-T1/E1 | ||
| NIM-1CE1T1-PRI | ||
| NIM-2CE1T1-PRI | ||
| NIM-8CE1T1-PRI | ||
| Asynchronous | NIM-16A | NFVIS 3.8.1 ISRv 16.8.1 |
| NIM-24A | ||
| T1/E1 Voice | Same as T1/E1 Data PID List | NFVIS 3.9.1 ISRv 16.9.1 |
| DSL | NIM-VA-B | NFVIS 3.9.1 ISRv 16.10.1 |
| NIM-VAB-A | ||
| NIM-VAB-M | ||
| GE | NIM-1GE-CU-SFP | NFVIS 3.9.1 ISRv 16.9.1 |
| NIM-2GE-CU-SFP |
Has that been independently validated? I work for a similar industry OEM and its pretty frequent that we will say something is not supported due to compatibility or testing guidelines but is found to be working by users that just have the ability to tinker. I was willing to give it a shot myself except I cannot find that NIM anywhere for an "affordable" price.
We haven't actually tried it but in doing the research and speaking with Cisco it's just not possible. I can confirm that earlier versions of the NFVIS software didn't even work with the 2 port gigabit NIMs.
From what I could tell with some brief RE of an old NFVIS it looks like the NIM gets connected to one of the 10Gb ports on the internal Marvell switch. I have been wanting to test this also, but I have not found a 10Gb NIM for anything close to a reasonable price to test. It seems possible that it may work, even if it doesn't work with NFVIS, or if it's unsupported.
I use some of the Marvell switch ports on the ENCS in a LAG with my main switch and that has been good enough for what I'm using it for.
I use some of the Marvell switch ports on the ENCS in a LAG with my main switch and that has been good enough for what I'm using it for.
Interesting... Functionally I assumed it would act like a PCI Passthrough since the NM-2G modules appear as if they are plugged into the ISR1000v directly. Either way I would assume it all comes down to whether or not the ENCS's NIM slot can functionally support the NM-10G. Could be a simple issue of the proprietary PCIe interface being x1 vs x4. Not sure if CISCO publishes anything about how that interface works.
From looking at the intel spec site the processor does have a limit of 32 PCIe lanes so it's quite possible that these two NIMs are only x1 and that is where the limitation on NIM compatibility comes from.
Last edited:
Correct! I installed proxmox v8.1 without almost changing anything just mapped the CD drive to a proxmox ISO and followed the install prompts. I haven't tested any of the SATA ports yet but I should have some low capacity 2.5 SSDs laying around.
I'm right now mostly focusing on getting the switch to bootstrap correctly with proxmox.
Code:
[79138.032478] ata1: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[79138.032711] ata1.00: ATA-7: SSDSA2SH032G1SB INTEL, 845C8862, max UDMA/133
[79138.032716] ata1.00: 62500000 sectors, multi 1: LBA48 NCQ (depth 31)
[79138.032990] ata1.00: configured for UDMA/133
[79138.033154] scsi 0:0:0:0: Direct-Access ATA SSDSA2SH032G1SB 8862 PQ: 0 ANSI: 5
[79138.033582] sd 0:0:0:0: Attached scsi generic sg7 type 0
[79138.033662] sd 0:0:0:0: [sdf] 62500000 512-byte logical blocks: (32.0 GB/29.8 GiB)
[79138.033688] sd 0:0:0:0: [sdf] Write Protect is off
[79138.033694] sd 0:0:0:0: [sdf] Mode Sense: 00 3a 00 00
[79138.033724] sd 0:0:0:0: [sdf] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[79138.033762] sd 0:0:0:0: [sdf] Preferred minimum I/O size 512 bytes
[79138.036111] sdf: sdf1 sdf2 sdf3 sdf4
[79138.036388] sd 0:0:0:0: [sdf] Attached SCSI diskIt also seems like CISCO doesn't make ANY 10G NIM modules that would fit this slot. It is the same as the list of NIMs for the 4400 series routers for the most part (at least a subset). Any of the "double wide" (SM-X) interfaces obviously won't work (which includes the 4400's 10 Gig interface... which is hilarious since the 4400 isn't capable of actually routing anywhere close to 10G).
Cisco 4000 Series Integrated Services Routers - Interfaces and Modules
There are a few on that list that are curious as they are the same form factor but not listed as compatible with the ENCS. I'm actually kind of curious if the 8 port switch module and the dual SATA slots are just the same as the ones that are built in on the ENCS. And if so whether you could use them as well.
Another interesting one is the UCS modules. The UCS-EN140N-M2 uses the same NIM interface on the 4400's. Wondering if you could just plug one into the slot on the 5412 and have two nodes in one box.
Which 10G NIM were you looking at?
I was looking at the C-NIM-1X part which is a single-port SFP+ module that looked like ti would physically fit - but it is meant for the Catalyst 8300 series which is the successor to this model (I think).
You are correct the 8300 is what Cisco is guiding us to as a successor for our 5400's at work. From a quick scan of the compatibility matrix for the 8300 I do see some of the same modules that the 5400 works with. That hints that the physical slot should fit. The big question is electronically. As I've said above if they are just custom PCIe interfaces and the interface on this 5400 is just a 1x vs the 8300 having a 4x or 8x or something that could be an issue. The other question is power Cisco isn't very forthcoming on the specs for these modules or the chassis interfaces I doubt a single SFP would really use anything more than basic bus power though. I agree though the $2K price tag is a little too much just to "test".
So one (well, two...) of the supported NIMs in the ENCS5412 platform are the NIM-1GE-CU-SFP and NIM-2GE-CU-SFP. Has anyone considered trying one of those in their unit to see if it's recognized by any OS other than NFVIS? Seems like that would be a heck of a lot less expensive (Cisco NIM-1GE-CU-SFP 1-Port Gigabit Ethernet for ISR 4300/4400 - 1 Year Warranty | eBay) a way to test if NIMs are even recognized instead of trying something that's not on the books as being supported... especially if someone has one lying around in their inventory somewhere (I don't, I checked).
Last edited:
Update on getting the switch online after boot.
Guide to setup a bootstrapping VM: Switch bootstrap VM | Notion
Python package for injecting configuration: GitHub - yeyus/marvell-11abbe00: marvell switch configuration
Guide to setup a bootstrapping VM: Switch bootstrap VM | Notion
Python package for injecting configuration: GitHub - yeyus/marvell-11abbe00: marvell switch configuration