$127 Cisco ENCS5412/K9 Xeon-D 1557 (12 core), 32G ram

[peramus]

On my ESXi NFVIS VM I have 4 E1000e interfaces plus the passed through X710 VFs. I bootstrap the switch with a different VM before NFVIS starts because I have other VMs using the internal switch and I didn't want to depend on NFVIS to do it.

The interface mapping happens in /etc/udev/rules.d/70-persistent-net.rules, I think I did change this to map to the MAC address instead of using the PCI ID because that changed for some reason when I was playing with different options. something like this

ACTION=="add", SUBSYSTEM=="net", ATTR{address}=="00:0c:29:xx:xx:xx", NAME="MGMT"

the real SSH port is 22222 at least on my NFVIS version. You have to clear the fw rules before it will respond though (iptables --flush works)

dmidecode reports this inside the NFVIS VM in case this helps to replicate it

Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
        Vendor: VMware, Inc.
        Version: VMW201.00V.20829224.B64.2211211842
        Release Date: 11/21/2022
        ROM Size: 2048 kB
        Characteristics:
                ISA is supported
                PCI is supported
                PNP is supported
                BIOS is upgradeable
                ACPI is supported
                Targeted content distribution is supported
                UEFI is supported

Handle 0x0001, DMI type 1, 27 bytes
System Information
        Manufacturer: Cisco Systems, Inc.
        Product Name: ENCS5412/K9
        Version: None
        Serial Number: FGLxxxxx07K
        UUID: xxxxxxxxx-xxxx-xxxx-xxxx-881e6fda766d
        Wake-up Type: Power Switch
        SKU Number: Not Specified
        Family: Not Specified

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
        Manufacturer: Intel Corporation
        Product Name: 440BX Desktop Reference Platform
        Version: None
        Serial Number: None
        Asset Tag: Not Specified
        Features: None
        Location In Chassis: Not Specified
        Chassis Handle: 0x0000
        Type: Other
        Contained Object Handles: 0

Handle 0x0003, DMI type 3, 21 bytes
Chassis Information
        Manufacturer: No Enclosure
        Type: Other

I did find a file in the root of the ISO called "Fake_dmidecode_data.bin" that appears to match what I'm getting on my VM. It appears that Proxmox doesn't have the option to blindly passthrough the hosts DMI data. But you can manually enter it... Only issue seems to be that the config file doesn't like the fact that Cisco put a comma in the Vendor name. Hoping it doesn't actually care.

 

[peramus]

I haven't been able to find any cisco "ISP" raid cards though. Does anyone have a cisco rep contact they could ask more info regarding this module?

BTW, do any of you guys know about the internal port on these? I'm interested in putting a raid card inside it if it's available to buy relatively cheap on somewhere like eBay. A card with mini sas would be ideal.

I actually have one of the cards but I've not been able to find anything else on ebay. The part number is 74-104764-01-A0 (The 6 MIGHT be a 5 it's kinds small and hard to read... Also the 01 might be D1 and A0 might be AD)

 

[foureight84]

There's no way to use a NIM from another model in this thing, huh? Are the NIM units proprietary from one model to the next?

 

[ccie4526]

There's no way to use a NIM from another model in this thing, huh? Are the NIM units proprietary from one model to the next?

They're not proprietary between models as much as whether the software support for the module is written into the code for the device. The NIMs can be used in any of the ISR4K routers because the code is written into IOS-XE to support them. The code for the ENCS is more limited, certain modules "drivers" are written into the NFVIS software, and that is how some of those modules are available under NFVIS, but not all of them. And if you use some software other than NFVIS on the ENCS, then the proprietary drivers are definitely not there.

 

[peramus]

So from doing a little research I found something (albeit unnoffical) that indicates that NFVIS seems to use IOS XE as a control plane which is probably how it talks to the NIMs as under normal operation under NFVIS they never appear to the HOST OS and present as if physically connected to the router VM.

As for the NIMS yes they are semi-specific. Usually they have a specific form factor across similar models from the same time period. They have changed over time (though what exactly is different from one generation of router to the next is hard to get from Cisco) While most NIMs that use the same physical format are compatible across similar platforms occasionally one device does not support a specific NIM that other similar devices do. Whether this is due to something inherent with the system (Think PCI Express lanes or version etc) or just something abstract where Cisco chose not to add drivers or some capability is another mystery sadly. Also assuming that no matter how a NIM "connects" to the system it still needs PCI bandwidth to communicate there's not a lot "leftover" after all the internal systems take what they need.

But all of that is kind of a moot point if we are no longer running Cisco's Proprietary NFVIS software on them. Since as of right now I cannot seem to see the module at all on the PCI bus. My earlier assumption was that all they were was fancy PCIe cards with a proprietary physical format. That doesn't seem to be the case. I currently have a NIM-1G-CU-SFP (one that I KNOW works under NFVIS) plugged into mine and I cannot see it at all under proxmox... I cannot even see the device showing upwhen I do an "lspci". So since Cisco doesn't want it to work under NFVIS and they defnitley don't care if it works on Proxmox or VMWare (or any other linux based OS) then they presumably do not publish the drivers to MAKE anything work (PLEASE someone prove me wrong and find the RPM on the NFVIS ISO Disk!!).

 

[rocketpanda40]

Well, unfortunately work's gotten stupid and I'm not gonna have any time to mess with this 8200 after all.

I threw it back on eBay – as much as I hate eBay as a seller – for a higher price just to see what happens. But if anyone on here wants it, happy to give it to you for what I paid for it.

 

[turbo]

If you need to reset the CIMC password you can do the following:
Connect to the CIMC serial interface and boot up the system.
When it prompts, press *** press enter and run boot current recovery.
Once it reaches the login go ahead and login as root (no password) and run the following:
mount /dev/cydisk0a3 /mnt/jffs2
mv /mnt/jffs2/avct_ems_cfg/etc/userdb /mnt/jffs2/avct_ems_cfg/etc/userdb_old
umount /dev/cydisk0a3
Reboot the system and it will reset the credentials to default aka admin/password.

FWIW this does not work on my updated CIMC. Cisco's update procedure to login with admin/password then doing emmc format p3 does still work

 

[turbo]

I have been trying to play with getting a NIM-LTEA-EA to work, but it does not appear on lspci or lsusb at all. Using the /opt/nic-pkg/nim-info command on NFVIS the box definitely SEEs the NIM, with the MAC address S/N etc. Poking around in CIMC logs at some point I could see a log entry where the CIMC was seeing the NIM, and switching some type of MUX from BMC to HOST.

I can't find any way to configure it through the web interface on NFVIS even though it's supposedly supported.

Going through some of the scripts, it communicates with the NIM through OS2BMC and using int-ngio, which is the first 10G port connected to the internal switch. Some protocol over ethernet framing? I have not been able to capture any data so far.

 

[ccie4526]

Curious if anyone has tried dropping a SAS SSD or HDD into their 5412 unit. The specs imply SAS support, but everything I've got in current inventory is SATA, so haven't been able to confirm. Some of these "cheap" SAS SSDs coming available might be interesting to add to these boxes.

Edit to add: Read back in the chain, and see that it was attempted. Apparently the addon RAID controller is required to enable SAS functionality. Bleh.

 

[autoturk]

if anybody needs any of these, I have a couple with hard drive trays in my garage that I'd like to get rid of. $60 each if you can pick up in the SF Bay Area (east bay), add shipping on top if you want it shipped.

 

[cwellman]

For anyone here who is using ESXi and just using an NFVIS VM for switch config? How is it setup exactly? I have NFVIS installed in a VM and have the Marvell switch passed through to it, is that all or do either of the X710 also need to be passed through?

 

[peramus]

For anyone here who is using ESXi and just using an NFVIS VM for switch config? How is it setup exactly? I have NFVIS installed in a VM and have the Marvell switch passed through to it, is that all or do either of the X710 also need to be passed through?

I’ve tried with proxmox to pass through the marvel switch (you want to save the x710’s for connectivity to VMs) but I got stuck trying to force the NFVIS to think it was actually on bare metal otherwise it installs and does not enable functionality for the marvel switch or any NIM modules. Also proxmox will not detect a known compatible NIM modules (I tried a 1G sfp unit we use on our fielded 5412’s at work). So sadly unless someone can figure out how to extract the necessary components from the actual NFVIS installation and come up with a basic interface the closest thing that seems to work is the previously mentioned instructions to built a VM which leaves you with a basic switch config unless you can edit the confusing XML that Cisco uses to provision them.

TLDR, NFVIS on a VM KNOWS it’s on a VM and won’t talk to the switch.

 

[turbo]

For anyone here who is using ESXi and just using an NFVIS VM for switch config? How is it setup exactly? I have NFVIS installed in a VM and have the Marvell switch passed through to it, is that all or do either of the X710 also need to be passed through?

Yes, this is possible. Pass through both the marvel switch and both X710 and It should work.

I would have sworn I documented this detail but I couldn't find it in this thread. The key for making NFVIS work under ESXi for me was adding the advanced setting board-id.reflectHost = "TRUE" in the VMX file. I don't know what the equivalent would be for Proxmox?

Once you have it working, you should find that you can pass through a VF of the XL710s and it will still work. If you replace the fake DMI info file to match the board I would expect that to work also.

 

[peramus]

Yes, that would be the likely fix... Proxmox does not have anything like this and I tried playing with different ways to manually edit the SMBIOS info (Takes me back to my old Hackintosh days) to no avail. So sounds like VMWare finally has a leg up over Proxmox on something.

 

[oneplane]

Yes, that would be the likely fix... Proxmox does not have anything like this and I tried playing with different ways to manually edit the SMBIOS info (Takes me back to my old Hackintosh days) to no avail. So sounds like VMWare finally has a leg up over Proxmox on something.

For Proxmox that's a combination of using data from the DMI or host ACPI tables, and if required, host CPU model. It's effectively just a string of text for the SMBIOS API so when the VM loads you can spoof what it is running on.

An easy way in the GUI is to click on edit SMBIOS. If you don't know the current values, dmidecode --type 1 will give you that.

 

[peramus]

For Proxmox that's a combination of using data from the DMI or host ACPI tables, and if required, host CPU model. It's effectively just a string of text for the SMBIOS API so when the VM loads you can spoof what it is running on.

An easy way in the GUI is to click on edit SMBIOS. If you don't know the current values, dmidecode --type 1 will give you that.

Precisely the rabbit hole I went down (Had to do some similar tricks to get a hacked OSX working on proxmox before. But there's SOMETHING that the NFVIS recognizes as QEMU and switches to it's "I'm running as a virtual machine" mode. Which still works for 90% of the functions but sadly it does not show the host switch no matter what you pass through to it.

 

[oneplane]

Precisely the rabbit hole I went down (Had to do some similar tricks to get a hacked OSX working on proxmox before. But there's SOMETHING that the NFVIS recognizes as QEMU and switches to it's "I'm running as a virtual machine" mode. Which still works for 90% of the functions but sadly it does not show the host switch no matter what you pass through to it.

Strange, it did work on my ENCS5412. I don't have it anymore but I do remember that I just ended up dumping the tables and importing them since I didn't really care about which details to keep/discard.

 

[peramus]

Strange, it did work on my ENCS5412. I don't have it anymore but I do remember that I just ended up dumping the tables and importing them since I didn't really care about which details to keep/discard.

There's a distinct possibility I missed a step somewhere. Might have to mess around with it again when I get some free time. Would be nice to use those 8 ports plus the SR-IOV ports.

 

[Navy_BOFH]

Strange, it did work on my ENCS5412. I don't have it anymore but I do remember that I just ended up dumping the tables and importing them since I didn't really care about which details to keep/discard.

There's a distinct possibility I missed a step somewhere. Might have to mess around with it again when I get some free time. Would be nice to use those 8 ports plus the SR-IOV ports.

Any chance y'all were able to get this working? I bought two of these to use as lab appliances but really wanted to get the switches working as well. They'd turn into a perfect "mobile lab" then.

 

[yeyus]

There you go I remember writing about the SMBIOS config Switch bootstrap VM | Notion