You are absolutely awesome! I have seen snippets of this setup elsewhere but never a complete guide or reference. Now to bring the appliances back to the bench!
$127 Cisco ENCS5412/K9 Xeon-D 1557 (12 core), 32G ram
- Thread starter RobstarUSA
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
Does anyone have access to the latest NFVIS updates they could share privately? https://software.cisco.com/download/home/286308693/type/286309317/release/4.15.5
It's been a long thread and after catching up, it seems that NFVIS isn't really necessary? I seems that some people installed baremetal linux and some loaded Cisco branded ESXi 8? But NFVIS has some magic stuff that makes the switch works? (Although I don't really plan on using this for routing.)
My docker server just went offline this morning and I can't power it back up again. Power supply probably went bad. So it's time to upgrade to this bad boy.
It's been a long thread and after catching up, it seems that NFVIS isn't really necessary? I seems that some people installed baremetal linux and some loaded Cisco branded ESXi 8? But NFVIS has some magic stuff that makes the switch works? (Although I don't really plan on using this for routing.)
My docker server just went offline this morning and I can't power it back up again. Power supply probably went bad. So it's time to upgrade to this bad boy.
I loaded the Cisco ESXI 7.0.3 image on mine, just not using the 8 switchports.
Gotta be careful with loading the newer NFVIS, images, they will update (and lock down) the BIOS where you can't F2 into it during bootup, you can only do BIOS stuff through the CIMC interface.
Check a certain cios dot dhitechnical site for the latest NFVIS.
Gotta be careful with loading the newer NFVIS, images, they will update (and lock down) the BIOS where you can't F2 into it during bootup, you can only do BIOS stuff through the CIMC interface.
Check a certain cios dot dhitechnical site for the latest NFVIS.
It's possible to roll back, I did it with mine.... would that I could remember how I did it now.... I've had it in a co-lo for the last two years, so that gives you an idea how long ago I did it. There may still be notes in this thread on how to do it.
You can downgrade. Surprisingly Cisco makes it fairly easy to switch back and forth between versions. The one caveat is that while you can upgrade without re-imaging the NFVIS you will have to wipe the system clean to downgrade. Also downgrading will not back down the CIMC and BIOS versions but they SHOULD be able to be downgraded via the CIMC but I've not tested this a whole lot. Also any NFVIS version after 4.15.X will disable the Marvell switch completely. Everything else still seems to work fine but you'll loose the switch entirely... Even though at one point cisco had the newer software (Up to 4.18.2) on the download page specifically for the 5400 ENCS. Don't ask how I found THAT little tidbit out.
I've been using proxmox on mine since I got it. We use NFVISon the systems I have at my work (including the replacement 8300-UCPE chassis) and I HATE IT. I describe it as cisco handed KVM and linux to a bunch of engineers and said "Make this unusable... and don't document anything about how it's supposed to work."
That being said you are correct you will loose the 8 port switch module and any access to the NIM Modules, but you gain full access of the management port, both of the built in SFP/RJ45 ports (you can only use one media type or the other and it switches to the SFP if one is inserted). You'll also see 2 x710 and 1 x552 10G port, the x710's are wired to the switch (if memory serves) and I have NO clue what the x552 is intended for but it's a backplane connection of some sort.
Proxmox has been VERY stable for me. I've had it up and running for a couple years with no issues. fan throttling seems to work ok and if you upgrade the ram (I went with 64G total) you end up with a pretty capable platform for hosting routing VMs, docker hosts and whatever storage based servers (NAS/nextcloud/immich etc). The lack of any real functional video card makes it a little weak for hosting a media server like plex/jellyfin unless you aren't planning on doing any transcoding etc. The storage is somewhat limited with one internal m.2 and 2 2.5" drive bays. If I remember correctly too there's a size limitation on the 2.5" I think it's a phsyical thing though like only 10mm or thinner. The M.2 slot is a regular NVME type not a SATA but due to the placement and cooling I would be careful what you put in there. I generally use mine as a boot drive for Proxmox and an internal storage for ISOs and Images etc. I have 2x 1TB HDDs installed in the 2.5" bays configured as a ZFS pool under proxmox there is an optional hardware raid card available that mine didn't come with but honestly for proxmox it's not necessary since ZFS prefers to manage the individual drives directly anyway. I can't attest to anything VMWare but I would assume the capabilities aren't much different. I don't see Cisco providing VMware/Broadcom with any of the proprietary drivers or software (and from what others on here have reported they didn't) That being said I'd have to see some kind of side by side comparison to prove that there is any benefit to one platform or another. Although Turbo did have some better luck than I did with getting an NFVIS VM to boot and recognize the switch module. Which may or may not be of interest. The other method described in this thread works but it's FAR from simple and definitely doesn't end up with anything I'd call convenient for configuration and management of the switch.
thanks, unfortunately their juniper ex files are old, the search continuesEver tried viewing the html after you fail login?
Reaching out to all.
Got 5412, what a machine. Added 64gb - great.
Installed ESXi 7 on it (big vcenter cluster here).
But been at IT whole week and still can't fire up that switch.
So far:
1. tried vm method. basically remote-bootd starts, I see both x710 interfaces become up/connected in esxi ui, heck I can even see CDP data from those ports indicating that they are at link-local 169.254.1.0 as expected. But whatever I did (passhru, srvio, portgrp 0, 2363, 4095), I can't push xml to endpoint. It just never responds (or arp even). Maybe it's esxi's quirk?
2. turned to @turbo method - installing NFVIS as vm. Well. Out of about 25 attempts it installed thru only once. Booted but no switch action whatsoever.
I need help.. Maybe someone can just share a working bootstrap VM for esxi...
Thanks, all
Got 5412, what a machine. Added 64gb - great.
Installed ESXi 7 on it (big vcenter cluster here).
But been at IT whole week and still can't fire up that switch.
So far:
1. tried vm method. basically remote-bootd starts, I see both x710 interfaces become up/connected in esxi ui, heck I can even see CDP data from those ports indicating that they are at link-local 169.254.1.0 as expected. But whatever I did (passhru, srvio, portgrp 0, 2363, 4095), I can't push xml to endpoint. It just never responds (or arp even). Maybe it's esxi's quirk?
2. turned to @turbo method - installing NFVIS as vm. Well. Out of about 25 attempts it installed thru only once. Booted but no switch action whatsoever.
I need help.. Maybe someone can just share a working bootstrap VM for esxi...
Thanks, all
I only tried ESXi 8, but make sure you set this in the vmx file before installing NFVIS. If the switch is already bootstrapped, passing through sriov should work. It needs to be tagged VLAN 2363, so don't set the portgroup to 2363 in VMWare
SMBIOS.reflectHost = "TRUE"Hey turbo. I dm'ed you. I'm not a complete idiot, of course I added that key. With that key iso fails to install (hangs). So I installed without it, added it afterwards. No switch action. Pls dm me. Switch is NOT bootstrapped. Help please.
int-ngio isn't populated (no udev match, even with editing it). sysconfig/network-scripts got a lot of messy crap put there by installer figuring out it's not on realhw interfaces? Really. 5th day and counting, non stop...
int-ngio isn't populated (no udev match, even with editing it). sysconfig/network-scripts got a lot of messy crap put there by installer figuring out it's not on realhw interfaces? Really. 5th day and counting, non stop...
It's been fun playing around with this thing. Currently rolling Promox (8.4, 9.2 installer doesn't seem to work with this server) with a customized almalinux image with the switch passthrough (based on NFVIS 4.15.5). PoE works and I'm creating a TUI to manage the switch. I'll post the repo soon.
is there a 4.15.6? I would be interested for a copy of that iso if anyone has it? the normal cios place doesn't have it yet.
EDIT: Repo is ready. GitHub - foureight84/encs5400-almalinux will require a 4.15.x NFVIS iso to build a custom Almalinux ISO and qcow2 for proxmox2. The ISO/qcow2 image will have the management TUI and there are instructions on how to install it to the Proxmox host. You should have control of PoE on the switch through the TUI. Thanks to @yeyus for the write up that made this a lot easier to make.
is there a 4.15.6? I would be interested for a copy of that iso if anyone has it? the normal cios place doesn't have it yet.
EDIT: Repo is ready. GitHub - foureight84/encs5400-almalinux will require a 4.15.x NFVIS iso to build a custom Almalinux ISO and qcow2 for proxmox2. The ISO/qcow2 image will have the management TUI and there are instructions on how to install it to the Proxmox host. You should have control of PoE on the switch through the TUI. Thanks to @yeyus for the write up that made this a lot easier to make.
Last edited:
Ok got it working... can confirm 4.15.6 does work. I do have a few questions....
1) How do you add devices to the LAGs?
2) I See that TE3&4 say something about the NIM. From everything I assumed about the NIM and the way it works with the actual cisco NFVIS (NGIO) I was under the assumption/impression that it was some sort of cisco proprietary PCIe Passthrough deal. Do you know if this exposes the two NIM ports to the switch somehow or is this just some sort of backside thing to communicate internally with the NIM module?
1) How do you add devices to the LAGs?
2) I See that TE3&4 say something about the NIM. From everything I assumed about the NIM and the way it works with the actual cisco NFVIS (NGIO) I was under the assumption/impression that it was some sort of cisco proprietary PCIe Passthrough deal. Do you know if this exposes the two NIM ports to the switch somehow or is this just some sort of backside thing to communicate internally with the NIM module?