Dell VEP/VMWare Edge/Velo Cloud SD-WAN/VeraCloud VEP1400/VEP1400-X firewall units

[paykuh]

You already got it working, but if you press the button on the PCB closes to then power plug long enough, the device will power off.
When powered on again, all interfaces will be available.

This doesn't seem to work for me. I acquired 2 Dell VEPs from scrap when I worked at Dell from one of the labs, so I am wondering if I have some sort of experimental build. Reading thru this thread, it seems my units are completely different from everyone else. Mine are just branded "DELL EMC" with no Model #, none of the firmware updates from Dell's website work on it, and also, those 2 buttons people are pressing that are by the power button... on mine, the buttons on are on the opposite part of the board, near the SFP ports, pressing the one closest to the SFP port does nothing, and the one furthest away from the SFP port just reboots the module.

In OPNsense, I am only able to see ix0 and ix1, which are the SFP ports. I have RJ45 SFPs in them, and when I connect either of them to my PC I get a link light, but when I connect them to my WAN / switch I get no link. So I'm assuming something odd is happening with speed negotiating. Any who, I'll probably end up tinkering with them for a few more hours. I've spent countless hours already trying to get them to work already.. they'll probably end up on eBay by the end of the week.

 

[nmpu]

This doesn't seem to work for me. I acquired 2 Dell VEPs from scrap when I worked at Dell from one of the labs, so I am wondering if I have some sort of experimental build. Reading thru this thread, it seems my units are completely different from everyone else. Mine are just branded "DELL EMC" with no Model #, none of the firmware updates from Dell's website work on it, and also, those 2 buttons people are pressing that are by the power button... on mine, the buttons on are on the opposite part of the board, near the SFP ports, pressing the one closest to the SFP port does nothing, and the one furthest away from the SFP port just reboots the module.

In OPNsense, I am only able to see ix0 and ix1, which are the SFP ports. I have RJ45 SFPs in them, and when I connect either of them to my PC I get a link light, but when I connect them to my WAN / switch I get no link. So I'm assuming something odd is happening with speed negotiating. Any who, I'll probably end up tinkering with them for a few more hours. I've spent countless hours already trying to get them to work already.. they'll probably end up on eBay by the end of the week.

Can you enter the BIOS? If so, post a terminal capture. Would also be interesting to see photos of the PCB. Please don't confuse the poor eBay buyers. They'll be back posting on this same forum. The preferred units are non-N 640/680 (620 if really cheap) or maybe the 510 LTE which was recently $17.49 NIB.

 

[TrevorH]

I don't understand why these things are so cheap in the USA and so expensive over here in .uk. I checked Ebay UK and the cheapest there is a 640 for £360 (US$480) and there are barely any others for sale - which probably explains why they're so expensive.

 

[nmpu]

This is more than I would pay, but here's some Edge 680s for $175 BIN or BO. No 256GB SATA SSD or power supply.

$100 x3 offer was auto-rejected.

 

[cvmagic]

Long time lurker, first time contributing to a thread. Picked up an Edge 610 on ebay last week, wanted to get the ethernet ports to work on Debian. Spent some time poking around and a few tokens with ChatGPT but managed to get the switch chip configured and the 6 ethernet ports accessible. You will have to build your kernel with DSA support, and use a custom kernel module to bitbang configure the Marvell switch chip. If you want to give it a try yourselves I have the generalized version of the code linked below.

GitHub - CVMagic/edge610: Kernel modules and scripts to enable general use of ethernet ports and disable watch dog on the SD-WAN Edge 610. This project was created with the assistance of an LLM.

Kernel modules and scripts to enable general use of ethernet ports and disable watch dog on the SD-WAN Edge 610. This project was created with the assistance of an LLM. - CVMagic/edge610

github.com

Note, it is possible to bitbang configure the switch chip with only user level code, but it will be limited to bridging between the host ports and the ethernet ports on the front, if that sounds like it would work for you, let me know and publish that one too. I have only tried this with Debian Trixie, but my goal was to set this up on VyOS which is why I started with Debian directly.

 

[Kirby perkins]

Awesome teardown photos, thanks for sharing! Seeing 48GB ECC SODIMM boot up on a 680 is wild for a box this size. Definitely makes these surplus units a lot more tempting

 

[individualeleven]

Hello all, I have created a plugin for OPNsense for these devices (X and non-X) based off of the init scripts in DiagOS. This allows the 620/640/680 devices to automatically bring up all the interfaces during boot and allows for fan (if available) and LED control (to signal "startup complete"). The plugin also can setup the Marvell switch chip in the 610, if a patched kernel is installed. Also, a dashboard widget is provided to show board ID, fan speed and temperature (separate from the CPU temp reading).

EDIT: Forgot to mention, the init script in the plugin also turns off the watchdog. The endgoal is building custom "serial" images of OPNsense that someone can just plug in and install without flashing a BIOS or messing with DiagOS.

GitHub - brandontolbird/plugins at sysutils/vep1400-hw

OPNsense plugin collection. Contribute to brandontolbird/plugins development by creating an account on GitHub.

github.com

 

[cvmagic]

Have you considered submitting a PR to see if this support can be integrated into the project itself?

 

[individualeleven]

Working on that right now.

 

[individualeleven]

I've also been wrapping my head around the OPNsense build tools and I've made some progress into building an install image for the Edge 610/VEP1400 (I may make one for VEP1400-X units, but since all they need is the plugin, I may not; depends on the "demand" I guess). I've gotten to the point where I can patch the kernel for ixgbe MDIO support, build the kernel (and the rest), have the build process put the extra files (including the pkg for the plugin) into the rootfs and get a bootable USB image out of it. Next, I need to figure out how to get it to automatically build the plugin and install it during the build process, but at the moment, it's "good enough" to be able to fully bring up the device. I've attached some screenshots where I was checking that everything was working and where I did the install of plugin in the installer live OS.

 

[ericloewe]

I finally got my hands on a pair of these units (and they miraculously arrived intact despite the courier's best efforts to destroy them both by yeeting the fragile package improvised by the seller over the fence...). White Edge 620 branding, running the VeloCloud software. Curiously enough, said software did not manage the watchdog, which was weird... Also, no DiagOS installed.

Anyway, after installing DiagOS and updating the system firmware and CPLD and PIC to the latest versions Dell offers for the VEP14xx, I ended up with the fairly typical situation where the SFP+ ports are correctly detected, but all the copper NICs aren't - and that's the part I really don't understand. Messing around with the buttons did solve the problem (more details on that later), but the whole situation is very bizarre:

1. DiagOS sees all NICs and they go as far as negotiating a link (did not test further)
2. The I350 isn't even enumerated by either the system firmware or OPNsense. In other words, they don't even show up as PCI devices (on FreeBSD with pciconf)
3. The 1GBase-T ports off the SoC's X553 do show up as PCI devices, enough so for the UEFI driver to interact with that part of the device without apparent issues... But then FreeBSD's driver errors out when trying to set them up and fails to attach. The ports are just completely dead.
4. Messing around with the buttons fixes it and all the NICs show up as expected both in the AMI firmware setup application.
   1. Specifically, the button closest to the NICs seems to only have reset the system firmware to default settings.
   2. The button farthest from the NICs did not seem to do anything different.
   3. Pressing both simultaneously left the unit hanging, and it wouldn't boot. A power cycle fixed that, and the unit booted up with all NICs visible in the system firmware setup menu and working in OPNsense.

So... What the hell did Dell implement in these things?

I can imagine specific customers wanting ports disabled, but surely that could just be managed with the typical system firmware options for PCIe devices. Is this some weird power gating thing? That's the only rational explanation I can come up with that fits the data.

Apologies if someone's actually gone down this rabbit hole and figured out the details, I may have missed it in the last 39 pages.

 

[individualeleven]

Got a second 620 this week and this one was a little unusual in that the USB ports were disabled. I'm guessing this is managed by the "CPLD" because there were listings in DiagOS for GPIO for disabling the USB, WiFi, and NVMe. Neat security feature I guess but a hassle when trying to boot a USB stick. I wound up pulling out the coin cell and hoping that would reset it and thankfully it did. I'm assuming pressing the reset button until factory reset appears like nmpu said would have worked as well. Like the factory reset option this enables all of the GbE interfaces without a CPLD i2c write, so I don't need that to add to Proxmox's start-up scripts.

One last thing, this second 620 confirmed something: they only have two LEDs: red and green, no blue. They boot up with yellow, not with white like the 610s and the 640+ units (I'm assuming based again on nmpu's post; I don't have a 640 or above to test). Weird that they chose that to cheap out on and seemingly for only this revision. It doesn't cause any problems for the OPNsense plugin as the i2c write is the same, the blue byte just doesn't do anything. I have already put a help snippet in the UI for the plugin warning users of this.

 

[nmpu]

1. Messing around with the buttons fixes it and all the NICs show up as expected both in the AMI firmware setup application.
   1. Specifically, the button closest to the NICs seems to only have reset the system firmware to default settings.
   2. The button farthest from the NICs did not seem to do anything different.
   3. Pressing both simultaneously left the unit hanging, and it wouldn't boot. A power cycle fixed that, and the unit booted up with all NICs visible in the system firmware setup menu and working in OPNsense.

So... What the hell did Dell implement in these things?

I can imagine specific customers wanting ports disabled, but surely that could just be managed with the typical system firmware options for PCIe devices. Is this some weird power gating thing? That's the only rational explanation I can come up with that fits the data.

Apologies if someone's actually gone down this rabbit hole and figured out the details, I may have missed it in the last 39 pages.

The exact function of those buttons remains unknown. I thought it was to select the active BIOS slot. That's not something I want to revisit once I've got everything working. I suspect there are multiple paths to the same outcome. With sufficient tickling, the device does a low-level reset and DXE suddenly appears in the boot process. DXE enforces stricter POST checking. For example, it will reboot when the Wi-Fi module is removed. That was not the case in earlier BIOS.

The next time someone updates an old device, they need to document the before and after and the steps in between.

 

[nmpu]

... this second 620 confirmed something: they only have two LEDs: red and green, no blue.

Interesting. I'll have to add this to my list of checks during an update. I've never owned a 620 for comparison.

 

[ericloewe]

The next time someone updates an old device, they need to document the before and after and the steps in between.

That's what I'm trying to do - without getting too bogged down, since I don't have infinite time. My units came with system firmware version 3.48.0.9-7 or so, and Dell's most recent updater flashed 3.50.0.9-22, even though it has images for both 3.48 and 3.50. I guess 3.48 is reserved for certain specific customer versions? Anyway, the I350 NICs did not show up in the setup menu before the update.

The exact function of those buttons remains unknown. I thought it was to select the active BIOS slot. That's not something I want to revisit once I've got everything working.

The firmware slots theory makes a lot of sense, since the buttons are labeled S1 and S2. What's not at all clear to me is when/how the firmware update tool flashes the second firmware EEPROM. After the update I always get the 3.50 version, regardless of what buttons I pressed, so either both slots are updated by the tool (using option 5?), or nothing's actually being switched. I should also mention that attempting to flash just the system firmware (option 1 of the official tool) leaves the device unbootable, and it recovers automatically using the backup slot, which still contains the old firmware.

Small correction, too:

The 1GBase-T ports off the SoC's X553 do show up as PCI devices, enough so for the UEFI driver to interact with that part of the device without apparent issues... But then FreeBSD's driver errors out when trying to set them up and fails to attach. The ports are just completely dead.

Whatever is happening is even weirder. The UEFI driver is happy to talk to the NICs, but can't actually flash the port LEDs (which it can do when everything is working).

 

[frollic]

Specifically, the button closest to the NICs seems to only have reset the system firmware to default settings.

if you keep the button pressed long enough, the device will power off.
when you power it on again (pull cable) the NICs will be back again.

 

[ericloewe]

Ok, slightly crackpot theory: Are all the units with missing NICs just suffering from corrupted settings? I just noticed that at least one of mine is complaining about the RTC battery. More than once, I've spent hours trying to diagnose mystery issues in systems that ended up being a bad RTC battery - my favorite is when one of four memory channels on my workstation decided to stop working. As it turns out, it was probably just the memory training cache that had gotten corrupted and was messing up the one channel.

Anyway, my second unit was complaining about the RTC battery and no amount of fiddling with the buttons would get the copper ports to work
(despite the reboots happening).

if you keep the button pressed long enough, the device will power off.
when you power it on again (pull cable) the NICs will be back again.

This is exactly what worked on my first unit, but just would not work on the second unit.

So I replaced the battery, naturally. And the thing would not boot afterwards, with the red LED blinking and a garbage character being written to the serial console - it looked a lot like a boot loop very early in the process. I think this just fixed itself after a long while - I did not time it, but it felt like more than 10 minutes. None of the three buttons seemed to accomplish anything during that time.
At the end of this painful process, the NICs show up again.

So, independently of the crackpot part, it sure sounds like some sort of reset to default (internal) settings is what enables the missing NICs.

 

[individualeleven]

Ok, slightly crackpot theory: Are all the units with missing NICs just suffering from corrupted settings?

No I don't think so. I think the missing NICs are an intentional option from DellEMC/VMware, like turning off the WiFi and the USB. Some of these were deployed as WFH VPN devices (link from a former VMware employee documenting this use case), so for those use cases, they would definitely want the GbE ports on since most home users don't have SFP+ gear, and turn off the USB so that they can't be booted from by accident or on purpose, or otherwise be abused (or if they don't need a USB LTE modem, which was hinted at in the stock VeloCloud UI). But for enterprise deployments (like a remote office branch), the GbE ports might be useless and/or a security risk, so they would be disabled and it would be SFP+ only to connect to the remote office's core switch and then to the ISP. I'm assuming that Dell or VMware provides a VeloCloud customization/configuration tool to select what options they want or don't want enabled.


DiagOS does a write to the CPLD to turn on all the i350 (and ix2 and ix3) GbE interfaces for a tech to do troubleshooting I'm guessing.


From /etc/rc.local_x in DiagOS:

case ${boardtype} in
"1000"|"1010"|"1011")
# For Gris platform reinit the Marvell phy and I350 MAC
velo_ver=$(/opt/dellemc/diag/bin/nvramtool -r -R 0x59 | grep VeloCloud | awk '{ print $NF
}')
if [ -z $velo_ver ]; then
:
elif [ $velo_ver = "1" ]; then
rm /etc/udev/rules.d/70-persistent-net.rules
rmmod ixgbe
i2cset -y 1 0x31 0x1 0xff
sleep 1
while [ -z "$(lsmod | grep igb)" ] && [ $count -le $timeout ]
do
echo 1 > /sys/bus/pci/rescan
echo "Rescan PCI bus..."
count=$(( count+1 ))
sleep 1
done
modprobe ixgbe
fi
;;

That i2cset command is what tells the CPLD to enable the GbE ports and unmask the PCI devices from the bus. However, it doesn't seem to be a permanent override, since on reboot the ports go back to disabled and the PCI devices go back into hiding (I have tested this on the 620 I'm using as my router now). I have incorporated that i2c/smbus write into the OPNsense plugin so that regardless of the device's past life, all of the ports are avaiable for OPNsense.

 

[ericloewe]

Some of these were deployed as WFH VPN devices (link from a former VMware employee documenting this use case)

Oh wow, that's massive overkill. I've seen Edge 620s deployed to serve decent offices (100ish employees) and connect them back to the rest of the corporate network.

I think the missing NICs are an intentional option from DellEMC/VMware

That's almost certain, it's just implemented in fairly unusual way.

I'm assuming that Dell or VMware provides a VeloCloud customization/configuration tool to select what options they want or don't want enabled.

I'd lean towards either a factory option or handled as part of the provisioning option the OSes for these things usually have.

I have incorporated that i2c/smbus write into the OPNsense plugin so that regardless of the device's past life, all of the ports are avaiable for OPNsense.

That's super useful, thanks for the effort. It actually gets them past the "too much hassle for non-tinkerers" line. Now, if only availability would improve again...

 

[individualeleven]

Oh wow, that's massive overkill. I've seen Edge 620s deployed to serve decent offices (100ish employees) and connect them back to the rest of the corporate network.

Yeah, I agree. If this actually became an external product offer, I have no idea, but I'd imagine if it did, they would have probably offered the 610 for most of those cases.

That link also contradicts my findings of the 620 not having a blue LED, since they use that to indicate HA status. Maybe later revisions dropped the HA ability and thus the blue LED? Dunno, but I doubt it. Maybe the blue LED is just burnt out on both 620s of mine.

That's almost certain, it's just implemented in fairly unusual way.

Yeah, this could have been done in the BIOS, but maybe the SoC didn't have the GPIO pins to do all of it, so that's why they added the CPLD and PIC.

I'd lean towards either a factory option or handled as part of the provisioning option the OSes for these things usually have.

You're right, this was probably handled by VeloCloud Orchestrator.

That's super useful, thanks for the effort. It actually gets them past the "too much hassle for non-tinkerers" line. Now, if only availability would improve again...

No problem, now I just need to find a place to upload the custom serial image for the 610 and the plugin pkg. Probably just gonna create a new repo on github, since I haven't heard anything from the OPNsense team on integrating the plugin. Yeah, most of what I can find are 610s and 620s, with a few 640s and 680s well outside my budget.