Dell VEP/VMWare Edge/Velo Cloud SD-WAN/VeraCloud VEP1400/VEP1400-X firewall units

frollic · May 15, 2024

6TTLAB said:
dd if=openwrt-21.02.0-x86-64-generic-ext4-combined.img bs=1M of=/dev/mmcblk0.

I'd start by using a recent version of OpenWRT....

Your eMMC could be b0rked though.
Should still be able to install it, if you manage to identify the bad blocks, and make sure no data's saved there.

nmpu · May 15, 2024

frollic said:
I'd start by using a recent version of OpenWRT....

I agree. Delete any existing partitions from the eMCC using fdisk. Try again with a current image. The EFI version might play nicer if you ever want to install anything else.

I also like to know why something won't work, but I'd just concede this battle. Put DiagOS on the eMMC. Put OpenWrt on the SATA. If you have the Wi-Fi version, you can remove the Wi-Fi module and replace with an NVMe drive. You also have USB 3 for external storage. I haven't tested the eMMC speed, but it may not be the best. The eMMC is also soldered to the PCB and can't (easily) be replaced. Many of these machines had a 5-year lease from VMware. Failure was on them. You can check the original ship date using the service tag. It's like buying a used car. If you repaste the CPU, the main components will likely live a long life. The storage (like tires) has limits. I will say that I've never seen any flash device fail. I don't use my eMMC mainly because it was easier to install Proxmox on an NVMe. The SATA is removeable, but requires complete disassembly.

ccie4526 · May 18, 2024

One thing I've been tossing around in the back of my brain... the power brick that came with my 620 did not make good power contact with the power input on the device, and the only power supply I had available that did make a good connection was a 12VDC 3A unit... obviously less than the 5A that the device calls for.... but I tried it anyway and it has been working fine.

Fast forward to thinking/wondering/wishing I could just put some sort of PoE adapter in line to power the unit, then I don't need to have this extra "wall-wart" brick power supply covering up ports on the back of my UPS... and since my switch has dual power supplies, I have a little more redundancy there.... but oh, my switch is only PoE+ (30W), not PoE++ (60W). And then I came across this document at Dell Japan:
https://japancatalog.dell.com/c/wp-content/uploads/SD_WAN_EDGE_600_spec_sheet.pdf which has this nice little graphic:

Oh, how interesting! So I *could* use a PoE+ to 12VDC adapter and feed the 620, and still be within output power limits of the switch!

I'm currently traveling, but when I get back home for a few weeks, I'mma go order a PoE+ adapter and see how things work out.

mirochulovesnetwork · May 25, 2024

Hi everyone.
I'm the guy who posted the bios dump from it's cpld corrupted vep1400 back then.
I've started tinkering with it and now i wonder if it's possible to restore the cpld to its original status.
Also what os are you running on your vep1400, OpenWRT? PFsense?
Thank you

bearcat · Jun 2, 2024

Ok, played with 610 and 620, here is my summary:

1. Diag-OS permanently disables watchdog, no worry even after reboot or installing/using other OS.
This is done via rc.local i2c commands

2. Edge 610
- All RJ45 ports are connected to switch chips so normal OS won't see it unless the OS supports DSA, from what I saw in this thread, pfsense plus or Openwrt can do it.
- SFP ports can be recognized in most OSs, support 1G or 10G SFP-RJ45 copper connectors

3. Edge 620
- Unfortunately, SFP ports can only work with fiber connector.
- Good news is, all RJ45 1G ports are on PCI bus and can be recognized by most OS.
- The easiest way to make it work,
a. Make an Openwrt USB drive and boot from it, it will automatically recog all interfaces, although you might need to manually bring interfaces up.
btw: I can use wle600 wireless also under Openwrt, this turns the unit into a full functional Openwrt wireless router.
b. Download Dell version of ESXi for Vep1400, this one can be installed and run via Serial, no need to modify grub.cfg
c. Or just follow Dell instruction to install Ubuntu server, you have to customize ISO to boot via Serial

octafruittart · Jun 3, 2024

oneplane said:

Good find! The other pins (not soldered on your model) are probably either a serial port connected to the PIC or the header for the LEDs that go behind the front panel logo on SD-WAN/Velo branded boxes.

Edit: maybe this is what the jumpers are for? Disconnecting Vcc and GND from he PIC... I'll see if I can find out with a multimeter. I also know the PIC comes with a bootloader and separate runtime application, and it needs a machine reboot to switch from one mode to another, which means the startup mode is probably managed by the CPLD? In any case, this means that reading it will be easiest with either the machine off, or with the PIC in bootloader mode while it is on.

If you have the time, check the following options:

1. Disabling the PIC watchdog via I2C, a Live Linux over USB should be plenty fast in booting before the WDT kicks in

2. Flashing the BIOS backup, or checking the BIOS ID using Afu, if the ID is the same as mine it should work

As for those with a broken CPLD: the bitstream is ready to test, so hit me up if I need to write some instructions for that.

Fun fact, the MCLR pin is indeed GPIO-controlled, GPIO mapping has a specific pin for it:

XML:

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="gpio-config.xsl"?>
<config xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="gpio-config.xsd">
<platform>VEP1400</platform>

<gpio>
     <gpio-chip>0</gpio-chip>
     <gpio-bits-delimiter>256</gpio-bits-delimiter>
     <gpio-name>CORE-GPIO</gpio-name>
     <gpio-interface>CORE</gpio-interface>
     <gpio-0map>0</gpio-0map>
     <gpio-bit>
          <bit>9</bit>
          <name>BOOT_OK</name>
          <gpio-direction>OUT</gpio-direction>
          <gpio-active>HIGH</gpio-active>
          <val>1</val>
     </gpio-bit>
     <gpio-bit>
          <bit>10</bit>
          <name>INT_88E6190</name>
          <gpio-direction>IN</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>1</val>
     </gpio-bit>
     <gpio-bit>
          <bit>11</bit>
          <name>GLB_RESETn</name>
          <gpio-direction>IN</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>1</val>
     </gpio-bit>
     <gpio-bit>
          <bit>12</bit>
          <name>CPU_GPIO_12</name>
          <gpio-direction>OUT</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>0</val>
     </gpio-bit>
     <gpio-bit>
          <bit>15</bit>
          <name>RST_CPU_88E6190_N</name>
          <gpio-direction>OUT</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>0</val>
     </gpio-bit>
     <gpio-bit>
          <bit>18</bit>
          <name>CPLD_WE</name>
          <gpio-direction>OUT</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>1</val>
     </gpio-bit>
     <gpio-bit>
          <bit>27</bit>
          <name>SSD_MODULE_PRESENT_N</name>
          <gpio-direction>IN</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>1</val>
     </gpio-bit>
     <gpio-bit>
          <bit>39</bit>
          <name>EN_USB3_PORT1</name>
          <gpio-direction>OUT</gpio-direction>
          <gpio-active>HIGH</gpio-active>
          <val>1</val>
     </gpio-bit>
     <gpio-bit>
          <bit>40</bit>
          <name>EN_USB3_PORT2</name>
          <gpio-direction>OUT</gpio-direction>
          <gpio-active>HIGH</gpio-active>
          <val>1</val>
     </gpio-bit>
     <gpio-bit>
          <bit>41</bit>
          <name>THERMAL_INT</name>
          <gpio-direction>IN</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>1</val>
     </gpio-bit>
     <gpio-bit>
          <bit>43</bit>
          <name>USIM2_DETECT</name>
          <gpio-direction>OUT</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>1</val>
     </gpio-bit>
     <gpio-bit>
          <bit>44</bit>
          <name>USIM1_DETECT</name>
          <gpio-direction>OUT</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>1</val>
     </gpio-bit>
     <gpio-bit>
          <bit>80</bit>
          <name>POE_xSystem_ok</name>
          <gpio-direction>IN</gpio-direction>
          <gpio-active>HIGH</gpio-active>
          <val>1</val>
     </gpio-bit>
     <gpio-bit>
          <bit>99</bit>
          <name>WIFI_MODULE_PRESENT_N</name>
          <gpio-direction>IN</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>1</val>
     </gpio-bit>
     <gpio-bit>
          <bit>100</bit>
          <name>PIC_PRG_MCLR</name>
          <gpio-direction>OUT</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>0</val>
     </gpio-bit>
     <gpio-bit>
          <bit>120</bit>
          <name>BOOT_WDT_EN</name>
          <gpio-direction>OUT</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>1</val>
     </gpio-bit>
     <gpio-bit>
          <bit>151</bit>
          <name>USIM2_CD</name>
          <gpio-direction>IN</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>0</val>
     </gpio-bit>
     <gpio-bit>
          <bit>152</bit>
          <name>USIM1_CD</name>
          <gpio-direction>IN</gpio-direction>
          <gpio-active>LOW</gpio-active>
          <val>0</val>
     </gpio-bit>
</gpio>
</config>

Hi all,

I (foolishly) cross flashed the cpld of a vep 1400/610 with that of a 1400x/620 etc. about a year and a half ago (file modification dates give around a "Nov 11 2022").

I went through the stages of grief until I found myself reading through Lattice docs and imagining a conversation centered around "You're putting an FPGA in all of them? ALL of them?". At that point the sunk time far exceeded the cost of the device and I decided it was time to move on.

Since then, it has been a fantastic, though inert, monitor stand. On a whim I went looking to see if there was any leakage or progress on the device and came across this rather extensive thread at STH. I have read or skimmed through most of it, and in doing so came across @oneplane 's post here about a testable bitstream.

Is that an option for the vep 1400? If so, would you (or anyone else) be willing to send it (one?) over? I have nothing to lose with this device and I truly don't mind if it is further bricked.

I do have bios dumps, from both flash chips, if anyone is interested. I may also have a complete dump of the emmc with the original installation/os floating around somewhere, though I have not tested it.

Looking at my recorded boot logs I was at

Code:

Version 2.19.1266. Copyright (C) 2019 American Megatrends, Inc.

BIOS Date: 06/24/2019 19:32:15 Ver: 0ACHI040

and

Code:

[    2.091387] vc: cpld rev=0x33 board=0x50 bid=0x00 edge610

The device itself seems to be a A03 revision.

Brood · Jun 9, 2024

Good Afternoon All,

I currently have the 640 up and running. All of the RJ45 ports are working and seen by Pfsense. I how ever have a issue to get the SFP connectors up and seen by Pfsense. What should I do to get them up and running as I would ideally switch to a SFP-gpon and have a 10gb sfp uplink directly to my switch

ccie4526 · Jun 9, 2024

Brood said:
Good Afternoon All,

I currently have the 640 up and running. All of the RJ45 ports are working and seen by Pfsense. I how ever have a issue to get the SFP connectors up and seen by Pfsense. What should I do to get them up and running as I would ideally switch to a SFP-gpon and have a 10gb sfp uplink directly to my switch

PF should detect the SFP ports as ix0 and ix1. Would be a good idea if you could screenshot your "interfaces" screen or ssh into the box and get a copy of "ifconfig". For example, from my 620:

Code:

ix0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: OUTSIDE
        options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>
        ether e8:b5:d0:xx:xx:xx
        inet 10.0.0.10 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::eab5:d0ff:fexx:xxxx%ix0 prefixlen 64 scopeid 0x5
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAN
        options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>
        ether e8:b5:d0:xx:xx:xx
        inet 10.50.51.233 netmask 0xfffffff8 broadcast 10.50.51.239
        inet6 fe80::eab5:d0ff:fexx:xxxx%ix1 prefixlen 64 scopeid 0x6
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Also keep in mind there may be some issues with some of the 10G-TX copper SFPs. Personally, I'm running a pair of Cisco SFP-10G-SR fiber SFPs in my 620, and it is quite happy with them.

Brood · Jun 9, 2024

Thank you I will have a look as I've tried a couple of different 1gb-t that work flawlessly in the 610 but is dead as a dodo in the 640. I am however ugrading to a newer switch with 10gb sfp+ connections so might look into the using a fibre connection to the switch instead of a copper connection

6TTLAB · Jun 12, 2024

nmpu said:
You need to use the dd command to copy the image/partition. sda is the internal SATA drive. sdb is your USB drive. mmcblk0 is the eMMC.

The original OpenWrt instructions show the following:

Code:

# Unpack image gunzip openwrt-*.img.gz # Identify disk (to replace sdX in the following command below) lsblk # Write image dd if=openwrt-21.02.0-x86-64-generic-ext4-combined.img bs=1M of=/dev/sdX

If you're happy with your current USB image, you should be able to use that. The command would be:

Code:

dd if=/dev/sdb of=/dev/mmcblk0

You would then change the boot device in the BIOS.

I have not tested this as I don't want to mess with a working system. I have verified that the dd command is available in OpenWrt. It's part of BusyBox. fdisk, parted, and resize2fs are available, but may need to be installed.

hi bro thanks your help

i have some problem lately .
can you send me openwrt support 10G network ?
thanks!

nmpu · Jun 12, 2024

Yes, OpenWrt recognizes 10Gb SFP on both my 640 and 680. However, the actual maximum firewall throughput might only be 5Gb due to CPU routing.

6TTLAB · Jun 12, 2024

nmpu said:
Yes, OpenWrt recognizes 10Gb SFP on both my 640 and 680. However, the actual maximum firewall throughput might only be 5Gb due to CPU routing.

openwrt cant't recognize 10G SFP+ port on 620?
The 620 and 640 have the same network adapter ixgbe?

nmpu · Jun 12, 2024

I'm assuming that a 620, 640, and 680 have identical network hardware/setup, but I don't personally own a 620 to test.

ccie4526 · Jun 13, 2024

nmpu said:
I'm assuming that a 620, 640, and 680 have identical network hardware/setup, but I don't personally own a 620 to test.

From a 620....

Code:

ix0@pci0:5:0:0: class=0x020000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x15c4 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X553 10 GbE SFP+'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 64 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(128) FLR RO
                 max read 512
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 0000c9ffff000000
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0180, VF RID Stride 0x0002
                     VF Device ID 0x15c5
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
                     P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
                     P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
                     P2P Direct Translated unavailable, Enhanced Capability unavailable
ix1@pci0:5:0:1: class=0x020000 rev=0x11 hdr=0x00 vendor=0x8086 device=0x15c4 subvendor=0x8086 subdevice=0x0000
    vendor     = 'Intel Corporation'
    device     = 'Ethernet Connection X553 10 GbE SFP+'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 64 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 128(128) FLR RO
                 max read 512
                 link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)
    ecap 0001[100] = AER 2 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 0000c9ffff000000
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0180, VF RID Stride 0x0002
                     VF Device ID 0x15c5
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
    ecap 000d[1b0] = ACS 1 Source Validation unavailable, Translation Blocking unavailable
                     P2P Req Redirect unavailable, P2P Cmpl Redirect unavailable
                     P2P Upstream Forwarding unavailable, P2P Egress Control unavailable
                     P2P Direct Translated unavailable, Enhanced Capability unavailable

I'm curious if the 640/680 have the same restriction of

Code:

link x1(x1) speed 2.5(2.5) ASPM disabled(L0s/L1)

as that might say even with 10G optics, the port is still going to be restricted to 2.5Gbps. Of course this might already be answered here in this forum, but then I'd have to read back through all 16 pages again. ;-)

nmpu · Jun 13, 2024

I see a 2.5 as well, but it's 2.5GT/s. The T is for Transfers. I can definitely pass more than 2.5Gb/s of network traffic.

Code:

05:00.1 Ethernet controller: Intel Corporation Ethernet Connection X553 10 GbE SFP+ (rev 11)
        DeviceName: sfp1
        Subsystem: Intel Corporation Ethernet Connection X553 10 GbE SFP+
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr+ Stepping- SERR+ FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin B routed to IRQ 17
        IOMMU group: 21
        Region 0: Memory at dea00000 (64-bit, prefetchable) [size=2M]
        Region 4: Memory at dee00000 (64-bit, prefetchable) [size=16K]
        Expansion ROM at df700000 [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] MSI-X: Enable+ Count=64 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00002000
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x1
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 260ms to 900ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES- TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [140 v1] Device Serial Number 00-00-c9-ff-ff-00-00-00
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 0
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration- 10BitTagReq- Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy- 10BitTagReq-
                IOVSta: Migration-
                Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 01
                VF offset: 128, stride: 2, Device ID: 15c5
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 00000000df900000 (64-bit, non-prefetchable)
                Region 3: Memory at 00000000df800000 (64-bit, non-prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [1b0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Kernel driver in use: ixgbe
        Kernel modules: ixgbe

The 2.5 has to do with PCIe bus transfer speed. However, even if the X553 has 2 PCIe lanes, it still looks bandwidth starved. If that's the case, then how can I push 12Gb/s using SR-IOV?

Mithril · Jun 13, 2024

ccie4526 said:

PF should detect the SFP ports as ix0 and ix1. Would be a good idea if you could screenshot your "interfaces" screen or ssh into the box and get a copy of "ifconfig". For example, from my 620:

Code:

ix0: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: OUTSIDE
        options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>
        ether e8:b5:d0:xx:xx:xx
        inet 10.0.0.10 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::eab5:d0ff:fexx:xxxx%ix0 prefixlen 64 scopeid 0x5
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
ix1: flags=1008943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST,LOWER_UP> metric 0 mtu 1500
        description: LAN
        options=48138b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,HWSTATS,MEXTPG>
        ether e8:b5:d0:xx:xx:xx
        inet 10.50.51.233 netmask 0xfffffff8 broadcast 10.50.51.239
        inet6 fe80::eab5:d0ff:fexx:xxxx%ix1 prefixlen 64 scopeid 0x6
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

Also keep in mind there may be some issues with some of the 10G-TX copper SFPs. Personally, I'm running a pair of Cisco SFP-10G-SR fiber SFPs in my 620, and it is quite happy with them.

Since you have a 620 and 10G to it, what's your internet speed and what does the CPU use look like when you are doing a speed test? Given the price bump the 640s seem to have on ebay they seem not likely worth it (double the cores doesn't mean double the performance for most things).

nmpu · Jun 13, 2024

Mithril said:
Since you have a 620 and 10G to it, what's your internet speed and what does the CPU use look like when you are doing a speed test? Given the price bump the 640s seem to have on ebay they seem not likely worth it (double the cores doesn't mean double the performance for most things).

I posted some numbers here. I've currently only got 500Gb/s internet provided by HOA. As long as I can match line speed, I'll be happy. I agree that for many tasks (including routing) more cores does not scale well. In the recent past, there were more 640s on eBay. Maybe the 'STH effect' has increased competition. Historically, I've scored some good deals. Even if it's not cutting edge, I believe the Dell hardware is likely to be more reliable than some generic Chinese seller.

frankharv · Jun 13, 2024

nmpu said:
However, even if the X553 has 2 PCIe lanes, it still looks bandwidth starved. If that's the case, then how can I push 12Gb/s using SR-IOV?

Worse yet 2.5GT is PCIe 1.0 speeds.
Two lanes of that = 500 Megabytes a second.

Now in saying that 12Gb sec= 1.5 Gigabytes per second.
So I don't see how two PCIe 1.0 lanes could achieve that.
In truth it sounds like PCIe 3.0 speeds which is acheivable.
x2 lanes of PCIe 3.0 offers 1.96GB/sec which sound right with overhead.

nmpu · Jun 13, 2024

I just booted into the official Dell DiagOS. It shows the same 2.5GT/s number. These network ports are internal to the SOC. While they do work with PCIe passthrough to a VM, I'm pretty sure there's no LVDS signaling going on inside the chip. How do you report PCIe lanes/speed when there are none?

arcadia420 · Jul 7, 2024

I am thinking about getting a 620. I've seen contradictory posts where one post recommends avoiding the EMC labeled black units while another says avoid the VMware labeled white units. Is there any consensus on this?

Dell VEP/VMWare Edge/Velo Cloud SD-WAN/VeraCloud VEP1400/VEP1400-X firewall units

New Member

Active Member

Active Member

New Member

New Member

New Member

Member

Active Member

Member

New Member

Active Member

New Member

Active Member

Active Member

Active Member

Active Member

Active Member

Active Member

Active Member

New Member