NEW! Topton 10Gb 2xSFP+ 4x2.5Gb i5-1240P

[EncryptedUsername]

You do have to pass the full SFP+ PCIe card through (not possible to only pass through the 10G ports individually).

A final note, this was an error on my part. I was selecting "all functions" when adding the PCI device in Proxmox. With that unselected, I was able to add one SFP+ port as passthrough and one as virtualized. Handy, depending on what you want to do with it.

 

[giacombum]

Do you think that 1235U is powerful enough (same PCI configuration of 1240U I guess) to do routing of 10G optic fiber? Or I need 1240U?

 

[EncryptedUsername]

Do you think that 1235U is powerful enough (same PCI configuration of 1240U I guess) to do routing of 10G optic fiber? Or I need 1240U?

Based on these spec sheets there is very little difference between the 1235U and the 1240U, other than power (and therefore heat).
For some reason, the 1235U consumes a lot more power (55W vs. 29W).

Intel® Core™ i5-1235U Processor (12M Cache, up to 4.40 GHz) - Product Specifications | Intel

Intel® Core™ i5-1235U Processor (12M Cache, up to 4.40 GHz) quick reference with specifications, features, and technologies.

ark.intel.com

Intel® Core™ i5-1240U Processor (12M Cache, up to 4.40 GHz) - Product Specifications | Intel

Intel® Core™ i5-1240U Processor (12M Cache, up to 4.40 GHz) quick reference with specifications, features, and technologies.

ark.intel.com

The 1240U is running my workloads flawlessly - (Proxmox VE running 5 guest workloads, one of which is pfSense with the 10 G ports passed through).
No guarantees, but seems like the 1235U should do just as well, but dissipate more heat in the process - so better cooling may be needed.

 

[dak64]

@EncryptedUsername I'm very tempted with this h/w and have been thinking of deploying pfSense or OPNsense in Proxmox. But the problem with FreeBSD and virtIO is an inhibitor. I was hoping that SR-IOV would solve it. I assume you checked out this in the Proxmox docs? And as of today nothing you have tried has got SR-IOV to work?

I have tested both pfSense and OPNsense on a Qotom box (10th gen core i7) with Proxmox 8 and the fastest throughput I can get over virtIO is 3-4 Gbps which is 1/10th what I can get between two linux VM or Containers on this box. Looking at the history in FreeNAS, the problem has been known for >10 years, so I doubt any prospect of a fix.

I can passthrough the WAN, but I want the LAN to connect to both the router and a proxmox bridge. I was hopeful the SR-IOV would allow that with decent performance, otherwise other VMs and containers will be limited by this virtIO bug.

If SR-IOV cannot work with pfSense/OPNsense then I will have to look for a linux-based router/firewall instead. Thanks.

 

[dak64]

There is a interesting and quite detailed article on getting SR-IOV working in both FreeBSD and Linux here.

 

[EncryptedUsername]

Technically, I did get SR-IOV working on this device - the issue I hit was that the virtual pfSense guest could not load the driver for the NIC. I eventually gave up trying and used PCI passthrough instead. I did not test OPNSense so I couldn't say if that might work or not. It's not an issue with the hardware - more of a guest OS support thing from what I could tell.

 

[EncryptedUsername]

I also tried the Netgate forum - but got no replies ...

Intel 10G Server NIC SRV-IO Drivers

Been doing some reading on this, and so far I haven't found any working solutions. I tried the hw.pci.honor_msi_blacklist=0 in /boot/loader.conf.local trick,...

forum.netgate.com

 

[dak64]

The more I look a FreeBSD the less I like it. The VirtIO driver is 10x slower than linux. The SR-VIO is not working (and reading the readme in the source, even if it was working it doesn't allow promiscuous mode, so tcpdump won't work... something I use quite a lot). I don't know if just lifting the Intel driver (which is provided as source code only) and dumping it into FreeBSD build tree would fix SR-VIO or not, but I would rather the FreeBSD team focus on VirtIO... but 10+ years in I doubt they will.

So for a router/firewall in Proxmox, it looks like OpenWRT may be the only option.

 

[thepsyborg]

So for a router/firewall in Proxmox, it looks like OpenWRT may be the only option.

VyOS still exists, so far. I'm sure they'll kill it off the rest of the way eventually, but they haven't yet.

 

[blunden]

VyOS still exists, so far. I'm sure Broadcom'll kill it off the rest of the way eventually, but they haven't yet.

Why would Broadcom have anything to do with it?

 

[zer0sum]

Technically, I did get SR-IOV working on this device - the issue I hit was that the virtual pfSense guest could not load the driver for the NIC. I eventually gave up trying and used PCI passthrough instead. I did not test OPNSense so I couldn't say if that might work or not. It's not an issue with the hardware - more of a guest OS support thing from what I could tell.

Proxmox + OPNsense + SRIOV interfaces work perfectly

I would think that PFsense would be fine as well, but didn't test that.

The more annoying issue is once you start using VF's none of the other guests can communicate.

[SOLVED] - communication issue between SRIOV VM VF and CT on PF bridge

I'm passing through a VF using SRIOV to a VM. This VM works fine, I can reach it from Proxmox (mgmt interface on vmbr0 of the PF) and other hosts on the network. However, a CT attached to the very same vmbr0 cannot reach the VM, but also everything else on the network. According to tcpdump...

forum.proxmox.com

 

[thepsyborg]

Why would Broadcom have anything to do with it?

Ah, sorry, my bad; Broadcom acquired VMWare and began systematically killing off small-scale VMWare/ESXI users at around the same time VyOS pulled its in-progress VPP support project from the public branch and I guess I had conflated the two events in my head.

I'm still pissed off about it, though; it's quite challenging to do full-featured one-box routing (meaning full-duplex routing+NAT+firewall minimum and preferably also a reasonable level of IDS/IPS) at 10Gbps with reasonably affordable consumer hardware and virtually impossible to do so beyond that speed without Vector Packet Processing, a FOSS ultrafast software networking dataplane developed by the Fast Data Project, a project of the Linux Foundation.

There were exactly two (general-purpose non-hardware-specific) consumer router OS projects working on implementing VPP: Netgate's TNSR (which, even if you don't morally object to supporting them after their conduct towards OPNSense, is expensive), and VyOS, which was and had always been fully free and open source, and was basically the hope for the long-term future of homelab routing, especially as residential internet speeds continue to increase (even if only for those of us lucky enough to live in the right handful of neighborhoods).

And then they pulled VPP support, spent months dodging questions about it, pulled free access to LTS release ISOs and then two months ago also pulled access to the repositories and build tools- as of right now the only free VyOS is the nightly rolling build- and then about a month ago released the VPP as an "add-on" exclusive to their paid subscription customers. Which start at 50% more expensive than the already-obscene TNSR and go up from there.

The Linux Foundation needs to take a good hard look at its Fast Data Project because while it's been a massive thing for a whole bunch of cloud and proprietary projects there is currently not even the most cursory attempt being made to actually benefit the wider open-source community.

 

[blunden]

Ah, sorry, my bad; Broadcom acquired VMWare and began systematically killing off small-scale VMWare/ESXI users at around the same time VyOS pulled its in-progress VPP support project from the public branch and I guess I had conflated the two events in my head.

I'm still pissed off about it, though; it's quite challenging to do full-featured one-box routing (meaning full-duplex routing+NAT+firewall minimum and preferably also a reasonable level of IDS/IPS) at 10Gbps with reasonably affordable consumer hardware and virtually impossible to do so beyond that speed without Vector Packet Processing, a FOSS ultrafast software networking dataplane developed by the Fast Data Project, a project of the Linux Foundation.

There were exactly two (general-purpose non-hardware-specific) consumer router OS projects working on implementing VPP: Netgate's TNSR (which, even if you don't morally object to supporting them after their conduct towards OPNSense, is expensive), and VyOS, which was and had always been fully free and open source, and was basically the hope for the long-term future of homelab routing, especially as residential internet speeds continue to increase (even if only for those of us lucky enough to live in the right handful of neighborhoods).

And then they pulled VPP support, spent months dodging questions about it, pulled free access to LTS release ISOs and then two months ago also pulled access to the repositories and build tools- as of right now the only free VyOS is the nightly rolling build- and then about a month ago released the VPP as an "add-on" exclusive to their paid subscription customers. Which start at 50% more expensive than the already-obscene TNSR and go up from there.

The Linux Foundation needs to take a good hard look at its Fast Data Project because while it's been a massive thing for a whole bunch of cloud and proprietary projects there is currently not even the most cursory attempt being made to actually benefit the wider open-source community.

I see. Yeah, those are completely unrelated.

Yeah, it's sad that the only free routing distribution with VPP is no longer available for free. It's obviously still possible to set up VPP yourself on a normal Linux distribution, but that's definitely a hassle that only a very select few will do.

The situation with the VyOS builds being available for free is improving soon though according to their recent announcement of VyOS Stream. Those builds are going to be available for anyone. For me, that's arguably even better than LTS releases.

Introducing VyOS Stream — a next step in the VyOS project evolution

that will improve the LTS release development process and make it easier for the community to participate in it.

blog.vyos.io

 

[athurdent]

Everyone still happy with their 1240P units? I‘m considering a HUNSN RJ50f one.
Had ROUFAWIT (or so) N100 from an amazon.de special offer which came with an unacceptably noisy additional fan on the SFP+ NIC, instantly crashed with 32GB (modules showed PASS in memtest86+) when passing through a 10G NIC in Proxmox and it had a very bad kind of “soily/earthy“ smell that wouldn’t wear off.
Send it back but really liked the form factor, low energy consumption, speed and and the low noise level with the 10G NIC fan unplugged, which support told me to do…

 

[georgelza]

Which Transceivers are you using.

Have you figured out what !10GbE chip is being used.

Got myself the U300E based Topton for my pfSense. 2.7.2

G

Well I am quite happy to post this update. My Topton box with the 1240P processor and 10G SFP+ ports arrived today.

TLDR: The device can push 20 Gbps (10 in and 10 out) with packet inspection enabled.

My setup: - two PCs, both with 82599 (X520) cards connected by fiber to the topton running pfSense 2.7.2 bare metal. The WAN interface is one of the 2.5 G ports.

The PCI speeds show for ix0 and ix1 as follows with pciconf:

> pciconf -lcv ix0
ix0@pci0:4:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x10fb subvendor=0xffff subdevice=0xffff
    vendor     = 'Intel Corporation'
    device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 64 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 max read 512
                 link x4(x8) speed 5.0(5.0) ASPM disabled(L0s)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 d42000ffffb1d8b3
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0180, VF RID Stride 0x0002
                     VF Device ID 0x10ed
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304
> pciconf -lcv ix1
ix1@pci0:4:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x10fb subvendor=0xffff subdevice=0xffff
    vendor     = 'Intel Corporation'
    device     = '82599ES 10-Gigabit SFI/SFP+ Network Connection'
    class      = network
    subclass   = ethernet
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[50] = MSI supports 1 message, 64 bit, vector masks
    cap 11[70] = MSI-X supports 64 messages, enabled
                 Table in map 0x20[0x0], PBA in map 0x20[0x2000]
    cap 10[a0] = PCI-Express 2 endpoint max data 256(512) FLR RO NS
                 max read 512
                 link x4(x8) speed 5.0(5.0) ASPM disabled(L0s)
    cap 03[e0] = VPD
    ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
    ecap 0003[140] = Serial 1 d42000ffffb1d8b3
    ecap 000e[150] = ARI 1
    ecap 0010[160] = SR-IOV 1 IOV disabled, Memory Space disabled, ARI disabled
                     0 VFs configured out of 64 supported
                     First VF RID Offset 0x0180, VF RID Stride 0x0002
                     VF Device ID 0x10ed
                     Page Sizes: 4096 (enabled), 8192, 65536, 262144, 1048576, 4194304

So they both report x4 lanes of PCIe 2.0 bandwidth available to each.

iperf3 shows this directly to the firewall (I used 10 threads: -P 10):

[SUM]   0.00-10.00  sec  11.0 GBytes  9.43 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  11.0 GBytes  9.43 Gbits/sec                  receiver

iperf3 shows this between PC1 and PC2 (routed through the firewall, and inspected by Suricata - Suricata is detecting things so I know its inspecting)

[SUM]   0.00-90.59  sec   100 GBytes  9.48 Gbits/sec                  sender
[SUM]   0.00-90.59  sec   100 GBytes  9.48 Gbits/sec                  receiver

And the pfSense Traffic graphs look like this:


I will include this in my eventual review on the Ali-Express page so that other folks will know. So that is that! Very happy with these results. On the weekend I will try virtualizing pfSense on top of Proxmox and see how that goes.

P.S. I used these SFP+ modules coded for intel and they just worked with 10Gtek Fiber Patch Cables (LC to LC OM3)

https://www.amazon.ca/dp/B01CN82LP8?ref=ppx_yo2ov_dt_b_product_details&th=1

 

[georgelza]

NIC Chipset:
vendor = 'Intel Corporation'
device = '82599ES 10-Gigabit SFI/SFP+ Network Connection'


Transceiver: 10Gtek 10 Gigabit SFP+ LC Multi-Mode Transceiver, 10GBASE-SR Module for Intel E10GSFPSR (850nm, DDM, 300m), now to find a supplier that ships to South Africa.

 

[casulo]

- Does this mini pc/router has GPIO, so power and reset can be done with pikvm?
- Does BIOS support bifurcation for multiple nvme disks on the pci slot?

They need to update this for newer cpus and memory sizes. and pci-e4.

 

[georgelza]

don't know answers for above...

Re my Tranceiver question, found people (elsewhere listed here) in USA having success with Gtek transceivers.

G

 

[georgelza]

anyone know/remember if the Topton's have space for a 2.5" SSD on the inside.
I got a 1Tb NVMe installed atm, wanting to add some more space... for ZFS or Ceph.

G

 

[blunden]

don't know answers for above...

Re my Tranceiver question, found people (elsewhere listed here) in USA having success with Gtek transceivers.

G

If 10Gtek isn't available in your region you can check if the brand "ipolex" is. They rebrand 10Gtek transceivers as far as I can tell. In fact, they are pretty open about 10Gtek being the manufacturer on the packaging.

As long as you get an Intel coded one, it will probably work fine.