Hey folks,
Not sure if this is specifically networking or the USFFs themselves. Haven't had any luck on Reddit though. This is mostly copy/pasted from that thread.
I'm running a Proxmox cluster on a quartet of old HP 260 G1 USFFs. They're very low power and cheap so I have no need to upgrade them.
Recently I've started using 2.5Gb networking - much as I would love to jump to 10Gb, cost/heat/not really being able to push packets that fast are issues I'm facing. I've added Realtek RTL8156-based USB3.0 NICs to each of the nodes, with a 2.5Gb/10Gb dumb switch between them. This serves two purposes - migration between the nodes via dedicated IPs, and shared storage over iSCSI from my TrueNAS machine via different dedicated IPs. The TrueNAS system is all Intel with 4x 2.5Gb I226 NICs and an X520-DA2 10Gb card. I currently have it connected via 10Gb DAC to the unmanaged switch. Each of the hypervisor NICs has two IPs - one for the migration network, one for iSCSI, all in the same layer-2 space (as I can't VLAN it). All 4 nodes connect to the same iSCSI IP on TrueNAS (which is separate to the web UI and file servers).
This setup works, but it doesn't reach the potential of 2.5Gb - in fact, it barely hits 1Gbps. This seems to be a known issue with the RTL8156 in the USB NICs. iperf3 does show the bandwidth hitting 2.43-2.50Gb in testing but in practise this doesn't seem to happen. During my overnight backups, I do see traffic hit a little over 1Gbps from TrueNAS when it does the dump to local disks before pushing to my other NAS, but doing a shared-storage migration of just the RAM between nodes only seems to reach 80-100MB/s. The TrueNAS machine can sustain 290MB/s during ZFS send/receive so I don't think it's a problem on that side. I did have to apply the steps in this post here: Fixing performance issues with Realtek RTL8156B 2.5GbE USB dongle in Ubuntu - CNX Software
From what I've heard, this is partially explained by the RTL8156, and partially by the USB connection. The 260s do have a Mini PCIe slot for optional wifi. I've managed to source some Intel I226 m.2 M-key cards from Ali Express. I thought I might be able to use a Mini PCIe to m.2 M-key adapter since it's all PCIe, but I'm having no luck - the 260 is not recognising any combination of cards. As the cards are pretty cheap, I bought a few different ones - some Realtek dual-port m.2, some Realtek single-port Mini PCIe. Unfortunately the latter turned out to be 1Gb cards, and testing them was the same result - no sign of life and no recognition in the OS. It's further complicated because the slots are half-length, so I generally have to break off the full-length section of the adapter which means it's not returnable if tests are negative. I have a 5th 260 G1 for testing with.
The slot does work, as I have one of the official HP wifi cards and that's recognised. I wondered if there might be an expansion card allow-list like Lenovo do (damn them), but I swapped in a spare non-HP wifi card and that was recognised too. I've considered whether the whole different-key setup with m.2 cards is the issue, but the fact that the native Mini PCIe card didn't work either is bugging me. Side note, I've tested the cards by getting a PCIe->m.2 M-key adapter and putting them into a spare machine, and all the NICs are detected. When plugged into the HPs, nothing is detected in those slots, native mini-PCIe or adapted m.2.
So I'm wondering, is it actually possible to add any NICs to these USFFs via the Mini PCIe slot? Or is there something that means it's not pure PCIe that I'm trying to tap into? I'm looking at another set of cheap Mini-PCIe NICs on AliExpress: https://www.aliexpress.com/item/1005006266147212.html but I'm hesitant to buy any more when I've had zero success with the first ones.
TIA!
Not sure if this is specifically networking or the USFFs themselves. Haven't had any luck on Reddit though. This is mostly copy/pasted from that thread.
I'm running a Proxmox cluster on a quartet of old HP 260 G1 USFFs. They're very low power and cheap so I have no need to upgrade them.
Recently I've started using 2.5Gb networking - much as I would love to jump to 10Gb, cost/heat/not really being able to push packets that fast are issues I'm facing. I've added Realtek RTL8156-based USB3.0 NICs to each of the nodes, with a 2.5Gb/10Gb dumb switch between them. This serves two purposes - migration between the nodes via dedicated IPs, and shared storage over iSCSI from my TrueNAS machine via different dedicated IPs. The TrueNAS system is all Intel with 4x 2.5Gb I226 NICs and an X520-DA2 10Gb card. I currently have it connected via 10Gb DAC to the unmanaged switch. Each of the hypervisor NICs has two IPs - one for the migration network, one for iSCSI, all in the same layer-2 space (as I can't VLAN it). All 4 nodes connect to the same iSCSI IP on TrueNAS (which is separate to the web UI and file servers).
This setup works, but it doesn't reach the potential of 2.5Gb - in fact, it barely hits 1Gbps. This seems to be a known issue with the RTL8156 in the USB NICs. iperf3 does show the bandwidth hitting 2.43-2.50Gb in testing but in practise this doesn't seem to happen. During my overnight backups, I do see traffic hit a little over 1Gbps from TrueNAS when it does the dump to local disks before pushing to my other NAS, but doing a shared-storage migration of just the RAM between nodes only seems to reach 80-100MB/s. The TrueNAS machine can sustain 290MB/s during ZFS send/receive so I don't think it's a problem on that side. I did have to apply the steps in this post here: Fixing performance issues with Realtek RTL8156B 2.5GbE USB dongle in Ubuntu - CNX Software
From what I've heard, this is partially explained by the RTL8156, and partially by the USB connection. The 260s do have a Mini PCIe slot for optional wifi. I've managed to source some Intel I226 m.2 M-key cards from Ali Express. I thought I might be able to use a Mini PCIe to m.2 M-key adapter since it's all PCIe, but I'm having no luck - the 260 is not recognising any combination of cards. As the cards are pretty cheap, I bought a few different ones - some Realtek dual-port m.2, some Realtek single-port Mini PCIe. Unfortunately the latter turned out to be 1Gb cards, and testing them was the same result - no sign of life and no recognition in the OS. It's further complicated because the slots are half-length, so I generally have to break off the full-length section of the adapter which means it's not returnable if tests are negative. I have a 5th 260 G1 for testing with.
The slot does work, as I have one of the official HP wifi cards and that's recognised. I wondered if there might be an expansion card allow-list like Lenovo do (damn them), but I swapped in a spare non-HP wifi card and that was recognised too. I've considered whether the whole different-key setup with m.2 cards is the issue, but the fact that the native Mini PCIe card didn't work either is bugging me. Side note, I've tested the cards by getting a PCIe->m.2 M-key adapter and putting them into a spare machine, and all the NICs are detected. When plugged into the HPs, nothing is detected in those slots, native mini-PCIe or adapted m.2.
So I'm wondering, is it actually possible to add any NICs to these USFFs via the Mini PCIe slot? Or is there something that means it's not pure PCIe that I'm trying to tap into? I'm looking at another set of cheap Mini-PCIe NICs on AliExpress: https://www.aliexpress.com/item/1005006266147212.html but I'm hesitant to buy any more when I've had zero success with the first ones.
TIA!
