QNAP QXG-10G2T-X710 discussion thread

[heromode]

i got a QNAP QXG-10G2T-X710 nic, which is a dualport intel X710 card. sr-iov capable. This thread is intended for any information about this card.

initially it seems like no temp and no fan speed readings, but that's to be expected.

At least there is a small fan that spins, i'm so tired of worrying about the airflow for cheap and hot SFP+ enterprise cards, and worrying about having to buy 40 EUR media converters that run close to 100 degrees celcius the moment you transfer a 100KB file over them. 10Gbe is where it's at baby!

this OP will be updated as needed, for now, initial readouts in linux below.

# lspci -s 02:00.0 -vvvvvvvvvvvvvvv
02:00.0 Ethernet controller: Intel Corporation Ethernet Controller X710 for 10GBASE-T (rev 02)
        Subsystem: QNAP Systems, Inc. Ethernet Controller X710 for 10GBASE-T
        Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 17
        IOMMU group: 1
        Region 0: Memory at 4014000000 (64-bit, prefetchable) [size=16M]
        Region 3: Memory at 4015808000 (64-bit, prefetchable) [size=32K]
        Expansion ROM at 91580000 [disabled] [size=512K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] MSI-X: Enable+ Count=129 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00001000
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop- FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 8GT/s, Width x4
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR-
                         10BitTagComp- 10BitTagReq- OBFF Not Supported, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp- ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR- 10BitTagReq- OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance Preset/De-emphasis: -6dB de-emphasis, 0dB preshoot
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+ EqualizationPhase1+
                         EqualizationPhase2+ EqualizationPhase3+ LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [e0] Vital Product Data
                Product Name: Example VPD
                Read-only fields:
                        [V0] Vendor specific:
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                End
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt+ UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [140 v1] Device Serial Number 02-f6-67-ff-ff-be-5e-24
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration- 10BitTagReq- Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy- 10BitTagReq-
                IOVSta: Migration-
                Initial VFs: 64, Total VFs: 64, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 272, stride: 1, Device ID: 154c
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 0000004015400000 (64-bit, prefetchable)
                Region 3: Memory at 0000004015910000 (64-bit, prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [1a0 v1] Transaction Processing Hints
                Device specific mode supported
                No steering table available
        Capabilities: [1b0 v1] Access Control Services
                ACSCap: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
                ACSCtl: SrcValid- TransBlk- ReqRedir- CmpltRedir- UpstreamFwd- EgressCtrl- DirectTrans-
        Capabilities: [1d0 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Kernel driver in use: i40e
        Kernel modules: i40e
       
# modinfo i40e
filename:       /lib/modules/6.1.0-27-amd64/kernel/drivers/net/ethernet/intel/i40e/i40e.ko
license:        GPL v2
description:    Intel(R) Ethernet Connection XL710 Network Driver
author:         Intel Corporation, <e1000-devel@lists.sourceforge.net>
alias:          pci:v00008086d0000158Bsv*sd*bc*sc*i*
alias:          pci:v00008086d0000158Asv*sd*bc*sc*i*
alias:          pci:v00008086d00000D58sv*sd*bc*sc*i*
alias:          pci:v00008086d00000CF8sv*sd*bc*sc*i*
alias:          pci:v00008086d00001588sv*sd*bc*sc*i*
alias:          pci:v00008086d00001587sv*sd*bc*sc*i*
alias:          pci:v00008086d00000DDAsv*sd*bc*sc*i*
alias:          pci:v00008086d000037D3sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D2sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D1sv*sd*bc*sc*i*
alias:          pci:v00008086d000037D0sv*sd*bc*sc*i*
alias:          pci:v00008086d000037CFsv*sd*bc*sc*i*
alias:          pci:v00008086d000037CEsv*sd*bc*sc*i*
alias:          pci:v00008086d0000104Fsv*sd*bc*sc*i*
alias:          pci:v00008086d0000104Esv*sd*bc*sc*i*
alias:          pci:v00008086d000015FFsv*sd*bc*sc*i*
alias:          pci:v00008086d00001589sv*sd*bc*sc*i*
alias:          pci:v00008086d00001586sv*sd*bc*sc*i*
alias:          pci:v00008086d00000DD2sv*sd*bc*sc*i*
alias:          pci:v00008086d00001585sv*sd*bc*sc*i*
alias:          pci:v00008086d00001584sv*sd*bc*sc*i*
alias:          pci:v00008086d00001583sv*sd*bc*sc*i*
alias:          pci:v00008086d00001581sv*sd*bc*sc*i*
alias:          pci:v00008086d00001580sv*sd*bc*sc*i*
alias:          pci:v00008086d00001574sv*sd*bc*sc*i*
alias:          pci:v00008086d00001572sv*sd*bc*sc*i*
depends:      
retpoline:      Y
intree:         Y
name:           i40e
vermagic:       6.1.0-27-amd64 SMP preempt mod_unload modversions
sig_id:         PKCS#7
signer:         Debian Secure Boot CA
sig_key:        32:A0:28:7F:84:1A:03:6F:A3:93:C1:E0:65:C4:3A:E6:B2:42:26:43
sig_hashalgo:   sha256
signature:      0A:3F:9D:B5:4F:8F:57:58:85:99:9C:59:51:51:89:52:0E:33:B9:4A:
                E3:D4:74:54:05:15:55:42:2A:49:B4:70:CA:2E:7D:C9:C0:A8:F5:E4:
                61:F0:30:BF:2A:9A:5E:C6:20:8D:C9:8F:EC:0E:81:D9:B6:40:87:08:
                D3:7A:AB:76:77:2E:AA:08:CE:D5:CF:3E:CE:FE:98:CA:98:9C:59:A1:
                C3:93:C7:EB:55:6F:25:B8:37:D9:E3:7C:D0:3C:BF:66:D5:14:FF:82:
                A1:4C:87:95:D1:70:3C:4A:98:D0:79:31:6B:99:EF:B7:04:79:C4:9C:
                71:78:63:AF:81:B2:FF:4B:E3:8F:B2:FC:72:76:79:D2:26:32:22:16:
                22:E2:DE:81:C4:BE:B6:0C:EA:ED:FD:E4:6A:AC:7F:95:42:52:F9:60:
                62:74:DA:A0:7E:9F:5D:6D:1E:3B:EF:96:7B:2B:E2:C4:7F:68:55:74:
                B3:57:66:B1:42:2F:D4:24:3C:B3:A7:34:2C:62:9F:AE:E8:55:1E:E9:
                C6:4A:80:13:6E:FA:A2:6E:73:8C:F8:72:09:50:88:66:0F:86:22:68:
                BF:1E:48:F7:A2:F4:8D:D5:E9:5B:A8:54:F8:D4:83:36:DE:FA:CE:6A:
                9B:5D:84:C6:45:5E:D3:03:9F:CF:54:30:7A:0F:01:C5
parm:           debug:Debug level (0=none,...,16=all), Debug mask (0x8XXXXXXX) (uint)


# ethtool enp2s0f0
Settings for enp2s0f0:
        Supported ports: [  ]
        Supported link modes:   100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  100baseT/Full
                                1000baseT/Full
                                10000baseT/Full
                                2500baseT/Full
                                5000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Unknown! (255)
        Auto-negotiation: off
        Port: Other
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: no

 

[heromode]

update:

My standard systemd proxmox script for sr-iov partitioning and detaching vf's from host seem to be somewhat working, i've modified the lspci command to list the vf's. I obtained the lspci -D -d 8086:154c command by trying various combo's from the output of lspci -vvm -nn -s 0000:01:02.0

#lspci -vvm -nn -s 0000:01:02.0
Device: 01:02.0
Class:  Ethernet controller [0200]
Vendor: Intel Corporation [8086]
Device: Ethernet Virtual Function 700 Series [154c]
SVendor:        QNAP Systems, Inc. [1baa]
SDevice:        Ethernet Virtual Function 700 Series [0000]
Rev:    02
ProgIf: 00
NUMANode:       0
IOMMUGroup:     158

Each physical port can be divided to 64 partitions iirc. This script creates 8 virtual functions per physical port, set static MAC's for them (VF's get random generated MAC's each time they are created), and then detaches 7 VF's per PF for use with VM's. I wanted to leave 1 VF per port for the proxmox host for use with things like iscsi. The actual PF is also used by proxmox as a bridged port, for VM's that use it via standard virtio.

If you add or remove pcie devices you need to check if the pcie address of your nic changes, and edit the script accordingly. I wish it wasn't so, but i have yet to figure out another way. Proxmox now supports device mapping for pcie passthrough devices, to enable migration support.

I wish the Proxmox team could make a GUI function for partitioning and detaching network devices as well. I don't have the skills or the time to get into all that.

nano /etc/systemd/system/sriov-vfs.service

[Unit]
Description=Enable SR-IOV and detach guest VFs from host
Requires=network.target
After=network.target
Before=pve-firewall.service
[Service]
Type=oneshot
RemainAfterExit=yes
# Create NIC VFs
ExecStart=/usr/bin/bash -c 'echo 8 > /sys/class/net/ens2f0np0/device/sriov_numvfs'
ExecStart=/usr/bin/bash -c 'echo 8 > /sys/class/net/ens2f1np1/device/sriov_numvfs'
# Set static MACs for VFs
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 0 mac 76:9e:17:83:39:e5'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 1 mac 46:2c:6d:24:6b:1b'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 2 mac 3e:47:48:12:ed:94'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 3 mac be:e3:6a:f3:8f:ac'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 4 mac 62:8f:3d:bb:02:08'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 5 mac ae:91:57:b9:14:7f'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 6 mac 5a:c2:08:a9:68:a7'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 7 mac b2:f0:18:af:cb:c5'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 0 mac 16:47:7c:a8:95:98'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 1 mac a6:c7:c5:7f:9c:22'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 2 mac b6:0f:45:34:5e:19'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 3 mac 2a:f7:37:84:31:30'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 4 mac 8a:fa:f8:c5:0b:93'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 5 mac b2:f5:d5:2f:79:06'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 6 mac c2:92:f5:fa:32:20'
ExecStart=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 7 mac 2e:fb:29:1e:48:31'
# Detach VFs from host
ExecStart=/usr/bin/bash -c 'echo 0000:01:02.1 > /sys/bus/pci/devices/0000\\:01\\:02.1/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:02.2 > /sys/bus/pci/devices/0000\\:01\\:02.2/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:02.3 > /sys/bus/pci/devices/0000\\:01\\:02.3/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:02.4 > /sys/bus/pci/devices/0000\\:01\\:02.4/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:02.5 > /sys/bus/pci/devices/0000\\:01\\:02.5/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:02.6 > /sys/bus/pci/devices/0000\\:01\\:02.6/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:02.7 > /sys/bus/pci/devices/0000\\:01\\:02.7/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:0a.1 > /sys/bus/pci/devices/0000\\:01\\:0a.1/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:0a.2 > /sys/bus/pci/devices/0000\\:01\\:0a.2/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:0a.3 > /sys/bus/pci/devices/0000\\:01\\:0a.3/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:0a.4 > /sys/bus/pci/devices/0000\\:01\\:0a.4/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:0a.5 > /sys/bus/pci/devices/0000\\:01\\:0a.5/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:0a.6 > /sys/bus/pci/devices/0000\\:01\\:0a.6/driver/unbind'
ExecStart=/usr/bin/bash -c 'echo 0000:01:0a.7 > /sys/bus/pci/devices/0000\\:01\\:0a.7/driver/unbind'
# List new VFs
ExecStart=/usr/bin/lspci -D -d 8086:154c
# Destroy VFs
ExecStop=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens2f0np0/device/sriov_numvfs'
ExecStop=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens2f1np1/device/sriov_numvfs'
# Reload NIC VFs
ExecReload=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens2f0np0/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 0 > /sys/class/net/ens2f1np1/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 8 > /sys/class/net/ens2f0np0/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c 'echo 8 > /sys/class/net/ens2f1np1/device/sriov_numvfs'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 0 mac 76:9e:17:83:39:e5'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 1 mac 46:2c:6d:24:6b:1b'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 2 mac 3e:47:48:12:ed:94'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 3 mac be:e3:6a:f3:8f:ac'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 4 mac 62:8f:3d:bb:02:08'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 5 mac ae:91:57:b9:14:7f'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 6 mac 5a:c2:08:a9:68:a7'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f0np0 vf 7 mac b2:f0:18:af:cb:c5'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 0 mac 16:47:7c:a8:95:98'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 1 mac a6:c7:c5:7f:9c:22'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 2 mac b6:0f:45:34:5e:19'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 3 mac 2a:f7:37:84:31:30'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 4 mac 8a:fa:f8:c5:0b:93'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 5 mac b2:f5:d5:2f:79:06'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 6 mac c2:92:f5:fa:32:20'
ExecReload=/usr/bin/bash -c '/usr/bin/ip link set ens2f1np1 vf 7 mac 2e:fb:29:1e:48:31'
ExecReload=/usr/bin/bash -c 'echo 0000:01:02.1 > /sys/bus/pci/devices/0000\\:01\\:02.1/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:01:02.2 > /sys/bus/pci/devices/0000\\:01\\:02.2/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:01:02.3 > /sys/bus/pci/devices/0000\\:01\\:02.3/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:01:02.4 > /sys/bus/pci/devices/0000\\:01\\:02.4/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:01:02.5 > /sys/bus/pci/devices/0000\\:01\\:02.5/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:01:02.6 > /sys/bus/pci/devices/0000\\:01\\:02.6/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:01:02.7 > /sys/bus/pci/devices/0000\\:01\\:02.7/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:01:0a.1 > /sys/bus/pci/devices/0000\\:01\\:0a.1/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:01:0a.2 > /sys/bus/pci/devices/0000\\:01\\:0a.2/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:01:0a.3 > /sys/bus/pci/devices/0000\\:01\\:0a.3/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:01:0a.4 > /sys/bus/pci/devices/0000\\:01\\:0a.4/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:01:0a.5 > /sys/bus/pci/devices/0000\\:01\\:0a.5/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:01:0a.6 > /sys/bus/pci/devices/0000\\:01\\:0a.6/driver/unbind'
ExecReload=/usr/bin/bash -c 'echo 0000:01:0a.7 > /sys/bus/pci/devices/0000\\:01\\:0a.7/driver/unbind'
ExecReload=/usr/bin/lspci -D -d 8086:154c
[Install]
WantedBy=multi-user.target

Enable with:

systemctl daemon-reload
systemctl enable sriov-vfs.service

Usage:

systemctl start sriov-vfs.service
systemctl stop sriov-vfs.service
systemctl reload sriov-vfs.service
systemctl status sriov-vfs.service

It seems to be working, however, after proxmox boot i get:

Mar 16 08:02:17 pve kernel: iavf 0000:01:02.0: Failed to communicate with PF; waiting before retry
Mar 16 08:02:22 pve kernel: iavf 0000:01:02.0: Hardware came out of reset. Attempting reinit.
Mar 16 08:02:22 pve kernel: iavf 0000:01:02.0: Multiqueue Enabled: Queue pair count = 4
Mar 16 08:02:22 pve kernel: iavf 0000:01:02.0: MAC address: 76:9e:17:83:39:e5
Mar 16 08:02:22 pve kernel: iavf 0000:01:02.0: GRO is enabled
Mar 16 08:02:22 pve kernel: iavf 0000:01:02.0 ens2f0v0: renamed from eth0

I don't know why this is..

after posting this i will modify my script and set new random MAC's, the above MAC's only used for this example.

UPDATE:

I was looking for the command to list the randomly generated MAC addresses for my VF's, in old script for solarflare it was easy..
Turns out that it seems Intel cards do not generate random mac's when partition, as seen by output of the command

ip link show dev ens2f0np0

2: ens2f0np0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master vmbr2 state UP mode DEFAULT group default qlen 1000
    link/ether 24:5e:be:67:fe:a4 brd ff:ff:ff:ff:ff:ff
    vf 0     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 1     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 2     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 3     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 4     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 5     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 6     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    vf 7     link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking on, link-state auto, trust off
    altname enp1s0f0np0

So it seems i have to find a random MAC address generator to set static MAC's
like this one
Previously with solarflare i just copied the one time randomly generated ones and made them permanent

If someone has more info about this please say what's going on.

UPDATE 2:

Seems like some, but not all of the MAC addresses generated by that web page don't work! So now i have to manually test setting the address for each VF, and see if it accepts it. What a pain

 

[heromode]

About the fan situation:
These are the results of my internet research work, which i have spent several hours of my life on.

The QNAP QXG-10G2T-X710 comes as standard with an EVERFLOW model T124010BL(2) fan, rated at 12V , 0.1A.

There is no fan control logic. Only DC 12v with a JST 2mm connector
It has rare asymmetrical mounting holes with spacing of 30mm x 30mm x 23mm:



The problem is that,

this fan is EXTREMELY GOD DAMN LOUD !!!

It is so loud, it penetrates all walls and doors throughout my whole apartement. It is a complete no-go for any homelab, unless that homelab has a dedicated concrete basement for the server stuff.

It sounds like a fighter jet ready for takeoff on an aircraft carrier. (well, almost)

QNAP made a serious error with the design of this cooling system, as they are generally sold as add-in cards for their NAS systems. With this fan, it turns a nice little QNAP NAS box into an annoying little shitbox that's so loud it's practically unusable in a home setting, which they are designed for.

I bought two of these cards used, afaik they've had several owners. One card came with the original fan, the other card came with a replacement fan like this:


I currently have the card with the replacement fan in my proxmox server, and it's very nice, very very quiet. But because this fan has 3 symmetrical mounting holes with a spacing of 26mm x 26mm x 26mm, it doesn't fit on the original heat spreader. So some previous owner has simply glued the fan to the aluminium heat spreader with superglue. And he's done a nice job as well, it looks nice and tidy, precision work all in all. I don't have a picture because the card is in use as i'm writing this.

I have already ordered and received from aliexpress the same fan pictured above, rated at 12V, 0.05A, and some superglue.
I have also ordered a resistor cable color RED which should be slowdown about 50%:



So once i receive the resistor cable, i plan to test it with the original fan and see how it goes, otherwise i will just attempt to glue on the 0.05A fan onto the other card as well, and call it a day. Anyone with knowledge on where to get quiet fans with mounting holes of 30x30x23mm, plus the tiny screws, please let me know

Here is a picture of my currently still unused card with the original fan + replacement fan, just waiting for the resistor cable to test, before finally probably getting the glued on fan on it:





I have attempted to source replacement fans with mounting holes of 30x30x23mm, but they are rare, i have found one ocurrence of this original fan on US amazon, currently unavailable, and one similar fan on aliexpress. But they're all 0.1A, and probably extremely loud.

I should not have spent the 30 minutes of my life even making this post, but if it helps someone else, maybe it was worth it.

 

[pimposh]

The very same fan QNAP uses also for E810/CX4/CX5/CX6/AQC113 flavours of cards therefore your effort may please many users out there.
Great job

 

[marcoi]

About the fan situation:
These are the results of my internet research work, which i have spent several hours of my life on.

The QNAP QXG-10G2T-X710 comes as standard with an EVERFLOW model T124010BL(2) fan, rated at 12V , 0.1A.

There is no fan control logic. Only DC 12v with a JST 2mm connector
It has rare asymmetrical mounting holes with spacing of 30mm x 30mm x 23mm:

View attachment 42522

The problem is that,

this fan is EXTREMELY GOD DAMN LOUD !!!

It is so loud, it penetrates all walls and doors throughout my whole apartement. It is a complete no-go for any homelab, unless that homelab has a dedicated concrete basement for the server stuff.

It sounds like a fighter jet ready for takeoff on an aircraft carrier. (well, almost)

QNAP made a serious error with the design of this cooling system, as they are generally sold as add-in cards for their NAS systems. With this fan, it turns a nice little QNAP NAS box into an annoying little shitbox that's so loud it's practically unusable in a home setting, which they are designed for.

I bought two of these cards used, afaik they've had several owners. One card came with the original fan, the other card came with a replacement fan like this:

View attachment 42523
I currently have the card with the replacement fan in my proxmox server, and it's very nice, very very quiet. But because this fan has 3 symmetrical mounting holes with a spacing of 26mm x 26mm x 26mm, it doesn't fit on the original heat spreader. So some previous owner has simply glued the fan to the aluminium heat spreader with superglue. And he's done a nice job as well, it looks nice and tidy, precision work all in all. I don't have a picture because the card is in use as i'm writing this.

I have already ordered and received from aliexpress the same fan pictured above, rated at 12V, 0.05A, and some superglue.
I have also ordered a resistor cable color RED which should be slowdown about 50%:

View attachment 42524

So once i receive the resistor cable, i plan to test it with the original fan and see how it goes, otherwise i will just attempt to glue on the 0.05A fan onto the other card as well, and call it a day. Anyone with knowledge on where to get quiet fans with mounting holes of 30x30x23mm, plus the tiny screws, please let me know

Here is a picture of my currently still unused card with the original fan + replacement fan, just waiting for the resistor cable to test, before finally probably getting the glued on fan on it:

View attachment 42525

View attachment 42526

I have attempted to source replacement fans with mounting holes of 30x30x23mm, but they are rare, i have found one ocurrence of this original fan on US amazon, currently unavailable, and one similar fan on aliexpress. But they're all 0.1A, and probably extremely loud.

I should not have spent the 30 minutes of my life even making this post, but if it helps someone else, maybe it was worth it.

i just bought a used qnap nas, which has two cards with the same fan and noise of the fan is crazy. I think the original owner never dusted so the fans barrings just went. I saw the same fan so i may just order it and glue it in like you said

I just found these on ebay, order the set since i need to replace 2 fan cards anyways

Lot 4: NEW Evercool 50mm x 10mm 2-pin Video Card Replacement Frameless Fan | eBay

DC Fan Size: 50x50x10mm. Fan Speed: 4500 RPM. Connector: 2 Pin with 6" Tail. Noise Level: 23 DBA.

www.ebay.com

If they work, probably with glue method I will let you know.

 

[heromode]

i just bought a used qnap nas, which has two cards with the same fan and noise of the fan is crazy. I think the original owner never dusted so the fans barrings just went. I saw the same fan so i may just order it and glue it in like you said

I just found these on ebay, order the set since i need to replace 2 fan cards anyways

Lot 4: NEW Evercool 50mm x 10mm 2-pin Video Card Replacement Frameless Fan | eBay

DC Fan Size: 50x50x10mm. Fan Speed: 4500 RPM. Connector: 2 Pin with 6" Tail. Noise Level: 23 DBA.

www.ebay.com

If they work, probably with glue method I will let you know.

Just so happens the resistor cables arrived today. They measure about 47 ohms and reduce the voltage to about 8.3V. And they do reduce the noise level. So much that i will probably just leave the resistor on, instead of gluing the other fan. This card will be in a separate room in a server.

But it doesn't reduce the noise level enough that i could be in the same room as the card. The 0.05A aliexpress fan is whisper quiet.

I find it amazing that manufacturers like QNAP pay zero attention to this, and effectively ruin their products with a $2 fan part. The noise is such that it would be impossible to spend any time in the same room as this fan is spinning 24/7

And there are thousands of fans with the symmetric 26x26x26mm screwholes. There seems to be exactly ONE fan available worldwide with 30x30x23mm.

Whoever chose this fan for the QNAP products is simply STUPID, and should be fired. And the QNAP management should also be fired for allowing this crap.



Anyways, the final thing i plan to test with this card, There is an intel page with links to various software for the x710 chipset, including management software:

Intel® Ethernet Controller X710 - Product Info, Docs, Support and...

i downloaded something from there, and tested with the standard in tree linux kernel module, it could not access the card, just said not an intel product.

Once one machine goes for OS change, i might take the opportunity to test running it with intel's proprietary kernel module, and see if i can access the card with any management software.

Oh one more thing, in addition to the weird kernel message after boot i noted above, which seems to be some linux kernel driver quirk, i get another funny message:

pve kernel: i40e 0000:01:00.0: DCB is not supported for X710-T*L 2.5/5G speeds

 

[heromode]

i just bought a used qnap nas, which has two cards with the same fan and noise of the fan is crazy. I think the original owner never dusted so the fans barrings just went. I saw the same fan so i may just order it and glue it in like you said

I just found these on ebay, order the set since i need to replace 2 fan cards anyways

Lot 4: NEW Evercool 50mm x 10mm 2-pin Video Card Replacement Frameless Fan | eBay

DC Fan Size: 50x50x10mm. Fan Speed: 4500 RPM. Connector: 2 Pin with 6" Tail. Noise Level: 23 DBA.

www.ebay.com

If they work, probably with glue method I will let you know.

Note that those seem to be 0.13A, even higher than the 0.10A original, so they might also be very loud.
I specifically looked for a 0.05A fan, which is what the glued fan is.

Edit: and just noticed, 50mm is too big, they won't fit.

 

[heromode]

Reminder to those ordering the resistor cables: the connector ir JST 2mm, not 2.54mm
In aliexpress the makers of the cables categorize them by color, red being the highest, about 50 ohm.. Not sure if they all do it the same way..

 

[pimposh]

Does this card overheat without it? There are many X710 adapters without a fan, right ?

 

[heromode]

Does this card overheat without it? There are many X710 adapters without a fan, right ?

i have no reading with sensors.

I previously had passively cooled solarflare SFP+ adapters, and i run a quiet server in a ATX case with 140mm fans. They got crazy hot, as in you burned your finger touching the heat sink. Finally one card died, and i got tired of SFP+.

Because i run a quiet server with slow case airflow, i want small fans on my nic's, otherwise the hot heat sink would bake other components in the next pcie slot.

 

[marcoi]

Note that those seem to be 0.13A, even higher than the 0.10A original, so they might also be very loud.
I specifically looked for a 0.05A fan, which is what the glued fan is.

Edit: and just noticed, 50mm is too big, they won't fit.

What size will fit?

 

[heromode]

What size will fit?

36mm

https://www.aliexpress.com/w/wholes...ling-Fan-DC12V.html?spm=a2g0o.detail.search.0

the transparent colored model 4010HS which say 0.05A on the label and has the eternity logo is the one that was glued on my card. I ordered that one, and it will fit, and is very quiet.

 

[marcoi]

36mm

https://www.aliexpress.com/w/wholes...ling-Fan-DC12V.html?spm=a2g0o.detail.search.0

the transparent colored model 4010HS which say 0.05A on the label and has the eternity logo is the one that was glued on my card. I ordered that one, and it will fit, and is very quiet.

ty just order 6 fans from one of the vendors on there. I like to keep some on hand if the other fans die.

 

[blunden]

Does this card overheat without it? There are many X710 adapters without a fan, right ?

The fact that it's a 10GBASE-T model, it might need active cooling.

The SFP+ model can usually get by with passive cooling in my experience, as long as you have at least some airflow.

 

[marcoi]

well i got my fans and so far three of the 6 i tested do not spin when on the nic or qxp qnap cards.
Not sure what the issue is. I have to try the other 3 fans to see if any of them work.
quite annoying if they are all duds.

 

[heromode]

well i got my fans and so far three of the 6 i tested do not spin when on the nic or qxp qnap cards.
Not sure what the issue is. I have to try the other 3 fans to see if any of them work.
quite annoying if they are all duds.

the model 4010HS ones?

 

[marcoi]

Yeah
edit- figu out the issue. Power leads were wrong place. I had to pull them out of plug and reverse them. Now the fans work

 

[heromode]

Yeah

Well i haven't even tested mine yet, but i will in a moment here..

To be honest, of course i can't check the technical specs on the glued on fan, but it looks exactly the same as the 4010HS. same size, clear plastic, same logo. I just assumed it must be the 0.05A one, based on the low noise, and observing the rotation speed when i watched it while powering on the server..

I'll test mine tonight.. report back

 

[pimposh]

Fanboys!

 

[heromode]

And bummer. Mine doesn't spin either.

Sorry for giving bad info, i just assumed it was the same fan. The glued fan has a big eternity logo on the top:

thumbnail-IMG-6108 hosted at ImgBB

So now i don't know which fan would work either. Other option is the resistor cable, it will take down the noise a fair deal.. If you get them, get the highest ohm.. about 50 ohm. that will take the rpm's down about 30%