(US) 90 dollar Wyse 5070 Thin client/mini-server?

yrxuthst

New Member
May 20, 2020
12
16
3
Really? Goldmont Atoms can do at least 16GB per DIMM?
Intel traditionally gimps the memory controllers on the consumer Atoms.

Got proof?
Here is one for the 5070 specifically: The WYSE 5070, a perfect little VyOS device – blog.kroy.io

Additionally for other J5005 boards:
 
  • Like
Reactions: Samir and hk92doom

hk92doom

Member
Jun 4, 2020
43
34
18
but nothing about 32 or beyond.
2x16GB of RAM for NUC and ASRock boards, built around the J5005/J4105, is nothing new. I think that whether it will work depends on the type of RAM that people have.

https://www.servethehome.com/intel-pentium-silver-j5005-benchmarks-and-review/
https://develmonk.com/2019/08/02/asrock-j4105-itx-16gb-and-32gb-memory-configuration-tests/
https://www.reddit.com/r/homelab/comments/flqcs6/asrock_j4105itx_32gb_success/?sort=old
https://www.reddit.com/r/HomeServer/comments/e97j4d/asrock_j5005itx_for_a_low_power_nas/?sort=old

Will this work for Wyse 5070 Extended? I'd love to hear it.
Some guy who normally runs 32GB on the J5005 server, if I understand correctly, tried increase it to 64GB o_O
Unfortunately, the board refused to boot.

 
  • Like
Reactions: Samir

WANg

Well-Known Member
Jun 10, 2018
979
578
93
2x16GB of RAM for NUC and ASRock boards, built around the J5005/J4105, is nothing new. I think that whether it will work depends on the type of RAM that people have.

https://www.servethehome.com/intel-pentium-silver-j5005-benchmarks-and-review/
https://develmonk.com/2019/08/02/asrock-j4105-itx-16gb-and-32gb-memory-configuration-tests/
https://www.reddit.com/r/homelab/comments/flqcs6/asrock_j4105itx_32gb_success/?sort=old
https://www.reddit.com/r/HomeServer/comments/e97j4d/asrock_j5005itx_for_a_low_power_nas/?sort=old

Will this work for Wyse 5070 Extended? I'd love to hear it.
Some guy who normally runs 32GB on the J5005 server, if I understand correctly, tried increase it to 64GB o_O
Unfortunately, the board refused to boot.

Here is one for the 5070 specifically: The WYSE 5070, a perfect little VyOS device – blog.kroy.io

Additionally for other J5005 boards:
Huh. Interesting. I stand corrected, then. 32GB DDR4 looks verified for the Goldmont+ Atoms, anything over 32 still looks to be unknown.

So this leads me to, as us Americans would put it “put our money where our mouth is”. Several things that needs to be done -

a) Test 16GB DIMM compatibility with the Wyse 5070, both as shipped, and with the latest BIOS. That can wait until later. I might have spares laying around at work.

b) Test 32GB DIMM compatibilty, both stand-alone, paired and in conjunction with the 2, 4 and 16GB configuration (yielding 34, 36 and 48GB configurations), once again with different BIOS configuration in play.

So here’s the tough, possibly painful question of testing 16/32GB SODIMM configuration with the 5070, and with the HP t740 thin client (which I bought earlier in the year at around 375 USD). The Ryzen embedded are said to have an official RAM ceiling of 32GB, which might or might not be true. If it’s based on the typical Raven Ridge memory controllers they should be able to support 32GB SODIMM units (so 64GB RAM should be possible). There are secondary vendors selling customized m715q Gen 2s on Amazon with 64GB RAM, and in terms of the embedded Ryzen there seem to be confirmation of 64GB RAM support from VirtuallyGhetto. If that’s the case I am going to pick up a pair of DDR4-2666 DIMM units (roughly 255 USD) and use that as part of the test. If the Wyse won’t take them, then the HP should.
 
Last edited:
  • Like
Reactions: Samir and hk92doom

hk92doom

Member
Jun 4, 2020
43
34
18
For the sake of completeness...
Users reported Dual Rank 2Rx8 memory to be compatible with J5005/J4105
260-Pin, SODIMM, non-ECC

Kingston ValueRAM stick (KVR24S17D8/16)
Corsair CMSX32GX4M2A2400C16 Vengeance 32GB kit (2x16GB)
Crucial CT2K16G4SFD824A 32GB kit (2x16GB)

ASRock J5005-ITX Memory QVL
DDR4 Corsair 3200 2400 16GB CMSX32GX4M2B3200C16
DDR4 Corsair 2666 2400 16GB CMSX64GX4M4A2666C18
DDR4 Crucial 2400 2400 16GB CT16G4SFD824A.16FB1
DDR4 Crucial 2133 2133 16GB MTA16ATF2G64HZ-2G1A1
DDR4 G.Skill 2666 2400 16GB F4-2666C18D-32GRS
DDR4 G.Skill 2133 2133 16GB F4-2133C15S-16GRS

*2GB DRAM per module is not supported.
 
Last edited:
  • Like
Reactions: Samir

WANg

Well-Known Member
Jun 10, 2018
979
578
93
Okay - so I could confirm that the 5070 showed up at my doorsteps this afternoon.

Unboxing 1.png

The machine is relatively new (production date of November 2019), and comes with its accessories (keyboard, mouse, power brick, stand). Below is a comparison with an HP t740. If this was a WD5070 extended model it'll be roughly the same thickness of the t740.

t740 comparison.png

Here is a teaser image from the teardown -

Opening it up 2.png

So things of interest -
a) Uses the Dell 7.4x5 center positive power brick (and can tolerate the HP 7.4x5 as well) commonly found on the Latitude 74x0 series
b) Passively cooled
c) The t5070 extended uses the same motherboard (you can see traces of the PCIe riser between the CR2032 clock battery and the chassis edge, there is a 48 pin breakout supporting a PCIe x8 riser slot), so yes, if you can find a CTO version on CDW on clearance pricing, go for it.
d) There's a second PCIe slot for a WiFi card or (possibly) an ethernet breakout
e) There are signs suggesting that SRIOV might also work.

Hysteresis Testing 1.png
Hysteresis Testing 2.png

The goal is to move the testing write-up to its own forum thread, and then followed by a more professional write-up.
 
Last edited:
  • Like
Reactions: Samir and hk92doom

hk92doom

Member
Jun 4, 2020
43
34
18
Okay - so I could confirm that the 5070 showed up at my doorsteps this afternoon.
...
The goal is to move the testing write-up to its own forum thread, and then followed by a more professional write-up.
I can't wait to see your review and setup guide.
 
  • Like
Reactions: Samir

WANg

Well-Known Member
Jun 10, 2018
979
578
93
It looks like yours did not include the M.2 standoff either. It needs a 6-32 outer thread, with a 4mm standoff.

This is a pretty common size for motherboard standoffs, you may have some laying around.

Here are the ones I used: https://www.amazon.com/dp/B00213KL5I/

These are 6mm but can be filed down.
Is it supposed to?

One of my already developing pet peeves about the 5070 is how it does not come with the M.2 retaining standoff. On my t740 it's in place and ready to go even for the unpopulated SATA slot. I mean, sure, it saves a few bucks, but c'mon, seriously?


t740+PCIe+Latch+Opened.jpg
 
  • Like
Reactions: Samir

WANg

Well-Known Member
Jun 10, 2018
979
578
93
I think most of Dell's stuff doesn't come with the standoff unless it has the m.2 drive, just mentioned it since my 5070 Extended didn't come with it either.
How much did that 5070 extended ending up costing you? I want to do a pricing comparison here.
 
  • Like
Reactions: Samir

WANg

Well-Known Member
Jun 10, 2018
979
578
93
I got it from CDW Outlet a couple weeks ago for $90 + shipping, same price as this one.
Which one did you order? Did the listing say "5070 CTO" and you lucked out with the extended model? Or does the listing explicitly say "5070 Extended" specifically?
 
  • Like
Reactions: Samir

yrxuthst

New Member
May 20, 2020
12
16
3
Which one did you order? Did the listing say "5070 CTO" and you lucked out with the extended model? Or does the listing explicitly say "5070 Extended" specifically?
The listing I got was titled "WYSE 5070 TC EXTENDED FCTO (BSTK)"
 
  • Like
Reactions: Samir

WANg

Well-Known Member
Jun 10, 2018
979
578
93
The listing I got was titled "WYSE 5070 TC EXTENDED FCTO (BSTK)"
Good to know.
What is your 5070 “big” currently running? I am doing a forum thread (and possibly something else) on the 5070 and I need to know if SRIOV is working or not. One of the big negative on the t730 is it’s inability to safely do SRIOV (no PCI Access Control Services to do PCIe device segregation), and on the t740, it’s functional but limited to 7 VFs in total (no Alternative Routing-ID Interpretation). In both cases its equivalent (VMDirectPath) doesn’t seem to work with VMWare ESXi. SRIOV VFs are super-useful on applications where you can, say, do line-speed virtualized firewalls...

I saw the proper flags enabled in the BIOS with Proxmox, but until I find a way to solder a PCIe x8 riser to the board or find another “big” on the cheap, I can’t exactly test it on the real.
 
Last edited:
  • Like
Reactions: Samir

yrxuthst

New Member
May 20, 2020
12
16
3
Good to know.
What is your 5070 “big” currently running? I am doing a forum thread (and possibly something else) on the 5070 and I need to know if SRIOV is working or not. One of the big negative on the t730 is it’s inability to safely do SRIOV (no PCI Access Control Services to do PCIe device segregation), and on the t740, it’s functional but limited to 7 VFs in total (no Alternative Routing-ID Interpretation). In both cases its equivalent (VMDirectPath) doesn’t seem to work with VMWare ESXi. SRIOV VFs are super-useful on applications where you can, say, do line-speed virtualized firewalls...

I saw the proper flags enabled in the BIOS with Proxmox, but until I find a way to solder a PCIe x8 riser to the board or find another “big” on the cheap, I can’t exactly test it on the real.
This took a bit as I have not used SR-IOV before, let me know if you need more information on anything here, or have other tests you'd like me to run.

Host OS: OpenSuse Tumbleweed, kernel 5.7.2-1-default.
Network card: SolarFlare SFN7002f

sfboot switch-mode=sriov pf-count=1 vf-count=16. The card supports up to 16 PFs and up to 240 VFs per PF.

I enabled VT-d in setup, and added kernel parameters intel_iommu=on pci=realloc. Without the second parameter, I get the error not enough MMIO resources for SR-IOV when I tried to activate the VFs, which seems to indicate that the firmware does not allocate resources properly for SR-IOV. I'm not sure if ESXI has an equivalent to this.

echo 16 > /sys/class/net/em2/device/sriov_numvfs. Result:

Code:
00:00.0 Host bridge: Intel Corporation Gemini Lake Host Bridge (rev 03)
00:00.3 System peripheral: Intel Corporation Device 3190 (rev 03)
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 605 (rev 03)
00:0c.0 Network controller: Intel Corporation Device 31dc (rev 03)
00:0e.0 Audio device: Intel Corporation Celeron/Pentium Silver Processor High Definition Audio (rev 03)
00:0f.0 Communication controller: Intel Corporation Celeron/Pentium Silver Processor Trusted Execution Engine Interface (rev 03)
00:12.0 SATA controller: Intel Corporation Device 31e3 (rev 03)
00:13.0 PCI bridge: Intel Corporation Gemini Lake PCI Express Root Port (rev f3)
00:14.0 PCI bridge: Intel Corporation Gemini Lake PCI Express Root Port (rev f3)
00:15.0 USB controller: Intel Corporation Device 31a8 (rev 03)
00:17.0 Signal processing controller: Intel Corporation Device 31b4 (rev 03)
00:17.3 Signal processing controller: Intel Corporation Device 31ba (rev 03)
00:1c.0 SD Host controller: Intel Corporation Celeron/Pentium Silver Processor SDA Standard Compliant SD Host Controller (rev 03)
00:1f.0 ISA bridge: Intel Corporation Device 31e8 (rev 03)
00:1f.1 SMBus: Intel Corporation Celeron/Pentium Silver Processor Gaussian Mixture Model (rev 03)
01:00.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (rev 01)
01:00.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (rev 01)
01:02.2 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:02.3 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:02.4 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:02.5 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:02.6 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:02.7 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.2 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.3 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.4 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.5 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.6 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.7 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:04.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:04.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
lspci -vvv -s 01:00.0
Code:
01:00.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (rev 01)
        Subsystem: Solarflare Communications SFN7x02F-R3 Flareon 7000 Series 10G Adapter
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 22
        IOMMU group: 12
        Region 0: I/O ports at e100 [size=256]
        Region 2: Memory at 80000000 (64-bit, non-prefetchable) [size=8M]
        Region 4: Memory at 89080000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at 89000000 [disabled] [size=256K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <64ns, L1 <8us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 25.000W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (downgraded), Width x4 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp+ ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00002000
        Capabilities: [d0] Vital Product Data
                Product Name: Solarflare Flareon 7000 Series 10G Adapter
                Read-only fields:
                        [PN] Part number: SFN7x02F
                        [SN] Serial number: 750000305371150257100264
                        [EC] Engineering changes: PCBR3:CCSA1
                        [VD] Vendor specific: 6.5.0
                        [V0] Vendor specific: 6.5.0
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                End
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [140 v1] Device Serial Number 00-0f-53-ff-ff-2e-2c-30
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
                IOVSta: Migration-
                Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 2, stride: 1, Device ID: 1903
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 0000000081000000 (64-bit, non-prefetchable)
                Region 2: Memory at 0000000089084000 (64-bit, non-prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [1c0 v1] Transaction Processing Hints
                Device specific mode supported
                No steering table available
        Capabilities: [290 v1] Latency Tolerance Reporting
                Max snoop latency: 3145728ns
                Max no snoop latency: 3145728ns
        Kernel driver in use: sfc
        Kernel modules: sfc
 
  • Like
Reactions: Samir and hk92doom

WANg

Well-Known Member
Jun 10, 2018
979
578
93
This took a bit as I have not used SR-IOV before, let me know if you need more information on anything here, or have other tests you'd like me to run.

Host OS: OpenSuse Tumbleweed, kernel 5.7.2-1-default.
Network card: SolarFlare SFN7002f

sfboot switch-mode=sriov pf-count=1 vf-count=16. The card supports up to 16 PFs and up to 240 VFs per PF.

I enabled VT-d in setup, and added kernel parameters intel_iommu=on pci=realloc. Without the second parameter, I get the error not enough MMIO resources for SR-IOV when I tried to activate the VFs, which seems to indicate that the firmware does not allocate resources properly for SR-IOV. I'm not sure if ESXI has an equivalent to this.

echo 16 > /sys/class/net/em2/device/sriov_numvfs. Result:

Code:
00:00.0 Host bridge: Intel Corporation Gemini Lake Host Bridge (rev 03)
00:00.3 System peripheral: Intel Corporation Device 3190 (rev 03)
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 605 (rev 03)
00:0c.0 Network controller: Intel Corporation Device 31dc (rev 03)
00:0e.0 Audio device: Intel Corporation Celeron/Pentium Silver Processor High Definition Audio (rev 03)
00:0f.0 Communication controller: Intel Corporation Celeron/Pentium Silver Processor Trusted Execution Engine Interface (rev 03)
00:12.0 SATA controller: Intel Corporation Device 31e3 (rev 03)
00:13.0 PCI bridge: Intel Corporation Gemini Lake PCI Express Root Port (rev f3)
00:14.0 PCI bridge: Intel Corporation Gemini Lake PCI Express Root Port (rev f3)
00:15.0 USB controller: Intel Corporation Device 31a8 (rev 03)
00:17.0 Signal processing controller: Intel Corporation Device 31b4 (rev 03)
00:17.3 Signal processing controller: Intel Corporation Device 31ba (rev 03)
00:1c.0 SD Host controller: Intel Corporation Celeron/Pentium Silver Processor SDA Standard Compliant SD Host Controller (rev 03)
00:1f.0 ISA bridge: Intel Corporation Device 31e8 (rev 03)
00:1f.1 SMBus: Intel Corporation Celeron/Pentium Silver Processor Gaussian Mixture Model (rev 03)
01:00.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (rev 01)
01:00.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (rev 01)
01:02.2 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:02.3 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:02.4 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:02.5 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:02.6 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:02.7 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.2 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.3 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.4 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.5 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.6 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:03.7 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:04.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
01:04.1 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (Virtual Function) (rev 01)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)
lspci -vvv -s 01:00.0
Code:
01:00.0 Ethernet controller: Solarflare Communications SFC9120 10G Ethernet Controller (rev 01)
        Subsystem: Solarflare Communications SFN7x02F-R3 Flareon 7000 Series 10G Adapter
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 22
        IOMMU group: 12
        Region 0: I/O ports at e100 [size=256]
        Region 2: Memory at 80000000 (64-bit, non-prefetchable) [size=8M]
        Region 4: Memory at 89080000 (64-bit, non-prefetchable) [size=16K]
        Expansion ROM at 89000000 [disabled] [size=256K]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [70] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 2048 bytes, PhantFunc 0, Latency L0s <64ns, L1 <8us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 25.000W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s (downgraded), Width x4 (downgraded)
                        TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range ABCD, TimeoutDis+ NROPrPrP- LTR+
                         10BitTagComp- 10BitTagReq- OBFF Via message/WAKE#, ExtFmt- EETLPPrefix-
                         EmergencyPowerReduction Not Supported, EmergencyPowerReductionInit-
                         FRS- TPHComp+ ExtTPHComp-
                         AtomicOpsCap: 32bit- 64bit- 128bitCAS-
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis- LTR+ OBFF Disabled,
                         AtomicOpsCtl: ReqEn-
                LnkCap2: Supported Link Speeds: 2.5-8GT/s, Crosslink- Retimer- 2Retimers- DRS-
                LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete- EqualizationPhase1-
                         EqualizationPhase2- EqualizationPhase3- LinkEqualizationRequest-
                         Retimer- 2Retimers- CrosslinkRes: unsupported
        Capabilities: [b0] MSI-X: Enable+ Count=32 Masked-
                Vector table: BAR=4 offset=00000000
                PBA: BAR=4 offset=00002000
        Capabilities: [d0] Vital Product Data
                Product Name: Solarflare Flareon 7000 Series 10G Adapter
                Read-only fields:
                        [PN] Part number: SFN7x02F
                        [SN] Serial number: 750000305371150257100264
                        [EC] Engineering changes: PCBR3:CCSA1
                        [VD] Vendor specific: 6.5.0
                        [V0] Vendor specific: 6.5.0
                        [RV] Reserved: checksum good, 0 byte(s) reserved
                End
        Capabilities: [100 v2] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
                AERCap: First Error Pointer: 00, ECRCGenCap+ ECRCGenEn- ECRCChkCap+ ECRCChkEn-
                        MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
                HeaderLog: 00000000 00000000 00000000 00000000
        Capabilities: [140 v1] Device Serial Number 00-0f-53-ff-ff-2e-2c-30
        Capabilities: [150 v1] Alternative Routing-ID Interpretation (ARI)
                ARICap: MFVC- ACS-, Next Function: 1
                ARICtl: MFVC- ACS-, Function Group: 0
        Capabilities: [160 v1] Secondary PCI Express
                LnkCtl3: LnkEquIntrruptEn- PerformEqu-
                LaneErrStat: 0
        Capabilities: [180 v1] Single Root I/O Virtualization (SR-IOV)
                IOVCap: Migration-, Interrupt Message Number: 000
                IOVCtl: Enable- Migration- Interrupt- MSE- ARIHierarchy+
                IOVSta: Migration-
                Initial VFs: 16, Total VFs: 16, Number of VFs: 0, Function Dependency Link: 00
                VF offset: 2, stride: 1, Device ID: 1903
                Supported Page Size: 00000553, System Page Size: 00000001
                Region 0: Memory at 0000000081000000 (64-bit, non-prefetchable)
                Region 2: Memory at 0000000089084000 (64-bit, non-prefetchable)
                VF Migration: offset: 00000000, BIR: 0
        Capabilities: [1c0 v1] Transaction Processing Hints
                Device specific mode supported
                No steering table available
        Capabilities: [290 v1] Latency Tolerance Reporting
                Max snoop latency: 3145728ns
                Max no snoop latency: 3145728ns
        Kernel driver in use: sfc
        Kernel modules: sfc
Huh. Interesting. This looks similar to my experiences with the t740, Proxmox 6 (Debian based) and the Flareon SFN7322F (it's basically the same as the SFN7122F, except the hardware ptp stamping is factory enabled). The important distinction is that when I tried to allocate more than 7 VFs the t740 will puke out a type ff on all devices beyond 0:01:07 (meaning that SRIOV spiked all access to all VFs beyond the 7th). It doesn't look like this is happening with yours. That being said, what it says on init and what it actually does when you try to use it...could be very different. The Mellanox ConnectX3 VPI card that I use (MCX354-FCBT) also reported successful creation of VFs up to and beyond 7...until I tried to use it, then it flagged it as type ff and shut off access to the extra VFs. You might want to install KVM on that Linux instance and create a VM with PCI offloading on a VF just to see if it works, both from 01.00.1 all the way to 01.00.7 and above. I wish that this stock 5070 have some embedded hardware that I can try to offload using SRIOV.

Oh - those Flareons tend to run hot - you will want to use a PCIe slot extender to keep it as far away from the thin client hardware as you can. I had hard lockups and crashes when fitted on the t730, and I didn't want to take chances with the t740. The same is probably a good idea with the 5070 "big".

 
  • Like
Reactions: Samir

yrxuthst

New Member
May 20, 2020
12
16
3
You might want to install KVM on that Linux instance and create a VM with PCI offloading on a VF just to see if it works, both from 01.00.1 all the way to 01.00.7 and above.
I spun up 8 VMs (vfio-pci/q35) with those VFs, and they all have a network connection. So that part seems to be working fine.
 
  • Like
Reactions: Samir

WANg

Well-Known Member
Jun 10, 2018
979
578
93
I spun up 8 VMs (vfio-pci/q35) with those VFs, and they all have a network connection. So that part seems to be working fine.
Any issues allocating/deallocating/reallocating VFs to those VMs? And what OSes are you using on those VM guest?
 
  • Like
Reactions: Samir

yrxuthst

New Member
May 20, 2020
12
16
3
Any issues allocating/deallocating/reallocating VFs to those VMs? And what OSes are you using on those VM guest?
None that I saw. I created the VFs as above, assigned them in virt-manager, then started the whole lot. After they launched I verified that they got a DHCP address and I could ping them. I just used an Ubuntu cloud image for the OS.
 
  • Like
Reactions: Samir