Solarflare dual 10Gb SFP+

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

WANg

Well-Known Member
Jun 10, 2018
1,302
967
113
46
New York, NY
Anyone know the difference between the S6120 and S7120 card? Are they as good as the Mellanox MCX354A-FCBT or Intel X520-DA2 card?
The S6120 is a 6 series Solarflare card , while the S7120 is a 7 series. The 7 and 8 series are still supported, while the 5 and 6s are considered EOL and do not receive the newer feature-sets. All should be able to use OpenOnload for accelerating 10Gb traffic.
 

mimino

Active Member
Nov 2, 2018
189
70
28
Yeah, @arglebargle went through the angle of setting up Proxmox, setting up VFs off the solarflares and then passing it onto a pfsense VM - it doesn't seem to work. Pretty sure it was a driver issue on the FreeBSD end of things.
Nice thread, too bad I missed it and had to learn the hard way. I could live with 8 VFs limitation, the bigger problem was that I couldn't bring up theVM without getting "Failed to reset the device" error. At this point I gave up and just did the pcie pass through. I picked up another series 7 card though, hoping to have a better luck with newer generation.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
Yeah, @arglebargle went through the angle of setting up Proxmox, setting up VFs off the solarflares and then passing it onto a pfsense VM - it doesn't seem to work. Pretty sure it was a driver issue on the FreeBSD end of things.
I got busy with other things the last couple of months but my conclusion with SR-IOV on any of these cards was basically "install DPDK" and handle >1G networking that way. Drivers just aren't available for SR-IOV on any *nix guest OS but Linux, and even on Linux it's *super* hit or miss getting anything working with VF passthrough that survives guest OS reboots.

The machine I tested on was only somewhat SR-IOV capable though, and I haven't had time to bring up my new HP server to finish testing. I think I'm just going to stick with DPDK for now, the driver situation on FreeBSD was pretty awful and I don't see anything short of pfSense rebasing onto the next major FreeBSD release improving that.
 

mimino

Active Member
Nov 2, 2018
189
70
28
I've obtained a pair of SFN7122F since, and while performing admirably the temperature got me concerned. With IR gun I measured 65C on the board surface, right below the chip. This appears to be the upper limit of its operating temperature :( Didn't want to fry it so I pulled it out, looking for some ways to get better airflow. Can you guys whoever runs this card check and report back what temperature you're observing?

Also, do you know if there's a way to get linux driver (or maybe solarflare utilities) to report the module temperature? I found this on freebsd site, but didn't get the desired output from sfxge driver: ⚙ D5240 sfxge: implement SIOCGI2C to read infromation from phy modules
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
I've been using low profile 40-60mm fans directly attached to the heatsink when I need to cool a network card in a non-server chassis. The cooling on the 40GbE Mellanox card in my desktop is ... it's ugly, but it stays cool enough to operate without issue.

For Mellanox cards I've been able to just use coarse screws to mount fans to heatsinks. I didn't have to do this with the solarflare cards, the thin clients I tested them in had enough airflow with the system fans on high that the NICs stayed safely inside their operating range. You could probably find a way to zip-tie a fan on if you had to.
 

mimino

Active Member
Nov 2, 2018
189
70
28
I've been using low profile 40-60mm fans directly attached to the heatsink when I need to cool a network card in a non-server chassis. The cooling on the 40GbE Mellanox card in my desktop is ... it's ugly, but it stays cool enough to operate without issue.

For Mellanox cards I've been able to just use coarse screws to mount fans to heatsinks. I didn't have to do this with the solarflare cards, the thin clients I tested them in had enough airflow with the system fans on high that the NICs stayed safely inside their operating range. You could probably find a way to zip-tie a fan on if you had to.
Thanks for the input. I might actually have to attach some fan to it if nothing else helps. How did you measure its temperature thought? Did you get any utilities/drivers to report it?
 

ewer0012

Member
Feb 10, 2019
81
25
18
I just picked up 2 of these (Sf329-9021-r7.2 10gb Dual Port Server Adapter Low Profile S6102) the other day to transfer data from my old storage to my new storage. They're working great so far.

Haven't seen the temps get higher than ~35C under full transfer load for hours in the server, and ~38C in the desktop chassis. Temps taken with an IR gun.

I've obtained a pair of SFN7122F since, and while performing admirably the temperature got me concerned. With IR gun I measured 65C on the board surface, right below the chip. This appears to be the upper limit of its operating temperature :( Didn't want to fry it so I pulled it out, looking for some ways to get better airflow. Can you guys whoever runs this card check and report back what temperature you're observing?
Is the temp the same on both cards?

I've been using low profile 40-60mm fans directly attached to the heatsink when I need to cool a network card in a non-server chassis. The cooling on the 40GbE Mellanox card in my desktop is ... it's ugly, but it stays cool enough to operate without issue.
I've had to do the same thing for some of my super old desktop builds in the past, including sometimes using superglue to glue the 4 corners of the heatsink to the fan. Super ugly, but effective. :)
 
  • Like
Reactions: Whaaat

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
Thanks for the input. I might actually have to attach some fan to it if nothing else helps. How did you measure its temperature thought? Did you get any utilities/drivers to report it?
You know, I really don't remember. I think I last played with my pile of 10GbE NICs back in September or October. I'm pretty sure I managed to pull the temperature somehow, I'd grab the SolarFlare driver package from their website and see if it includes a temperature utility.
 

ewer0012

Member
Feb 10, 2019
81
25
18
That could be. When you have multiple devices, it's always best to test them so that you can start establishing a baseline. A quick Google search didn't really return anything on those cards running super hot as the 'norm'. I did see (I believe in this thread) and through my Google search that 65C is the very top of their operating range, so I imagine something is going on.

Give the second card a try, my bet is on a defect of some kind, or maybe the heat sink wasn't properly installed. I had an old MSI card with passive cooling that ran hot enough to cook bacon and couldn't figure out what was going on with it till I physically inspected it and 2 of the screws on one side of the heat sink weren't screwed in all the way, so the heat sink wasn't fully touching the GPU. Maybe you've got the same thing? I've also seen the 'pads' of thermal compound doubled up under the heat sinks, which caused them to run stupid hot.

That's really all I can think of off hand, but I don't want to hijack this thread anymore than I already have. I'd be happy to continue troubleshooting with you in a PM if you need.
 

arglebargle

H̸̖̅ȩ̸̐l̷̦͋l̴̰̈ỏ̶̱ ̸̢͋W̵͖̌ò̴͚r̴͇̀l̵̼͗d̷͕̈
Jul 15, 2018
657
244
43
That could be. When you have multiple devices, it's always best to test them so that you can start establishing a baseline. A quick Google search didn't really return anything on those cards running super hot as the 'norm'. I did see (I believe in this thread) and through my Google search that 65C is the very top of their operating range, so I imagine something is going on.

Give the second card a try, my bet is on a defect of some kind, or maybe the heat sink wasn't properly installed. I had an old MSI card with passive cooling that ran hot enough to cook bacon and couldn't figure out what was going on with it till I physically inspected it and 2 of the screws on one side of the heat sink weren't screwed in all the way, so the heat sink wasn't fully touching the GPU. Maybe you've got the same thing? I've also seen the 'pads' of thermal compound doubled up under the heat sinks, which caused them to run stupid hot.

That's really all I can think of off hand, but I don't want to hijack this thread anymore than I already have. I'd be happy to continue troubleshooting with you in a PM if you need.
I have a pair of those and I can confirm that they do run hot. They're designed for forced chassis airflow, but they ran fine at 50C+ when I was playing with them in my thin clients. I'm almost certain there's a temperature polling utility in the linux driver package, I'm pretty sure I remember polling temperature while I was playing with their 6 and 7 series cards and noting that the 7 series ran much hotter than the 6.
 
  • Like
Reactions: ewer0012

Michaelding313

New Member
May 18, 2021
6
7
3
Apologies for reviving this old thread. It was very helpful for myself while researching these solarflare cards. Allow me to add somethings I wish i knew before now I got my hands on some of these cards.

Ebay listing, which is the "SFC7501" mystery part number. These cards are missing the timing chip (the one on the top right of the PCB, or maybe just an ambient temp sensor) and the PPS connections.

xillinx bought solarflare a while back and it seems like they are hosting the drivers.

The manual is super helpful and that's where I found how the card publish temperatures

Ubuntu 20.04 have sfc drivers pre-installed, and seems to be able to read temperatures just fine using the built in "sensors" package. Example output here:
Code:
$ sensors sfc-pci-0300
sfc-pci-0300
Adapter: PCI adapter
1.2V supply:                          1.20 V  (min =  +1.15 V, max =  +1.25 V)
                                              (crit max =  +1.27 V)
0.9V supply:                          1.02 V  (min =  +0.85 V, max =  +1.20 V)
                                              (crit max =  +1.20 V)
0.9V supply (ext. ADC):             990.00 mV (min =  +0.85 V, max =  +1.20 V)
                                              (crit max =  +1.20 V)
Controller PTAT voltage (ext. ADC): 467.00 mV (min =  +0.00 V, max =  +0.60 V)
                                              (crit max =  +1.00 V)
0.9V die (ext. ADC):                988.00 mV (min =  +0.96 V, max =  +1.02 V)
                                              (crit max =  +1.20 V)
in5:                                     N/A
in6:                                     N/A
in7:                                  1.17 V  (min =  +1.10 V, max =  +1.30 V)
                                              (crit max =  +1.40 V)
in8:                                  1.78 V  (min =  +1.70 V, max =  +1.90 V)
                                              (crit max =  +2.00 V)
Controller heat sink:                    N/A  ALARM
Controller board temp.:              +41.0°C  (low  =  +0.0°C, high = +70.0°C)
                                              (crit = +85.0°C)
Regulator die temp.:                 +64.0°C  (low  =  +0.0°C, high = +90.0°C)
                                              (crit = +105.0°C)
0.9V regulator temp.:                +50.0°C  (low  =  +0.0°C, high = +85.0°C)
                                              (crit = +95.0°C)
1.2V regulator temp.:                +48.0°C  (low  =  +0.0°C, high = +70.0°C)
                                              (crit = +85.0°C)
Controller die temp. (ext. ADC):         N/A  (low  =  +0.0°C, high = +100.0°C)  ALARM
                                              (crit = +115.0°C)
0.9V supply current:                  3.19 A  (min =  +0.00 A, max =  +9.00 A)
                                              (crit max = +10.00 A)
1.2V supply current:                588.00 mA (min =  +0.00 A, max =  +1.60 A)
                                              (crit max =  +2.00 A)
Seems to be missing controller core temps or the like, maybe this can be inferrred from controller board temp? This is with the card just sitting idle without any connections.
I don't have a 6 series card to compare so can't comment on relative temps, judging by the current on supply rails it seems to idle at 5 ish watts. (I hope). In my specific case the card is in a HP elitedesk800 G1 sff, against the powersupply intake. Airflow is non zero but very low.

Hot spot there is probably one of the regulators or something.
1623605574398.png

Heatsink is a poor indiator of temperature, anodized aluminimum is still shiny in thermal.
1623605833386.png

Update, the sfc drivers that come with ubuntu is verion 4.1, which is old. Installing new drivers is a pain on ubuntu server 20.04 at least.
I was able to eventrually install the latest driver that is packaged with onload. That's the only driver package that compiled properly. Redhat source packages from the official drivers don't like multiple "-" seperator returned from uname -r, which, short of compiling and installing a new kernal, is hard to fix.

Clone:

Manuall install libcap-dev on ubuntu systems, (libcap-devel is not a thing in apt)
and run

./scripts/onload_install

as root
 
Last edited:

Michaelding313

New Member
May 18, 2021
6
7
3
Seems like there are many of these cards on ebay with short brackets, and full height brackets are hard to get by itself (it is not standard with other card). My girlfriend drew up printable full height bracket.
Works for holding the card in place. Plastic obviously doesn't provide any emi shielding. Probably not a big deal however.

 

ViciousXUSMC

Active Member
Nov 27, 2016
264
140
43
41
Somewhat a side bar but related.

First thing I was thinking is "ooo a card for PFSense" but it seems that it perhaps does not work well?
The newest upgrade from 2.4.5 to 2.5.1 has broken my Intel NIC for netmap/suricata in inline mode. It crashes the box.

So specifically I need to track down a card that works well with netmap.

Also just a curious question, other than the ease of running fiber, or the "just because" if your using PFSense stricly as a router/firewall for WAN and your WAN is less than 1gbps, any advantage at all using 10gb vs 1gb ethernet? I can't think of any. But I often am in the "just because" camp.
 

Michaelding313

New Member
May 18, 2021
6
7
3
Havent tried this for Pfsense yet

Honestly using 10G for pfsense is probably a net negative if you don't have 10G wan.
More power consumption, worse driver support for cheap old ebay cards just due to age.
Low power 10G cards are $$$, cheap cards are hot. Maybe the connectx 3s are better? I hear they are nice.

Having SFP onboard is a nice to have if you have gigabit wan comming in on fiber, which I do from my ISP. I get a single mode fiber and ISP supplies a bidi transciever of their spec. (It is supposed to plug in to the ISP's box but can be taken out and used).
Maybe a 1G SFP card would be better for that purpose honestly. Power saving would pay a card like that back in a year or 2.

Some arguments could be made if you have 10G intranet that you want routed with firewall, that would fall into the "just because" camp I feel. Leaving the inter-vlan connectivity to the switch is honestly more sensible. Less latency and less power consumption overall.
 
Last edited:

ViciousXUSMC

Active Member
Nov 27, 2016
264
140
43
41
Havent tried this for Pfsense yet

Honestly using 10G for pfsense is probably a net negative if you don't have 10G wan.
More power consumption, worse driver support for cheap old ebay cards just due to age.
Low power 10G cards are $$$, cheap cards are hot. Maybe the connectx 3s are better? I hear they are nice.

Having SFP onboard is a nice to have if you have gigabit wan comming in on fiber, which I do from my ISP. I get a single mode fiber and ISP supplies a bidi transciever of their spec. (It is supposed to plug in to the ISP's box but can be taken out and used).
Maybe a 1G SFP card would be better for that purpose honestly. Power saving would pay a card like that back in a year or 2.

Some arguments could be made if you have 10G intranet that you want routed with firewall, that would fall into the "just because" camp I feel. Leaving the inter-vlan connectivity to the switch is honestly more sensible. Less latency and less power consumption overall.
I do have 10gb intranet for my ESXi server and my main desktop, but I am not using the firewall to inspect traffic between them.
I did have it split out in VLANS once upon a time, but it was nothing but trouble.

IOT, Clients, Servers
Too many IOT devices could not function unless your phone or something is on the same layer 3 network, some of my servers like Plex also were not working right to do local streaming.
PFSense didn't like to give out DHCP to VLANS unless it had a directly connected interface assigned to that IP range.

So in the end I flattened it back out to a Single Layer 3 Domain.

I had clients set the switches vlan interface as the default gateway, and had the switch set the firewall as the default gateway, then the firewall had staic routes configured back to the switches vlan interfaces. It worked technically the way it was supposed too, but not in practice.

the Connect X3 is what I use in my server and my desktop, have not had a problem with them so far and been in use for atleast about 4 or 5 years.
 

Pakna

Member
May 7, 2019
50
3
8
Somewhat a side bar but related.

First thing I was thinking is "ooo a card for PFSense" but it seems that it perhaps does not work well?
The newest upgrade from 2.4.5 to 2.5.1 has broken my Intel NIC for netmap/suricata in inline mode. It crashes the box.

So specifically I need to track down a card that works well with netmap.

Also just a curious question, other than the ease of running fiber, or the "just because" if your using PFSense stricly as a router/firewall for WAN and your WAN is less than 1gbps, any advantage at all using 10gb vs 1gb ethernet? I can't think of any. But I often am in the "just because" camp.
I am running a 10G SMF-only setup with pfSense - WAN is symmetric 1G. I'd be running fibre even if I had a 0.1G WAN.

Aside from nicety of uniformity and consistency, my biggest motivation was that I could cut out the ISP router and directly plug in the SFP ONT into the 10G NIC on the firewall. That alone should account for any possible net power loss, though....honestly, between 1 G copper and fibre, I don't think it's more than a few watts of delta - if that. Unless your electricity provider is sending you power from an orbital satellite via microwave link, you won't notice it on your power bill.

You're also cutting out the electro-optical conversion - not sure if/what latency improvement that confers but there is that.

Next, fibre is impervious to EM intereference or induction noise. Fibre with just a basic 3 mm PVC jacket is, in my experience, at least as resilient as any CAT6. You can go crazy from there and combine aramyd, steel or even kevlar cladding - or any combination of these. These are not only good for outdoor applications but for indoors - as critters/rodents don't care for these materials all that much.

Another thing is you don't care about cable length or cable run - MMF goes a few hundred meters; with SMF you _start_ with 10 km. You could literally wrap the SMF cable around your house and if you didn''t exceed your bend radius, it would work like a charm (but please don't consider doing that).

Additional positive feature is that by running a setup like that your structural cabling is virtually future-proof - for life. You no longer care what speed ISP is providing - you can take it. Even with outrageous network speed increases in the next 20 years, I'd be shocked if we come even close to saturating an OM3/4/5 fibre, let alone an SMF fibre. In fact, I don't even know of a technology that would replace the fibre (speed and capacity-wise).

If there are any downsides to copper - you have to keep termination points fairly clean (for obvious reasons). Also, splicing/termination is a pain (but you shouldn't worry about that as you'll buy factory terminated cables).

In short - if you have the option, go with it. You won't be sorry.
 

ViciousXUSMC

Active Member
Nov 27, 2016
264
140
43
41
I am running a 10G SMF-only setup with pfSense - WAN is symmetric 1G. I'd be running fibre even if I had a 0.1G WAN.

Aside from nicety of uniformity and consistency, my biggest motivation was that I could cut out the ISP router and directly plug in the SFP ONT into the 10G NIC on the firewall. That alone should account for any possible net power loss, though....honestly, between 1 G copper and fibre, I don't think it's more than a few watts of delta - if that. Unless your electricity provider is sending you power from an orbital satellite via microwave link, you won't notice it on your power bill.

You're also cutting out the electro-optical conversion - not sure if/what latency improvement that confers but there is that.

Next, fibre is impervious to EM intereference or induction noise. Fibre with just a basic 3 mm PVC jacket is, in my experience, at least as resilient as any CAT6. You can go crazy from there and combine aramyd, steel or even kevlar cladding - or any combination of these. These are not only good for outdoor applications but for indoors - as critters/rodents don't care for these materials all that much.

Another thing is you don't care about cable length or cable run - MMF goes a few hundred meters; with SMF you _start_ with 10 km. You could literally wrap the SMF cable around your house and if you didn''t exceed your bend radius, it would work like a charm (but please don't consider doing that).

Additional positive feature is that by running a setup like that your structural cabling is virtually future-proof - for life. You no longer care what speed ISP is providing - you can take it. Even with outrageous network speed increases in the next 20 years, I'd be shocked if we come even close to saturating an OM3/4/5 fibre, let alone an SMF fibre. In fact, I don't even know of a technology that would replace the fibre (speed and capacity-wise).

If there are any downsides to copper - you have to keep termination points fairly clean (for obvious reasons). Also, splicing/termination is a pain (but you shouldn't worry about that as you'll buy factory terminated cables).

In short - if you have the option, go with it. You won't be sorry.
Will do if I can, looked at my ONT and dont see any SFP type connections to use, looks like raw optical wire going into some kind of laser termination area. I have to unscrew it and see what it looks like inside but dont want to break anything lol.
 

TheServerGuy

Member
Dec 18, 2016
48
38
18
My Centurylink "modem" has an SFP connection available, fwiw. I don't use it though, running ethernet straight from ONT to pfsense and works fine with pppoe.
 

Pakna

Member
May 7, 2019
50
3
8
Will do if I can, looked at my ONT and dont see any SFP type connections to use, looks like raw optical wire going into some kind of laser termination area. I have to unscrew it and see what it looks like inside but dont want to break anything lol.
Some routers (like Nokia's) will have an LC connector, some non-standard and some (like Actiontec) will come with an SFP port. I believe ISP should be able to accomodate your needs. I would not unscrew anything unless it's last resort and you're ready to potentially be charged for a new router (that may be expensive).