SFP choices for use with Supermicro X10SDV-7TP4F

[RepairBill]

Hi! I am evaluating a Supermicro X10SDV-7TP4F motherboard as part of a 5018D-MHR7N4P system, and I have a question about using the SFP+ ports.

This motherboard has 2x 1GBE Ethernet ports on RJ-45s, and two empty SFP+ cages. The SFP+ ports are said to be supported directly the Intel D-1500 SOC. The goal is to make all four Ethernet ports available to VMware on this system.

My problem is that I don't know which interconnects or drivers I need to use in VMware to support the SFP+ interfaces. I tried installing a 10Gtek ASF-GE-T SFP module and running RHEL 6.9 as a first test, but RHEL complains that it does not have the correct driver for the device. Realizing that I don't even know the part number of the Ethernet device that I need a driver for, I decided to ask here and see if anyone who has experience with this hardware can share some experience or knowledge about a setup like this one.

TIA, Bill

 

[Patrick]

Usually, DACs work without issue. I have Intel coded SFP+'s from FS.com, I believe, in the D-1500 systems.

 

[RepairBill]

Usually, DACs work without issue. I have Intel coded SFP+'s from FS.com, I believe, in the D-1500 systems.

Hi Patrick. I could try using DACs instead of interface modules such as the one I mentioned, All I would need is a switch that had SFP+ and 1GBE copper to connect to the other devices on the network, perhaps with link aggregation. Ideally I could find a switch with, say, 6 SFP+ ports and 12 GBE ports to connect servers to the networks - no luck finding such a switch yet, as most have only two SFP ports or all SFP ports.

If you have used DAC connections with VMware on these systems, do you happen to know which driver(s) are needed? And do these drivers ship with VMware, or do they need to be obtained separately from from Intel, Supero, or VMware?

Thanks!
--Bill

 

[fohdeesha]

Ideally I could find a switch with, say, 6 SFP+ ports and 12 GBE ports to connect servers to the networks - no luck finding such a switch yet,

do you care about buying used? https://forums.servethehome.com/index.php?threads/brocade-icx6450-icx6610-etc.21107/

 

[RepairBill]

do you care about buying used? https://forums.servethehome.com/index.php?threads/brocade-icx6450-icx6610-etc.21107/

As long as the vendor is reputable, and the price is low enough that I can afford to buy spares, it's definitely something I would consider. I still have to figure out if it makes more sense to buy copper converters or additional switches.
Thanks for the tip!

 

[compuguy]

Hi! I am evaluating a Supermicro X10SDV-7TP4F motherboard as part of a 5018D-MHR7N4P system, and I have a question about using the SFP+ ports.

This motherboard has 2x 1GBE Ethernet ports on RJ-45s, and two empty SFP+ cages. The SFP+ ports are said to be supported directly the Intel D-1500 SOC. The goal is to make all four Ethernet ports available to VMware on this system.

My problem is that I don't know which interconnects or drivers I need to use in VMware to support the SFP+ interfaces. I tried installing a 10Gtek ASF-GE-T SFP module and running RHEL 6.9 as a first test, but RHEL complains that it does not have the correct driver for the device. Realizing that I don't even know the part number of the Ethernet device that I need a driver for, I decided to ask here and see if anyone who has experience with this hardware can share some experience or knowledge about a setup like this one.

TIA, Bill

Yeah, I've been trying to figure this out on a similar Xeon-D server from Supermicro, the SYS-E300-8D. First thing to know is that version of RHEL may be too old to recognize those SFP ports (They come up on newer Linux kernels as Intel Corporation Device 15ac). In Windows they are recognized as Intel x552/x557). You are either going to have to try a newer kernel (RHEL/Centos 7) or compile a newer release of the igxbe drivers.

On the subject of gigabit sfp transceivers, I haven't come across one that works at all yet. I bought a 6com 6C-SFP-M RJ45 adapter that identifies itself as a Intel module, a E1GSFPT-A (whatever that is). It recognizes the module, but the link light on the back of the server never goes green. I would recommend you follow these other STH threads, because other people are having similar problems with the Xeon-D SFP ports:

https://forums.servethehome.com/ind...ibility-intel-x553-mellanox-connectx-2.21477/
https://forums.servethehome.com/index.php?threads/connecting-xeon-d-10g-base-t-to-sfp-switch.16319/

On the Vmware ESXi front, Tinkertry has a guide to installing a vib to get the SFP ports working: How to download and install the Intel Xeon D 10GbE X552/X557 driver/VIB for VMware ESXi 6.x, works with the X540 PCIe card too | TinkerTry IT @ Home

Usually, DACs work without issue. I have Intel coded SFP+'s from FS.com, I believe, in the D-1500 systems.

It looks like these Xeon-D SFP/SFP+ ports have a high chance of working if its Intel coded. The next best thing is Finisar or Avago coded SFP/SFP+ transceivers. I had success with a Finisar and a Avago (Foxconn) made Cisco branded SFP-10G-SR. I'm still looking for a working Gigabit RJ45 SFP module. Supermicro support has been mixed. One person told me a Finisar SFP fiber module worked with my board. The other told me that it doesn't support SFP modules, only SFP+.

 

[RepairBill]

I will continue to do research and experiment with various options including a DAC connection to a switch with SFP or SFP+ ports. Thanks to all for the comments and links. I would love to hear about any additional experiences that other may share. It looks like this will be a medium to long-term project ;-)

The gigabit SFP module that I tried is Intel-coded, and may just work with the right drivers - I'll report back once I have time to do further experiments along those lines.

I have raised this issue with Supermicro support, but have received no response so far. It's disappointing that they do not provide documentation or a tested devices list for their SFP+ ports on this motherboard. Hopefully their support folks will be able to find someone in the company that knows what was tested and what works. I guess this is a good argument for why IEEE standards such as 802.3 are more desirable than the less formal manufacturer agreements that led to the current situation with SFP/SFP+ implementations.

 

[RepairBill]

Supermicro support has responded that they have not validated any 1GB SFP transceivers, but suggested that I try the Aquanta AQS-109. Unfortunately, this part does not seem to be generally available - I found one listing, but with no inventory and an estimated 12 week lead time.

They did state that the board is compatible with their SSE-F3548SR switch, but a search does not turn up any mention of that switch at resellers or dealers. The datasheet is here: https://www.supermicro.com/datasheet/datasheet_SSE-F3548S.pdf

Does anyone have any experience with the Aquanta AQS-109 transceiver or the Supermicro SSE-F3548S switch?

Thanks again to compuguy for the links and Patrick for hosting this great site. - it looks hopeful that there may be a way to connect these SFP+ ports to my networks, but a lot of experimentation will be required, and there will probably be leftover parts to sell on eBay ;-)

 

[compuguy]

I will continue to do research and experiment with various options including a DAC connection to a switch with SFP or SFP+ ports. Thanks to all for the comments and links. I would love to hear about any additional experiences that other may share. It looks like this will be a medium to long-term project ;-)

The gigabit SFP module that I tried is Intel-coded, and may just work with the right drivers - I'll report back once I have time to do further experiments along those lines.

I have raised this issue with Supermicro support, but have received no response so far. It's disappointing that they do not provide documentation or a tested devices list for their SFP+ ports on this motherboard. Hopefully their support folks will be able to find someone in the company that knows what was tested and what works. I guess this is a good argument for why IEEE standards such as 802.3 are more desirable than the less formal manufacturer agreements that led to the current situation with SFP/SFP+ implementations.

It took a bit of back and forth, but one person from Supermicro support told me the server doesn't support SFP. Another support person recommended me to try a Finisar FTLF8524P2BNV SFP module: FTLF8524P2BNV Finisar Corporation | Optoelectronics | DigiKey

Supermicro support has responded that they have not validated any 1GB SFP transceivers, but suggested that I try the Aquanta AQS-109. Unfortunately, this part does not seem to be generally available - I found one listing, but with no inventory and an estimated 12 week lead time.

They did state that the board is compatible with their SSE-F3548SR switch, but a search does not turn up any mention of that switch at resellers or dealers. The datasheet is here: https://www.supermicro.com/datasheet/datasheet_SSE-F3548S.pdf

Does anyone have any experience with the Aquanta AQS-109 transceiver or the Supermicro SSE-F3548S switch?

Thanks again to compuguy for the links and Patrick for hosting this great site. - it looks hopeful that there may be a way to connect these SFP+ ports to my networks, but a lot of experimentation will be required, and there will probably be leftover parts to sell on eBay ;-)

Supermicro support mentioned to me that the AOM-AQS-107-B0C2-CX they sell in the Supermicro Store should work: Aquantia 10G/s High Speed SFP+ Ethernet Optical Transceiver.

 

[RepairBill]

Thanks for the pointers to the optical transceivers. I may have to go that route, but my first goal is to find a way to use the SFP+ ports to interoperate with an existing 1GBE copper network. Hope to have time in the next day or two to run copper transceiver tests. Support did inform me today that the SFP+ interfaces on the SOC should appear as an Intel X550 controller, so that will help in identifying drivers.

 

[Blinky 42]

I know the Fiberstore Copper SFP+ modules work when coded as Intel. I have a few and use them in Intel cards and Supermicro boards with SFP+ cages - they link up and work fine at 10G and 1G speeds in my X10SDV-TP8F

They are expensive however Customized 10GbEBASE-T Copper SFP+ RJ45 Transceiver | FS.COM
The LR and SR 1G/10G modules coded as Intel also work and are a lot less expensive
Intel E10GSFPSR 1000BASE-SX and 10GBASE-SR SFP+ Transceiver 300m | FS.COM
Intel E10GSFPLR 1000BASE-LX and 10GBASE-LR SFP+ Transceiver 10km | FS.COM

The 1G copper Mikrotik modules *DO NOT* work in the Supermicro motherboards (so far). In Linux the driver sees an unknown vendor on the SFP+ and disables the port. I have a few of the 10G Mikrotik modules but they are in use elsewhere and I don't recall if I tested them in the SM motherboards before putting them in a Juniper switch to use.

 

[compuguy]

Thanks for the pointers to the optical transceivers. I may have to go that route, but my first goal is to find a way to use the SFP+ ports to interoperate with an existing 1GBE copper network. Hope to have time in the next day or two to run copper transceiver tests. Support did inform me today that the SFP+ interfaces on the SOC should appear as an Intel X550 controller, so that will help in identifying drivers.

In Windows, it comes up as a x552/x557 Intel controller, according to TinkerTry https://www.servethehome.com/lab-confirmation-intel-xeon-10g-lan-x552-x557at/

I know the Fiberstore Copper SFP+ modules work when coded as Intel. I have a few and use them in Intel cards and Supermicro boards with SFP+ cages - they link up and work fine at 10G and 1G speeds in my X10SDV-TP8F

They are expensive however Customized 10GbEBASE-T Copper SFP+ RJ45 Transceiver | FS.COM
The LR and SR 1G/10G modules coded as Intel also work and are a lot less expensive
Intel E10GSFPSR 1000BASE-SX and 10GBASE-SR SFP+ Transceiver 300m | FS.COM
Intel E10GSFPLR 1000BASE-LX and 10GBASE-LR SFP+ Transceiver 10km | FS.COM

The 1G copper Mikrotik modules *DO NOT* work in the Supermicro motherboards (so far). In Linux the driver sees an unknown vendor on the SFP+ and disables the port. I have a few of the 10G Mikrotik modules but they are in use elsewhere and I don't recall if I tested them in the SM motherboards before putting them in a Juniper switch to use.

I just got my order from Streakwave, and the SYS-E300-8D has no issues with the Mikrotik S+RJ10 module (10GBASE-T Copper). It doesn't like the S-RJ01 (as Blinky 42 said above). Same thing happens with my BlackBox LFP416 (10/100/1000 RJ45) and 6Com Gigabit RJ45 transceiver. The odd thing is that it doesn't complain about compatibility if you plug in one supported transceiver into one of the SFP+ slots + the incompatible transceiver. It still doesn't work, but it allows me to read the transceiver using ethtool at least. I looked up the OUI for the S+RJ10, and it comes up as "NetSys Net & Sys Co., Ltd.".

 

[RepairBill]

Quick Update: In VMware 6.0.0 U2, installing the VMware ixgbe driver
(net-ixgbe_4.5.3-1OEM.600.0.0.2494585.vib) causes VMware to see the adapter. It is listed in the UI as an "Intel(R) Ethernet Connection X552 10 GbE SFP+", which is as expected.

However, the 10Gtek ASF-GE-T (INT) SFP module, while seen as an interface, does not recognize a 1GbE network connection - it appears to be disconnected, despite the switch on the other end of the CAT 6 wire indicating that a 1GbE connection is active.

So I guess that this SFP module, at least, is not compatible with this motherboard. Since Supermicro does not have a list of approved copper transceivers of this type, I guess I'll have to buy a 10G switch and try a DAC next. Going to optical would be a last resort.

 

[RepairBill]

Another Update: I ordered a Ubiquity US-16-XG switch, and after two days of fiddling to get the switch configured on my network, I was able to get one of the SFP+ ports on the Supermicro XD10-SDV motherboard to talk to the switch using a DAC cable (Cablematters 104033-2m-US).

Next up is to see how this all works together. My first impression of the Ubiquity switch is not good - probably the most pathetic "out-of-box-experience" I have ever... well... experienced. But that's another discussion.