Datto s4p4 (asrock D2143D8UM) NIC options

[zachj]

The internet can’t seem to agree whether these have proprietary mezzanine connectors or if they are standard (ex: OCP) connectors.

im wondering if instead of spending a bunch of money on sfp transceivers I can just replace the rj45 mezzanine with an sfp mezzanine?

 

[RolloZ170]

I can just replace the rj45 mezzanine with an sfp mezzanine?

you mean replace sfp with rj45 to use 10G max. ?

sfp mezzanine need sfp transceivers to work.

got it: you want to use Direct Attach SFP Cables.

 

[louie1961]

You can, technically, replace the RJ45 mezzanine with an SFP+ mezzanine. The problem is if you can find them they are stupid expensive. BTW, the board that is in these Datto units is NOT 100% identical to the ASrock Rack D2143D8UM. They are silk screened with a different model number and some of the ports are different. Mostly minor things like one of the USB ports on the side of the board, things like that. I was able to flash mine with the 1.11 bios (see here) but the board still identifies as an S4P2134, not a D2143D8UM. But that bios solved a lot of issues for me. There are other posts on this forum about these boards that may answer your OCP question. See here

I am not sure what kind of connectors are on the board:

 

[zachj]

I want to remove the mezzanine PHY containing two rj45 ten gigabit ports and replace with a mezzanine PHY containing either two or four sfp+ ports.

the x527 and x557 PHY only mezzanine cards are cheap as hell on eBay but it’s super unclear whether they are physically or electrically compatible with the asrock motherboard. They look to me like standard OCP 2.0 A/B/C mezzanine connectors.

I went ahead and bought one on eBay. Will see if my server catches fire

 

[etorix]

Logically, if AsRock Rack wanted to lock the mezzanine to their own parts ONE single connector would have been enough; the trio only makes sense if these are standard OCP 2.0 A/B/C and you're welcome to plug absolutely any OCP 2.0 mezzanine you see fit.

Fingers crossed for your experiment…

 

[louie1961]

I would love to switch my RJ45 mezzanine out for SFP+. Please report back if it works.

 

[zachj]

Ok so I have mixed results and am looking for suggestions…

I purchased a quad sfp+ port ocp 2.0 type C mezzanine card (PHY only):

Intel X527-DA4 Ethernet Network Connection OCP X527DA4OCPG1P5

I can confirm it is physically compatible and my server did not emit any of its precious blue smoke.

I can further confirm that it lights up when cables are installed and in BIOS all four ports are visible and report as connected when they are connected (and disconnected when they are not connected).

I can also confirm that TCPDUMP shows ARP traffic.

unfortunately that’s aaaallllll I can confirm because nothing else works. DHCP doesn’t work (request isn’t received at the firewall and so response is never sent). Static ip assignment doesn’t work.

the googles are going on and on about upgrading the nic firmware but since the MAC embedded in the SOC was being used by the previous dual port rj45 adapter I don’t see how the firmware could change the result.

the googles are also going on and on about turning off the offloads but I don’t see how those are relevant either (for the same reason as above—the offloads worked just fine when using the rj45 PHY).

there’s a jumper on the motherboard that Google tried to tell me is responsible for configuring whether the installed PHY is rj45 or sfp+ but I think Google is hallucinating because the motherboard manual says that jumper is to instruct the motherboard whether the installed card is a passive ocp 2.0 type C network PHY or if instead it’s an active type a/b PCIe card.

does anyone have any brilliant ideas??

 

[zachj]

Progress update: I believe the hardware works and the problem is between brain and keyboard.

I haven’t got it to work in proxmox but it does work in Ubuntu. I haven’t got it to work in a bond but it does work as singleton network connection.

Oh and my eBay seller sent both of my PHYs slammed full of 10g sfp optics. Not bad for twenty bucks

 

[zachj]

I’ve got one host working in proxmox with two of its four sfp ports in a bond.

The other host is giving me grief so I think I’m going to reimage it.

In any case I think this proves the motherboard does indeed accept standard OCP 2.0 mezzanine cards. $20 for a quad port SFP+ PHY is a helluva lot less money than two rj45 transceivers so I call this a win!

 

[louie1961]

so are all those PHYs supported by the SoC or is there an independent NIC chipset taking up some PCI lanes?

 

[zachj]

I still need to get my other server working properly. The plan was to use two of the ports on each server in a bond connected to an l2 switch and the other two ports on each server in a bond for host-to-host communications (storage/migration). So I haven’t tested the other two ports yet.

the SOC is supposed to support four ports and asrock sells a quad sfp port PHY, and both the bios and proxmox show all four ports. So I expect that all four ports will work.

The ocp 2.0 mezzanine spec allows for both “passive” PHY-only cards as well as full fat PCIe devices. There are mezzanine type a/b cards that implement full PCIe network controllers (the most obvious of which is connectx because clearly that isn’t integrated into the SOC). Confusingly there are also type a/b cards that implement a full PCIe intel 710 NIC—no earthly idea why.

I believe any type c mezzanine is using the SOC integrated network MAC.

 

[webauthn]

I have a few of these boards and have successfully been able to use the Intel OCP 2.0 X527-DA4 cards for 4x10g SFP+ from the SoC - works without any issue at all. These use the same interface as the stock card.

I can also confirm that the Mellanox ConnectX-4 Lx CX4421A OCP 2.0 cards which are PCIe based work as well without issue, for 2x25g SFP28, and this config does not interfere with the PCIe lanes which are used for the actual PCIe slots.

FWIW, I'm running the ASRock Rack firmware and not the branded Datto firmware on these boards, but the OCP 2.0 cards worked before I changed that.

 

[986box]

was there any special handling needed for either Intel or Mellanox to work?

Any idea if Dell branded Mellanox ConnectX-4 Lx will work?

 

[zachj]

On my end no. The PHY just worked right out of the box. No need to even enter the bios

 

[986box]

Does anyone know where to source the fasteners for OCP card? Mine arrived unattached.