It is indeed different. So ethtool and some other utilities will confirm that the ports on the C3000 SoC are actually in 'backplane mode', which are internally both connected to the switch. By default, those ports also support a variation of SR-IOV which makes it look like the SoC actually has 4 ports, but it doesn't. (well, there are C3000 SKUs that allow for 4x10GbE but the 1G base units Dell makes are all the lower end C3000 I think)
The reason most operating systems will happily allocate 4 interfaces is because the C3000 came around just before mainstream DSA, but just around mainstream DPDK, so it does have the hardware, but they just assumed the OS would use some special SDK sauce to properly configure the interfaces.
In the 10G version that's automagically solved because it just uses 2x10G-KR or 4x10G-KR depending on the model, and then (IIRC) i350 PCIe NICs for the remaining ports. I don't have one turned on here right now, but I remember lspci and pciconf reporting the different NICs on the VEP1425 and up. I also have a VEP1485 here which probably does the same.
What I did forget to add in my diagram is the eth0 interface between the C3000 and the Switch is also the MDIO interface which is used to control the switch chip. The MDIO tool from Dell, and the other tools do make a distinction between setting MAC registers, and PHY registers, so I'm assuming one set of registers is used to control port and link settings, while the other one does forwarding and ethernet stuff. As per
https://www.kernel.org/doc/Documentation/devicetree/bindings/net/dsa/marvell.txt there generally is a single base address and then everything else is offset from there over the MDIO bus. The GPIO line between the SoC and the Switch can be used to reset the switch, but also to receive a 'switch is ready' signal. The FPGA can also do this, so maybe it's just a simple enable/disable/reset pin on the switch that is used and not a fancy bus.
The Lattice FPGA probably also has some direct lines to the SFP+ ports, I haven't checked yet, but it controls powerup/powerdown, light power levels and for all ports it also controls the LED settings. I think there must be some sort of real-time interface between the FPGA and the switch because how else would it know whet LEDs to blink and at what speed. (or the Marvell is driving the LEDs directly, but I don't think it can?)
Edit: some random picture (from eTeknix?) shows there are two dual-port devices on the C3000 in many cases, and without any additional information, the OS does indeed not know how it is actually wired up. Ironically, this was actually standard back in the day, to have a separate MAC and PHY etc. and you'd have to tell the OS which ones you have and how you connected them. In the Dell case, I wouldn't be surprised if they have one link of each of them connected to the switch to make better use of hardware assistance, but this is just a guess.
Edit2: Another somewhat random datasheet for a COM Express module:
https://www.c-dis.net/media/images/upload/DFI-DV970-COM-Express-Basic-DataSheet.pdf it shows a bunch of stuff specific to them:
- PHY for X557 or CS4227/CS4223 and Controller for Intel® I210AT/Intel® I210IT
- 2 x Independent 10GbE Media Access Controller (Sku dependent)
- Supports up to Two 10GBASE-KR Interfaces and Max up to Four 10GbE MAC Ports
Supports Intel® X557-AT/AT2/AT4 10GbE PHY
- Supports Inphi CS4227 (2 port)/CS4223 (4 port) (maximum bandwidth is 20Gb when 4 ports of 10G interface active at the same time)
- 1 x Intel® I210AT (10/100/1000Mbps) (normal temp.) or
- 1 x Intel® I210IT (10/100/1000Mbps) (wide temp.)
But it does show that 10GBASE-KR interface which is what is used (at 2.5G speeds) with that Marvell chip.
Edit3: Here is an image from Mouser, while not totally correct it does show what the acronyms mean and where in the chain they tend to be:
The only thing we'll probably not get access to any time soon is the datasheet and specs for the Marvell switch chip, because they NDA it so hard you might as well never buy the chip.
This means that while native configuration for the network ports on VEP1400 (non-X) models is available, it's just not turned on. Even nightly builds from VyOS and OpenWRT don't have it by default. OpenWRT does have it for device-specific builds, but for x86 they aren't enabling it by default.
