Newbie lost in the 10Gbps wilderness...

[Nicoloks]

Hi All,

Quick background. I currently have a 1Gbps network and just in the process of "upgrading" my Proxmox home lab to use 3 x HP EliteDesk 800 G3's which will have a x16, x4 and two by x1 PCIe 3.0 slots. I already have the x16 slots populated with LSI 9200 8e PCIe 2.0 x8 cards to go to the external storage for each node. I've never had a requirement to go beyond basic consumer grade 1Gbps switches before for my needs, so still trying to get my head around all the 10Gbps options while keeping the budget tight. Really only looking for 10Gbps to connect my 3x Proxmox nodes for now, however uplifting the rest of my network to 2.5Gbps would be a nice bump (but not really essential), and likely as much as I'm ever realistically get any functional use out of for the foreseeable future.

I'd really appreciated some help with is the following in regards to;

• Identify suitable low profile 10Gbps network adapter running PCIe 3.0 x4 for my Proxmox nodes. I was eyeing off the TP-Link TX401 as this is what my local computer shop stock. Have also been told to look at used SFP+ options, however they all seem to run x8 PCIe lanes and usually do not come with transceivers. Not really sure what I am looking for to make sure I purchase transceiver modules that are compatible with the cards I might by. Alo, Given my Proxmox nodes are going to be the only real use case for me to see valuable gains in uplifting to 10Gbps, would it be possible to just get 3 x dual SFP+ cards and daisy chain my Proxmox nodes together?
  
  
  
• My core switch is currently an old 24 port Netgear JGS524Ev2 with only a third of the ports used. I think I would be wanting to look at another 24 port, however with all 2.5GbE ports with 4 x SFP+ options for 10GbE? Would really appreciate some guidance on this one as find myself quickly getting lost in the eBay lottery of used options.

**EDIT**

Just demonstrating how all over the place my head is on this, currently thinking I'll likely go for 3 x single SFP+ PCIe 3.0 cards, a MicroTik CRS305-1G-4S+IN and some RJ45 SFP+ transceivers to suit. Being unmanaged, I'd keep this Proxmox network isolated from my main network. Does this make sense for achieving 10GbE for a low price point?

 

[MountainBofh]

Since you're going to need a new switch, you got some choices to make.

1. Do you want to standardize on a SFP+ switch, or a 10gb RJ45 switch? The later will increase your costs considerably for both the switch, and the cards. SFP+ is a lot cheaper.

2. Do you need a managed switch, or would unmanaged work? How many ports do you really need? 24 ports all at 2.5gb is going to push your switch cost up some.

One possibility is to get a cheap 8 port all SFP+ switches (they are on Aliexpress for under $100, and Amazon for $160), link all your proxmox stuff together on the SFP+ switch, and then get one of the 8 port 2.5 switches with a SFP+ uplink port ($50-80 Amazon or Aliexpress), and put all your non proxmox stuff on that switch. You'll have less ports overall, but thats a cheap way to get high speed ports to everything.

If you have your heart set on a 24 port switch thats all 2.5gb with a handful of faster ports, there are some Dell N2224X-ON switches on ebay for $700. Not cheap, but you'll have a bunch of 2.5gb ports with 4 SFP28 ports added in for fun.


As far as your network card is concerned......
If your slot is only 4x pci-e, your options are going to realistically going to be either a used Mellanox 4x (these are GREAT cards btw, and they're dirt cheap on ebay for $30-35), or an Aquantia based one (the TP-Link one you referred to is such a card). The Intel 10gb cards can be flakey if you don't give them 8x pci-e lanes. I run my Mellanox on a 2x slot and I have zero problems getting full 10gb out of it.

For transceivers, the Mellanox's will work great with anything. Some of the big name vendor switches (Cisco is bad in particular) can get picky on transceivers, but I wouldn't sweat this too much since its easy to get transceivers that will lie to the switch and tell it whatever it wants to "hear".

Pick your switch setup first, then your network cards. Once you have that figured out the transceiver choice should be easy enough.

 

[Nicoloks]

Thanks so much for the reply.

Seems the SFP+ is clearly the way to go, just makes me wonder why the 10GbE RJ45 switches exist if they offer less flexibility at increased cost.

I've just done an audit of my port usage, and you are spot on. All 2.5GbE would be a massive overkill. It is only really my servers (all in the same room) and my main PC (in study 10 meters away) that'd benefit from 2.5Gbps, much less 10Gbps. No other devices would get any sort of justifiable advantage for the cost.

I think I'll go for the option I put in my edit in my first post. Thanks for the tip on the Mellanox cards, from what I can see they seem like they'll do the trick. Had no idea transceivers could be configured like that. Anything I should be looking out for when buying off eBay or Aliexpress for these?

 

[sko]

Some of the big name vendor switches (Cisco is bad in particular) can get picky on transceivers

that's simply FUD. True, IOS will give a warning about an unsupported transceiver and older versions even disabled the port, but you can simply override this behaviour and disable warnings by issueing service unsupported-transceiver.
I've used transceivers from various vendors in cisco switches without any problems. But given that you can get them new at only ~20$ nowadays e.g. from fs.com there's really no reason to go for used ones, especially if you don't have a cleaning tool for them. I've had plenty used transceivers in my hands that were stored without dust caps and went almost blind because of dirt on the lenses...

But given OP wants to use non-standard copper transceivers: they often give problems as they are WAY above the allowed specs for power draw on SFP+ ports, which is 1.5W. Also those transceivers are more expensive and wiring is (for shorter patch cables) around the same, but the longer the cable run and the more interconnects you have, the cheaper fiber will get in comparison. There's absolutely no sense in *buying* anything copper-based for 10G

 

[Nicoloks]

Thanks for the insight.

So if I understand correctly, the only place it makes sense to stick with RJ45 would be where I need to interface with an existing Cat5/6 network?

Given I am looking at something simple like a MicroTik CRS305-1G-4S+IN that is isolated to just my Proxmox nodes (so physically right next to each other), I'm guessing my most straightforward option would be to just use a DAC between the MircoTik and the card in each Proxmox node?

 

[louie1961]

Do you really need 10gbs? What kind of hard drives do you have in your Proxmox nodes? If you have spinning drives in your Proxmox nodes, you would probably be well served with 2.5gbps, save a bunch of money and generate a lot less heat.

I just upgraded my main Proxmox node to 10gbps networking and really haven't noticed any benefit for my uses. But the transceivers get awfully hot, by comparison.

 

[Nicoloks]

I run all SSDs for my latency/bandwidth sensitive workloads, I do still run a few spinners for things like email and backups. Not expecting much advantage in the day to day, it'd only really be when evacuating a node I'd think any real benefit would show. Regular 2.5GbE would certainly have been simpler from my position.

No matter now, I found a used CRS305-1G-4S+IN as well as 3 x Mellanox ConnectX-4 cards going cheap on eBay. Just need to get myself 3 x DAC cables now and I think I should be set for a while. I can't see my Proxmox environment getting bigger, so 10Gbps should see me through for the foreseeable future.