100GbE small server to 3-4 workstation, no switch?

[RyanCo]

Hey All-

Looking for some feedback on an idea I have.

I'm looking to set up 3-4 workstations to have 100GbE access to a single server without a 100GbE switch by having 2 dual qsfp28 PCIe cards in the server and directly connecting all the workstations. The workstations do not need 100GbE connections to each other, and can use a seperate gigabit lan for that and internet etc.

The server will have quad u.2 in Raid 0 plus a HDD mirror array, plus a 16TB HDD for backup and likely a couple other HDDs for archival. I'm eyeballing a 1st gen threadripper (1900x?) for enough PCIe lanes. I do know that PCIe 3.0 wouldn't do full bandwidth to both qsfp28 ports, but that's OK, the Raid will not be fast enough to saturate dual 100GbE anyway. Low network latency is plus for my application.

This is for a recording studio, the 3-4 workstations will use the server for both loading projects, and for centralized storage of sample libraries. Sample libraries are especially sensitive to latency and read times. All the computers are in a AC controlled machine room, and close together so I was just thinking copper qsfp cabling.

Any input appreciated!

 

[Rand__]

It probably will work but I dont see the need for 100G?
56G (CX3Pro) would be simpler and work as well.

RDMA is on the table I assume? So Win Server/SMB shares?

 

[RyanCo]

Hello Rand

Thanks for the response. Yes I think 56G would be sufficient, although it seems there are more 100G QSFP28 cards on ebay available more cheaply (unless I'm missing something?). Also the quad u.2 array should be capable of somewhere around 10-12GB/s. The basic load on the server is WAY below 100G 99% of the time, but fast session load times are desirable. One piece of this is I'm looking at going back to Win from having had been on hackintoshes. If all the audio engineers working in the studio see crazy fast session load times, it will help them to be more sold on Win (lol). Pro Tools and some other audio apps cache the full session in the workstations ram, so the faster it can do that one read the better. Write speeds are basically a non-issue for this.

And yeah either Win Server or even Win Pro with something like SoftRaid (which is what I'm using now). It looks like some of the qsfp28 cards support Win 10 Pro as well. There will likely be some mac/osx clients here and there, which I could connect via RJ45/10GbE.

RDMA I suppose, but there isn't a small set of data that is accessed more frequently than the rest in this use. I'm pretty out of the loop on all of this with windows, and far from a guru with networking, so please forgive me if there is asinine thinking here!

 

[Rand__]

If latency is the key here then RDMA is the way to go. That can be done (RoCE2) with CX3 Pros or newer cards.
If you get CX4 or newer for cheaper the go for them.

No idea re RDMA support in W10 Pro, but it might be there, you might want to check.
If you need it in the first place I don't know, but its usually the best solution for ultra low latency (unless you can go Infiniband)

 

[Falloutboy]

If latency is the key here then RDMA is the way to go. That can be done (RoCE2) with CX3 Pros or newer cards.
If you get CX4 or newer for cheaper the go for them.

No idea re RDMA support in W10 Pro, but it might be there, you might want to check.
If you need it in the first place I don't know, but its usually the best solution for ultra low latency (unless you can go Infiniband)

No it's not, RDMA AFAIK is a workstation skew only feature.

 

[RyanCo]

Doing some googling here...getting up to speed.

So it looks like if the file server is running win server 2019 and the workstations are on windows 10 for workstations, the connection from the latter can use RDMA to the former. That's doable.

So is this the type of 56g card you are recommending?

Mellanox IBM p8 ConnectX3 PCIe3 56Gb FDR Infiniband Adapter MCB194A-FCAT 00RX852 | eBay

Find many great new & used options and get the best deals for Mellanox IBM p8 ConnectX3 PCIe3 56Gb FDR Infiniband Adapter MCB194A-FCAT 00RX852 at the best online prices at eBay! Free shipping for many products!

www.ebay.com

The price is reasonable.

 

[RageBone]

small warning, that ebay listing is the wrong card and mislabeled. That is a connect -IB infiniband only card, and low height bracket.
There is no mention of Ethernet in the listing, so its not mislabled / misleading. Even the table entry only states FDR IB.

Also, I think you need RoCE V2 for SMB Direct which Rand confirmed, is only on CX3 PRO and up.
So, cx3 Non PRO cards would then be a surprise.

Cheapest current options i know are HPs Flexlom Cards with a custom Adapter / Riser, dropping prices below 20$/€ per card.
HP 544+ FLR FLR for Flexlom what ever.
The 544+ is a normal card from HP and you could get those.

In the past, the 544 s sometimes went for sub 30$ those are the non pro equivalent.

 

[RyanCo]

HP 544+ FLR FLR for Flexlom what ever.
The 544+ is a normal card from HP and you could get those.

Ok awesome, thanks. Bit by bit I'm getting up to speed here. At first I thought Rand was talking about CX4 connectors, so that gives you an idea of how out of the loop I am.

So these guys then?

Ethernet/Network Cards Network Cards 40 Gbps Maximum Data Rate for sale | eBay

Get the best deals on Ethernet/Network Cards Network Cards 40 Gbps Maximum Data Rate and find everything you'll need to improve your home office setup at eBay.com. Fast & Free shipping on many items!

www.ebay.com

And this is 56g with the IB? I'm in a little over my head here, so reading a lot between posts.

At ~40 per card, it's seems worth it to give this a shot before taking a look at 100g options.

 

[Rand__]

Those are probably only Connect-X3 (non pro) cards, so only RoCEv1 (or IB). If thats sufficient for you, only you can know that.
CX4 is Connect-X4, the ne t Gen. after CX3, they are available in 40G/100G (x8/x16 slots).
They can be had for ~$150 (100G)

O/c you can also use CX5 or CX6 (200G) if you find them cheap enough

In the end only you know whether you can run a small IB deployment (or want to) or if ETH/RDMA (RoCE) is sufficient (way simpler)... or even only ETH without RDMA

 

[RageBone]

So, i was of the impression that Mellanox was ok / good with naming stuff.
Witj connectx 3 and 3 pro, that falls appart because the difference can be a single letter.

HPE 544 is cx3 vs HPE 544+ being cx3 Pro.
MCX354A FCBT being cx3 vs MCX354A FCCT cx3pro.

and i would grabb the pros, roce v2 seems very important to me.
So those cards you linked are still the non pro version.

A factor why RDMA is in my opinion a must for you is that it helps reduce the load on the cpu. without it, you could rather quickly, run into single thread / cpu performace issues.

In terms of your planed setup with multiple nics in the server, i have something like that and have tinkered around with it a lot.

The jellyfish NAS that made some waves in the youtube world like 2 years ago, did it the static IP way with each nic having its own different network.

Current option i have running on a freenas is a Brdige that makes it basically a software switch. I have the inpression that that isn't as optimised as it could be but i digress.

Spend some time of trying to get a Link Agregation Group working in a splitt configuration and it does work ocasinally.
Needs some code to work permanently though, and that code does not exist yet.

 

[NPS]

I'm eyeballing a 1st gen threadripper (1900x?) for enough PCIe lanes.

You could also consider the Xeon W-21xx/22xx. I think they have enough PCIe lanes, really high clock speeds, support for ECC RDIMMs and some solid mainboards. The Fujitsu/Kontron D3598-B could fit your needs. I got myself a used W-2135 for 150€ and am building a NAS of that.

 

[RyanCo]

A factor why RDMA is in my opinion a must for you is that it helps reduce the load on the cpu. without it, you could rather quickly, run into single thread / cpu performace issues.

Yeah, once I realized that this was a feature of RDMA, it went from "do I really need that?" to "Ok, this is a no brainer".

So if I'm understanding correctly, CX3 Pro or newer would be the way to go for sure. With the 3Pro, we could get 56gbps configured as IB (trickier?) Or 40gbps with ethernet/RDMA (easier?). Going to CX4, It could be 100gbps with either ethernet or IB config and either has a RDMA function?

And for both ethernet/RDMA and IB I would need Win Server 2019 for the Server and Win 10 Pro for Workstations on the workstations?

And are these 544+/CX3 (seems a little tricky because they put the 544+qsfp together)?

HPE HP 764736-001 InfiniBand FDR 2P 10GB 40GB QSFP Network Adapter 764284-B21 | eBay

Hp - Infiniband Fdr/ethernet 10gb/40gb 2-port 544+qsfp Adapter. Ports : 40gb Ethernet / Infiniband Fdr Qsfp X 2. Interfaces : 2 X 40gb Ethernet / Infiniband Fdr - Qsfp. Designed For : Hpe Proliant Dl180 Gen9, Dl560 Gen9 Entry, Xl170r Gen9, Xl190r Gen9.

www.ebay.com

I'm thinking it makes the most sense to try this out with the CX3 and make sure there isn't an achilles heal for this whole plan before spending more.

Thanks!

 

[RageBone]

The main difference with IB is that you need a so called "subnetmanager" running on a machine in network who "manages" the network.
Without one, no network.
And then you need your software to actually support and use IB, there are ways to get protocol support for ip stuff on ib called ipoib but that is its own can of worms with potential cpu perf issues.

So having Ethernet Cards or VPI cards that can be set to Ethernet simplifies that.

The official Ethernet standard only does 40Gbit/s aka 40GbE. Mellanox one upped them with an ib FDR14 equivalent 56GbE mode.
cards will default to 40, you can set them to 56 through software.
On linux, its the ethtool that can list compatible modes and change that to 56.

Another thing to know is that Linux recently got client support for SMB Direct and Multichannel since those go together hand in hand.
So Linux clients are an option as well.

Linus tech tips "recently" did a video about his new Mellanox stuff, worth a watch i guess, the least you will learn is what windows will work and what issues he had.

In terms of specific network cards, if you are going for the often cheaper oem cards like those HP ones, search for their HP Partnumber.
And have a look at this thread.

https://forums.servethehome.com/index.php?resources/mellanox-nics.47/

 

[i386]

So, i was of the impression that Mellanox was ok / good with naming stuff.
HPE 544 is cx3 vs HPE 544+ being cx3 Pro.
MCX354A FCBT being cx3 vs MCX354A FCCT cx3pro.

Mellanox nics have a "good" and documented naming system. It's the oems that have weird or multiple product names.

I'm not sure where you got the "544" as a product name. Hpe uses a different naming system for parts:

This is a screenshot from the mellanox hpe reference guide from june 2020

 

[RageBone]

Well MCX354A FCBT seems to me to be the CX3, the FCCT cx3pro, so a single letter in the part that shows you what speed and height bracket the card has, suddenly tells you of a chip difference?

Never mind that.
To the 544 weirdness. HPE goes by that.
Here a small screenshot of the Mellanox / nvidia to HP OEM card list of CX5 cards. Sadly they don't list cx3.
"HPE 547FLR-QSFP VPI 40/50Gb 2-port Adapter (HPE FlexLOM)10/25/40/50Gb879482-B21"
And the other screenshot from the HPE support site.

Does that answer where those come from?

 

[RyanCo]

Thanks everyone-

It's a little confusing, but it looks like a HPE 764284-B21 is CX3Pro (right?), And available at reasonable prices. I see some on ebay for $30...The actually branded FCCT is high enough price wise that I would probably opt for CX4 over it.

And then is it necessary to flash these to the newest FW from mellanox? Can I flash and access the settings for 56gbe and RDMA via windows?

So if those cards are right, it's just those and QSFP+ cables and that's the hardware side of this?

Thanks again

 

[RyanCo]

Hello all-

I got the 764284-B21 cards and a 10/40 VPI/FDR qsfp+ cable (also Mellanox brand). They both pop up with the HPE drivers in win 10, but show as Mellanox 5.5 in info. Right now I'm just trying to get them working without RDMA, for the moment my 'server' is a X299 Pro that doesn't have drivers for WS2019. Once I can complete some testing, I'll likely to an Epyc just for PCIe lanes.

So, when I connect the qsfp+ cable, a light comes on, but it's static, and on both computers it shows the cable as unplugged. Did basic config that I would do for 10gbe (set IP's etc), but still unplugged and not working. All the guides I manage to find just show how to enable RDMA. Any guides for just getting it working? I did install OF, pretty new to powershell here.

Thanks

 

[i386]

In devicemanger under system devices Change the Port types Form Auto to Ethernet for the mellanox device

 

[RyanCo]

Awesome, well they see each other now.

But I seem to be stuck in some sort of endless permissions/credentials loop. Both computers can ping each other, the workstation can see the server, and access another folder I shared on it, but not the drive I'm trying to share. The server doesn't see the workstation at all under network. I think I made a credentials/names issue because the server was a clone of the workstation drive. Maybe that's what got me it trouble. I changed the key/COA on the server (which is just win 10 pro for now).

Any other suggestions

 

[i386]

To see other hosts under network you will need the correct configuration for the domain/working group