Unneeded home server upgrade I don't need but want to do

[Octopuss]

Depending on how your zpools are set up you might not see any speed boost with NVMes:

10GbE => 10 Gbit/s => 1.25 GB/s
SATA => 6 Gbit/s => 0.75 GB/s (say 0.5 GB/s per SATA SSD to be more realistic)

So with four drives in raidz1, which gives streaming read and write speed of [number of data drives in the pool] times [speed of the slowest drive], you're likely already limited by the network. (Or at least not by the drive interface.)

That's a fair point. It might still be of some hypothetically real use if there are multiple sources accessing the data at the same time. But you're most likely right - but there is still the upgrade itch argument

No such thing as an AIC that implements bifurcation. You either have UEFI support for bifurcation or you need to use a PCIe switch card -- and the latter can certainly be four M.2 slots on an x8 AIC (random example: NV9524-4I).

Looks like I will need to upgrade the server first anyway, haha I guess I better look around to assess how much can I sell the current hardware for.
The upgrade wouldn't be terribly expensive (not counting the SSDs) though.
I'm thinking either Supermicro H13SAE-MF or Asrock B650D4U, that's the most expensive part.
For CPU, Ryzen 7600, that's not very expensive.
RAM, I don't know, isn't DDR5 pretty cheap these days? I won't need more than 32GB anyway.

I guess I can offset close to half of the cost with the current hardware. Or at least I hope these server MBs and CPUs keep some value for a long time and don't drop close to zero like desktop components do.

 

[Blue4130]

That's a fair point. It might still be of some hypothetically real use if there are multiple sources accessing the data at the same time. But you're most likely right - but there is still the upgrade itch argument


Looks like I will need to upgrade the server first anyway, haha I guess I better look around to assess how much can I sell the current hardware for.
The upgrade wouldn't be terribly expensive (not counting the SSDs) though.
I'm thinking either Supermicro H13SAE-MF or Asrock B650D4U, that's the most expensive part.
For CPU, Ryzen 7600, that's not very expensive.
RAM, I don't know, isn't DDR5 pretty cheap these days? I won't need more than 32GB anyway.

I guess I can offset close to half of the cost with the current hardware. Or at least I hope these server MBs and CPUs keep some value for a long time and don't drop close to zero like desktop components do.

If you go with the CPU do you plan on using an HBA or AIC that has a switch chip for the nvme drives?

 

[Octopuss]

You mean the Ryzen? If it turns out neither the motherboard support bifurcation, I guess the AIC would be better. Was it here where someone told me a HBA treats NVMe drives like SAS or something, not making use of all the NVMe features or something?

 

[kapone]

No such thing as an AIC that implements bifurcation. You either have UEFI support for bifurcation or you need to use a PCIe switch card -- and the latter can certainly be four M.2 slots on an x8 AIC (random example: NV9524-4I).

You're right, I kinda phrased that poorly, even though we were thinking the same thing.

When I said "AIC" I was referring to add-in cards (in this case) that implement a PLX switch (since no bifurcation support on OPs motherboard), and yes they do come with 4x m.2 slots, but I was kinda focussing on not limiting the bandwidth to the SSDs and keeping them at x4. Hence my point about two SSDs per x8 slot.

 

[Octopuss]

Side question:
I don't think I need more than 32GB RAM for this kind of server, but does it matter if I go with 4800 or 5600MHz? I believe I want ECC/unbuffered RAM (that's what I remember from TrueNAS forums, something to do with ZFS), and there's not a whole lot to choose from around.

 

[nexox]

Looks like I will need to upgrade the server first anyway

One of those PCIe switch cards will work in your current board, you'll be limited to the ~64Gbps bandwidth of a 3.0x8 slot for the NVMe drives, but unless you're planning to go to a 100G network that should be pretty good still.

I don't think I need more than 32GB RAM for this kind of server, but does it matter if I go with 4800 or 5600MHz?

More RAM means more cache space for recently-accessed files, memory speed isn't going to make a noticable difference.

 

[Octopuss]

I might just turn to Ebay, googling for server memory in this country sucks ass. Most eshop don't sell this, some are overpriced as hell, and some have all the stuff miscategorized, jesus effing christ!!!
I guess I might just grab the cheapest 4800MHz modules I can find for my use case. For desktop, I would carefully scout for highest frequency with the lowest latency modules, but there's no such luxury in this category, lol.

 

[nexox]

ECC UDIMMs are usually pretty difficult to find, and as I understand it DDR5 ECC in general is still pretty expensive, but it's still not necessarily a component to cheap out on (this is one reason why I'm just running DDR4 RDIMMs everywhere, cheap, reliable, and widely available.)

 

[Octopuss]

Yeah. I could really use some tips for specific modules from someone who was perhaps shopping for something recently.
The worst part is half the eshops doesn't even seem to know what ECC means and have stuff misplaced all over the place.

 

[jode]

Yeah. I could really use some tips for specific modules from someone who was perhaps shopping for something recently.
The worst part is half the eshops doesn't even seem to know what ECC means and have stuff misplaced all over the place.

That's why I ended up with Genoa EPYC. Taking the total cost for mobo, CPU, RAM into account I found that a used 64-core Genoa system with 12x DDR5-4800 RDIMMs is the next step in my hardware journey. Not saying this is the right step for you, but you should probably contemplating such a move as part of your comparison shopping.

 

[Octopuss]

lol I don't even need 6 cores, let alone 64 Not to mention EPYC probably eats 50 times more power than Ryzen. I don't need a nuclear bomb to dig a hole in the floor

 

[nexox]

If your current CPU is adequate then just take the power consumption hit from a PCIe switch card and keep the existing platform, that'll be cheaper overall and plenty fast enough for a 10G network (or probably even 25G, though software and networking tend to start causing bottlenecks by then even if your drives and PCIe bandwidth are sufficiently fast.)

 

[jode]

lol I don't even need 6 cores, let alone 64 Not to mention EPYC probably eats 50 times more power than Ryzen. I don't need a nuclear bomb to dig a hole in the floor

I didn't think so.

But if you look at AMD's 8004 line of CPUs you may find an 8-core CPU for the SP6 socket with 128 Gen5 PCIe lanes with 8 channel memory that sips power.
It's worth considering if you find yourself limited in expandability (storage, graphics, network) or memory (speed or capacity).

 

[Octopuss]

Only problem is the CPU alone cost as much as the motherboard I am looking for....used. So a MB for an EPYC will likely cost twice as much. That's not going to fly as appealing as it is from specs point of view.

 

[jode]

Only problem is the CPU alone cost as much as the motherboard I am looking for....used. So a MB for an EPYC will likely cost twice as much. That's not going to fly as appealing as it is from specs point of view.

You checked - that's all I could ask for.

I see that used Genoa prices start dropping. Apparently, the 8004s are keeping their value better. There must be a reason for it...

 

[Octopuss]

Just like this crappy Xeon I have right now kept quite some value for its age for several years. Today it cost next to nothing, but two years ago it was totally worth selling

Also, as an upgrade you do once in many years, the EPYC price alone might not be a complete showstopper, but I am pretty sure finding a Micro-ATX board for it would be a huge problem. If that even existed. And the few boards I noticed (irrelevant of form factor) used friggin' RDIMMs. I had no idea that shit still existed! I thought it was a failed technology from 20 years ago. And I certainly don't want to know what this cost, lol.

 

[Blue4130]

You mean the Ryzen? If it turns out neither the motherboard support bifurcation, I guess the AIC would be better. Was it here where someone told me a HBA treats NVMe drives like SAS or something, not making use of all the NVMe features or something?

Something that I don't think you have mentioned is how much storage are you looking at total? That will possibly dictate what CPU is best. If you only need 16TB or so, that can be had across 4 ssd's and then you can use a standard Ryzen and motherboard, potentially even if it doesn't support bifurcation. (though ti would be easier if it did)

BUT if you are wanting a dozen nvme drives things will get more complicated, either requiring an HBA and backplane or a couple AIC with switch chips and potentially cpu with more than 24 pcie lanes.

 

[alaricljs]

So here's that m.2->PCIe slot I mentioned earlier: https://www.aliexpress.us/item/3256803389623446.html

 

[Octopuss]

Something that I don't think you have mentioned is how much storage are you looking at total? That will possibly dictate what CPU is best. If you only need 16TB or so, that can be had across 4 ssd's and then you can use a standard Ryzen and motherboard, potentially even if it doesn't support bifurcation. (though ti would be easier if it did)

BUT if you are wanting a dozen nvme drives things will get more complicated, either requiring an HBA and backplane or a couple AIC with switch chips and potentially cpu with more than 24 pcie lanes.

Oh I did, indirectly. 4x1,92TB SSD in RAIDz1 (I believe). I have no idea what that comes to, but it's enough.
Four cores is all I need really. I do need a server motherboard though because of ECC and IPMI.

 

[Blue4130]

Oh I did, indirectly. 4x1,92TB SSD in RAIDz1 (I believe). I have no idea what that comes to, but it's enough.
Four cores is all I need really. I do need a server motherboard though because of ECC and IPMI.

Now that you have stated it directly, do you have the drives already? Are you looking at m.2 format? That could potentially limit what AIC you can use since as far as I am aware, they only come in 22110, not 2280. Not every carier or motherboard can take 4 drives of that size.