I would like to upgrade from HDD to SSD pools. Please rate my plan

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
P

Partofthesolution

Member
Dec 3, 2020
46
12
8
Hi all,

this is my current setup, running with ESXi : Server-config 2020/2021 Current state | Wunschlisten Geizhals Deutschland

I have a Napp-IT VM and the HBA and seven 10TB-drives in passthrough mode.

A more detailed topic about this build can be found here: https://forums.servethehome.com/ind...0d4u-2l2t-build-for-esxi-and-nas-usage.30993/

I would like to upgrade the HDD based pool to flash only with the following parts:

1x HighPoint SSD7580B - HBA for aight NVME drives ( Rocket 1580 |HighPoint Technologies, Inc. ; HighPoint SSD7580B | Preisvergleich Geizhals Deutschland )
4x HighPoint 8654-8611-205 - 1x SFF-8654 (host) to 2x SFF-8611 Oculink (device) cables ( HighPoint 8654-8611-205 | Preisvergleich Geizhals Deutschland )
1x ICY DOCK MB873MP-B - Enclosure for aight M.2 NVME drives ( ICY DOCK MB873MP-B | Preisvergleich Geizhals Deutschland )
Nx Samsung OEM Datacenter SSD PM9A3 2TB ( Samsung OEM Datacenter SSD PM9A3 1.92TB, M.2 ab € 172,00 (2023) | Preisvergleich Geizhals Deutschland )
or
Nx Samsung OEM Datacenter SSD PM9A3 4TB ( Samsung OEM Datacenter SSD PM9A3 3.84TB, M.2 ab € 300,42 (2023) | Preisvergleich Geizhals Deutschland )
1x New enclosure with 1x 5,25" (not yet defined)

The new flash drives will serve a ZFS pool for local VM and CIFS network drives over 10G for up to three private Workstations.

The old HBA and HDDs will take place in a backup system.


What I dont like is the price of the HighPoint HBA, which is ~1500EUR at the moment and the price of the ICY DOCK, which is ~700EUR. But as far as I can see, there is now cheaper NVME enclosure on the market. A broadcom 9500 based HBA would be much cheaper, but I currently dont know how to attache more than four NVMEs to one of these HBAs in a consumelike system.
 
M

mrpasc

Well-Known Member
Jan 8, 2022
493
261
63
Munich, Germany
My 2 cents:
- the Icy dock does not offer sufficient cooling if packed with 8 enterprise NVME ssd. They will throttle as soon as some load is coming.
- the icy dock is VERY picky about HBA. Those Broadcom Tri-Mode ones do not work (they do state this in bold on their site). So forget absolut the 9400/9500/9600 series
- the icy dock is very picky about cable’s too. Check the compatibility guide carefully.

We tried the icy dock @work and shifted to U.3 as for much better cooling.
 
  • Like
Reactions: Partofthesolution
CyklonDX

CyklonDX

Well-Known Member
Nov 8, 2022
857
282
63
if you want inplace upgrade of your zfs pool. Its not going to work. (You can only replace disks with greater, or same size disks) If you copy over the data then its not an issue - copy data from old pool to new pool.

Personally i think its a waste of cash. Unless you really have something that will benefit you from extra performance. I think you would be better off price wise just getting nvme's for your l2arc, zil, slog.
 
  • Like
Reactions: Partofthesolution
P

Partofthesolution

Member
Dec 3, 2020
46
12
8
Thanks @mrpasc ! Your tips are gold. I checked the compatibility list of the IcyDock and only one Raid controller from Highpoint is listed at the moment: HighPoint SSD7580 A and B. Bad, because I need a HBA.


@CyklonDX Thanks for this advice, but I dont want to upgrade inplace, so everthing is fine.

I have some reasons why I would like to switch to a Flash only pool.
1. HDDs are dying from time to time and need replacement. They are some years old and dont have waranty anymore. Before i replace them with HDDs again, I better switch to SSDs and use the HDDs in a backup system which is not online 24/7.
2. I have a RaidZ2 with 7 3,5" HDDs. I have helium filled HDDs which were very silent first, but they are getting louder over time i think. And they produce to much temperature for a living room, which was a problem for the HDDs last summer. I simply dont have aircon.
3 I have some performance issues. If one client is performing an IO intensive read (a file search or indexing task is enough), the poool is unuseable for other clients.

Napp-IT is equiped with 16GB ram (=ARC). I already gave it up to 64GB for tests (have 128GB in this system), but this doesnt make much difference. L2ARC is just SSD-based ARC. If more ARC has no impact, than L2ARC will also have no impact. So, L2ARC is not the solution here it think.

ZIL/SLOG is a technique to optimize write operations. But I dont have problems with write operations only, I (also) have problems with read operation. Only to solve write issues doesnt solve the read issues.

I think the main problem is that I only have one RaidZ2 with 7 disks in and the HDDs are just to slow for this setup. It is possible that encryption is also a contributing factor.
 
A

ano

Well-Known Member
Nov 7, 2022
654
272
63
very good drives (pm9a3)
also look at AOC-SLG3 and AOC-SLG4 series for atteaching them, you get 1:1 then, but highpoint should work

and as pointed out THEESE NEED COOLING, actually not bad for theese 7mm ones, vs other 15mm and other versions/brands, but they require airflow

and 2nd, you need cpu! zfs with them eats cpu, but works very well, filling a 10gig pipe is easy, 10x of the drives on zfs, and enough cpu, you fill 2x100
 
Last edited:
  • Like
Reactions: Partofthesolution
P

Partofthesolution

Member
Dec 3, 2020
46
12
8
I have one PM983 2TB (previous model) in this system and I like it. All VMs are stored on this drive :)

I understand that cooling is a main topic. Will have an eye on it.


Whats enough CPU to serve 10G ? Do you think a Ryzen 7 PRO 4750G can do this job?
 
A

ano

Well-Known Member
Nov 7, 2022
654
272
63
yes, 10Gig is easy with that cpu

it upsets me when people write 2TB for 1.92TB devices btw ;) guessing it hurts someone elses head as well, 2TB are consumer stuff, 1.92TB enterprise (most of the time at least)

so when people say 2TB and pm983 Im confused if your not using a pm983, or are just bad at numbers :D
 
P

Partofthesolution

Member
Dec 3, 2020
46
12
8
Haha, good point. I am just lazy, thats the reason why I write 2TB instead of 1.92TB. Because its nearly the same and I can never remember if its 1.92 :)

I will pay attention in the future to prevent irritation ;)
 
A

ano

Well-Known Member
Nov 7, 2022
654
272
63
and a tip, the 7.68TB, are priced very competitive vs 3.84TB, so unless you need the iops, you will save on power, number of drives needed etc.

the 3.84 has more IOPS (marginally) but again with cpu. 7.68TB might be a good idea
 
I

i386

Well-Known Member
Mar 18, 2016
4,250
1,548
113
34
Germany
samsung doesn't sell "datacenter"/"enterprise" ssds with plp to end customers, only to oems.
 
I

i386

Well-Known Member
Mar 18, 2016
4,250
1,548
113
34
Germany
You must log in or register to reply here.
Top Bottom