RHEL8 VDO NAS Performance

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

matkisson

Member
Apr 11, 2017
32
7
8
36
I just built a 'custom' NAS using RHEL8 in a CSE-216 chassis. Hardware is:

2x2630L v2
96GB PC3-10600R
Supermicro X9DRD-7LN4F-JBOD
8x1TB BX500 (Raid5) **Will be scaled to 2x24TB**
3x480GB SM953 (Raid5) ** Currently adding 3 more once they arrive**

NFS and SMB are working great but I am curious if I should change the CPU. I cant find any documentation on the way VDO operates and I dont know where my bottleneck is. I get about 24Gbps with compression and dedupe enabled, and I can saturate the 40Gb link with them disabled. With compression/dedupe my CPU usage is only about 15% with 24Gbps throughput. My question is should I look into a faster lower core count CPU? Any thoughts on diagnosing?
 

dandanio

Active Member
Oct 10, 2017
182
70
28
What is that VDO you are talking about?
Besides, it is hard to give you any recommendation without knowing more about your config. Is that RAID HW of SW? What brand? What is the max. theoretical through put of the drive you have there. What are your plans with "2x24TB"? Raid 0? Raid 1? We need more details when you ask for advice. If you are just gloating over your system, do not use the question mark. :)
 

matkisson

Member
Apr 11, 2017
32
7
8
36
VDO - A look at VDO, the new Linux compression layer

Similar (I know its different) to XFS file system compression/dedupe. The VDO is on a software raid. Im not concerned about the 24x2TB array (typo), thats just general file storage. Im trying to see if anyone knows about the mechanism behind VDO compression or if anyone has already been down this rabbit hole. Either it is highly threaded and the system is not utilizing the CPU, or it is more geared towards core speed in which case the CPU is holding it back.

Really not here for a measuring contest, and this isnt gloating.
 

matkisson

Member
Apr 11, 2017
32
7
8
36
In case it helps, kicked off a clone of a VM on one of the servers. The drives are capable of much high speeds than 384MB/s per drive. You can see the CPU spike when data is incoming.
 

Attachments

RTM

Well-Known Member
Jan 26, 2014
956
359
63
I do not recall having heard/read anywhere on the forum about people actually using VDO, so there may be limited help.
That all said, perhaps some of the typical wisdom from ZFS also applies here.

Primarily I was thinking, if you have tried disabling deduplication (without also disabling compression), as far as I know people generally consider that troublesome on ZFS to the point that it is not recommended.

With regards to the CPU's, my honest opinion is that you should do the CPU upgrade, if only because it should be a relatively small increase in price compared to the disks you are putting (or planning to put) into the machine. While you are at it, you might want to add some extra RAM, DDR3 RAM is cheap.

Another aspect that might be relevant for you, is that given you have two CPU's, NUMA could be a factor. If the disks are connected to the SAS2308 controller that again is connected to one of the CPU's, consider that a result may be that if compression is done on the other CPU. This means that data will have to go from one CPU to the other, the end result can be lower performance. A solution to this (if it is a problem) might be CPU pinning (forcing the software to use specific cores).
 

matkisson

Member
Apr 11, 2017
32
7
8
36
I have though about NUMA but I need to get into the BIOS to change some tings there. The CPU is looking like the likely culprit, and possibly the 1333Mhz RAM.

Unfortunately the onboard 2308 is not playing nice with RHEL8 so for the moment I have an HP H240 in the chassis connected to the SAS expander backplane (Need to get that fixed to clear up another PCI slot...). The 2.5" drives are just there for capacity and im not concerned with their performance, 2-4Gbps throughput is plenty. My focus at the moment is the NVME raid. Which, I forgot to mention, I am using the Supermicro AOC-SLG3-2M2-O adapters and splitting the slots.

I will give the NUMA settings a try and get back with more information (may take some time, I am actively using the network at the moment). Also I have a pair of 2637 v1 cpus comin in so i can see if the core speed is affecting it. Hopefully I wont need chips over ~80 watts, actually going for 'low' wattage on this build.

Edit: because I posted the screenshots above I figured I would edit this. Turns out my nexus 3064 was not enforcing jumbo frames. The traffic below is on the NVME pool.
1593890630505.png
 

Attachments

Last edited:

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,394
511
113
I've not used it in anger myself, but I'm tangentially aware of it. It operates as a device mapper module (dm-vdo) and was originally a Permabit thing. Compression/decompression is done inline by the VDO module and is LZ4-based. The dedupe claims to use much less RAM than you might on ZFS - they claim 1GB of memory can service 10TB of storage depending on the type of index being used, but it seems that there's also a significant portion of the index that's stored on the discs themselves rather than in RAM.

Numbers from RH themselves never looked especially great - in their own example, IO performance is pretty much halved. It would need testing but I suspect that there's significant portions of the code that might not scale very well, hence you can end up getting bottlenecked on CPU usage - you can frequently eyeball this by using tools like top or htop to show you per-CPU usage. Your CPUs are relatively old, so if there's anything in the stack that's single-threaded (or can't scale past a handful of threads) then you're likely to bottleneck, and if that's the case a CPU upgrade is unlikely to get you a significant amount of extra performance.

If I had to guess I'd say the culprit here is more likely to be the dedupe portion, since it's relatively trivial to parallelise LZ4 compression whereas it looks like the dedupe component has both a RAM-based and a disc-based index to check.

If performance is your goal VDO is probably best avoided, but if you have a workload that you think is going to benefit from compression and/or dedupe it'll be worth investigating how it behaves once dedupe is turned off.
 

matkisson

Member
Apr 11, 2017
32
7
8
36
Well, after about 6 months of operation (and a few drive failures) I can honestly say this thing is a champ. Heres a quick run down of changes from the initial design and reasons.

First and foremost, I ended up swapping out the backplane to the CSE-216 chassis. The backplane with the expander, and raid controller, did not work right with RHEL8 as it used the SAS2008 chip. I did not realize this until I attempted to add more than 8 drives. (Still dont know why it would show all 8 if there was no support to begin with ...). Swapped the backplane to the BPN-SAS-216A and picked up 3 HP H240 HBA cards.

I also 'upgraded' the functionality for the storage to run Nextcloud. I despised the way Nextcloud stored files so, I added the NAS to my AD, configured samba and NFS, and mapped the 'local NFS/Samba shares' so that I could maintain control of the file structure and manage permissions from my DC. Temporary file upload locations were added to the NVME storage so file uploads would be a bit smoother (even upload to SATA SSD were slow for some reason).

The final nail in the coffin was the 2nd system being stood up. Once both systems were stood up, NFS shares (excluding the ESXi datastore) were RSYNC'd for backup purposes and the windows shares were converted to a mirrored DFS. The end result is a 2 NAS setup, live backup of files to a second (much lower power) location, complete integration with AD, and using NextCloud and ACME SSL certs, I have some friends able to utilize my NAS from anywhere they are.

... though the cert expired a month ago ... I need to fix that ....

@EffrafaxOfWug
Good call on the dedupe. I underestimated the amount of overhead it would cause. I had expected some loss, but when I got the 3 additional NVME and migrated to RAID6 there was no performance change with dedupe enabled. I rebuilt and disabled dedupe, performance was up about 50% compared to the 3 NVME RAID5. I swapped the CPUs and RAM (went to 48GB PC3-12800R) as well.

I'll be writing up a guide incase anyone is interested once I have time. Need to finish up this class work first.
 

Attachments

  • Like
Reactions: gb00s and Evan