NFS storage for proxmox

[razvanc.mobile]

Hello,
I've recently purchased a Dell M1000e blade system, that i'm planning to use for proxmox vms.
I want to use an external box as storage, connected to the blade using 10G networking.

The box in question is an Intel chassis with 12x3.5 drives that i had used previously as a freenas box, but has been re-purposed.

My plan is to use 12 drives probably in a raidz3 setup, connected to a lsi 9211 HBA. OS will be centOS, with zfsonlinux.

Also i plan on using 2 ssd's connected to either the secondary port of the HBA (first one goes to the builtin expander in the server) or motherboard directly (have to check if those ports are sata3 first).

I've been scouring ebay for some cheap intel small s3700's, to use for slog and l2arc, but being in europe, i don't have decent access to US market, shipping is usually a killer. So i will probably settle for 2 x samsung SM863's 240GB, which although way too big, can serve the purpose.

Now comes the question of the correct setup for these ssd's.

The box will have some low-voltage quad-core cpu's (L5630 or L5640), and 64-128GB RAM.
What's the best way to setup the 2 ssd's to speed up the underlying pool?

OP them to 160 GB or less, and make 2 mirrors, one 16GB for a mirrored slog and one of an 128GB for l2arc?
Is mirroring important? or just use one for slog and one for l2arc?

Plan is to connect by NFS, and sync=always.

Thank you.

 

[Patrick]

That S3710 400GB deal was probably the way to go if you can use shipitto or a similar service.

How many SATA ports do you have available? I would not look to mirror L2ARC since it is really a cache not a primary data storage tier.

 

[razvanc.mobile]

plenty of ports available, basically 6 from the motherboard(which might be sata2 only) and 4 from the second sas port of the hba.

Sent from my SM-G930F using Tapatalk

 

[Patrick]

If you can get a lower-cost but larger L2ARC drive, that may be a decent idea. I could see, with the blade servers, using a decent amount of L2ARC space.

Maybe look for a deal on S3700 100GB drives for ZIL and a larger SSD for L2ARC such as a PM863.

 

[razvanc.mobile]

Comming back to this

The storage box is almost finished

It's config is
dual L5630, 64 G ram
LSI 9211-8i HBA
2x240GB Samsung SM863 OP'd to 200G
12 x (assortment of 1 TB Constellation ES and HGST enterprise drives, all SATA)

I've done some testing, but haven't made up my mind how to configure the storage pool.
i know i'll have 2x16GB partitions from the ssd's mirrored for ZIL, and the rest of the ssd's given as roundrobin cache devices.

But i can't decide about the spinners, pool of mirrors or raidz2-3 ?
Given their size, i'd have either ~6TB as a pool of mirrors, or 9-10TB as a pool of raidz3,2.

What would be best for VM storage via NFS?

 

[gea]

What you want for VM storage is iops.
If you build a pool from a single Raid-Z2/3 vdev you have around 150 iops for the whole pool. This is like one disk as you must position every disk for every io.

If you use 6 mirrors instead, you have around 6 x 150 iops=900 write iops and around 1800 read iops (as ZFS can read from both disks of a mirror).

So raid-Z is a nogo with VM storage unless you use SSD only storage where a good single SSD can give 20k iops and more.

 

[ttabbal]

I agree with gea, you definitely want mirrors for VM and most performance tasks. Raidz can do alright with large files in sequential access modes, but a large mirror pool will smoke it for most normal things. And most file access isn't sequential, particularly VMs.

 

[razvanc.mobile]

So basically, even if the zfs pool will store large (50-100GB) qcow files, the access pattern is still considered random?

 

[ttabbal]

Yes. Just like the access on a real disk is random for the OS etc.. Sequential access is actually kind of rare on most computers. Other than media players, I don't see much of it.

 

[razvanc.mobile]

OK, I'll go with a pool of mirrors then. Thank you very much.
Any other recommendations given my hardware?

64 GB ram, 16 GB ZIL, 2x160GB L2arc, 6 TB pool. Anything looks off here?

 

[ttabbal]

Looks pretty good to me. Some people don't like running SLOG and L2ARC on the same drives. I suggest watching the I/O stats and perhaps split them up if needed. It's very dependant on the exact use your pool sees and the level of sync writes.