Crappy subject, but I can't really think of how better to word it, so forgive me. I'm wondering how to go about determining bottlenecks in my system, and then using that to determine what to upgrade. First, my FreeNAS box (current):
Supermicro SC846 24 bay Chassis w/ 920W -SQ PSUs & SAS3/SATA BP (6Gbps)
Supermicro X10SDV-4C-TP4F board (4C/8T xeon-d processor)
48GB DDR4 ECC RAM (it's expensive...this was already a lot)
LSI (Broadcom) SAS 9207-8i HBA
18x WD RED 8TB drives in RAIDZ2 configs (3x 6Drive R-Z2)
2x Intel DC S3510 120GB drives mirrored for boot
FreeNAS 11.1-U5
So first things first. Using IPERF, I can get ~9.4Gbps over the 10G interface, however when using NFS or ISCSI, I max at around 2.2Gbps. (reading off SSDs into the pool, or from pool onto SSDS) During this test, I see CPU usage is ~80% across all threads in FreeNAS. This is sequential transfers using either DD or copying a large file (~80GB). This tells me first I probably need to upgrade the Xeon-D to something better, fine. But how would I go about measuring that? does clock speed mater more? Cores? (I'm mostly using either ISCSI or NFS storage)
Secondly, in D-t-D operations (this is in my home lab), I routinely conduct two functions. Copying inside the pool (1 large ~80TB pool) using Rsync shows speeds around 30-40MBps. For 18X drives in 3 stripes, I'd expect a lot faster. The second is using the pool for ISCSI storage for Kubernetes pods. Performance here is acceptable (I'm definitely not maxing out 10Gbps speeds), but I notice that I have a lot of ARC misses. Would adding something like an Optane 900p help either of these?
My main goal is as low power as possible, followed by decent performance. When I rebuild/build the next iteration of my FreeNAS system, I'm looking at either one of the newer Xeon-D platforms with more cores, or a Single Xeon E5-1630/1650 or something, unless someone says something else is more than adequate for 10Gbps Seq R/W? When I look at processor/board specs, I'm not sure what my limiting factor in the (inter-pool copying and NFS/ISCSI), so I'm not entirely sure what would be a decent upgrade. I definitely don't want/need dual processors, but either an E3, Xeon-D or something else?
TL;DR: given drives capable of multi gigabit transfers, which component(s) would offer the best upgrade path with least power? Do I need a Xeon E5-16xx processor? Is it Ghz related? Core related? or something else? Can I get by with a higher core count Xeon-D processor or do I need to go full on Xeon -E3 or E5?
Thanks!
Supermicro SC846 24 bay Chassis w/ 920W -SQ PSUs & SAS3/SATA BP (6Gbps)
Supermicro X10SDV-4C-TP4F board (4C/8T xeon-d processor)
48GB DDR4 ECC RAM (it's expensive...this was already a lot)
LSI (Broadcom) SAS 9207-8i HBA
18x WD RED 8TB drives in RAIDZ2 configs (3x 6Drive R-Z2)
2x Intel DC S3510 120GB drives mirrored for boot
FreeNAS 11.1-U5
So first things first. Using IPERF, I can get ~9.4Gbps over the 10G interface, however when using NFS or ISCSI, I max at around 2.2Gbps. (reading off SSDs into the pool, or from pool onto SSDS) During this test, I see CPU usage is ~80% across all threads in FreeNAS. This is sequential transfers using either DD or copying a large file (~80GB). This tells me first I probably need to upgrade the Xeon-D to something better, fine. But how would I go about measuring that? does clock speed mater more? Cores? (I'm mostly using either ISCSI or NFS storage)
Secondly, in D-t-D operations (this is in my home lab), I routinely conduct two functions. Copying inside the pool (1 large ~80TB pool) using Rsync shows speeds around 30-40MBps. For 18X drives in 3 stripes, I'd expect a lot faster. The second is using the pool for ISCSI storage for Kubernetes pods. Performance here is acceptable (I'm definitely not maxing out 10Gbps speeds), but I notice that I have a lot of ARC misses. Would adding something like an Optane 900p help either of these?
My main goal is as low power as possible, followed by decent performance. When I rebuild/build the next iteration of my FreeNAS system, I'm looking at either one of the newer Xeon-D platforms with more cores, or a Single Xeon E5-1630/1650 or something, unless someone says something else is more than adequate for 10Gbps Seq R/W? When I look at processor/board specs, I'm not sure what my limiting factor in the (inter-pool copying and NFS/ISCSI), so I'm not entirely sure what would be a decent upgrade. I definitely don't want/need dual processors, but either an E3, Xeon-D or something else?
TL;DR: given drives capable of multi gigabit transfers, which component(s) would offer the best upgrade path with least power? Do I need a Xeon E5-16xx processor? Is it Ghz related? Core related? or something else? Can I get by with a higher core count Xeon-D processor or do I need to go full on Xeon -E3 or E5?
Thanks!