I try very hard to not cross post (ie make same post here as over at the freenas forums), but in this case i feel its important enough to break my "rule"- There is a different perspective/experience level over here on STH forums and i really wanted to get that input on this important question. (i will delete this thread if users object to my cross post, tks) </disclaimer></apology>
so i have a extra intel p3605 1.6tb PCIe nvme card, and a few extra slots, so im considering adding it as a L2ARC, but wanted to get some feedback as to if i should even add an L2ARC. (i was all ready to do it, and even have tested it on a separate FN test server i keep, but after reading quite a bit, im having second thoughts as to if i should even run a l2arc).
The server specs/pool types are at the bottom (see SPOILER "Main FN BOX SPECS" ), it does have 256g of fast ecc ram. My usecase for the files on this FN box (physical box btw):
MY PLAN IS TO SPLIT THE L2ARC (ie ~400gb each for 3x of these pools)- please, i would like this discussion to mainly focus on if i should add an L2ARC, not on the wisdom (or lack of) of splitting/sharing a single 1.6tb drive l2arc. Thanks
pool name="he8x8TBz2": (has split optane slog)
SMB file access during work hours (from 4x windows pcs, here at my home office)
Backups via SMB and NFS - (ie veeam for some vms onsite, some offsite)
maybe 5x low use VMs running via NFS from it.
about 20tb of video files, that a plex VM accesses via SMB share
pool name="ssd3x6x480GBz2": (has split optane slog)
~ 15x VMs running via NFS from it. (all service type VMs, many windows 2012r2)
2x large PRTG VMs store their files here via NFS - (PRTG = snmp network motioning app)
2x heavy disk IO "torrent" VMs running 24/7 (use this ssd pool via NFS as temp/scratch download storage, then move out to other HDD pools via SMB when download completed)
pool name="red4tbENCz1": (has split optane slog)
3x milestone NVRs (cctv) archive their video to it daily (about 75gb per day)
light daily Backups via SMB - (ie some FTP / syncbackpro)
1x low use VMs running via NFS from it.
pool name="he5x8TBz1": (no slog)
ONLY ~ 20tb of video files, that a plex VM accesses via SMB share
random output of arcstats:
zpool status:
Some stats on the p3605;
(i know diskinfo is a write/slog relevant test, so not really relevant for a L2ARC drive, but i figured i would post at the bottom anyway)
so i have a extra intel p3605 1.6tb PCIe nvme card, and a few extra slots, so im considering adding it as a L2ARC, but wanted to get some feedback as to if i should even add an L2ARC. (i was all ready to do it, and even have tested it on a separate FN test server i keep, but after reading quite a bit, im having second thoughts as to if i should even run a l2arc).
The server specs/pool types are at the bottom (see SPOILER "Main FN BOX SPECS" ), it does have 256g of fast ecc ram. My usecase for the files on this FN box (physical box btw):
MY PLAN IS TO SPLIT THE L2ARC (ie ~400gb each for 3x of these pools)- please, i would like this discussion to mainly focus on if i should add an L2ARC, not on the wisdom (or lack of) of splitting/sharing a single 1.6tb drive l2arc. Thanks
pool name="he8x8TBz2": (has split optane slog)
SMB file access during work hours (from 4x windows pcs, here at my home office)
Backups via SMB and NFS - (ie veeam for some vms onsite, some offsite)
maybe 5x low use VMs running via NFS from it.
about 20tb of video files, that a plex VM accesses via SMB share
pool name="ssd3x6x480GBz2": (has split optane slog)
~ 15x VMs running via NFS from it. (all service type VMs, many windows 2012r2)
2x large PRTG VMs store their files here via NFS - (PRTG = snmp network motioning app)
2x heavy disk IO "torrent" VMs running 24/7 (use this ssd pool via NFS as temp/scratch download storage, then move out to other HDD pools via SMB when download completed)
pool name="red4tbENCz1": (has split optane slog)
3x milestone NVRs (cctv) archive their video to it daily (about 75gb per day)
light daily Backups via SMB - (ie some FTP / syncbackpro)
1x low use VMs running via NFS from it.
pool name="he5x8TBz1": (no slog)
ONLY ~ 20tb of video files, that a plex VM accesses via SMB share
random output of arcstats:
Code:
oot@freenas:~# arcstat.py
time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c
03:28:22 46G 2.5G 5 1.8G 3 748M 87 583M 1 182G 181G
# arc_summary.py
System Memory:
0.09% 225.63 MiB Active, 5.13% 12.80 GiB Inact
92.35% 230.43 GiB Wired, 0.00% 0 Bytes Cache
2.19% 5.46 GiB Free, 0.24% 617.35 MiB Gap
Real Installed: 256.00 GiB
Real Available: 99.97% 255.93 GiB
Real Managed: 97.49% 249.51 GiB
Logical Total: 256.00 GiB
Logical Used: 92.87% 237.74 GiB
Logical Free: 7.13% 18.26 GiB
Kernel Memory: 3.77 GiB
Data: 98.90% 3.73 GiB
Text: 1.10% 42.53 MiB
Kernel Memory Map: 249.51 GiB
Size: 7.81% 19.48 GiB
Free: 92.19% 230.03 GiB
Page: 1
------------------------------------------------------------------------
ARC Summary: (HEALTHY)
Storage pool Version: 5000
Filesystem Version: 5
Memory Throttle Count: 0
ARC Misc:
Deleted: 1.64b
Mutex Misses: 1.82m
Evict Skips: 1.82m
ARC Size: 73.21% 181.94 GiB
Target Size: (Adaptive) 73.23% 181.99 GiB
Min Size (Hard Limit): 12.50% 31.06 GiB
Max Size (High Water): 8:1 248.51 GiB
ARC Size Breakdown:
Recently Used Cache Size: 94.96% 172.80 GiB
Frequently Used Cache Size: 5.04% 9.18 GiB
ARC Hash Breakdown:
Elements Max: 7.23m
Elements Current: 79.66% 5.76m
Collisions: 314.86m
Chain Max: 6
Chains: 440.55k
Page: 2
------------------------------------------------------------------------
ARC Total accesses: 46.83b
Cache Hit Ratio: 94.56% 44.28b
Cache Miss Ratio: 5.44% 2.55b
Actual Hit Ratio: 94.34% 44.18b
Data Demand Efficiency: 53.34% 2.62b
Data Prefetch Efficiency: 12.03% 844.63m
CACHE HITS BY CACHE LIST:
Anonymously Used: 0.19% 85.99m
Most Recently Used: 4.48% 1.98b
Most Frequently Used: 95.29% 42.19b
Most Recently Used Ghost: 0.01% 3.38m
Most Frequently Used Ghost: 0.03% 14.49m
CACHE HITS BY DATA TYPE:
Demand Data: 3.15% 1.40b
Prefetch Data: 0.23% 101.60m
Demand Metadata: 96.60% 42.77b
Prefetch Metadata: 0.02% 8.12m
CACHE MISSES BY DATA TYPE:
Demand Data: 47.93% 1.22b
Prefetch Data: 29.16% 743.03m
Demand Metadata: 22.68% 578.00m
Prefetch Metadata: 0.23% 5.84m
Page: 3
------------------------------------------------------------------------
Page: 4
------------------------------------------------------------------------
DMU Prefetch Efficiency: 7.83b
Hit Ratio: 9.43% 738.60m
Miss Ratio: 90.57% 7.10b
Page: 5
------------------------------------------------------------------------
Code:
root@freenas:~ # zpool status
pool: freenas-boot
state: ONLINE
scan: scrub repaired 0 in 0 days 00:00:04 with 0 errors on Sat Sep 14 03:45:04 2019
config:
NAME STATE READ WRITE CKSUM
freenas-boot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
errors: No known data errors
pool: he5x8TBz1
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
he5x8TBz1 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/f9ad6c75-d748-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/fac4063a-d748-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/febf393e-d748-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/02df6e7a-d749-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/044e4834-d749-11e9-bae4-00259084f1c8 ONLINE 0 0 0
errors: No known data errors
pool: he8x8TBz2
state: ONLINE
scan: scrub repaired 0 in 1 days 07:56:31 with 0 errors on Wed Sep 11 12:55:02 2019
config:
NAME STATE READ WRITE CKSUM
he8x8TBz2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/ab4a8be7-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/ac4d1939-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/ad4d7a28-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/ae4b5c46-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/af54390e-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/b05a4b41-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/b15cf9b3-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/b2675d33-c451-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
gptid/e1eba74e-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/e34d7ec3-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/e4a114e3-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/e608021e-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/e727453f-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/eb0401da-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/eed1a7b2-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/f013e8e4-d1bc-11e9-bae4-00259084f1c8 ONLINE 0 0 0
logs
gptid/ed098586-c7bc-11e9-975e-00259084f1c8 ONLINE 0 0 0
errors: No known data errors
pool: hus9x4TBz2
state: ONLINE
scan: none requested
config:
NAME STATE READ WRITE CKSUM
hus9x4TBz2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/9a817388-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/9d201743-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/a077cd9e-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/a800a13f-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/b4bd6736-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/c2588d0b-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/cfea05aa-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/dabb68f5-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
gptid/e76a9705-c456-11e9-bbf0-00259084f1c8 ONLINE 0 0 0
errors: No known data errors
pool: red4tbENCz1
state: ONLINE
scan: resilvered 487G in 0 days 01:41:38 with 0 errors on Mon Sep 2 01:56:54 2019
config:
NAME STATE READ WRITE CKSUM
red4tbENCz1 ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0
gptid/85a9cf3a-cd40-11e9-975e-00259084f1c8.eli ONLINE 0 0 0
gptid/7afcb936-c450-11e9-bbf0-00259084f1c8.eli ONLINE 0 0 0
gptid/7ca5439e-c450-11e9-bbf0-00259084f1c8.eli ONLINE 0 0 0
gptid/7e4df0e4-c450-11e9-bbf0-00259084f1c8.eli ONLINE 0 0 0
gptid/7f8f24d1-c450-11e9-bbf0-00259084f1c8.eli ONLINE 0 0 0
logs
gptid/e82c363e-c7bc-11e9-975e-00259084f1c8 ONLINE 0 0 0
errors: No known data errors
pool: ssd3x6x480GBz2
state: ONLINE
scan: scrub repaired 0 in 0 days 00:29:33 with 0 errors on Mon Sep 16 02:03:06 2019
config:
NAME STATE READ WRITE CKSUM
ssd3x6x480GBz2 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/b238bbd8-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b2e8a1de-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b382351d-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b421e71c-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b4d79d37-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b580104e-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
raidz2-1 ONLINE 0 0 0
gptid/b670b4d0-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b714597f-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b7d8bca7-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b88979e7-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b94e1216-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/b9f0c93b-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
gptid/bb0fa8d3-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/bbb3f9cf-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/bc74e749-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/bd35b67b-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/bde00cf9-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
gptid/be9f1fbc-d82e-11e9-bae4-00259084f1c8 ONLINE 0 0 0
logs
gptid/f166ac78-c7bc-11e9-975e-00259084f1c8 ONLINE 0 0 0
errors: No known data errors
root@freenas:~ #
Some stats on the p3605;
(i know diskinfo is a write/slog relevant test, so not really relevant for a L2ARC drive, but i figured i would post at the bottom anyway)
Code:
root@freenas:~ # diskinfo -wS /dev/nvd0
/dev/nvd0
512 # sectorsize
1600321314816 # mediasize in bytes (1.5T)
3125627568 # mediasize in sectors
131072 # stripesize
0 # stripeoffset
INTEL SSDPEDME016T4S # Disk descr.
CVMD4414004X1P6KGN # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM
Synchronous random writes:
0.5 kbytes: 22.4 usec/IO = 21.8 Mbytes/s
1 kbytes: 22.1 usec/IO = 44.3 Mbytes/s
2 kbytes: 20.4 usec/IO = 95.9 Mbytes/s
4 kbytes: 15.4 usec/IO = 253.8 Mbytes/s
8 kbytes: 20.5 usec/IO = 380.7 Mbytes/s
16 kbytes: 26.9 usec/IO = 581.8 Mbytes/s
32 kbytes: 36.4 usec/IO = 858.3 Mbytes/s
64 kbytes: 49.5 usec/IO = 1263.3 Mbytes/s
128 kbytes: 94.9 usec/IO = 1316.8 Mbytes/s
256 kbytes: 168.1 usec/IO = 1487.0 Mbytes/s
512 kbytes: 316.9 usec/IO = 1577.7 Mbytes/s
1024 kbytes: 648.5 usec/IO = 1542.0 Mbytes/s
2048 kbytes: 1288.0 usec/IO = 1552.8 Mbytes/s
4096 kbytes: 2586.7 usec/IO = 1546.4 Mbytes/s
8192 kbytes: 5110.6 usec/IO = 1565.4 Mbytes/s
root@freenas:~ #
root@freenas:~ # smartctl -a /dev/nvme0
smartctl 6.6 2017-11-05 r4594 [FreeBSD 11.2-STABLE amd64] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Number: INTEL SSDPEDME016T4S
Serial Number: CVMD4414004X1P6KGN
Firmware Version: 8DV1RA13
PCI Vendor ID: 0x8086
PCI Vendor Subsystem ID: 0x108e
IEEE OUI Identifier: 0x5cd2e4
Controller ID: 0
Number of Namespaces: 1
Namespace 1 Size/Capacity: 1,600,321,314,816 [1.60 TB]
Namespace 1 Formatted LBA Size: 512
Local Time is: Sat Sep 21 16:47:45 2019 CDT
Firmware Updates (0x02): 1 Slot
Optional Admin Commands (0x0006): Format Frmw_DL
Optional NVM Commands (0x0006): Wr_Unc DS_Mngmt
Maximum Data Transfer Size: 32 Pages
Supported Power States
St Op Max Active Idle RL RT WL WT Ent_Lat Ex_Lat
0 + 25.00W - - 0 0 0 0 0 0
Supported LBA Sizes (NSID 0x1)
Id Fmt Data Metadt Rel_Perf
0 + 512 0 2
1 - 512 8 2
2 - 512 16 2
3 - 4096 0 0
4 - 4096 8 0
5 - 4096 64 0
6 - 4096 128 0
=== START OF SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
SMART/Health Information (NVMe Log 0x02, NSID 0xffffffff)
Critical Warning: 0x00
Temperature: 30 Celsius
Available Spare: 100%
Available Spare Threshold: 10%
Percentage Used: 1%
Data Units Read: 2,757,905 [1.41 TB]
Data Units Written: 1,053,105 [539 GB]
Host Read Commands: 6,052,800,385
Host Write Commands: 8,210,809,402
Controller Busy Time: 0
Power Cycles: 388
Power On Hours: 19,775
Unsafe Shutdowns: 1
Media and Data Integrity Errors: 0
Error Information Log Entries: 0
Error Information (NVMe Log 0x01, max 64 entries)
No Errors Logged
Main system FINAL (still testing/building as of aug 2019)
OS: FreeNAS-11.2-U5 (stable) (boot volume is a mirror of 60gb intel 520 SSDs)
MB: X9DR3-LN4F+ rev 1.20A
CPU: 2x E5-2637 v2 (4c/3.5ghz)2x e5-2640v2
RAM: 8x 32gb (256g@1866mhz) Samsung M386B4G70DM0-CMA4 (ecc + is on MB's QVL list)
Case: 4u 24bay (cse-825)
PSUs: 2x 920sq (Supermicro)
HBAs: 1x- lsi 9207-8i / 2x- lsi 9207-8e (all are at FW= 20)
10g NIC: CHELSIO T520-CR (2x 10g sfp+)
NVMe: 280g Optane p900 (as a speed test pool or zil for now)
DISKS:
MANYx 8TB HGST "He8" SAS 7200rpm (HUH728080AL5200)
21x 480gb HGST SSD (HUSMR1650ASS204)
6x 200gb HGST SSD (HUSMM8020ASS201)
9x 4tb HGST SAS 7200rpm (HUS724040ALE640)
5x 4tb WD RED SATA 5400rpm
23x 3tb HGST SAS 7200rpm (HUS724030ALS640) (might not use these)
POOLs:
* 16x 8tb - 2x vDEVs of 8x-disk in Z2
(on main chassis bays via lsi 9207-8i to BPN-SAS2-846el1, name="he8x8TBz2")
* 5x 8tb - Single vDEVs of 5x-disk in Z1
(on main chassis bays via lsi 9207-8i to BPN-SAS2-846el1, name="he5x8TBz1")
*18x 480gb SSD - 3x vDEVs of 6x-SSDs in Z2
(on 2u "disk shelf2" via lsi 9207-8e to BPN-SAS-216el1 name="ssd3x6x480GBz2")
* 9x 4tb - Single vDEV of 9x-disks in Z2
(on 3u "disk shelf1" via lsi 9207-8e to BPN-SAS2-836el1 name="hus9x4TBz2")
* 5x 4tb - Single Encrypted vDEV of 5x-disk in Z1
(on main chassis bays via lsi 9207-8i to BPN-SAS2-846el1, name="red4tbENCz1")
Disk Shelf 1: xx
Case: 3u 16bay (Supermicro)
PSUs: 2x 920sq (SM)
Expander/BP: BPN-SAS2-836el1
Other: no MB, just a on/off SW and a Noctura Fan dial/knob
DISKS: 15x 3tb HGST SAS 7200rpm (HUS724030ALS640)
Disk Shelf 2: SSDs (ie pool "ssd3x6x480GBz2" disks are here)
Case: 2u 24bay 2.5" bays (Supermicro)
PSUs: 2x 920sq (SM)
Expander/BP: BPN-SAS-216el1
Other: no MB, just a on/off SW and a Noctura Fan dial/knob
DISKS: 19x 480gb HGST SAS3 SSDs (HUSMR1650ASS204)
Disk Shelf 3: xx
Case: 4u 24bay (Supermicro)
PSUs: 2x 920sq (SM)
Expander/BP: BPN-SAS2-846el1
Other: no MB, just a on/off SW and a Noctura Fan dial/knob
DISKS: 15x 3tb HGST SAS 7200rpm (HUS724030ALS640)
Other:
10g SW: Ubiquiti ES‑16‑XG (not unifi sw)
1g SW: D-Link DGS-1510 (24x 1g eth + 4x 10g SFP)
ePDU: 3x Eaton G3 EMA115-10 (per outlet power monitoring / switching)
UPS: 3x APC SMT-1500 (w AP9631 Network MGMT cards, FN connected/working)
Rack1: 42U StarTech Open Frame 4-Post- Adjustable Depth
Rack2: 25U StarTech Open Frame 4-Post- Adjustable Depth - on "furniture" sliders so can slide around
OS: FreeNAS-11.2-U5 (stable) (boot volume is a mirror of 60gb intel 520 SSDs)
MB: X9DR3-LN4F+ rev 1.20A
CPU: 2x E5-2637 v2 (4c/3.5ghz)
RAM: 8x 32gb (256g@1866mhz) Samsung M386B4G70DM0-CMA4 (ecc + is on MB's QVL list)
Case: 4u 24bay (cse-825)
PSUs: 2x 920sq (Supermicro)
HBAs: 1x- lsi 9207-8i / 2x- lsi 9207-8e (all are at FW= 20)
10g NIC: CHELSIO T520-CR (2x 10g sfp+)
NVMe: 280g Optane p900 (as a speed test pool or zil for now)
DISKS:
MANYx 8TB HGST "He8" SAS 7200rpm (HUH728080AL5200)
21x 480gb HGST SSD (HUSMR1650ASS204)
6x 200gb HGST SSD (HUSMM8020ASS201)
9x 4tb HGST SAS 7200rpm (HUS724040ALE640)
5x 4tb WD RED SATA 5400rpm
23x 3tb HGST SAS 7200rpm (HUS724030ALS640) (might not use these)
POOLs:
* 16x 8tb - 2x vDEVs of 8x-disk in Z2
(on main chassis bays via lsi 9207-8i to BPN-SAS2-846el1, name="he8x8TBz2")
* 5x 8tb - Single vDEVs of 5x-disk in Z1
(on main chassis bays via lsi 9207-8i to BPN-SAS2-846el1, name="he5x8TBz1")
*18x 480gb SSD - 3x vDEVs of 6x-SSDs in Z2
(on 2u "disk shelf2" via lsi 9207-8e to BPN-SAS-216el1 name="ssd3x6x480GBz2")
* 9x 4tb - Single vDEV of 9x-disks in Z2
(on 3u "disk shelf1" via lsi 9207-8e to BPN-SAS2-836el1 name="hus9x4TBz2")
* 5x 4tb - Single Encrypted vDEV of 5x-disk in Z1
(on main chassis bays via lsi 9207-8i to BPN-SAS2-846el1, name="red4tbENCz1")
Disk Shelf 1: xx
Case: 3u 16bay (Supermicro)
PSUs: 2x 920sq (SM)
Expander/BP: BPN-SAS2-836el1
Other: no MB, just a on/off SW and a Noctura Fan dial/knob
DISKS: 15x 3tb HGST SAS 7200rpm (HUS724030ALS640)
Disk Shelf 2: SSDs (ie pool "ssd3x6x480GBz2" disks are here)
Case: 2u 24bay 2.5" bays (Supermicro)
PSUs: 2x 920sq (SM)
Expander/BP: BPN-SAS-216el1
Other: no MB, just a on/off SW and a Noctura Fan dial/knob
DISKS: 19x 480gb HGST SAS3 SSDs (HUSMR1650ASS204)
Disk Shelf 3: xx
Case: 4u 24bay (Supermicro)
PSUs: 2x 920sq (SM)
Expander/BP: BPN-SAS2-846el1
Other: no MB, just a on/off SW and a Noctura Fan dial/knob
DISKS: 15x 3tb HGST SAS 7200rpm (HUS724030ALS640)
Other:
10g SW: Ubiquiti ES‑16‑XG (not unifi sw)
1g SW: D-Link DGS-1510 (24x 1g eth + 4x 10g SFP)
ePDU: 3x Eaton G3 EMA115-10 (per outlet power monitoring / switching)
UPS: 3x APC SMT-1500 (w AP9631 Network MGMT cards, FN connected/working)
Rack1: 42U StarTech Open Frame 4-Post- Adjustable Depth
Rack2: 25U StarTech Open Frame 4-Post- Adjustable Depth - on "furniture" sliders so can slide around
Last edited: