My rack!

[xnoodle]

This is in a bomb shelter connected to my basement, has a kitchen, bathroom, generator, pump and blower fan. Noise isn't a problem!
Neither is space in the rack, not even using half of the 42U, so I'm using the shelves as working surfaces, as the switches aren't full depth.
Have 100TB+ of raw storage (but less then half is in use).

Current setup is:
Shelf 1:
Dell 15" monitor
Dell 2724 switch

Shelf 2:
Fulcrum Micro Focalpoint 10gbE switch (behind the keyboard)
USB keyboard

2 x APC Smart-UPS 750VA 2U 120V

Chassis 1: Norco 4220

• Supermicro X9SCM-F
  • CPU: E3 1230
  • RAM: 16 GB (4 x 4 DDR3 ECC UDIMM)
  • HBA: Areca ARC-1680LP (ESXi datastore for VMs)
    • 2 x Seagate Barracuda 7200.11 ST31500341AS 1.5TB in RAID1
  • HBA: M1015 x 2 (Passed through to a VM)
  • Onboard ICH with reverse SATA cables (Passed through to a VM)
    • 12 x 4TB drives in RAIDZ2

Chassis 2: Dell C6100

• Node 1: 2 x L5520, 96 GB of 1.35V RAM, dual port 10gb SFP+ (TCK99), IBM BR10i SAS1068 card, 2 x 2TB 7.2K drives
• Node 2: 1 x L5520, 48 GB of 1.35V RAM, 30 GB OCZ Vertex SSD
• Node 3: 1 x L5520, 48 GB of 1.35V RAM, dual port 10gb SFP+ (TCK99), IBM M1015, Intel 160GB G3 SSD
• Node 4: 1 x L5520, 48 GB of 1.35V RAM, Intel 160GB G3 SSD

Chassis 3: Norco 4020 (not in rack, handle can be seen to the right)

• PICMG 1U backplane
  • HP SAS Expander
    • EMPTY

Notes:
Was experiencing data corruption with using the DAS and a internal SFF-8087 cable. Maybe need to get a SFF-8088.. but I don't know where I'm going to stash a internal SFF-8087<->SFF-8088 converter in the C6100.. or get a SAS2X08 card with a external port.
Have 14 2TB and 10 3TB drives unplugged currently

TODO:
Put 10GB Switch into use. Was drawing too much power while I was powering the C6100 stuff down and up.
Migrate VMs off Chassis1 (ESXi) to Chassis2.Node1 (Xen)
Fix data corruption issue and use Chassis1 as DAS to Chassis2.Node3
Fix data corruption issue and use Chassis3 as backup DAS to Chassis2.Node4 (need to figure out how to turn on chassis as needed)

 

[nry]

Very nice!

Slightly jealous of the Dell C6100, would love one of these

 

[xnoodle]

Notes:
Was experiencing data corruption with using the DAS and a internal SFF-8087 cable. Maybe need to get a SFF-8088.. but I don't know where I'm going to stash a internal SFF-8087<->SFF-8088 converter in the C6100.. or get a SAS2X08 card with a external port.

So, swapped controller, ram and cable and still experiencing checksum errors in ZFS. Likely culprit is either the backplanes or the SAS expander now. frown.

 

[Lost-Benji]

Hey Noodles, can you try something for me?

Take your systems off the UPS's and direct feed then from mains. The 750VA's are on the too small side of the friendship. something of at least 1KVA or better should be in play. 750VA will only see about a 500-550W output meaning that there limts maybe too close and you are getting dirty/noisy power into your other systems. Might be a long-shot but still stands with the power limits your facing.

What are the details on the HDD's in use?

Besides the the limited power you running on, you may be having corruption issues thanks to running SATA drives on SAS cables and backplanes. SAS be default runs higher signalling voltages however, SATA does not and is a lot more touchy to excessive cabling and connections. I by my own findings have always run an expander either directly attached to the drives or the shortest SFF-8087 cables to the backplane. The SFF-8088 cable can be longer as this will still signal to the HBA at SAS levels.