Exploring logical options for LSI 12Gb/SAS HBA's for high performance storage target...

[Sniper_X]

Background Info:

After a long delay, I am continuing my "InfiniBeast" project of building a super fast, (5GB+/sec) storage target.

I am basing it on an Ultrastar Data60 Hybrid Storage Platform.

Although I might change this plan, currently, the build will mimic/re-create the build-out outlined in this whitepaper.
(The complete list of reference architecture designs are here - scroll down a bit to "Reference Architecture")

Rationale for this post:

I am looking for advice on obtaining the SAS HBAs required for this build and know that many vendors OEM the LSI/Broadcom/Avago controllers.
This affects pricing quite a bit.

I have a great deal of experience with flashing and using these controllers, so the idea of re-flashing and/or using alternately named cards is no problem for me.

The build I'm looking at calls for using LSI 9400-8e HBA's.
I'm okay using those exact cards.

However, I also know that newer cards have been released, and choosing correctly would allow me to continue to use the HBA's further into the future as I upgrade the storage servers I build to front this platform.

Currently I like the 9500 series since these are PCIe 4.0 and the 9400 series is only PCIe 3.1.

The Question:

What is the best (i.e. easiest and least expensive) way to obtain an LSI 9500, or better LSI card without straying too far from the 9400 series HBA prescribed?

I am drawn to using this table at LSI/AVAGO that outlines their SAS/SATA Storage I/O Controllers (IOCs) to help select my HBA platform.

However, I'm open to other ideas.

 

[Sniper_X]

I have purchased the LSI 9400-8e HBAs and installed them in a Windows 10 PC on running a gigabyte motherboard w/32Gb RAM.
(It's my test system to flash cards and perform other testing)

With both HBAs installed, flashed to the latest firmware and BIOS, all 4 cables installed to the WD Ultrastar Data60 target, I could see the drives.
(It's quite a thing to see two HBAs list out all the drives during post. - LOL)

I then saw that Windows saw all the drives in Disk Manager, and they were presented FOUR TIMES!.
(Of course, this was because multipathing wasn't set up on Windows 10)

I highlighted the first set of drives (ignoring the other 3 sets of presentation), and created a RAID 0 using Windows.

Once the (HUGE) RAID 0 drive was online, I ran Atto benchmark on it.

I was stunned to see that with no tuning, no special drivers, and no preperation other than "turning it on", I was able to see the system benchmark at...

~5.2 GIGABYTES per second on both READ & WRITE tests!

In fact, I ran TWO instances of Atto, with the same settings, started them at the same time and the I/O was split between them, but it did bump the aggregate throughput to 6GB/sec. at several points!

This speed was seen consistently starting at the 128k record sizes and this continued all the way to the end of the test.

I'm now running TrueNAS Core on the same system and running tests there, familiarizing myself with that platfom, but who know what I will end up using in the end.

I'm looking at ZFS, Lustre, and CEPH based storage platforms. (I'm open to suggestions).

I am now in search of a dual node server to build into an HA controller/filer head.

 

[ano]

zfs probably best bet, since you only have 1

 

[Sniper_X]

zfs probably best bet, since you only have 1

Only 1 JBOD or 1 server?

The JBOD is VERY redundant internally, and can be zoned into virtual JBOD units.
With so many drives, I had intended to split it up.

 

[ano]

how many zones though? ceph usually likes 4+, but enjoys 12+

I have a few of the data102 incoming here, looking forward to play with them, usually ran supermicro (and have a few of the new ones as well)

 

[Sniper_X]

how many zones though? ceph usually likes 4+, but enjoys 12+

I have a few of the data102 incoming here, looking forward to play with them, usually ran supermicro (and have a few of the new ones as well)

Well, good question.
I have only just turned this on in the last 48 hours and am in the initial stages of learning how to use it.
I still need the WD tools to manage it, but have fired it up in a minimal way so far with impressive results.

Regarding the 102's and also the other units you mention, what do you have?
I'm interested in what more I might want to add to this setup

 

[Sniper_X]

For the record, I was able to capture the performance results of the Windows test PC using RAID0 created across 32 drives (the maximum a stripe or span can be in windows) using disk manager.

Here is what I got...

A Single instance of ATTO benchmark:




Here is TWO instances of ATTO running at the same time, performing in 100% sync together:

Windows Perfmon recorded almost 7GB per second of I/O on the back-end during the run of this test.