SAS/HBA SFF-8088 Maximum bandwidth and recommendations

[ericloewe]

You could, that's not an issue, though LSI SAS 9300s have gotten pretty cheap. You just need suitable SFF-8088 to SFF-8644 cables, or SFF-8087 to SFF-8643, depending on how you want to write things out.
SAS3 is mostly run over SFF-8643/8644, which is more popular these days. You may recognize SFF-8643 as the connector used in the early days of NVMe to wire up U.2 disks, before the current zoo of higher-density connectors displaced it from most applications. It's still used all over the place for SAS and SATA, though.

 

[vocalicacorn187]

Weird note on this-just saw that core only supports a single SAS JBOD, and scale is needed for multiple enclosures. Not sure how that is documented.

I don't believe Core would be able to tell how SAS Expanders you are using.

I am running Core currently with 2 SAS expanders. The same Intel E91267-203 RES2SV240 that is being recommended. I am running them both off of an LSI 9201-16i. One is running 16 HDD and the other is running 8 HDD. The one expanded is inside of my main case, and the other is in a Node 804 that I am using as a disk shelf. When I first set it up I was using 8088 cables to adapter brackets. I ended up running into UDMA CRC errors with that setup. I am now just running 8087 cables directly to the expander from the HBA. No more errors. I do have new 8088 cables on order, but I don't even know if that was the issue. It may have been the two 8087 cables in the disk shelf that I am not using now, or the adapters. Debating if I even want to investigate further or just leave for now since it is working.

 

[StrongFall]

The setup I ended up going with:
- Broadcom HBA 9500-8e - approx 170 GBP
- Intel Storage Expander RES3FV288 - Approx 100 GBP
- SFF-8643 to 4 DATA - Approx 10 GBP
- SFF-8644 to SFF-8644 - Approx 20 GBP

I will be building my enclosure in the next few days and putting all this to the test

 

[gregsachs]

Good luck-don't forget about fan control, and you can jumper the power supply for free:

How to Jump a PSU | bit-tech.net

A quick guide on how to power up and test specific components (like a pump) without turning on your whole system.

bit-tech.net

 

[StrongFall]

The setup I ended up going with:
- Broadcom HBA 9500-8e - approx 170 GBP
- Intel Storage Expander RES3FV288 - Approx 100 GBP
- SFF-8643 to 4 DATA - Approx 10 GBP
- SFF-8644 to SFF-8644 - Approx 20 GBP

I will be building my enclosure in the next few days and putting all this to the test

I've finally found the time to put this together and I noticed something odd when testing.

The drives connected through the HBA seem to be significantly slower (IOPS) compared to being connected directly. I would imagine due to the additional complexity there would be SOME amount of reduction in performance but the levels I'm seeing seem unworkable.

I have tested 4 SSDs connected through the HBA and connected directly to the motherboard via SATA.

4KB Ran Read (QD1) x1000	4KB Ran Write (QD1) x1000	4KB Ran Read (QD32) x1000	4KB Ran Write (QD32) x1000	Connection Type
11.1	0.7376	38.6	16.6	HBA
238	119	238	119	Direct

The above is a table comparing my IOPS performance of a single drive with the only difference being connection type. This drive when connected directly achieves 238,000 IOPS but when connected via the HBA only achieves 11,100 IOPS which is more than a 95% decrease.

I assume this is not normal at all?

Do you have any ideas on what would be good steps towards diagnosing and resolving the issue?

The temps of the drives do not exceed 30C and both the HBA and Expander are not hot to the touch.

I feel like it might have something to do with the cables since they are the only part of the setup that uses generic branded equipment
10Gtek® 12G External Mini SAS HD SFF-8644 to SFF-8644 Cable, 1-Meter(3.3-ft)
YIWENTEC Internal HD Mini SAS (SFF-8643 Host) to 4X SATA (Target) Hard Drive Cable (8643 SATA 0.5M)

EDIT: Forgot to mention - I am seeing the "CRC error count" increase over time whilst the drives are connected via the HBA

EDIT: My current next steps are to try a different 8643 to SATA cable - I have purchased 2 from different brands (10Gtek and StarTech)

 

[ericloewe]

EDIT: Forgot to mention - I am seeing the "CRC error count" increase over time whilst the drives are connected via the HBA

Yeah, your cabling setup is inadequate. PHY retries == fewer IOPS.

I feel like it might have something to do with the cables since they are the only part of the setup that uses generic branded equipment
10Gtek® 12G External Mini SAS HD SFF-8644 to SFF-8644 Cable, 1-Meter(3.3-ft)
YIWENTEC Internal HD Mini SAS (SFF-8643 Host) to 4X SATA (Target) Hard Drive Cable (8643 SATA 0.5M)

The expander is in the external chassis, right? 1.5 m is way over the 1-meter limit for SATA, which is often problematic enough even for runs that approach it from the shorter end - so the only sane solution is to have the expander in the external chassis, so that the exterior run is SAS, which does have signalling tolerant of such runs.

 

[StrongFall]

The expander is in the external chassis, right? 1.5 m is way over the 1-meter limit for SATA, which is often problematic enough even for runs that approach it from the shorter end - so the only sane solution is to have the expander in the external chassis, so that the exterior run is SAS, which does have signalling tolerant of such runs.

Yeah, the expander is in the external chassis. The run from the expander to the actual drive is 0.5M using the SFF-8643 to 4xSATA cable.

The expander from the external chassis is then connected using 2xSFF-8644 (1M each) to the HBA which is plugged into the host via PCI.

I thought the SATA limit only applied to between the expander and the actual drive - which in this case is 0.5M, under the SATA limit?

Are you trying to say the entire run from drive to HBA should be less than 1M?

 

[ericloewe]

The expander from the external chassis is then connected using 2xSFF-8644 (1M each) to the HBA

Try a single cable.

thought the SATA limit only applied to between the expander and the actual drive - which in this case is 0.5M, under the SATA limit?

It does, it's just important to be clear about these things, don't want go down the rabbit hole if it's a too-long cable run.

 

[StrongFall]

Try a single cable.

Just gave this a try - it did not make a difference

 

[ericloewe]

It could be just bad breakout cables, I guess. Also worth checking that all firmware is up to date.

 

[gregsachs]

Any ability to use breakout cable direct to the HBA and test that way? Would isolate issue to the cable..

 

[StrongFall]

It could be just bad breakout cables, I guess. Also worth checking that all firmware is up to date.

The firmware on the HBA is pretty old - version 14 but version 20 is available. I think I am going to hold fire on this. According to the seller, it was a fully functioning HBA so I'm hoping it's just bad cables.

Any ability to use breakout cable direct to the HBA and test that way? Would isolate issue to the cable..

Thanks for the suggestion - I just purchased a SFF-8643 to SFF-8644 converter (not expander). This should allow me to remove the storage expander from the equation for testing purposes.

Out of interest, should I be expecting any performance decrease when comparing a drive connected directly via SATA to the motherboard vs through storage expanders, HBAs etc?

EDIT: Just updated the HBA from version 14.00.00.00 to 29.00.00.00 (latest) - this did not improve the situation

 

[nexox]

Out of interest, should I be expecting any performance decrease when comparing a drive connected directly via SATA to the motherboard vs through storage expanders, HBAs etc?

HBAs tend to operate exactly at the maximum speed of whatever generation they support, onboard SATA isn't so consistent, but there's an overhead penalty for transporting SATA over SAS (I forget exactly how much, something like 8 or 10%,) so unless your onboard SATA is pretty bad then the HBA may be a little slower if your drive can saturate the link. Expanders don't usually have much of a performance impact, as long as you don't hit the bandwidth limits of the link to the HBA.

 

[nabsltd]

The firmware on the HBA is pretty old - version 14 but version 20 is available. I think I am going to hold fire on this. According to the seller, it was a fully functioning HBA so I'm hoping it's just bad cables.

I bought a breakout cable one time and got sent a reverse breakout. They look pretty much identical, but the reverse would not let me see drives reliably.

 

[ericloewe]

A reverse-breakout cable would not work at all, unless it's a super dodgy cable that's miswired to half forward, half reverse breakout.

 

[StrongFall]

Just finished a bunch of testing and it looks like the IOPS issue is not related to the HBA/Expander/SATA cables at all.

The issue is the filesystem used. In my earlier posts I was comparing the IOPS reading from two of the same drive, one was connected through the HBA and the other was connected directly via SATA. The other big difference was that one was XFS and the other was NTFS.

I guess it's time to go down the filesystem rabbit hole and try to figure out why this is happening haha

These are the commands I have been using for IOPS testing

# 4KB Ran Read (QD1)
# 4KB Ran Write (QD1)
# 4KB Ran Read (QD32)
# 4KB Ran Write (QD32)

fio --name TEST --filename=deleteMe.fio --rw=randread --size=500m \
     --io_size=10g --blocksize=4k --ioengine=libaio \
     --direct=1 --numjobs=1 --runtime=60 --group_reporting --fsync=0 --iodepth=1 | grep IOPS

sleep 5

fio --name TEST --eta-newline=5s --filename=deleteMe.fio --rw=randwrite --size=500m \
     --io_size=10g --blocksize=4k --ioengine=libaio \
     --direct=1 --numjobs=1 --runtime=60 --group_reporting --fsync=0 --iodepth=1 | grep IOPS

sleep 5

fio --name TEST --eta-newline=5s --filename=deleteMe.fio --rw=randread --size=500m \
     --io_size=10g --blocksize=4k --ioengine=libaio \
     --direct=1 --numjobs=1 --runtime=60 --group_reporting --fsync=0 --iodepth=32 | grep IOPS

sleep 5

fio --name TEST --eta-newline=5s --filename=deleteMe.fio --rw=randwrite --size=500m \
     --io_size=10g --blocksize=4k --ioengine=libaio \
     --direct=1 --numjobs=1 --runtime=60 --group_reporting --fsync=0 --iodepth=32 | grep IOPS

 

[recursed]

Regarding enclosure, a very important question is rack mount or desk mount, and noise tolerance.

Something like the old Xyratex 1235, Netapp DS4243 or 4246 are enclosures with SAS expanders that work well, but can be noisy. Supermicro also makes. Avoid the Netapp 448x, which is an odd bastard. The EMC KTN-STL3 are also pretty dated at this point.
Also make sure you get the trays/caddies, controllers, and power supplies. There are a LOT of 2.5" units on ebay now, cheaper than 3.5", so if you are looking at SSD storage you can do well.
Here is a SAS3 2.5" x24 unit for ~$150 shipped in the US.

HPE HP 3PAR 8000 Storage JBOD Disk Array 12Gbs 24x 2.5" SAS SSD H6Z26 QR491 | eBay

HPE HP 3PAR 8000 Storage JBOD Disk Array 12Gbs. Can also be used as a server JBOD, connect with an external SAS card and a high density mini SAS cable. 24x 2.5" SAS / SSD Trays Installed, No Hard Drives.

www.ebay.com

You will see most units look very similar, that is because most are OEM by seagate at this point I believe.

Listing claims that enclosure is sas only no sata. Can anyone confirm or deny?

 

[gregsachs]

Listing claims that enclosure is sas only no sata. Can anyone confirm or deny?

I have not played with the HPE unit. I have a Netapp ds2246 in my basement that definitely works with SATA drives, with a caveat, they are only seen by one of the two controllers due to not having dual path. I think there may be some trays with SATA/SAS interposers that would solve that problem, but i don't need dual path. My understanding is that SATA support is a requirement of the SAS specification, and that is why SATA drives can plug into SAS connections, but SAS drives won't plug into SATA connections, but i wouldn't swear to it. .