Capped at total 2gigabyte/s--SAS2-846-EL1 blackplane bottleneck?

[bouys]

Hey folks,

I've been a data hoarder for a while but recently I've taken the plunge into moving my unraid server into an old supermicro setup I bought off the 'bay. Formerly I'd had a desktop motherboard with a couple HBAs and pass-through backplanes so I had no issues with bandwidth. The machine has:

X9DRi-F motherboard
supermicro BPN-SAS2-846EL1 backplane
LSI 9211-8i (pci-e 2.0 x 8 lane) HBA
17 3.5" 5400rpm HDDs

Unraid needs to read from all the drives at once so overall bandwidth can be a bottleneck. When reading from all the drives at once I find it's capped at almost exactly 2 gigabytes/s. I want to go up to 24 drives, which would be limiting them to less than 100mb/s each. That'd be a pretty significant impairment. So I need to figure out where the bottleneck is, but it doesn't seem to be as simple as it could be.

Backplane. The EL1 has 3 SFF-8087 ports on the back. The server came with an LSI raid card connected to two of the ports. It came with two SFF8087 cables which I'm using.

As I understand it, the backplane lets you connect two SFF8087 connectors from the HBA to double the bandwidth, which should give me 4gb/s. Explained here: https://forums.servethehome.com/index.php?threads/home-server-build.18782/#post-182595

I only get 2gb/s, which says only one SFF8087 is being utilized. But performance seemed to fall off with just one SFF8087 connected (maybe I connected the wrong one?).

I have the latest firmware for the motherboard and HBA, not sure how to (or if I can) upgrade the firmware on the backplane. It seems like it could be a compatibility issue with the HBA, but I've tried it with a 9201 I had lying around, and I got the same bottleneck.

Any suggestions as to where I could go from here?

As an aside, I'm getting poor write performance with sata SSDs connected to the motherboard's sata iii ports. The motherboard's not that old; I would expect them to work at full speed (~500megabytes /s) but write speeds are poor (about 175 megabytes/s).

Thanks in advance!

 

[bouys]

If anybody's looking for an answer to this question, I don't really have one. To upgrade the firmware on the backplane, I think you have to e-mail supermicro and beg for it. The GUI utility didn't work on windows 10 when I tried it, either.

But I did figure it out, more or less. I got an H220 HBA (PCIe 3.0, LSI SAS 2308) and it seems to work great and isn't limited by a single SFF8087.

I did update its firmware to the latest LSI using the old flasher trick.

 

[vangoose]

SAS2-846EL1 backplanes have a single-port expander that accesses all hard drives and supports cascading.

SAS2-846EL2 backplanes have dual-port expanders that access all the hard drives. These dual-port expanders support cascading, failover, and multipath.

The improvement you see in performance is from PCI-E version.

 

[Spartacus]

For the slow SSD are you doing a passthrough/direct mounted disk? or is it a cache drive?

 

[bouys]

It's a cache drive, though I'm now pretty sure the issues I was getting were caused by low single-threaded performance of the CPU.

 

[Spartacus]

Single or multiple drives? Sounds like CPU as you ascertained

 

[bouys]

It was multiple drives actually, RAID0 btrfs. When I run two threads with a slightly faster cpu (2650 v2) I get full speed.

 

[pro_lamer]

It was multiple drives actually, RAID0 btrfs. When I run two threads with a slightly faster cpu (2650 v2) I get full speed.

What was the previous CPU?

Sent from my phone

 

[bouys]

A 2620 v1. The performance of the 2650 isn't what I'd expect but it's better.

The prior unraid server had an i7 6700k which I thought was overkill, but it had very high good single-thread performance and could push a single-threaded samba transfer at the full 1gigabyte/s.

I just didn't realize samba was this single-thread cpu limited after always having had either A) slow network or B) really fast single-thread CPU.

 

[koisama]

But I did figure it out, more or less. I got an H220 HBA (PCIe 3.0, LSI SAS 2308) and it seems to work great and isn't limited by a single SFF8087.

What firmware version do you have on your sas2308 card? I have a server with almost exactly same hardware, just with a 2308 HBA, and my bandwidth is limited to roughly 1.8GB/s when both ports are connected and half that if I disconnect one of the ports.

Backplane firmware version is 55.14.11.00, HBA firmware is 20.00.07.00

 

[azev]

If you are using sata drives then you are limited to only 1 connection. Typically the performance offered by expander would be more than enough to support spinners arrays. If you insist on getting the max performance on each drive, you will have to swap your expander to passthrough (A) model, and use multiple conntroller to support 24 drives.

 

[koisama]

Yes, I'm using consumer WD SATA drives. I expected to have a performance bottleneck due to expander, but 1.8GB/s from two physical ports seems way too low.
Speed figures seem very similar to this thread: https://forums.servethehome.com/ind...ders-60-ibm-lsi-chip-intel-alternative.11365/
This, together with OP saying that he fixed it by replacing a HBA, makes me think that the issue can be solved without replacing a backplane and buying more HBAs.

edit: found a way to get the link info, 4x 6Gb/s initiators in total. Numbering order suggests that only 2 links on each port are used.

 

[koisama]

Alright, I solved it. One of the cables was originally connected to J2. After moving it to J1 I got all 8 links from two ports.

 

[heavyarms2112]

Yes, I'm using consumer WD SATA drives. I expected to have a performance bottleneck due to expander, but 1.8GB/s from two physical ports seems way too low.
Speed figures seem very similar to this thread: https://forums.servethehome.com/ind...ders-60-ibm-lsi-chip-intel-alternative.11365/
This, together with OP saying that he fixed it by replacing a HBA, makes me think that the issue can be solved without replacing a backplane and buying more HBAs.

edit: found a way to get the link info, 4x 6Gb/s initiators in total. Numbering order suggests that only 2 links on each port are used.

How did you check the count of links?