PCIe 4.0 HBA's?

jeffarese

New Member
Sep 10, 2019
8
0
1
I was wondering if there's going to be HBA's that use PCIe 4.0 so there can be double the ports in a single HBA, it would be good for a x570 powered NAS.
 

Evan

Well-Known Member
Jan 6, 2016
3,252
560
113
I am sure we will see them soon, HBA from LSI today support even NVMe so PCI4 makes total sense then.
 

jeffarese

New Member
Sep 10, 2019
8
0
1
Yes there will be.

In 2020, as part of the U.3 transition, we discussed here https://www.servethehome.com/toshiba-aggressively-pushes-pcie-gen4-nvme-ssds/ one of the big benefits is that backplanes work across different types of drives. Usually, the storage guys are a bit behind the platform guys. Ice Lake Xeons in 2020 is when all of the current systems get re-designed.
Nice!

I think that makes X570 a very interesting choice for a low-power yet powerful NAS, since you could easily add more drives than you could fit in almost any case with some PCIe 4.0 lanes

What do you think?
 

nezach

Active Member
Oct 14, 2012
205
113
43
PCIe 4.0 cards from LSI will be based on Aero/Sea (3816/3916) controllers. They have been working on adding mpt3sas driver support for a while.

Here is one recent commit describing different performance modes that will be supported by new controllers

scsi: mpt3sas: Introduce perf_mode module parameter · torvalds/linux@ca7e1e9

The most interesting bit of info from that commit:

Code:
4k Random Read IO performance numbers on 24 SAS SSD drives for above three
permormance modes. Performance data is from Intel Skylake and HGST SS300
(drive model SDLL1DLR400GCCA1).

IOPs:
 -----------------------------------------------------------------------
  |perf_mode    | qd = 1 | qd = 64 |   note                             |
  |-------------|--------|---------|-------------------------------------
  |balanced     |  259K  |  3061k  | Provides max performance numbers   |
  |             |        |         | both on lower QD workload &        |
  |             |        |         | also on higher QD workload         |
  |-------------|--------|---------|-------------------------------------
  |iops         |  220K  |  3100k  | Provides max performance numbers   |
  |             |        |         | only on higher QD workload.        |
  |-------------|--------|---------|-------------------------------------
  |latency      |  246k  |  2226k  | Provides good performance numbers  |
  |             |        |         | only on lower QD worklaod.         |
  -----------------------------------------------------------------------

Avarage Latency:
  -----------------------------------------------------
  |perf_mode    |  qd = 1      |    qd = 64           |
  |-------------|--------------|----------------------|
  |balanced     |  92.05 usec  |    501.12 usec       |
  |-------------|--------------|----------------------|
  |iops         |  108.40 usec |    498.10 usec       |
  |-------------|--------------|----------------------|
  |latency      |  97.10 usec  |    689.26 usec       |
  -----------------------------------------------------
 

sweden

New Member
Dec 6, 2020
2
0
1
Any new information regarding this? I can't seem to find any news about these cards
 

i386

Well-Known Member
Mar 18, 2016
2,512
703
113
32
Germany
Broadcom 95xx cards are PCIe 4.0 and have been available for some time now.
The new interesting features are pcie 4.0, u.3 and for the raid controllers ddr4 2666MHz support

HBAs:
Broadcom 9500-8i: HBA 9500-8i Tri-Mode Storage Adapter
Broadcom 9500-8e: HBA 9500-8e Tri-Mode Storage Adapter
Broadcom 9500-16i: HBA 9500-16i Tri-Mode Storage Adapter
Broadcom 9500-16e: HBA 9500-16e Tri-Mode Storage Adapter

Raid controllers:
Broadcom MegaRAID 9560-8i: MegaRAID 9560-8i
Broadcom MegaRAID 9580-8i8e: MegaRAID 9580-8i8e
Broadcom MegaRAID 9560-16i: MegaRAID 9560-16i
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,390
494
83
Has anyone actually got their hands on one of the 9500 HBAs yet?

I saw that the 9500-8i was specced to use an anaemic 6W rather than the 10-15W I'm used to for my other 8i's and I wondered if anyone could verify. That combined with the single-port x8 SFF-8654 would make it a good fit for the sort of compact NAS box that I do a fair few of.
 

i386

Well-Known Member
Mar 18, 2016
2,512
703
113
32
Germany
Not yet..

(I want to get a 16 port pcie 4.0 raid controller and see what happens when I connect all ports to a 846 sas3 expander backplane :D
16x 12GBit/s 196 GBit/s ~ 19.6 GByte/s
x8 pcie 4.0 ~ 15.7 GByte/s)
 

Evan

Well-Known Member
Jan 6, 2016
3,252
560
113
Has anyone actually got their hands on one of the 9500 HBAs yet?

I saw that the 9500-8i was specced to use an anaemic 6W rather than the 10-15W I'm used to for my other 8i's and I wondered if anyone could verify. That combined with the single-port x8 SFF-8654 would make it a good fit for the sort of compact NAS box that I do a fair few of.
if 6w is true that’s a nice saving for sure.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,390
494
83
It would be indeed. I was getting close to being tempted to think about buying one (and a 8654 -> 2x8643 cable) to try out.

Reading through the spiel though I see they're making a big thing about signed firmware, and looking through the downloads there's also a dearth of tools like sas[2|3]ircu and sas[2|3]flash which makes me think this generation are going to be far, far less crossflash-friendly. Broadcom will be broadcom I suppose...
 
  • Like
Reactions: Sleyk

Sleyk

Well-Known Member
Mar 25, 2016
1,126
505
113
Stamford, CT
I got my hands on a new Dell HBA345. Its not a full SAS 4.0 card, but it uses the new gen connections. The issue is not the cards or the availability itself, but actually, the cables to connect these new cards.

As you will have read, these cards are coming with a new SFF connector: The SFF 8654. It essentially allows for double the number of drives connected to the port.

I actually did some crazy stuff, and I have it posted to my blog. I planned to make a new STH post and will link it once I get a chance.
 

Sleyk

Well-Known Member
Mar 25, 2016
1,126
505
113
Stamford, CT
It would be indeed. I was getting close to being tempted to think about buying one (and a 8654 -> 2x8643 cable) to try out.

Reading through the spiel though I see they're making a big thing about signed firmware, and looking through the downloads there's also a dearth of tools like sas[2|3]ircu and sas[2|3]flash which makes me think this generation are going to be far, far less crossflash-friendly. Broadcom will be broadcom I suppose...
You are absolutely right. Broadcom is seemingly going out of their way to stop this crossflashing business we so much like :.)

The cables are a shit show as well. No one has even attempted to make the cables we need. Well, I have, and will post about it soon.
 

EffrafaxOfWug

Radioactive Member
Feb 12, 2015
1,390
494
83
You are absolutely right. Broadcom is seemingly going out of their way to stop this crossflashing business we so much like :.)
I think the only surprise I have here is that broadcom didn't try and ruin all of this much sooner...!

The only cables I've seen here in rightpondia with SFF-8654 connectors (I was looking specifically for a SFF-8654 -> 2xSFF-8643) are those made by Broadcom themselves, I dare say other vendors will follow in time if this is still an emerging standard.

They don't seem to have any SFF-8654 -> 2xSFF-8077 cables in existence and there's two types of SFF-8654 -> 2xSFF-8643 cables it seems - one for NVME (p/n 05-60002-00) and one for SAS/SATA (p/n 05-60003-00) - so it looks like mixing and matching of SAS/SATA and NVME on the same controller might not be possible.

I was rather hoping for U.3 connectivity so we could start getting unified backplanes that could take NVME, SAS and SATA drives with a single set of HBAs and cables but it seems we're still a fair way from that.
 
  • Like
Reactions: Sleyk

Sleyk

Well-Known Member
Mar 25, 2016
1,126
505
113
Stamford, CT
I think the only surprise I have here is that broadcom didn't try and ruin all of this much sooner...!

The only cables I've seen here in rightpondia with SFF-8654 connectors (I was looking specifically for a SFF-8654 -> 2xSFF-8643) are those made by Broadcom themselves, I dare say other vendors will follow in time if this is still an emerging standard.

They don't seem to have any SFF-8654 -> 2xSFF-8077 cables in existence and there's two types of SFF-8654 -> 2xSFF-8643 cables it seems - one for NVME (p/n 05-60002-00) and one for SAS/SATA (p/n 05-60003-00) - so it looks like mixing and matching of SAS/SATA and NVME on the same controller might not be possible.

I was rather hoping for U.3 connectivity so we could start getting unified backplanes that could take NVME, SAS and SATA drives with a single set of HBAs and cables but it seems we're still a fair way from that.
Haha, you're definitely on the right track my friend. It seems like Broadcom is not having the previous-gen crossflashing business like before. If you notice, they have been patching the SAS2/3 flash utility to death for years, then realizing they couldn't really stop us, they are now trying to get everyone on Storcli and such utilities, and these utilities seemingly make it harder to crossflash these cards. Seems they will be doing things much differently than say, Dell for instance.

I am working on a way to figure out how to crossflash using the newer utilities, so that may be a route we can take in the future as well.

As far as mixing SATA and NVME on the same card, the newer cards will actually support this. Well, to be fair, it was supported with the current "tri-mode" cards as well. I know that they will allow for a mix and match though, as I read it somewhere in my deep-dive of these new controllers...

In any case, we have to wait to see. I know that Dell will follow the pattern of LSI/Broadcom of course, and to tell you the truth, they already have, so we may be also out of luck there as well, unless they give us another SAS flashing utility, like say a SAS4flash or something. I'm not gonna keep my hopes up though.

As far as cables go, I actually had the idea to make SFF 8654-8i to SATA and SAS cables. They're insane. :.)

Think double the amount of ports, lol. ;)

They are actually already done, I just was deciding how to post it to avoid a C&D letter from Broadcom themselves. Trying not to get sued! As I mentioned before, I also got my hands on a new Dell "interim-gen" card. It's super nice, but it requires a special version of the new standard SFF 8654 connector to work correctly. Well, frankly speaking, I need a special part that only Amphenol has currently licensed from TE connectivity exclusively.

Those bitches. Keeping it all for themselves. I'm trying to source the particular part now, but they are of course keeping it under wraps and refusing to sell it to me, or even send me samples. I have some other contacts I'm working with, and I hope they can come through for me.

I'm essentially done with the write-up on my personal tech blog, so will post it here soon. I was also considering loaning Pat the card so he could do an official STH review if he would like, but we'll see. Without proper cables (for that particular card) it might not be worth it though.
 
Last edited: