Broadcom 9500-8i, NVME U.2/U.3, Tri-Mode
- Thread starter john389
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I had a question for any of you owners of a 9400 or 9500 HBA that is unrelated to the main topic. There are multiple reports out there of people using older HBA chipsets such as 2208/2308/3008 where the HBA prevents the processor from going into deeper idle states than C2 or C3. I have not seen anything from anyone with a more modern chipset such as the 3408 or 3808 chipset. Broadcom advertises more power saving features starting with the 3408 such as " – Slumber and partial power mode support for SAS/SATA devices ", " Programmable SAS link power down", and " – Variable PCIe bandwidth negotiation". I was curious if any of you have investigated what idle states your system can achieve with powertop or similar program?
@Stankyjawnz: I can perhaps - can't promise it though - test that for you in the coming weeks, no idea when yet. However, not being able to achieve higher C-States, i.e. > C3, typically happens whenever you have high performance I/O chips in your system, be they for data storage or networking. You just introduce latency into your system, when the CPU continually enters deeper sleep states, which take more time to wake from again. The question is then how much additional energy saving you could in theory achieve by enabling deeper C states relative to problems introduced in your system. Most HBAs and networking equipment, such as SFP+ and SFP28 cards, where I know the problem from as well, are all built for enterprise environments, where latency problems are to be avoided. So you probably won't have any luck there trying to force something that is not designed for it.
However, as you can see from the datasheets - I haven't measured it yet - the newer 9400 and especially 9500 series HBAs use a lot less power than previous generations, so if you can upgrade your card, you'll perhaps save more power than what you would get from deeper C states - which you then also get if it is so designed.
9500-8i, 6W typical
9400-8i, 10W typical
9300-8i, 13W typical
Of course the max value is quite a bit higher. So newer chips are more efficient and I've stopped buying the older chipsets a few years ago.
EDIT:
The newer 9500 series chips, i.e. SAS3808 and SAS3816, are also found in, for example, Lenovo and Dell HBA cards.
I think it is the Dell HBA350i, Dell HBA355i, Lenovo 440-8i and Lenovo 440-16i. Those you might get a lot cheaper than the original Broadcom versions. Before buying read up on the SlimSAS low profile cables you'll need for the Dell versions.
However, as you can see from the datasheets - I haven't measured it yet - the newer 9400 and especially 9500 series HBAs use a lot less power than previous generations, so if you can upgrade your card, you'll perhaps save more power than what you would get from deeper C states - which you then also get if it is so designed.
9500-8i, 6W typical
9400-8i, 10W typical
9300-8i, 13W typical
Of course the max value is quite a bit higher. So newer chips are more efficient and I've stopped buying the older chipsets a few years ago.
EDIT:
The newer 9500 series chips, i.e. SAS3808 and SAS3816, are also found in, for example, Lenovo and Dell HBA cards.
I think it is the Dell HBA350i, Dell HBA355i, Lenovo 440-8i and Lenovo 440-16i. Those you might get a lot cheaper than the original Broadcom versions. Before buying read up on the SlimSAS low profile cables you'll need for the Dell versions.
Last edited:
@john389 Thank you for the offer to test. I scoured the internet and could not find anyone who had examined the 9500-8i c-state. I actually have a very low performance use case of 12 3.5" hdds. When trying to research this topic I saw numerous reports that SAS2008/2308/3008 could add 30w to idle power. At my power rates that would be ~40 dollars per year. It was conceivable to me that I could breakeven with a 9500 card over several years if it idled considerably better. Without data I thought the 9500 card to be too risky to spend the extra money on the chance it idled better, so I ended up purchasing a SAS2008 chipset card.
To my surprise my system is still idling fairly well, C7 for ~88% of the time. Idle power went from 35 to 44w when I added the hba which appears to be from the card itself and then a few extra watts for random cycles that wake up the cpu. This is a new board to replace an old E5-2660v2 system and I have not swapped it out and connected it to the backplane yet though so hopefully adding the drives does not change the idle situation too much.
Power consumption seems to be a complicated thing to get information on online, there is so much variation from board to board and different platforms. This is with LGA 1700, 12th gen intel on debian 12.
If you still feel like sharing your results I would think folks would be interested, like I said I had never seen the 9500 reported and many of the power obsessed NAS builders shun HBAs entirely, going to sata add-in cards instead. I think as more 9400 and 9500 cards hit the second hand market there will be more interest from nas builders from power consumption. To me the SAS2008 and 3808 chips both appear viable low power nas cards.
To my surprise my system is still idling fairly well, C7 for ~88% of the time. Idle power went from 35 to 44w when I added the hba which appears to be from the card itself and then a few extra watts for random cycles that wake up the cpu. This is a new board to replace an old E5-2660v2 system and I have not swapped it out and connected it to the backplane yet though so hopefully adding the drives does not change the idle situation too much.
Power consumption seems to be a complicated thing to get information on online, there is so much variation from board to board and different platforms. This is with LGA 1700, 12th gen intel on debian 12.
If you still feel like sharing your results I would think folks would be interested, like I said I had never seen the 9500 reported and many of the power obsessed NAS builders shun HBAs entirely, going to sata add-in cards instead. I think as more 9400 and 9500 cards hit the second hand market there will be more interest from nas builders from power consumption. To me the SAS2008 and 3808 chips both appear viable low power nas cards.
That's because those things are non-related
Bullshit, I have each of mentioned generations plus 3408 and 3816 and none of them impacts the CPU ability to demote C states
Last edited:
They are.
Numerous drivers & cards (not necessarily LSI) are known to prevent deep C-States, e.g. due to the lack of ASPM
Ok, I double checked and I have ASPM disabled system-wide in BIOS, so this have nothing to do with CPU ability to demote cores into C6. What's next assumption?
Save your passive aggressive replies for someone else.
Yes, I can confirm that I also have a ConnectX-3 card installed in this system and the card itself supports ASPM L0s. You can see that SAS3408 (LSI 9400) also supports both ASPM L0s and L1. Even the Fusion ioMemory SX350 that doesn't support ASPM at all cannot change the behavior of the CPU's power saving - it reaches both C6 and PC6 states nice and easy.
The original question was about the hardware LSI chipsets limitation, not some rare OS handicaps. Here is the answer: LSI SAS chipsets (SAS2008 and newer) do support ASPM and has nothing to do with CPU ability to switch between C-states
Hey guys, I was doing research on this card and was wondering about the performance of NVMe drives on this card -- How does the read/write speed looks like comparing to direct PCIe connection?
I am asking because I also saw another thread saying the performance is poor (700MB/s read) from this thread https://forums.servethehome.com/ind...ance-on-broadcom-9440-8i-lenovo-530-8i.23534/
I am asking because I also saw another thread saying the performance is poor (700MB/s read) from this thread https://forums.servethehome.com/ind...ance-on-broadcom-9440-8i-lenovo-530-8i.23534/
Yeah, apparently they present NVMe devices as SCSI devices, negating half the advantage of NVMe. Sort of like AHCI PCIe SSDs from the early days of PCIe SSDs a decade ago. Tri-mode and U.3 is nothing short of a farce by Broadcom and Microchip to sell absurdly expensive crap to be able to save a little bit of cash on backplanes versus U.2.
Similar story, I have a Quantum R6000 (actually three of them). Pretty nice specs, but I was also worried because even in JBOD mode, linux just saw /dev/sdX devices, and I expected /dev/nvmeX. Thanks for confirming that Tri-Mode is just a farce 
And I only got it to about 4.0 GB/s write speed with a raid0 mdadm on the 4 NVMe drives
if curios, these are the specs:
And I only got it to about 4.0 GB/s write speed with a raid0 mdadm on the 4 NVMe drives if curios, these are the specs:
- Mainboard: Asrock X570D4-2T
- CPU: AMD 3950X
- RAM: 128GB DDR4
- NIC: 2x10GBaseT + 2x100G QSFP28 + 1 IPMI
- Storage: BROADCOM 9560-16i + 4x3.84TB Kioxia U.2 NVMe + 512GB boot M.2 NVMe
Last edited:
Same deal with FreeBSD. They show up as direct attach devices not nvme. I have only dealt with the SAS9400 and tri-mode via SFF8643.Thanks for confirming that Tri-Mode is just a farce
I will note that I saw the same speeds with devices on controller or PCIe direct. No raids. Single drive speed.
Ran into similar issues with the Tri-Mode card and ended up using a pure NVMe switch adapter.
So far, everything works fine.
So far, everything works fine.
So i have a 30TB nvme which is split into 5 namespaces. Using a Trimode hba isnt the way to go to see all namespaces?
i have a broadcom 9400 and when i connected it, it only saw one namespace, compared to a intel S2600WG mothereboard which saw all namespaces.
Thanks.
i have a broadcom 9400 and when i connected it, it only saw one namespace, compared to a intel S2600WG mothereboard which saw all namespaces.
Thanks.
None of the trimode hba support namespaces in the sense that they will only ever see max. 1 namespace, if any.
The reason, as stated before, is that the nvme drives appear as scsi devices, which have no support for namespaces. So the issue is probably limitations in one or more of: hardware, firmware and driver.
The reason, as stated before, is that the nvme drives appear as scsi devices, which have no support for namespaces. So the issue is probably limitations in one or more of: hardware, firmware and driver.
Hello together,
his post might not be the newest, but it appears to be the best source for information about the cable that i could find.
Im just building my first nas and ordered a 9500-8i
Now since its just a normal case i need to directly attach a cable without backplane.
After searching countless of hours for 8654 to 8482 cables i realized that the u.3 should be compatible (complete newbie in this area)
Now this cable seems to be the one i need, but looking at the Broadcom spec sheet i wonder what the two additional smaller cables are for, could anyone help me out?
I appreciate the help.
his post might not be the newest, but it appears to be the best source for information about the cable that i could find.
Im just building my first nas and ordered a 9500-8i
Now since its just a normal case i need to directly attach a cable without backplane.
After searching countless of hours for 8654 to 8482 cables i realized that the u.3 should be compatible (complete newbie in this area)
Now this cable seems to be the one i need, but looking at the Broadcom spec sheet i wonder what the two additional smaller cables are for, could anyone help me out?
I appreciate the help.