HBA/IT controller

[tazzmn]

Hello,

I am currently working on building a system. I have 6 18TB SAS drives. My current Adaptec 7 card only shows them at 16 TB which I believe is causing all my errors I am seeing on my ZFS. I have tried with and without the memory card that comes with the Adaptec 7 card. I have a very hard time believing 6 drives under 30 days old are all bad. I am looking for a SAS card that supports IT/HBA mode, 12 gbps and prefer something that runs cooler or can run cool due to additional addons. Any help would be greatful. Thanks.

-tazz

 

[BLinux]

I think the 18 vs 16 difference is the difference between TB and TiB. Use this calculator and enter 18 TB to see it is 16.37 TiB. TB to TiB Conversion Terabyte to Tebibyte Calculator

Maybe explain what you mean by "causing all my errors" ??? Also, what OS are you using with ZFS?

BTW, SAS-3 spinning HDDs will not be able to take advantage of 12Gbps, they will still only transfer at around 2Gbps (about 250MB/s).

 

[tazzmn]

I am using Ubuntu 22.04. I have SAS drives rated at 12 gbps. All drives end up with errors and fall out of the zfs.

 

[tazzmn]

But even a working 6 gbps SAS would be nice. Currently I have the Adaptec Series 7 6G SAS/PCIe3 ASR-71605 SFF8643.

 

[BLinux]

I am using Ubuntu 22.04. I have SAS drives rated at 12 gbps. All drives end up with errors and fall out of the zfs.

Ok, you need to start looking at your Linux logs to see what's going on under the ZFS software layer. I show how to do this for LSI controllers in this part of the video:

You'll need to replace the "mpt[23]sas" part with the Adaptec driver name. But the rest should still apply. Those messages should tell you what is going on with your hardware.

Also, check to see if you have incrementing read/write corrected and/or uncorrected errors on those SAS drives.

 

[tazzmn]

I am seeing lots of:

Sep 20 12:43:31 tazz-desktop kernel: [ 146.796161] aacraid: Host bus reset request. SCSI hang ?
Sep 20 12:43:31 tazz-desktop kernel: [ 146.796168] aacraid 0000:04:00.0: outstanding cmd: midlevel-0
Sep 20 12:43:31 tazz-desktop kernel: [ 146.796170] aacraid 0000:04:00.0: outstanding cmd: lowlevel-0
Sep 20 12:43:31 tazz-desktop kernel: [ 146.796170] aacraid 0000:04:00.0: outstanding cmd: error handler-1
Sep 20 12:43:31 tazz-desktop kernel: [ 146.796171] aacraid 0000:04:00.0: outstanding cmd: firmware-0
Sep 20 12:43:31 tazz-desktop kernel: [ 146.796171] aacraid 0000:04:00.0: outstanding cmd: kernel-0
Sep 20 12:43:31 tazz-desktop kernel: [ 146.828150] aacraid 0000:04:00.0: Controller reset type is 3
Sep 20 12:43:31 tazz-desktop kernel: [ 146.828154] aacraid 0000:04:00.0: Issuing IOP reset

 

[BLinux]

I am seeing lots of:

Sep 20 12:43:31 tazz-desktop kernel: [ 146.796161] aacraid: Host bus reset request. SCSI hang ?
Sep 20 12:43:31 tazz-desktop kernel: [ 146.796168] aacraid 0000:04:00.0: outstanding cmd: midlevel-0
Sep 20 12:43:31 tazz-desktop kernel: [ 146.796170] aacraid 0000:04:00.0: outstanding cmd: lowlevel-0
Sep 20 12:43:31 tazz-desktop kernel: [ 146.796170] aacraid 0000:04:00.0: outstanding cmd: error handler-1
Sep 20 12:43:31 tazz-desktop kernel: [ 146.796171] aacraid 0000:04:00.0: outstanding cmd: firmware-0
Sep 20 12:43:31 tazz-desktop kernel: [ 146.796171] aacraid 0000:04:00.0: outstanding cmd: kernel-0
Sep 20 12:43:31 tazz-desktop kernel: [ 146.828150] aacraid 0000:04:00.0: Controller reset type is 3
Sep 20 12:43:31 tazz-desktop kernel: [ 146.828154] aacraid 0000:04:00.0: Issuing IOP reset

That sounds like you have a bad controller that is freezing up and resetting itself frequently. Which would explain your problem... when the controller resets and goes away, all the I/O requests will fail until it is online again.

I don't know about Adaptec cards, but with LSI cards, this behavior can happen when:

1) card is overheating
2) card is physically defective, like bad chip or bad BGA solder joints

You can try to see if you can provide better cooling to the card, and if that doesn't help, then I think your Adaptec card is dying.

 

[tazzmn]

Yeah I am starting to lean towards the controller. I had read online prior to installing it that they get hot so I put a case fan in my empty pci slot area and angled it up at the card so i dont think its a overheating issue. I am now also seeing scsi status=2 like 8 times when I boot up. Any 16 drive SAS cards you guys recommend? I will be running zfs with it. I only have 8 drives ATM but I want a card that can handle 16 since I can upgrade to that many in my case. Thanks.

 

[mrpasc]

16 Port SAS HBA cards are still expensive as they are seldom. Try searching your preferred sources for Broadcom 9300-16i it-mode.
One example from the UK: Broadcom 9300-16i

 

[BLinux]

16 Port SAS HBA cards are still expensive as they are seldom. Try searching your preferred sources for Broadcom 9300-16i it-mode.
One example from the UK: Broadcom 9300-16i

Keep in mind the 9300-16i is dual SAS3008 chips + PLX switch. My guess is that it dissipates about 25-30W, maybe more, which is quite a bit of heat. Have to keep that cool. The best choice if using HDDs is probably the 9201-16i as it runs much cooler, but genuine ones are hard to find, and they are expensive.

If you have available PCIe slots, might consider 2x -8i cards instead.

 

[itronin]

I have been a fan of the tri-mode cards for a while now (like lsi 9400-16i). These are HH cards unlike the 9201-16i which is very much a full height card. The 9400-16i is also SAS3 but you won't see a perf difference with spinners, just enterprise SAS SSD's.

Honestly I'm just using them as -16i SAS HBA's on my CSE-836. for a while they were ~250USD on the bay. still show up periodically at that price. I bought a few from a local US seller but they ran out of inventory. Last couple I purchased from China and have worked fine.

 

[UhClem]

I have been a fan of the tri-mode cards for a while now (like lsi 9400-16i).
...
Honestly I'm just using them as -16i SAS HBA's on my CSE-836.

Saw a great price on the "special" NVMe cables for that HBA. [DiLinKer LSI94xx Series Dedicated 2 Port HD Mini-SAS(SFF-8643 8X) to 2 Port U.2 (SFF-8639) NvMe SSD Cable] $14
(no personal experience)
-----
"Better to have and not need, than to need and not have." --Kafka

 

[Fritz]

I own zero Adaptec cards. Learned a long time ago that they're nothing but trouble.

Got in on the recent deal on a LSI 9300 16i and it works like a charm.

 

[Markess]

Hello,

I am currently working on building a system. I have 6 18TB SAS drives. My current Adaptec 7 card only shows them at 16 TB which I believe is causing all my errors I am seeing on my ZFS. I have tried with and without the memory card that comes with the Adaptec 7 card. I have a very hard time believing 6 drives under 30 days old are all bad. I am looking for a SAS card that supports IT/HBA mode, 12 gbps and prefer something that runs cooler or can run cool due to additional addons. Any help would be greatful. Thanks.

-tazz

What mode is your Adaptec card in for these drives? 7 Series can operate in passthrough or RAID mode on the same BIOS. For ZFS, each drive needs to be set to passthrough, not RAID, individually for every drive in the pool. Many (All?) 7 series cards can be operated in passthrough for some drives and RAID for others at the same time, so you should pull up the card's BIOS at boot and check each drive individually. Its been a while, but there may also be a setting to put the entire controller in passthrough, you'd have to check you card's BIOS.

You don't want to use memory for any drives operatiing in passthrough mode. If all your drives will be used for ZFS, its best to leave the module off. If you've got Cache RAM active on the card it will definitely not work right for ZFS and you'll get tons of errors.

There are people that swear their 7 series works flawlessly in ZFS for both BSD and Linux based systems, while others say they are nothing but trouble. I never had an issue with mine (71605), but retired it after a year in favor of a new motherboard with an onboard LSI controller.

 

[nihonjin]

@tazzmn just a guess... but which is your setup? do it has a good cooling? because what happens to you could be because the controller goes too hot and you didn't put a fan over it.

 

[i386]

I own zero Adaptec cards. Learned a long time ago that they're nothing but trouble.

while others say they are nothing but trouble

More (great imo) adaptec controllers for me

Spoiler