SATA drives behind expander still a no-go?
- Thread starter Rand__
- Start date
Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.
I have a number of ZFS-based systems that use SATA drives behind expanders and they work fine. I would not recommend any SAS gen1 (3Gb/s) gear, but the later LSI SAS2 and SAS3 stuff is fine (LSI 2008 controllers and later). I would also stick to reasonably reliable newer (~2013 and later) drives, i.e. not first-gen 1TB WD green drives.
I think the primary reasoning against this in the past is that a balky SATA drive that is failing and experiencing command timeouts (due to either controller or media failures) can cause the expander to drop multiple disks out of the array, not just the failing drive.
Linux raid and ZFS handle this pretty gracefully. The Linux mpt2sas driver will generally reset the entire expander and then try to start talking to the drives again. This causes a brief hiccup. If the failing drive continues to timeout you'll see this repeatedly. Occasionally, if you are unlucky, some other good drives may not come back from a reset.
Again, Linux raid and ZFS generally handle this without data loss. If these resets go on too long, you may lose drives out of the array and enter a degraded state and need to reboot the system to get them back. But it's unlikely you'll lose the filesystem, assuming you have protection (mirroring, raidz, raidz2, etc.)
This sounds awful but, unless you are trying to run an extreme commercial uptime environment, it's not a big deal. If all of your drives are in good health, you won't ever see a problem. If a drive starts failing, swap it as soon as you notice it causing errors.
I don't know the state of this on other ZFS-supporting OSes. They may be more or less forgiving on the LSI SAS driver-level reset behavior. If so, I would expect you would get to the "degraded array -- time to reboot" state quicker but you still probably wouldn't lose any data. At that point, swap the drive that is causing the problems.
I think the primary reasoning against this in the past is that a balky SATA drive that is failing and experiencing command timeouts (due to either controller or media failures) can cause the expander to drop multiple disks out of the array, not just the failing drive.
Linux raid and ZFS handle this pretty gracefully. The Linux mpt2sas driver will generally reset the entire expander and then try to start talking to the drives again. This causes a brief hiccup. If the failing drive continues to timeout you'll see this repeatedly. Occasionally, if you are unlucky, some other good drives may not come back from a reset.
Again, Linux raid and ZFS generally handle this without data loss. If these resets go on too long, you may lose drives out of the array and enter a degraded state and need to reboot the system to get them back. But it's unlikely you'll lose the filesystem, assuming you have protection (mirroring, raidz, raidz2, etc.)
This sounds awful but, unless you are trying to run an extreme commercial uptime environment, it's not a big deal. If all of your drives are in good health, you won't ever see a problem. If a drive starts failing, swap it as soon as you notice it causing errors.
I don't know the state of this on other ZFS-supporting OSes. They may be more or less forgiving on the LSI SAS driver-level reset behavior. If so, I would expect you would get to the "degraded array -- time to reboot" state quicker but you still probably wouldn't lose any data. At that point, swap the drive that is causing the problems.
I have several supermicro SC847E26 and E16 (45 slot, SAS2 expander) and SC417E16 (88 slot) running.
In two of these (E26) a mix of dual-attached SAS (HUS724040ALS640) and consumer-grade SATA drives (HDS72302,MD04ACA4) are getting along gracefully.
One SC417 is populated with Crucial_CT1024M5 and TOSHIBA_Q300, and only the Q300's are having slight hiccups. This is due to the internal structure of this particular TLC device, every few minutes it takes SECONDS to process the flushes and trims the zfs layer sends them.
The Host (FreeBSD, LSI SAS2308) doesn't notice aside from not being able to talk to them for 2-3 seconds. But I won't add any TLC SSDs to that setup and stick to MLC.
In two of these (E26) a mix of dual-attached SAS (HUS724040ALS640) and consumer-grade SATA drives (HDS72302,MD04ACA4) are getting along gracefully.
One SC417 is populated with Crucial_CT1024M5 and TOSHIBA_Q300, and only the Q300's are having slight hiccups. This is due to the internal structure of this particular TLC device, every few minutes it takes SECONDS to process the flushes and trims the zfs layer sends them.
The Host (FreeBSD, LSI SAS2308) doesn't notice aside from not being able to talk to them for 2-3 seconds. But I won't add any TLC SSDs to that setup and stick to MLC.
Hello I walked in from another thread curious on a similar topic. Why do you not recommend SAS gen 1, is it ONLY for quirky expander behavior or are there other reasons? I was going to use a system on SnapRAID without ZFS at all. Because SnapRAID requires manual commands to create snapshots, raid parity, and restore I was hoping even occasional quirks like that might be no big deal. (provided they didn't happen in the middle of creating data - though i'm assuming since it's file based instead of block based, it would just sit waiting for a response/assuming the reset behavior doesn't take multiple seconds to finish)
Do you know if it's possible to run for instance one SAS and three SATA drives off the same "4 lane" cable coming out of an SAS card which I believe is an 8088 cable? (i've seen cables which go to four SATA connectors, and those which just lead to I think they are 8087 cables, but not mixtures)
Wow okay, I didn't know about any of that. (incl the 2tb limit) SAS2 it is, thank you.
The 3gig limit isn't so bad when all I want to do at first is saturate gigabit, but 2tb is not enough.
There isn't a space limit on SAS2 is there?
The 3gig limit isn't so bad when all I want to do at first is saturate gigabit, but 2tb is not enough.
There isn't a space limit on SAS2 is there?
I saw a post of yours where you said that you are using SATA drives in a CSE-847E26... Are you having any problems using SATA drives vs using the the SAS only recommendation?
I recently purchased a second hand CSE-847E26-RJBOD1... without knowing about dual expander backplanes... According to Supermicro... only SAS drives should be used with this setup. Also, without knowing about the dual expander, I purchased 20 HGST 3TB HUA723030ALA640 HDDs.
Luckily, these HDDS are in the Supermicro list of HDD tested with SAS2/SAS3 Expanders, but only for 846EL2, not the 847EL2.
I am planning on using a Dell UCS-70 = 6Gbps SAS HBA to connect to the CSE847E26... It is uses the LSI2008 Chipset.
So do you think I will be ok?
I recently purchased a second hand CSE-847E26-RJBOD1... without knowing about dual expander backplanes... According to Supermicro... only SAS drives should be used with this setup. Also, without knowing about the dual expander, I purchased 20 HGST 3TB HUA723030ALA640 HDDs.
Luckily, these HDDS are in the Supermicro list of HDD tested with SAS2/SAS3 Expanders, but only for 846EL2, not the 847EL2.
I am planning on using a Dell UCS-70 = 6Gbps SAS HBA to connect to the CSE847E26... It is uses the LSI2008 Chipset.
So do you think I will be ok?
I have a SAS2846EL2 and a SAS2847EL2.
I am still waiting for Supermicro to get back with me about the proper internal cabling between the backplanes... The manual says To configure this, you need to connect the J0 connector of both BPN-SAS2-847EL1 and BPN-SAS2-
846EL1 backplanes to cables in section E4. The other two cables are connected
to J1 of BPN-SAS2-847EL1 and J2 of BPN-SAS2-847EL1 to make the chassis
cascade ready for future... but this is for the SC847J E16 Series.. it doesn't mention the E26 series.
If I use SATA drives will I have access to all 45 slots (both backplanes) or will I be limited to just one backplane?
The main reason I am inquiring about this is because I am using the E26 (dual expander) vs the E16 (single expander). Apparently SATA drives are allowed (by Supermicro) on the E16 but not the E26.
... from the manual:
There is also dual version of the same expander which has two expander chips so
there are two paths to each drive behind the expander. It is mandatory to use SAS
drives when using the dual expander backplane. This is because the SAS drives
have dual connectivity, therefore enabling the redundancy feature. No SATA drives should be used in dual expander backplanes.
Thanks.
I am still waiting for Supermicro to get back with me about the proper internal cabling between the backplanes... The manual says To configure this, you need to connect the J0 connector of both BPN-SAS2-847EL1 and BPN-SAS2-
846EL1 backplanes to cables in section E4. The other two cables are connected
to J1 of BPN-SAS2-847EL1 and J2 of BPN-SAS2-847EL1 to make the chassis
cascade ready for future... but this is for the SC847J E16 Series.. it doesn't mention the E26 series.
If I use SATA drives will I have access to all 45 slots (both backplanes) or will I be limited to just one backplane?
The main reason I am inquiring about this is because I am using the E26 (dual expander) vs the E16 (single expander). Apparently SATA drives are allowed (by Supermicro) on the E16 but not the E26.
... from the manual:
There is also dual version of the same expander which has two expander chips so
there are two paths to each drive behind the expander. It is mandatory to use SAS
drives when using the dual expander backplane. This is because the SAS drives
have dual connectivity, therefore enabling the redundancy feature. No SATA drives should be used in dual expander backplanes.
Thanks.
Last edited:
SAS disks have 2 channels, one on each expander, thereby providing two data paths. SATA disks by comparison are only single channel and will therefore only be accessible from a single expander. The dual expander option is to allow for HA setups. SATA disks get away with being used on single expanders because of the basic compatibilty between the interfaces, but really they weren't meant to be used on SAS backplanes
I got a response from Supermicro...
The simple way is to hook up internal cables as following diagram (1 cable connected to J0 on rear backplane as input and another cable connected from J1 of rear backplane to J1 or J0 of front backplane. That way all hard drives (On front and rear backplane) will be detected with 1 external cable connected to HBA on the server.
I also inquired with supermicro about the E26 backplane...
If you already had CSE-847E26-RJBOD1 (E26) versions. No need to purchase the E16 version.
E26 version is E16 plus the second port expanders to support failover and multipath. You don’t need to do anything to it if you don’t use failover and multipath for now.
The simple way is to hook up internal cables as following diagram (1 cable connected to J0 on rear backplane as input and another cable connected from J1 of rear backplane to J1 or J0 of front backplane. That way all hard drives (On front and rear backplane) will be detected with 1 external cable connected to HBA on the server.
I also inquired with supermicro about the E26 backplane...
If you already had CSE-847E26-RJBOD1 (E26) versions. No need to purchase the E16 version.
E26 version is E16 plus the second port expanders to support failover and multipath. You don’t need to do anything to it if you don’t use failover and multipath for now.