Experiences with NetApp DS4243

[NablaSquaredG]

A friend of mine is considering to buy NetApp DS4243s - I'm not convinced yet...

How loud are these?

And the most important questions:
Are they vendor locked to NetApp branded HDDs?

I guess there are lots of things we / he needs to be aware of...
As far as I've understood, these things are 3Gb/s with the original NetApp controllers? But the controllers can be replaced with Dell ones that add 6G/s support.

 

[andrewbedia]

I have one DS4243. The first thing I did was put in a pair of Dell Compellent 6Gbps controllers (as you've referenced) that use normal SFF-8088. Sometimes these can be found cheap. Example here Genuine Dell Compellent HB-SBB2-E601-COMP 0952913-07 6GB SAS EBOD Controller OEM 787421252224 | eBay

Sometimes the DS4246 controller is cheaper. If you have the 8088-8088 cables already, it may be worth getting the Dell Compellent controllers. If not and you need to buy cables anyway, DS4246 controller may be a cheaper way to go. If you're running spinning hard drives, also consider you may be happy with 3Gbps performance.

In terms of quietness... I don't have a sound meter, but it is very noticeably quieter than HP StorageWorks MSA60 / D2600. I wouldn't put it in my livingroom, but it's reasonably quiet.

I have only used SAS drives with mine. I threw the SATA interposers out. I have used a mix of netapp and non-netapp 4TB and 8TB SAS disks in mine and never had trouble. I have no idea how SATA drives work in DS4243 or if there are weird caveats.

Good luck

 

[gregsachs]

I've run sata drives in a ds2446, the 24x2.5 version. They only showed for one controller, not both, since single path. Proper sata->sas interposers would fix that. Never had issue. All the SBB documentation I've been able to find says that the expander and smarts are all in the controller/IOM module, not in the unit itself. That said, I know "people" have said they had issues putting the Dell IOM units in.

 

[ItsaMeDS4243]

With both power supplies and IOMs installed, it makes about the same amount of noise as one of those plug in air filter things. It's fan noise, but mine doesn't make any fan whine, it's mostly wooshing. I'd consider it very tame given that it idles at ~250w when populated with drives, but by modern desktop computer standards it's far from silent.

Now if you detatch one of the power supplies or IOMs (physically remove, not just unplug), it will make a scary amount of noise even for a rack mount device. I think I actually measured something like 70w extra power draw just from the fans when I tried pulling one of the IOMs to save a bit of power. Not recommended.

I can also report that the HB-SSB2-E602 is also not drive locked, I picked up a 601 and 602 and currently just have the 602 being used. Both can be had for under $30 on the bay, and the 601 has much wider availability.

 

[gtcdma]

I have 2 of these shelves. The first one is older and came with IOM3 controllers, the other one is newer and came with IOM6s. I think it might be actually a DS4246, but it doesn't have "DS4246" written anywhere on it.

I also picked up a Dell HB-SBB2-E601-COMP controller. It worked fine installed into the older shelf, but not in the newer one: the yellow light was on and the fans spun like crazy.
I tried to get it to work by installing it in different slots, together with an IOM3, IOM6 or both, etc, but nothing helped until I swapped the power supplies from the older shelf. Apparently the Dell unit can't properly communicate with the newer PSUs.

The older PSU part number is 82562-12 / HB-PCM01-580-AC and the newer one is 114-00087 / Delta TDPS-580BB A
Both of them work fine with IOM3 or IOM6, it's only Dell unit that has issues with the newer one.
Newer PSUs are also more efficient (35W vs 50-60W idle with no disks, 2 PSUs, 1 IOM).

 

[gtcdma]

Now if you detatch one of the power supplies or IOMs (physically remove, not just unplug), it will make a scary amount of noise even for a rack mount device. I think I actually measured something like 70w extra power draw just from the fans when I tried pulling one of the IOMs to save a bit of power. Not recommended.

I can't confirm this. Yes, they need 2 PSUs minimum, but work just fine with one IOM, no noise increase.

 

[gtcdma]

I have only used SAS drives with mine. I threw the SATA interposers out. I have used a mix of netapp and non-netapp 4TB and 8TB SAS disks in mine and never had trouble. I have no idea how SATA drives work in DS4243 or if there are weird caveats.

I have a wild mix of different SAS and SATA drives, 4TB-12TB, 512e and 4kn. No caveats so far. I'm not using interposers.

Also, it looks like there is absolutely no difference between IOM6 and Dell 601-COMP with regard to performance and drive compatibility. I'm pretty sure both of them are made by the same OEM.

 

[ari2asem]

are these units also easy in use with Windows 10 enterprise and any LSI 6gbps HBA ?

or is it difficult with some proper drivers ??

 

[gtcdma]

are these units also easy in use with Windows 10 enterprise and any LSI 6gbps HBA ?

or is it difficult with some proper drivers ??

I didn't try it on Windows, but I suppose it should just work as well without any special drivers.

 

[gregsachs]

I had a netapp ds2246 show under hyper-v 2016 with no issues, connected to a 9286-cv-8e

 

[Preston Youngsley]

I have 2 of these shelves. The first one is older and came with IOM3 controllers, the other one is newer and came with IOM6s. I think it might be actually a DS4246, but it doesn't have "DS4246" written anywhere on it.

I also picked up a Dell HB-SBB2-E601-COMP controller. It worked fine installed into the older shelf, but not in the newer one: the yellow light was on and the fans spun like crazy.
I tried to get it to work by installing it in different slots, together with an IOM3, IOM6 or both, etc, but nothing helped until I swapped the power supplies from the older shelf. Apparently the Dell unit can't properly communicate with the newer PSUs.

The older PSU part number is 82562-12 / HB-PCM01-580-AC and the newer one is 114-00087 / Delta TDPS-580BB A
Both of them work fine with IOM3 or IOM6, it's only Dell unit that has issues with the newer one.
Newer PSUs are also more efficient (35W vs 50-60W idle with no disks, 2 PSUs, 1 IOM).

reviving this thread from the dead, as it has the most info I have found in regards to my request. I have 2 DS424x's in service in my home rack, and I've recently been trying to get some power consumption numbers down. I'm running them with 2x PSUs (X518A-R6), plugged in and powered, 1 IOM 6 Module, and blanks to stop the fans from ramping up. The first shelf is full of SATA disks that range from 2TB - 18TB. I'm just stating the fill up the 2nd shelf. I know Xyratex was the OEM for alot of different shelves and I've noticed that many have the same form factor of PSUs. From some precursory reading and searching it seems that Netapp provided two part numbers for PSUs for the DS424x series of disk shelves.
X518A-R6 |114-00070 / HB-PCM01-580-AC - PSU W/FANS,580W,AC,DS424x,R6

X522A-R6 |114-00026 / Delta TDPS-580BB A - PSU W/FANS,Estar,580W,AC,DS424x


My original plan was to grab a X522A and test it out with a Kill-a-watt to see if the Estar rated PSU is any more efficient with my setup. Not sure how much difference this would potentially make, as the comparison of power draw was @ idle with no disks. Further searching down the rabbit hole led me to more PSUs with the same form factor, but I am unsure if their pinouts would remain the same, but I would assume so much. Two of the additional units stood out specifically to me -
SP-PCM01-HE580-AC / Delta TDPS-580AB - PSU W/FANS, 580W, 80+Silver - These seem to come out of Dells or HPs and often have a -AC-DELL / -AC-HP PN listed.

SP-PCM02-HE580-AC / Flextronics XRT-S-0580ADU00-103 / 0996223-02 - PSU W/FANS, 580W 80+GOLD - These seemed to be mainly from HPs and often have the -AC-HP PN listed.

These two units from Dells and HPs seem to be the same form factor, but the PCM01 is 80+ Silver, and the PCM02 is 80+Gold Rated. I know that Energy star != Gold / Silver Efficiencies, but I was wondering if anyone knows A) if the Dell / HP PSUs will work in a netapp, (are pinouts the same. Does anyone have a datasheet for any of these PSUs?) B) if the 80+Gold or 80+Silver is more efficient than the Energystar Netapp PSU, C) if this exercise is insane and i'm probably not going to save any power and just spend money on stupid old psus.

 

[richard.dzavoronok]

I'm too trying to get down consumption of ds4243, I'll check what PSU's I'm having tomorrow and report back consumption.
Also I read on reddit that HB-SBB2-E601-COMP (while having sff8088 ports, for which cables are a lot cheaper than qsfp-sff) has some additional fan/voltages control as it is communicating with PSU and can be quieter and consume lower energy. I have no idea if it's PSU version dependent though.

 

[Preston Youngsley]

Got a single TDPS-580BB / X522A in for testing and it looks promising. I've only done some surface level quick testing. I would say it's not the most scientific or exact measure approach, but a good enough measure to prove the point. The unit slots in with no issues as expected, as it is a netapp P/N for this PDU.

Testing methodology- Single PDU, slotted in, connected to the wall. 3x PDUs slotted in, not connected to the wall. 1x IOM6 Controller. Controller blanks to keep the fans from ramping. Tested Idle W and VA with no drives, as well as connected with 2x 18TB SATA drives. Wattage and VA were measured with a kill-a-watt type device and after the array had been powered on for a few minutes and the start up procedure and fan check have been completed. I am in the US running AC 120V.

X518A-R6 - No Drives + Idle - 60.02W / 52.15VA
X522A / TDPS-580BB - No Drives + Idle - 37.32W / 39.16VA
rough power savings @ idle with an empty shelf ~ 23W / ~13VA

X518A-R6 - 2x 18TB Drives / Connected to Unraid but idle - 73.47W / 75.02VA
X522A / TDPS-580BB - 2x 18TB Drives / Connected to Unraid but idle - 52.95W / 54.07VA
rough power savings @ idle with 2x 18TB SATA disks ~20.5W / 21VA

I'd say the TDPS-580BB / x522A are worth the upgrade to save some power. Most I found were in the $45US range, but some ebayers accepted offers of prices as low as $18US.

I was able to also test a SP-PCM02-HE580-AC. The unit physically slots in with no issues and powers the shelf on. That's about where the test stopped. Once the IOM is initialized the fan on the PCM cranks up to maximum RPM. Any PCM that is plugged in will also ramp up its fans. The fans in the SP-PCM02 are also different and make more of a higher pitched whine noise as opposed to the whoosh of the Netapp Units. I did not measure any wattage pull, as the fans were unbearably loud. Imagine an electric leaf blower plugged and blowing in behind the rack. My thoughts are that the PCM does not talk to the IOM properly and thus just defaults to full speed. I would think maybe this could be remediated with a different controller, either the HP3 Par controller or a Dell Comp controller, neither of which i have on hand right now. Extrapolating results from this test, I'd also venture to guess that the SP-PCM01 80+ Silver rated would most likely do the exact same thing.


TL;DR - TDPS-580BB / X522A can save some wattage and is a worthy upgrade if you can find them for a low enough price. The HP / Dell PCMs slot in physically and will power the shelf without issue, however the PCM does not communicate with the Controller properly and the fan speed ramps up to maximum RPMs. The noise is unbearable for regular use.

 

[richard.dzavoronok]

Thanks for your effort. I didn't checked my PSUs as two Delta ones are in shelf that I need to be powered up.
Maybe SP-PCM02-HE580-AC are for xyratex/dell controllers?
I suggest to read this thread on reddit:

Spoiler: Thread

https://www.reddit.com/r/DataHoarder/comments/ehpfur

OP in that thread did test a few things that are worth trying I guess? Specifically another savings from smart control of fans with SBB2 controller. I found some of them for 14 bucks on US ebay. Here in EU they are going for 90 euros which sucks.

Edit: media insert from reddit thread hidden behind Spoiler

 

[Preston Youngsley]

From my minimal investigations, it seems that the SP-PCM02-HE580-AC is from an HP 3PAR M6720, even then, most of those came with the PCM01 version that is 80+ Silver.

Being a glutton for punishment and exhausting my potential savings on science, I grabbed a set of Dell Compellent SAS Controllers to test. I ended up with some very interesting findings. More reading has shown that the compellent controller requires 2x psus powered before it will spin up any disks, not sure the truth to this, but it may explain some of my experiences.

Backstory:
I have 2 DS424x Shelves.
The original one I picked up years ago, we'll call this OG, has Xyratex 2008 on the PCB. OG came with 2x IOM3's, 4x X518a-R6 Power One branded PSUs. A later IOM6 (single IOM6 Controller, 3x Blanks) upgrade slotted in fine with no issues, worked on first boot;
The newer shelf I got recently, we'll call this FNG, has Xyratex 2010 on the PCB. FNG also came with 2x IOM3's, 4x X518a-R6 Power One branded PSUs.a later IOM6 (single IOM6 Controller, 2x Blanks, one IOM3 flipped into the blank position with no physical connection just to serve as a blank) upgrade slotted in fine with no issues, worked on first boot
Both shelves show power savings when moving to a TDPS-580BB / X522A psu. Currently running 2x TDPS-580BBs plugged in to the wall and switched on in both OG and FNG, with 2x X518a-r6 psus slotted in, but not plugged in to the wall (essentially blanks). Plugged in PSUs are in Top left and bottom right positions when facing the rear of the chassis.
When I swap the IOM6 for the Compellent in OG, with no other changes to PSUs or anything, the shelf immediately ramps up the two plugged in PSUs fans, and stays there, however the shelf ID registers and the device seems to be functional (I've only done idle testing so far, no usage testing yet). When I swap the IOM6 for the compellent in FNG the fans ramp up, but no shelf ID registers, after 30 seconds the fans go into full on combustion mode. The shelf ID never registers with the 2010 unit.

Since I only have one SP-PCM02-HE580, (looking for another one to satisfy proper testing) I have limited testing to one plugged in PSU, the PCM02. The rest of the 3x psus are slotted in but not plugged in, serving as blanks.
OG with a Compellent controller and one PCM02 will spin up fans in a more normal speed and fashion, register the shelf id. after 30 seconds the non plugged in slot 4 PSU will ramp its fan into full on combustion mode, while the PCM02 fan stays relatively tame. Unit seems functional outside of the excess noise from psu4.
FNG with a compellent controller and one PCM02 will spin up fans quickly, no shelf id registers, after 30 seconds the shelf goes in to full power fan mode. excessive noise, not usable.

So here is where I'm at. . . My understanding is that the 2010 and later Xyratex PCBs were more desired for IOM6 upgrades / high capacity drives. I have never had any capacity issues with my OG 2008 PCB, Ive got some 18TB and 14TB drives running fine. Im wondering what if any differences there are between backplane revisions. Some lockout must exist, either on the PSU, the PCB, or SAS Controller. I'm wondering if the HP SAS controllers are another route or if they are just clones of compellent. It seems I need at least another SP-PCM02-HE580 to test the psu4 fan speed issue when deploying in OG. The saga continues...

 

[Dravor]

Got a single TDPS-580BB / X522A in for testing and it looks promising. I've only done some surface level quick testing. I would say it's not the most scientific or exact measure approach, but a good enough measure to prove the point. The unit slots in with no issues as expected, as it is a netapp P/N for this PDU.

Testing methodology- Single PDU, slotted in, connected to the wall. 3x PDUs slotted in, not connected to the wall. 1x IOM6 Controller. Controller blanks to keep the fans from ramping. Tested Idle W and VA with no drives, as well as connected with 2x 18TB SATA drives. Wattage and VA were measured with a kill-a-watt type device and after the array had been powered on for a few minutes and the start up procedure and fan check have been completed. I am in the US running AC 120V.

X518A-R6 - No Drives + Idle - 60.02W / 52.15VA
X522A / TDPS-580BB - No Drives + Idle - 37.32W / 39.16VA
rough power savings @ idle with an empty shelf ~ 23W / ~13VA

X518A-R6 - 2x 18TB Drives / Connected to Unraid but idle - 73.47W / 75.02VA
X522A / TDPS-580BB - 2x 18TB Drives / Connected to Unraid but idle - 52.95W / 54.07VA
rough power savings @ idle with 2x 18TB SATA disks ~20.5W / 21VA

I'd say the TDPS-580BB / x522A are worth the upgrade to save some power. Most I found were in the $45US range, but some ebayers accepted offers of prices as low as $18US.

I was able to also test a SP-PCM02-HE580-AC. The unit physically slots in with no issues and powers the shelf on. That's about where the test stopped. Once the IOM is initialized the fan on the PCM cranks up to maximum RPM. Any PCM that is plugged in will also ramp up its fans. The fans in the SP-PCM02 are also different and make more of a higher pitched whine noise as opposed to the whoosh of the Netapp Units. I did not measure any wattage pull, as the fans were unbearably loud. Imagine an electric leaf blower plugged and blowing in behind the rack. My thoughts are that the PCM does not talk to the IOM properly and thus just defaults to full speed. I would think maybe this could be remediated with a different controller, either the HP3 Par controller or a Dell Comp controller, neither of which i have on hand right now. Extrapolating results from this test, I'd also venture to guess that the SP-PCM01 80+ Silver rated would most likely do the exact same thing.


TL;DR - TDPS-580BB / X522A can save some wattage and is a worthy upgrade if you can find them for a low enough price. The HP / Dell PCMs slot in physically and will power the shelf without issue, however the PCM does not communicate with the Controller properly and the fan speed ramps up to maximum RPMs. The noise is unbearable for regular use.

I'm curious if you have added more drives since and what power consumption you are seeing. Right now I have a r720xd, r710xd, and a r530 all of which are fully populated.

Am using unraid and the r710 is running like 130w avg. So reading in another post that this 424x run around 250w idle it scared me a bit.

But it appears with the right psu's these idle pretty low actually.

 

[aero]

Chiming in here because I'm a proud owner of a ds4246. I'm trying to test maximum throughput between the ds4246 and a host with a lsi2008. I've got both ports on the HBA connected to a single IOM6; trying to give it the most lanes possible rather than dual-path by going to both IOMs. I think I'm encountering a bottleneck around 3GB/s, but it's hard to tell exactly since I have a mixture of zfs pools and a couple random ssd and I'm trying to run FIO in parallel on all of them.

below is the output from lsiutil and I noticed one link (PhyNum 36) negotiated at 3Gb/s, but what is that link?

SAS2008's links are 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G

B___T SASAddress PhyNum Handle Parent Type
500605b0079b4a70 0001 SAS Initiator
500605b0079b4a71 0002 SAS Initiator
500605b0079b4a72 0003 SAS Initiator
500605b0079b4a73 0004 SAS Initiator
500605b0079b4a74 0005 SAS Initiator
500605b0079b4a75 0006 SAS Initiator
500605b0079b4a76 0007 SAS Initiator
500605b0079b4a77 0008 SAS Initiator
500a09800253bfbf 0 0021 0002 Edge Expander
0 96 5000c500a650f13e 9 0022 0021 SAS Target
0 95 5000c500a6cfbc26 10 0023 0021 SAS Target
0 94 5000c500a6cb2a7e 11 0024 0021 SAS Target
0 66 5000cca26a4054b2 12 0025 0021 SAS Target
0 71 5000cca26a3fe0ae 13 0026 0021 SAS Target
0 92 5000cca2510dd902 14 0027 0021 SAS Target
0 93 5000cca26a408aa2 15 0028 0021 SAS Target
0 91 5000cca26a4063a6 16 0029 0021 SAS Target
0 90 5000cca2510bf78a 17 002a 0021 SAS Target
0 89 5000cca2510d5c1a 18 002b 0021 SAS Target
0 88 5000cca2510b792a 19 002c 0021 SAS Target
0 86 5000cca2430ef026 22 002d 0021 SAS Target
0 87 5000cca24409a146 23 002e 0021 SAS Target
0 103 5000cca25d527f76 24 002f 0021 SAS Target
0 105 5000cca25d52f106 25 0030 0021 SAS Target
0 100 5000cca25d4d04aa 26 0031 0021 SAS Target
0 97 5000cca25d52f8ae 27 0032 0021 SAS Target
0 69 5000cca25d528346 28 0033 0021 SAS Target
0 101 5000cca25d52934a 29 0034 0021 SAS Target
0 99 5000cca25d51f0b6 30 0035 0021 SAS Target
0 98 5000cca25d521c6a 31 0036 0021 SAS Target
0 64 500a09800253bfbe 36 0037 0021 SAS Initiator and Target
0 104 5000cca2510d7c7e 8 0038 0021 SAS Target
0 106 58ce38ee20199cd7 20 0039 0021 SAS Target
0 107 58ce38ee20188b3b 21 003a 0021 SAS Target

Type NumPhys PhyNum Handle PhyNum Handle Port Speed
Adapter 8 0 0002 --> 3 0021 1 6.0
1 0002 --> 2 0021 1 6.0
2 0002 --> 1 0021 1 6.0
3 0002 --> 0 0021 1 6.0
4 0002 --> 7 0021 1 6.0
5 0002 --> 6 0021 1 6.0
6 0002 --> 5 0021 1 6.0
7 0002 --> 4 0021 1 6.0

Expander 37 0 0021 --> 3 0002 1 6.0
1 0021 --> 2 0002 1 6.0
2 0021 --> 1 0002 1 6.0
3 0021 --> 0 0002 1 6.0
4 0021 --> 7 0002 1 6.0
5 0021 --> 6 0002 1 6.0
6 0021 --> 5 0002 1 6.0
7 0021 --> 4 0002 1 6.0
8 0021 --> 1 0038 1 6.0
9 0021 --> 1 0022 1 6.0
10 0021 --> 1 0023 1 6.0
11 0021 --> 1 0024 1 6.0
12 0021 --> 1 0025 1 6.0
13 0021 --> 1 0026 1 6.0
14 0021 --> 1 0027 1 6.0
15 0021 --> 1 0028 1 6.0
16 0021 --> 1 0029 1 6.0
17 0021 --> 1 002a 1 6.0
18 0021 --> 1 002b 1 6.0
19 0021 --> 1 002c 1 6.0
20 0021 --> 1 0039 1 6.0
21 0021 --> 1 003a 1 6.0
22 0021 --> 1 002d 1 6.0
23 0021 --> 1 002e 1 6.0
24 0021 --> 1 002f 1 6.0
25 0021 --> 1 0030 1 6.0
26 0021 --> 1 0031 1 6.0
27 0021 --> 1 0032 1 6.0
28 0021 --> 1 0033 1 6.0
29 0021 --> 1 0034 1 6.0
30 0021 --> 1 0035 1 6.0
31 0021 --> 1 0036 1 6.0
36 0021 --> 36 0037 1 3.0

Enclosure Handle Slots SASAddress B___T (SEP)
0001 8 500605b0079b4a70
0002 0 500a098002548f2c 0 64

ioc1 LSI Logic SAS2008 B2 MPT 200 Firmware 14000700 IOC 0

SAS2008's links are 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G, 6.0 G

B___T___L Type Vendor Product Rev SASAddress PhyNum
0 64 0 EnclServ NETAPP DS424IOM6 0191 500a09800253bfbe 36
0 66 0 Disk HGST HUH721010AL42C0 A3Z4 5000cca26a4054b2 12
0 69 0 Disk HGST HUS726040ALS214 MS00 5000cca25d528346 28
0 71 0 Disk HGST HUH721010AL42C0 A3Z4 5000cca26a3fe0ae 13
0 86 0 Disk HGST HUS726040ALS214 MS01 5000cca2430ef026 22
0 87 0 Disk HGST HUS726040ALS214 MS01 5000cca24409a146 23
0 88 0 Disk HGST HUH721010AL4201 bT04 5000cca2510b792a 19
0 89 0 Disk HGST HUH721010AL4201 bT04 5000cca2510d5c1a 18
0 90 0 Disk HGST HUH721010AL4201 bT04 5000cca2510bf78a 17
0 91 0 Disk HGST HUH721010AL42C0 A3Z4 5000cca26a4063a6 16
0 92 0 Disk HGST HUH721010AL4201 bT04 5000cca2510dd902 14
0 93 0 Disk HGST HUH721010AL42C0 A3Z4 5000cca26a408aa2 15
0 94 0 Disk SEAGATE ST12000NM0027 E002 5000c500a6cb2a7e 11
0 95 0 Disk SEAGATE ST12000NM0027 E002 5000c500a6cfbc26 10
0 96 0 Disk SEAGATE ST12000NM0027 E002 5000c500a650f13e 9
0 97 0 Disk HGST HUS726040ALS214 MS00 5000cca25d52f8ae 27
0 98 0 Disk HGST HUS726040ALS214 MS00 5000cca25d521c6a 31
0 99 0 Disk HGST HUS726040ALS214 MS00 5000cca25d51f0b6 30
0 100 0 Disk HGST HUS726040ALS214 MS00 5000cca25d4d04aa 26
0 101 0 Disk HGST HUS726040ALS214 MS00 5000cca25d52934a 29
0 103 0 Disk HGST HUS726040ALS214 MS00 5000cca25d527f76 24
0 104 0 Disk HGST HUH721010AL4201 bT04 5000cca2510d7c7e 8
0 105 0 Disk HGST HUS726040ALS214 MS00 5000cca25d52f106 25
0 106 0 Disk TOSHIBA PX05SMB080 0101 58ce38ee20199cd7 20
0 107 0 Disk TOSHIBA PX05SMB080 0101 58ce38ee20188b3b 21

Also, when looking at the stats I'm comparing what FIO is outputting to what I see in zpool iostat and regular iostat. Is zpool iostat truthful? It looks to me like the write speed listed in zpool iostat is aggregate of all disks, so if I have a pool of mirrored vdevs, the real actual write throughput is half of what that is showing (as confirmed by FIO)?

 

[aero]

I'm going to respond to myself after further testing....

I'm still curious what phy36 is exactly, and why it's negotiated at 3Gb/s.

I still can't tell if I'm being limited by the # of SAS lanes or negotiated speeds, but these numbers kind of seem to indicate to me that maybe...

Here are some benchmark numbers...
"aggregated individual" means that I individually tested each array, then added up the values to give an idea of the theoretical max
"aggregated parallel" means I tested all arrays simultaneously and then added up the values

with one SAS cable connected between the host and IOM
aggregated individual writes - 3.23GB/s
aggregated parallel writes - 1.93GB/s
aggregated individual reads - 3.64GB/s
aggregated parallel reads - 2.13GB/s

with two SAS cables connected between the host and IOM
aggregated individual writes - 3.34GB/s
aggregated parallel writes - 2.62GB/s
aggregated individual reads - 3.7GB/s
aggregated parallel reads - 2.8GB/s

I'm still below what I think should be the theoretical max, and below 3GB/s; although using both connections definitely helps some. I think testing would be more definitive if I had more SSDs in the mix. For the current setup I blew away one of my raidz2 pools in lieu of a md raid 0, 2 SSDs in raid 0, and finally a zpool of mirrors.

 

[UhClem]

Chiming in here because I'm a proud owner of a ds4246. I'm trying to test maximum throughput between the ds4246 and a host with a lsi2008 ...

and

with two SAS cables connected between the host and IOM
... aggregated parallel writes - 2.62GB/s
... aggregated parallel reads - 2.8GB/s

There's a strong possibility (but I only have 2nd-hand empirical evidence) that the LSI2008 itself does not have the CPU/Data_Handling_Capacity to fully achieve the PCIe x8 Gen2 throughput ceiling (real-world 3.2-3.4 GB/s).

 

[aero]

That's funny you should say that. I was thinking the exact same thing, and was scouring the net to see if anyone had real world benchmarks specifically for the 9200-8e, and I couldn't find anything. Yes, it is old...maybe time for an upgrade to sas-3. However, I was also thinking about upgrading to IOM12's, until I saw they run ~$1500 on ebay...insane!

edit: found some youtube benchmarks from artofserver, and a post on STH from Patrick himself...they seemed to max out around 2.2 GB/s
edit: 9207-8e on the way!