EMC KTN-STL3 15 bay chassis

Netix

New Member
Sep 17, 2022
1
0
1
Hi guys. Found a KTN-STL3 locally for around 200$ CAD. Was wondering if that would be a good DAS for Unraid and a Plex server?

All caddy are included and they seems to be : 303-115-003D. The good one for SATA drive right ?

The sellers also sell a Dell 012DNW PCIe SAS 6Gbps HBA for 30$. Would that be a good HBA to pair with that ?

As I understand, unRAID can't seem to do multipath so I will only be able to plug one of the cable SAS cable instead of two right?

Thanks again for everything !
 

bonox

Member
Feb 23, 2021
69
15
8
since no other answer, I'll put in a partly justified, partly plucked from the nethers answer:

Aside from the fibre channel interface to the backplane on these chassis, the main reason for a sata interposer is to magic a single interface sata drive into two. That said, it's fairly common knowledge that sata drives with the -003D sas interposer will only work when connected to the A controller on top. So it's fairly pointless exercise trying to run dual pathing with sata drives except for trying to cover the loss of an HBA or cable, but if that's a concern for you, you should probably really be investing in SAS drives and dual pathing the storage system from end to end.

The H200 is a SAS2008 chipset, meaning that it should be able to take an IT firmware if you want a proper HBA, meaning it's a reasonable choice. Unraid, Truenas etc don't like to play with drives hiding behind IR firmware.

Read up here
for info on what cards can be turned into what.

TL;DR - only use one cable and the HBA is a reasonable choice.
 
  • Like
Reactions: Netix

mower

New Member
Mar 31, 2022
8
5
3
So as an additional note. The SATA drives only have one port on them, and the SAS drives have two ports on them. The difference in the connectors appears to be that the SATA connector has a physical gap between the power and (primary) data plugs, but the SAS drives do not. And if you look at the SAS drives, the plastic that spans the gap has data? connectors on the other side. I'm reaching out on a limb here, but I'm pretty sure that is the secondary port.

Now if we come back to the dual controllers on the KTN-STL3 drive shelves, the lower controller connects to the primary port, and the upper controller connects to the secondary port. So IF you are connecting up SATA drives, you will only be able to see them on the lower controller because they only have the primary port. And as bonox mentioned above, a single connection is just fine unless you are running SAS drives, and need to have redundant bus connections for high availability or something (and have a system that understands that you have two ports to the same device).

I have been running these shelves with a single LSI HBA, and daisy chained two shelves though the lower controller and have had no problems.. Nice units..
 

odabajo

New Member
Oct 8, 2022
1
0
1
Hi!

Just a newbie here. I have just register myself on the forum to ask some questions about this enclosure.

Are SAS caddies compatible with SATA drives if always using the bottom controller?

I have also read on this thread that if the PSU is 3rd gen or 3rd gen VE the unit is more quiet.

In your experience, is it quiet enough to be in the same room you will be using to work (for example)? Remotely working with computer, not other noisy machines.

I’m looking into buying some of these for my homelab but I’m a bit worried about the noise.

Thanks!
 

bonox

Member
Feb 23, 2021
69
15
8
they have been in my experience, but i've not mixed sas and sata together in the same enclosure/array.

as for noise, i find them quite quiet compared to the 1U and 2U servers they're next to. There is obvious fan noise but it's not crazy like the servers.
 

mower

New Member
Mar 31, 2022
8
5
3
Yes, I have been using them in a mixed configuration of SAS and SATA drives on the bottom controller only with no problems. (I actually just added my third shelf to my system :) (Chia farm.. (hopefully that doesn't draw any rebuke :))

It is a note that you have to have both power supplies powered and both controllers installed for the system to operate normally. If you pull one of them (like the second controller to save power) they seem to go into a fail-safe mode and crank the fans to maximum. If someone knows how to work around that, it would be nice to drop the redundant controller and save a few watts :)..

As for noise, Like bonox said, they are quieter than my server that is running them by quite a bit (they wake up with full fan speed, and then calm down (probably after they get a good temperature reading I assume)). I wouldn't suggest them in a living room kind of environment, but in a office they are relatively quiet.. (note that my chia far doesn't really hammer on them very hard, so if you have a high load environment for them, they might perk up the fans.. just to be clear :)
 

pzwahlen

New Member
Jun 24, 2021
7
1
3
Hi everyone, first time poster here, please be gentle ;-)

I have a setup made of one Dell R620 running RockyLinux 9 and one KTN-STL3 full of 3TB SAS HGST drives (formatted in 512 bytes) and it is causing me great headaches.

I am trying to run OpenZFS (2.1.6) and while it works perfectly well under normal use, as soon as I enable ZED (for monitoring) or request fault/locate led status, then I start getting mpt2sas errors and IO errors, up to the point where I completely loose the enclosure under /sys/class/enclosure. This happens in about 10 seconds and a full log of such an event is attached.

What I have tried so far:
- I initially tested with a Dell PERC H810 reflashed to IT mode. I have then switched to an LSI 9207-8e (running P20) without luck.
- As this HBA is not officially supported on RHEL9, I tested on RHEL8 and RHEL7, without luck (exact same errors on the 3 OS)
- I changed the single SAS cable that I am using and tried controller B instead of controller A, without success
- I actually have a second enclosure and have the problem on both.
- I have tested all the disks for read/write badblocks for several days.
- I have tested my memory with Memtest86+ for several days as well.

I first had the impression that ZED was the culprit and decided to stop using it. Later, I discovered that running the command "zpool status -c locate_led" (as a non-root user) would also crash this setup and this makes it too fragile for production. I therefore need to keep looking for a solution (or call it a day and give up with this setup).

This "zpool status" command is mostly iterating over the block devices and issuing "cat /sys/.../locate" and the strange part is that so far I have been unable to crash the system by manually hitting the /sys interface with "cat" or "echo" commands on the fault/locate leds. Only zed/zpool seem to be able to trigger that behavior.

Also, I have been unable to crash a small pool with up to 4 VDEVs, only larger pools seem to have this weakness.

I think my question is: did any of you experience strange SAS errors while running led-related commands on this enclosure and did you ever find a solution ?

Thanks for any hint/input. Cheers. Patrick!
 

Attachments

pzwahlen

New Member
Jun 24, 2021
7
1
3
Thanks @Stephan for shimming in.

I did try "mpt3sas.max_queue_depth=10000" a couple weeks ago, and just tried it again today, without luck.

I also tried both the circle and square connectors of controller A and it doesn't make a difference. Is there a recommendation here ?

I am now recording the output of "strace zpool status -c slot" and "strace zpool status -c locate_led". Both commands end up running the exact same bash script, but the first one doesn't trigger errors, whereas the second one very reliably crashes my 6 devices pool.

Cheers. Patrick!
 

pzwahlen

New Member
Jun 24, 2021
7
1
3
Some more updates (and cross-posting from zfs-discuss)

I have received 2 more identical JBODs and using the same drives, I am
getting a different behavior, confirming 100% where the problem sits.

I have tried adding VDEVs one by one (no redundancy) and zed/zpool
status starts triggering the problem at 5+ drives. On these "new"
JBODs, I am not losing the drives/enclosure anymore and I am getting a
new mpt2sas error that I couldn't decode:

mpt2sas_cm0: log_info(0x3112011a): originator(PL), code(0x12), sub_code(0x011a)

Also, I have discovered that performing sequential "cat" to
/sys/.../{locate,fault} doesn't trigger the problem. However, doing
parallel "cat" generates the errors. On a pool with 7 VDEVs:

for i in $(seq 0 6); do (cat /sys/class/enclosure/*/$i/locate); done # ===> No errors
for i in $(seq 0 6); do (cat /sys/class/enclosure/*/$i/locate &); done # ===> Errors

The cards in the newer JBODs are flagged Rev13
 

pzwahlen

New Member
Jun 24, 2021
7
1
3
Quick update,

The old/problematic controller are marked Rev13, too. They have much smaller "seeds" and serial numbers, that all.

The problem really seems to be limited to parallel actions to the /sys filesystem so I have worked around it by adding a lock/mutex to the ZFS scripts (using flock).

I have posted my quick fix in the zfs-discuss mailing list here: Topicbox
 
  • Like
Reactions: Stephan

Stephan

Well-Known Member
Apr 21, 2017
561
360
63
Germany
Thanks for the update! Kudos also for going down that rabbit hole.

So is the error in LSI controller firmware or within the enclosure firmware?

In essence, does the EMC KTN-STL3 need strictly serialized (and slow?) SES accesses from the host?
 

pzwahlen

New Member
Jun 24, 2021
7
1
3
The problem is definitely with the expander's firmware, inside the enclosure.

I could test with 4 enclosures, 2 "broken" and 2 "semi-broken". When sending parallel ses commands to the broken ones, the mpt2sas drivers ends up loosing the enclosure completely (they disappear from /sys/class/enclosure under Linux).

On semi-broken ones, I get another mpt2sas error but I don't loose the enclosure. If I swap the expanders between enclosures, the problem follows the expanders (and not the actual enclosure).

If I tweak ZFS to only send sequential commands (with my flock hack), then all enclosures behave "normally".
 

Bert

Active Member
Mar 31, 2018
497
203
43
43
they have been in my experience, but i've not mixed sas and sata together in the same enclosure/array.

as for noise, i find them quite quiet compared to the 1U and 2U servers they're next to. There is obvious fan noise but it's not crazy like the servers.
As for noise, Like bonox said, they are quieter than my server that is running them by quite a bit (they wake up with full fan speed, and then calm down (probably after they get a good temperature reading I assume)). I wouldn't suggest them in a living room kind of environment, but in a office they are relatively quiet.. (note that my chia far doesn't really hammer on them very hard, so if you have a high load environment for them, they might perk up the fans.. just to be clear :)
I have been trying to use these disk shelves with SATA drives but no luck. I think sata drives do not even initialize since blue light do not come on with sata drives. Is there a special trick here?
 

bonox

Member
Feb 23, 2021
69
15
8
blue light? isn't that just power to the shelf? The caddies just have yellow/green lights (don't trust me here; i'm colourblind)

They do make specific sata caddies - don't know if the interposer part number is the same as the sas unit though. Also something to consider is that i've not tried 4k sector drives on these shelves. If you're using new one with 4Kn it could be an issue.

Most likely problem with sata is using the wrong controller/port. Have you tried both ports on both controllers?

I guess also don't discount the possibility that the expander has an old firmware or is broken.
 

papkee

New Member
Nov 28, 2022
8
0
1
Hi all,

Having some issues with my KTN-STL3, a 9200-8i, HGST H7240AS60SUN4.0T drives, and the 303-115-003D interposers.

The drives don't seem to spin up properly, or at least aren't full coming online. Here's what dmesg has to say:
Code:
[  537.342364] sd 1:0:5:0: [sdh] Spinning up disk...
[  538.350836] ..................................................................................................not responding...
[  637.725088] sd 1:0:5:0: [sdh] Read Capacity(16) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[  637.725099] sd 1:0:5:0: [sdh] Sense Key : Not Ready [current] [descriptor]
[  637.725104] sd 1:0:5:0: [sdh] Add. Sense: Logical unit not ready, initializing command required
[  637.725888] sd 1:0:5:0: [sdh] Read Capacity(10) failed: Result: hostbyte=DID_OK driverbyte=DRIVER_OK
[  637.725894] sd 1:0:5:0: [sdh] Sense Key : Not Ready [current] [descriptor]
[  637.725897] sd 1:0:5:0: [sdh] Add. Sense: Logical unit not ready, initializing command required
The drives show up with an lsscsi, but without size information:
Code:
[1:0:5:0]    disk    HGST     H7240AS60SUN4.0T A3A0  /dev/sdh   35000cca07321dee0  /dev/sg7        -
Am I out of luck with these drives? I'm not sure where the breakdown is occurring, but others have claimed to use the same model drives with the same HBA without issues, and the only difference in my setup is the KTN-STL3.