EMC KTN-STL3 15 bay chassis

[negative15gigabytes]

I picked up two of these shelves a week ago(each shelf has a different number, 046-004-212-A05, 046-004-212-A04). Both came with two different types of interposers. I have the 303-116-003D and 250-136-911C. Can I use SAS with both?

 

[Gomie]

anyone able to use software to control HDD led with ktn-stl3? or is there a good way to look for a particular drive?

I have great luck with blinking drives off my Dell PERC h830. I have some drives in RAID arrays and some passed through as non raid drives. I have three of the stl3 chassis daisy chaned off of one external port and then a net app 4246 off the other controller port.

 

[tuffghost]

Hi Everyone, thanks for this great thread, there was a ton of information in here that let me grab a few of these shelves and easily hook them up to my Unraid box.

They seem to be working 100% fine but just noticed that the bottom shelf, most of the drives have an amber light where the top ones don't. This is on empty bays too. If I take bays from the top and bring them down it doesn't make a difference so it isn't the trays / interposers I don't think.

In the end, It doesn't really matter since things appear to be working fine but just wondering if anyone has seen them behave like that and what it might be?

They are all using EMC 303-115-003D interposers to a pair of LSI 9207-8e. And from what I can tell the controllers in the back have the same part numbers.Acts the same way with two power cables or a single one.

 

[jakobmorckchristensen]

Hello all, thanks for all the nice info..

my setup.
Hp Z800 (running super stable. see video on youtube (They spared no expenses! HP Z800 Workstation (PWJ184) )

dual cpu. Intel(R) Xeon(R) CPU X5650 @ 2.67GHz
dual gpu. nvidia quadro/nvidia geforce (transcode)
dual NVMe card. w. ssd boot. Mirroring. AXAGON PCEM2-D (system Truenas Scale TrueNAS-SCALE-22.12.2 )
Broadcom 5761 PCI-Express LAN Controller
192 gb ram.
psu.800w
DELL MJFDP LSI Quad Port PCIe SAS HBA SAS9201-16e Raid P20
see video on youtube (Something strange with LSI 9206-16e... | How to fix LSI firmware settings) Art of Server



Storage Expansion#1
2x lindy quad drive enclosure SAS (2xSAS cable from SAS9201-16e to lindy quad drive enclosure )
( 1 x RAIDZ3 | 8 wide | 931.51 GiB )



Storage Expansion#2 SAS
1xEMC KTN--STL3 15 Bay Chassis (1xSAS cable from SAS9201-16e to modul B OO)
1 x RAIDZ3 | 9 wide | 3.64 TiB
1 x RAIDZ2 | 6 wide | 2.73 TiB


see video on youtube (Teardown of a Disk Storage System EMC CX3-80 (PWJ62))



all running smooth...
Acronis Cyber Protect home (pc backup) from pc to truenas scale)
synology hyper backup rsync (synology backup to truenas sacle)
Jellyfin (transcode HW)
nextcloud
Windows (SMB) Share
Virtual Machines 5 x Windows (Remote Desktop) , 2 x Linux.(VNC)

Jakob.

 

[BrassFox]

blue light? isn't that just power to the shelf? The caddies just have yellow/green lights (don't trust me here; i'm colourblind)

They do make specific sata caddies - don't know if the interposer part number is the same as the sas unit though. Also something to consider is that i've not tried 4k sector drives on these shelves. If you're using new one with 4Kn it could be an issue.

Most likely problem with sata is using the wrong controller/port. Have you tried both ports on both controllers?

I guess also don't discount the possibility that the expander has an old firmware or is broken.

About the 4k sectors: I never even thought about this.

Some years ago I had made a project out of harre-formatting all my storage data (SATA, shucked) drives to have 64kb sectors, back when I had them all individually wired to an HBA. After making the step up to these DAEs I just kept doing that 64kb format whenever adding a drive without thinking much about it. Is there an optimum sector size the the KTN-STL3’s?

There’s so little information available on these units and most of the the good stuff is right here. I’m very grateful for that.

Still looking for a good way to control those damn fans, too. I’ve gone as far as to crack open the psu’s and rewire every fan through individual buck converters. Thus far, the damn thing senses my trickery, throws a fan fault, and I am foiled. Have not given up yet though.

 

[doppler_shift]

Hi Everyone, thanks for this great thread, there was a ton of information in here that let me grab a few of these shelves and easily hook them up to my Unraid box.

Thinking about doing something similar, but one thing I haven't found much information about is how to properly rack-mount these units into a rack meant for servers (mine is 29" deep). Do the EMC rails (100-560-184) work or are they too short? is there an alternative?

 

[bonox]

About the 4k sectors: Is there an optimum sector size the the KTN-STL3’s?

I've never heard of 64kB sector sizes for drives. Doesn't mean they don't exist, just that i'm unfamiliar with them. The sector size is the physical minimum storage unit on the disk. A cluster or block size is one to many multiples of the sector size, and those should be dependent on the data you're storing - 64kB or 1MB block sizes for example are a file system abstraction rather than how the disk presents itself to the controller. A 64kB block 'format' won't actually change the physical on disk sector structure, but low level formatting between 512B and 4kB will change the physical disk sector structure.

I'm aware that there are issues with 4k sector size drives on some sas2 raid/it controllers. Latest firmware for LSI units is supposed to fix them, but i've had variable success. The sas3 controllers i've tried are immune. Same would apply to firmware on expanders, which is why I make the comment with respect to the sas2 STL3

 

[bonox]

is there an alternative?

I use full depth rack shelves rather than rails, mostly two units per shelf, but I use three supports on each side of the shelf rather than just one on each end as is usual. I wouldn't trust a cantilevered shelf with these either.

 

[Nikika]

Hi Everyone,
I would like to ask, does it matter with which cable I connect LSI 9200-8e IT mode and EMC KTN-STL3 ?
for now I'm using Dell EMC 038-003-786 Amphenol 1M/3' Mini SAS SFF-8088 Mini SAS Cables with two grooves, but I'm wondering if I can also use three-groove ones?

 

[oneplane]

I'm thinking about getting one or two of them as well (especially since they are going so cheap: EMC2 VNX 15 Bay Disk Enclosure Disk Array - Dual Controllers - KTN-STL3 + PSU OR 818212955243 | eBay ) but I'm wondering if anyone has them running with SATA SSDs? I've seen some mentions of SAS SSDs and some formatting issues, but not just a handful of bare SATA SSDs (like the ECO CloudSpeed models). I wouldn't be doing it for the bandwidth but more for the seek time mixing about 10 HDD's and 5 SSD's per chassis (maybe a few different ratios). In theory the controller and multiplexer/multiplier shouldn't really care what's attached, but you never know...

 

[Sean Ho]

For SATA SSDs on SAS HBA, an issue is RZAT/DRAT support for passing TRIM through the SATA/SAS translation layer. Many enterprise/DC SATA SSDs have RZAT, e.g., S3500/3600. Very few consumer SSDs have it, and very few say whether they do or not. You can check if a drive reports it using hdparm.

 

[araemo]

Hi Everyone, thanks for this great thread, there was a ton of information in here that let me grab a few of these shelves and easily hook them up to my Unraid box.

They seem to be working 100% fine but just noticed that the bottom shelf, most of the drives have an amber light where the top ones don't. This is on empty bays too. If I take bays from the top and bring them down it doesn't make a difference so it isn't the trays / interposers I don't think.

In the end, It doesn't really matter since things appear to be working fine but just wondering if anyone has seen them behave like that and what it might be?

They are all using EMC 303-115-003D interposers to a pair of LSI 9207-8e. And from what I can tell the controllers in the back have the same part numbers.Acts the same way with two power cables or a single one.

Did you figure out the Amber LEDs on one of your enclosures? For the empty bays, I'd bet if you carefully reseat the empty drive trays the LEDs will come on on those too (that got all 15 of mine to light up. ).

I recently purchased 2 of these, and on the first one - all the trays have the amber LEDs on full-time. When I got the second one I thought it wasn't working at first because the amber lights don't come on. Turns out both seem to work almost perfectly (my SATA drives seem to randomly negotiate at 3gbps instead of 6gbps.. but 3gbps is still faster than they can transfer so I'm not too worried about it). So, I figure the interposers are not seeing something they expect to see in my first enclosure, but they do see it on the second one, or for some reason an 'identify' code or something is set on one of my enclosures? But I pored through the output of sg_ses -a for both enclosures, and found only 3 elements (unfortunately, all 'Vendor specific element type' entries) that are different between the two (besides things like slot status for ones that have drives vs don't, SCSI addresses, fan rpms, and temperatures).

Enclosure with orange tray LEDs off	Enclosure with orange tray LEDs on
[3,17] Element type: vendor specific [0x81] Enclosure Status: Predicted failure=0, Disabled=0, Swap=0, status: OK Vendor specific element type, status in hex: 01 2a 11 20	[3,17] Element type: vendor specific [0x81] Enclosure Status: Predicted failure=0, Disabled=0, Swap=0, status: OK Vendor specific element type, status in hex: 01 2a 11 e0
[3,18] Element type: vendor specific [0x81] Enclosure Status: Predicted failure=0, Disabled=0, Swap=0, status: OK Vendor specific element type, status in hex: 01 2a 12 20	[3,18] Element type: vendor specific [0x81] Enclosure Status: Predicted failure=0, Disabled=0, Swap=0, status: OK Vendor specific element type, status in hex: 01 2a 12 e0
[12,-1] Element type: vendor specific [0x81] Enclosure Status: Predicted failure=0, Disabled=0, Swap=0, status: OK Vendor specific element type, status in hex: 01 ff 00 20	[12,-1] Element type: vendor specific [0x81] Enclosure Status: Predicted failure=0, Disabled=0, Swap=0, status: OK Vendor specific element type, status in hex: 01 00 00 20

I tried doing sg_ses --set for these bytes but it just told me to get lost.

ledctl locate/locate_off does turn the LEDs from solid amber to flashing amber and back.

All other lights/displays on both seem to match, including the rear side items. Both are working perfectly with my SATA drives. I tried pulling the "B" controller from each and didn't get any change to the drive tray LEDs.

As others in the thread suggest, it's probably proprietary EMC control messages, but it'd be nice if we could reset these to get rid of the errors.

I may try pulling the B controller from my chassis with the LEDs off, and putting it in the other chassis in place of the A controller I'm using now.

 

[Fiberton]

About the 4k sectors: I never even thought about this.

Some years ago I had made a project out of harre-formatting all my storage data (SATA, shucked) drives to have 64kb sectors, back when I had them all individually wired to an HBA. After making the step up to these DAEs I just kept doing that 64kb format whenever adding a drive without thinking much about it. Is there an optimum sector size the the KTN-STL3’s?

There’s so little information available on these units and most of the the good stuff is right here. I’m very grateful for that.

Still looking for a good way to control those damn fans, too. I’ve gone as far as to crack open the psu’s and rewire every fan through individual buck converters. Thus far, the damn thing senses my trickery, throws a fan fault, and I am foiled. Have not given up yet though.

Change your powersupplies to ones that are PWM. They quiet way down. This I know work correctly. P/N: 071-000-553 - 3rd Generation VE

 

[AlistairM]

Would someone be able to run a sanity check over these items I'm about to purchase to make sure they would work together please?

3 times EMC KTN-STL3
45 times Seagate Exos X18 Enterprise 16TB SAS
2 times LSI 9200-8e 6Gbps
8 times SFF-8088 to SFF-8088
3 times 100-560-184 Rail Kit (I have a 29 inch deep rack, I'm not sure if these are compatible or not).


I plan to link the JBODs together to my existing NAS as such:



Assuming this is all good, if I were to continue extending the setup in the same way, adding more JBODs to the daisy chain, how many would I theoretically be able to add?

 

[preston_brown]

I'm not too far off that. Looks fine.

 

[AlistairM]

I'm not too far off that. Looks fine.

Thanks! - Any thoughts on the rails or the ability to add more JBODs to the chain?

 

[preston_brown]

You can definitely add more EMCs as JBODs. I have 5. The hard drives are slow, unlikely to ever saturate 6GBPS.

I am not sure on the rails. Pretty sure those aren't "sliding" rails just ones you either extend or shrink to fit your cage length. And your cage is more than deep enough to support the EMCs.

 

[bonox]

unless you've got software that support multi-pathing, the right side of the connection tree is pointless. The Cards will say in their literature something like "Supports up to 254 devices" or some such, which means keep adding shelves up to that point, after which that's what you'd use the second card for. BTW, with your proposed setup, apps like Truenas will show you have 90 disks installed.

 

[AlistairM]

unless you've got software that support multi-pathing, the right side of the connection tree is pointless. The Cards will say in their literature something like "Supports up to 254 devices" or some such, which means keep adding shelves up to that point, after which that's what you'd use the second card for. BTW, with your proposed setup, apps like Truenas will show you have 90 disks installed.

Thank you very very much, I wasn't aware of any of that. Yes, I'm using Truenas Scale, and according to here, you're right, it doesn't support multipathing. Is it worth changing to another OS so I can use multipathing?

Also, does the fact that there is a loop in each of the daisy chains impact multipathing at all? Should I keep the loop, get rid of the loop since Truenas Scale doesn't support multipathing...?

 

[bonox]

unless you've got monetary issues with downtime I wouldn't go for multipathing, especially with a copy on write filesystem. And if I did have hefty SLA's, I go for clustering first since you cover the server going down in addition to the data path.

get rid of the HBA to top shelf link as well. If you want more bandwidth, make the top shelf attach only to port B of the HBA, and the remaining two shelves chained on port A