Problems with a Dell TL4000 SAS tape library + LTO drives

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

gargravarr

Member
Jul 1, 2021
36
1
8
Hi folks. I don't see a dedicated Tape subforum so I guess this is the closest?
I got my hands on a Dell TL4000 tape library - 48-slot, 4 drive slots (3 included) and 2 PSUs. The library itself seems to be working - diagnostics pass, the robot moves tapes around etc. I got into the web UI without issue and updated the firmware to latest. It is very slow, taking >15 minutes to boot up from cold. However I'm having trouble getting it to work with my Linux machines.
It has 1x LTO-3 and 2x LTO-4 tape drives installed plus an empty LTO-3 sled, though one of the LTO-4 drives doesn't work. I've been trying to get things working with the LTO-3 - none of my LTO-3 tapes have labels, but I printed my own for a box of unused LTO-2 tapes and they were recognised by the library.
I have multiple problems here so I hope I can describe them in a semi-logical order. I think they're all related but they may not be.
  1. If I connect the LTO-3 or -4 tape drive to my NAS via SAS, I can only see the tape drive. It doesn't expose a SCSI LUN for the robot. I'm not sure if this is a problem with my SAS card (Adaptec ASR-78165 in HBA mode) but this machine is going to be my intended tape server.
  2. If I connect it to a different machine with a Dell H200 SAS HBA, it does expose the necessary LUN for the robot. I can move tapes between the magazine and the drive. However, if I try to read or write anything off the drive using tar with a tape loaded, I get errors that no medium is loaded. Sometimes the robot even unloads the tape and stores it without being instructed.
  3. I have a Dell PowerVault 114X with an LTO-5 drive in it and plenty of LTO-5 tapes. I really want to upgrade to LTO-5. The drive is the same IBM series as in the TL4000 and has the correct physical interfaces. However, if I connect the drive to a spare sled and install it in the library, there are two issues:
    1. The library recognises it only as the LTO generation written on the sled (i.e. 3 or 4), not 5
    2. The library throws an error after trying to initialise the drive on install or boot - code 'FB 0x' where x is the tape slot
Googling around for the last issue suggests that despite the tape drive physically interfacing to the sled, being in the same series and being Dell OEM, there are some internal changes to the drive that go beyond firmware - a standalone drive is not able to work in a tape library. I found a single reference to setting the DIP switches on the drive - number 5 needs to be ON for a library. Sure enough, the drives I got with it had this set, but setting it on my LTO-5 drive didn't make a difference.
I've tried updating the firmware on the drives themselves, which made no difference. I know this is an old library, but I see Dell are still providing firmware and their documentation details it working with LTO-8 drives so it's got a pretty long service life.
The fact that I can't even get it to work in a fairly stock configuration (fully initialised OEM LTO-3 drive and LTO-2 tapes) makes me wonder if there's a fault with this library that I can't fix.
My servers run Devuan 4 (Debian 11 without systemd) on a 5.10 kernel. SAS otherwise works on the NAS - I have SAS HDDs, and I'm able to use the PowerVault 114 via the same card.
As a last point, I know this library is a rebadged IBM TS3200. I noticed on eBay there's a number of HP LTO sleds for sale and they look identical to the ones I have - also a rebadged library for HP? Would an HP LTO-5 sled work in a Dell library? Can I flash some firmware to make this thing work as I want it?

Many thanks in advance.
 

Zombielinux

Member
Jun 14, 2019
71
21
8
Oh boy. A lot to unpack here.

Slow boot up process: Yes that's normal. Doubly so if your tapes don't have barcodes since it has to load each one.

As far as I know, the LTO3 drives only have SCSI interfaces, not SAS. I could be wrong in this. Does it work as expected when you have just the two LTO4 drives? Maybe start with just the first LTO4 drive that works.

LTO2 labels are different than LTO4 labels, there is different information encoded in the barcode.

Regarding your issues:

1) See above. Try with only LTO4 drives at first, preferrably just the working one. Put it into I think the bottom slot (could be the top, memory is a bit fuzzy as to how they're ordered in software)

2) When you use the `tar` command, which device are you accessing?

3) The SAS sleds don't actually have any brains on them to identify the drives. They just have some voltage converters/isolators. Something else is going on. I recommend trying one drive at a time.

There are arcane references to the DIP switches. For an LTO4 SAS drive to be recognized they need to be in the orientation: OFF, OFF, OFF, OFF, ON, OFF, OFF. That seems to be the same pattern for the LTO5 drives.

Lastly, Yes, its a TS3200. No you can't use HP drives. PM me for tape drive information the elites don't want you to know!
 
  • Like
Reactions: edge

jd456

New Member
Sep 16, 2020
11
13
3
Have you got any information on flashing HP ESL drives to MSL libarys or HP ESL/MSL drives to dell TL2000 libarys @Zombielinux ? This would be very usefull to me as well
 

Zombielinux

Member
Jun 14, 2019
71
21
8
Have you got any information on flashing HP ESL drives to MSL libarys or HP ESL/MSL drives to dell TL2000 libarys @Zombielinux ? This would be very usefull to me as well
From the research I did, it doesn't appear possible. If I recall, HP uses Quantum drives, while dell uses IBM. Not sure how that works with LTO7 or 8 where IBM is the only remaining player.
 

gargravarr

Member
Jul 1, 2021
36
1
8
Oh boy. A lot to unpack here.

Slow boot up process: Yes that's normal. Doubly so if your tapes don't have barcodes since it has to load each one.

As far as I know, the LTO3 drives only have SCSI interfaces, not SAS. I could be wrong in this. Does it work as expected when you have just the two LTO4 drives? Maybe start with just the first LTO4 drive that works.

LTO2 labels are different than LTO4 labels, there is different information encoded in the barcode.

Regarding your issues:

1) See above. Try with only LTO4 drives at first, preferrably just the working one. Put it into I think the bottom slot (could be the top, memory is a bit fuzzy as to how they're ordered in software)

2) When you use the `tar` command, which device are you accessing?

3) The SAS sleds don't actually have any brains on them to identify the drives. They just have some voltage converters/isolators. Something else is going on. I recommend trying one drive at a time.

There are arcane references to the DIP switches. For an LTO4 SAS drive to be recognized they need to be in the orientation: OFF, OFF, OFF, OFF, ON, OFF, OFF. That seems to be the same pattern for the LTO5 drives.

Lastly, Yes, its a TS3200. No you can't use HP drives. PM me for tape drive information the elites don't want you to know!
Yep, I had no idea I'd encounter quite so many issues. I mean, we use much bigger tape libraries at work (>100PB onsite) and a team member maintains it, but I'd hoped something smaller would not generate as many curses!

The library boots slowly even though I have only a small number of labelled tapes installed. Is the robot checking each and every slot in turn for presence of a tape? Guess that would explain the slowness. The labels are printed barcodes stuck to the cartridge with sticky tape. I made sure to print the correct barcode for each generation of media, e.g. all the LTO-2 tapes have L2 as the final 2 characters in the barcode.

The IBM LTO-3 drives are actually a mix of Fibre Channel, Parallel SCSI and SAS, to suit the customer. All of the drives I have are definitely native SAS. I have a similar LTO-3 drive shucked from a different tape library that's native Fibre Channel. I also have a standalone Tandberg LTO-3 drive that's SAS, which I added to the PowerVault 114.

I can try with just the LTO-4 drive.

1) Yes, the slots are 1-4 from the bottom upwards, so the lowest slot is 1.

2) Just a simple tar -tf to see if there's anything on the loaded tape gives a media error. If I try mt status, I get the same, the drive errors out saying there's no media.

3) I don't think so, because like I say, the LTO-5 drive installed into the LTO-3 sled actually shows up to the library as an LTO-3 drive. I have no idea how, but there is a fairly long circuit board on the side of the sled, maybe there's a chip somewhere on it that's flashed with the drive information.

Yeah, DIP switch 5 ON, all others OFF, matches what I've seen.

I didn't want to use HP drives, I wanted to use HP sleds, since I can't find TL4000 LTO-5 sleds without them already having the drive installed and commanding a huge price tag (~£1,000).
 

Zombielinux

Member
Jun 14, 2019
71
21
8
Have you loaded the mtx module and associated commands? IIRC mt is the much older version of the software.

I'm not sure about LTO3 sleds, but I know the LTO4 sleds have no such eeprom or memory of any sort. My source: The stock LTO4 TL4000 FH sled sitting on my desk! It worked fine for SAS LTO5 drives before I moved to FC.

HP Sleds should be just fine. Because there's no circuitry on there at all.
 

gargravarr

Member
Jul 1, 2021
36
1
8
Have you loaded the mtx module and associated commands? IIRC mt is the much older version of the software.

I'm not sure about LTO3 sleds, but I know the LTO4 sleds have no such eeprom or memory of any sort. My source: The stock LTO4 TL4000 FH sled sitting on my desk! It worked fine for SAS LTO5 drives before I moved to FC.

HP Sleds should be just fine. Because there's no circuitry on there at all.
As far as I can tell, mtx is specific to the robot - it allows for tape loading and movement operations. I'm not 100% sure but I don't think it has commands for the drive once the tape has been loaded. So you have to use mt and mtx together. But in any case, yes, on the machine I had this working on, the mtx module was loaded and working, and I could use mtx to move tapes between slots or the drive.

I wonder if there could be something different about the HH sleds, because I'm struggling to explain why this problem exists. When I plug in the LTO-5 drive, it clearly shows the details of the LTO-3 drive that was previously in the sled. I think the same happened in the LTO-4 sled as well.

Incorrect to say the HP sleds have no circuitry, they clearly have the same long PCB to the side of the drive to interface the tape drive to the library. Like I say, they look exactly the same as the Dell sleds superficially.
 

Zombielinux

Member
Jun 14, 2019
71
21
8
Are you fully rebooting (pull power) between swapping drives? Because its not hot swappable in my experience.

I should rephrase. The "circuitry" on there is just signal conditioning, line drivers, and isolators between the drive and the end of the long pcb. The part numbers indicate no memory of any sort is on that PCB.
 

gargravarr

Member
Jul 1, 2021
36
1
8
Are you fully rebooting (pull power) between swapping drives? Because its not hot swappable in my experience.

I should rephrase. The "circuitry" on there is just signal conditioning, line drivers, and isolators between the drive and the end of the long pcb. The part numbers indicate no memory of any sort is on that PCB.
The library does have some hot-swap capability AFAICT, it will recognise when a sled is removed and reinstalled, and will attempt to initialise the drive once plugged in fully.

However, I have completely cold-booted the library and the problem persists. It only identifies what was last in the sled.

So there are some electronics on that PCB, but no processing or memory of any kind that you're aware of?
 

Zombielinux

Member
Jun 14, 2019
71
21
8
The library does have some hot-swap capability AFAICT, it will recognise when a sled is removed and reinstalled, and will attempt to initialise the drive once plugged in fully.

However, I have completely cold-booted the library and the problem persists. It only identifies what was last in the sled.

So there are some electronics on that PCB, but no processing or memory of any kind that you're aware of?
There is not. I guarantee it. I'm an EE by trade, so scrounging part numbers is right up my alley.
 

gargravarr

Member
Jul 1, 2021
36
1
8
Unless it's not the sled, but the slot, then. I may have to look more closely at what it thinks is in the slot and what was last in it...
 

Zombielinux

Member
Jun 14, 2019
71
21
8
Unless it's not the sled, but the slot, then. I may have to look more closely at what it thinks is in the slot and what was last in it...

Don't forget to mix the order up too. Sometimes things get stuck in firmware, especially arcane firmware like tape libraries.
 

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,140
594
113
New York City
www.glaver.org
Hi folks. I don't see a dedicated Tape subforum so I guess this is the closest?
Sure. I'll try to address your points in order. I have a few of these exact units on a variety of systems.
I got my hands on a Dell TL4000 tape library - 48-slot, 4 drive slots (3 included) and 2 PSUs. The library itself seems to be working - diagnostics pass, the robot moves tapes around etc. I got into the web UI without issue and updated the firmware to latest. It is very slow, taking >15 minutes to boot up from cold.
The back of each "pocket" in the magazine has a barcode that signals "nobody home" to the barcode reader in the picker. That is more picky (no pun intended) than reading tape labels, since it is further away. The pickers are the problem area in that library. You might want to remove the cover (a variety of Torx + Philips screws on the sides, top and back) and run through the power-up "naked" to see if it having trouble with a specific magazine or left vs. right sides. There is a plastic slide strip attached to the inside metal base of the library. That can get gunked up and slow the picker down - look for dark / skid marks on the plastic.

These libraries will NOT work without feet on a table top or rack shelf (the official rack mount brackets are fine). If it sits on the bottom of the chassis it will warp slightly and cause you no end of picker problems.

Similarly, NEVER ship the library without the shipping lock inserted and securely taped down with the picker in the "home position". If the shipping lock has been lost (it is supposed to be stored in the notch in the back blank faceplate above the controller card) you cam make your own by bending a standard-size paper clip into a "T" shape.
However I'm having trouble getting it to work with my Linux machines.
All of my systems that have these are FreeBSD or OpenVMS.
It has 1x LTO-3 and 2x LTO-4 tape drives installed plus an empty LTO-3 sled, though one of the LTO-4 drives doesn't work. I've been trying to get things working with the LTO-3 - none of my LTO-3 tapes have labels, but I printed my own for a box of unused LTO-2 tapes and they were recognised by the library.
Check out my blog post here for info on how I print my own labels.
I have multiple problems here so I hope I can describe them in a semi-logical order. I think they're all related but they may not be.

If I connect the LTO-3 or -4 tape drive to my NAS via SAS, I can only see the tape drive. It doesn't expose a SCSI LUN for the robot. I'm not sure if this is a problem with my SAS card (Adaptec ASR-78165 in HBA mode) but this machine is going to be my intended tape server.

If I connect it to a different machine with a Dell H200 SAS HBA, it does expose the necessary LUN for the robot. I can move tapes between the magazine and the drive. However, if I try to read or write anything off the drive using tar with a tape loaded, I get errors that no medium is loaded.
On FreeBSD, the changer shows up as a "ch" device and a "pass" device. I need to use the pass device with MTX. This is a library configured as 2 logical libraries, with one LTO-3 SCSI and one LTO-4 SAS drive (yup, connected to the same host but one drive/changer pair is dedicated to an OpenVMS VM):
Code:
sa0 at mps0 bus 0 scbus1 target 10 lun 0
sa0: <IBM ULT3580-HH4 G361> Removable Sequential Access SPC-4 SCSI device
sa0: Serial Number 906880xxxx
sa0: 600.000MB/s transfers
sa1 at mpt0 bus 0 scbus2 target 4 lun 0
sa1: <IBM ULT3580-TD3 93GP> Removable Sequential Access SCSI-3 device
sa1: Serial Number 121023xxxx
sa1: 160.000MB/s transfers (80.000MHz, offset 127, 16bit)
ch0 at mps0 bus 0 scbus1 target 10 lun 1
ch0: <IBM 3573-TL F.11> Removable Changer SPC-3 SCSI device
ch0: Serial Number 00X4U78Pxxxx_LL1
ch0: 600.000MB/s transfers
ch0: Command Queueing enabled
ch0: 23 slots, 1 drive, 1 picker, 3 portals
ch1 at mpt0 bus 0 scbus2 target 4 lun 1
ch1: <IBM 3573-TL F.11> Removable Changer SPC-3 SCSI device
ch1: Serial Number 00X4U78Pxxxx_LL0
ch1: 160.000MB/s transfers (80.000MHz, offset 127, 16bit)
ch1: 21 slots, 1 drive, 1 picker, 3 portals

<IBM ULT3580-HH4 G361>             at scbus1 target 10 lun 0 (pass7,sa0)
<IBM 3573-TL F.11>                 at scbus1 target 10 lun 1 (ch0,pass8)
<IBM ULT3580-TD3 93GP>             at scbus2 target 4 lun 0 (sa1,pass9)
<IBM 3573-TL F.11>                 at scbus2 target 4 lun 1 (pass10,ch1)
Sometimes the robot even unloads the tape and stores it without being instructed.
That shouldn't happen. If the library thinks it needed to do something to avoid damage, the yellow tape icon LED on the front panel should be lit and the LCD should display an error message.
I have a Dell PowerVault 114X with an LTO-5 drive in it and plenty of LTO-5 tapes. I really want to upgrade to LTO-5. The drive is the same IBM series as in the TL4000 and has the correct physical interfaces. However, if I connect the drive to a spare sled and install it in the library, there are two issues:

The library recognises it only as the LTO generation written on the sled (i.e. 3 or 4), not 5
These libraries aren't that bright when it comes to changing the quantity or type of tape drives - hot swap only seems to work for the same type in the same slot it came out of. Sometimes a few reboots will clear it up, sometimes I give up and just reset the library to defaults and let it find the new drive that way.

There is nothing on the sled that identifies the drive - all it does is handle power control and supply to the drive, operate the LED on the sled and connect the drive's loader interface port to the library backplane.

HP does something bizarre (even for HP) where they sell the IBM library but put Tandberg tape mechanisms in IBM trays. All of your library drives need to have the IBM style 7-segment display by the tape slot. A Tandberg drive (regardless of personality) will cause the library to throw a "Wrong drive type" front panel error on power-up.

In theory, you could work around this by getting the library controller card from a HP MSL 4048 and putting that in. When the library tells you there's a configuration conflict and asks whether to use the configuration from the library or the controller card, select the controller card. I have never tried this because I suspect the library will then complain about non-HP drives. I have switched between IBM and Dell controller cards successfully.
The library throws an error after trying to initialise the drive on install or boot - code 'FB 0x' where x is the tape slot
Googling around for the last issue suggests that despite the tape drive physically interfacing to the sled, being in the same series and being Dell OEM, there are some internal changes to the drive that go beyond firmware - a standalone drive is not able to work in a tape library. I found a single reference to setting the DIP switches on the drive - number 5 needs to be ON for a library. Sure enough, the drives I got with it had this set, but setting it on my LTO-5 drive didn't make a difference.
Depending on the LTO generation, the library / standalone personality may be stored in the drive VPD (Vital Product Data) area along with the serial number and OEM flavor. Some drives can be modified with switch 5, some can't.

Going from a library to non-library drive is far more common, which explains the large number of empty sleds available on eBay (and the high cost of drive faceplates).
I've tried updating the firmware on the drives themselves, which made no difference. I know this is an old library, but I see Dell are still providing firmware and their documentation details it working with LTO-8 drives so it's got a pretty long service life.
It still works and is "functionally stable" (meaning don't expect any firmware updates). Dell changed that library from the IBM flat back face to the Dell "bunch of shotgun holes" faceplate style and they didn't want to change again to their new "Darth Vader" corporate faceplate standard. Also, IBM discontinued the underlying machine type (3573) and replaced it with a new version (6173) and Dell didn't want to stock a completely different set of spares.
The fact that I can't even get it to work in a fairly stock configuration (fully initialised OEM LTO-3 drive and LTO-2 tapes) makes me wonder if there's a fault with this library that I can't fix.
I would suggest trying with a single drive in the lowest position and doing a full factory reset. Tape some cardboard over the open back panel slots where you removed the other tape drive(s) to ensure proper airflow.

The library is pretty good at logging errors. Once you have reset it, you can clear the logs (if they weren't already cleared) from the web interface so you know anything in there is from your most recent testing. The service password is SER001 if you didn't know that already.
As a last point, I know this library is a rebadged IBM TS3200. I noticed on eBay there's a number of HP LTO sleds for sale and they look identical to the ones I have - also a rebadged library for HP? Would an HP LTO-5 sled work in a Dell library? Can I flash some firmware to make this thing work as I want it?
See above - HP uses Tandberg mechanisms in their version of the library and I don't know if IBM mechanisms would even be recognized by a HP-personality library.

You can probably cross-flash the controller card somehow, but they're inexpensive enough on eBay that I never bothered to figure out how. And both the IBM and Dell personalities are happy with either IBM or Dell drives, or even a combination.

Once you get it working, you might check out the "Backups" section of my write-up of my RAIDzilla 2.5 here. A library-aware backup shell script is included there.
 

gargravarr

Member
Jul 1, 2021
36
1
8
Wow, there's a lot of info here, thank you very much for your post Terry!

I moved the LTO-5 drive to the bottom slot. On booting the library up, it still showed an LTO-3 drive there and error 'FB 01'. I just rebooted it and the library is once again scanning all slots. Wonder if it'll make any difference to shut down and reboot a few times.

All my drives are IBM 3580 series AFAICT - they do indeed all have the 7-segment displays.

Interesting, the robot just stopped moving and the library now shows Media Attention, which is expected with the LTO-2 tapes still in slots, but the drive itself shows Initialised and Ready. It still shows as an LTO-3 drive but the library shows no error against the LTO-4 tapes I have loaded, just the LTO-2 ones. This is promising.

Whether or not I'm able to write anything to the tape remains to be seen.
 

Terry Kennedy

Well-Known Member
Jun 25, 2015
1,140
594
113
New York City
www.glaver.org
All my drives are IBM 3580 series AFAICT - they do indeed all have the 7-segment displays.
Ok. Try connecting them to the host via their SAS, FC or SCSI cable and see what they report themselves as. For example, my drives report:
  • IBM ULT3580-HH6 KAJ9 - IBM personality LTO6 HHv2 SAS drive
  • IBM ULT3580-HH4 G361- Dell personality LTO4 HHv2 SAS drive
  • IBM ULT3580-TD3 93GP - Dell personality LTO3 FH SCSI drive
It is possible that what the drive is reporting is different from what the library is reporting. That generally happens if a sled was swapped without the library noticing the change in type. As I said, sometimes I've had to reset the library to factory defaults to get it to notice (or forget about) a drive.

Interesting, the robot just stopped moving and the library now shows Media Attention, which is expected with the LTO-2 tapes still in slots, but the drive itself shows Initialised and Ready. It still shows as an LTO-3 drive but the library shows no error against the LTO-4 tapes I have loaded, just the LTO-2 ones.
Does the inventory (in the web interface) show the correct labels (including media type) in the correct slots, with all the others shown as empty?

Have you checked for a sticking picker as I suggested (and confirmed the library is on feet if on a table top or shelf)?

If you updated the firmware from something very old (I forget the point at which this was changed) you can get a media attention by not having a properly labeled cleaning tape in the dedicated cleaning slot. If you don't have such a tape, make sure you have "Auto clean enabled" un-checked and "Active slots: 45" in Configure Library / General.
 

gargravarr

Member
Jul 1, 2021
36
1
8
So the LTO-5 drive registers as
IBM ULTRIUM-HH5 H971
I updated the firmware myself using the Dell update packages. Took a while to figure out which one had the LTO-5 firmware in it but it updated to this revision successfully. This shows in dmesg and in ITDT.

There's still no
LUN exposed for the robot when hooked to my NAS, nor does ITDT detect any kind of library. This is pretty weird as it DOES show when connected to the other server via a different card. Wonder if the ASR-7000 series SAS card doesn't support multiple LUNs? Or I need to change a setting?

Every time I start the library up, however, the thing eventually shows 'Drive Logged Out' whenever I try it with the LTO-5 drive. I did a factory reset as well; it doesn't seem to have helped. Also, the 'Service' user does not work with the password 'SER001'.

There is a cleaning tape in slot 48 which is properly detected. The labelled tapes show up correctly, as do the empty slots, it just takes a long time. The library is in the bottom of my rack on the original rails/shelf so it should be flat.
 

Zombielinux

Member
Jun 14, 2019
71
21
8
Have you reflashed the controller firmware to the latest? Sometimes even just a reflash will get things moving again.
 

gargravarr

Member
Jul 1, 2021
36
1
8
Have you reflashed the controller firmware to the latest? Sometimes even just a reflash will get things moving again.
I upgraded the library controller firmware to latest, yes. I could try again, I guess. I'm about to try putting the LTO-4 drive in and see what happens.