Lots of Optical Drives

[Winter]

I am building a system for my students and I to do projects with. One of the data sources we will be moving into the system currently resides in 9,000 mixed optical disc formats.

To help speed this up Ive gotten 20 blu-ray drives. SATA port multipliers kind of work to run these, but are not reliable enough.

Can anyone recommend controller card that can handle ATAPI or if anyone has any suggestions Id be very grateful.

 

[pricklypunter]

I've never used any of these cards, so take this with a pinch of "suck it and see". There are lots of PCIe 20 port SATA cards on amazon, but they are pricey. It's cheaper using 2x 12 port cards instead, if you have the slots available. I have no clue if they support ATAPI devices or not, but if they are true SATA ports, they should

 

[MountainBofh]

would a LSI HBA sas card handle ATAPI devices ?

 

[Stephan]

Jesus Christ and a cookie 9000 optical discs. Might get three Inter-Tech Y-5508 cases for the drives and three el cheapo IBM M1015 LSI controllers. Case can hold 8 external 5.25" drives. Controller is really the cheapest of the cheapest, but totally sufficient. Do not mess with port splitting, never works alright and not worth the pain when a controller is like 10 bucks.

Flash to IT mode like so: https://www.servethehome.com/ibm-serveraid-m1015-part-4/

For the SFF-8087 connector you need a 1 to 4 forward (not reverse! looks the same) breakout cable so 5 in total for 20 drives. Buy one first and test, if brand works, buy the rest.

To save money and if you have some serverish board already with at least three PCIe x8 slots, like an old Broadwell Xeon board, you might also try a more ad-hoc approach: Get a Streacom BC1 V2 Open Benchtable, mount a 800W PSU below it, mount the board on top and install the three M1015 on it. The bench table comes with small 4 cm holding clamps. Use these to mount a 120 cm fan with 1500 rpm or more on the long side to blow straight at and through the M1015s. To cool away the 30 watts those will generate. Put drives in heaps, fix somehow to a table so they don't wander off, and connect them up to PSU and the controllers. Heaps probably not more than four drives per heap and with foam between table and each drive to cushion vibrations this will generate.

And then let the discjockeying commence...

Might even write a script that polls a drive if ready, if so tries to mount it and copy off everything to a directory named DATE-VOL-LABEL, then unmounts and ejects it again. Write some logs to check for errors. So you will just sit there, and whenever a drive ejects, you stop sipping on the pina colada and put in the next disc.

 

[Peter Blanchard]

CD/DVD duplication devices can turn up for cheap. In themselves probably not what you need but the cases might be useful to you.

I remember the time it took me to rip a couple of hundred CDs. 9000 disks? I shudder to think how long that might take.

 

[Stephan]

I calculate around 100 hours, or 10 days.

 

[Peter Blanchard]

Something that occurred to me - how good are Blu-ray drives at reading older format disks? I can remember situations when DVD-writer would fail to read damaged CD but a CD drive had no problem.

I've been binning PATA optical drives today.

 

[drdepasquale]

would a LSI HBA sas card handle ATAPI devices ?

Optical drives rarely work on LSI SAS cards

 

[Stephan]

https://www.reddit.com/r/DataHoarder/comments/fchhlo
Should work, if flashed to IT mode that is. Which I mentioned. If in doubt, start with 1 controller and 1 splitter cable as proof of concept. Probably wise anyhow. Of course the blu ray drives might act up and refuse to play nice with the LSI SAS controller. In that case you may want to look at JMB585 PCIe cards with around 5 ports.

 

[rtech]

If the LSI card does not work

Get some old motherboards with maxed out SATA out and insure the extra slots are not handled by LSI but by intel controllers
For example my Z420 has 10 sata ports no LSI here

 

[Winter]

First off. Thank you to everyone who has replied so far.

I've got power and cases and such. I actually have a 18 port external duplicator case that I used for an older version of this.

This newer version is 20 slimline drives mounted paired in a rack chassis with 10 5.25 bays. I've got a dedicated 850 watt power supply on this case with a loopback cable so when I turn the power on at the switch it turns right on.

First I tried what worked on my last version of this (15x full height dvd drives) which was 3 sata port multipliers hooked to 3 of these USB3.1/3.0 To ESATA (6Gb) Adapter,JMicron Chipset Port Multiplier, eS3U31 | eBay which worked beautifully for years.

The new one I tried 4 multipliers and 4 of those adapters which kept dropping out, when turned out to be the bd-rom drives higher data rates were blinking the adapters.

I then put in a motherboard that handled port multiplication and removed the usb step and it sees all the drives, but if any 1 drive has a read error that set of 5 will cycle as a group and you have to restart 5 reads.

Then I tried these https://www.amazon.com/dp/B09KDLKYRN?psc=1&ref=ppx_yo2ov_dt_b_product_details

Which ran into a similar issue.

Then I tried two of these Glotrends SA3112J. But I think these run as a multiplier as well.

My next attempt will be 20 of those sata to usb adaptors, but I'm going to run 1 drive to an adapter, and then each adapter gets a dedicated slot on a StarTech pexusb3s44v since that card has a dedicated usb controller for EACH port on the card and doesn't share the line.

If that bombs I was going to try an older AsRock X99 Extreme11 board since it has 18 sata ports already. But I'm trying to avoid just building a whole computer for just this since I've got a PowerEdge R720 that can handle this work in the background already if I can get this addon set built for it. I've also got an R920 that I'm rebuilding for my students AI work that I can lean into as well if needed

Based on the recommendations so far:

The IBM ServeRAID suggestion. - I've got LSI9211 cards that didn't work, but I'll hunt down a 9207 as well and give it a try.

In that case you may want to look at JMB585 PCIe cards with around 5 ports. - I've tried the JMB575 but I'll try this as well thank you.

Keep them coming if anyone else has ideas. I'll source parts this weekend and report back on what does and doesn't work.

 

[rtech]

@Winter
That Asrock board accomplishes 18 SATA ports with LSI 3008 controller

 

[alaricljs]

The JMB575 IS a port multiplier. The 585 is a controller.

 

[unwind-protect]

Optical drives rarely work on LSI SAS cards

I run a BR writer on a SAS2008 right now.

 

[Winter]

The JMB575 IS a port multiplier. The 585 is a controller.

Would these work as a set then?

JMB585 Card
JMB585 M.2 Adaptors

 

[alaricljs]

That first card's description is confusing but at least the M.2 slots are keyed as I would expect, they're sata only so no. The 585 M.2 adapters won't slot into that card. You'd need a motherboard supporting bifurcation and one of https://a.co/d/0PRkfe0 or similar.

 

[Stephan]

The big card looks like it has some PCIe multiplier but I can't see PLX logo so probably all m.2 slots sharing a lane. Maybe also chip bugs, would not buy. Only reliable PCIe multipliers at least I know is by PLX.

 

[Peter Blanchard]

My next attempt will be 20 of those sata to usb adaptors, but I'm going to run 1 drive to an adapter, and then each adapter gets a dedicated slot on a StarTech pexusb3s44v since that card has a dedicated usb controller for EACH port on the card and doesn't share the line.

How many of the Startech cards can you get in the R720 etc?
Do they support OS other than Windows? I've had issues using USB 3 cards with Linux boxes.

Something I don't really understand is why USB peripherals are not always recognised by host computer, especially when using a USB hub. Is software or hardware?

Waiting sometimes is enough but in other cases plug/unplug of cables is required. Sometimes only restarting the host works. This morning had to deal with mouse not being recognised by Lenovo laptop running Windows. Have had trouble with webcams with that machine as well.

USB 3 has sufficient bandwidth to use a hub connected to multiple optical drives but it could be a lot of hassle.

Would external USB optical drives be an option to increase ingest?

Would batching work by media type be more efficient? For example, if drives have trouble with, say, CD-RW, sort the problems out roughly at the same time.

If you batch by media type, you could use external CD/DVD drives.

Do you have a CD/DVD repair thingy? I can remember ripping several hundred CDs and it proved invaluable for those that had been treated roughly.

 

[Winter]

How many of the Startech cards can you get in the R720 etc? <- I can go up to 22 cards. I've got an external chassis for extra cards that I use for my NPU cluster for my students and I to train AI models on.

Do they support OS other than Windows? I've had issues using USB 3 cards with Linux boxes. <- For this I use Windows Pro for Workstations. It doesn't handle giant RAM pools as well as a server install, but for this it will allow me to use my whole drawer of imaging software.

Something I don't really understand is why USB peripherals are not always recognised by host computer, especially when using a USB hub. Is software or hardware? <- You can have too many "layers" and larger hubs tends to be several small hubs stacked in the case.

Waiting sometimes is enough but in other cases plug/unplug of cables is required. Sometimes only restarting the host works. This morning had to deal with mouse not being recognised by Lenovo laptop running Windows. Have had trouble with webcams with that machine as well.

USB 3 has sufficient bandwidth to use a hub connected to multiple optical drives but it could be a lot of hassle. <- 2-3 max BD-ROM at full bore. On paper it should handle way more, but 2.5ish seems to be the sweet spot.

Would external USB optical drives be an option to increase ingest? <- I've already got 36 drives. I've got 5 externals if I wanted to add them, but I'm already adding 20 USB controllers to this thing.

Would batching work by media type be more efficient? For example, if drives have trouble with, say, CD-RW, sort the problems out roughly at the same time. <- My scripting is agnostic for this. Image the drive regardless of type, then sort by image type, mount virtually, extract all files, copy them to the storage pool, change out all the discs when the last drive finishes.

If you batch by media type, you could use external CD/DVD drives.

Do you have a CD/DVD repair thingy? I can remember ripping several hundred CDs and it proved invaluable for those that had been treated roughly. <- Other than just aging of the doping in the discs themselves, these have all be kept in acid-free padded sleeves inside a climate controlled steel cabinet set made just for that to keep light off of them.