SAS controller - 50TB Raid XX - advice for Noob.

[11Blade]

I'm looking for some advice/view points. Just for background. Building a home rack setup to store and work on some large data set.

I bought a used Supermicro 16bay CSE836 - 16bay SAS server case that had a SM X7DCL motherboard with a single processor, 8Gb RAM and a SAS controller (LSI 3081ER) - I flashed newest FW onto the LSI controller. The backplane of this case is a BPN-SAS2-836EL (listed as a 6Gbps backplane).

Currently in slots 0 + 1, I installed pair of Seagate 15k 146Gb drives and managed to install ubuntu 16.04LTS without much difficulty.

Slot 2, I installed a WD 3Tb RE SAS drive - which when installed, lights up solid blue - not seen by LSI configuration utility.

Plan
I would like to populate the remaining 13 drives with 4 or 5 TB SAS drives to store approximately 50TB of lab data at some level of redundancy.

Questions
1. Since the LSI part does not support >2 Tb drives - should I expect the controller to see the drive in the config startup utility as a 2TB drive - could this brand new drive be bad out of the box(since it is showing solid blue when plugged in)?

2. I clearly would like to choose a more appropriate controller that will handle the 16 drives - Can you guys recommend a cost-effective alternative that gets me to 6Gbps and manages the 16 drives/backplane. I am not queasy about flashing firmware/bios

3. RAID question. The slot 0+1 drives will be my system drives, the slot 2 drive is a scratch space drive for working on data pulled from the larger dataset. The data in question is about 45TB. What is the RAID recommendation to protect against data-loss if there is a drive failure. I have the data on Tape backup at another location so I am guarded against permanent data loss. How would you configure the raid for decent performance but primarily reliability. Should I do this at the controller level or do it in the OS instead?

I been primarily a windows user for a very long time, but recently been using tools for research in linux and now just begun scratching the surface of both server hardware, storage and the OS. Please excuse my ignorance.

This site has been a great resource for bargains and knowledge. Your input and opinions would be greatly appreciated.

11Blade

 

[pricklypunter]

While your backplane is 6Gbps capable, your controller isn't. It will likely only ever see disks up to 2TB in size. If you are in doubt about your disk, try plugging it into one of the onboard SATA ports to test it. They are only 3Gbps ports, but should allow you to test disk functionality.

For controller replacement, you'll be wanting an IBM M1015 or Dell H200/310 cross flashed with LSI 2811 IT firmware, or if your lucky a cheap 16 port card, but they are spendy, even secondhand. You'll then have direct disk access to all 16 disks. If you go with SAS disks, you can use both ports for Multipath which provides HA and gives you a healthy bump in bandwidth.

As for raid, if you have anything in the way of a modern CPU at your disposal, then deal with your redundancy in software. I would say go with ZFS, as it's the more mature product with modern features, but others have a soft spot for BTRFS or ReFSv2 with are also copy on write filesystems etc, even MDRaid/ LVM is pretty reliable, albeit lacking a few things the next gen file systems offer. Each filesystem has it's foibles, obviously, so you'll have to evaluate each to see if it's a good fit for your use case. If you do decide on ZFS, your obvious OS choices are FreeBSD/ OpenZFS or Linux/ ZoL. If you go the Micro$oft route, you have Storage Spaces/ ReFS to help you get things in order.

If it were me, I would be looking to replace that mainboard with an SM X9 series board with a couple of decent CPU's in it. The E5-2670 is popular just now, readily available, relatively inexpensive and pack good bang for buck. Toss in 64-128GB RAM and you'll have a really nice box that will handle just about anything you throw at it

 

[11Blade]

Pricklypunter,

Thanks for the response. Couple of followup questions..

The SAS backplane has 16 slots and I assumed since the controller can address 256 SAS devices that it can handle all 16 slots on the backplane - This sounds like a completely incorrect assumption since you mentioned I should find a cheap 16 port card. (That's distressing!!) - Essentially I was sold a system with a card that could only work with 8 drives

I do plan on using SAS drives just for performance + bandwidth plus reliability..

I was trying to keep system price down because I have to drop money on 10-13 more drives to get 50-70Tb. You would not recommend hardware RAID for redundancy and drive failure backup? I figured a 100 dollar RAID card would save me from spending money on beefier processor/MB combo. (the current processor and ram suffice for my work, the more intensive computational stuff I can offload to a different server.

So.

1. ZOL software raid better than hardware Raid?
2. Can I address all 16 bays of my backplane with a LSI- 3081ER?
3. If not, can I use a pair of H200/310 or M1015 to address the whole back plane (16 drives)?
4. what if I want to attach a SAS JBOD to this to copy the data onto my new array - would I need yet another SAS controller/HBA to provide an external port?

regards

11Blade

 

[pricklypunter]

Ok, I'll start by apologising perfusely. I was half asleep and busy doing loads of other stuff when I responded to you last night and lost the plot half way through. I kept coming back and adding to/ editing my response before posting. For some reason, even though I understood which backplane you had to start with, I went ahead and answered you as if you had one of the direct attached backplanes. Hence my mentioning the need for more than 1 HBA or a 16 port version, although a 16 port card brings options to the table that the smaller cards can't. Also, I failed to qualify why I would change the mainboard you have, before mentioning the HBA cards in my response. Ok, with the humble pie out the way, I'll try and answer you again in a more sensible manner this time

The backplane you have is a 6Gbps capable extender backplane. ANY suitable SAS HBA that can address 16+ endpoint devices will be able to handle all 16 bays in your chassis with a single 8087 port. You can use both ports on the card for multipath/ HA type scenarios, if you are using SAS disks. SATA disks don't support multipath. As for bandwidth, unless you plan on using SAS SSD's in your arrays, having a top notch 12Gbps card on a 8 lane PCI-e 3.0 bus is overkill and I might add, a waste of your money. Totally disregarding for a moment that your x7 mainboard would not handle the bandwidth required to keep up with it.

You will see a benefit using 6Gbps capable cards though with SAS disks and a healthy bump in performance when using SATA disks. The card you mentioned having, the LSI3081e-r, is only capable of 3Gbps and will be using PCI-e 1.1, which is fine in your x7 mainboard, but if you later move up to a more modern board, anything else on the bus that you plug that card into, will also have to run at PCI-e 1.1. So you would in all likelihood be replacing it at your first upgrade.

Taking your questions in order:

ZoL, or more to the point, is ZFS generally better than hardware raid? I would say that depends. If you were using a really old CPU that can't keep up with parity calculations, then no, you would be better off using a hardware raid implementation. But just about any remotely modern CPU built in the last 5 years, perhaps a bit further back, would just take that in its stride without breaking a sweat. So if you have one, I would say ZFS is a good choice, if it fits your use case well. Along with redundancy, it also brings with it a set of next gen filesystem features that are hard to say no to, copy-on-write, check-summing etc, but there are other robust filesystems out there also worthy of consideration. Filesystem choice is like a game of snakes and ladders, they all have upsides and downsides. Careful consideration should be given to what you have available to you, what you have experience with, what fits better, i.e has more upsides for your use case and how brave you are trying something completely new to you

Will the LSI 3081e-r card address 16 endpoints? Sure, but at only 3Gbps you are limiting your disk performance. To be fair here, this is not only a limitation of the 3081e-r card you have in front of you. In this case, your mainboard is also only capable of sucking 250Mbps per the PCI-e 1.1 bus specification, so it's a double whammy for you I'm afraid. The only way you will get the best out of your disks, is with a more modern combo of HBA and mainboard. I should have mentioned your mainboard limitation in my last post before prattling on about possible HBA choice.

Using either the H200/310 or an M1015 would get you to 6Gbps and make the best use of your disks, but your mainboard will only do 250Mbps with a wind behind it, so it's a bit of a moot point if you can't upgrade the mainboard. But to answer your next question along with this one, if you want to use 16 SAS/ SATA disks in your 836 chassis, only a single port on the card is required, because the chasiss has an expander backplane. If you want to also make use of i/o multipath, you will need SAS disks and at least one HBA using both ports. To address external storage, you will need another card, preferably one with external ports, rather than having to mess with port adapters etc. However, if you were lucky enough to have a 16 port card, you could use a port adapter for your external jbod and the other two ports for your chassis backplane.

Sorry about the confusion, hopefully some of what I just wrote is helpful

 

[wildchild]

Kind of missing the obvious here.
Zol = freebsd ?
If yes then i agree, but if not...
I'd go with either freebsd , but better yet omnios .
Latter has better cifs intergration , crossbow has a battle tested iscsi/nfs implementation and zfs is native to it

 

[TuxDude]

Just want to clarify a bit regarding multipath SAS, and increasing your bandwidth to your drives (assumes a controller + MB upgrade to at least PCIe 2 / SAS2 speeds, eg a M1015)

First off, the OP is not clear on exactly which backplane is installed in the server - specifically whether it is a BPN-SAS2-836EL1 or BPN-SAS2-836EL2 - if you want to use multipath SAS you will require the BPN-SAS2-836EL2 model, which has two expander chips on it, and 6 SFF-8086 ports (3 wired into each expander chip) Using dual-ported SAS drives, one port of each drive is wired to each expander, which then both need to be connected to one or two HBAs - this gives you two complete paths from the HBA(s) to the drive and is mostly only useful for redundancy. Using multipathing will also cause every disk to show up twice in your OS, which will require a bit of extra work to turn back into a single device (some OS's handle this better than others). My recommendation is to stay away from multipathing unless you really need the extra redundancy.

If all you want is more bandwidth for your drives, you do NOT want to go the multipath route. SAS supports bonding connections together to increase bandwidth at a very low layer, and will take care of it automatically for you if you just make the physical connections. The best example is the single SFF-8086 cable that everyone around here is familiar with - it contains 4 separate SAS lanes, but if you connect it between an HBA and an expander, you get a 4x6G=24G link for all of the drives on that expander to share - regardless of whether you are using SATA or SAS drives, etc. You can put a second SFF-8086 cable in there to double that, creating a 48G link from a 8-port SAS2 HBA to a SAS2 expander - now you have enough bandwidth back to the HBA for quite a few SATA3/6G SSDs even - all with only a single expander chip and no multipathing complexity.

And with these Supermicro expanders, there is still that 3rd SFF-8086 port on the backplane (2 more if you are doing this all multipathed, 3 ports per expander chip). I've never tried putting a third 4-lane SAS connection back to an HBA as I don't have a 16-port SAS controller to test it with, but in theory it would give you a 72G link between the two devices. But what I have tested is using that extra port to daisy-chain off to even more devices, either directly to drives with a breakout cable, or to an additional SAS expander (possibly routed to another chassis even).


In the end, I would recommend grabbing a cheap M1015 to stick in the current system, just for the >2GB drive support. Thats going to be the cheapest way to move forward with bigger drives, and if performance is good enough for you then just leave the rest as-is. If you need more bandwidth then also upgrade the MB/CPU/RAM to something with at least PCIe 2

 

[TuxDude]

Kind of missing the obvious here.
Zol = freebsd ?

Nope... ZoL = ZFS on Linux

 

[gigatexal]

make it raid - 0 and hold on to your hat! haha

just kidding

 

[pricklypunter]

Lol @gigatexal

Good catch Tux, my brain just assumed it was the el2 backplane he had. Also, you are correct, bonding is what he needs for bandwidth, not Muitipath. I swear my brain is not switched on just now

 

[wildchild]

Nope... ZoL = ZFS on Linux

Of course .. i know.
But correct me if i am wrong , but i thought zol isnt in the kernel. Therefor less performant than in freebsd or illumos

 

[pricklypunter]

It is not in the Linux Kernel, and very unlikely to ever be, at least officially, unless the powers that be sort out their licensing differences. IF it is less capable of performing as a result of not being baked right into the Kernel, I can't say I have noticed the difference. Now if having ZFS baked into the Kernel is a requirement for some other reason, perhaps native encryption for example, then you would be better off running Solaris to begin with

 

[fractal]

You said that you are primarily a Windows user.

Does your application that deals with the data sets run under Linux?

Or are you going to export the data volume over the network and use it on your Windows machine?

If the latter, would you consider installing FreeNAS / NAS4Free / OpenFiler / ... some NAS appliance software that boots from USB stick?

That plus an 8-16 port SAS card should give you access to your large data sets from your windows machine with a minimum of work assuming you have the correct backplane as discussed by TuxDude. Just one thing -- the IBM M1015 on eBay are going for more than the LSI equivalent so do shop around.

 

[TuxDude]

Of course .. i know.
But correct me if i am wrong , but i thought zol isnt in the kernel. Therefor less performant than in freebsd or illumos

It is not part of the upstream linux kernel due to licensing issues, but it is implemented as a native kernel module - its not going through FUSE or anything like that, and performance should be close to other implementations. It might be more RAM-hungry than other implementations as it doesn't share the normal linux filesystem cache - it does its own ARC thing.

 

[PigLover]

The ZoL license issue is actually starting to break down too. Canonical has included ZoL as part of their Distro starting in Ubuntu 16.04. Its not clear that this is compliant with the license ZFS is released under - but the only group with standing to challenge them (Oracle) has publicly stated they have no intention of mounting a challenge.

Sent from my SM-G925V using Tapatalk

 

[gigatexal]

"not in kernel ... will be slow" = optimising before a perf problem presents itself is a no-no. Don't do that.

ZFS on Linux compiles to a kernel module like anything else, and Ubuntu, as has been said, as of 16.04 makes it stupid easy to run.

 

[wildchild]

It is not in the Linux Kernel, and very unlikely to ever be, at least officially, unless the powers that be sort out their licensing differences. IF it is less capable of performing as a result of not being baked right into the Kernel, I can't say I have noticed the difference. Now if having ZFS baked into the Kernel is a requirement for some other reason, perhaps native encryption for example, then you would be better off running Solaris to begin with

Been running omnios for the last few years now.. rock solid, fast...
But initially had a choice between zol and illumos.
Ended up choosing omnios still happy with that choice

 

[pricklypunter]

Been running omnios for the last few years now.. rock solid, fast...
But initially had a choice between zol and illumos.
Ended up choosing omnios still happy with that choice

I seem to remember something about Omnios/ Illumos having kernel memory issues that affected ZFS ARC, has that been fixed now?

At the end of the day, it's horses for courses...and there's lots of both to choose from

 

[gea]

The basic ZFS code is identical on all Open-ZFS platforms as they all use the Opensolaris code that was forked to Illumos but Illumos is more or less the reference platform now where improvements were added quite often first. Last year there was an improved L2Arc code with a bug. It was discovered shortly after it was included in OmniOS and then fixed within a week.

I am not aware of an Arc problem on any OpenZFS platform.

 

[pricklypunter]

Yea, I just remember it had something to do with kmem hogging memory slabs in the kernel that wasn't being released and was putting a stranglehold on the available memory for the ZFS ARC among other things. It wasn't a problem with the ARC as far as I know. I don't use either OS, so haven't really paid much attention to them, can't even remember now where I seen the article that referred to it