Do I need a SAS card?

[Jonnyswboy]

Sorry if this is the wrong section, I am new to this.

What I have:
SuperMicro X10DRU-i+
BPN-SAS3-826EL1 12-Bay SAS Backplane
12 Sata SSD's

I connected the backplane to the motherboard but I am starting to think that I need a SAS add in card because the Supermicro only mentions a SATA controller on the product page. If so what card do I need? The backplane has 4 HD Mini-SAS connectors on it, but the server only shipped with two SAS cables. Do I need a card with 4 connectors to use all 12 drive bays?

Do I need to spend this much? LSI 9305-16i PCI-Express 3.0 x8 SAS Host Bus Adapter - Newegg.com

Thank you!

 

[gea]

Each miniSAS cable is good for 4 Sata/SAS disks.

Your main options to connect the backplane:
- use a 16i SAS card ex 9305-16i and connect both with 3 x miniSAS cables for 12 disks

You can also use regular Sata mainboard port to connect such a backplane. You then need a "reverse breakout cable sata->miniSAS"

 

[i386]

Yes, you will need a sas hba/raid controller for this single expander backplane: the sata protocoll doesn't "understand" how to talk to sas expanders.
An 8 port hba/raid controller will work.

 

[Jonnyswboy]

Thank you for the replies. Can anyone recommend a good cheap 16 port SAS raid card?

Can I use two 8 port HBA cards and still do software raid under proxmox?

 

[BlueFox]

Each miniSAS cable is good for 4 Sata/SAS disks.

Your main options to connect the backplane:
- use a 16i SAS card ex 9305-16i and connect both with 3 x miniSAS cables for 12 disks

You can also use regular Sata mainboard port to connect such a backplane. You then need a "reverse breakout cable sata->miniSAS"

This is wrong with the backplane you have. You cannot hook it up to SATA ports and you would be wasting your money getting a 16 port HBA. Any 4 or 8 port HBA will do (all 12 drives would still be hooked up). If you are running hard drives, you will not be bandwidth limited even with a 4 port HBA, so no need to spend more money on something with more ports.

You also will not be able to use multiple HBAs with the backplane since it has a SAS expander integrated.

 

[Jonnyswboy]

This is wrong with the backplane you have. You cannot hook it up to SATA ports and you would be wasting your money getting a 16 port HBA. Any 4 or 8 port HBA will do (all 12 drives would still be hooked up). If you are running hard drives, you will not be bandwidth limited even with a 4 port HBA, so no need to spend more money on something with more ports.

You also will not be able to use multiple HBAs with the backplane since it has a SAS expander integrated.

So I could get a 4/8 port HBA card and still use all 12 drives and any version of raid I want using software under proxmox. If that's true I would save a lot of money. I thought one SAS port = 4 sata ports though? Could you recommend me the cheapest solution for my use case?

Thanks a lot for the help!

 

[BlueFox]

Yes, a 4 port SAS HBA will let you use all the drives. The backplane has an integrated SAS expander. Think of it like a USB hub for disks.

A multilane SAS port consists of 4 lanes, which among other things can be used to attach 4 SATA devices, however isn't restricted to that. With a 4 port HBA, you would have 4 lanes between it and the SAS expander. Bandwidth would be shared between 12 drives, but 4.8gb/s combined is hardly an issue with hard drives.

As for cards, an LSI 9341-4i can be had for $75 on eBay. If you want cheaper, you could get an 8 port 6gbit HBA like the LSI 9211-8i for about $40. You would need to use 2 cables to get the same amount of bandwidth then.

 

[Jonnyswboy]

Yes, a 4 port SAS HBA will let you use all the drives. The backplane has an integrated SAS expander. Think of it like a USB hub for disks.

A multilane SAS port consists of 4 lanes, which among other things can be used to attach 4 SATA devices, however isn't restricted to that. With a 4 port HBA, you would have 4 lanes between it and the SAS expander. Bandwidth would be shared between 12 drives, but 4.8gb/s combined is hardly an issue with hard drives.

As for cards, an LSI 9341-4i can be had for $75 on eBay. If you want cheaper, you could get an 8 port 6gbit HBA like the LSI 9211-8i for about $40. You would need to use 2 cables to get the same amount of bandwidth then.

So with the SAS9211-8i would that bottleneck my 12 SATA 6GB SSDs using two cables? What would to total bandwidth be in that case? And just to confirm I could use any version of RAID I want in proxmox even with an HBA card?

 

[BlueFox]

It would, but not by much, and only on sequential reads that involve all the drives concurrently. You would have 4.8gb/s of bandwidth between all of the drives. You are free to use any software RAID you want as the drives will show up as 12 individual disks that you can do with as you please.

 

[Jonnyswboy]

It would, but not by much, and only on sequential reads that involve all the drives concurrently. You would have 4.8gb/s of bandwidth between all of the drives. You are free to use any software RAID you want as the drives will show up as 12 individual disks that you can do with as you please.

I think I will go with that then. Last question. My backplane has 4 HD Mini-SAS ports on it, does it matter that I am only using 2 of 4 ports? And would it matter which two I plug into the HBA card so the expander works?

I really appreciate the help, I learn best hands on so I stupidly dropped a lot of money on this stuff with embarrassingly little knowledge.

 

[nivedita]

So I could get a 4/8 port HBA card and still use all 12 drives and any version of raid I want using software under proxmox. If that's true I would save a lot of money. I thought one SAS port = 4 sata ports though? Could you recommend me the cheapest solution for my use case?

Thanks a lot for the help!

One SAS port = 4 SATA ports only if the drives are direct-attached. The backplane you have has an expander chip, which is basically like an ethernet switch for SAS. It connects to all 12 drives on the backplane, and has 2x4 "upstream" and 2x4 "downstream" links additionally. The upstream links connect to your HBA, connecting both just gives you more bandwidth, which is unnecessary for HDD's since they generally won't exceed a 1x4 link (24Gbps). The downstream links are for connecting a second backplane (like an external JBOD).

 

[nivedita]

I think I will go with that then. Last question. My backplane has 4 HD Mini-SAS ports on it, does it matter that I am only using 2 of 4 ports? And would it matter which two I plug into the HBA card so the expander works?

I really appreciate the help, I learn best hands on so I stupidly dropped a lot of money on this stuff with embarrassingly little knowledge.

Read the manual for the backplane for how to connect. It does matter which ports you use, two are for HBA and two are for connecting additional downstream JBODs.

 

[nivedita]

It would, but not by much, and only on sequential reads that involve all the drives concurrently. You would have 4.8gb/s of bandwidth between all of the drives. You are free to use any software RAID you want as the drives will show up as 12 individual disks that you can do with as you please.

This is actually a question I've had -- with SATA drives attached to an expander, does the HBA/expander (assuming both HBA and expander are SAS3) link still operate at 12Gbps or does it get limited to 6Gbps?

 

[Jonnyswboy]

One SAS port = 4 SATA ports only if the drives are direct-attached. The backplane you have has an expander chip, which is basically like an ethernet switch for SAS. It connects to all 12 drives on the backplane, and has 2x4 "upstream" and 2x4 "downstream" links additionally. The upstream links connect to your HBA, connecting both just gives you more bandwidth, which is unnecessary for HDD's since they generally won't exceed a 1x4 link (24Gbps). The downstream links are for connecting a second backplane (like an external JBOD).

This is a great explanation, thanks.

Can you verify the logic behind my current plan:

LSI SAS9341-4i
I would be limited to 8000MB/s by PCIe 3.0 8X so for 12 drives it would be 666MB/s per SSD. (perfect)

This is a hardware RAID card I believe, so I would have the option of making 1 big hardware raid and playing with it in Proxmox
or
do JBOD+ZFS in Proxmox and doing a software RAID.

Are there any disadvantages using HW raid over software raid with this card? I have 2x E5-2680 V4 totalling 56 threads with 128GB DDR4.

 

[nivedita]

The PCIe slot will not be the bottleneck. 4-lane SAS3 will be 4x12=48Gbps=4.8GB/s, and I think but not 100% sure that it will actually run at 4x6=24Gbps=2.4GB/s with SATA drives. Some SAS expanders can buffer data from 6Gbps devices to resend it at 12Gbps to the RAID card, but I'm not sure if the expander chip in the SMC backplane has that capability.

 

[Jonnyswboy]

The PCIe slot will not be the bottleneck. 4-lane SAS3 will be 4x12=48Gbps=4.8GB/s, and I think but not 100% sure that it will actually run at 4x6=24Gbps=2.4GB/s with SATA drives. Some SAS expanders can buffer data from 6Gbps devices to resend it at 12Gbps to the RAID card, but I'm not sure if the expander chip in the SMC backplane has that capability.

So I would only get 200Mb/s per drive? I think I'll try to get a card with two SAS ports then to double the bandwidth.

 

[kapone]

Some SAS expanders can buffer data from 6Gbps devices to resend it at 12Gbps to the RAID card, but I'm not sure if the expander chip in the SMC backplane has that capability

It doesn't.

 

[kapone]

So I would only get 200Mb/s per drive? I think I'll try to get a card with two SAS ports then to double the bandwidth.

No.

You're confusing mbps vs MBps vs GBps...

A SATA III device is 6gbps, which is ~600MBps. With a 12 port expander using 2x SAS links (each would negotiate at 4x6gbps), you have a total of 48gbps (which is ~4.8GBps) of bandwidth between the expander and the HBA. With a single SAS link it'd be 24gbps, which is ~2.4GBps.

So...with 12 disks...the math works out to:

Single SAS link (4x6gbps) = 2gbps (which is ~200MBps) per drive.
Dual SAS links (8x6gbps) = 4gbps (which is ~400MBps) per drive.

 

[itronin]

What SSD's do you plan to use?
If you are really concerned about effective throughput and have not purchased the SSD's you might look at used SAS3 SSD's instead of SATA.

You mentioned proxmox and so I am assuming some virtualization - but is there a predominant workload that will be utilizing your storage?

More details may get you some more finely targeted recommendations.

 

[Jonnyswboy]

What SSD's do you plan to use?
If you are really concerned about effective throughput and have not purchased the SSD's you might look at used SAS3 SSD's instead of SATA.

You mentioned proxmox and so I am assuming some virtualization - but is there a predominant workload that will be utilizing your storage?

More details may get you some more finely targeted recommendations.

They're Sandisk x300 128Gb sata drives. I plan on running pfsense, Plex, some game servers, FTP, Web hosting. I just want the most speed possible
for future use cases.