Two Intel Expanders, Same Chassis. ( RES2CV240 + RES2CV360 )

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

lumdol

Member
Jun 9, 2016
30
2
8
39
I have a 4U chassis which I can fit 36 2.5" (4x 8 + 1x4). I'll be using SATA drives, but may migrate to SAS.

I'll be using ZFS to manage the array. So, the 1x4 will house the SSDs and hot spares

I presently have an Intel RES2CV360 and RES2CV240 which are laid out like this:





From what I gather in other forum posts the 36 port seems to only work with up to 24 disks?

The manual states "
  • Cables from the SAS Expander to the hot swap backplane must be connected in order: A–D for 16-drive configurations, and A–F for 24-drive configurations.
I believe this is for Dual Link plus one outgoing for daisy chaining?

So, my plan presently is:

RES2CV360 H + I to Atto H680 (Two ports)
RES2CV360 G to RES2CV240 F

RES2CV360 A - F for first 24 disks (I figure I will put the SSDs up front in A of the RES2CV360, but not sure it's relevant).
RES2CV240 - A-C for next 12 disks


I am left with RES2CV240 D + E ..... One or both of which I can use for daisy chaining but presently I only need one. However would it make more sense to "dual link" from one expander to the other?

Clarification: By dual link I was primarily interested in redundancy. Previously I had the chassis with two separate SAS cables going into the two ports of the H680. However, it seems some with SAS disks use a single SAS port with two connections? I'm not sure if my references to dual link and multi path are semantically correct.

The expansion / daisy chain link will be for a slower backup unit of 3.5" disks which will not saturate 6Gbps and has its own expander.

Am I missing something? Curious what others have done / what would be the optimal setup.

I would like to get a full idea before ordering cabling as I'd like to keep the chassis as clean as possible and optimize airflow.
 
Last edited:

lumdol

Member
Jun 9, 2016
30
2
8
39
PS, what's odd is that Intel doesn't seem to have a real utility for the 'G' connector in the RES2CV240 in their complete suite of products. They don't seem to ever mention it's purpose.



Except to say in some instances it's used for mounting? Seems like a lot of trouble for a mounting configuration.


Also, conflicting forum information regarding exactly how many drives each of these supports.

From one post:

"The 24 port sas expander will only support 16 drives, the 36 port will only support 24 drives.
The other ports are for connecting to the controller and daisy chaining to another sas expander."

Later in the same post: "supports 20 drives not 16." and so on.

Intel's product brief clearly states:
The RES2CV360/240, powered by an LSI* SAS2x36/24 36 or 24- port SAS-2 expander, enables the connection of up to 32 or 20 directly attached SAS or Serial ATA (SATA) devices, and provides table routing to support connections for up to 1024 SAS addresses. Each expander phy is individually configurable and performs SAS and SATA transfers based on the speed of the host or target at either 6.0Gb/s, 3.0Gb/s or 1.5Gb/s. The RES2CV360/240 supports management functions such as discovery and enclosure management, and can be cascaded to accommodate as many drives as the associated SAS/RAID card allows.

( Link)

Can someone confirm one way or another in practice and if so what the ideal cabling configuration would be? Also, perhaps firmware version is relevant?
 
Last edited:

lumdol

Member
Jun 9, 2016
30
2
8
39
Here is what I’ve found from Intel’s JBOD 2000 storage chassis user manual. Obviously these are set up with redundant connections, but it looks like the same principles can apply.

Seems clear despite contrary forum posts that the 36 and 24 port expanders support 32 and 20 disks respectively.

Still eager to hear others experiences and suggestions for my particular use case.
 

Attachments

lumdol

Member
Jun 9, 2016
30
2
8
39
What’s interesting is that it seems from what I see that they are not linking the expanders internally in the chassis.
 

i386

Well-Known Member
Mar 18, 2016
4,217
1,540
113
34
Germany
"The 24 port sas expander will only support 16 drives, the 36 port will only support 24 drives.
The other ports are for connecting to the controller and daisy chaining to another sas expander."
A 24 port expander could connect up to 20 hdds/ssds and "single link" to raid controller/hba or up to 16hdds/ssds and dual link to raid controller/hba.
A 36 port expander could connect up to 32 hdds/ssds and single link to raid controller, with dual link up to 28 devices can be connected.
However would it make more sense to "dual link" from one expander to the other?
Somebody should try to dual link two expanders and then dual link one of the expanders to a hba or raid controller.
 
  • Like
Reactions: lumdol

lumdol

Member
Jun 9, 2016
30
2
8
39
From what I have read online dual link will only benefit when using SAS drives, not SATA from a performance perspective.



I believe this is a semantics problem again.

Dual path = cable redundancy
Dual domain = hba, expander and cable redundancy

Both can equate to bandwith gains.

While dual domain is apparently limited to SAS disks, dual path is not.

From HP’s documentation:

Dual-domain SAS uses an open port on an HP Smart Array controller that can support dual-domain SAS. The second port on the Smart Array controller generates a unique identifier and can support its own domain. To take advantage of multiple domains, SCSI devices must be dual-ported and connected to pathways in both domains. SAS drives can meet that requirement.
Dual-domain SAS implementations can tolerate failure of a host bus adapter (HBA), external cable, or expander. They can also survive failure in a spanned disk (JBOD) environment or in RAID environments. Dual-domain SAS solutions offer higher reliability, performance, and data availability than SATA solutions.

Note they say performance. However linking in this configuration would not be useful with SATA disks.

Not sure if HP specific drawback or not.

However, not to fear:

Figure 6 shows a topology for connecting one port on a controller to an external storage enclosure. This single controller port incorporates four lanes for a total maximum throughput of 2400 MB/s. The storage enclosure contains an internal 36-port expander that supports cascading an additional enclosure in a 1+1 configuration. Figure 6 also shows a topology for connecting both ports to the enclosure. That doubles the bandwidth, which is important for some workloads.

Page 7, here: https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c01613420

So it appears doable.

The lack of dual connection in the cascade configuration is a limitation of the available expander ports, not the expander or architecture.

As long as you have the ports you’re good to go.

I’m doubtful I’ll be really saturating that but I was primarily curious why anyone would dual link to their first enclosure / expander and then compromise that with a single link to the second enclosure / expander. Or why that seems to be the default config.
 
Last edited:

lumdol

Member
Jun 9, 2016
30
2
8
39
For dual link cascade example see figure 7.5.4 Page 46 of this document:

JBOD 2000 Manual

Seems the link speed I negotiated by the HBA primarily so unless one expander will saturate the 2500MBps there’s no need to use dual path with the expanders.

Dual connections from one expander to the next might also mess with the drive mapping.

The ATTO guide is particularly interesting, showing vastly increased throughput using disks that won’t even saturate the port. THIS IS BIG! I have not heard anyone discuss this performance gain with dual path except when describing total bandwidth.





Adaptive Path Optimization
https://www.atto.com/software/files/techpdfs/TechBrief_Adaptive_Path_Optimization.pdf

Since I’m using the H680 this squarely applies in my case.

I’m unsure of the adaptive path benefits with other controllers.
 

Tiberizzle

New Member
Mar 23, 2017
25
11
3
124
The LSI SAS2x36 in general does not give a **** where you connect things as it is a full SAS switch capable of supporting many topologies, although I haven't bothered to experiment with an Intel branded one and they theoretically could have limited it with custom firmware.

There is a difference between dual link to the controller and dual link to drives. SATA drives will not support dual link between the expander and drive without an interposer (and even then won't provide additional throughput for obvious reasons), but dual link between at the very least any LSI-chipset controller and the expander should provide additional paths (and throughput) to SATA drives behind the expander. I have been using a Chenbro SAS2x36-based expander in this configuration for the better part of a decade now and have previously confirmed all links are used.

One obscure fact about expanders is that since it is a channel switched rather than packet switched protocol SAS3 controllers connected to SAS3 expanders connected to SATA3/SAS2 disks *do not* benefit in any way. In fact the entire SAS expander will operate at the speed of the slowest connected interface in most cases, so watch out for SATA2 disks or disks dropping to a lower negotiated rate due to checksum errors. I believe there are some specific combinations of newer Broadcom chipsets this is not true for but it is true for all SAS2 and most shipping SAS3 products.

Here's a bit more on the crazy topologies the SAS2x36 supports -- Storage: SAS

Note that most of the images there are from LSI's own documentation and it is clearly implied it supports more than 24 drives per expander. There are also e.g. hardware examples of the SAS2x36 being used with 3-4 simultaneous controller connections and nothing but expanders downstream.

In short, I wouldn't worry about what Intel says to connect to what until it isn't working :p
 
Last edited:

lumdol

Member
Jun 9, 2016
30
2
8
39
Yes but one of my primary questions still eludes me.... being, why does all the SAS documentation never show downstream expanders dual linked to one another.... almost always dual link from controller to expander then single link daisy chaining expanders.
 

Tiberizzle

New Member
Mar 23, 2017
25
11
3
124
At the protocol level there's very little difference between the 4 lane wide link and the 8 lane wide link, or for that matter 2,3, 5, 6 or 7 lane wide links.

If it generically supports 8 lane wide ports at all it "could" support wide ports facing anything including other downstream expanders, although the specification is technically not prescriptive on how many circuits of downstream devices connections should be attempted through or how they should be balanced (and I believe some early implementations begin to show suboptimal blocking/contention behavior in larger networks due to this not being fully specified even as all paths are "active", you can think of this as degrading to an active/passive failover configuration in practice due to poor path selection, but also see below which addresses this in part). There would be very little technical reason for a vendor not to support 8-lane wide ports between expanders if they already support 8 lane port widths for initiators (controllers) and wide ports at all for targets (i.e. dual link SAS disks). You have to essentially do extra work to prevent it from functioning from that position.

SAS as noted earlier is a circuit / channel switched network, so for larger networks all real world and recommended topologies resemble a Clos network:


This maximizes the number of possible non-blocking paths that can be simultaneously open and is provably the optimal configuration. Any other topology will permit fewer combinations of simultaneous paths to be open due to :math:

Nonblocking minimal spanning switch - Wikipedia

Internally each SAS expander will resemble this diagram in its implementation of the internal cross bar switch, and to continue to expand the network with maximum path diversity the layers must also be interconnected in this manner. It might be helpful to view the SAS expander as a single layer of the network implemented in ASIC.

Additionally, in simpler configurations where the number of simultaneous non-blocking paths is not a factor, these topologies are still preferred because they minimize path length and points of failure. Connecting x8 to x8 to x8 vs x4, x4 to x4, x4 with an x4, x4 cross connection makes the core expander a single point of failure in the first case, and increases path length for half of the paths to the second expander vs the second case.

One might want to conclude that there are the same number of single hop paths in both cases, we've just moved half of them from the first expander to the second. But consider the case when you add an additional layer, x8 to x8 to x8 to x8, and it should become clear that we are not just shifting equal path length destinations between expanders except in the degenerate case of 3 node topology, and that the Clos layout significantly reduces path length and maximizes diversity as network size increases.

Consider also the case of two controllers connected dual link to 2 expanders, connected dual link to a single expander, connected to 16 disks. This would work but only x8 circuits could be opened to the disks at any given time through core expanders, and the more optimal topology would reapproach the Clos pattern favoring a mix of x8 and x4 wide port direct paths to every expander and allow the full x16 circuits available from the controllers to reach the disks in the maximum possible number of path combinations.

You do not see anyone providing examples of the x8... topologies in the documentation because supporting the Clos layout is more complex from a routing perspective, implicitly suggests support of the simpler topology, and in literally no case would you actually want to use the x8... over the x4, x4 to x4, x4 with cross x4 topology when the latter is supported. But the x8... case should, barring absolutely braindead implementations, be supported anywhere you see support for the Clos layout suggested by the documentation. These examples are plentiful in the protocol specs and chipset documentation, as in e.g. page 3 of the data sheet for the LSI SAS2x36 chipset:

https://docs.broadcom.com/docs-and-downloads/sas-expander-ics/lsisas2x36/LSISAS2x36-PB.pdf
 
Last edited:

lumdol

Member
Jun 9, 2016
30
2
8
39
Thanks. You've been a wealth of information.

My concern was that firmware / configuration differences on the Intel branded expanders for their own servers might have altered the available functionality of the LSI chip.

If you'd be so kind would you make a recommendation for cabling my chassis as seen below?

It's an intel 24 port and 36 port, feeding 36 2.5" 1tb spinning disks. I'd like to have x8 to my controller, plus ports for expansion.

My guess based on your responses would be one controller connection to each expander, disks split relatively equally between each, with the expanders linked internally with one cable, and then the remainder outgoing from each expander?

 

Tiberizzle

New Member
Mar 23, 2017
25
11
3
124
I would recommend the following topology with the parts you have on hand:

From the 24 port expander, 4 ports for cross connection to other expander, 4 ports for upstream, 16 ports for drives
From the 36 port expander. 4 ports for cross connection to other expander, 4 ports for upstream, 20 ports for drives

This topology would survive the loss of a controller <-> expander link with only reduced throughput.

2.4GBps is the maximum data throughput for 4x6Gbps due to the 8/10 encoding as you noted earlier, so in theory the slight imbalance in drive distribution is relevant and would result in a slight (~10%) bottleneck for linear access near the outer zone of modern 7.2K drives, but in practice both ZFS and the use of a single controller will shift the bottleneck further upstream.

I would not recommend expanding this topology by daisy chaining as it should already be expected to bottleneck for linear access in outer zone and limit scrub/resilver throughput.

If you want to expand beyond the 36 drives I believe discarding the 24-port expander and moving to dual 36-port expanders in every chassis with 2 controllers using dm-mpio would provide best results with fewest number of links:

In NAS head:
2 controllers with 1 x4 downstream to each expander

In 2x 36-bay chassis:
2 36-port expanders each with x4 upstream, x4 internal cross, and x4 to other chassis

This topology would provide the full x16 dual controller bandwidth for every combination of downstream devices, tolerate simultaneous loss of almost every expander-expander or expander-controller link, and provide a single expander path to every disk. Any alternate topology with the same or lesser number of upstream/downstream links would for sure have fewer single expander paths and I believe would suffer greater loss of bandwidth in some multiple link down scenarios, and of course the pathologically bad alternatives with fewer links would lose all available paths to some disks much sooner. You do have some extra links on the 36-port expanders which could be utilized, at greater cost in cabling, to offset inefficiencies of some alternate layouts.

Finally, I would recommend against using a SAS/SATA ZIL at all with an array this size and investment. Optane devices are so well-matched to the ZIL requirements for the price that it is reasonable to conclude any other choice is objectively wrong.

I believe the 900p is still in the sweet spot here. It absolutely does have power loss protection, the retroactive removal of it from the specification is market segmentation / standard Intel cash grab and nothing else.

Intel responded to the criticism of the specification update which removed PLP from the 900p as follows:
As an enterprise part, the Intel® Optane™ SSD DC P4800X offers multiple data protection features that the Intel® Optane™ SSD 900P does not, including DIF data integrity checking, circuit checks on the power loss system and ECRC.
So, it still offers PLP by Intel's own admission, it lacks "circuit checks on the power loss system". Whatever the **** that means. It's literally the same board with identical components and a different sticker. They're just shifting the goal posts somewhere intentionally ambiguous and unverifiable so the suckers won't feel quite so cheated. In practical terms the difference is whether or not they will replace the device under warranty after it has already eaten your data in a power loss event, but without additional data I would have to conclude the probability of that happening is the same for both devices as it is an inherent property of this type of solid state memory.

It is worth foregoing hotswap capability for the ZIL devices, even, if your NAS head does not have NVMe hot swap capability, and just using a pair of mirrored Optane 900p. Additionally, assuming failure of power loss protection exists at all, it is implied to be probabilistic for the 900p since we gain only "circuit checks" with the 4800x, mirrored ZIL would mitigate this.
 
Last edited:

lumdol

Member
Jun 9, 2016
30
2
8
39
Thanks! I was close, but no cigar.

I was thinking of moving to 2x 36 ports and using the 24 port for a backup chassis anyhow (see the WD red situation below).

Presently I also have two Intel S3700 100GB and a couple of 240gb enterprise Sandisk SSDs.

This build has lived happily for some time in a slightly different configuration in a different chassis, and now getting around to expanding it and migrating to rack mount form factor. I was using S3700s since about 2015.

However my workspace then became a sea of power bricks, boxes, whirling fans and cabling due to using thunderbolt expansion.

Thus I’m putting it all into a quiet rack to save myself some space and sanity.

My plan using 2.5” disks originally was to eventually migrate to a full ssd compliment when the price and performance became reasonable. However performance with my workload is such that I don’t feel overly compelled to buy 30 1tb SSDs.

The original configuration was:

4x 5 disk raidz1 HGST 1tb
2x hot spares
2x s3700 mirrors (from memory 20 or 30gb overprovisioned from 100gb disks)
2x Sandisk mirrors (forget model but enterprise) for l2arc (from memory 80gb overprovisioned from 240gb)

So, I’m nowhere close to saturating even one port. Even if I was, I’m capped just under it with thunderbolt2.

Nonetheless 5 disk raidz1 seems to be a nice sweet spot for my needs. I haven’t felt need to reconfigure.

The expansion is to add 2x 5 disk raidz1 for a total of 6 vdevs

This is then backed up nightly to a shelf of WD Reds, so I will need daisy chaining capability or to add another HBA any way you shake it.

Presently I’m looking into potential 8x or 12x 2U shelves that’ll fit a 26” rack for this purpose to remove yet another box and power brick from my desk.

Are the optanes just head and shoulders above?

I was thinking of striped mirrors with 4x S3700 for ZIL. They are heavily overprovisioned and the two current ones have over 95% life in them still according to smart. I believe I still have two spares somewhere as I purchased a lot of them being liquidated from a datacenter some time back.

I have a spare half height pci-e slot in the sonnet xmac, but had considered using it for another HBA down the road instead of a pci-e ssd.