Built in vs add on, and SAS cabling issues

[ynari]

I've taken the plunge on a pair of fairly reasonably priced E5-2690 v1 cpus, now to get a board that comes with it. I want IPMI, 6GB/s SAS (currently using SATA, but SAS should provide more backplane/hot swap options, and is compatible), and dual twin slot GPUs , plus at least one other add on card.

I was looking at the Supermicro X9DR7-LN4F-JBOD. I note, however, that there are very reasonably priced second hand 12Gb SAS controllers out there already, and that perhaps it's just as good to get an add in card, if I find a cheaper motherboard with a decent number of slots (6-7). SAS3 is completely backwards compatible with SAS2/SATA, isn't it?

Thoughts? I'm not a fan of Asus because they haven't cared for consumers in the past, or for Intel and their qualified list of PCI-e 3.0 cards. So there's a variety of Supermicro options if I can find them in stock, and there's the Gigabyte GA-7PESH2 (stingy on slots, appears to run them all off one CPU - not happy with that) and the GA-7PESH3 - which is looking good, if expensive (and also very difficult to find), but has normal SAS ports so I'd need to budget another £60 or more for SAS to SAS cables, instead of the mini SAS to SAS cables used and included on most other motherboards.

Sidenote : I'm very impressed by what this person (chune's house build) has done with the GA-7PESH3. Similar to my existing set up, but on steroids. Although when I look at it, the KVM box does not support HDCP..

 

[Patrick]

How many drives are you looking to support? Hard drives or SSDs? Do you have SAS drives?

 

[ynari]

At the moment it's just four hard drives in software (Linux mdadm) RAID10 backed by a RAID 1 SSD cache, because that's the maximum my current system supports. I'm open to more in the future possibly. The main reason to go SAS initially was to ensure 6Gb speed. At some point it would be nice to move to a ZFS backend under FreeBSD, but Xen under FreeBSD does not currently support PCI passthrough.

I'm not intending to change my system for a bit, so it's nice to have expandability.

 

[J--]

Personal opinion here, but X10 is nearing a better value proposition at this point. The v3/v4 CPUs are much better on power, to the tune of 100W+ with actual loads (that's probably like $120/year in electricity depending on where you live). The board you're suggesting looks like isn't coming in under $500, whereas I was able to pickup my X10DRi (granted w/o the LSI chip onboard) for <$300. So add a $75 LSI 3008 chipped controller, and you're still spending less.

I haven't priced out 8-core v3/v4 CPUs to find something equivalent to the gen1 2690s, but 8-core is near minimum in the Haswell/Broadwell platforms.

The real driver will be how much RAM you need, as DDR3 is still much cheaper than DDR4.

 

[ynari]

I definitely hear you - I'm in the UK, by the way. The reason I decided to go for the 2690 is that it still has a relatively high clock speed, that doesn't seem to be bettered by later chips unless new instruction sets are used. Obviously for specific multithreaded loads the later chips are a lot faster, but I'm mostly running a couple of desktop VMs with hardware passthrough, plus a number of other server VMs.

If I chose a chip that was overall faster, then yes, it wouldn't necessarily be vastly more expensive (v3 chips are hovering at 600 quid when I looked, which is expensive but not unimaginable). If, however, I look for one with a decent clock speed it becomes awfully expensive very quickly (1500UKP+ each, way more than I want to spend).

RAM - not too much, really. Between 64 and 128GB, more would be overkill.

I'm very interested in the LSI 3008 chips, trying to find more detail on them. I note the 2308 bundled in various cards/motherboards supports SR-IOV in addition to 6GB/s. The ability to pass through one SAS channel to a VM sounds like a 'nice to have' I'd definitely like.

I can find a couple of second hand/reasonably priced Supermicro boards, just trying to make a decision before I pounce on them. Also trying not to source it from the US, where you lucky people have a plentiful supply, as we get stuffed on import charges..