Refreshing the ESXi server

Build’s Name: WorkHorseII
OS: ESXi (or HyperV) to host Windows Server, Windows, Linux, BSD, FreeNAS, pfSense, unTangled, etc.
CPU: 2x Intel Xeon E5-2670 (V1)
Motherboard: Intel S2600CP2J
Chassis: Intel P4216xxMHGR
RAM: 128GB (16 x 8GB)
Drives: Various; Seagate 4TB for FreeNAS
Power Supply: Default per chassis (redundant 750W)

I'm looking to refresh, clean up, or move on. This server sadly has sat unused and disconnected. A few things to resolve:

1) maybe upgrade to E5-26?? V2 CPUs

2) Drive storage options (SuperMicro M28SAB-OEM, Intel FUP8X25HSDKS (hack/mod), Intel FUP8X25HSDKO (eol), etc.) This will determine how I configure the box. 1x,2x, or 3x SAS controllers, or use the Intel "240" or "360" Expander? That in turn determines which cables I use and where I route them.

3) PDU Auxillary PCIe Power to Graphics Cards (custom cables?)

4) LSI 2008 detects that the 5x 4TB Seagate HDDs are alive, but not their capacities?!?

Just got ESXi 6.7 installed to 32GB Lexar USB flash.

Wow, Java is the framework for accessing IPMI on the Intel S2600CP2J? Nothing at all like Supermico's IPMI that I'm used too on several boards.

Have ~24-28GB free on the USB boot device, but I need a different drive volume for datastorage?!? Yeah, sorry, it's been long while. I thought I did just that with 2x 1.0 TB HDDs with ESXi 5.0-5.5 hardware - maybe mistaken? Also, trying to figure out the whole pass through and possible LSI 2008 limitations to drive capacity - I can see 5x Seagate HDDs from within ESXi, but they just don't have any capacity displayed. Is this an issue with sector sizes - from 512b to 4K as I recall, or a drive capacity issue?

Add-On Cards Installed:
1x 6-port 1.0 GbE card
2x LSI 8i 6G SAS controllers (IT mode)
1x Intel "360" SAS Expander (tight - no cables) designed to be single or dual ported? It there a manual for the Intel "360" SAS Expander? Where do the I2C cables go?

This is what I got:

USB Flash Drives: various USB 2.0 and USB 3.0/3.1 flash drives upto 128GB; likely to use the "nano" (Sandisk USB 32GB Fit or similar) - using 32GB Lexar (purple plastic slider) USB flash drive.

Drives: (mix'n'match and move around if needed)
256GB Toshiba Q Pro series SSD (atm, Win10 boot drive)
480GB SandDisk Ultra II SSD (floating)
500GB Samsung 850EVO SSD (floating, Raspberry Pi)
others I'm sure..

~10x 4.0TB 2.5" 15mm (shucked or can be) Seagate HDD

Add-on Cards Available:
1x nVidia Quadro 4000 - the one available PCIe x16 slot is limited to half sized cards (otherwise the RAM interferes)
2x Infiband dual-port 40GB controllers (no cables)
1x Intel 8i 6G SAS (B0 stepping) SAS controller (locked/fixed to Intel S2600CP - NO cross flashing)
1x Intel "240" SAS Expander
(various SAS to SAS and SAS to SATA cables)

What I still don't know with both Intel Expanders is: 1) Will 2x miniSAS double bandwidth performance (like they do with HP Expanders)? 2) Any known issues or bottlenecks using the Intel SAS Expander(s) with SATA drives?

I'm wanting to keep the "lower" 16 bays strictly for FreeNAS drives (if I can). As I still have 3 open 5.25" bays, I'm looking for drive bay options. Do I try to low bid on an Intel FUP8X25HSDKS Spare 8x2.5" Hot-swap Drive Bay to fit into 2x 5.25" bays to hack and modify? Or find some other IcyDock Solution (like the one listed on the front page the other day)?

In my digging, I discovered that PCIe Slot5 is on CPU socket 2, all the others are on CPU socket 1. I have both socket's occupied - Does this even matter for a VM like pfSense/unTangled, etc??

-- Near future upgrade options:
$359.99 - Matched Pair Intel Xeon E5-2680 V2 SR1A6 2.80GHz 25MB 10 Core LGA2011 CPU
$279.99 - Samsung 32GB 4Rx4 PC3L-10600L-09-11-C0 RAM
$52.99 - Kingston KTD-PE318/16G 16GB Memory RAM PC3-14900R
$49.99 - Myricom 10G-PCIE-8A-S [single port] Network Adapter
$52.99 - Myricom 10G-PCIE-8AL-S [single port] Network Adapter
$104.99 - Myricom 10G-PCIE2-8B2-2S [dual port] Network Adapter
$34.99 - Chelsio CC2-N320E-SR Dual Port Network Adapter Card
$44.99 - IBM 5769 Chelsio 10GB PCI-e Single Port Network Adapter 46K7897

 

Drive bay options:

Try If you can find a good deal on a2u8x25s3<*>
Those are designed for 2u rackmount servers and should fit in the 2x 5.25 bay space. Securing may be a challenge.

I think newer variants can support NVME also.

You have already listed SM and ICYDOCK options.

Intel SAS HBA - with 2308 C0 chip.
Though it's locked to Intel MB, you can flash to LSI 9205 P20.07 just not 9207

PCIe power cables.
Check kalleyomalley for P4000L or P4000M electric accessories kit. Been about 2 years but I think they accepted very reasonable offer for the kit.

If not PM me and I can check my parts bin for spares.








Sent from my Nexus 6 using Tapatalk

 

Yeah, I think the wiring is straight forward, just a re-purposed P4 CPU power plug with two grounds and two 12v (?) power. So long as ground and power are kept straight, I should be able to hack, modify, or even build out my own. Just need the right crimping tools to build my own. Stopped into two computer stores - one had a box of goodies. The other store should have some as well, I just barely missed the tech as he stepped out for an on-site service call.

P.S. - May seem like a dumb question, why do these Intel (4x 3.5, 8x 3.5, 8x 2.5, 16x 2.5) HSBP kits include SAS to SATA when it seems that SATA to SAS cables (mobo SATA) or SAS to SAS cables (mobo or controller) is what is really needed? So are these included Intel SAS to (silver) SATA cable(s) forward or reverse breakout? I have two different breakout colors/styles - one set, that is shorter, has silver SATA cables (part of the kit) and other set, that is longer, has blue SATA cables. I think they are all forward breakout cables.