Purpose / end state goal:
I intend to build 2 new ESXi Servers on Xeon E3-1200v3 architecture in order to run a high efficiency network with minimal “always on” physical network hardware. This means no always on physical switch and favoring multiple NICs to form a soft-switch to cover down. Both servers will run ESXi 5.5 connected with Infiniband. One server will run as a router/NAS (ZFS) and possible domain box. The NAS server will be a single spinning disk for frequently accessed files and 3-4 SSD ZFS array for VM storage and PXE boot targets. The second server will be a backup and mass storage server running Windows 2012 Essentials with iSCSI targets to a OpenIndiana ZFS array. In the beginning, the entire setup will be consolidated to one server to spread out cost. If it runs well, I may keep it to one server. If not, I may offload the large array and Windows 2012 portion to a C6100 node until the second server is built. Separate servers are preferable to me to minimize always on drives for power and wear considerations.
One of the primary goals is to keep power requirements to a minimum. Currently, my Pfsense router and server is hosted on a Core2 Quad utilizing an APC SMX1500 UPS to power up and shut down the network on a schedule. This became important to me after I saw my power bills. This beast sucks 250w with the switch at idle and can easily pull upwards of 500w. With the new Xeon chips idling around 22 watts, I figure I can keep idle power requirements to below 70w with 2 network cards PCIe cards, 4 SSDs and a spiny disk. The second server will contain 1 Infiniband card, 1 HBA, and 4 growing to 8 x 4TB spiny disks over raid 50 (two raidz1 arrays in a stripe as space is needed). This may become 4 x 4tb and 4 x 6tb drives if the 6tb drives are released to consumers on time. All said and done, the first server running the router and NAS will idle low for always on activity. The second server will be a bit less efficient with the number of drives, but will only run when I accessing movies or automatic backups are conducted. I think 70w always on and 140w with storage server at idle sounds much better than the 250w or more likely 330w of the old system.
Another primary goal is to minimize expense and maximize performance by consolidating SSDs on the network into a single ZFS array. This will prevent each machine from requiring its own SSD thereby reducing the quantity of SSDs I must buy and keep powered on. For this to work well, network speed is a must hence the reason for infiniband. The fat pipe infiniband provides will allow multiple operating systems to be remotely hosted without bogging down the network.
A secondary goal would be to achieve this package in the form of a cluster. This may be a bad idea as I’m utilizing different hardware (add-on cards) across both servers. Achieving HA will be impossible with the storage array since it’s specific to one server due to hardware. The two ZFS servers must stay on their respective ESXi host. Vmware Virtual SAN may be a solution, but I have done zero research to date to understand its capabilities and how it works.
I’ve been working on setting up Infiniband in what little spare time I have with varying degrees of success. Since I’ve posted in multiple other posts, I thought I would consolidate in a separate thread and document my trials and errors. If you have suggestions or advice, please send them my way as it is most welcome.
The Test Bed:
Dell C6100
-4 Nodes, each w/ 24GB ram and 2 x Xeon 5520 processors, SAS Mezz adapters and 2 x 250GB HDDs. (reconfigured to 2 nodes with 48GB ram for testing)
-2 nodes have Mellanox ConnectX-2 VPI adapters. (all testing has now been moved to these hosts IOT achieve consistency with the Mellanox adapters. HDD swaps are conducted to test differing configurations.)
-2 nodes have HP ConnectX VPI adapters (HP adapters work well, but driver support sucks for Windows 7 bare medal)
Operating systems in use:
ESXi 5.5
Windows 7
Windows 8
Windows 2012 Essentials
OpenIndiana
PFsense 2.1
Ubuntu
Testing:
Most of my testing to this point has been with 10Gbe which has been pretty good and very user friendly. The main issue I’ve had is saturation of 10Gbe, high latency, and the price and power requirements of 10gbe switches and network cards. If my planed setup is going to work well, I believe infiniband is the answer.
Initially all speed tests shall be performed with primo ramdisk operating in IO mode and 14.5gb of mixed size file transfers. Each test is performed 10 times minimum with erroneous data point thrown out.
This is not the most scientific process, but intended just to get an idea of how everything works. At some point, I need to learn how to use iometer in order to get more consistent and descriptive results. For now, I’m looking at very simple setups to get a baseline quickly and determine what testing I want to conduct in the future. Once I determine the way forward, I will update each post to include iometer results.
I intend to build 2 new ESXi Servers on Xeon E3-1200v3 architecture in order to run a high efficiency network with minimal “always on” physical network hardware. This means no always on physical switch and favoring multiple NICs to form a soft-switch to cover down. Both servers will run ESXi 5.5 connected with Infiniband. One server will run as a router/NAS (ZFS) and possible domain box. The NAS server will be a single spinning disk for frequently accessed files and 3-4 SSD ZFS array for VM storage and PXE boot targets. The second server will be a backup and mass storage server running Windows 2012 Essentials with iSCSI targets to a OpenIndiana ZFS array. In the beginning, the entire setup will be consolidated to one server to spread out cost. If it runs well, I may keep it to one server. If not, I may offload the large array and Windows 2012 portion to a C6100 node until the second server is built. Separate servers are preferable to me to minimize always on drives for power and wear considerations.
One of the primary goals is to keep power requirements to a minimum. Currently, my Pfsense router and server is hosted on a Core2 Quad utilizing an APC SMX1500 UPS to power up and shut down the network on a schedule. This became important to me after I saw my power bills. This beast sucks 250w with the switch at idle and can easily pull upwards of 500w. With the new Xeon chips idling around 22 watts, I figure I can keep idle power requirements to below 70w with 2 network cards PCIe cards, 4 SSDs and a spiny disk. The second server will contain 1 Infiniband card, 1 HBA, and 4 growing to 8 x 4TB spiny disks over raid 50 (two raidz1 arrays in a stripe as space is needed). This may become 4 x 4tb and 4 x 6tb drives if the 6tb drives are released to consumers on time. All said and done, the first server running the router and NAS will idle low for always on activity. The second server will be a bit less efficient with the number of drives, but will only run when I accessing movies or automatic backups are conducted. I think 70w always on and 140w with storage server at idle sounds much better than the 250w or more likely 330w of the old system.
Another primary goal is to minimize expense and maximize performance by consolidating SSDs on the network into a single ZFS array. This will prevent each machine from requiring its own SSD thereby reducing the quantity of SSDs I must buy and keep powered on. For this to work well, network speed is a must hence the reason for infiniband. The fat pipe infiniband provides will allow multiple operating systems to be remotely hosted without bogging down the network.
A secondary goal would be to achieve this package in the form of a cluster. This may be a bad idea as I’m utilizing different hardware (add-on cards) across both servers. Achieving HA will be impossible with the storage array since it’s specific to one server due to hardware. The two ZFS servers must stay on their respective ESXi host. Vmware Virtual SAN may be a solution, but I have done zero research to date to understand its capabilities and how it works.
I’ve been working on setting up Infiniband in what little spare time I have with varying degrees of success. Since I’ve posted in multiple other posts, I thought I would consolidate in a separate thread and document my trials and errors. If you have suggestions or advice, please send them my way as it is most welcome.
The Test Bed:
Dell C6100
-4 Nodes, each w/ 24GB ram and 2 x Xeon 5520 processors, SAS Mezz adapters and 2 x 250GB HDDs. (reconfigured to 2 nodes with 48GB ram for testing)
-2 nodes have Mellanox ConnectX-2 VPI adapters. (all testing has now been moved to these hosts IOT achieve consistency with the Mellanox adapters. HDD swaps are conducted to test differing configurations.)
-2 nodes have HP ConnectX VPI adapters (HP adapters work well, but driver support sucks for Windows 7 bare medal)
Operating systems in use:
ESXi 5.5
Windows 7
Windows 8
Windows 2012 Essentials
OpenIndiana
PFsense 2.1
Ubuntu
Testing:
Most of my testing to this point has been with 10Gbe which has been pretty good and very user friendly. The main issue I’ve had is saturation of 10Gbe, high latency, and the price and power requirements of 10gbe switches and network cards. If my planed setup is going to work well, I believe infiniband is the answer.
Initially all speed tests shall be performed with primo ramdisk operating in IO mode and 14.5gb of mixed size file transfers. Each test is performed 10 times minimum with erroneous data point thrown out.
This is not the most scientific process, but intended just to get an idea of how everything works. At some point, I need to learn how to use iometer in order to get more consistent and descriptive results. For now, I’m looking at very simple setups to get a baseline quickly and determine what testing I want to conduct in the future. Once I determine the way forward, I will update each post to include iometer results.
Last edited:
