Help me build a VMware home lab

[VR Bitman]

The Mikrotik L3 switch arrived at my doorstep today. I have to say it has an impressive feature set for the price.
One thing that bothered me a bit was that the default management subnet is unusual (or is it?). I actually had to first connect via MAC and change it to be able to comfortably access the web interface via browser on my home network.
Also, now that I'm looking at the web interface... the configuration of this thing doesn't seem to be for the faint of heart!

 

[VR Bitman]

I have successfully won an auction for a used HP DL380 G6 server. For the price of 389€, expensive shipping included, I get:
- 2U server
- 4x PCI-E slots
- quad 1GbE NICs (HP NC382i onboard)
- HP iLO 2 management interface
- 2x 146GB SAS disks
- DVD-RW optical drive
- HP Smart Array P410i
- Intel SATA controller
- ATI ES1000 GPU

As for RAM and CPU it has only 10GB DDR3 and a low-end Xeon E5504. Nothing impressive here, but I will upgrade it in time. I specifically wanted it to be underpowered so that the initial cost is as low as possible.
Nobody else bid on it. I reckon it's because it seems to not be worth it thanks to the CPU and RAM, but I find that quad onboard NICs are rare. Many servers also don't have management interfaces, optical drives or other optional features.

 

[Patrick]

You can upgrade the CPU fairly easily to something significantly faster if you really wanted.

 

[VR Bitman]

You can upgrade the CPU fairly easily to something significantly faster if you really wanted.

Yes it was exactly my plan. I'm planning on swapping it with a hexacore X5650 in the near future.

 

[Mike]

Curious. Are you getting it shipped from the US or did you find a cheap EU seller?

 

[VR Bitman]

I guess it's a "cheap EU seller". Germany.

 

[mervincm]

The Mikrotik 226 is indeed a bit unexpected for many of us. I too had to resort to the mac for the initial connection post ip configuration.

The HP 380 series servers are the industry standard 2U box. a Gen 6 is exactly where I would start if you want to play with real servers at the lowest price. The G5's are considerable less powerful, more noise and not worth your time IMO. G7 and G8 are still much more $$$. They are not than picky on RAM, but I have had issues with trying non HP storage in it, so be careful there. Tons of slots so you can pick up small sticks for nearly nothing if you come across them. with 18 DIMM, you can fill it up with 4's and 8's pulled as others upgrade. Be aware that if you have a single CPU box, you can only use 1/2 of them.

Hopefuly you got lucky and the ILO is licensed, as it's not cheap separately.

Did you decide what you will connect to your 10G-E ports? I would suggest leave one for your storage server and 1 for a visualization host. I bought Broadcom cards as they were inexpensive and on the VMware support list.

 

[VR Bitman]

Thanks for the advice, mervincm. Much appreciated.

I don't think I'll use 2 CPUs because power consumption would really go up.

I'm beginning to think that the Mikrotik was a mistake. 2 ports are not going to cut it. I am planning to create a 3-node cluster, there is no way I can connect everything with 10Gb links.
Anyway, what model of Broadcom cards have you actually purchased?

 

[mervincm]

Sound like you really need 6 10G port
3 - one to each node
1 - storage server
1 - uplink to the rest of your network
1 - spare (migrations to new hardware, etc)

Try thinking about it this way
While the Mikrotik is not going to do everything for you, it is surely good enough until you buy all that gear, and get enough data on it that you really need it all on 10G. Right now a 10g switch with more than 4 ports is VERY expensive, noisy and likely consumes 5-10 times the power of your Mikrotik. Once you get this 6+ port 10G switch, the Mikrotik slides into the role of using the "uplink to the rest of your network" and providing a spare 10G connection. I bet getting to your end state, where you honestly need the 10G everywhere will take a while, and having delayed that purchase you will see cheaper better options come available.
This is what my thinking was for buying it anyway

My Broadcom cards are IBM rebadged BCM957710A1020G

 

[VR Bitman]

Hmmm a search for the model number isn't turning up many results.. I've tried other searches as well but with little success.

 

[Dev_Mgr]

Look for broadcom 57710. That will yield more results for that card.

 

[VR Bitman]

Alright for now I'm going to focus on getting the cluster up and running, I will upgrade everything to 10G further down the road. I'd like to thank everyone who replied so far.

Through a local group I was able to find a cheap open frame rack (30U or so) that I will be picking up next week for 70€, which is great.

Next, I got a DELL server. It is a R710 model with the following specs:
- 2U server
- 2x 570W PSU
- 4x PCI-E slots
- quad 1GbE NICs
- DELL iDRAC 6 Express management interface - I will upgrade this to the Enterprise version via the add-on management port
- no disks and no HDD cages - I will add 2x SAS disks and 6x cages to "fill all holes"
- no optical drive - no real need for it, but I will add a DVDRW drive anyway
- PERC 6i
- GPU: I don't remember / haven't checked (it's of little importance for now)
- 8GB DDR3 1333MHz RAM (2x4GB)
- 2x low-end Intel Xeon E5504, quad core 2GHz, 4MB, 64bit
- 2x front USB ports, 2x rear USB ports
- 1x front VGA port, 1x rear VGA port
- SD card slot: I don't remember / haven't checked

The cost was 380€ shipping included. I've noticed that - what do you call them - the side mounts for the front bezel are missing. I'd like to add them, but I'm not sure what to search / how to find them..

Anyway, both the HP DL380 G6 and the DELL R710 look excellent, as new. I haven't powered them on, yet.

To complete my 3-node cluster I am now looking to buy the IBM System X3650 M2, with the same CPU as the previous two servers.
The only thing that confuses me about the IBM model is whether the physical management port for the IMM is standard or not. Some documents I found indicate that it might be and that advanced features are enabled with a small add-on chip, but a seller that I contacted on Ebay about an interesting deal told me that their server has no remote management capabilities.

 

[VR Bitman]

Confirmed, the management port is standard on the IBM X3650 M2. Excellent.
I found a good deal from another European seller on Ebay and now I have an IBM product on the way for 315€, shipping included. There are no disks, but that's OK. The hypervisor will run on SD cards or USB drives.

I have powered on the two servers that I already have for some quick tests and they both work just fine. The only issue I could find was that one of the Nidec fans of the HP server makes an unusual noise, so I will replace it.
I've also realized that the HP server does not have the secondary riser card, so I've added that to my list of future upgrades.

Here are some pictures:

I still need to figure out how to find the side mounts for the bezel of the R710.

 

[VR Bitman]

Today I received the third and final server, and IBM System x3650 M2.
As I mentioned before total cost shipping included was 315€, and here are the specifications:

- 2U, 2-socket server
- 1x 675W PSU
- 4x PCI-E slots
- 4x 1GbE NICs (Broadcom BCM5709 onboard)
- IBM IMM standard management interface
- no disks
- DVD-RW drive
- Megaraid SAS 8808E RAID controller
- Matrox MGA G200e 8MB GPU
- 8GB (4x2GB) DDR3 1333MHz RAM
- 2x low-end Intel Xeon E5506, quad core 2.13GHz, 4MB, 64bit
- 2x front USB ports, 2x rear USB ports
- 1x front VGA port, 1x rear VGA port
- 3x fans

A couple of pictures:

It was a bit dusty, but I dismantled it, cleaned it a bit, put it back together and finally powered it on. It works perfectly and has no signs of damage.

I have also succeeded in finding the lateral pieces necessary to mount the rails for the R710 server. After a session of trial and error that took me some time, I discovered a keyword that will turn up results on Ebay: ears. Searching for "R710 ears" turns up 3 relevant results. The part number is GW660 but using it as a search term yields little success.

If all goes well, tomorrow I will pick up the rack cabinet and start mounting the hardware.

I am now focusing on designing a low-cost, high-capacity storage solution with decent performance. This will be "homebrewed".

 

[VR Bitman]

The laboratory is finally starting to take shape with the rack cabinet that I picked up yesterday.
Here it is with the Mikrotik mounted:

The challenge is now to build the storage solution.
What I have in mind is a Micro-ATX system that will cost precisely 300€.
These are the components I was thinking of:

- motherboard ASRock Q1900M (integrated Intel J9100 quadcore CPU) 58€
- 2x4GB IBM DDR3 PC3-12800 CL11 ECC DDR3 90Y3154 47J0164 33,5€
- NIC Intel PRO / 1000 PT Dual Port | 2x Gigabit LAN | PCIe x4 35€
- PSU Silverstone SST-ST30SF semi-fanless 300W 74€
- case Fractal Design Define R4 100€

The key component of course will be the RAID controller. I plan to use the PERC 6i which I already have in the DELL R710 server.
Right now there are 2 questions I do not have the answer for..
1. will it work with the ASRock motherboard? all of the slots are x1 electrical
2. will it support 6TB (and higher) capacity drives (perhaps with a firmware update)?

 

[pyro_]

While the raid card will fit in the 16x slot it will be very limited in bandwidth by it only being 1x electrical

 

[mervincm]

I would come up with a design that included a 10gigE card for your storage server. If you are gonna roll your own, I wouldn't have it any other way. Also why atom? your servers are gonna dwarf any CPU power wasted in here. You will want the more PCIE lanes that other CPUs include imo.

 

[Dev_Mgr]

None of the Dell PERC6i firmware updates offered support for drives larger than 2TB. However, I'm not sure if you can cross flash it to the LSI equivalent and if that then will support larger drives.

 

[VR Bitman]

I wanted to keep costs as low as possible. My original plan did include 10G, but then I shelved the idea fearing that finding an affordable 10G card would be an impossible task.
I have now discovered the Brocade 1020, so I'm including that in the project.

What CPU and motherboard would you guys recommend, keeping in mind that cost is a priority? The storage server must not cost more than 500€ (excluding drives).
As regards the RAID controller, if the PERC 6/i is not adequate, I was looking at the (apparently) popular IBM M5014 (LSI 9260-8i). Would this product be a good choice for very high capacity drives and great performance?

 

[pyro_]

You might also want to look at the mellanox 10gb cards as they can be found fairly cheap as well