PigLover's playground

[PigLover]

Finally got everything racked/stacked how I like it. Well, not really, since everything changes weekly - but its much better than it was! Ordered a self-assembly rack from StarCase.com sized to fit a small space in my garage. Didn't really plan for the new switch (Cisco SG500X-48 - 48Gbe + 4 10Gbe SFP) and that C6100 Patrick found for us. But its all worked out real nice.

A few pics:

The whole thing tucked away in its little corner. It looks crooked because it is. Well, it isn't, but there is a small rise in the concrete floor near the wall and when you park it, it lists. Its all level and square when you pull it out, really, it is!

And pulled out to access the back:

Cabling into the house, etc, hides in this little cabinet above it.
24 port pp to outlets inside the house, fiber pp to outlets at my desk, 24 port pp to rack. Also a small POE switch to power a few things (2 WiFi APs, outside cameras).
Lower half has Comcast cable modem, Mikrotik router and HD Home Run 3-tuner cable TV tuner for MythTV

Lower part of the rack - servers (bottom to top)
APC SmartUPS 1500 RT (2U gap below it is for extra APC battery shelf when I find the right deal)
Norco 4220 with 20x2TB Hitachi, X8SIA-F, X3450, 32GB ECC, 3x M1015, Intel 10GB X520-DA2. Bare metal Solaris 11.1 ZFS server.
Dell C6100 w/4 nodes, each with 2xL5638, 48GB ECC & 10Gbe (currently Mellanox, but Intel mezz cards on the way - thanks Justin!). All lit up as 4-node Proxmox VE cluster, but no VMs built yet.
I-Star USA 4U case with 11x 3.5" hotswaps, 2x 2.5" hotswaps & 600w PSU. Currently no electronics. Will be backup server soon.
Norco RPC-270. Currently empty, nothing, just taking up space and pretending to be a shelf (used to house ESXi server).

Upper part of rack - Network (bottom to top)
Avocent 17" rackmount KVM (there is a 16 port Trendnet KVM switch in the back of rack). All console ports connected, including all 4 nodes of the C6100
Juniper EX2500 24 port 10Gbe SFP+ switch (yum!). Loud and power hungry, but 24 ports 10Gbe - love & hate live together here. Haven't cabled all the C6100 10Gbe - waiting on the Intel Mezz cards first.
Cisco SG500X-48. Was going to retire the Juniper and just use the 4x 10Gbe on this baby 'till the 6100 came along...
24 port pp cabled to pp in closet. This keeps the "house" wire demarced from the rack. Connected by the fat bundle of cables (the smaller bundle in the white split inner-duct is the 6 fiber pairs from the upper pp in the closet)

A shot with the Avocent opened up.
I'm a big believer in remote management via IPMI and/or vPRO/AMT - but sometimes you just can't live without a real console! Besides - I got it cheap because the left rail was damaged and it wouldn't slide back in (amazing - somebody gave away a $400 KVM shelf for <$100 over something that could be fixed with large needle nose pliers, 2 minutes and a couple of grunts).

And the hot-swaps inside the I-Star 4U.
I have 12 2TB Hitachi's to fill it with, but looking for a new MB/CPU. Might buy 3TB or 4TB drives if the prices would get anywhere near pre-flood levels...but I might be waiting a really long time for that to happen!

Its a decent setup. Also has two rackmount outlet strips mounted in back, one near the bottom and one near the top (stupid ones - no monitoring). The Trendnet KVM switch is mounted near the top on the back side (and the ugly bundles of cables you can see in the side view are the KVM cables). I'm not completely happy with now I have the power run. I need to run data cables on both sides in back so I've got power and data running parallel on the back post - a no-no in "proper" cabling. I also need to figure out some kind of swing-arm for the Norco and I-Star cases as they need to slide out sometimes and the cables get all tangled & stressed. But other than that I am happy with how it turned out.

Biggest problem is power - the entire rack draws about 400w nominal idle load from the SmartUPS and goes up towards 850w when I have things running (ugh!). Might need to invest in solar before I buy any more toys!

 

[gigatexal]

drool man, very nice.

 

[zunder1990]

what is your power bill like.

 

[PigLover]

what is your power bill like.

Umm, not too good. Add a swimming pool pump and kids that don't know how to turn off lights and its ugly. Living in PG&E country - with the undisputed highest electric rates in the country - and, well...

OTOH, I changed over to a time-of-day price plan designed for owners of electric cars (rate plan E-9A for fellow PG&E'rs). It has really low rates during off-peak hours (7 hours/day weekdays and 19 hours-day winter weekends, so over 50% of the hours are off-peak. Works real good for 7x24 power pigs like me. I still pay through the nose...

 

[Jeggs101]

Nice rack!

 

[wyluliraven]

I like how you built that out.

Loud, power hungry and powerful, just the way it should be!

 

[nry]

Some very nice kit there, wouldn't mind that Cisco switch

What do you use all this kit for?

 

[PigLover]

Its mainly what I called it - a playground. I like to play around.

Main "production" use is the ZFS server, which has every DVD/Blu-Ray I've purchased ripped (~5TB), video of various sports teams that I've done highlight videos for (~4TB), thousands of my wife's photos (~2TB), recorded TV & Movies (~3TB), and various backups, downloads, home security camera recordings, etc. Its about 50% full.

The things I actually care about on the virtualized server are VMs supporting MythTV DVR backend (Mythbuntu), telephones (freepbx), security cameras (Blue Iris NVR), and a web server I use to share videos/photos with.

Other than that its just a toy for a tinkerer...

Oh - and that Cisco switch. Its actually a pretty awesome switch. I got it hoping to "retire" the Juniper 10Gbe. If I hadn't added the C6100 I would have. I could have been real happy with 48 GigE and 4x 10Gbe (with 10Gbe to the ZFS Server, 1 virtual host server, 1 to my desk and one "spare"). At ~$1,500 street brand new its actually a really good deal. My deal was better .

I do wish I had gotten the POE version as I've been adding more and more POE and it seems silly to run separate injectors or a separate switch just to support them.

 

[nry]

Impressive! Those C6100 servers seem pretty neat, don't see them on eBay here in the UK very often, I would like one of them to replace my 5 ESXi nodes, but guessing they are pretty noisy.

I am currently looking for a 24 port L3 switch with atleast 2x 10G ports. Seems to be hard to find for my budget

 

[Beaubeshore]

How is the cooling for this. Building a house and looking to put the rack in the garage. Confined space is not an airflow problem.

 

[alex1002]

Amazing!! Nice setup!!

 

[PigLover]

How is the cooling for this. Building a house and looking to put the rack in the garage. Confined space is not an airflow problem.

Haven't had a problem yet. Based on past years summer temps I don't expect to have one. But its not lived in the rack through a full summer either...

The APC unit on the bottom does have the Ethernet management card with environmentals and I have a temperature sensor near the top-rear of the rack. I've been collecting and tracking the temps. We'll see how it does when it gets warm this summer. Its actually pretty temperate here (N. CA / East Bay) so I think I'll be OK - but as I noted in another thread - you'd probably have trouble with it if you lived somewhere that it gets hot/humid and stays that way all day & all night.

 

[Beaubeshore]

Haven't had a problem yet. Based on past years summer temps I don't expect to have one. But its not lived in the rack through a full summer either...

The APC unit on the bottom does have the Ethernet management card with environmentals and I have a temperature sensor near the top-rear of the rack. I've been collecting and tracking the temps. We'll see how it does when it gets warm this summer. Its actually pretty temperate here (N. CA / East Bay) so I think I'll be OK - but as I noted in another thread - you'd probably have trouble with it if you lived somewhere that it gets hot/humid and stays that way all day & all night.

I am southern California.... what is the ambient temp of your garage. humidity isn't a problem. I am just worried about a 90 degree day. Do you have a plan for summer or will you just run it?

 

[PigLover]

I let it run on. We have regular runs of 100+ for a few days or maybe a week, but it generally cools down at night it's a sea breeze. Inside temps of my closed garage rarely get above 85, maybe near 90ish a couple of days/year.

My garage also has part of the house above it to insulate it from direct solar heating. If you have a one-story or garage away from the house it might be more troublesome on hot days.

I just let it run on....

If you are near-coastal SoCal - say within 30 miles of the water and not in one of the valley areas you should have a real similar climate. If you are in one of the valleys it might get too hot in the summer. YMMV.

 

[mrkrad]

Tell us about the switch? Does it support DCBX? vlan assist port virtualization qos?

$2500 is a wicked price. I picked up 4 2920 POE+ [uses an hp 575 watts server power supply] with 40gbps interswitch link ring and 2 x 2 10gbase-T (or 2x 2 SFP+ 10gbe) with cables for about $7500 - these even support what is essentially like infiniband - software defined networking where the switch essentially moves packets, but your vm tells it where to move them (very much like infiniband where connections establish port 1 goes to port 36. Pretty useless for me but hay why not.

Whilst i have 16 10gigabit ports splayed across 4 switches with a pretty beefy ISL link, I can't help but be jealous of that single 24 port TOR switch.

Alas without DCBx the best features of these swanky new CNA's are lost. so I'm hoping someone out there has some recommendations on cheap used (or HP brand) TOR switches.

I've got a loaner XS7224S (opposite of the XS7224 which is 20 ports 10gbase-T and 4 ports SFP+) that has 20 ports of SFP+ and 4 ports 10GBASE-T - which with the L3 software is about $10K - far out of my budget.

tell us more about that awesome switch? is it front to back or back to front that are cheap? power draw? I noticed 40 watts on the 2920-48 POE+ with a max of 550 watts when POE+ enabled. Do the 10 gigabit switches use up a bunch more heat? I can imagine a 10GBASE-T would tear a hole in your power and heat profile and quite honestly 10GBASE-SR SFP+/DAC cards are way cheaper.

 

[PigLover]

Your message is a bit confusing. Which switch intrigues you? The Cisco SG500 or the Juniper EX2500?

 

[uberguru]

drool man, very nice.

Looking at your signature..so you mean to tell me one can use different CPUs on each node of the C6100?
I thought the video i saw recently comparing C6220 and C6100 states that the upgrade from C6100 to C6220 was the ability to have different CPUs in each node?

 

[PigLover]

Looking at your signature..so you mean to tell me one can use different CPUs on each node of the C6100?
I thought the video i saw recently comparing C6220 and C6100 states that the upgrade from C6100 to C6220 was the ability to have different CPUs in each node?

Each node on the C6100 is a separate motherboard and, except for the common power, power management, fan control and alarming is independent in every way. You can run different CPU types on each node. Currently, due to adding an HP DL180 to the rack and needing the CPUs, I have two nodes with 2x L5638 and two nodes with 1x L5620 (the other two L5638s are now in the DL180).

I've never bothered to the video so I have no idea what point the Dell rep on the video was trying to make.

 

[PRIMEZ]

Are you using POE camrea? Which model if so?

 

[PigLover]

Are you using POE camrea? Which model if so?

El-cheapo ones. Dahua IPC-HDB3200C. Three right now on my front door, back door and back entrance to the garage. They work ok but lack some features I want like IR illuminators for night time.

I use a windows-based NVR program called Blue Iris to manage them. It's ok, but lacks some things I'd like to play with like facial/person recognition. I run a windows 7 VM in the cluster with the Blue Iris NVR, use motion detection triggered clips and store them directly to a share on the NAS. Works pretty well. I've been able to track the comings & goings from my house on my android and ipad even when traveling in Africa and Europe.