Renderfarmer's DIY Render Farm

[rubylaser]

A very cool thing about vray is that it lets me do both network rendering and distribited rendering. so I can either split a single frame amongst all of my machines or give each machine a single frame of an animation to work on.

This is a great feature. A while ago I used to manage a Blender render farm, and none of the render engines at that point allowed an individual frame to be split between render nodes. Also, this is a fantastic home setup. I'm very jealous, and impressed by the quality of your work.

 

[renderfarmer]

This is a great feature. A while ago I used to manage a Blender render farm, and none of the render engines at that point allowed an individual frame to be split between render nodes. Also, this is a fantastic home setup. I'm very jealous, and impressed by the quality of your work.

Thanks. Yeah, Distributed rendering is pretty cool, especially since the vast majority of my work is single large images I use it a lot. It's not a linear increase in speed with Vray because the light cache can't be split up. So your workstation ends up doing a lot of the heavy lifting, and of course it locks up your workstation. It would be much better if you could designate a sacrifical node to handle that but it's better than nothing.

 

[Patrick]

Thanks. Yeah, Distributed rendering is pretty cool, especially since the vast majority of my work is single large images I use it a lot. It's not a linear increase in speed with Vray because the light cache can't be split up. So your workstation ends up doing a lot of the heavy lifting, and of course it locks up your workstation. It would be much better if you could designate a sacrifical node to handle that but it's better than nothing.

Very interesting! Too bad you couldn't do something like use a VM on a beefy system to partition out say 2 cores for other work while the rest rendered.

I have always wanted to get render farm information on the main site. I tried a few years ago, but if you read my Microsoft Surface to Surface Pro thoughts, I'm not so good at "art" (also why the STH logo is :-/) Ended up having to put into the category of a use case for the hardware that I currently cannot show off appropriately.

 

[renderfarmer]

Very interesting! Too bad you couldn't do something like use a VM on a beefy system to partition out say 2 cores for other work while the rest rendered.

I have always wanted to get render farm information on the main site. I tried a few years ago, but if you read my Microsoft Surface to Surface Pro thoughts, I'm not so good at "art" (also why the STH logo is :-/) Ended up having to put into the category of a use case for the hardware that I currently cannot show off appropriately.

Yeah, the distributed function requires your workstation to be the initiator and take part. It's still better than nothing and with this much power I don't end up waiting around very long ;-)

Really high end packages like SolidAngle's Arnold and Pixar's Renderman don't have any DR functionality whatsoever. They're purely focused on Visual Effects and Animation. I find it ironic because I hear that some of the frames for movies can take 24hours to render. I pitty the artist that has to wait around that long to see a finished frame before he can make some minor changes and then wash, rinse, repeat...

 

[renderfarmer]

I have always wanted to get render farm information on the main site. I tried a few years ago

If I'm not busy with work and you have something specific in mind I'd be happy to help you out.

My current pet project is getting diskless booting to work. It would be so much easier to administer my nodes if I could make a change once and clone a bunch of VHD's rather than logging into each node and doing it long hand.

 

[wyluliraven]

Patrick, I vote we keep this guy.

renderfarmer, awesome setup! very well executed and thought out. I was honestly expecting an AMD setup, not quite a 2P E5 setup however it made me smile!

My only question/critique in regards to the nodes and host, is why are you not running these chips in quad channel? Cost or oversight? You could potentially shave quite a bit of time allowing the machine full memory bandwidth.

 

[renderfarmer]

renderfarmer, awesome setup! very well executed and thought out. I was honestly expecting an AMD setup, not quite a 2P E5 setup however it made me smile!

Thanks! I actually tested AMD chips when I was planning this farm (Early 2012) and I found them to be a worse value proposition for rendering than Intel's offerings.

My old farm was two dual Nehalem X5650 machines. For the new farm I had a choice of either E5 (Romley) or Magny Cours. I picked up a second hand Magny Cours 6128 off of ebay and an Open box UP board from newegg for testing. I then put together a Romley system (had to buy new as they had just shipped after months of delays). I took three different rendering scenarios (interior, exterior, studio) and ran them all at 4 different outputs 1k-6k on all three machines. On a per GHz basis the E5 came out to 1.09x as fast as the Nehalem while Magny Cours came in at 0.77x. I then spec'd out systems with both chips and I had to go with a quad socket Magny Cours system to be competitive with a similarly priced E5. But there was another big factor in my value decision: resellabilty.

The last straw for AMD was when I tried selling the Magny Cours back on eBay... It was up there for nearly a month before someone bought it from me and I was asking for a very low price. I could flip any intel chip in 15min flat at that kind of a discounted price...

For the past year the farm has been filled with E5-2620 chips. I only recently swapped them out for 8 cores. I'll be putting up the E5-2620s up on ebay tomorrow.

My only question/critique in regards to the nodes and host, is why are you not running these chips in quad channel? Cost or oversight? You could potentially shave quite a bit of time allowing the machine full memory bandwidth.

Great catch. I have the dimms arranged in that way because that's how the mobo manual says to arrange them. I tried doing one stick per bank but it still registers as Dual Channel.

The bright side is that I studied this carefully and my findings were that there is no impact on rendering speed whatsoever irrespective of the RAM configuration.

I ran a rendering test using four RAM configs on the same system.

1. 4x8GB ECC RDIMM @ 1600 (dual channel)
2. 4x8GB UDIMM @ 1600 (dual channel)
3. 4x8GB UDIMM @ 1333 (dual channel)
4. 4x8GB UDIMM @ 1600 (quad channel)
5. 8x8GB UDIMM @ 1333 (quad channel)

All rendering times were within 0.2% of each other and not necessarily favoring the 1600 or quad channel; statsictially insignificant.

 

[wyluliraven]

For power savings, going from Westmere X5650s to Romley E5s is a significant jump. In regards to how the Magny Cours performed, that sounds about right, although those 6128 are the babies of the bunch. You obviously did your due diligences in your day to day executions and requirements. Something that is almost a lost art nowadays.

At least your MC chips moved on eBay, basically only the folding crews buys them now, or people heavily reliant on multiple, _multiple_ VM environments.

Interesting that memory configuration doesn't play into the matter with your rendering, but hey! It saved you some cash right?

 

[renderfarmer]

For power savings, going from Westmere X5650s to Romley E5s is a significant jump. In regards to how the Magny Cours performed, that sounds about right, although those 6128 are the babies of the bunch. You obviously did your due diligences in your day to day executions and requirements. Something that is almost a lost art nowadays.

It cost me less than $50 (mostly shipping fees to return the mobo and heatsink) + a little time to find out that I was buying into the right platform.

At least your MC chips moved on eBay, basically only the folding crews buys them now, or people heavily reliant on multiple, _multiple_ VM environments.

Yeah, and I didn't lose much more than the eBay/Paypal fees. I got the CPU for like $80 back when they were $350 or so retail.

Interesting that memory configuration doesn't play into the matter with your rendering, but hey! It saved you some cash right?

That always surpised me too. But it makes sense given how CPU biased ray tracing is. I'm sure that at some point the memory would be a bottleneck but if it isn't on an E5-2670 then I don't know what would... I did save about $50/32GB.

 

[nitrobass24]

Do you know if your software would run on a Kerrighed machine?

 

[renderfarmer]

Do you know if your software would run on a Kerrighed machine?

I tried installing Fedora 18 yesterday (and I mean all frickin' day) and wasn't able to in the end. So I wouldn't have very high hopes for Kerrighed which looks to be based on a much older kernel.

I've often thought of clustering my machines but in the end, Vray already has a descent distributed rendering mode (doesn't scale linearly but it's still a significant boost). I usually have several shots as deliverables to render of any one scene so sending each frame to a different machine makes optimal use of them.

 

[awedio]

C l e a n !

 

[Lost-Benji]

I want that rack, it's so cute.

Seem to be a little rare though....

 

[renderfarmer]

I want that rack, it's so cute.

Seem to be a little rare though....

I got mine form CompSource. It's a great rack, only thing it's lacking is a pass-through from front to back - meaning, if you have a switch with power/signal connectors on opposit sides there is no means to pass a cable to the back from the front unless you leave a 1U slot open or drill a hole in the base.

 

[Lost-Benji]

Here is Aus, not many of them. I found a few for $700+ AUD + freight.

Was going to look at building something like it so I can run my workstation (4RU Chenbro case) in it along with my servers, UPS and networking gear for when I go to LAN parties. I like the idea of just wheeling it in, hook up cables ad play. Same for when I go do shows or concerts (sound editing/playback).

 

[renderfarmer]

Here is Aus, not many of them. I found a few for $700+ AUD + freight.

Was going to look at building something like it so I can run my workstation (4RU Chenbro case) in it along with my servers, UPS and networking gear for when I go to LAN parties. I like the idea of just wheeling it in, hook up cables ad play. Same for when I go do shows or concerts (sound editing/playback).

Sounds good, but keep in mind the rack alone weighs well over 50Kg. The casters are ok to move the case from one end of your server room to the other but I wouln't really consider it portable beyond that.

 

[Lost-Benji]

I was looking to build something like it, but with larger pneumatic tyres for clearing cables, door jams and loading into ute or van. Frame was to be Aluminum with fixed side panels lined with sound deadener and form absorption. Front and rear doors where going to be double-glazed laminate glass front, solid metal or same glass again but only as a half-door style. All cables where to pass-through a lower section of rear that is fixed and has dust/sound restrictions.

Nothing worse than listening to a Boeing 747 at ground-idle sitting near you (I run up to 10M of active USB extensions, sound, and video).

 

[Eric1b]

I was looking to build something like it, but with larger pneumatic tyres for clearing cables, door jams and loading into ute or van. Frame was to be Aluminum with fixed side panels lined with sound deadener and form absorption. Front and rear doors where going to be double-glazed laminate glass front, solid metal or same glass again but only as a half-door style. All cables where to pass-through a lower section of rear that is fixed and has dust/sound restrictions.

Nothing worse than listening to a Boeing 747 at ground-idle sitting near you (I run up to 10M of active USB extensions, sound, and video).

I’m planing to build a render farm for myself and just find a render farm company in the UK on Google. Here is some data from the website:

Six nodes withi7-4770 CPU:

Idle noise: ~46-48 dB
Idle consumption: ~ 150 Wh
CPU's temperatures: ~ 25-30 °C

100% load noise : ~48-50 dB
100% load (rendering) consumption : ~ 650 Wh
100% load (rendering) CPU temperatures : ~ 65-70°C

It doesn't look like a noisy system. I'm sure it isn't for everyone. They are selling cases too. Worth to check their website : Modular Creation render farm

 

[MiniKnight]

@Eric1b to me that is like perfect. Too bad @Spotswood doesn't make something like that.

 

[gigatexal]

yeah but that system is like 1500 USD with current exchange rates? Meh. Better to have a bunch of boxes with a giant noctua cooler on it and call it good.