The BookShelf Server

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

Kurly_B

New Member
May 26, 2013
17
0
0
The Bahamas
Build’s Name: The BookShelf
Operating System/ Storage Platform: Funtoo / XEN / ZFS
CPU: (2) E5-2620
Motherboard: Supermicro X9DRI-LN4F+
Chassis: Wood You 78x42x14 Bookshelf
Drives: 4 x Seagate SV35 2 TB, 4 x WD WD20EARS 2 TB
RAM: 32GB Kingston 8GB ECC Registered 1600
Add-in Cards: LSI 9201-16i, Silverstone USB 3.0 addin card, Old g210 nVida Video
Power Supply: Corsair HX 1050
Other Bits: (3) 4 x 3.5 Drive Rack, Silverstone 4 x 2.5 + slimline dvd in 5.25 Rack.

Usage Profile: Back-ups, remote file access, hosting Windows 7 VM transcoding, Windows Home server 2011 for itunes server and front end to zfs storage, pfSense Firewall.

Other information…
Building this into a bookshelf to consolidate all the devices, info and services that I currently have, Including the 7 - 8 misc hard drives that I have laying around containing pictures, videos, archived emails, old bussiness docs, etc.
Basically setting up a central space for myself, and my family, to store and manage our digital lives.

I'm waiting on the rest of my parts, currently have
Motherboard and heat sinks - (2) Supermicro P0050AP4
Waiting on the cpu's, ram, and power supply to be delivered, should get them next week.

I will post some images as soon as I find a image hosting service. And suggestions?
 

gavin

New Member
May 26, 2013
23
0
0
Kurly,

Looking forward to watching your project & subscribed!
I got the shelves in my zfs 'server' project but your server's specs much better.
Have you done any other wood project before?

RE image hosting site -- I use this free one, working okay so far TinyPic - Free Image Hosting, Photo Sharing & Video Hosting the only annoyance is you need to verify-human each time. If you are a serious uploader could be an annoyance. Not too bad for just a few.
 

Kurly_B

New Member
May 26, 2013
17
0
0
The Bahamas
Yeah, I saw that..... Funny Stuff.
I love the hanging bottle.
I would do something like that if I didn't have a one year old..... She would have the bottle open, server on the floor and crying because she couldn't play with it anyone.

I currently have the server running with a different board in the setup right now. This is an upgrade, a massive one.

This is the sketckup layout for this board, haven't downloaded the pics from the camera yet.



This is the back of the shelf air flow layout.



This is everything with door not on.



As you can see, I've been planning this a long time, since last year in fact.

Now that I have the board, I'm taking it to work so the carpenters and I can plan the mounting, for it. We have some nice ipe we're installing on the floors, nice stuff dense as hell, high silica content. It will make a great board mount. BTW this board is 13" x 13.5", not really a standard mount in anything but Supermicro Revision 'M' case.

Any feed back on the air flow will be appreciated. The intake fans are Corsair SP120 62.74 CFM fans. The Exhaust fans are Corsair AF120 39.88 CFM. I'm hoping to build up the static pressure inside the box. Intake = 376.44 CFM - Exhaust = 119.64 CFM + whatever the power supply is. I'm looking at this from an HVAC view point, as I'm familiar with those things.

I will use this site as my build log.
 

Patrick

Administrator
Staff member
Dec 21, 2010
12,511
5,792
113
Wow! This looks really cool!

Had a few thoughts/ questions:

  • Something I was thinking when I read this is that the airflow is going to change if you stick a PCIe card in there, especially a large hot one like a GPU. Ever think about flipping the motherboard 90 degrees counterclockwise?
  • Cable lengths are going to be significant. Not something that is a huge deal but those are fairly long runs
  • What are you doing regarding dust?

Looks great though!!!
 

Kurly_B

New Member
May 26, 2013
17
0
0
The Bahamas
Yeah, I think so too.
But I kept buying an item here, piece there, all, each and every item or part was a to fit a consumer board. And living where I do, Customs and Duty, local suppliers would charge me the same for a consumer level setup, I know, I checked. Example, they wanted $400 for a I5 V1. Not even joking. And everything that I want to play with is in the server realm, IOMMU, SR-VIO, VT-D PCI Pass Through, ZFS. So I said screw it.
I know it's over kill to a degree, but building a foundation for the future. I plan to use this setup for the next 5 years, or more if I can get away with it. Computers are only obsolete if they don't do what you want.
 

Kurly_B

New Member
May 26, 2013
17
0
0
The Bahamas
Current P5N-T

This is my current system, which I have partially disassembled.



This is a ISO view airflow mockup with the LSI and Future (4) single slot video cards.



Here's the side view.



The airflow might be an issue for the 1st cpu, I might construct a duct to route the air around the expansion cards.



I never finished the original build because I knew that I was going to change the motherboard.
Now I can complete the build as the parts come in.

Tonight I will be making the templates for the carpenters to fab the ipe board mount next week.

Here's a guestimate of the final look



Current Parts Pic:
 
Last edited:

Lost-Benji

Member
Jan 21, 2013
424
23
18
The arse end of the planet
I can see a headache coming, venting from and too the rear of the bookshelf will mean that you will need to keep the bookshelf well off the wall and you will also likely get airflow just recirculating. Pulling air from low to the floor is also just the same as having a floor-dwelling PC that fills with dust way to quick. Filtering is a must.

May I suggest drawing airflow from and through the bottom shelf (the one at bottom of the bay you are going to use) and venting waste heat out one or both sides at top of bay?
If you feed the PSU and VGA out say the right side (as you look at it) and use say a pair of good 120mm fans as draws in the bottom, providing you run a perspex or polycarbonate front panel, it will see a positive pressure inside to keep dust out and positive airflow through the PSU and the GPU card.

Oh, if you are going to Overkill it with that board, I personally would ditch the slow CPU's in favour of a faster single chip for now, add another later when demand for slots and power increases.
Also, allow for a decent mid-range GPU nVidiot card to allow on-GPU trans-coding or offline media conversion.
 

Kurly_B

New Member
May 26, 2013
17
0
0
The Bahamas
The bookshelf is 6 inches from the wall and vented at top.
As far as the CPU's go, already ordered them, they will be on island Monday - Tuesday, along with the ram, and power supply.
I was planning on an ATI 7750 single slot cards. The internet consensus is that xen works best with ATI Cards doing PCI Forwarding.
From what I can gather with nVidia cards, you need to use their professional cards to do gpu pass through.
Here's the xen wiki link
Xen VGA Passthrough - Xen
I Currently have an old nVidia G210, and was going to try it.
 
Last edited:

gavin

New Member
May 26, 2013
23
0
0
Kurly,
Looking good, I like your drawings too, what software did you use for them?

How did you decide on the current exhaust design because I will need some in my project but I haven't even started ventilation research yet. I was thinking like a single big pipe for each channel (one for intake, one for exhaust).

Are your fans going to be on 24/7? I briefly looked into sensor based automated fan control systems with PWM (copied some links below) but haven't tried any of these yet.
Due to article age (2001 / 2003 etc.) some microchips not even available in the market any more - one of the links offer a method of identifying a replacement chip on a huge electronics supplier web site but when I gave it a quick try I ended up with 900 results, so all this still require time to investigate / decide / test etc.

DIY Fan Temperature Control How-To
Temperature Controlled Fan System for Cabinet
SPCR • View topic - Custom microcontroller fan controller - work in progress
SPCR • View topic - DIY Temperature Controlled Case Fan Controller Design
SPCR • View topic - PWM to Voltage bridge
Guide - PWM Fan controller | bit-tech.net

From these drawings, I have a suspicion that you may have difficulty with keeping this cool, details surely depend on server location & climate/server load etc. but just a preliminary thought...
Also from what I briefly read about air control, rather than having one big box, it might be a better idea to have air channels and direct each fan to a specific purpose. As you too probably have done - this specialized fan channel structure is what I observed in some high class server cases. [You have only one acrylic looking cover, so it might be better to have more perhaps].

Otherwise looking all good!
 

Kurly_B

New Member
May 26, 2013
17
0
0
The Bahamas
AirFlow

The air flow is an issue. The more I think about it, the more I'm glad I haven't actually cut the bookcase yet.

I relaid out the components vertically, io port on top, in sketchup, but the cpu's are inline. The 2nd cpu flows into the 1st cpu.
This would be the same with the IO ports to the bottom.

Splitting the heat sink discharges by ducting the separately might be the best way, or get (2) liquid coolers like the H80.
This board has the normal square lga 2011 mounts. That would solve the cpu air flow issue.
 

Lost-Benji

Member
Jan 21, 2013
424
23
18
The arse end of the planet
Why the need for four GPU's?
As for nVidiots, I have windows based servers including one with Win8 on it. I have a small 430 card in it for media playback on TV but I also RDP into it for P2P management/downloading. I have used things like Handbrake but found it was crap at processor use/scheduling. I have found "Any Video Convertor" to be a better option as it can use 100% of the CPU and cores as well as have a GPU offload to nVidiots (CUDA) for X264/H264 encoding. Works well for me.

The idea of single slot cards are nice but they still have the same issue as double-slot cards and the lack of good airflow.

Have you considered water cooling?
Example, use blocks on both CPU's and the GPU/GPU's and then run a heat-exchanger RAD in the same chamber as the board. This will allow a sealed chamber for the system board (and associated attached bits) to keep dust and moisture (humid area with sea and salt spray equals short life-span).
Another option is total immersion cooling, turn system vertical (DIMM's and GPU's Vertical) with the I/O panel and GPU monitor ports facing up.
immersion PC cooling - Google Search
The ultimate cooling and keeping things free of dust and corrosion.
 

Kurly_B

New Member
May 26, 2013
17
0
0
The Bahamas
nVidiots, that's funny.
Yeah my g210 transcodes fine. The (4) gpu are just for future maybe, haven't committed to anything.
I wanted to have the option at a later date. This is my hobby, fun, therapy, what have you. Just wanted the options, cause lets face it, how often does one build a box like this. In the last 2 years I've got into bash scripting heavily ( it keep me sane during my wife's first pregnancy and delivery), virtual machine creation / management, transcoding (handbrake) scripting, zfs scripting. Just playing, something away from my real job and the kids.

This build is really just to get my digital house in order, and then have enough power left over for my to setup vm 's and play with stuff, networking, pxe, dns, firewalls, what not. That's why I don't think it's overkill. Enough room for daddy to still play and not have to worry about mommy and the kids watching itunes streams, music, and pics. In Nassau, space is a premium, my condo is 600 square feet, including the balcony. Not alot of room for boxes laying around, hence the integrated bookshelf thing.

After your posts, I really started looking at the cooling, and did a redesign, here it is:






Still working out the whole hard drive side air flow thing. Not sure how to route it all.
 

PigLover

Moderator
Jan 26, 2011
3,184
1,545
113
Looking very nice.

You are up to 11 fans (10 120/140mm intake/exhaust fans and the one embedded in your PSU). You're going to have a devil of a time keeping that many fans "living room quiet". You should make sure you have rock solid fan speed controls built in 'cuz you're going to constantly have to balance running them slow & quiet vs fast enough to get the job done.

Rotating the MB will certainly give you better airflow over the PCIe cards (GPU or otherwise). But with the new design how do you get the cables out? Keyboard and mouse could be wireless USB dongle, which would very cute. But there's video, sound, whatever else comes off the PCIe slots. And how are you routing SATA cables from the MB to all those disks? Will your cabling arrangement leave you the option to hotplug remove/replace a drive? Or will the power or SATA cables from neighboring drives be in the way? You've put a lot of forethought into airflow patterns, which is good, but before you actually start building you need to put an almost equal amount of thought into cable management.

You also probably want USB and USB-card-reader somewhere on the bookshelf so you can plug things in w/out having to open it up.
 
Last edited:

gavin

New Member
May 26, 2013
23
0
0
I like the new design but as PigLover commented with all these fans (if they are not silent/slow fans) it might get too noisy.
One question, what software are you using for drawings?
 

Biren78

Active Member
Jan 16, 2013
550
94
28
Yeah, I think so too.
But I kept buying an item here, piece there, all, each and every item or part was a to fit a consumer board. And living where I do, Customs and Duty, local suppliers would charge me the same for a consumer level setup, I know, I checked. Example, they wanted $400 for a I5 V1. Not even joking. And everything that I want to play with is in the server realm, IOMMU, SR-VIO, VT-D PCI Pass Through, ZFS. So I said screw it.
I know it's over kill to a degree, but building a foundation for the future. I plan to use this setup for the next 5 years, or more if I can get away with it. Computers are only obsolete if they don't do what you want.
I am surprised there isn't a way for people to send you stuff from the US inexpensively. At that cost you could buy a plane ticket with 10 chips easily.
 

Kurly_B

New Member
May 26, 2013
17
0
0
The Bahamas
I just use the free version of Sketchup. Does everything that I need. Revit would be nice, but at $5,000 a seat, I think I'll just keep using Sketchup.

@gavin - Yeah I know the fan / noise issue is coming down the pipe. Really not sure about how to deal with it. Still working it out. The Board has (8) PWM fan connectors, but I don't have any PWM fans. I could just bite the bullet and get some Noctua PWM fans later on. I do not want anything the I have to fiddle with. I could build a thermistor / temperature relay fan control that only turns on certain fans when needed.
Basically, the fan will be addressed after I get the thing setup and during the working out the kinks phase, with the reduction in air flow change of direction, it might not be that bad. The expanded metal at the filter will disrupt it, by how much I will have to see. Air flow disruption = noise.

@ Mr PigLover - I'm drawing the cable routing right now, have to plan that to minimize the air flow problems. I have a nice route for the internal cabling, nice thing about it is the LSI uses (4) SFF-8484 connectors, and the onboard SCU uses (1) SFF-8484. And I plan to use the (2) onboard sata 3 as boot drives, so that is only (7) cables coming out of the motherboard area.
As for the IO stuff. After the initial setup, this is headless, dedicated IPMI.
Power is really simple, power supply cables are about the perfect length, no coiling needed, just some strapping to the sides. Hard drive power is not to bad, a little bit of kludging, but nothing ugly.
I wil finish the drawing and post it. Still waiting on parts to even start it up. Calm before the storm.

BYW PigLover, nice name..... LOL. Love to hear the story on that one.

Thanks everyone for your input.
 

Kurly_B

New Member
May 26, 2013
17
0
0
The Bahamas
@ Biren78 - It's not the shipping, it's customs and duty. 10% on the item including the shipping. Example, $200 cpu, $50 shipping, C & D is $25. So the item just cost me $275, that's 33% over cost of item.
10% duty is only on computer parts, other stuff is anywhere from free to 85% duty. And they change the duty rates on stuff a lot. Last year computer parts were duty free.
Don't even get me started on food. (2) people - (1) kid = $300 a week at the store.
Cost of a ticket to the States is $225, rent a car = $75, fuel = $20, food = $30 , now your taking money $350 just to pick up an item. And deal with DHS, not really fun.

I just kinda gave up and start using a freight forwarder, buy in the states and ship it to their address, but it cost money. This build will cost my 33% over cost state side. $2000 = $3000. Just the way it is. Thats why I don't thinks it's over kill. Buy it once and be done with it.
 
Last edited:

Lost-Benji

Member
Jan 21, 2013
424
23
18
The arse end of the planet
Hey Kurls, I was looking at your revised layout and you are nearly there and certainly on the money.

The boards are designed to have airflow from the front edge to the back edge (bottom to top in your new layout) allowing airflow to run along and through the DIMM's and other devices.

I see the other gents have mentioned the fan-count. There is a very simple way to cut down on them, loose 7 and your golden. Take away the 4 that are behind the drives and loose the fans at the top of the enclosure. You only need the 6 fans across the bottom areas to "PULL" air through the dust filters. You have already covered the system so the airflow is guided. On the drive side, two options; the first is to mount the drives on their edges to allow vertical flow past them. The other is to actually extend the shelf front or back edge to duct the air up either the front and then through or up and behind to come forward. Picture the airflow snaking its way forward and backward to eventually vent through the top.

As for the engine side, just the 3x 120's or 140's will flow enough airflow to keep things sane and happy. User something like the Nocturas or Arctics for the CPU fans. The system will sit very quiet and maybe be noticeable when you start flogging it with a workload.
 

Kurly_B

New Member
May 26, 2013
17
0
0
The Bahamas
Funny you mention the fans, I was researching the Noctura 140mm last night, and added 4 of them to my amazon cart. Not sure about the model though, any insight. The NF-A14 PWM seems to be the more general use fans, has all the latest advances in their fan technology, and are reasonably priced @ $23.95 per fan. ( as far as Noctura fans can be ).
The NF-A14 PWM fans are:
82.5 CFM per fan x ( 4 ) = 330 CFM @ 1500 RPM @ 24.6 dB
68.0 CFM per fan x ( 4 ) = 272 CFM @ 1200 RPM @ 19.2 dB
Is this enough air flow?

BTW, anyone try this power monitor or have a recommendation for one. It has a 15 amp capacity, the max for a typical residential outlet. I want to know the total system draw @ idle / basic serving, and under full load / transcoded / kernel compiling / benchmarking.