Building my first server, need some guidance

[Keljian]

I run 10gbit from my desktop to the switch and from the switch to the server.
Internet is routed through pfsense in a vm on the server, files are also served across the same 10gbit line

Perhaps look at a similar setup?

 

[ridney]

That will be the preferred setup but I don't have a 10G switch at the moment. So this P2P setup is just to tide me over until I can add a 10G switch.

 

[ridney]

So both options would look like this (simplified plan view)

Option 1

Option 2

The point here is I need to have access to Cisco 3825 Router. With option 1, if the server is down I still have access to the router. With option 2, when the server is down then I lose all connections. But for option 1, I have to run a new cable to the room where I have my desktop which is a real pain haha

What is your advise? Do you have any other ideas?

 

[ridney]

Oh man, bad news. I got the cpu cooler and installed. Fired up and no post, no display just clicking sound from the motherboard speaker. I removed one by one ram, fans, everything and still no go. I tried to power it up with just the cpu and still the same. Removed the cpu and powered up with just the board still the same clicking sound. Used another psu with just the board and still the same. Oh man, this totally sucks!

 

[ridney]

Any ideas where I can purchase this board X9SRH-7TF? I found one in Wiredzone but shipping is $120! This board sells locally here in Singapore but it's S$1000

 

[pricklypunter]

Have you bench tested the board outside the case?

 

[Alfa147x]

Thanks for the pictures and write up of your experience. With that rack do you have trouble with temps?

 

[ridney]

Have you bench tested the board outside the case?

I was in contact with supermicro tech support yesterday and they have concluded there was a problem with the board and will need to be sent for repairs. I was really depressed by it but that's life and another lesson learnt. Never buy used motherboard as much as possible. The board will be sent today, hopefully will get it back in a month.

Thanks for the pictures and write up of your experience. With that rack do you have trouble with temps?

Without AC on, these are the temps I am getting from my Dell R210 at idle

I think the rack case will be about 5-8c above ambient temps. Currently, it is like 27c room temp. So far I am not having any problems with it.

 

[ridney]

First of all, a lot of photos. Again! Apologies haha

While I'm still waiting for the motherboard to return from repairs, some of the items have arrived

Bought these Sata cables for like $4 from ebay. It says that it is certified for 1.5 gbps but not sure that it matters though I guess all cables are the same.

Plugged them into the board but now a problem emerges, it looks like the sata plugs would be interfering with the 8 bay 2.5 drive. It can't fit with these sata cables.

Closer look (oops camera has no macro, sorry)
The 8 bay drive was listed as 86mm height and this 5.25 bay cage is about 85mm so just about the same height. It looks pretty close and it's not gonna work!

I need a low profile cable that is low enough to let the 8 bay drive fit in. So gathered all sata cables I had and compared each one, I think the blue ones (from my dead R210 thank you very much) would work

Let's try it and see, it has some space that's good but I'm still not sure if it's enough. Will try to find more sata cables

Okay, unto the next one. The 8 bay drive has arrived! Wohoo. The most awaited one of all

Wrapped in plastic and some padding. Accessories (screws and keys) were taped just behind. Seems fine

It's a beauty! Front view with the 8 hot swap bays

Side view with the mounting holes. The bay is all aluminum with beautiful brushed metal but unfortunately we can't see this as this bay goes inside the case. It looks really well done, the craftsmanship is very good.

View of the back. With a 70 x 15mm fan. I have tested it and it is completely silent which is wonderful.

Brushed aluminum yum yum

2.5" HDD Tray. Listed as 15mm maximum thickness but I think it can take more thicker drives

Underside of tray. 4 mounting holes for the HDD

Some vent holes on the front which is good, not sure if it's enough

And the drives on the tray. 16tb of goodness in such a small package

Back plane with trays removed

3 x 5.25 Bay cage. Let's fit it in

It is tight fit. I may have scratched the brushed aluminum. Never mind the scratches on the table haha

Backside, plugging in 2 x sata power plugs needs a little work

 

[ridney]

Now unto the next one, Noctua NH D9DX i4 3U cpu heatsink/fan

Accessories

This is a 3U heatsink

4 heatpipes and two ILM Narrow brackets. Each bracket gives you the option to choose the direction of the fan in relation to the memory dimms. If the fan will be perpindicular to the memory dimms then two slots would not be available as the fan will over hand on top of it.

Just what I thought! Enough space for the 2.5" drives with the sata and power cables plugged in. Pretty happy at this point.

And enough space for the two dimm slots too! I thought I can't remove them without taking out the heatsink but it turns out just fine.

The heatsink is tall and I can see it just enough touching the case cover when closed. It is also pretty silent too with 92mm fan at 2000 rpm max

And here it is almost complete. I am supposed to be testing the bios and flashing the lsi 2308 now but sadly my worst fear has come true haha I am just relieved that the motherboard is on its way for repairs and hopefully they can fix it.

 

[ridney]

My 2U rack fan has arrived! It is an AC Infinity Cloudplate T7
CLOUDPLATE T7, Quiet Rack Fan System 2U with Thermal Control

What's in the box? Rack fan, a 12v dc adapter (autovolt 100-240v), 4 sets of rack screws and a manual.

The whole unit is enclosed with a powder coated steel case. Very sturdy too

Label at the back

3 screws on each on top and 3 on the bottom secures the case

Default configuration is to exhaust air out of the rack case but my intention is to push air in to help the two 140mm rack exhaust fans. A total of 38 screws to remove to access the fans and flip them

Took a few mins a set of extra pair of hands, i finally managed it.

The LCD is also very bright and can change the temps from F to C.

And now to install this into the rack

 

[ridney]

It has been a long time that I have not touched the rack and change something with it so now it is very dusty and needs to be cleaned. I don't like the feeling of installing a brand new server (although used parts haha) into the rack without cleaning the entire thing. So it's time for some cleaning.

Vacuumed much of the dust from the foam and wiped out the major dirt from the rack though not all haha I'm really getting old and just want to be done with it

So this is how it looks like now. I couldn't resist and installed the server chassis haha. It looks so big inside the rack

 

[ridney]

Well supermicro just emailed me and it's a bent pin that's gone wrong with the motherboard. Charged me $50 for it but thank God it can be repaired. Hopefully can get it back by next week.

 

[ridney]

Well nothing much is happening as I'm still waiting for the motherboard to come back from repairs. They have said it would take at least a month as it will be outsourced to some third party vendor. I think it's still two weeks and no word from them yet.

So I have a little update going, I received the chinese knock-off X540-T2 from ebay and have installed them in my PC. It is working great as of the moment but couldn't test at 10g as I have no 10g switch but 1g is working quite well with the supplied drivers, it seems to detect it as a genuine intel card so I'm happy about it.

I'm still going ahead with directly connecting this to the server once I get the board back. I have already ran some aditional cat6 cables to my room and is ready. If this setup works great I might be looking at a 10g switch. I have my eyes on a Cisco SG350XG-2F10 for a while and it's one of the items I have in my "wants" bucket list. 10g is really not necessary but I can't help it haha.

So here are some photos to share (again)

My camera doesn't really have macro or does not work well (it's a Canon G1X) so photo quality is not so great

The card looks like it is genuine though I don't have a genuine one but from my inspection, the build quality is really good. It looks like an expensive card.

Installed in my PC

I used to have a crossfire gpu cards in there but since I finished this little watercooling project some years ago I just got lazy to maintain it. It used to look better before but age has caught up and I just don't care anymore haha

 

[ridney]

My motherboard comes with an onboard lsi 2308 and I was thinking of getting an hba card, possibly another lsi 2308, for some additional drives. Or is it better to get a sas expander?

I am looking at getting this (below link). Is this fine? What sort of cable am I going to use it with?

Intel SAS2308-8i (9217-8i) 6Gbps 8Port HBA PCI-E Server SATA SAS Controller Card

 

[ridney]

just a little update

 

[ridney]

so after almost 3 months of waiting i have finally received my repaired motherboard (X9SRH-7TF) from supermicro and everything is working fine except for two dimm slots that doesn't accept memroy modules. it would not post past B7/B9 code and based from my research it's a memory error. i guess that my processor has been damaged from the bent pin as my memory modules are all fine after testing each one by one. now, i am only able to populate 6 dimms out of 8 for a total of 64gb, not too bad but someday i may have to get another E5 2670 for this.

so during these 3 months of waiting i was able to purchase

• 4 x 480gb SM943T ssds (used as single drive datastores for VMs)
• Intel X540-T2 (10gb nic for desktop, counterfeit from ebay)
• LSI 9217-8i (additional HBA, onboard mobo has onboard SAS2308, total of 16 sata 6gbps ports, Intel branded/counterfeit from ebay)
• Netgear S3300-28X-POE (heavily used from china, really regret this purchase)
• QNAP TS-831x (to replace my QNAP NAS and relegate as a backup server, comes witth 2 x SFP+)

luckily, i was able to flash both HBAs to IT mode without any problems using the uefi shell. the onboard HBA was easier to flash compared to the PCIE card as i had to delete the IR bios and firmware before i can flash it with IT. it was nerve wracking thinking if the system is powered off i would break the HBA card. anyways, i was able to install esxi 6.0 and everything worked great, no additional driver installs.

here are some photos i want to share

my all-in-one server

my current rack setup, top to bottom

1. 24 CAT6 patch panel
2. Cisco 2504 Wire Lan Controller
3. Cisco SG200-26 Swtich
4. Service Provider ONT
5. Netgear S3300-28X Switch
6. Cisco UC520 CME + POE Injectors
7. AIO Server
8. Dell R210 - powered down/backup
9. Cisco 3825 Router
10. AC Infinity fan

The noise coming from this rack is acceptable to me, i can still hear the fans but it's not so annoying anymore compared to when i had 2 x Dell R210 running. I might change the fans on the netgear switch as this is the only one that is screaming from this rack. I have ordered some replacement fans and hopefully it would make it a lot more quieter.

Just below the rack is where it put my backup nas, QNAP TS-831x. It is trully whisper quiet.

idle temps of AIO server from IPMI

And finally my XPEnology, smart working like a charm

unfortunately, i am having some trouble with a couple of my 2.5" Seagate 4tb drives. It seems they randomly disconnect from the array and I have to keep rebuilding it. not sure what to do here, so far speed is fine it's just this disconnect errors that makes the volume crash.

update: just finished rebuilding the array and decided to increase memory to 16gb and now performance has significantly improved. this is with ssd cache turned off (6 x 4tb raid 6)

so i guess that's it. this build is officially over. big thanks to STH and to everyone who tried to help. appreciate it

 

[Kybber]

unfortunately, i am having some trouble with a couple of my 2.5" Seagate 4tb drives. It seems they randomly disconnect from the array and I have to keep rebuilding it. not sure what to do here, so far speed is fine it's just this disconnect errors that makes the volume crash.

Just a thought: Could this be related to drives going to sleep or TLER? I don't know how XPEnology handles such cases, but this might explain your issues.

 

[ridney]

Im not sure honestly. first i built a storage pool under windows 10 using these same drives and my volume crashed after a week. then i moved them to my qnap and built a new array and after a day it disconnects again and crashed the volume. so regardless of software it just disconnects. out of the 6 drives, 3 of them suffer from this. smart reports drives are fine just having a few udma errors.