So...how many 2.5" disks can you put in a Chenbro NR40700?

[nexox]

I can't help but think those fans would be a lot more efficient with some kind of cowling, even just partially with curved triangles coming in from the sides between fans and maybe some large fillets in the corners.

 

[kapone]

I can't help but think those fans would be a lot more efficient with some kind of cowling, even just partially with curved triangles coming in from the sides between fans and maybe some large fillets in the corners.

I thought about that, but the spacing is quite tight now, there's literally ~0.3" of space between fans and the box walls (vertical side). When I was doing temporary testing, it did look like the fans work just fine with no cowling (remember these are ceiling fans...). I think it might be the curvature of the fan blades and the fact they have ~5" of space to draw air from. On a ceiling, it'd be even less.

 

[nexox]

Right, a ceiling fan isn't going for any static pressure either, not sure what your setup will need, but allowing (relatively) high pressure air to spill over the blade tips to the low pressure side of the blades won't be great for forcing airflow through restrictions. Come to think of it the blade tips passing each other might also make some extra noise too, depends on rotation speed.

 

[kapone]

Right, a ceiling fan isn't going for any static pressure either, not sure what your setup will need, but allowing (relatively) high pressure air to spill over the blade tips to the low pressure side of the blades won't be great for forcing airflow through restrictions. Come to think of it the blade tips passing each other might also make some extra noise too, depends on rotation speed.

That certainly is a concern, and I won't know till I finish the frames and mount them. But yeah, static pressure is not needed in this case, just airflow. All of the compute sleds (They're all open frame, with no chassis) have active CPU coolers on them (Dell...) and are spaced 3U apart.

Only the low power cluster is spaced 1U apart and those boards do have passive heatsinks on the CPUs, but these really are low powered (ranging from 35-65w CPUs) and will never really be working too hard.

The low power cluster is really just a playground for testing stuff.

 

[kapone]

Mounted the first frame with 3x fans. *WHEW* No turbulence issues, the airflow is fantastic, and while not exactly silent (there's a boatload of airflow, most of the sound is just air being beaten), it's very quiet. From more than 2ft away, there's no issues with sound.

My dimensions for the frame were just a hair off, but nothing that was a deal beaker.



Now build the other one...gah! But, I'm stoked! This is rack level cooling and it seems to be working great from a physical standpoint. Once the compute sleds are in, will take measurements before/after.

 

[kapone]

Getting there...The second box frame with 3x fans is in, the "big" compute nodes are in (dual CPU, 256 to 512GB RAM), the two "workstations" that can handle multiple GPUs are in...just gotta finish the "small" nodes cluster.

In total:

- 9x big compute nodes
- 7x small compute nodes
- 2x GPU workstations.



p.s. Yes, the space is messy right now, it'll get cleaned up.

 

[kapone]

btw...the HP PDBs use 16 gauge wires...they're thick...and stiff...and a pain in the rear to work with. And I had to repin ~14 PDBs...my hands are sore!

 

[kapone]

The pattern is full. Both, storage and compute racks finished, wired up and tested.

(Ignore that sole yellow ethernet cable just hanging out there. It's temporary)



(The compute rack below is too tall and too close to take a single shot)







Filters in place for both racks.

 

[itronin]

Its beautiful!
As always maximizing your real-estate!
some of your shelves appear to have two nodes - those that do are they TMM's?
You gonna swap the red twisties for velcro?
Those 40/56 DACS certainly are thickboys aren't they?
I started collecting those and then decided to look harder for AOC's.

 

[kapone]

Its beautiful!
As always maximizing your real-estate!
some of your shelves appear to have two nodes - those that do are they TMM's?
You gonna swap the red twisties for velcro?
Those 40/56 DACS certainly are thickboys aren't they?
I started collecting those and then decided to look harder for AOC's.

Gracias. It's designed to be...functional, not a homelab (and it's not just a lab, it's production infra as well) that lives only on Instagram...

So, the shelves where you have two "nodes", is actually just two separate motherboards. The "small" nodes use Supermicro X9SPU-F motherboards (E3-1270 v2/16-32GB RAM, don't need more horsepower for these), which is a "baby AT" form factor, according to Supermicro. i.e. It's ~8" wide (but 13" long). So, that board and an ITX board fit perfectly on a single shelf. I'm using Gigabyte B75-TN thin mini ITX boards in these, because they take direct 12v in (no ATX PSU). Each Supermicro board has a 160w pico-psu.

The entire "small" cluster runs off of 12v, using a mining breakout board and a 750w HP PSU. (This is mounted on the side of the compute rack, I installed the 750w version of the PSU, since I took this picture).

These small machines get used for a bunch of things that don't need serious horsepower and are not always active, but turned on/off automatically via IPMI. The Gigabyte boards support S3 sleep and WOL, so they get woken up using WOL.

As far as the twisties vs velcro...I didn't know what the final wiring layout is gonna look like, until it got done...so twisties it is... Someday I might clean them up with velcro strips...but that's way down the priority list.

Yeah, I'm not fond of the QSFP DAC cables, but...they're functional and they were cheap. I think I paid like $4-5/ea. for 3m 56gb versions.

 

[kapone]

Ahh...one of the two Chenbros is...misbehaving.

Before I racked everything, everything was tested outside, double checked and then racked (I'm not particularly fond of re-racking 100lb boxes...). The primary storage server (the other Chenbro) was installed, configured and was being put through its paces for the last month or so, and it has been performing beautifully.

The other Chenbro...I fired it up 2 days ago to install and configure it as a mirror, and...the power button just makes go into a turn-on-off loop. Doesn't even get to the BIOS or anything. What the hell??

- Took it out of the rack to inspect and see if something's shorting out or something...and it turns on just fine outside the rack. WTH?
- Put it back in the rack - back to the loop...grrrr....
- Pulled it out, took off the lid to see if something is not right...and pushed it back in without the lid...back to loop...grrr..
- After several debugging tries of pulling the server out (but not completely) and pushing it back in...if I pull it out just about half way....it'll boot just fine. If push it back in where more than half the server is in the rack....back to loop!

This is very weird. Yeah, there's wires on the "sides" of the chassis inside, as you can see:

(This is an older pic, there's actually more stuff in the chassis now. I had to add two 12v-5v step-down converters as the 5v supply wasn't enough from the HP PDUs).

But...nothing is touching anything that it's not supposed to, from a visual inspection. There has to be something that's shorting out once the server is fully pushed into the rack and there's pressure from the sides and/or top/bottom, but doesn't seem easy to figure out.

Both servers were built almost identically, so I'm a bit flummoxed. Other than tearing up the server and rebuilding it again, anyone have any bright ideas?

 

[nabsltd]

There has to be something that's shorting out once the server is fully pushed into the rack and there's pressure from the sides and/or top/bottom, but doesn't seem easy to figure out.

In my experience, any properly racked server won't get any "pressure" from the sides, top, or bottom unless something is out of spec (e.g., rack width, rails bent, etc.) or not installed correctly. Even if there is something a bit off, I've never seen it be able to do enough to the inside of a chassis (e.g., bend the motherboard or move a wire). By the time it's off far enough to do that, it should be really hard to rack the server.

But, I still say look at everything in the actual "rack hardware" realm. If you can, move the rails to another location in the rack (even up or down part of an RU might be enough) to test for a weird out of spec issue.

 

[kapone]

In my experience, any properly racked server won't get any "pressure" from the sides, top, or bottom unless something is out of spec (e.g., rack width, rails bent, etc.) or not installed correctly. Even if there is something a bit off, I've never seen it be able to do enough to the inside of a chassis (e.g., bend the motherboard or move a wire). By the time it's off far enough to do that, it should be really hard to rack the server.

But, I still say look at everything in the actual "rack hardware" realm. If you can, move the rails to another location in the rack (even up or down part of an RU might be enough) to test for a weird out of spec issue.

Yeah, I know. I just have a bad feeling that it's going to be something trivial, but to find it, I'll end up rebuilding the damn thing.

 

[kapone]

Yeah, I know. I just have a bad feeling that it's going to be something trivial, but to find it, I'll end up rebuilding the damn thing.

The PWR BTN (front panel) cable polarity was reversed in this Chenbro...

Still not sure why that would cause these issues, but that seemed to have fixed it (without rebuilding the damn thing...)

 

[kapone]

Been optimizing the networks at home. I have two WAN networks now, a FIOS 2gb symmetric and a Comcast metro circuit at 10gb symmetric (+ they throw in an additional 1gb line because...hey why not? )

Optimized the FIOS network this weekend:



Getting almost 2.5gb/s up and down with zero jitter. And Ashburn VA is like a hop skip and jump from me, so the pings vary between 3-5ms depending on the time of the day.

Pleased with the weekend work.

 

[jode]

Been optimizing the networks at home. I have two WAN networks now, a FIOS 2gb symmetric and a Comcast metro circuit at 10gb symmetric (+ they throw in an additional 1gb line because...hey why not? )

Optimized the FIOS network this weekend:

Getting almost 2.5gb/s up and down with zero jitter. And Ashburn VA is like a hop skip and jump from me, so the pings vary between 3-5ms depending on the time of the day.

Pleased with the weekend work.

The speed tests work great on FIOS in NoVA. I have noticed that recently (for the last couple of months) I cannot get better than 100mbit from any actual service I use. Examples include Linux upgrades (Fedora, Debian, etc.) that dropped from 70-80 MB/s to ~11MB/s.

 

[kapone]

The speed tests work great on FIOS in NoVA. I have noticed that recently (for the last couple of months) I cannot get better than 100mbit from any actual service I use. Examples include Linux upgrades (Fedora, Debian, etc.) that dropped from 70-80 MB/s to ~11MB/s.

I can’t say the same. I’ve downloaded from Microsoft, Apple, Canonical, etc at > 700MBps at times.