12 Node, 24 Drive Bay, LCD with BMC in 3U Case with 10G Switch

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

JustinH

Active Member
Jan 21, 2015
124
76
28
48
Singapore
Hi All,
I thought you all might like this Build I just finished.



Build’s Name:
Operating System/ Storage Platform: Linux (armbian)
CPU: 6 x NanoPI M4v2, 5 Raspberry 3 CM+, 1 RK3399 Development Board with 7" LCD and 4G/LTE Uplink
Chassis: intel p4400
Drives: 3 x 8 2.5" Bay SATA Icydock (total 24 Drives)
RAM: 25Gig total
Add-in Cards: SATA HAT for the NanoPI SBC's
Power Supply: Redundant 480W PSW
Other Bits: Lots!

Background:
I had a few of the NanoPI M4 boards and decided to make a cluster out of them! This sort of grew out of control compared to what I've ended up with!

The CPU on the NanoPI's are 64-bit Dual Core Cortex-A72 + Quad Core Cortex-A53 and clock upto 2Ghz (on the A72) and 1.5Ghz on the A53. Memory is 4Gb and also has Ethernet, USB3.0 and Wifi/Bluetooth. I've added a 64Gb EMMC drive (as boot) and the SATA hat which gives 4 SATA ports and its total is around $110 in total.

The Raspberry CM3+ boards are the 32Gb EMMC version, and loaded onto a MiniNodes Cluster Board, which has a USB Ethernet Chip per board, and a onboard Ethernet switch, so only one uplink is required.

The RK3399 Board is also from FriendlyArm, and similar specs to the NanoPI M4 - Specifically 64-bit Dual Core Cortex-A72 + Quad Core Cortex-A53 and clock upto 2Ghz (on the A72) and 1.5Ghz on the A53. Memory is 4Gb and also has Ethernet, USB3.0 and Wifi/Bluetooth. It also contains a NVME adapter with a Adata XPG SX8200 Pro 512Gb SSD drive on board and a 4G/LTE modem. Additionally it has lots of expansion. I'm using this as a BMC/Management Node for the NanoPI's and CM3+ nodes.

The case is from a old workstation I had, a Intel P4400 Case - It has dual redundant 480W powersupplies and a Power Distribution Board. All the NanoPI's and the Mininodes Cluster Board runs off 12V, so I rewired the ATX/SATA and GPU Leads from the powersupply to fit the Nodes. I used a dremel to cut out the front door and fit the LCD, which has a short lead back to the RK3399 board. Means I can open that door as well. LCD is touch screen

For the Switch - Its a Mikrotik CCR1016-12S-1S+. (its really a router, but I'm using it in a mixed mode - All ports connected upto the Cluster are switched, and then the "Uplink" port (10G) is routed) I had to purchase a lot of RG45 SFP adapters for this!

Harddrive wise - Its a mix of old SSD drives I had lying around. Mostly 250G or 512G consumer (Samsung Evo, HP Rebranded, and a few cheap chinese brands). Final purchase will be to populate with 1Tb SSD's soon. - I'm searching for a decent deal on used drives here).

OS wise, the NanoPI's and CM3+'s run Armbian (essentially debian) for Arm based boards and currently I'm managing/monitoring the cluster with a Puppet and Zabbix.
The RK3399 is currently running Android in kiosk mode cycling through some Grafana dashboards and also able to talk to the powersupplies via i2c so I can switch on/off from the LCD or monitor power usage. I do plan to install armbian on this as well eventually and turn it into a full puppet/influx (or zabbix)/BMC type node (see below).

Remaining Steps:
  • I designed a something akin to a BMC Board to control power to individual nodes, monitor individual consumption, Drive upto 8 PWM fans with Temp Control and also has 12 Serial to USB Interfaces on it. That board has been stuck in China for about 8 weeks now. Hopefully when I get it (and I didn't screw up) I'll add it in, so I can (mainly) do work when I need console access (All Boards are configured as headless, with a Serial Console port).
  • Fans - Currently only one Noctua Fan mainly to move some air over the CM3+ boards and that Ethernet Switch Chip, which is by far the hottest thing in the case. The NanoPi's averaging around 40 degress (in around 35 degree ambient temp) and under load they are reaching around 70 degress (they start throttling at 80 degrees). Once the BMC board is in, I'll get some better fitting fans to get airflow through the entire case.
  • HDD - Get some used enterprise 1Tb (preferable) drives in there, instead of the mis-match of old stuff I have now.
  • I killed one of the NanoPI M4's (hence the hanging cables) and waiting for a replacement (so its really 11 Nodes right now!)
Challenges:
  • Occasionally I'll have a drive drop out. I suspect its the SATA cables I got (cheaply) from China, but I've just spent some time rerouting the cables to try and issolate the SATA cables from power/ethernet in case that was causing interference. If I still get drop outs, then I'll try some standard cables. I really hope its not the ICYDock though.
  • The Network Bandwidth on the CM3+ boards is pretty low. Not anywhere near what I'd expect from a RPI 3 based board. I have to investigate if its a driver/kernel issue, or just the USB Ethernet Chip they use.
  • the RJ45 SFP adapters I got also are a bit suspect. Cheap ones from China (using the same model as the legit Mikrotik RG45 SFP Adapters) and I have to force the Link Speed. I also don't get a Link Light on the NanoPI's but it still works. If I can find a few cheap alternative adapters I may try them out just for completeness sake.
Performance:
  • I've setup Gluster in on the NanoPI's with Samsung Evo 860 Drives and pushing just over 5GBytes on uplink from the switch when reading files. Writing files, it will start around 4G and eventually settle around 3.2G, so I'm pretty impressed with the SATA performance of these NanoPI's. I'll be looking forward when I get some decent drives and try again.
  • As mentioned, the Network performance of the CM3+ Cluster isnt what I expected. I'm currently booting via the EMMC storage and NFS mounting home directories and Docker Containers.
Finally - I'm sure a lot are wondering what I'm going to use this for. There is a actual business case behind this, and I built it up along some of those requirements. I'm going to use this setup for a few months, and see how it performs and how stable it is. Should it work out - I plan to build about 10 of these for my clients. The Nano PI's will be doing some "heavy" lifting, while the CM3+ boards are "mangement" nodes (web interfaces etc). The RK3399 would be a "operator" node and has a LTE/4G modem to allow remote management if needed.

Cost: Excluding the case, HDD and Mikrotik Switch (which I already had), this has cost around $1100 (including shipping). I actually think that's pretty effective given what its capable of. I've contacted a few suppliers in China to build some custom cases for this as well, and they are quoting around $400/piece which is reasonable. For actual production, I'd probably use a 1G switch would keep the cost down as well.

More Pics are available at Imgur: The magic of the Internet
 
Last edited:

ajs

Active Member
Mar 27, 2018
101
36
28
Minnesota
Have you measured power consumption at all? I would be curious to see what this setup draws idle and under load.
 

TXAG26

Active Member
Aug 2, 2016
397
120
43
Very interesting! That looks like a very clean build! Post some pictures of the inside if you can.
What types of workloads would benefit from this type of setup, compared to something like latest generation 28 core Intel Xeon/32 core AMD Epyc?