Intel NUC Thunderbolt networking for super low-power cluster?

billc.cn

Member
Oct 6, 2017
41
7
8
I am planning an all-flash virtual SAN cluster, so I need a minimum of two storage nodes with 10Gbps+ between them and a witness node where at least gigabit network. However, I also have a 200W total power budget which limits the node choices a lot.

As an experiment, I built a Xeon E3 1240L V5 server, which uses 33W idle and 51W at full power. It satisfies my power requirements but cost me £450 (w/ 16GB gold bars RAM sticks) after weeks of eBay sniping. It may not sound a lot, but 450x3 is definitely over my budget.

Then, I came across Intel NUCs, especially the NUC7 i5/i7 ones with Thunderbolt 3. I found a couple of eBay deals for complete, used kits with RAM and NVME SSD for <£400 each. They are not only cheaper but use almost half of the energy as well.

The only drawback is the limited expansion, but I found out you can do peer-to-peer networking over Thunderbolt and drivers are available for all major OSes. I hope with one (relatively) cheap Thunderbolt hub on each, I can connect three of them in peer-to-peer mode. (Or falling back to P2P between two and 1Gbe to the witness box.)

Has anyone attempted similar setup or Thunderbolt networking? Do you think it is doable?
 

ideabox

Member
Dec 11, 2016
69
25
18
33
Linux drivers for thunderbolt a flaky, the driver stated last time i tried last year that it should not be used in production. I know its a home lab but just keep that in mind.
The other drawback is only 1 port on each nuc and no Thunderbolt Switch/Hub exists, But you can do direct connect the 2 nodes and witness the other over 1gb.
Another option would be to grab a thunderbolt to pcie box and through a dual port 10gb card in each?

Usually some second hand Thunderbolt2 boxes come up cheap and you can grab a Thunderbolt3 to Thunderbolt2 adapter from apple circa$30.
 

billc.cn

Member
Oct 6, 2017
41
7
8
Thanks for the insight. I don't worry about the flaky drivers so much, but not be able to interconnect more than two nodes is a significant limitation.

All Thunderbolt boxes I can find on eBay are at least £100 which will push the total cost beyond that of the E3 setup. I have also considered wiring the internal m.2 connector to the outside and put a standard PCI-E NIC on it, but decided it's too much of a hassle.
 

WANg

Well-Known Member
Jun 10, 2018
848
480
63
Hmmm...A pair of HP t620 Plus thin clients, a pair of Mellanox x4 single port 10GbE adapters and a length of Twinax?
 

billc.cn

Member
Oct 6, 2017
41
7
8
Hmmm...A pair of HP t620 Plus thin clients, a pair of Mellanox x4 single port 10GbE adapters and a length of Twinax?
I was researching T620s yesterday after seeing the post about it here. The power numbers do look very good, but the CPU seems a bit weak for virtualisation purposes. I guess it could be used for the witness host though.

The biggest problem is the high price/low availability of the Plus model in the UK...
 

WANg

Well-Known Member
Jun 10, 2018
848
480
63
SuperMicro E200/E300-8D? those have 10GbE networking built-in and are full Xeon-D machines.