Nvidea/GB10 Cluster sizes

[georgelza]

Hi guys.

This is going to be a set of questions... various posts, as I unpack it myself.

First... Was wanting to do it / cluster as a 5 node, then came across a couple of comments, mentioning Tensor Parallel saying, 1 2 4 or 8...
What would be the impact if I could say only get hands onto 6 nodes... not 8.

Was thinking of adding a 2nd generic Intel/K8S cluster on the side, connected to the GPU cluster via 10GbE, allowing all GPU processing to be purely on the Spark's and all non / generic workload on the x86-64.

For models, and scripts etc, was looking at having a x86-64 as a jump box, doubling as a NAS, aka NFS mount... -> 4TB HDD... could go 2 x 4TB or single 1 x 8TB NVMe stick... Also allows me to run all monitoring etc from here, instead of on the GPU Cluster itself.

Fire away.

G

 

[larrysb]

I think the law of diminishing returns will probably work against you from a practical standpoint. The GB10 box is great from the "has a lot of memory perspective", but it is slow memory. You can cluster them with the 200gb interfaces, but that's not a super fast link and the more you cluster, the less it scales. Then you get into switches and so forth, and pretty soon, you have a power-hungry, heat-generating, noisy rack of equipment running, which isn't probably going to be fast enough.

It is one thing to build it as a demonstration, and another if you're trying to do real work.

If you have a real workload for all that, what is the value of the workload?

Used RTX Pro 6000 / 96gb workstation cards are starting to appear on the used market in the $4k range, about what a Spark costs. A cluster of those would likely be a better option for a large workload. They're a lot faster. I think a pair or maybe a quad in the right host box, would be pretty awesome.

 

[georgelza]

... I hear what you saying...
But it's not answering my question.
a cluster might not scale for a single user, but multiple users / concurrency will scale, especially if you bring into scope technologies like KAI with pod placement/affinity and vCluster to virtualise the larger environment, vs placing a pod on a GPU node and that pod taking that entire node for the entire duration it runs, even it only actually required a very small amount of processing.

there is other things/concepts that can be proven at this scale, processes tested before being replicated to enterprise H100/B100, .. 200 size.

G

 

[TrevorH]

Have you already seen https://www.servethehome.com/big-cl...gb10-cluster-marvell-cisco-ubiquiti-qnap-arm/ ?

 

[larrysb]

Well the best place for answering the question would be the Nvidia developer forum. There are a number of bleeding-edge folks who’ve pushed the clustering to the limits, and there are git repos.

I think clusters wind up being even, but don’t know why not odd numbers.

But that would be the place to ask.

 

[georgelza]

will go ask. thanks.thought maybe someone here might know
G

 

[Mashie]

will go ask. thanks.thought maybe someone here might know
G

With the b12x kernel you can run non-power of two numbers of nodes. It is a recent development so not many have a lot of experience with it yet, nor bought additional nodes. I have 4 nodes and may go to 5 in the future.

But yes, there is a guy on the Nvidia dev forums running a cluster of 6 Sparks.

 

[georgelza]

... as this is to be a "piece" of def as a blog to enable developers... cost is a issue and well 6 nodes might cross the funding line allot easier than 8, allot of switch are 8 ports... leaving no upload ports when 8 node cluster is build, or 8 nodes for management plus upload.. it might look insignificant, but from a blog/funding it's big.

lets see.

thanks.
G

 

[Mashie]

... as this is to be a "piece" of def as a blog to enable developers... cost is a issue and well 6 nodes might cross the funding line allot easier than 8, allot of switch are 8 ports... leaving no upload ports when 8 node cluster is build, or 8 nodes for management plus upload.. it might look insignificant, but from a blog/funding it's big.

lets see.

thanks.
G

If you spend 40k on Sparks and a MikroTik switch, then a few hundred on a 10G switch for management will be a rounding error.

Must be a very busy blog if the adverts will pay for that hardware.

 

[csingley]

Hey georgelza, I haven't gotten my hand on these Sparks to play with ... but look, these weren't really designed to scale, and they won't excel at that. These were really designed for single-node (or sharded) inference not multinode training workloads, I think - if you need performance, this ain't it, chief.

But for demo purposes - as a lab environment - if you want a mini scale model to start playing with cluster design & operations, pay your dues, bark your shins, develop some workflows and some infrastructure you can transfer to production and scale up/out - i.e. you don't really care about absolute performance as much as you care about topology - then these will work, although they are expensive.

Mostly people configure the onboard (slow) NICs as a north/south network in band for host access & control plane, then configure the (fast) Mellanox cards as a different pure east/west network reserved for RDMA data plane. I would start with a static link config not switched, i.e. just plug a Spark's ConnectX HCA into its neighbors with a DAC, daisy chain. Run it like that until you get the basic deployment/configuration/operations well in hand... which will take you quite a while if you've never done any of this before. Then, and only then, will there be a solid case for investing in a fast switch on interconnects, and working out the patterns for dynamically dispatching workload to 2 vs. 4 GPUs via DDP or what have you.

When you get there, you can just plug these GPU nodes right into an existing k8s cluster, if you got one ... just apply an appropriate taint so they will repel scheduled work by default unless you deliberately request them. Do not mix your RDMA switch chocolate with your main network switch peanut butter.

Tensor parallelism during training.... man... if you have cleared away all the other blockers and dependencies and bottlenecks, so that now your big problem is squeezing high performance out of massive frontier models and exploring those tradeoffs... at that point you will be deep into "first world problems" territory, and you will already be having the kinds of conversations you need to do the engineering. If you ask me, first get settled in at base camp before you attempt the summit.

IMO you would do well to linger on the upfront survey of requirements/specs/assets/etc. Do not assume that any of this gear is plug&play; GPU compute is nowhere near as mature as CPU systems. As the poster above said, in multinode collective communications you are gonna run into real issues with ARM shared memory... it's hard to achieve true parallelism of compute & network when they are contending for the same memory! You would really rather profile those limits up front, rather than discovering them after you're $x0,000 deep into sunk costs.

But don't let me talk you out of it, it sounds really fun and interesting! By all means friend, get after it, hit it like a ton of bricks. Just buckle up; it's gonna be a bumpy ride

 

[bayleyw]

TP=6 gives you a 768GB system which will run Kimi K2.7, GLM5.2, and DeepSeek V4 Pro. Looking at the details, both Kimi and DeepSeek use 384 experts which will shard across 6 ranks, but GLM5.2 uses 256 which will not. All three are MLA so the attention tensors and KV-cache will be replicated across ranks.

 

[georgelza]

Hi csingley

And this, I think is the place for these smaller buildings blocks, figure out how to build the very big very expensive environments, figure out how it works, whats works, what does not, whats the bottlenecks you did not think of, how to work around them, these expensive production scale environments are seriously expensive if they are standing still, so use a smaller environment to figure out how to build and scale.

G

Hey georgelza, I haven't gotten my hand on these Sparks to play with ... but look, these weren't really designed to scale, and they won't excel at that. These were really designed for single-node (or sharded) inference not multinode training workloads, I think - if you need performance, this ain't it, chief.

But for demo purposes - as a lab environment - if you want a mini scale model to start playing with cluster design & operations, pay your dues, bark your shins, develop some workflows and some infrastructure you can transfer to production and scale up/out - i.e. you don't really care about absolute performance as much as you care about topology - then these will work, although they are expensive.

Mostly people configure the onboard (slow) NICs as a north/south network in band for host access & control plane, then configure the (fast) Mellanox cards as a different pure east/west network reserved for RDMA data plane. I would start with a static link config not switched, i.e. just plug a Spark's ConnectX HCA into its neighbors with a DAC, daisy chain. Run it like that until you get the basic deployment/configuration/operations well in hand... which will take you quite a while if you've never done any of this before. Then, and only then, will there be a solid case for investing in a fast switch on interconnects, and working out the patterns for dynamically dispatching workload to 2 vs. 4 GPUs via DDP or what have you.

When you get there, you can just plug these GPU nodes right into an existing k8s cluster, if you got one ... just apply an appropriate taint so they will repel scheduled work by default unless you deliberately request them. Do not mix your RDMA switch chocolate with your main network switch peanut butter.

Tensor parallelism during training.... man... if you have cleared away all the other blockers and dependencies and bottlenecks, so that now your big problem is squeezing high performance out of massive frontier models and exploring those tradeoffs... at that point you will be deep into "first world problems" territory, and you will already be having the kinds of conversations you need to do the engineering. If you ask me, first get settled in at base camp before you attempt the summit.

IMO you would do well to linger on the upfront survey of requirements/specs/assets/etc. Do not assume that any of this gear is plug&play; GPU compute is nowhere near as mature as CPU systems. As the poster above said, in multinode collective communications you are gonna run into real issues with ARM shared memory... it's hard to achieve true parallelism of compute & network when they are contending for the same memory! You would really rather profile those limits up front, rather than discovering them after you're $x0,000 deep into sunk costs.

But don't let me talk you out of it, it sounds really fun and interesting! By all means friend, get after it, hit it like a ton of bricks. Just buckle up; it's gonna be a bumpy ride