8x 4090 motherboard recommendations

[iron-bound]

Ring Attention with modified softmax split's the kv cache across devices, it also does a better job than fsdp, as it allows all devices to compute and pass the results.

Striped Attention: Faster Ring Attention for Causal Transformers

To help address the growing demand for ever-longer sequence lengths in transformer models, Liu et al. recently proposed Ring Attention, an exact attention algorithm capable of overcoming per-device memory bottle- necks by distributing self-attention across multiple devices. In this paper, we...

arxiv.org

 

[bayleyw]

Ring Attention with modified softmax split's the kv cache across devices, it also does a better job than fsdp, as it allows all devices to compute and pass the results.
View attachment 36663

Striped Attention: Faster Ring Attention for Causal Transformers

To help address the growing demand for ever-longer sequence lengths in transformer models, Liu et al. recently proposed Ring Attention, an exact attention algorithm capable of overcoming per-device memory bottle- necks by distributing self-attention across multiple devices. In this paper, we...

arxiv.org

RingAttention is for long sequence lengths, I don't think it works well for less than 16K-ish sequence?

 

[iron-bound]

Depends on your device zero, if you can't run 16k with a batch size, it let's you scale.

 

[bayleyw]

Depends on your device zero, if you can't run 16k with a batch size, it let's you scale.

Right, but if you're only training at 4k context it doesn't help you.

 

[ufear]

I have a G292-Z20 for sale that takes 8 dual slot GPUs in 2U, threw the box though - can order it from piospartlab - if you want to collect it in NL let me know!

 

[bayleyw]

I have a G292-Z20 for sale that takes 8 dual slot GPUs in 2U, threw the box though - can order it from piospartlab - if you want to collect it in NL let me know!

Definitely not going to hold 8x 450W 4090's...

 

[iron-bound]

Four cards fit under the 2200W power supplies and under-volting is also recommended.

 

[kgcdctx]

I have a G292-Z20 for sale that takes 8 dual slot GPUs in 2U, threw the box though - can order it from piospartlab - if you want to collect it in NL let me know!

I built one of these. It will hold 8 data center GPUs, dual slot, but it won’t 4090.