RingAttention is for long sequence lengths, I don't think it works well for less than 16K-ish sequence?Ring Attention with modified softmax split's the kv cache across devices, it also does a better job than fsdp, as it allows all devices to compute and pass the results.
View attachment 36663
Right, but if you're only training at 4k context it doesn't help you.Depends on your device zero, if you can't run 16k with a batch size, it let's you scale.
Definitely not going to hold 8x 450W 4090's...I have a G292-Z20 for sale that takes 8 dual slot GPUs in 2U, threw the box though - can order it from piospartlab - if you want to collect it in NL let me know!
I built one of these. It will hold 8 data center GPUs, dual slot, but it won’t 4090.I have a G292-Z20 for sale that takes 8 dual slot GPUs in 2U, threw the box though - can order it from piospartlab - if you want to collect it in NL let me know!