soo anyone try k2.6 locally yet?

[T_Minus]

thoughts?

 

[TrashMaster]

thoughts?

Yes, come to the BlackwellPerformance rtx6kpro discord there is a channel dedicated to its testing

 

[T_Minus]

I wish I had enough rtx6kpros or any nvidia gpus to fit this, sadly Ill need to wait for a ~400gb mac version

for now i'm messing around with it on their api to learn the pitfalls

 

[Patrick]

This takes a lot of memory! It is loading on the 8x GB10 cluster right now, IIRC, using a 4-bit quant.

 

[T_Minus]

This takes a lot of memory! It is loading on the 8x GB10 cluster right now, IIRC, using a 4-bit quant.

let us know how that works once loaded

 

[Patrick]

let us know how that works once loaded

It is at least running now. 936W, including the two switches.

 

[TrashMaster]

It is at least running now. 936W, including the two switches.

View attachment 48595

Please run the MMLU-Pro benchmark and compare with the models baseline score to see how brain damaged NVFP4 is.

That quant format has been a singular disappointment in both performance and accuracy of model output across the blackwell performance community.

 

[Patrick]

That quant format has been a singular disappointment in both performance and accuracy of model output across the blackwell performance community.

Totally get it. One of the reasons we have the GB10 cluster is to run models at FP8 instead of going to 4-bit.

 

[tinfoil3d]

Patrick do you have any results / insights something to share on K2.6 or did you try running DS4 Pro? How many nodes that would take? I'm very interested to see how yuge models perform locally and what are the minimum specs for those. I am just starting ofc but the pace and features that are being pushed out this year look so mind blowing