Small update, since this turned into a useful hardware lesson.
The 31B text tests are not the only thing I have been doing with this box. I also have a local image-classifier workflow running on it. The current production path is llama.cpp with Gemma 4 E4B multimodal, behind a small local wrapper, reading image files from a mounted NAS share and returning structured JSON labels. The latest test was a corpus-scale image classification run, not just a short prompt benchmark.
I also tested the official Gemma 4 E4B MTP path through a small local OpenAI-compatible Transformers sidecar. It worked technically, and the existing wrapper could talk to it, but it was not worth switching to for the classifier. In a production-style 13-image A/B, primary labels matched 11/13, watchlist behavior matched 12/13, and the best tuned run only improved end-to-end wrapper time by about 12%. So MTP remains interesting, but it is not my default classifier path.
The bigger discovery was thermal. Once I started a real corpus-scale image classification run, the RTX 4000 SFF Ada hit about 88C at the full 70W limit with the fan already at 100%. I paused the run, stopped the GPU services, and dropped the power cap.
Important caveat: this is not the stock NVIDIA cooler. My card is using the n3rdware aftermarket single-slot cooler, with liquid-metal thermal compound as part of that conversion. The stock NVIDIA cooler may not show the same behavior. In my setup, though, the reduced-size cooler is clearly the limiting factor during sustained CUDA work.
So the next step is not a different model yet. It is a proper low-power sustained-work profile: probably starting around 50W, then measuring throughput and temperature before scaling the corpus run again. Undervolting may be worth investigating, but I am going to treat that as a planned driver/tooling experiment rather than changing it in the middle of a live batch.
The MS-01/RTX 4000 SFF combo is still useful. The lesson is that short benchmark runs and sustained corpus work are not the same thermal wo
rkload, especially after converting the GPU to a single-slot cooler.
The 31B text tests are not the only thing I have been doing with this box. I also have a local image-classifier workflow running on it. The current production path is llama.cpp with Gemma 4 E4B multimodal, behind a small local wrapper, reading image files from a mounted NAS share and returning structured JSON labels. The latest test was a corpus-scale image classification run, not just a short prompt benchmark.
I also tested the official Gemma 4 E4B MTP path through a small local OpenAI-compatible Transformers sidecar. It worked technically, and the existing wrapper could talk to it, but it was not worth switching to for the classifier. In a production-style 13-image A/B, primary labels matched 11/13, watchlist behavior matched 12/13, and the best tuned run only improved end-to-end wrapper time by about 12%. So MTP remains interesting, but it is not my default classifier path.
The bigger discovery was thermal. Once I started a real corpus-scale image classification run, the RTX 4000 SFF Ada hit about 88C at the full 70W limit with the fan already at 100%. I paused the run, stopped the GPU services, and dropped the power cap.
Important caveat: this is not the stock NVIDIA cooler. My card is using the n3rdware aftermarket single-slot cooler, with liquid-metal thermal compound as part of that conversion. The stock NVIDIA cooler may not show the same behavior. In my setup, though, the reduced-size cooler is clearly the limiting factor during sustained CUDA work.
So the next step is not a different model yet. It is a proper low-power sustained-work profile: probably starting around 50W, then measuring throughput and temperature before scaling the corpus run again. Undervolting may be worth investigating, but I am going to treat that as a planned driver/tooling experiment rather than changing it in the middle of a live batch.
The MS-01/RTX 4000 SFF combo is still useful. The lesson is that short benchmark runs and sustained corpus work are not the same thermal wo
rkload, especially after converting the GPU to a single-slot cooler.
