Hello,
I'm IT, my partner is an educator for a local university.
I'm looking into a home setup for a local AI for LAN access and wanted to get some recommendations.
Server Hardware:
Mac Mini M5 Pro 64 GB
AI Engine:
oMLX running on the Mac Server. A 27B-32B model.
RAG/Workstation access:
AnythingLLM
Workstations:
Windows - We will be accessing the LLM via windows workstations, if that's a factor.
Number of users accessing it:
2 (with occasional spikes to 4)
Responsiveness: We don't need Frontier model responsiveness. The idea is to find a reasonable
blend of AI capabilities and keep it responding in a reasonable time frame for tasks. A few mins is fine.
We don't need chat-level responsiveness for these tasks. In fact, a more capable model running a bit slower
may serve this better (ie. 70B-tight, instead of 32B-Comfortable). I won't know until I get the machine and can
experiment, unless anyone here has direct experience with this.
The main use of the server will be dropping documents/textbooks/education materials into a RAG repository,
then using the AI to generate things like Exam Questions, Study Guides, and other focused materials derived
from the Textbooks stored in RAG.
I would also use it to work with technical documents (I'm Senior IT, so it can get deep, fast) to assist me with creating
flashcards, summarizing long-form documents, and digging into the 400 page manual for the 1 switch I need. I would
also occasionally have a need for a capable model to whip up some Powershell scripts.
I'm highly technical, but still looking into a local AI model for the home network, so I'm trying to find sources of information to see
if what I'm lining up actually suits our need.
Questions:
1. oMLX seems the best candidate for Macs, as it is built for Mac Metal.
I do understand it only runs MLX models and not the seemingly more common GGUF.
Would this be a factor in practical terms? ie. The the best models in my range not offered in MLX?
2. AnythingLLM seems like a perfect fit for our need. RAG, and accessible over the network. Is it?
3. Would this server support multiple specialized models? (ie. one for research/material generation, one for Coding, etc)
This would not be required to be loaded at the same time, if that's impactful.
What did I miss? What other factors am I failing to account for?
I don't mind RTFM, but it's all new to me, so I'm asking for assistance in focusing my research
into this.
Thanks!
I'm IT, my partner is an educator for a local university.
I'm looking into a home setup for a local AI for LAN access and wanted to get some recommendations.
Server Hardware:
Mac Mini M5 Pro 64 GB
AI Engine:
oMLX running on the Mac Server. A 27B-32B model.
RAG/Workstation access:
AnythingLLM
Workstations:
Windows - We will be accessing the LLM via windows workstations, if that's a factor.
Number of users accessing it:
2 (with occasional spikes to 4)
Responsiveness: We don't need Frontier model responsiveness. The idea is to find a reasonable
blend of AI capabilities and keep it responding in a reasonable time frame for tasks. A few mins is fine.
We don't need chat-level responsiveness for these tasks. In fact, a more capable model running a bit slower
may serve this better (ie. 70B-tight, instead of 32B-Comfortable). I won't know until I get the machine and can
experiment, unless anyone here has direct experience with this.
The main use of the server will be dropping documents/textbooks/education materials into a RAG repository,
then using the AI to generate things like Exam Questions, Study Guides, and other focused materials derived
from the Textbooks stored in RAG.
I would also use it to work with technical documents (I'm Senior IT, so it can get deep, fast) to assist me with creating
flashcards, summarizing long-form documents, and digging into the 400 page manual for the 1 switch I need. I would
also occasionally have a need for a capable model to whip up some Powershell scripts.
I'm highly technical, but still looking into a local AI model for the home network, so I'm trying to find sources of information to see
if what I'm lining up actually suits our need.
Questions:
1. oMLX seems the best candidate for Macs, as it is built for Mac Metal.
I do understand it only runs MLX models and not the seemingly more common GGUF.
Would this be a factor in practical terms? ie. The the best models in my range not offered in MLX?
2. AnythingLLM seems like a perfect fit for our need. RAG, and accessible over the network. Is it?
3. Would this server support multiple specialized models? (ie. one for research/material generation, one for Coding, etc)
This would not be required to be loaded at the same time, if that's impactful.
What did I miss? What other factors am I failing to account for?
I don't mind RTFM, but it's all new to me, so I'm asking for assistance in focusing my research
into this.
Thanks!
Last edited:
