I have a limited understanding of local LLMs, but from what I gather 128GB is not enough to match DeepSeek v4 Flash or GPT 6.1 Luna. And these models are so cheap right now. What's the use case besides having your data locally?
While personally I’m not using them for work, just tinkering, 128GB can run qwen 3.8 flash next at q4(~111GB gguf). That model seems to be quite capable from what I’ve read.
It’s quite evident that the api models can and have been nerfed at the provider’s discretion, then there’s the abliteration aspect too(api models refusing what they consider “harmful”) which makes the case for local.
"Having your data locally" is definitely the biggest one.
- The general consensus seems to be that the frontier models are kind of (understandably) "benchmaxxed" towards coding benchmarks and have suffered in some other areas like writing, art, etc. Lot of tweaked/specialized models on huggingface
- Getting around the censorship in commercial models
- While local models don't match frontier coding agents, you can run them for multiple 24/7 with a long running goal (The fans in Macs are presumably not built to run 24/7, but this can be mitigated to a considerable extent IME by simply pointing a desktop fan at it, and you can throttle things in software. On the discrete GPU side of things, you can usually get big power savings without much performance impact by throttling the GPU clock; things are usually bottlenecked on memory)
- Even small 27B/35B models are getting darn good, definitely more than powerful enough for classification etc... you don't need 128GB to run them
- Right now, providers like OpenAI are running at huge losses. At some point, they are going to need to turn a profit, at which point they may get significantly more expensive, and "hosting your own" may make more economic sense than it does today -- and be a valuable professional skill, even.
- Even more speculative: perhaps the future for many applications is training and running your own hyper-specialized models. Right now, training your own is nowhere near cost effective on self-hosted hardware vs. renting GPUs
Note: whether any of these use cases are "worth it" is up to the individual. I'm not trying to sell anybody on the idea. I don't personally do much local LLM stuff at the moment, but it's attractive and I've been close to pulling the trigger...
I'm just trying to understand what the particular uses people have for them.
My raw data alone is terabytes, and it takes hundreds of GB sometimes to run very simple analyses. So there's no way I can have a local AI orchestrating the analyses locally.
But I realize some people have different kinds of data. So I'm interested to hear how exactly people use local models.
The point where it'd make sense for me is if someone makes a compact machine with 512 of unified memory in a couple of years and it costs less than a down payment for a house...
It's more "software & gear discussion" than "use cases", admittedly, I think because the people looking for software/gear discussion already have use cases in mind that they're trying to achieve.
For an M5 max top bin with 128GB Ram:
- mac studio: $5,899.00 (with 2TB ssd bump to match the default macbook sku)
- macbook 16: $7,399.00
Portability gets a hit but I feel you can have a thin client and let the studio do the heavy lifting back at your home/studio(heh).