I stopped chasing bigger local models, and a 27B model on my 8GB GPU does more of what I need
Source ↗
👁 1
💬 0
Running local models on a smaller card almost always comes with a ceiling, so you really have no choice but to live somewhere around the 4B to 9B mark. Every few weeks a new Qwen release seems to land on Hugging Face with bigger numbers attached, and the urge is to grab the largest version that might squeeze onto my card. Usually, it doesn't fit. Or it does, but with half the model sitting in system RAM, and answers crawl out a word at a time.
Comments (0)