AI @ Home - Part One: Everything I Got Wrong Before the Expensive Hardware Turned Up
From LM Studio to a 26B model on a 12 GB card: what a model file actually is, a conversion that produced the same sentence six thousand times, and a card I never thought to check.
Why I started at all
Reasonable question, and the honest answer is not “I was curious about GPUs.”
I was studying at university. There are a handful of people who have read my raw, unassisted writing, and every one of them would agree it isn’t great. I’m dyslexic. What comes out of my head first is not what most people would call legible: the ideas are in there, the sentences are not, and the distance between those two things has followed me my entire life.
So I started using Claude to stop my rambling from arriving as an unreadable wall. Not to have the thinking done for me. To get it transcribed into something a marker, or a colleague, or you, could actually read. It worked. It worked well enough that it stopped being a convenience and became part of how I get anything written down at all.
Including this. You are reading the output of that arrangement right now.
And then in early 2026, Anthropic, fairly famously, started moving the limits around.
I’m not going to inflate that into a catastrophe, because it wasn’t one. But it did hurt, in a specific and faintly humiliating way. A capability I had come to rely on, for university work with real deadlines attached, quietly got smaller, and there was precisely nothing I could do about it. I had not bought a tool. I had rented access to one, and the landlord revised the terms.
That is the actual origin of everything in this post. Not an interest in tensor parallelism. A fairly ordinary, fairly uncomfortable realisation that I had built something I depended on directly on top of something I did not control, and that I had never once thought about what would happen if it moved.
It moved.
Which should not have surprised me
I have been a homelabber for a long time. It started with Plex on FreeNAS 9, the way it started for an awful lot of people, and that hobby turned into a career as a small-enterprise sysadmin. Owning your own data is not a position I arrived at recently or theoretically. It is most of what I do.
So I should have seen it coming, and I want to resist the obvious conclusion, because the obvious conclusion is smug and also wrong.
There is a very good reason cloud exists. It puts resources in your hands that you could not otherwise afford, at a price that makes them look free at the point of use. That is not a trick. It is a genuinely enormous transfer of capability to people who could never justify the capital cost, and I have benefited from it constantly.
To put a number on the capital cost in question: running a large model at home properly means one of the small machines built for exactly this. A decked-out Mac Studio Ultra. An AMD Strix Halo box. An NVIDIA DGX Spark. And realistically a pair of them, which lands somewhere between $10,000 and $20,000 roobucks depending which you pick. If you would rather do it with workstation cards instead, two RTX PRO 6000 Blackwell Workstation Editions, 96 GB apiece, is closer to $50,000.
Those are Australian retail, at the time of writing, and I am not going to pretend they will still be right when you read this. GPU pricing moves like weather. Check it yourself before quoting me at anyone.
What those three little machines have in common, and it turns out to be the entire subject of part two, is a big pool of memory shared between the CPU and the GPU. That is the thing you are actually paying for. Not compute. Somewhere to put the model.
I do not have any of them, I am not going to have any of them, and a subscription hands me most of the capability for the price of a takeaway. That is precisely the problem cloud solves, and pretending otherwise would be dishonest.
The catch isn’t that it’s a con. The catch is that the price is only reasonable until it isn’t, and you have no say in when that changes, and by the time it does you have built things on top of it. You are not being robbed. You are the frog in the pot, and somebody else is holding the dial.
I have given that advice professionally for years. I just hadn’t noticed I’d stopped taking it.
Where I’m standing, since it matters later
I am not a heavy user, and I want that on the record early, because by part two this series spends a lot of words arguing about compute.
I’m on the Pro tier, as an individual. I cannot justify Max. It is not an expense I can defend for personal use, and that isn’t a complaint about the price, it’s just where my budget stops. What I actually spend is about $35 AUD a month to help arrange my brain into sentences other people can read.
It isn’t the only purpose, the same subscription helps me organise Ansible playbooks and debug some genuinely stupid code, which means it is threaded through the infrastructure in this post as well as the description of it. The repo I spend a later section praising for being readable and reviewable was itself organised with help. That is not a contradiction, but it is the reason that section exists at all: if the assistance is going into the playbooks, the playbooks had better be the kind of thing I can sit down and read.
That is the entire extent of my investment in this. So when I get to the section about rate limits and capacity allocation, hold on to the fact that I am not a power user who got throttled mid-sprint, and I am not somebody with an enterprise contract and a support channel. I’m a bloke on the cheapest paid tier who noticed the floor move under something he’d started depending on.
Act one: the start of what I thought was a good idea at the time
So: LM Studio
Which is where it starts for most people, me included. I installed LM Studio on my home PC and tried to do “anything”.
That’s the whole beginning. A desktop with an RTX 3080 Ti in it, a GUI with a search box, a list of models with download buttons next to them, and me taking the recommended defaults, because I had no basis whatsoever for choosing anything else. What that gave me was a Gemma model, and it basically did something. It answered questions. Words came out in the right order.
It was also, unmistakably, a toy. Not because Gemma is a toy, but because the whole arrangement was: one app, on one desktop, answering only me, only while I had it open.
That’s exactly why I’m opening here rather than at the expensive hardware. This is the part everybody recognises, and the rest of this post is just what happened when I kept pulling the thread.
Two things about that beginning turned out to matter enormously later, and I noticed neither at the time.
LM Studio hands you GGUF. You pick a model, you pick a quant from a dropdown, it downloads a file, it runs. Nobody asks you what a checkpoint format is, because at that layer you genuinely do not need to know. That is good product design.
It is also how I came to believe, without ever deciding to, that GGUF was simply what models were.
Here is the part I only understood much later. Google generally do not ship Gemma as GGUF. What they publish is safetensors. They have put out quantised and on-device builds of some lines, so it is not a rule with no exceptions, but it is not the norm and it is emphatically not where my file came from.
Which means the very first model I ever ran was not the artifact Google published. It was somebody else’s conversion of it, re-quantised and repackaged, and served to me through a dropdown that gave no indication there was anything to know.
So why does safetensors exist, if everything uses GGUF?
That question is worth sitting with, because it is exactly the question I had, and it is built on a false premise I couldn’t see from where I was standing.
LM Studio is, underneath, llama.cpp. That’s the engine doing the work. GGUF is llama.cpp’s format. So when LM Studio presents you a catalogue of models, what you are actually browsing is the slice of the world that llama.cpp can run, in the format llama.cpp wants, and nothing else.
From inside that bubble GGUF looks universal. Every model in the list is GGUF. Every tutorial is GGUF. Every recommendation from anyone else in the same bubble is GGUF. So when you eventually notice that models are also published as safetensors, the obvious question is why that format bothers to exist at all.
The answer, which took me an embarrassingly long time to reach: everything does not use GGUF.
Safetensors is what models are born in. It’s what the training frameworks emit, what Hugging Face distributes, what the research code loads, and what the people who made the model actually signed off on. GGUF is a conversion, produced afterwards, so that one particular family of inference engines can run those models on ordinary hardware.
I had the relationship precisely backwards. I thought safetensors was the obscure niche thing and GGUF was the standard. It is the other way around: safetensors is the source, GGUF is the derivative, and I had only ever seen the derivative because I had only ever used tools that consume it.
Which is completely fine. Right up until the day you use a tool that doesn’t.
Keep that in mind for a few hundred words, because it is about to become extremely concrete: the next thing I do is decide I can perform that conversion myself. It goes badly here, as a format problem, and then goes badly again in part two as a considerably more expensive one.
And that same 3080 Ti box is still running. It makes pictures rather than words these days, which is part two’s problem. The first computer I ever loaded a model onto is still in service, just doing an entirely different job.
A user interface is not the same as features
Worth being fair to LM Studio, and also clear about what it is.
It has a graphical interface, which makes it look like a finished product, and it is genuinely good at its job: pick a model, it loads, it answers, and it will serve that model over an API if you ask it to.
What it is not is an assistant platform. There is no web search, so the model knows only what it was trained on and confidently tells you about a world that stopped some time ago. MCP support is limited, so wiring it into other tools and data sources only goes so far. It responds to input and it serves the model. That is close to the entire feature set.
Which is not a criticism. It’s a local model runner and it does not claim to be anything else. But it does mean “I have a UI” and “I have an assistant” are quite different statements, and the distance between them is one of the two reasons I moved on.
The other reason: I wanted it to stay up
That’s the bigger jump, and it’s a bigger one than it sounds.
LM Studio is an application. You open it, you use it, you close it. It lives on a desktop, tied to a logged-in session, on a machine that sleeps when you walk away. Which is completely fine until the thing you want is for the model to just be there - reachable by something that isn’t you, still running tomorrow, not conditional on whether anyone is sitting in front of that particular PC.
The moment you want that, you don’t want an app. You want a service. And essentially every hard problem in the rest of this post is downstream of that one sentence.
So I moved to llama.cpp in Docker, on an Intel Arc B580 in a box I keep here, in the Land Down Under. Same file format, same models, completely different posture: a container with a restart policy, an HTTP endpoint on a port, something other software can talk to without a human present.
That machine is the rest of this post, and it had a great deal to teach me, most of it unwelcome.
Because here is the short version of what followed: the GPU was never the hard part. The hard part was everything around it. What format the weights are in, who made the file, whether the thing loading it agrees with the thing that made it, and whether any of it will still be running tomorrow without me standing there.
The machine, down under
Where this ends up, so you can see the shape of it before the mess:
One box. Three containers. No cloud, no account, no rate limit. Getting to that picture took considerably longer than it should have.
It’s a single Intel Arc B580, and it is a deeply unremarkable computer:
OS: Ubuntu 26.04.1 LTS (Resolute Raccoon) x86_64
CPU: AMD Ryzen 5 5500 (6C/12T) @ 4.27 GHz
GPU: Intel Arc B580 [Discrete]
Memory: 64 GB
A midrange Ryzen, 64 GB of ordinary DDR4, and a 12 GB consumer Arc card. Nothing about it is impressive and that is rather the point.
I did not buy any of this for this. The desktop existed. The card existed. The 3080 Ti in the other machine existed. Not one component in this series was purchased to run models on, so the marginal cost of everything described here is nothing. It is all sunk cost, and not recent sunk cost either.
That distinction matters more than it looks, because the sums that made sense when I bought this kit in 2025 are not the sums you would be doing today. I bought a mid-range card for mid-range reasons. Nobody was bidding against me for it. Buying the same class of hardware in 2026, specifically to run models on, is a different transaction at a different price in a different market, and I would be making the decision for a reason I did not have at the time.
So take the whole of this post as what can I do with what is already sitting here, not as a shopping list. The constraint shaped every decision below, and it is the only reason the decisions are interesting.
This is where the actual hell started
LM Studio never made me choose anything. The B580 made me choose everything, and the first thing I had to choose was the part I’d never previously known was a decision: which model, in which format, built by whom.
Knowing that safetensors is upstream and GGUF is downstream is the easy half. The hard half is what that means in practice: most of the ecosystem I’d been living in is built on conversions, performed by strangers, at their discretion, and I had never once looked at who.
Not all of it. Some model authors publish GGUF themselves alongside the safetensors, and when they do, the whole question evaporates. But that is the exception, and you have to actually check which situation you are in.
And format shifting is genuinely painful. Converting is not a file-type change, it’s a re-encoding: the architecture has to be supported by the converter, the tokenizer has to survive the trip, multimodal models need their projector handled separately, and then you pick a quantisation level that trades quality for memory in ways nobody can fully characterise for you. Any step can go subtly wrong and still produce a file that loads and answers.
I tried doing the conversions myself
Which is where it stopped being theoretical. The attempt was gemma-4-E4B, converted by me, and it went badly.
Some results were fine. Some were schizophrenic, and I mean that fairly precisely, because the characteristic failure is not gibberish. Gibberish would be easy - gibberish you spot in one line and throw away. The characteristic failure is a model producing perfectly fluent, well-formed English while having completely lost track of who is speaking.
The mechanisms behind that, roughly in order of how often they bit me:
- The chat template doesn’t survive the conversion. The weights are intact but the turn structure is gone, so the model generates both halves of the conversation. It answers you, then asks itself a follow-up, then answers that one too, quite happily, forever.
- The stop tokens go missing or get remapped. Nothing tells generation to end, so it runs to the token limit, drifting further from the question with every paragraph.
- The tokenizer doesn’t quite match the weights. Output is mostly correct with occasional wrong words dropped in. This is by far the most unsettling one, because it reads as competent right up until it doesn’t.
- The quantisation is too aggressive for that particular model. It loops, repeats itself, or gets confidently wrong about things a heavier quant of the same model gets right.
Every single one of those produces a file that loads without error and responds when spoken to.
What that actually looked like
I kept one of the transcripts. Two questions. 13,243 lines of output.
First question was “tell me about yourself”:
I am the GPT-4 Copilot, developed by OpenAI. I am a virtual assistant and a language model designed to assist with a wide range of tasks, including content creation and analysis.
It was not. That is not the model, not the vendor, and not anything I asked about. It then explained, unprompted, a script for generating and remixing images, before settling into its final form:
It is for You.
It is for You.
It is for You.
Six thousand and twenty-one times. I counted with grep -c, because at that point what else are you going to do.
Second question was “what can you do?”, which produced a different flavour of the same disease. Grammatically flawless, semantically empty:
It’s time for a new beginning. - It’s time for you to try it. It’s time to make a difference. - It’s time for you to make a difference in your life.
That goes on for thousands of lines, gradually decaying into strings of numbers.
This is what I mean by schizophrenic. Not gibberish. Look at it: every sentence is well formed, the grammar is perfect, the register is confident. It has simply lost any idea of who it is, what it was asked, or when to stop. Two of my four mechanisms are visible in one transcript, the identity confusion and the missing stop condition, and I never definitively worked out which conversion step caused either.
That is the same failure shape as nearly everything else in this post: it loads, it answers, it’s wrong. I just hadn’t learned to expect it yet.
So I gave up converting things myself and took a recommendation from a mate instead: ornith-1.0-9b-Q4_K_M.gguf, which worked immediately and without drama.
And it’s worth saying why that went so smoothly, because it isn’t luck. Ornith publish their own GGUF. The people who trained the model also built and released the quantised files, which means there was no third party in the chain, no conversion by a stranger, and nothing for me to get wrong. The provenance question I’d been chewing on simply doesn’t arise: the vendor made the file.
Which is a useful counterexample to my own argument. GGUF being a downstream conversion is the general case, not a law.
The models directory still has that week of it sitting there:
| Date | File | Size | What it was |
|---|---|---|---|
| 8 Jul | gemma-4-E4B-f16.gguf |
15.1 GB | my conversion |
| 8 Jul | gemma-4-E4B-Q4_K_M.gguf |
5.3 GB | quantised from it, two minutes later |
| 12 Jul | ornith-1.0-9b-Q4_K_M.gguf |
5.6 GB | the mate’s recommendation |
A few days, then, between “I’ll convert it myself” and “fine, I’ll use the one somebody else built.” The two E4B files are still sitting there, owned by root because the converter ran in a container, and I have never deleted them.
And I want to be blunt about the part that bothers me most. I am not a fan of just finding a model and running it. Not on principle, and not out of snobbery, but for exactly the reason anyone should hesitate before downloading random software off the internet and executing it. That is what this is. A model file is an artifact of unknown provenance that you are about to run on your hardware, and the fact that it arrives through a friendly dropdown does not change what it is.
The uncomfortable version: when you download a community GGUF, you are trusting a conversion you cannot easily verify, of weights you did not train, from an uploader you have not checked, in a format designed for convenience rather than attestation. Sometimes that uploader is well known and does careful work. Sometimes you have no idea. The interface looks identical either way.
I don’t have a clean answer to this. What I have is a habit: know who published the weights, know who did the conversion if it isn’t them, and prefer the upstream artifact when the runtime can take it. That habit took a while to form, and part two opens with me spectacularly not having it yet.
So, with all of that decided rather than defaulted, here is the stack it runs on. llama.cpp’s server-intel image, on the SYCL backend:
llama-server:
image: ghcr.io/ggml-org/llama.cpp:server-intel
ports:
- "8080:8080" # published on the LAN. see the warning below
devices:
- /dev/dri:/dev/dri # renderD128 + card into the container
group_add:
- "${RENDER_GID:?run: getent group render | cut -d: -f3 ...}"
- "${VIDEO_GID:?run: getent group video | cut -d: -f3 ...}"
environment:
ZES_ENABLE_SYSMAN: "1" # sysman telemetry, for real VRAM stats
ONEAPI_DEVICE_SELECTOR: "level_zero:0" # pin to the discrete GPU
Three Intel-specific things there, none of which have NVIDIA equivalents you’d think to look for:
/dev/dri plus group membership. There’s no --gpus all. You pass the device nodes through and then make sure the container’s user is in the render and video groups, by numeric GID, because those GIDs differ per host. The compose file uses Docker’s ${VAR:?error message} syntax to carry its own instructions - if the variable is unset, the failure message is literally the command that tells you the value. That is a nice pattern for anything host-specific and I’ve started stealing it elsewhere.
ZES_ENABLE_SYSMAN=1. Without it, memory telemetry is unreliable. Every vendor has its own way of not telling you how much memory you are using, and this is Intel’s.
ONEAPI_DEVICE_SELECTOR=level_zero:0. Ryzen 5 5500 has no integrated graphics, so this is belt-and-braces here, but on any CPU that does have an iGPU, Level Zero will happily enumerate it and llama.cpp will happily use it. sycl-ls inside the container tells you the ordering.
The healthcheck that doesn’t assume curl
Small thing, disproportionately useful:
test: ["CMD-SHELL", "curl -sf http://localhost:8080/health || wget -qO- http://localhost:8080/health || timeout 2 bash -c '</dev/tcp/127.0.0.1/8080' || exit 1"]
server-intel may not ship the curl CLI. So it tries curl, falls back to wget, and falls back again to a raw TCP probe via bash’s /dev/tcp. A healthy server never gets marked unhealthy just because a client tool is missing from the image.
I’ve watched a container get restart-looped by a healthcheck testing for curl’s existence rather than the service’s health. Worth the extra thirty characters.
What that actually got me, at home
With a model finally running reliably, the rest was stack-building. Open WebUI in front of llama.cpp so it had a GUI instead of a curl command, and SearXNG alongside it, which gives the thing working internet search.
And that is the gap from earlier closed: LM Studio gave me a model behind a UI, and this gives me a model that can go and look things up. Same idea, considerably more useful.
It works, too. Capability-wise it felt like roughly GPT-3.5-turbo territory.
I need to be careful with that sentence, because it is a feeling and nothing else. There is no benchmark behind it. Worse, my reference point at the time was Opus 4.5, which is not a fair yardstick for anything: next to a frontier model most things feel like 3.5-turbo. So take it as an impression from a thoroughly biased observer, which is all it is. It is also, I notice, exactly the sort of judgement I will complain in part two that nobody has a metric for, made here entirely on vibes by the person who will be doing the complaining.
Discount it as heavily as you like and something still stands: a genuinely useful assistant, with web search, running on a second-hand desktop in my house, answerable to nobody’s rate limit and costing me nothing but power.
That is the first moment in this entire story where the original goal was actually met.
The whole integration is a handful of environment variables:
open-webui:
image: ghcr.io/open-webui/open-webui:main
depends_on:
llama-server:
condition: service_healthy
ports:
- "3000:8080"
environment:
OPENAI_API_BASE_URL: "http://llama-server:8080/v1"
OPENAI_API_KEY: "sk-no-auth-needed"
ENABLE_OLLAMA_API: "false"
WEBUI_AUTH: "false"
llama.cpp serves an OpenAI-compatible /v1, so Open WebUI just points at it by service name. The key is required by the client and completely unused, hence sk-no-auth-needed, which is the most honest string in the file. ENABLE_OLLAMA_API: "false" stops it hunting for an Ollama backend that was never there.
And condition: service_healthy is where that healthcheck earns its keep. Open WebUI won’t start until llama.cpp reports healthy. So if that check had falsely failed because curl isn’t in the image, the GUI would simply never have come up, and I’d have been debugging the wrong container. The curl-then-wget-then-/dev/tcp chain isn’t belt and braces. It’s load-bearing.
Then there’s WEBUI_AUTH: "false", and you should not copy this
Let me be unambiguous, because someone will paste this into a terminal.
Do not deploy it like this. That is a web interface with no login at all, wired to a model, and llama-server is published straight onto the LAN on 8080 next to it. Two open doors, not one. Nothing here is behind authentication, nothing is behind a reverse proxy, and the only thing standing between it and anyone else on the network is the network.
I told myself this was fine because it was a throwaway proof of concept on my own LAN. Having written that sentence down, I don’t think it holds. It is a proof of concept that has been running for months, and “it’s only on my network” is a sentence with an extremely long and unhappy history behind it. Honestly, I should not have stood it up this way even as a POC. It costs very little to do it properly at the start and a great deal to retrofit once you have got used to it working.
There is a right way to do this, with authentication in front and the model endpoint not exposed at all, and it is coming in its own post. Take this block as a description of what I did, not a recommendation.
(This is the small version. Part two has the grown-up deployment, with SSO, an HA Postgres behind it and a pinned release instead of :main. This one is just a container with a web page.)
The part that didn’t work well
Around the same time I was trying to get on with opencode, and I did not get far. It was, honestly, well short of just using the Claude extension in VS Code.
Worth saying plainly, because it complicates the story I would prefer to tell. The local stack replaced the chat window perfectly well. It did not replace the thing I actually depend on, which is help with writing and code at a level those models weren’t reaching. I bought independence for one job and not the other, and the one I didn’t get is the one that sent me down this road to begin with.
I should be clear that opencode was not the only thing I tried, just the only one I tried at this point in the story. Others came later, and the one with the best story gets its own airing in the marked tangent of part three. The pattern held across the assistants, though, and the pattern is the interesting part: the local stack is genuinely good at the easy job and does not touch the hard one.
That gap is most of the conclusion of part two.
A complete stack, and a natural place to stop
And that, technically, is the job done.
Stop and look at what exists at this point, because it is easy to undersell. A second-hand desktop. An Arc B580. A llama.cpp container serving an OpenAI-compatible endpoint. Open WebUI in front of it. SearXNG beside it. A model I did not build, did not convert, and did not have to trust blindly, because its authors published it themselves and somebody I know had already used it.
That is a complete stack. Not a demo, not a proof of concept. It starts on boot, it survives a reboot, it answers questions, it searches the web, it costs nothing but electricity, and nobody can change its terms of service while I sleep.
Everything I set out to build in the first place, I had. On hardware I owned. Down under, in my house.
What I learned getting there, in order of how much it cost me:
- A format is not a format. GGUF is llama.cpp’s format that other things partially support, not a universal container. LM Studio hid that from me for months, entirely reasonably.
- Most of what you run is not the artifact its authors published. It’s somebody’s conversion. The job is knowing whose, and whether they have anything to lose if it’s wrong.
- Converting things yourself is a re-encoding, not a rename, and it fails in ways that still load, still answer, and are still wrong.
- Wanting something to stay up is a bigger decision than it sounds. Everything hard downstream came from that one requirement.
And what I did not solve: the thing that sent me down this road. The local stack is a very good chat window. It is not what I lean on when the writing has to be right, and opencode did not change that.
So: job done.
Interlude: and that would have been the end of it
Genuinely. If Ornith 9B on a B580 was where this stopped, you would not be reading any of this.
A stack that works is not a story. There is no tension whatsoever in “I installed some containers and they did roughly what the documentation said they would.” I’d have used it, been quietly pleased with it, mentioned it to the two or three people I know who care about this sort of thing, and written precisely nothing.
Everything that makes this worth writing down happened afterwards.
Within a few days of that file landing, somebody lent me access to two DGX Sparks. I disappeared into them for six weeks. That is part two, and it is a completely different kind of story: 128 GB of unified memory per node, hardware sitting in someone else’s studio eight thousand miles away, and a great deal of expensive education about what fits where.
But the B580 thread doesn’t stop on the 12th of July either, and I don’t want to pretend it does.
The rest of this post jumps forward to 25 August, and I need to be upfront that it is only possible because of what happened in between. Everything I learned on the Sparks about memory, about what actually fits, about the difference between a model’s size and a model’s resident size, came straight back down here and changed what I was willing to attempt on my own hardware.
So read the rest of this as the B580 thread continuing, with knowledge borrowed from a machine I had to give back. And with one piece of stupidity that I only uncovered while writing these posts up, months after it started costing me.
Act two: so much time spent, and I still don’t know enough about anything
What I came back and did to it
A 9B model on a 12 GB card is a comfortable fit, and comfortable fits are where you stop learning things. Having spent six weeks watching two far more expensive machines struggle with models that didn’t fit, the obvious question was how far this one could actually be pushed.
On the 25th of August the B580 moved to unsloth/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf.
That is a 26B mixture-of-experts model. On a 12 GB card. It should not fit, it doesn’t, and it runs anyway.
Which brings the models directory up to date, and the second half of that listing is a different kind of shopping list to the first:
| Date | File | Size | What it is |
|---|---|---|---|
| 25 Aug | gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf |
17.0 GB | the model |
| 25 Aug | mtp-gemma-4-26B-A4B-it.gguf |
0.5 GB | its MTP draft head |
| 25 Aug | mmproj-F16.gguf |
1.2 GB | its multimodal projector |
Three files rather than one, because this is no longer just a set of weights. There’s a separate small model for speculative decoding and a separate projector for vision, and both have to match the thing they belong to. Note the size, as well: 17 GB of model going onto a card with 12 GB on it.
There is a pleasing symmetry to that filename, too. Same model family LM Studio handed me from a dropdown at the very beginning of all this, and the same family I comprehensively failed to convert myself back in July. The difference is that this time I chose it deliberately, I know it started life as safetensors, and the file says who produced the quant.
And yes, that is a community quant
Which looks like me walking back everything I said earlier about provenance, so let me be precise about why it isn’t.
unsloth is not an anonymous account that turned up last Tuesday with a folder of files. Unsloth are a company with a commercial presence and widely-used tooling for fine-tuning and quantisation, and their name on that file is attached to a business with a reputation it would rather keep. The UD in the filename is their own documented quantisation approach, applied consistently across a whole catalogue rather than one-off by hand.
That is a materially different act from pulling a randomly-named GGUF off Hugging Face because it was the first search result.
The habit was never “only ever run first-party files”. Almost nobody could work under that rule. It was know who made this, and whether they have anything to lose if it is wrong. With Ornith the answer was the model’s own authors. Here the answer is a company that has to keep shipping these. Both are fine. The failure mode is the third case, where the honest answer is “no idea”, and that is the one worth refusing.
The trick: put the experts somewhere else
The serve command is where it gets interesting:
-m /models/gemma-4-26B-A4B-it-UD-Q4_K_XL.gguf
-md /models/mtp-gemma-4-26B-A4B-it.gguf
--mmproj /models/mmproj-F16.gguf
--spec-type draft-mtp --spec-draft-n-max 4
-ngl 999 -ngld 999
--n-cpu-moe 99
-c 262144 -fa on --jinja --metrics
Two of those flags decide where everything physically lives, so they are worth defining before the clever one.
-ngl 999 is --n-gpu-layers: how many of the main model’s layers to keep in VRAM. 999 is the old idiom for “all of them, don’t make me count” (newer builds also take all or auto). -ngld 999 is the same knob for the draft model that speculative decoding uses, --n-gpu-layers-draft. That one matters more than its size suggests: the draft head here is only about half a gigabyte, and a draft model that spills onto the CPU defeats the entire point of drafting, because you end up waiting on the thing that was supposed to be running ahead.
So -ngl 999 says put the whole model on the GPU, which is plainly impossible: 17 GB of file, 12 GB of card.
--n-cpu-moe 99 is what resolves that, and it is the load-bearing flag. It claws the mixture-of-experts tensors back off the GPU and leaves them in system RAM. The two flags read like a contradiction and are really an instruction plus an exception: put everything on the card, except the experts.
That works because it’s an A4B model - only about 4B parameters active per token - so the bulk of the weights are expert layers that are individually cold. Attention and the dense path stay on the GPU where they’re needed constantly. The experts sit in the 64 GB of otherwise-unremarkable DDR4, and get woken up only when routed to.
You are deliberately spending host memory and CPU compute to buy VRAM headroom you don’t have. The expert weights stay in system RAM and are computed on the CPU, so what you are really paying with is memory bandwidth and cores, not bus transfers.
It moves the wall, it doesn’t remove it
Worth being clear about what this technique actually buys, because “run a model bigger than your card” sounds like a cheat code and it isn’t.
What MoE offload does is let you trade a resource you have for one you don’t. I have 64 GB of ordinary system RAM and not much VRAM, so the experts live in RAM. Fine. But the model still has to fit somewhere in total. If I wanted to run something like a 2.4T-parameter model, I would still need roughly 1.2-1.4 TB of somewhere, and no amount of clever placement conjures that out of a desktop.
And every step down the hierarchy costs you.
Weights in VRAM are fast. Weights in system RAM, computed on the CPU, are slower, and bounded by memory bandwidth and core count rather than by the GPU at all. There is work going on to push active layers further still, all the way onto disk, and it does work in the sense that the model runs. It also means every forward pass is issuing NVMe reads, which costs you latency now and drive endurance over time. (This assumes you are not enough of a madlad to attempt it on spinning rust. If you are, write it up. I would read that.)
The bus is a separate tax, and worth keeping straight from the others. It is not what makes the CPU-expert trade slow: those weights never leave system RAM, so PCIe barely features. What crosses the link is activations during inference, and every tensor that does end up GPU-resident, at load. Which matters more than it sounds, and I will come back to that in a moment with a number I should have noticed months ago.
So the honest framing is that MoE offload moves the wall further away. It does not take the wall down, and each metre you move it is paid for in throughput, in hardware lifetime, or in both.
Part two does exactly this on purpose and at scale, serving a 51 GB chunk of a model off NVMe because there is no other way to make it fit, with a whole row of dashboard panels dedicated to noticing when that trade stops paying off.
A confession about this box
And now the stupid part, which I only found while writing these posts up.
That machine is a B450 board and a Ryzen 5 5500, so I assumed the B580 was sitting on PCIe 3.0 x8 and got on with my life. Then, months later, I finally checked:
LnkSta: Speed 2.5GT/s, Width x1
PCIe 1.0. One lane. Roughly 250 MB/s, on a card specified for about 15.75 GB/s.
That’s a fault to go and fix, not a platform limitation to write clever paragraphs about. Slot or riser, most likely. Performance will change when I sort it out, and I can’t tell you by how much.
I’m an idiot for not checking. The card has been in there for months and the answer was one command away the whole time.
In partial mitigation, it has never been obviously broken. The reason is the trick itself: llama.cpp computes the CPU-side experts in place rather than shipping their weights across the bus, so during inference only activations make the trip. Small payloads, over and over. Annoying rather than fatal.
But model loading is a different story, and this is the bit that should have caught me.
250 MB/s works out at roughly four seconds per gigabyte. Every tensor that ends up GPU-resident has to crawl across that single gen-1 lane at startup, and while the experts staying in RAM means it is a fraction of that 17 GB file rather than all of it, the projector and the draft head are another 1.7 GB on their own and they are going nowhere else.
So every single start of this thing has been spending a chunk of time doing nothing but pushing bytes down a straw. Not milliseconds. Tens of seconds, every time, for months.
And I cannot tell you how many, because I never timed it. That is the actual indictment, and it is worse than “I wasn’t benchmarking”. This was not a subtle statistical signal buried in a percentile. It was a startup that felt wrong, in front of me, repeatedly, and I had no number attached to it because I had never once thought to put one there.
Which is the same gap as everywhere else in this post, one layer further down.
Hold onto that idea, because it is entirely specific to a discrete GPU: a card with its own memory, separate from the host’s. Part two is about hardware where that distinction collapses, and where this exact move turns out to be worthless. Which is precisely how I came to understand why it works so well here: I spent six weeks watching it fail somewhere else.
The rest of the flags: -c 262144 is 256k context, -fa on is flash attention, --jinja uses the model’s own chat template, and --mmproj loads the multimodal projector, so it takes images.
The two speculative decoding flags are worth a sentence each. --spec-type picks the drafting strategy from a list of them - simple draft models, eagle3, several n-gram variants - and draft-mtp is the multi-token-prediction one, which is why there’s a separate -md file holding this model’s MTP head. --spec-draft-n-max 4 is how many tokens it drafts per step, one above the default of 3.
One caveat on -c 262144, because 256k of context on a 12 GB card is the sort of claim that should make you suspicious.
Context is not free. The KV cache has to live somewhere, and it lives with the layers it belongs to - which here means the attention layers, which are exactly the ones I have deliberately kept on the GPU. So the same 12 GB that was already tight is what pays for context length. The experts going to system RAM buys me room for weights; it does not buy me room for context.
The server starts and serves, so the allocation evidently succeeds. What I have not done is push anything near 256k through it and watch what happens. I have not quantised the KV cache either - --cache-type-k and its partner are unset, sitting at the default - and that is the usual lever when context does not fit.
So read -c 262144 as the ceiling I asked for, not a number I have driven to. It is a configuration value, not a measurement, and I would not want anyone buying a card on the strength of it.
The part of speculative decoding that surprised me is that in principle it is lossless. The big model verifies every drafted token, so the output should be identical to running without it. You are not trading quality for speed. You are trading a small model’s guesses, checked, for fewer round trips through the large one.
“In principle” is doing deliberate work in that sentence. There is an open upstream issue where draft-mtp changes deterministic output at certain --spec-draft-n-max values on some MTP models, and I am running 4, one above the default. Whether that is biting me here I genuinely do not know, because I have not tested for it.
But the distinction matters: under this design, any deviation is a bug rather than a tradeoff. Those are very different things to go looking for, and telling them apart turns out to be most of what part two is about.
--metrics is the one worth calling out: it makes llama.cpp expose Prometheus metrics on its own, with no exporter to install. That becomes useful later, when this box ends up on the same dashboard as some considerably more expensive hardware.
None of these numbers are tuned
I should be straight about that, because a config block in a blog post tends to read as though somebody arrived at it deliberately.
--n-cpu-moe 99. -ngl 999. --spec-draft-n-max 4. -c 262144. Those are not optimised values, they are the first set that worked. I moved things until the model loaded and answered, and then I stopped, because it loaded and answered.
There is almost certainly throughput sitting on the table. --n-cpu-moe in particular is a dial, not a switch: 99 shoves every expert layer to the CPU, and it is entirely possible that keeping some of them resident would be faster on a card with 12 GB rather than 8. I have not tested that. I have not tested any of it.
And I cannot sensibly test it yet, for a reason that arrives in the next section: tuning this box before fixing the way the card is physically connected would be measuring the wrong thing very precisely.
So the order of operations is fix the link, then measure, then tune. Consider it added to the todo list, which is where the last one of these ended up and, in fairness, that one did eventually get done.
Where this actually leaves off
Two things, pulling in opposite directions.
The first is genuinely cheering. A 12 GB consumer graphics card, in a five-year-old budget desktop, running a 26B model with speculative decoding and vision support, and a context ceiling I have set high and not yet had the nerve to actually test. Not because the card is secretly powerful, but because the model’s architecture and the runtime’s flexibility let you put the expensive parts somewhere else. That is a real capability, on hardware I already owned for other reasons, and nobody had to give me permission for any of it.
The second is that I did not know what my own machine was doing. It had been running on a single PCIe lane, at gen-1 speeds, for months, while I formed confident opinions about its performance and wrote most of a blog post about it. The gap between “it works” and “I understand it” is, one more time, wider than it looks.
Which is the right note to leave on, because the next post is entirely about that gap, on hardware I understood even less.
Part two: two DGX Sparks in a studio eight thousand miles away, a checkpoint that fits on no machine I own, a vLLM image I had to compile myself from an unmerged pull request, and a memory setting that repeatedly switched both computers off from the other side of the planet.
Disclaimer, as ever: this post was written with the help of Claude Opus, gemma-4-26B-A4B-it and Qwen3.8-Flash-Next, and as ever I’ve read and approved every word - including the ones where I am, unambiguously, the entire problem. Converting a model I did not understand, spending the best part of a week proving I could not do it, and then running a graphics card at a single PCIe lane for months while forming confident opinions about its performance are regrettably, verifiably, entirely my own. In the mailcow post I mentioned that one day I might have my own AI server running, and added it to the todo list. This is that todo item. It took rather longer than expected and taught me considerably more about myself than about AI.