AI @ Home - Part Two: Two Borrowed Supercomputers and Everything That Went Wrong With Them

Somebody lent me two DGX Sparks. A checkpoint that fits on no machine I own, a vLLM image compiled from an unmerged pull request, and a memory setting that switched both hosts off.

Published: September 23, 2026Updated: September 23, 2026

Part one ended with a complete stack on a second-hand desktop: an Arc B580, llama.cpp, Open WebUI, web search. One box, down under, answering only to me. It also jumps forward at the end to show where that box got to, using things I only learned here.

This is the bit in the middle, and it starts the day after.

Two DGX Sparks turned up on loan, with the vague brief of “see what you can do with them,” which is the kind of plan that sounds complete right up until the hardware is powered on and you have no idea what you’re actually going to serve with it.

Where the hardware came from, up front: I don’t own these. They are Dell loaner units supplied to Jeff at Craft Computing, and I have them on loan from him in return for the findings written up here. The Ansible inventory group is called craft-gb10 for exactly that reason. Everything below is what I actually hit, including the parts that don’t flatter the hardware.

I want to be clear about what that means, because it isn’t a small thing. A pair of GB10s is wildly beyond what I would otherwise have my hands on. My homelab budget does not stretch to 256 GB of unified memory across two Blackwell parts with a 200 Gb/s interconnect in the box - and neither does most people’s, though I’ll concede some budgets genuinely do and I’m not one of them. The failure modes below are ones I would simply never have encountered on my own kit. Jeff didn’t have to hand this to a stranger on the internet to go and break. He did. That’s the only reason this half of the story exists.

This is the build log for what happened next: an Ansible role that deploys vLLM as an OpenAI-compatible endpoint, an Open WebUI front end with real SSO and a real database behind it, a Grafana dashboard that taught me something about colour, and a procession of models that each failed in their own educational way before one of them worked.

If you want the short version: the GPU was never the hard part. The hard part was that a GB10 has 128 GB of unified memory, which means the GPU’s memory and the host’s memory are the same memory, and almost every piece of tuning advice on the internet assumes they are not.

What I was actually building

Here is the part that shapes every decision below: none of these machines are in the same place. Not the same rack, not the same building, not the same continent, not the same hemisphere.

Three sites, two continents, two hemispheres, four completely different ways of not having enough memory:

• YoloColo, in Washington State - aifrontend-sr-1, doing a lot of jobs at once: Open WebUI, the Ansible control node, and the monitoring stack. Alongside it, aiembed-sr-1, a CPU-only VM on a Xeon 6138 that does nothing but embeddings. To be clear about who owns what, since the next bullet needs the contrast: YoloColo is my colo provider - the racks and the power are theirs, but everything of mine in them is hardware I own outright. And to be clear about what’s new here: the site already ran a full stack - Patroni Postgres, Traefik, the lot - long before any AI existed in it. This project bolted two nodes onto an estate that was already working. Do you think I’m made of money?
• Craft Studio, in Oregon - the two loaner GB10s, reached over Twingate. They are one clustered engine, not two servers, for reasons that took me a month to earn. Worth being clear about what “on loan” means here: the Sparks never shipped anywhere. They sit in Jeff’s studio, and what I was actually lent was access. Which is why there’s a tunnel in the diagram rather than a courier receipt.
• The land down under - the Arc B580, reached over WireGuard, and next to it a separate box with an RTX 3080 Ti running ComfyUI. Also where I am, which makes those the only machines in this post I can walk over and physically unplug.

YoloColo to Craft Studio is a neighbourly hop down the coast. YoloColo to down under is the wrong side of the Pacific, and that link is a fact you design around rather than one you optimise away.

There’s a pleasing absurdity to the topology that I want to name rather than hide: I sit down under, administering a control node at YoloColo, which drives GPUs at Craft Studio that I don’t own, and also reaches back down under to a graphics card sitting a few metres from my chair. The shortest physical distance in the whole system is the one with the most network between it.

Both remote links are one-way, and that is the single most load-bearing property in this whole build. aifrontend-sr-1 can reach the GPU nodes. The GPU nodes cannot reach back. Nothing out there can initiate a connection inward.

Most of the time that is invisible. Where it stops being invisible is monitoring, because a large slice of that ecosystem assumes agents push. Anything push-based was off the table before I evaluated it, which is why the design ended up pull-only. More on that when we get to Prometheus.

Nothing is exposed to the internet except the Open WebUI route, and that sits behind Authentik. Both inference endpoints answer on their private sides only.

Four ways to not have enough memory

The split that runs through this entire post:

Node	Where	Hardware	Memory it has	Runtime	Serving
gb10-01 + gb10-02	Craft Studio	2x GB10	128 GB unified, per node	vLLM, self-built	Qwen3.8-Flash-Next, FP8
docker-sr-1	Down under	Arc B580, B450 + R5 5500	12 GB VRAM + 64 GB host	llama.cpp, SYCL	gemma-4-26B, GGUF
aiembed-sr-1	YoloColo	Xeon 6138 VM	host RAM, no GPU at all	vLLM, patched CPU	Qwen3-VL-Embedding-8B
the 3080 Ti box	Down under	RTX 3080 Ti	12 GB VRAM	ComfyUI	SD 3.5

That memory column is the reason this series has a thesis. Every one of those is the same question asked of different hardware: what fits, and what do you give up to make it fit? The answers contradict each other. The correct move on one of those machines is actively wrong on another, and I did not understand that when I started. Attempt one is what the misunderstanding looked like.

A note on the vLLM role’s intel branch. The vllm-lab role does carry a vllm_backend: intel path that builds a vLLM XPU image from source and passes through /dev/dri. It is written and kept working, but it is not what the B580 is running today. The B580 runs llama.cpp, for reasons the next section gets into.

Why Ansible, actually

Worth answering directly, because it’s a fair question: why put all of this in Ansible instead of just doing it? It’s a handful of machines doing a handful of jobs. I could have SSH’d in and typed.

Because I can read Ansible.

Everything in this build was done with AI assistance. I say so plainly in the disclaimer at the bottom, and it’s been true since the first playbook of this project. The question everyone asks about AI-written code - how do you handle errors introduced by the AI tools - is my question too, about every file in that repo. Part three spends a whole section on the person who gave the best answer to it. My answer is his answer, and the reason that answer works here at all is the format: I read it.

But “I read it” only works if the thing is readable, and if you get to read it before it does anything. That’s the actual argument here, and it has almost nothing to do with configuration management:

• It is declarative and legible. A role states the end state it wants. I can read one and know what it will do to a machine without running it, and without being an Ansible expert.
• It is separated from the thing it acts on. I write and review on the control node, in git, on a different continent from the hardware. The code and its execution are not in the same place, and there is a deliberate gap between “this exists” and “this has happened”. Review lives in that gap.
• It runs twice. Idempotency means I can read it, run it, read what changed, and run it again to confirm nothing else moves. None of that depends on my memory of what I intended.
• It is in version control. Every change to these machines has a commit, a diff, and a reason attached.

The alternative, which is what most people do and what I did at the start, is to paste commands into a terminal and watch what happens. That works. It also means the only record is your shell history, the only review is whether it threw an error, and anything AI-suggested becomes a thing you executed rather than a change you approved.

You should never run code from any source arbitrarily. Not from a blog post, not from Stack Overflow, not from a model. That principle did not change when the source became an AI. It just became far easier to violate, because the output now arrives fluent, confident, correctly indented, and in volume.

Ansible is the mechanism by which I force myself to read it first. That is the entire reason.

And it is the same instinct as refusing to run a GGUF from an uploader I’ve never heard of. Same question at a different layer: what is this, who made it, and have I actually looked at it?

And the operational version of that, because “I review it” is worth nothing if the thing being reviewed can also run it. How do I use an AI without letting it - like any good intern with domain admin privileges - delete the production database? I don’t let it. From the earliest days of this project, when Claude was writing these playbooks for me, the workflow has been: the AI writes, I review, I commit, I push to git - and then a human being, me, pulls and executes on the target. The AI has no credentials, no SSH keys, no seat at any terminal in the estate. Not “trust but verify.” It cannot verify what it cannot touch. Even when its answer is exactly right, I only ever run what I have personally checked.

The uncomfortable half of that sentence is the word personally. Zero Trust, full bloody stop - but the chain of zero trust has to terminate in somebody, and the person who gets to run what nobody else has read is, at that moment, fully trusted by everyone. If I can’t trust myself, or any other person, to review before executing, on what principle should I trust the AI to be exempt from a rule humans were never trusted to skip? The AI isn’t the weak link in this workflow. It’s the only part of it that provably cannot reach the trigger. I’m the part that can. That’s not an argument for letting it run things. That’s the argument for why I check my own work like it belongs to someone I don’t trust. It does now.

The GB10 journey, in two stages

Before the individual attempts, the shape of the thing, because it took weeks and it splits cleanly in half.

Stage one: two machines, several models, no clustering. The Sparks arrived as two independent servers and I treated them as exactly that. Each pulled its own model and served its own API, and I worked through a run of candidates trying to find something that fit in 128 GB and was worth running. That stage produced most of the lessons in this post and almost none of the impressive numbers.

Stage two: two machines, one model. Eventually I wanted to run something that could not fit on a single node at all. That meant fusing both Sparks into one engine with the model split across them, over the cable that had been sitting there unused the whole time. Different class of problem entirely, and in the end it needed a container I had to build myself.

	Stage one	Stage two
Topology	2 independent servers	1 clustered engine, TP=2
Models	Ornith-1.0-35B, Qwen3.6-35B-A3B, Nemotron-3-Super-120B	Qwen3.8-Flash-Next, 180B
Image	stock NGC and vllm-openai tags	built from an unmerged PR
The 200 Gb/s link	idle	carries every token, twice per layer
Hardest question	what fits	what fits, and what kills the host

Stage one runs through the next three attempts. Stage two is attempt four, and it’s where the hardware finally gets used for what it obviously was for.

Stage one, attempt one: Ornith-1.0-35B, or how I learned that GGUF is not a universal solvent

The first model I pointed at it was deepreinforce-ai/Ornith-1.0-35B, in GGUF, and I reached for it without really thinking about it.

Look at the dates and you can see exactly why. ornith-1.0-9b-Q4_K_M.gguf landed on the B580 on 12 July, which is where part one finishes, and the vLLM repo’s first commit is 13 July. I had something that had just worked, so I reached for the same model, in the same format, one size up, and pointed it at hardware roughly forty times more expensive.

That is not a stupid instinct. It’s just an untested one.

I should be honest about why, because it’s the actual lesson of this section and it isn’t a technical one. GGUF is what I knew.

Remember the LM Studio dropdown from the top of this post. That’s where the habit was formed: pick a model, pick a quant, download a file, it runs. Then my own B580, sitting in a land down under, runs llama.cpp, where GGUF is again the native format and the workflow is again “pick a quant, point the server at the file, go.”

Two systems, both of which had worked, both of which taught me the same wrong lesson. So when new hardware arrived I reached for the file type that had never once let me down, on a completely different runtime, and treated a format as though it were a universal container.

Which is the joke, really. By this point I had learned to be careful about who built a model file. I had not yet learned that the file type itself was a choice with consequences, rather than just how models are shipped.

The reasoning I gave myself at the time sounded fine: Q6_K is around 28.5 GB against roughly 69 GB for bf16, near-lossless, and leaves far more of the 128 GB pool for KV cache. On paper the full 256k context fits.

vLLM disagreed:

GGUF model with architecture qwen35moe is not supported yet

No quantisation level changes that. It is not a size problem or a memory problem. vLLM’s GGUF loader has an explicit list of architectures it understands, and this MoE variant is not on it. I spent longer than I would like to admit trying different quants before accepting that the error message meant exactly what it said.

The mistake underneath it: vLLM is not llama.cpp, and GGUF is not a portable format so much as llama.cpp’s format that other projects partially support. llama.cpp treats GGUF as the target and does the conversion work to get architectures into it. vLLM treats safetensors as the target and supports GGUF as a side path with a much shorter list. Same file extension, entirely different level of commitment behind it.

The fix was to stop being clever and serve the safetensors base repo, which is what the model card distributes anyway. The role grew a vllm_model_source switch for it:

• gguf downloads a single .gguf file and serves it from a local path
• hf lets vLLM pull the whole sharded safetensors repo by ID into a mounted cache

That takes bf16 weights at roughly 69 GB, which leaves about 40 GB for KV cache, which gets you to about 128k of context rather than 256k. A real downgrade from the plan, arrived at by discovering that the plan was impossible.

The container tag whack-a-mole

While that was happening I was also cycling through NGC container tags, in this order:

Tag	Outcome
nvcr.io/nvidia/vllm:26.06-py3	didn’t work
nvcr.io/nvidia/vllm:25.12.post1-py3	didn’t work
nvcr.io/nvidia/vllm:25.11-py3	worked

Rolling backwards through container tags to find one that runs is not a satisfying engineering experience, but on brand new silicon it is the normal one. The GB10 is Blackwell, sm_121, and the ecosystem was still catching up. A later CUDA 13.2 bump moved things forward again.

Worth spelling out the mechanism, because it explains why the roulette even exists. The box ships DGX OS 7.5 - Ubuntu 24.04 underneath, NVIDIA’s 580 open kernel modules out of the box. And a published vLLM image carries its own CUDA, which on this architecture has to match the host driver’s CUDA series. The documented escape hatch, CUDA forward compatibility, simply does not work here - I have no deeper theory than “it does not,” which is the honest state of that particular knowledge. So the driver got upgraded to the 595 open modules to line the host up with the image, rather than the other way round.

The alternative, for completeness, is to stop taking the published image at all and build vLLM yourself against the CUDA that is actually installed on the host. If that sentence sounds familiar, it is: it is exactly the road stage two ends up walking, for a completely different reason. On day-0 silicon there seem to be only two stable states - match the host to someone else’s image, or build the image to match the host. The middle, where you just pull the latest tag and it runs, is not one of them.

Hence the lesson I’d actually pass on: pin the tag in group_vars and treat it as a variable you are actively managing, not as a detail you set once. It changed four times in two weeks.

One other unglamorous job falls out of all this pinning: the model gets pre-fetched into the cache before Compose starts, so that the first docker compose up isn’t also a multi-hour download with no progress output. Getting that task right was two container-shaped lessons of its own - the image’s entrypoint silently swallowing the command you thought you were running, and Docker’s auto_remove deleting a run-once container before anything could read why it failed. Both cost real time and neither is about AI. If you take one thing from that episode: before you debug your flags, check what the image’s entrypoint already prepends to them. The playbooks themselves aren’t going in this post - if anyone ever actually wants the detail, that’s a post of its own, or the day I decide to publish them.

Stage one, attempt two: Qwen3.6-35B-A3B-FP8, the one that just worked

The move that actually solved the memory problem wasn’t a flag, it was a checkpoint. Qwen/Qwen3.6-35B-A3B-FP8 is the same class of model, quantised to FP8, which is native on Blackwell. vLLM reads the quantisation straight out of the checkpoint config, so there is nothing to configure:

vllm_model_source: "hf"
vllm_model_repo: "Qwen/Qwen3.6-35B-A3B-FP8"
vllm_served_model_name: "Qwen3.6-35B-A3B-FP8"

vllm_dtype: "auto"
vllm_gpu_memory_utilization: 0.90
vllm_max_model_len: 262144

FP8 weights land at roughly 35 GB instead of 69. That leaves about 75 GB for KV cache, and the model’s full native 256k context fits with room to spare. The thing I’d failed to get by picking a clever GGUF quant, I got for free by picking a checkpoint the runtime supports natively.

The serve flags came off the model card:

vllm_extra_args:
  - "--enable-prefix-caching"        # big win on shared/repeated prompts
  - "--reasoning-parser=qwen3"       # <think> reasoning content
  - "--enable-auto-tool-choice"
  - "--tool-call-parser=qwen3_coder" # Qwen3.6 tool-call format, per the card

--tool-call-parser is worth flagging because it is easy to get wrong quietly. The earlier config had qwen3_xml, carried over from the Ornith card. Point the wrong parser at a model and tool calls don’t error, they just come back as prose that looks like a tool call, and your client sits there wondering why nothing executed. Read the card, use its parser.

Thinking mode is on by default here. Clients turn it off per request with chat_template_kwargs.enable_thinking=false rather than it being a server-side decision, which is the right split.

The digression I owe you about quants

I have been saying “quant” for most of this post the way a digger says “klicks” - completely confident, never once defined, and quietly assuming the whole room converted it in their head the same instant I said it. It’s load-bearing for everything below, so here is the interjection.

A model’s weights are just numbers, and “precision” is how many bits each of them is stored in. A model is trained with the weights at high precision - usually bf16, which is two bytes per parameter - because small changes to those numbers are how learning happens. Running the thing, though, doesn’t need two bytes. It can tolerate the numbers being stored roughly, rounded to a smaller set of possible values. That rounding is quantisation, and a quant is one particular recipe for doing it: which set of values, how many bits, how the errors get distributed across the matrix.

The arithmetic is the whole game, and it’s mercifully simple. Two bytes per parameter, 35B parameters, roughly 70 GB. One byte - FP8 - and it’s 35 GB. Half a byte and it’s about 17. What you give up in exchange is the interesting part, and the honest answer is it depends on the recipe, the model, and you should measure: usually a little quality, occasionally something specific and weird, and always a bit more bookkeeping about which runtime and which hardware actually understand the format you picked. Which is the entire whack-a-mole section, stated in one sentence.

And there is a trapdoor in that arithmetic that part one already made concrete for me. “Get the biggest model, squeeze it into the smallest quant it’ll fit” sounds like free lunch: a 35B at Q4 beats a 9B at full precision on any leaderboard, and the card says it fits, so ship it. Sometimes it does. Sometimes the squeeze is where the model lives, rather than where it visits, and what comes out the other side is that schizophrenic thing from part one - fluent, confident, and having lost the thread of its own identity. Coherence is not one number sitting in the weights; it’s a pattern distributed across all of them, and aggressive rounding is precisely the operation that thins a distributed pattern out. It gets worse one layer down the stack, too: a distillation trains a small student on the outputs of the squashed model, so every incoherence the quant introduced becomes ground truth the student learns from, faithfully, with its own quantisation errors stacked on top. Quantisation noise is a thing you inherit, not a thing you filter.

I want to actually measure all of that one day - same checkpoint, ladder of quants, the same awkward prompts run down it, the coherence failure put on a curve instead of asserted. It is a real experiment with real axes and I already have the hardware to run it.

Three meanings of "format" in one sentence, and I still picked this one to stand on. The digression above is the wound; the meme is the discharge note.

Then the naming, because of course the naming is a zoo. The llama.cpp world - LM Studio’s dropdown, my B580 - names quants like a tyre specification: Q4_K_M, Q6_K, where the number is bits-ish and the suffixes are one engineer’s opinion about where to spend the bits. The rest of that sentence is genuinely all you need to read those names in this post. The datacentre world speaks in number formats instead - FP8, and NVFP4, where the letters name an actual hardware-defined way to store a number, and the GPU has a circuit that reads it fast. One sidebar inside the sidebar worth having: NVFP4 is NVIDIA’s, in both senses - their format, and the hardware that executes it fast lives in their shops. An NV prefix on a quant name is a compatibility promise and a vendor lock-in, printed on the same label. Run it on the intended silicon and it’s magic; wander off the map and you’re back to the whack-a-mole table, asking which build knows what. Two ecosystems, two vocabularies, one idea: the weights are allowed to be approximate, and how much you approximate them is a dial, and this entire series has been me turning the dial into a wall. The disclaimer at the bottom of this post has been joking about “assuming a format was a format” for exactly this reason. It isn’t one dial with a smooth dial-plate; it’s a dial with detents, and several of them are labelled in a language - and now a brand - the other dial-plate doesn’t speak.

With that said, attempt three picks the most exotic detent available.

Stage one, attempt three: Nemotron-3-Super on the second node

With one Spark serving happily, the second one was sitting there doing nothing, so it got nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4 - a 120B MoE with 12B active, in NVFP4, following the DGX Spark recipe off the model card.

This one is interesting because almost everything about it is different from the Qwen deployment, and the role absorbed all of it through variables rather than branches:

vllm_image: "vllm/vllm-openai:v0.20.0"
vllm_command_style: "openai"        # this image's entrypoint IS the api_server
vllm_max_model_len: 1000000         # 1M context

vllm_extra_env:
  VLLM_NVFP4_GEMM_BACKEND: "marlin"
  VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
  VLLM_FLASHINFER_ALLREDUCE_BACKEND: "trtllm"
  VLLM_USE_FLASHINFER_MOE_FP4: "0"

Different image, different entrypoint convention, different quantisation format, different GEMM backend, and a context window with six zeroes in it that needs an explicit environment variable to unlock, because vLLM assumes a number that large is a mistake.

It also needed a reasoning parser that doesn’t ship with vLLM at all - a super_v3_reasoning_parser.py living in the model’s own Hugging Face repo. That produced two more role features that have turned out to be generally useful:

vllm_fetch_files:
  - url: "https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Super-120B-A12B-NVFP4/resolve/main/super_v3_reasoning_parser.py"
    dest: "{{ vllm_base_dir }}/super_v3_reasoning_parser.py"
    mode: "0644"

vllm_extra_volumes:
  - "{{ vllm_base_dir }}/super_v3_reasoning_parser.py:/app/super_v3_reasoning_parser.py:ro"

Fetch arbitrary files to the host with HF bearer auth applied automatically, and bind-mount arbitrary paths into the container. Between those two and vllm_extra_env / vllm_extra_args, the role can serve a model with bespoke requirements without anyone editing the role.

That config now lives commented out in host_vars/gb10-02.yml, kept as the reference for the NVFP4 recipe, because that node has since been given a different job. More on that at the end.

And this is where the machines started dying

Nemotron is where I first hit out-of-memory on a GB10, and it became the defining problem of this whole project.

Not because running out of memory is surprising. Because of how it failed.

It killed the machine, not the process.

That is not what is supposed to happen. A healthy Linux box handles memory exhaustion by picking the greediest process, killing it, and carrying on. You lose a service, you keep a server, you go and read the logs to find out what happened.

What I got instead was a node that switched itself off. Not a hang I had to force my way out of, not a wedged box needing the power button held in - just a machine that was running, and then was off. Turning it back on was the entire recovery procedure.

Which is somehow worse than a lockup, because it looks so tidy. No SSH, no console, nothing in the logs written afterwards, and no evidence of a struggle. A server that was working, and then simply wasn’t.

“Just turn it back on” is doing a lot of work in that sentence

Here is the part that turned an annoyance into the defining problem of the project.

Those machines are approximately 13,000 km away. Eight thousand miles, an entire ocean, and about twenty-six hours of flying. They are not under my desk. They are not in a rack I have a key to. They are in someone else’s studio, in a different country, in a hemisphere where the sun is up when mine is down.

So the recovery procedure is trivial and completely out of reach. The single simplest physical act in all of computing - pressing a power button - is the one thing I categorically cannot do.

And the machines belong to the person whose studio they’re in. Which means every crash is not just downtime. It is a message to Jeff. It is somebody else’s afternoon, interrupted, to go and press a button on hardware he lent me for free so I could write about it. There is a limit to how many times you can do that before you have thoroughly worn out a kindness.

If you think I’m being dramatic for effect here, go and ask Craft Computing how many messages he received from me that amounted to nothing more than “sorry mate, could you turn it back on?” I’m not going to print the number. He has it. He was, throughout, considerably more gracious about it than I would have been.

And while his name is on the screen: if you have read this far into this much madness - unified memory allocation, netplan stanzas, a genuine argument about Grafana’s colour assignment - then you are, fairly precisely, the person his channel is made for. Go and have a look. That isn’t part of the arrangement and nobody asked me to say it. It’s just true, and you’d probably have found him eventually anyway.

That’s the pressure that shaped everything downstream. Not “the server keeps falling over”, but “the server keeps falling over and I cannot pick it up.” Remote hands you do not employ, on a timezone that does not overlap with yours, attached to a favour you would rather not keep spending.

It is why the eventual fix in stage two is three independent layers rather than one setting, why one of those layers exists purely to guarantee I keep an SSH route in, and why there is a playbook whose entire job is bringing these nodes back up cleanly. None of that is engineering rigour for its own sake. It is what you build when the alternative is asking a favour.

My working theory at the time, and I want to be clear it was a theory: this is unified memory biting again. When the GPU reservation and the host’s RAM are the same 128 GB, memory handed to CUDA is not memory the kernel can take back. Push the reservation high enough and the OS is left trying to reclaim from a pool where most of it is untouchable. Everything blocks waiting on memory, including whatever would have resolved the situation.

The immediate fix was blunt and it worked: drop gpu-memory-utilization from 0.90 to 0.80. Give the operating system enough room that it never gets into the fight in the first place.

Two things about that are worth saying plainly.

It is a mitigation, not an understanding. I moved a number until the symptom went away, which is a completely legitimate thing to do when you need the machine working today, and is not the same as knowing why. I did not know why. I had a hypothesis and a number that helped.

And on any normal machine, this setting cannot do this to you. On a discrete GPU, gpu-memory-utilization bounds what happens on the card. Set it wrong and you get a CUDA OOM, an error message, and a process that dies while the host shrugs. On a GB10 the same knob decides how much memory the host is allowed to keep, which turns a tuning parameter into something that can take the whole box down.

I get properly to the bottom of this later, in stage two, where it stops being survivable by tuning and needs three layers of actual protection. The short version, for now: the theory was broadly right, the kernel’s own OOM killer never gets a chance to run, and 0.80 was buying me headroom rather than fixing anything.

The startup cost nobody warns you about

Still in stage one, and applying to every model above: somewhere in here I noticed that recreating a container cost tens of minutes before the API answered, every single time, and that this was not a download problem, because the weights were already cached.

vLLM defaults VLLM_CACHE_ROOT to ~/.cache/vllm. Inside a container, that is /root/.cache/vllm. It dies with the container. So every docker compose up --force-recreate was throwing away every torch.compile artefact and every captured CUDA graph, and rebuilding them from cold.

vllm_compile_cache_dir: "{{ vllm_base_dir }}/compile-cache"

Bind that to /vllm-cache, point VLLM_CACHE_ROOT at it, and compiled kernels survive the container. Delete the host directory to force a rebuild.

This is a completely generic containerisation mistake - ephemeral cache inside an ephemeral filesystem - and I want to be honest that I didn’t spot it from first principles. I spotted it because I got annoyed enough at the wait to go looking for what it was recomputing.

Stage two: making the two Sparks into one machine

Back to the Sparks, and this is the half the hardware was always for.

Stage one treated the GB10s as two independent servers, which is what they look like: two boxes, two power cables, two of everything. They are not two servers any more. They are one.

The model that forced it is Qwen/Qwen3.8-Flash-Next-FP8: 180B parameters, 6B active, 256k native context, and a 185.5 GB checkpoint. A GB10 has 128 GB. It does not fit on one node, and no amount of cleverness makes it fit on one node.

So both Sparks now run one engine with tensor parallelism split across them. gb10-01 is rank 0 and the only node that answers on port 8000. gb10-02 runs --headless, holds the other half of the weights, and has no API of its own. Every other argument is identical on both ranks by construction, because the template emits one list and varies only the rank and --headless. Shards cannot disagree about the engine config that way.

No Ray. Current vLLM doesn’t depend on it - it’s absent from requirements/common.txt, absent from the official image, and run_cluster.sh is gone from main. Coordination is plain torch.distributed: rank 0 binds a port on the fabric, the other rank dials in. No daemon, nothing to install, nothing to supervise. Deploy order doesn’t matter either. Whoever comes up first waits at the rendezvous.

The cable, and the trap hiding in it

This only works because the two Sparks are cabled directly to each other, QSFP to QSFP, no switch. Every token crosses that link twice per layer. Over the management LAN the model would run and would be unusably slow.

No switch also means no PFC and no ECN to configure, which removes the genuinely hard part of most RoCE deployments. What it does not remove is this:

enp1s0f0np0    <-- use this one
enP2p1s0f0np0  <-- and NOT this one

Those are not two names for one interface. They are two separate netdevs, each with its own MAC, its own ifindex and its own RoCE device, attached to the same physical port. LLDP proves it by showing each as the other’s neighbour, which is only possible because they share a wire.

Address both and two of the host’s interfaces land in the same broadcast domain. ARP resolves to whichever answers first, and NCCL is then free to bind a different rail than the ranks agreed on. Nothing errors. It just gets slow and non-deterministic, which is the worst category of fault. The deploy asserts that only one interface holds a fabric address, because this is not something you want to discover from a throughput graph.

The fabric addressing is derived rather than hand-written: each node keeps its management host octet on a different network, so 10.0.0.21 becomes 10.10.10.21. Adding a node needs no addressing decision and the two schemes can’t drift apart.

The image did not exist, so I built it

Here’s where it got expensive.

Qwen3.8-Flash-Next declares architecture Qwen4ExpForConditionalGeneration, which is not in vLLM main. No qwen4_exp.py, no Qwen4 entry in the registry. A general tag loads the config and then rejects the architecture outright.

vLLM ships day-0 support for launches like this as a purpose-built branch image, so there was one. But the day-0 image couldn’t serve the model within memory, because the thing that makes it fit is VLLM_PLE_MMAP, and that lives in an unmerged pull request. The architecture registration and the memory fix existed in exactly one place: PR #54129.

So the repo grew a playbook that builds vLLM from source on both nodes. Three things went wrong, and all three are the kind that waste hours rather than failing fast:

Pinning a commit SHA was worse than not pinning. The first version pinned the PR head directly and broke within a day. The author pushed, the old commit stopped being reachable from any ref, and the checkout died with reference is not a tree. It now fetches refs/pull/54129/head from upstream, which GitHub maintains for the life of the PR and always resolves to current. A pin to an unreachable object is worse than no pin, because a clone cannot fetch it at all.

The default base image is x86-only. vLLM’s Dockerfile defaults BUILD_BASE_IMAGE to a pytorch/manylinux2_28-builder tag that publishes no arm64 image. Docker pulls it on a GB10 anyway and the first RUN dies with exec format error, preceded by a platform-mismatch warning that is trivially easy to scroll past in a build log. PyTorch ships the aarch64 builder under a different repository, not a different tag, so the pinned default can’t just be reused.

And the one that actually hurt: torch_cuda_arch_list defaults to '7.5 8.0 8.6 8.9 9.0 10.0 11.0 12.0'. GB10 is compute capability 12.1, which is not in that list. A default build succeeds. It takes hours. It produces an image with no kernels for this GPU. Nothing tells you until you try to serve. It has to be passed as plain 12.1 and not 12.1a, because vLLM’s CMake declares bare versions and derives the suffixed form itself for the kernels that need it.

The build takes hours, runs on both nodes in parallel, and stops the vLLM stack first, because nvcc will take every GB it can get and leaving a ~100 GB model server running alongside it is how you get OOM-killed at hour three. The log goes to a file on the node rather than through Ansible, because a vLLM build log runs to hundreds of thousands of lines and piping it back means the whole thing prints on failure, burying the one line you need.

The memory budget, finally measured

This is the part I’m most pleased with, because it stopped being guesswork.

gpu-memory-utilization at 0.90 OOM-killed both hosts. So did 0.85. The FP8 checkpoint is 185.5 GB, which is 92.8 GB of weights per rank, 71% of a node before a single KV block exists. The N-gram table is ~51 GB of that and already FP8, so there was no quantisation headroom left to find.

One unit note before the budget proper, because this post has spent its whole life insisting that a format is not a format, and decimal GB versus binary GiB is the same trap in a lab coat. The panel size: the spec I remember says 128 GB, the MemTotal I measured says 121.6 GiB - which is 130.6 decimal GB, so either the marketing figure was quietly binary all along or I have been quoting the sticker instead of the chip. I have not torn the box open to find out; the arithmetic uses the measured number, because that is the one the kernel actually hands out. Everything in this section is decimal GB, checkpoints and nodes alike - conversions, not fresh measurements, so treat the decimals as rounded. Which is also how one checkpoint is both 185.5 GB and 172.78 GiB, and how I got to write both of those numbers in the same post without lying.

VLLM_PLE_CPU_OFFLOAD does not help here, for the reason this whole post keeps circling: host and device are the same pool, so it relocates the table instead of freeing it. VLLM_PLE_MMAP genuinely removes it, by turning the table into page cache served from NVMe - evictable memory rather than anonymous. With it, resident weights drop to ~67 GB per rank.

Then a real measurement fell out of a failure. At util 0.80, earlyoom SIGTERMed vLLM on gb10-02: available memory held steady near 32% through weight loading, then fell to 7.87% in under 40 seconds. That drop puts a number on everything vLLM’s reservation doesn’t account for - CUDA context, NCCL buffers, three vLLM processes, surviving page cache. It’s about 16 GB (14.8 GiB as the panel reports it).

Which turns the whole thing into arithmetic:

available at peak = 130.6 - (130.6 x util) - 15.9

util	reserved	KV cache	free at peak
0.80	104.5 GB	37.3 GB	10.2 GB	killed
0.75	98.0 GB	30.8 GB	16.7 GB	thin
0.70	91.4 GB	24.2 GB	23.3 GB	current
0.65	84.9 GB	17.7 GB	29.8 GB	most headroom

All figures decimal GB, node included - the measured MemTotal is 121.6 GiB, which is 130.6 GB, and the marketing “128 GB” is the sticker, not the chip.

Note what that 32 GB drop in 40 seconds actually is: it’s the KV cache allocation, not weight loading. The weights arrived gradually and were already in. So the dangerous moment is the end of startup, not the middle, which is the opposite of where I’d been watching.

And gpu-memory-utilization now has two jobs. What it reserves is KV cache. What it does not reserve is the page cache the mmap’d table is served from. Set it too high and the table stops fitting in cache, so lookups fall through to NVMe on every miss. The server works. It’s just slow. That is the hardest possible failure to notice, and it’s why this sits at 0.70 rather than the 0.85 the KV numbers alone would argue for.

Which is also, finally, what the NVMe row on the dashboard is for. Sustained read throughput with low GPU utilisation means the table is thrashing, and the fix is to lower util, not raise it.

When the OOM killer never runs

Here is the answer to the thing that had been killing these machines since Nemotron.

A healthy Linux box answers an out-of-memory condition by killing something and carrying on. These nodes were going down instead. Back in stage one I guessed that unified memory was the culprit and bought myself room by dropping gpu-memory-utilization to 0.80. That guess was directionally right and the mitigation was real, but neither was the mechanism.

The mechanism is that the kernel was livelocking before its own OOM killer could act: reclaim frees nothing, every task blocks on memory, and the killer never gets scheduled.

That is why there was never anything in the logs. The process that was supposed to write the explanation was itself waiting on memory that was never coming back. And it is why lowering the utilisation appeared to fix it: I wasn’t preventing the livelock, I was staying far enough away from the cliff to not fall off it. At stage two’s memory pressure, no value of that knob is far enough away. Once you’re there, nothing in userspace runs, so anything that helps has to fire earlier. There’s now an oom_guard role, deployed before vLLM so the protection exists before the thing it protects against, with three independent layers: earlyoom as a userspace killer, oom_score_adj so the kernel picks the vLLM container if it does run, and an sshd drop-in at OOMScoreAdjust=-900 so there is always a way back in.

Two configuration details in there are worth more than the rest of the role:

The sshd drop-in is the one I’d defend hardest, and it is entirely a consequence of the 13,000 km. OOMScoreAdjust=-900 tells the kernel that whatever else it sacrifices under pressure, it does not get to take my way in. On a machine I could walk to, that setting is a nicety. On this one it is the difference between a bad afternoon and an apologetic message asking someone to go and press a button.

earlyoom ANDs its memory and swap conditions. It fires only when free memory and free swap are both under threshold. These nodes keep most of their 17 GB of swap free, so with the stock setting the swap test is never satisfied and earlyoom never fires, no matter how bad memory gets. The swap threshold is now 100% so that memory alone governs. A protective daemon that is running, healthy, and structurally incapable of ever acting is worse than not having it, because you think you’re covered.

And the trigger has to sit below normal free memory or it kills a healthy server. vLLM reserves its utilisation share, so a working node runs with roughly (1 - util) free. Trigger above that and you cause precisely the outage you were preventing. The role asserts the relationship instead of trusting whoever edits the numbers next.

One more unified-memory sting. You cannot bound this with a container memory limit, because CUDA unified memory is not charged to cgroups. A mem_limit would bound page cache and host-side anonymous memory and would not touch vLLM’s ~110 GB reservation at all. The containment has to be a userspace killer and OOM scoring, because the normal tool for this job silently doesn’t apply.

Two smaller things that cost real time

Serve flags are now checked against the image before deploying. A flag the build doesn’t recognise makes vLLM exit 2 during argument parsing, and restart: unless-stopped turns that into a crash loop that reads exactly like a hang - which you discover forty minutes later, on the wrong node. --swap-space was a V0-engine flag, is gone from current vLLM entirely, and cost a deploy cycle. The check reads --help=all rather than --help, because vLLM tiers its help and the plain summary lists almost no flags, which made every flag look unrecognised the first time. It also only fails when confident: if it can’t produce something that looks like a flag listing, it prints SKIPPED and lets the deploy through, because a guard that blocks good work is worse than the bug it prevents.

There’s a reboot playbook now, because memory commitments are not always returned after a failed or killed load and a reboot is the only reliable way to get them back. It uses down rather than stop deliberately: restart: unless-stopped means a merely-stopped container comes back at boot, and the node would return committing ~109 GB before you could look at it, which is the opposite of what you rebooted for.

Watching it: Prometheus, Grafana and one lesson about colour

Monitoring runs on aifrontend-sr-1, the same box as Open WebUI and the Ansible control node. Everything is pull, and that is not a stylistic preference - it’s the one-way Twingate and WireGuard paths from the top of this post cashing out. Prometheus reaches across and scrapes. Nothing at either remote site ever has to find its way home:

aifrontend-sr-1 (Prometheus) --> gb10-0X :9100  node_exporter   host CPU/mem/disk/net
                             --> gb10-0X :9400  dcgm-exporter   GPU util/power/temp
                             --> gb10-0X :8000  vLLM /metrics   serving metrics

Component	Version	Where
Prometheus	v3.7.3	aifrontend-sr-1, Compose
Grafana	13.0.7	same Compose project
node_exporter	1.7.0	GB10s, apt + systemd
dcgm-exporter	4.8.3	GB10s, Compose

Scrape targets are generated from the metrics_targets inventory group, so adding a node to the inventory is the only step needed to scrape it. Targets are labelled instance=<inventory hostname> rather than ip:port, which is what makes the dashboard readable - and which matters more than usual when the addresses on the other side of a Twingate resource are not the addresses you think in.

vLLM needs no exporter. It serves Prometheus metrics on its own port, and /metrics is not behind the API key, because vLLM’s auth middleware only guards paths starting with /v1. Prometheus scrapes it unauthenticated and nothing needs configuring.

llama.cpp is the same story with --metrics, which is why the B580 can join this without a second monitoring design. Two runtimes from different projects, both speaking Prometheus, both scraped over a tunnel by the same server. The serving metrics don’t line up one-for-one between them - different projects, different names - so a genuinely unified dashboard row is still work I haven’t done.

The dashboard is committed JSON - written from my description by the same AI assistance as everything else in this series, read by me before it deployed - and auto-provisioned into a vLLM Lab folder. Grafana rescans every 30 seconds, so UI edits get overwritten on the next deploy. If you want to keep a change, export it back into the file. That is a deliberate trade: the repo is the source of truth, and the cost is that you cannot tweak a panel in the browser and have it stick.

DCGM cannot tell you about GPU memory here

This one is architectural and it caught me out. DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_FB_FREE come back empty or mirrored on a GB10, because there is no discrete framebuffer to report on. It is a unified memory part. There is one 128 GB pool.

So the dashboard has no VRAM panel. Model weights and KV cache show up under Host, memory used, against the shared pool, alongside everything the operating system is doing. Utilisation, power, temperature and energy from DCGM are all valid. Memory simply isn’t a GPU metric on this hardware.

Colour is bound to the node, and that is not decoration

Every per-node panel carries an explicit override tying gb10-01 to blue and gb10-02 to orange. I added those after taking a screenshot in which the same node was green in the GPU row and yellow in the Host row.

Grafana assigns colours by series order within each panel, and Prometheus does not return series in a stable order across different queries. So a dashboard whose entire purpose is “scan down the rows and correlate GPU behaviour against host behaviour” was silently re-labelling which node you were looking at, per row, per refresh.

Nothing errors. The numbers are all correct. The dashboard is just quietly lying to you about which machine you’re worried about.

If you add a third node, it needs a third override, or it falls back to order-based colour and reintroduces the whole problem.

Metric names are versioned, and nobody tells you

The panels were written against vLLM 0.20 metric names, from the V1 engine. On an older V0 build the same panels read empty, because the names changed:

V1 (current)	V0 (older)
vllm:kv_cache_usage_perc	vllm:gpu_cache_usage_perc
vllm:inter_token_latency_seconds	vllm:time_per_output_token_seconds

An empty panel looks identical whether the metric is missing, the target is down, or the name is wrong. Worth confirming your panels populate before you start trusting them to tell you a model is healthy.

One more small trap: counters carry the _total suffix Prometheus adds, so it is vllm:generation_tokens_total in your query even though the vLLM source and docs call it vllm:generation_tokens.

The front door: Open WebUI

An OpenAI-compatible endpoint on a private network is a curl target, not a product. The part that makes it usable by an actual human is Open WebUI on aifrontend-sr-1, deployed from a separate docker-ansible repo.

Four decisions in there that I’d defend:

No local AI in the container. ENABLE_OLLAMA_API and ENABLE_OPENAI_API are both off in the deployment. Model providers get added in the admin UI afterwards, pointing at the vLLM endpoint. The container does not need to know at deploy time what it will eventually talk to.

Authentik OIDC, with local signup disabled. ENABLE_SIGNUP: false, ENABLE_OAUTH_SIGNUP: true, and OAUTH_MERGE_ACCOUNTS_BY_EMAIL so an SSO identity lands on the existing local user with the same address rather than creating a duplicate. The login form stays as break-glass in case OIDC is unavailable. First Authentik user to sign in becomes the admin, so sign in yourself immediately after deploying.

External Postgres instead of SQLite, and pgvector instead of Chroma. The data lives in the Patroni HA cluster, and the same database backs both the application tables and the vector store for knowledge bases and RAG. One backup story instead of two, and a container that holds no state worth keeping.

That one has a real prerequisite attached, and a real discount on my own importance. The openwebui database, its owner role, and CREATE EXTENSION vector are provisioned by a different repo entirely and have to exist first. And the Patroni cluster behind it needs a confession, because I can hear the objection: surely a homelab chatbot does not need a highly-available database. Correct. It does not. Postgres is not a requirement either - the stock container with its SQLite file would be the honest engineering answer for this job, and a chatbot with web search needs roughly as much database as a shopping list. I went overkill anyway, on both counts, for the reason that explains most bad decisions in homelabs: I was building Patroni for other things anyway, and once you’ve built a thing, everything is the thing you’ve built. I like to hurt myself this way. As Hannibal used to say: overkill is underrated. Once the machinery exists, the marginal cost of putting the chatbot in it is one CREATE DATABASE, and that excuse - the cheapest possible justification wearing the clothes of an architecture decision - is how about half of this estate got built. And switching to Postgres starts a fresh schema - existing SQLite data is not migrated for you. Do that on day one, or plan a migration.

SearXNG, and the rate limit that came from the other direction. Web search reached internally by service name, self-hosted, no API key, no per-query invoice. Which is when I found out that one SearXNG instance means one egress IP, and the search engines rate-limit egress IPs - so I’d escaped Anthropic’s limiter only to be rate limited by the engines themselves. The fix was the estate’s standard trick: stop having one of a thing. It’s now a small HA cluster, which is as much about spreading query volume across more egress addresses as it is about uptime - a category of HA that only shows up once you’ve made yourself the customer of someone else’s free tier. Not going to dwell on it, but it got me thinking that “highly available” quietly means different things when the bottleneck is somebody else’s goodwill. And the UI of a member is still published on a host port for local testing, which is a thing to firewall rather than a thing to be relaxed about.

Open WebUI does not write log files

It only logs to the container console. That is upstream issue #16700, and it is still open.

The workaround in the Compose file is a command override that keeps the app as PID 1, so Docker’s SIGTERM still reaches it on stop, while teeing its combined output to a bind-mounted host directory:

command:
  - /bin/bash
  - -c
  - 'exec bash start.sh > >(tee -a /var/log/openwebui/openwebui.log) 2>&1'

tee passes the stream through, so docker logs and the image’s own healthcheck are unaffected, and a logrotate policy handles the host files. The exec matters: without it you get bash sitting in front of the app as PID 1, eating the stop signal, and a ten-second wait before Docker kills the container instead of it shutting down cleanly.

The routing is split across two playbooks, deliberately

Open WebUI publishes a port on the aifrontend host. It runs no Traefik of its own. The frontend hosts’ Traefik terminates TLS for ai.nevergoingtogiveyou.up and proxies back to that port using the file provider.

Which means a route change needs both deploy_aifrontend.yml (the app) and deploy_frontend.yml (the Traefik route). That’s a footgun and I know it. It is the price of not putting a second reverse proxy on a host that already sits behind one, and I think it’s the right trade, but it goes in the runbook rather than being something you’re expected to remember.

The other half of the RAG story - the CPU-only embedding node, the chat-template trap, and the most instructive 200 OK of my career - outgrew this post and now lives in part three, alongside what the knowledge base is actually for.

And the newest one: pictures

Sitting next to docker-sr-1 down under is a separate box with an RTX 3080 Ti, running ComfyUI and SD 3.5.

I’m including it mostly because of what it does to the table at the top of this post. It’s a fourth memory architecture: 12 GB of discrete VRAM, same as the B580, but a completely different runtime with completely different constraints, generating images rather than tokens. Diffusion has an entirely separate relationship with memory than autoregressive decoding does - no KV cache, no growing context, but a working set that spikes hard on resolution and batch size.

It is also plugged into Open WebUI, and I want to be honest about how that reads. It works: the button appears, the prompt goes in, a picture comes out. But image generation is not a focus of Open WebUI - it’s a bolt-on that technically functions - and as of v0.11.3 that is the correct expectation to hold. Anyone who actually wants to make things with images will end up in the node graph, learning what ComfyUI gives you directly: control over the pipeline, the samplers, where the money in VRAM actually goes. Or they’ll pick a front-end built for the purpose, because Open WebUI is not it, whatever its existence next to a working ComfyUI backend suggests. It’s a preview of the capability, not the tool. I keep it wired up because “retrieve a meme from the corpus of my own prompts” is a fine use, and I would not trust it with one that mattered.

Honesty note: this one is not automated yet. Everything else in this post is deployed by Ansible from a repo. ComfyUI is currently a box I set up by hand, and it will stay that way until it’s earned a role. I’d rather say that than imply a tidiness that doesn’t exist.

Where things stand

Working, and being used daily:

• vLLM deployed by Ansible on GB10 hardware at Craft Studio, API-key protected, models swappable through group_vars
• Qwen3.8-Flash-Next-FP8 - the 180B MoE - running tensor-parallel across both Sparks at full 256k context from the custom-built image. That is what answers when I ask it something. Qwen3.6-35B and Nemotron-3-Super were the stages that got the stack working; the NVFP4 recipe is kept in host_vars as the reference it earned, but it isn’t serving anything
• llama.cpp on the Arc B580 down under, serving a 26B MoE off a 12 GB card by pushing experts to host RAM
• Prometheus, Grafana and DCGM, with a committed dashboard and colours that mean something
• Open WebUI at YoloColo with real SSO, an HA database, pgvector RAG and web search
• A CPU-only embedding node behind a shim that makes Open WebUI’s stock API speak the model’s actual chat-template contract
• ComfyUI and SD 3.5 on a 3080 Ti down under, by hand, not yet automated

Honest about what isn’t done:

• This is not stable, and “it runs daily” was doing a lot of work above. There is a single instance of the model, on a two-node cluster with no redundancy by design - that’s what tensor-parallel means here. Mid-use, it crashed. Restarting the services and reforming the cluster took 19 minutes, start to finish, and I timed it because I’d been wondering. That is the availability story: not “always up”, but “comes back inside twenty minutes if I’m watching”. For one user that’s survivable. Any deployment where someone else depends on it wants multiple model hosts behind something that routes around a dead one, which is a bigger architectural conversation than this estate was ever sized for - part four picks up some opinions about what that should look like.

• Nothing is benchmarked, and it mostly can’t be yet. The cluster is configured for 256 concurrent sequences and serves about one person. Latency percentiles over a handful of single-user requests are an anecdote, not a distribution. The panels are correct and currently uninformative, which is a different problem from being wrong.
• The trans-Pacific link is unmeasured. Serving tokens from the other side of the Pacific has an obvious latency floor, and I have not characterised what that does to time-to-first-token or to streaming as experienced through Open WebUI. I have a strong suspicion and no numbers, which is the same thing as not knowing.
• The two runtimes are not comparable and I’m not going to pretend otherwise. Different hardware classes, different runtimes, different quantisation, different continents. Any “GB10 vs B580” number would be theatre.
• Nothing on the metrics path authenticates. node_exporter, dcgm-exporter, vLLM and llama.cpp /metrics are open on their ports, and Prometheus has no auth. The tunnels and the firewalls are the entire access control story. Acceptable given who can reach those ports; not acceptable anywhere else.
• The serving image is built from an unmerged PR. It is the only place the architecture registration and the memory fix both exist, and it is not a release, will not get fixes, and has to be rebuilt when the branch moves. That’s a deliberate trade with a shelf life, not a resting state.
• NCCL on RoCE is still assumed, not proven. A silent fallback to TCP over the same cable serves correctly and is merely slow, which is exactly why it needs an explicit check against the HCA’s own counters. It hasn’t had one.
• The second QSFP rail is cabled and idle. Both ports are connected peer-to-peer. One is in use.

What’s next

The open question is what to do with the B580. It works, it serves a 26B model off a 12 GB card, and it does it on llama.cpp with a hand-maintained Compose file rather than through the Ansible that manages everything else.

Two directions, and I genuinely don’t know which is right:

• Bring it under the vLLM role on the intel backend that’s already written. One runtime, one role, one config language across both sites. Costs me --n-cpu-moe, which is llama.cpp’s flag and the only reason that model fits on that card.
• Leave it on llama.cpp and give it an Ansible role of its own. Keeps the memory trick, admits the estate is genuinely heterogeneous, and stops pretending one abstraction fits both.

I’m leaning toward the second, because the first would mean giving up the thing that makes the machine useful in order to satisfy a tidiness I invented. But it does mean two runtimes to keep current, which is exactly the drift I said I was avoiding when I kept the intel branch alive in the first place. Consistency is not free, and neither is the exception.

And a closing note on the arrangement at the top: the Sparks are a loan, so at some point they go back. That has been a useful constraint. It’s kept everything in Ansible and nothing in my head, because “reproduce this on hardware I no longer have” is the test every part of this had to pass by default.

And then they go home

Here is the part that matters most, and it is the least satisfying thing in this entire post.

The Sparks are a loan. At some point they go back to Craft Studio and stay there. And when they do, I will not be able to run Qwen3.8-Flash-Next. Not “it will be slower”. Not “I’ll drop to a smaller quant and cope”. I will not have the hardware, at all. That checkpoint is 185.5 GB, it needs two 128 GB nodes and a direct 200 Gb/s cable between them, and I own precisely none of those things.

Read the whole of stage two back with that in mind. The clustering, the RDMA fabric, the two-netdev trap, the image I compiled from an unmerged PR over several hours, the memory budget I derived from watching an earlyoom kill happen. All of it is work I did on hardware I have to give back.

What I actually get to keep

If this year goes well, I might get a B70. And with it, a permanent home for Qwen3.8-27B: a real model, running properly, on hardware I own, that isn’t going anywhere and doesn’t need anyone’s permission.

I’d take that happily. It is a genuinely good outcome and it is more than most people running this stuff at home have got.

It is also, in round numbers, a fraction of what I have been running, on a fraction of the hardware, still serving the same one person.

Why not just buy the capability

Because look at the numbers.

Part one puts a pair of these small machines at $10,000 to $20,000 roobucks depending which you buy. Two DGX Sparks sits at the top of that range, near $20,000. That is the hardware this post is about, and it is not money I have. The loan is not a shortcut past a purchase I was going to make anyway. It is access to a tier I was never going to reach.

The other route people suggest when you say you want to run large models locally is workstation cards. An RTX PRO 6000 Blackwell Workstation Edition carries 96 GB of GDDR7, which is a serious amount of memory on a single card, and two of them would put me in genuinely capable territory without any of the RDMA and clustering work above. Two of those is somewhere around $50,000, on the same at-time-of-writing Australian retail basis as part one.

So the ladder goes: I cannot afford the $20,000 option, and the alternative to the $20,000 option is the $50,000 option. Buying my way out of this is not a plan, it’s a fantasy with a spreadsheet attached.

Which is worth sitting with when anyone frames self-hosting as the answer to depending on somebody else’s API. It is an answer. It is an answer that starts in five figures, before power, before cooling, before the fact that the thing you build will serve one person and sit idle most of the day. The economics that make a rate limit annoying are the same economics that make the alternative unaffordable, and I don’t think you get to be glib about either one.

This is the frog-and-pot problem from the top of this post, seen from the other side. The reason you stay in the water is not that you’re inattentive. It’s that getting out costs five figures and the water is, right now, still fine.

The maths that does not currently work

Worth being blunt, because “just self-host it” gets said a great deal by people who have not done the arithmetic.

Nothing in this series was bought for it. The B580, the desktop around it, the 3080 Ti box: all of it existed already, doing other jobs. The Sparks are a loan. My actual capital outlay across everything in these posts is zero, and if I had needed to spend anything, most of it would not have happened.

Because if you do have to buy, the sums are unkind. Call a 5090 several thousand dollars. Call a subscription that does most of what an end user wants around $200 a month, and mine is $35. Even at the higher figure you are a couple of years from the hardware winning. At my figure the hardware never wins at all: it will be obsolete long before it has paid for itself, and I will have spent the money up front for the privilege.

So on today’s prices, buying hardware to escape a subscription is a straightforwardly bad financial decision. I want that said plainly, in a series that could otherwise be mistaken for an advertisement for doing exactly that.

Except that the whole calculation rests on the subscription price staying where it is. I do not believe it will, and I don’t think it can.

This post is, among other things, an accounting of what it physically takes to serve one model to one person. Two GB10s, entirely committed, 185.5 GB of weights, a custom-compiled runtime and a 200 Gb/s cable, so that one bloke down under can ask it things. Whatever I am paying a month, it is not that. Somebody is absorbing the difference, and the earlier argument about peak pricing and who pays with what is really an argument about how that gap eventually closes.

Which flips the comparison in an uncomfortable way. CAPEX is a number you can see, that you pay once, for something you keep and control. OPEX is a smaller number you can afford, on terms that are not yours, which has only ever moved in one direction for every other subscription any of us has held.

I still can’t afford the CAPEX. That’s the trap: knowing which way it’s going does not put $5,000 in my pocket.

The plan I can actually execute

So here’s the realistic version, which is a lot less impressive and considerably more likely to happen.

The B70 is not a someday-maybe. It is something I will be able to afford at some point, and it will probably go into a used EPYC board.

That’s the standard homelab answer, and the reason isn’t the CPU. It’s PCIe lanes. Consumer platforms run out of them almost immediately: you get one card at full width and after that you are making compromises about bandwidth, or about how many devices you can have at all. I have a 4.0 card running at PCIe 1.0 x1 right now, for reasons I still haven’t tracked down, so I am not speculating about this. A second-hand EPYC board has lanes to spare. Which means whatever I buy this year has somewhere to grow, and I can add a card later instead of replacing an entire platform to do it.

That matters enormously when your budget arrives in instalments rather than in a lump.

Realistically it will be EPYC Rome, which is now about four generations back, and that brings its own compromise: it’s DDR4.

That matters more here than it would for most workloads, and it’s worth being precise about why. Think back to the --n-cpu-moe trick, where MoE expert layers get pushed out to system RAM so a 26B model fits on a 12 GB card. That trick is covered at the end of part one; the short version is that every one of those expert layers is computed on the CPU, reading its weights straight out of system RAM. It is a memory bandwidth problem, not a capacity problem.

Cheap second-hand EPYC gives me enormous capacity. Lots of DIMM slots, lots of lanes, plenty of room to hold a large model in system memory for very little money. What it does not give me is bandwidth. DDR4 on a four-generation-old platform is a great deal slower than what a modern DDR5 machine has, and unimaginably slower than the unified memory on a GB10, which is the entire reason a Spark can hold 128 GB right next to the GPU and still be useful.

So the trick that makes a small card punch above its weight is precisely the trick that DDR4 punishes.

The other trade is running cost, and it’s real. A DGX Spark is a genuinely efficient way to have that much memory sitting next to a GPU. A second-hand EPYC platform with cards hanging off it is not: it draws considerably more, at idle and under load, and it does so continuously, because the whole point is that it stays up. That is somebody else’s decommissioned datacentre hardware, designed in an era when the electricity bill was not the buyer’s problem.

So the honest summary of my upgrade path: worse performance per watt, higher running cost than the thing I’m handing back, four-generation-old silicon, memory that is plentiful and slow, arriving in pieces as I can pay for them.

It’s what I can do. And what I can do beats what I’d need $50,000 for, on the simple grounds that one of them is going to exist.

There is something clarifying about that. Every argument in this post about capacity allocation, about one big model versus eighty-five small ones, about who gets to decide what the hardware serves - I have been making those arguments from inside a temporary situation. When the loan ends, I stop being someone with a compute allocation problem and go back to being someone with a compute ceiling. Those are different problems and only one of them is interesting.

And it doesn’t solve the testing problem

The thing a B70 and a 27B model will not give me is the ability to test any of this properly.

I still won’t have parallel load. I still won’t be generating the concurrency that would make a latency percentile mean something. I will have a permanent, reliable, genuinely useful endpoint that I cannot meaningfully benchmark, for exactly the same reason I couldn’t benchmark the Sparks: there is one of me.

So the honest state of things is that I have learned an enormous amount about serving models at a scale I will not have access to, and I still cannot answer the simplest question anyone would ask about any of it, which is how well does it actually perform under real use. I have never had real use. I have had me, and a very good dashboard.

So what was it all for

The capability doesn’t survive the loan. That’s just true.

What does survive is everything that isn’t the hardware. The Ansible survives, and it is vendor-switchable and model-switchable by design specifically so that it would. The understanding survives: I know what unified memory does to gpu-memory-utilization now, I know why a livelock leaves no logs, I know that a format is not a format and a 200 OK is not a correct answer. The monitoring survives and will be waiting when there is something worth watching.

And the habit survives, which might be the real answer. I started this because something I relied on moved and I had no plan for it. I end it with the same dependency, an honest account of it, and a considerably better idea of what building the alternative would actually cost.

Because I did try. That’s what the opencode detour was, right back at the beginning: an attempt to move the hard job onto something I controlled. It didn’t take. The local stack is genuinely good at being a chat window with search attached, and it is not what I lean on when the writing has to be right. I have spent twenty-odd thousand words and two borrowed supercomputers establishing that I still need the thing I was trying to need less.

That is not nothing. It is also not a 180B model, and I want to be straight with you about the difference.

The part I’m least sure about

There was a section here, and it outgrew the post. The Zitron detour, the dyslexia disclosure, the rate-limits-as-somebody-else’s-gpu-memory-utilization argument, and everything I actually think about what all this has done to how I work - none of it is build log, so none of it belongs in one. It lives, considerably expanded, in Part Four: I’m Not Sure Any Of This Was A Good Thing. The build ends here; the arguing is down the road.

Thank you, Craft Computing

None of the above happens without Jeff at Craft Computing putting two Dell DGX Spark loaners in my hands and asking for findings rather than flattery.

I want to be specific about why that matters, rather than just nodding at it. This blog is a homelab blog. The hardware I write about is hardware I paid for, which means it is hardware I could afford, which puts a hard ceiling on what I get to learn. Everything genuinely interesting in this post sits above that ceiling. The unified memory behaviour, the day-0 container ecosystem still catching up to new silicon, DCGM having no framebuffer to report on, the whole class of problem where the GPU’s memory and the host’s memory are the same 128 GB - none of that is reachable from a consumer GPU and a hopeful attitude. You cannot read your way into those. You have to have the machine.

So if you got something out of this post, the credit for it being possible at all goes to Jeff:

• YouTube: youtube.com/CraftComputing
• Patreon: patreon.com/c/CraftComputing

I’ve already told you once in the middle of this post to go and watch the channel, so I won’t labour it. The Patreon is the other half: it’s what funds more of this sort of thing, and more people getting to poke at hardware they’d never otherwise touch is straightforwardly good for everyone who reads blogs like this one.

Cheers, Jeff. I’ll give them back in roughly the condition they arrived in, and I am genuinely sorry about all the messages.

---

Disclaimer, as ever: this post was written with the help of Claude Opus, gemma-4-26B-A4B-it and Qwen3.8-Flash-Next - the last of which is the model this entire post is about building, which pleases me more than it probably should - and as ever I’ve read and approved every word. Assuming a format was a format, setting gpu-memory-utilization to 0.90 and then, undeterred, to 0.85, and every single message Jeff received consisting entirely of “sorry mate, could you turn it back on?” are regrettably, verifiably, entirely my own. The cold spare from two posts ago is still untested. There are now two additional machines I cannot physically reach, and one of them is not mine.