Desperately Seeking LLMs: What Actually Works on an 8 GB Laptop GPU

There is an old film called Desperately Seeking Susan. After the last few weeks, I feel as though I have been making the less glamorous sequel: Desperately Seeking LLMs.

Not the best model on a leaderboard. Not the model with the largest context number on its card. Not even the model with the most impressive five-token benchmark. I wanted models that actually work on my hardware: a Ryzen 7 5800H laptop with 32 GB of RAM and an NVIDIA RTX 3070 Laptop GPU with 8 GB of VRAM, running Ollama under Linux.

That distinction became the whole story.

We pulled, imported, repaired, tuned, benchmarked and sometimes deleted a long list of models. Some were genuinely excellent models that were simply terrible matches for this machine. Some looked fast until we gave them a real prompt. Some offered 128k context in theory and collapsed before getting there. Some speculative decoders made their models slower. Q5 repeatedly cost speed without giving us evidence of better answers. One setting made Ollama report 100% GPU usage while performance fell to 0.13 tokens per second.

The failures were more useful than the victories. They taught us what this laptop is, what it is not, and how little a model name or advertised context window tells you about the experience of using it locally.

The machine sets the rules

This is laptop-class hardware, not a hidden workstation:

• AMD Ryzen 7 5800H, 8 cores and 16 threads
• 32 GB system RAM
• RTX 3070 Laptop GPU with 8 GB VRAM
• one NVMe drive
• Mageia Linux, headless
• a locally built Ollama runtime with its bundled CUDA libraries

Eight gigabytes of VRAM is enough to do interesting work, but not enough to pretend model architecture does not matter. A model that fits mostly on the GPU behaves very differently from one whose weights must stream from system memory on every token. The CPU's memory bandwidth then becomes the ceiling.

That is why sparse mixture-of-experts models became the stars of this experiment. A 30-35 billion parameter MoE may activate only about three billion parameters per token. Ollama can keep attention and other useful tensors on the GPU while the CPU handles experts. A similarly sized dense model must touch essentially all of its weights for every generated token. On this laptop, that difference is the boundary between roughly 30-70 tokens per second and 1-4.

This is not a universal ranking of model quality. It is a ranking of architectural fit.

Failure number one: “100% GPU” at 0.13 tokens per second

Our first major discovery was that a reassuring metric can hide a disaster.

Ollama had been started with CUDA unified memory enabled. A large model appeared to be entirely on the GPU, but the GPU only has 8 GB. The excess was being paged across PCIe while inference ran. The dashboard looked wonderful. The model did not.

A dense 27B model generated at 0.13 tokens per second. Removing GGML_CUDA_ENABLE_UNIFIED_MEMORY=1 brought it to 3.05 tokens per second-a 23-fold improvement from deleting one supposedly helpful option.

Three tokens per second was still not good, but it was honest. The model was now partitioned instead of pretending that system RAM was VRAM.

That episode established the first rule: reported GPU residency is not the same thing as useful GPU residency. Paging weights over PCIe can be catastrophically worse than explicitly leaving them on the CPU.

It also exposed a cluster of tuning folklore. Several environment variables that sounded authoritative did nothing in our Ollama build. For dense models, the effective control was the per-model num_gpu parameter. For MoE models, explicitly forcing num_gpu was usually the wrong move: it replaced Ollama's fine-grained tensor placement with a coarse layer split and could produce an out-of-memory failure even at a lower layer count.

The right instruction depended on the architecture:

• Dense model: find the highest layer count that survives a real prompt.
• MoE model: let Ollama autofit individual tensors; do not force a layer count.

Dense models: good models behind a bandwidth wall

Several of the most educational failures were dense models around 24-31B. They ran correctly. We tuned them correctly. They were still too slow.

Qwen3.8 27B

The dense Qwen3.8 27B Q4 model improved from 3.10 to 3.72 tokens per second after layer tuning. Its MTP variant reached roughly 6.3 tokens per second, which was a real gain but nowhere near the 20-25 tokens per second I wanted for comfortable interaction.

The arithmetic explained why. Roughly 12 GB of weights remained CPU-resident, and the laptop could stream them at around 38 GB/s. That implies about 0.32 seconds per token, or approximately 3.1 tokens per second. We had not missed a magic flag. We had reached the memory-bandwidth wall.

The family was deleted, reclaiming 34.93 GiB.

Granite 4.2 30B

Granite 4.2 was a dense 29.3B model. Its stock 128k configuration allocated around 17.4 GB merely for the KV cache and offloaded only 10 of 65 layers. On a short prompt it could show 3.06 tokens per second, but short prompts concealed the real cost.

After tuning, the 8k version reached 4.30 tokens per second on a tiny test. At 7,754 prompt tokens it fell to 2.45 tokens per second, with 112 seconds before generation. The 16k tier managed 1.34 tokens per second at 14,680 tokens and took 228 seconds to get through the prompt.

We tested the layer ceilings; one more layer caused an OOM. We tested thread counts; six threads helped. There was nowhere meaningful left to go. Granite was deleted, and its unique 17.7 GB weight layer went with it.

Devstral Small 2 24B

Devstral sounded promising because it was smaller and designed for coding. It was still a dense 24B model.

The tuned 16k tag produced 2.42 tokens per second at 14.2k tokens, with 49 seconds of time to first token. At 25.4k, generation fell to 1.35 tokens per second and prompt processing took 293 seconds. We deliberately did not create a misleading 64k or 384k tag just because the metadata allowed it.

Devstral was deleted, reclaiming its unique 15.18 GB layer.

Gemma4 31B

Gemma4 31B provided the same lesson with a useful twist. Q4 and Q3 plus MTP could produce about 5.2-5.8 tokens per second on a short prompt. At 13.3k, Q4 dropped to 2.17 tokens per second with 251 seconds of prompt processing; Q3 managed 2.43 with 193 seconds.

It remains interesting as a quality experiment, but not as a fast daily driver. We also learned that 16 SMT threads were 28% slower than eight: 2.38 versus 3.32 tokens per second in one comparison. More CPU threads are not automatically more throughput when memory contention is the bottleneck.

These were not defective models. They were the wrong physical shape for an 8 GB mobile GPU.

The older Qwen coder that should have been faster-but was not

The most recent disappointment was Unsloth's Qwen3-Coder 30B-A3B Instruct. It was an MoE with only about 3B active parameters, slightly smaller and older than the Qwen3.6 family. The expectation was reasonable: perhaps it would be the faster coding specialist.

We pulled both Q4 and Q5 and built the usual fast, 64k and 128k tiers. The measurements disproved the hypothesis.

Its architecture used full attention, so the KV cache grew from 1.632 GB at 32k to 6.528 GB at 128k. As context grew, weights were forced out of VRAM: Q4 went from 49/49 offloaded layers to 35/49; Q5 ended at 33/49. At 111k, both were effectively unusable.

The newer Qwen3.6, KAT and CoderX models were dramatically faster on the same hardware. “Older” and “smaller” did not mean leaner in the dimension that mattered.

All six tuned tags and both source tags were removed. We recovered 36.69 GiB. This was perhaps the purest failure of the project: a sensible idea, a clean test, an unambiguous negative result and a complete deletion.

Context windows are budgets, not badges

The number printed on a model card tells you what its positional encoding may support. It does not tell you what your GPU can support at a useful speed.

Context consumes KV cache. Compute buffers consume VRAM. Speculative decoding needs its own context. Every megabyte allocated to one of those is a megabyte that cannot hold model tensors. On a small GPU, a context increase can move just enough weights to the CPU to cause a performance cliff.

Poolside's Laguna XS 2.1 gave us the most spectacular example. At a nominal 262k context it generated 32.22 tokens per second on a short prompt, then only 2.14 tokens per second on a real 108k prompt-a 15-fold collapse. The short test had never exercised the memory layout that users would encounter.

Context caps therefore became performance parameters. For Laguna and Ornith, reducing the maximum context freed KV memory that Ollama could use for more expert tensors on the GPU. We built honest tiers around measured floors: approximately 20k and 32k for a 35-token-per-second floor, and 32k and 48k for a 30-token-per-second floor, depending on the model.

There is an unpleasant catch. When a prompt exceeds a configured cap, Ollama can silently truncate it instead of returning an error. A 47,750-token prompt sent to a 32k-capped model returned HTTP 200 while evaluating only about 16k prompt tokens. A “fast” context cap is therefore not free; it trades away information, and the interface may not warn you.

This is why the regular model families now have explicit fast, 64k and 128k variants. The name communicates the budget instead of pretending one configuration is ideal for everything.

The short-prompt trap

Many bad configurations looked good for 19 tokens.

Ollama decides tensor placement when loading a model, but some buffers scale with the actual prompt. We repeatedly found a num_gpu value that loaded, benchmarked beautifully on a tiny request and then crashed on a realistic one. Qwen3.8 had to move from 28 to 27 layers; its MTP variant from 27 to 26. A Gemma4 12B 128k configuration at 42 GPU layers reported 23.15 tokens per second and then OOMed on every real long prompt.

The same trap applies to time to first token. A model may decode at 40 tokens per second after taking two minutes to ingest the prompt. For coding and agent work, that waiting time is often more important than another two tokens per second during generation.

We therefore measured three things separately:

• cold prompt processing speed
• time to first token
• generation speed at realistic depth

We also had to defeat Ollama's prefix cache. Repeating the same prompt could report more than 130,000 prompt tokens per second, a technically correct cache hit and a completely useless prefill benchmark. Cold measurements required varied prompts and model reloads.

Batch size: one knob, opposite answers

num_batch became one of the most valuable-and model-specific-controls.

For KAT Coder Q4, increasing it from 512 to 4096 raised prefill from 318 to 1,066 tokens per second and reduced time to first token from 44.6 to 13.3 seconds, while decode moved only from 43.6 to 42.9 tokens per second.

For Ornith around 48k, moving from batch 512 to 2048 increased prefill from 262 to 656 tokens per second and cut time to first token from 182 to 72.8 seconds, with essentially unchanged generation.

Laguna could not afford the same setting. Its wider compute buffer displaced too many expert tensors and broke the throughput floor. At its 128k allocation, the correct direction was smaller, not larger.

There is no universal “best batch size.” It is a negotiation between prefill throughput, compute-buffer memory and which tensors remain on the GPU at that specific context cap.

Speculative decoding: sometimes magic, sometimes a tax

Multi-token prediction and draft models can transform a model, but only when the accepted future tokens cost less than they save.

The Gemma4 12B dense model was the happy case. Its MTP acceptance was about 61%, and throughput improved by 58%. The Gemma4 26B-A4B MoE with QAT and MTP became one of the early successes, running around 48 tokens per second in its standard tier.

Ornith was the opposite. Acceptance around 32% made it approximately 20% slower. Nemotron shipped with MTP depth two, but depth two was slower than turning MTP off for the relevant workload; it could also OOM at batch 4096 because the draft context consumed the remaining VRAM.

Depth mattered too. CoderX improved at draft depth one, declined at depth two and became worse at its shipped depth three. KAT's MTP variant won on long generations but could lose on tiny ones because speculative setup has a fixed cost.

Even acceptance rate was not enough to choose a quant. A Q8 drafter raised acceptance from about 61.2% to 66.5% but did not improve throughput and consumed another 200 MB. Better prediction is irrelevant if producing the prediction costs too much.

The rule became simple: explicitly enable the head, sweep the depth on the target hardware, and benchmark the intended prompt/output shape. Never assume that an MTP label means acceleration.

Q5: higher precision, lower speed, unproven benefit

I kept mistyping Q5 as “A5,” but the models themselves were consistent: the higher quant was slower.

KAT Coder Q5 was 11-14% behind Q4 in our throughput tests. Unsloth Qwen3.6 Q5 ran at 33.67 / 30.18 / 23.26 tokens per second across the fast, 64k and 128k workloads; Q4 achieved 37.06 / 33.87 / 24.89. The old Qwen3-Coder Q5 was slower at every tier and lost more layers at high context. LFM2.5 Q6 was also slower than its Q4 sibling.

Higher precision is real, but a quality advantage was not measured. We did not run a sufficiently rigorous coding evaluation to claim one. That matters: “more bits” is not the same as “better enough to justify the cost.” On this machine Q4 is the default until an actual task demonstrates otherwise.

Ollama and the workstation needed tuning too

Model parameters were only half of the work.

The MoE models execute many experts on the CPU, and the laptop was using a balanced platform profile with a powersave governor. Switching the platform profile, CPU governor and energy preference to performance improved throughput by 9.5%. That was larger than most parameter tweaks, so it was made persistent as a small system service.

The price is fan noise and power consumption. It is also a diagnostic: if every MoE suddenly becomes about 10% slower, the power profile is the first thing to check.

Thermals complicated the measurements. An unchanged Ornith configuration fell from 31.13 to 29.92 tokens per second over 40 minutes. Across the session, thermal drift reached roughly 9%-larger than many alleged optimization wins. The GPU was sitting at 82°C and the driver reported hours of accumulated thermal slowdown.

That forced a better method. We stopped trusting three-run medians for throughput guarantees, heat-soaked the machine, used five repetitions and judged a floor by the worst run. One candidate appeared to meet 30 tokens per second during the sweep, but the finished tag later recorded a worst run of 29.5. We rejected it and added more margin.

Other system-level experiments failed cleanly:

• Q4 KV cache was slower than Q8 for Ornith, 28.73 versus 31.13 tokens per second, because dequantization cost more than the freed memory saved.
• Two-way parallelism improved wall time for two tiny chats by 1.36×, but on long workloads aggregate throughput fell and one request ran at 6 tokens per second while the other got 44. Concurrency returned to one.
• Forcing the number of CPU MoE experts could gain about 3%, but the control was server-wide and another model OOMed under a nearby value. A fragile global win was not worth destabilizing every model.
• The CUDA development toolkit was unnecessary. Ollama uses the NVIDIA driver and its own bundled runtime; several gigabytes of compiler tooling would not make inference faster.

The fastest reliable system was not the one with the most aggressive settings. It was the one with fewer dishonest abstractions.

The models that earned their disk space

After all those failures, several families proved that local inference on this laptop can be genuinely good.

LFM2.5 8B-A1B

Liquid AI's small MoE was the speed champion. Q4 reached about 200 tokens per second on a short prompt, 149.66 at 26.7k and 79.85 at roughly 106.6k. Cold prefill reached thousands of tokens per second. Its Q6 sibling remained fast, but slower, with no measured quality gain.

Gemma4 QAT and MTP

The 12B QAT plus MTP model ran around 69-75 tokens per second in its fast form and remained practical at long context. A 128k tier deliberately disables its drafter because the apparent short-prompt advantage disappears at depth. The 26B-A4B MoE variant delivered around 48 tokens per second and offered a compelling general-purpose balance.

CoderX 27B-A3B

CoderX, an expert-pruned Qwen3.6 coding model with MTP depth one, achieved 46.03 tokens per second at 27.1k, 41.50 at 53.8k and 35.06 at 108.2k. The 128k tier is extraordinarily tight-only 99 MiB free after loading-and batch 4096 OOMs its draft context, but batch 2048 works.

We have measured speed, not proven that its pruning makes it a better coder. It remains an experiment with excellent hardware fit.

KAT Coder 2.5

KAT Q4 produced 40.90 tokens per second at 27.1k and 28.76 at 108.2k. Its MTP variant improved the long-generation result to 43.63 and 34.19, using depth one rather than assuming more speculation was better. Q5 stayed as a higher-precision option, but Q4 was the measured speed pick.

Unsloth Qwen3.6 35B-A3B

Unsloth's Qwen3.6 35B-A3B was the model we most wanted to make work. The imported GGUF had a malformed generic Ollama template path, so its renderer and parser metadata had to be repaired before tools, thinking and vision behaved correctly.

Once fixed, Q4 delivered 37.06 tokens per second at 27.1k, 33.87 at 52.7k and 24.89 at 108.2k. All three tiers used batch 4096 successfully. Q5 was slower but functional.

That is the difference between Qwen3.6 and the older Qwen3-Coder experiment: nearly 25 tokens per second versus less than 2 around 108-111k context. Similar headline parameter counts do not imply similar memory behavior.

Laguna, Ornith, Nemotron and North Mini

Laguna XS 2.1 and Ornith 1.5 became the models that taught us how to offer explicit throughput floors. Both can run beyond 128k, but useful tiers must balance speed against silent truncation and time to first token.

Nemotron 3.5 Lightning has an architectural party trick: only six of 54 layers require context-growing KV, so enormous contexts are relatively cheap. Its throughput-roughly 22-32 tokens per second-was not as strong as Laguna, but its long-context behavior was instructive. North Mini Code also remained viable, with around 37 tokens per second at baseline and an interleaved sliding-window architecture that made a tuned 128k tier possible.

None is “the winner” in every category. That is precisely why the model store contains families instead of one universal tag.

The smaller failures around the edges

Not every dead end was a throughput curve. Several were integration, packaging or storage failures, and they changed how we work just as much.

Liquid AI's 191 MB DSpark speculative sidecar looked like a nearly free upgrade for LFM2.5. Ollama 0.32.15 identified it as DFlash instead, and the bundled runner asserted on its first decode even when paired with the official target. Forcing the type did not make it compatible. The failed tag and 5.1 GB of staging files were removed.

With North Mini Code, Unsloth's Q3 quant reduced the model from 17.32 to 13.36 GiB. We expected fewer expert bytes to mean faster generation. It did not: prefill improved by 17%, but decode stayed the same or became slightly slower because Ollama spent none of the freed VRAM on useful extra tensors. Keeping 14 GB and accepting untested three-bit quality for a prefill-only gain made no sense, so Q3 was deleted the same day.

Our own Modelfile technique caused another expensive mistake. Using FROM <blob path> looked as if it would reuse an existing weight layer; Ollama imported a second copy instead. Across Qwen, Gemma and Laguna experiments, the duplication reached tens of gigabytes. Rebuilding from model tags and verifying shared digests reclaimed 37,087 MiB without changing throughput. The lesson was painfully literal: never infer deduplication from a Modelfile that happens to point at the same bytes.

Third-party GGUF metadata was inconsistent too. The Unsloth Qwen3.6 import needed native Qwen renderer/parser metadata before tools, thinking and vision were represented correctly. Nine KAT tags later needed a similar repair because live tool calls worked while Ollama's model API advertised only plain completion. A model can be computationally healthy and still be broken at the interface boundary.

We even discovered that the shell was using a stale Ollama 0.6.6 client against the current server because /usr/local/bin won the PATH lookup. Removing that old binary and pointing the command at the self-contained current installation eliminated a quiet client/server mismatch. Meanwhile, a completely different “MiniMax H3” experiment turned out to be a video-diffusion model rather than an Ollama LLM; it could generate five 1920×1088 frames, but a 243-frame clip demanded one 31.88 GB Vulkan allocation from an 8 GB GPU. Some searches end before the first token.

Even prompts can be over-tuned

We also tested whether a clever universal system prompt would improve reliability across the fleet. It did not.

A no-system-prompt baseline passed 16 of 21 small compliance tests. A compact generic persona also passed 16. A more elaborate XML-style prompt fell to 14. Mandatory headings broke otherwise runnable Nemotron code. Strict phase instructions interfered with KAT's JSON. Other models wrapped exact JSON in Markdown or exposed thinking when the interface expected clean output.

Request-level JSON schemas helped more than a grand personality imposed on every model. Publisher templates and samplers generally deserved to remain intact. The prompt audit reinforced the larger lesson: optimize the layer where the evidence says the problem exists.

What “failure” means now

The graveyard contains good models, bad configurations and bad assumptions. They should not be confused.

Granite, Devstral and dense Qwen did not fail at intelligence; they failed the interactivity requirement on this memory hierarchy. Qwen3-Coder did not fail to run; its full-attention KV growth made it a poor long-context choice here. Q5 did not fail to preserve information; it failed to demonstrate enough benefit to pay for lower throughput. Ornith's MTP did not fail to predict; it failed to predict cheaply. Unified memory did exactly what it promised and still produced the worst inference experience of the project.

That framing matters because local AI advice is often too broad. “This model is fast” is incomplete. Fast on which GPU, at what quant, with how much context, after how much prompt processing, at what temperature, with how many layers displaced, and using what definition of success?

On this laptop, the emerging answer is clear:

• Prefer MoE models with about 1-4B active parameters.
• Treat dense 24-31B models as quality experiments, not interactive defaults.
• Use Q4 unless a measured task proves that Q5 or Q6 earns its cost.
• Benchmark at the intended context depth, not on a greeting.
• Measure cold prefill and time to first token separately from decode.
• Let Ollama autofit MoE tensors; tune dense layer counts explicitly.
• Sweep MTP depth and batch size per model.
• Account for power policy, thermal decay and prefix caching.
• Name model tiers honestly: fast, 64k and 128k are different operating points.
• Delete experiments that do not justify their disk space, but keep their measurements.

The search is not over. It probably never will be. New models will arrive with better routing, cheaper attention, improved quantization and fresh opportunities to make the same mistakes.

But “desperately seeking” no longer means blindly downloading whatever is praised online. It means testing the claim that matters: can this model, in this quant, at this context, on this laptop, return useful work before I lose patience?

That is a much harder benchmark than a leaderboard-and a much more useful one.