Munich

27 October 2026

Developing with Local AI ModelsHow Far Can You Get?

Source code on a laptop screen with Claude, vLLM and Gemini icons visible inside its transparent base.

Rainer Hahnekamp Soverius AI

1

45-minute session including questions. Forum 1. Uses the canonical light Soverius HTML layout and the existing Graz teaching visuals. No organiser template or feedback slide. Published session title verified on 26 October 2026: Developing with Local AI Models: How Far Can You Get? Official iJS Munich programme: https://javascript-conference.com/muenchen/programm-muenchen/ . Cover layout matches the current iJS testing deck at /Users/rainerh/Documents/Soverius/iJS-Munich-2026/testing/index.html, including the official event logo and Munich identity. Approved cover illustration added 27 October 2026: simple vector laptop cutaway with source code on the screen and the requested Claude, vLLM and Gemini brand icons inside the base. The illustration is conceptual, not a verified runtime configuration. Claude and vLLM marks reuse existing deck assets; Gemini mark is the original inline footerSpark SVG from https://gemini.google/about/ . All icon data is embedded in the local SVG for offline playback.

Soverius AI — We build production-ready AI systems

Why local?

Reasons to Run Models Locally

€

Cost Control

No provider charge per token.

Privacy

Keep code and context inside.

Independence

Reduce provider dependency.

3

Cost Schema: Local vs. Frontier

4

Three presenter-controlled steps through the shared arrows. Entry shows the axes and legend; there is no autoplay. 1. Draw only the first quarter of both curves: Local and a low, flat Frontier subscription. Within the included usage budget, the subscription is cheaper in this illustrative comparison. 2. Extend both curves. Frontier reaches the end of its included allowance, then API Token Costs rise and eventually cross above Local. The shaded area marks the region where the illustrative frontier cost exceeds local cost. 3. Reveal Automatic Loops with a rightward arrow toward greater token usage. Loops can be used from the beginning; they do not start at a particular point on this chart. As soon as they are used, repeated model calls consume more tokens and make exceeding the subscription allowance more likely. The horizontal axis is Tokens, not time. The arrow indicates consumption direction and is not tied to the subscription/API bend. Starting a loop does not itself trigger API billing; provider-plan and tool rules determine how extra usage is charged. Previous reverses one teaching step. Double arrows skip slides; returning starts at step 0. Reduced-motion mode shows the same states without line-drawing motion. The local curve starts higher because infrastructure must be purchased, then rises gently with energy and operating costs. The comparison assumes a fixed accounting period, model/input-output/cache mix and workload within local capacity. It is schematic, with no measured break-even price or token count. Capacity upgrades, model quality, utilisation and engineering effort affect real costs. Plan/usage reference: https://support.claude.com/en/articles/12429409-manage-usage-credits-for-paid-claude-plans (inspected 26 October 2026).

Typical Required Components

AGENT

Claude Code
Google ADK
Mastra
LangGraph

calls

INFERENCE ENGINE

llama.cpp
vLLM
ONNX Runtime

runs

MODEL

Gemma
Qwen
Muse Glimmer

5

Three roles in a typical local agent application: the model predicts tokens, the inference engine loads and runs the weights, and the agent layer manages the application workflow and tools. Gemma, Qwen and Meta’s Muse Glimmer are example models and model families. llama.cpp, vLLM and ONNX Runtime are example inference engines. The ONNX browser example uses ONNX Runtime Web through Transformers.js. Claude Code is a ready-made coding agent; Google ADK, Mastra and LangGraph are frameworks for building agents. These are examples of each role, not a claim that every combination connects without configuration. Simple prompting applications can call the inference engine directly without an agent framework. Logo sources: adk.png: https://raw.githubusercontent.com/google/adk-python/main/assets/agent-development-kit.png vllm.png: https://raw.githubusercontent.com/vllm-project/vllm/main/docs/assets/logos/vllm-logo-only-light.png langgraph.svg: https://raw.githubusercontent.com/langchain-ai/langgraph/main/.github/images/logo-light.svg (standalone symbol) qwen.jpeg: https://cdn-avatars.huggingface.co/v1/production/uploads/6215ca5692c0ecfba9186921/hrRM50-6XcdWgg2AKpENG.jpeg mastra.svg: https://mastra.ai (official header symbol, static first frame) claude: https://commons.wikimedia.org/wiki/File:Claude_AI_symbol.svg llama.cpp: https://github.com/ggml-org/llama.brand gemma: https://github.com/google-deepmind/gemma/blob/main/docs/images/icon.svg onnx-runtime.svg: https://onnxruntime.ai/images/logos/onnxruntime/ORT_icon_for_light_bg.svg (official icon for light backgrounds; https://onnxruntime.ai/docs/reference/citing.html)

CPU and GPU Inference

CPU1× ComputeIntel Core i9-14900K

Hard Disk or SSD≈0.1–7 GB/sBarraCuda HDD
870 EVO / 990 PRO SSDSwap · No Compute

System RAM≈90–100 GB/sDual-Channel DDR5-5600

PossibleOften too slow

GPU≈30–100× ComputeRTX 4070 Super – RTX 5090

VRAM≈500–1,800 GB/sGDDR6X – GDDR7

PracticalUsually 5–10× faster

You can serve a model larger than 1 TB from disk, but do not expect a response anytime soon.

6

Start with the simple answer. A model can fit on a hard disk or SSD, but storage performs no model computation. The storage bandwidth range combines an ordinary hard disk at roughly 0.1 to 0.2 GB/s, a SATA SSD around 0.5 GB/s and a fast consumer NVMe SSD around 7 GB/s. llama.cpp can memory-map the model file and ask the operating system to load the required pages into system memory. If the model is larger than the available RAM, those pages must be loaded and discarded repeatedly. That swapping can make inference extremely slow. CPU plus system RAM can execute the model without a GPU. The 90 to 100 GB/s figure represents a normal dual-channel DDR5 desktop system, and the CPU serves as the 1-times compute baseline. GPU plus VRAM is the preferred path because both components are faster. Current consumer GPUs suitable for local inference span roughly 500 to 1,800 GB/s of memory bandwidth, while their parallel matrix throughput can be tens of times higher than a desktop CPU. Do not multiply those two ratios. Each phase is limited by its slower resource. As a rule of thumb, full GPU inference is often five to ten times faster than CPU-only inference for token generation. Prompt processing can show a larger difference. The precise speed depends on the model, quantization, hardware and runtime. Unified-memory systems combine RAM and VRAM into one pool, but the same principle remains: memory bandwidth and GPU compute determine the useful speed. Reference hardware behind the displayed bandwidths: - CPU and system RAM: Intel Core i9-14900K with dual-channel DDR5-5600, 89.6 GB/s, rounded to about 90 GB/s. - GPU lower endpoint: NVIDIA GeForce RTX 4070 SUPER, 504 GB/s. - GPU upper endpoint: NVIDIA GeForce RTX 5090, 1,792 GB/s. - Hard disk: Seagate BarraCuda ST2000DM008, up to 220 MB/s. - SATA SSD: Samsung 870 EVO, up to 560 MB/s. - NVMe SSD: Samsung 990 PRO, up to 7,450 MB/s. Transition: now look at three concrete hardware classes for local inference. Sources: https://www.seagate.com/files/www-content/product-content/barracuda-fam/barracuda-new/en-us/docs/100817550L.pdf ; https://semiconductor.samsung.com/news-events/news/samsung-introduces-latest-in-its-worlds-best-selling-consumer-sata-ssd-series-the-870-evo/ ; https://download.semiconductor.samsung.com/resources/brochure/990_PRO_Series_Brouchure_Web_Version_1.0.pdf ; https://www.intel.com/content/www/us/en/products/sku/236773/intel-core-i9-processor-14900k-36m-cache-up-to-6-00-ghz/specifications.html ; https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4070-family/ ; https://www.nvidia.com/en-sg/geforce/graphics-cards/50-series/rtx-5090/ ; https://github.com/ggml-org/llama.cpp

Local Inference Hardware

Consumer

RTX PRO 4000

24 GB VRAM≈ €2,500 per GPU

Mac Studio

36-512 GB Unifiedfrom €2,999

Ambitious Consumer

ASUS Ascent GX10

128 GB Unified Memory ≈ €4,850 per system

Enterprise

$NVIDIA RTX PRO 6000 Blackwell graphics card$

RTX PRO 6000 Blackwell

96 GB VRAM per GPU ≈ €17,700 each

7

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. These examples show practical local-inference options rather than a complete buying guide. The consumer column deliberately contains two different approaches. NVIDIA's RTX PRO 4000 Blackwell is a discrete GPU with 24 GB of GDDR7 VRAM and 672 GB/s of memory bandwidth. Twenty-four gigabytes is a useful starting point for quantized coding models, although the usable model size also depends on the context and KV cache. The RTX PRO name belongs to NVIDIA's professional range; consumer here describes the size of the local setup, not NVIDIA's market category. Austrian retail listings put the card at roughly €2,500. This buys the GPU, not a complete computer. A Mac Studio is the integrated alternative. Apple's current configurations use M5 Max or M5 Ultra and range from 36 GB to 512 GB of unified memory. The 512 GB figure is a configurable maximum, not the base configuration. M5 Max configurations span 36 to 128 GB. M5 Ultra configurations span 96 to 512 GB. Apple starts the M5 Max system at €2,999 and the M5 Ultra system at €6,599 in Austria. Apple says the 512 GB option will become available at the end of October 2026, and the final price of that option is not yet shown. Unified memory is shared by CPU and GPU, so it is not directly equivalent to dedicated VRAM, but the larger configurations can keep very large quantized models in memory. The wide range is important: Mac Studio is one product family, not one fixed capacity. The middle class is the ASUS Ascent GX10. Its NVIDIA GB10 Grace Blackwell Superchip uses 128 GB of coherent unified LPDDR5x memory. This gives a single developer far more model capacity than a 24 GB graphics card, although its 273 GB/s memory bandwidth is lower than the bandwidth of high-end discrete GPUs. Current Austrian retail listings start near €4,850 for the 1 TB system. ASUS originally announced a €3,499 recommended price, so availability and street pricing matter. The enterprise class uses RTX PRO 6000 Blackwell GPUs. Each GPU provides 96 GB of GDDR7 ECC memory. Austrian retail listings for the NVIDIA Workstation Edition currently start around €17,700 per card. The three cards in the picture would therefore represent more than €50,000 in GPUs alone. A workstation can contain several cards. Multiple GPUs add capacity and throughput, but one model does not automatically see them as one simple memory pool. The inference runtime must split or replicate the model, and interconnect bandwidth matters. All prices on the slide are rounded snapshots checked on 7 September 2026 and can change. Transition: move from physical machines to the model sizes and deployment classes they can support. Sources: https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-4000/ ; https://geizhals.at/nvidia-rtx-pro-4000-blackwell-900-5g147-2570-000-a3488409.html ; https://www.apple.com/at/newsroom/2026/08/apple-introduces-new-mac-studio-with-m5-max-and-m5-ultra/ ; https://www.apple.com/at/mac-studio/specs/ ; https://www.asus.com/networking-iot-servers/desktop-ai-supercomputer/ultra-small-ai-supercomputers/asus-ascent-gx10/ ; https://www.idealo.at/preisvergleich/OffersOfProduct/208466423_-ascent-gx10-asus.html ; https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000-family/ ; https://geizhals.at/nvidia-rtx-pro-6000-blackwell-workstation-edition-900-5g144-2500-000-a3476784.html Image sources: RTX PRO 4000 and RTX PRO 6000 images supplied by the presenter; https://www.apple.com/v/mac-studio/specs/b/images/specs/hero__d0yd117u6qy6_large_2x.jpg ; https://dlcdnwebimgs.asus.com/gain/c66958f8-57de-4563-b0db-bd317bfaf1f3/w800 Example configurations - not endorsements or product recommendations.

Model Size by Deployment Class

8

Slide 8: published closed frontier estimates, 27 October 2026. All open-model marker heights are calculated from log10(total main-model parameters in billions), with 1B at the bottom and 10T at the top. Main-model embeddings and lookup tables are included; separately reported auxiliary prediction heads are excluded. The Gemini Nano marker is a 2-4B range. Label offsets are relative to their marker, and x positions within a category simply keep similarly sized models distinct. There are no label connector lines. Visible model labels contain only names, parameter counts and essential availability status. Published closed-frontier estimates, checked 27 October 2026 and included at the user's explicit request. Visible labels contain approximate total parameters, the word Estimate and clickable AI BENCHY attribution. The publisher calls these low-confidence catalog estimates carried forward from earlier model families. They are not vendor disclosures or measured checkpoint sizes. The panel is unscaled; row position does not imply parameter count. No new estimate is calculated here. GPT-6.1 Sol: approximately 2T total parameters. AI BENCHY calls this a low-confidence catalog estimate carried forward from GPT-6 Sol. OpenAI has not disclosed the count; the linked announcement establishes identity and availability, not the parameter figure. Published estimate: https://aibenchy.com/model/openai-gpt-6-1-sol-low/ Claude Opus 5.5: approximately 5T total parameters. AI BENCHY calls this a low-confidence catalog estimate carried forward from the prior Opus 5 family. Anthropic has not disclosed the count; the linked announcement does not establish it. Published estimate: https://aibenchy.com/model/anthropic-claude-opus-5-5-low/ Gemini 4 Argon: approximately 1.2T total parameters. AI BENCHY calls this a provisional low-confidence catalog estimate carried forward from the Gemini Pro family. Google has not disclosed the count and describes Argon as a new model rather than a fine-tune of a prior model. The inherited estimate should not be interpreted as proof of a shared architecture. Published estimate: https://aibenchy.com/model/google-gemini-4-argon-low/ The three values above are the publisher's estimates, not our calculations. Only total parameters are shown. We do not adopt AI BENCHY's separate active-parameter or architecture guesses. Epoch continues to list parameters for these models as Unknown: https://epoch.ai/models/gpt-6-1-sol https://epoch.ai/models/claude-opus-5-5 https://epoch.ai/models/gemini-4-argon Grok 4.7: 2.1T, visibly labelled Musk Claim and linked to the original public statement of 28 July 2026. This is a pre-release founder statement, not an independently verified parameter estimate for the released checkpoint. Original rendered post inspected directly in the preceding source audit. Release announcement and developer guide do not specify a numeric count. Primary claim: https://x.com/elonmusk/status/2082123925283041545 Release: https://x.ai/news/grok-4-7 Developer guide: https://docs.x.ai/developers/grok-4-7 The old 5T / 10T / 6T / 3T benchmark-derived figures remain removed: they were our unvalidated extrapolations. The anpaure/frontier-model-sizes snapshot covers older versions and is not the source for the currently displayed estimates. Epoch's Grok 4.7 page currently displays 2.1e9, conflicting with the original founder's 2.1T claim; it is not numeric corroboration. Source provenance and limitations are retained under frontier_evidence in model-deployment-data.json. Selected memory configurations are documented here, rather than on the visible plot: Bolt four-bit capacity example; Gemma/Qwen Q4 and Super VL NVFP4 ~4-bit; Kolibri official FP8 ~8-bit; original Apertus and Soofi BF16 ~16-bit. These are example released/configured formats, not architectural requirements or a claim that other quantizations do not exist. Actual memory also includes quantization scales, higher-precision tensors, runtime, caches, vision components, optional draft models and the operating system. Hardware categories are representative capacity examples at the selected formats, not benchmarked guarantees for every device or full native context. Soofi retains Closed Beta; Kimi K3.1 Pro retains Weights Pending. Gemma 4 E2B: 5.1B includes per-layer embeddings; effective parameter count is 2.3B. Embedded deployment, not a claim of Chrome Prompt API support. Source: https://huggingface.co/google/gemma-4-E2B Gemini Nano: Chrome describes hardware-selected variants such as 2B and 4B; draw a range, not an exact midpoint model. Source: https://developer.chrome.com/docs/ai/understand-built-in-model-management Gemma 4 26B A4B: Replaces Gemma 4 12B on 27 October 2026. Google publishes 25.2B total language-model parameters / 3.8B active for the 26B-A4B version, plus a separate ~550M vision encoder. The main-model count determines the plotted height. Official QAT Q4_0 GGUF is available: 14.4 GB model plus 1.19 GB vision projector, about 15.6 GB combined before context/runtime. Consumer 24-32 GB remains a capacity example, with practical context constrained by remaining memory. Quantization source: https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-gguf/tree/main . Source: https://huggingface.co/google/gemma-4-26B-A4B-it Qwen3.8: 27B language-model parameters. Four-bit weights approximately 13.5 GB. The 24-32 GB category is a capacity example with reduced practical context, not a promise of full 262K context. Source: https://huggingface.co/Qwen/Qwen3.8-27B Nemotron 3.5 Bolt: 74B total, 9B active. Moved to the workstation/server category at the user's request. The previous 2-bit IQ2 example is replaced by a four-bit capacity estimate: approximately 37 GB of raw weights before scales, cache, runtime, vision and optional draft model. This is an illustrative 64-192 GB workstation capacity placement, not a measured runtime. NVIDIA also publishes a highly compressed nominal 24 GB IQ2 variant with a documented roughly 28 GB runtime recipe for 32 GB or larger GPUs; that special configuration is not the deployment example shown here. Source: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF Soofi S: Model architecture section reports 31.6B total, despite 30B family shorthand. Original checkpoint is gated closed beta; official files approximately 63.2 GB. BF16 capacity planning therefore belongs in the workstation/server group, with more than 64 GB once overhead is included. No unverified 4-bit distribution assumed. Source: https://huggingface.co/Soofi-Project/Soofi-S-Base Kolibri 1: 78,103,074,560 total parameters, 3,457,573,120 active. Official FP8 footprint about 78 GB. Provider minimum includes H200/B200/B300 or multiple 80 GB accelerators. This is not a claim of verified 128 GB GB10 support. Source: https://huggingface.co/Aleph-Alpha/Kolibri-1 Nemotron 3.5 Super VL: Provider reports 120B total, 12B active. Official deployment includes DGX Spark. Mixed-precision NVFP4, not all tensors at four bits. Source: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 Qwen3.8 Flash Next: Main model: 125B language backbone plus 51B n-gram embeddings = 176B. Optional 4B MTP adds 4B to checkpoint accounting (180B including MTP). Plot excludes auxiliary prediction heads consistently with the main-model convention. Four-bit weight estimate about 88 GB before runtime; practical deployment needs shortened/quantized context. Source: https://huggingface.co/Qwen/Qwen3.8-Flash-Next Apertus 1.5: Swiss AI Initiative (ETH Zurich, EPFL and CSCS), Switzerland. Current 1.5 release of the 70B family, with text/image/audio input and up to 262,144 context tokens. The official Hub reports approximately 72B BF16/F32 stored parameters after expanding input embeddings for multimodality; plot the rounded stored main-model count (72B), including embeddings, rather than treating the 70B family name as an exact count. No separate auxiliary prediction model is assumed. BF16 weights alone are about 144 GB, requiring the upper end of the 64-192 GB category or multiple accelerators; context and runtime add memory. Original weights require acceptance of published access conditions; this is not Soofi-style closed beta. Main-model source and parameter metadata checked 27 October 2026. Source: https://huggingface.co/swiss-ai/Apertus-v1.5-70B DeepSeek V4.1 Flash-1017: Latest released update verified 27 October 2026: Flash-1017, a minor update to V4.1 Flash. Base card specifies 552B backbone and a separate 196B Engram conditional-memory table. The plotted approximate main-model-plus-lookup count is 748B = 552+196; excluding auxiliary DSpark/MTP and separate vision components. This matches the chart convention used for Qwen n-gram embeddings. Do not plot only 552B while calling it total main model including embeddings. Official repository has weights and MIT license; full FP8/mixed distribution has ~763B stored tensors including auxiliary components. Sources: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash ; https://api-docs.deepseek.com/news/news261017/ . Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017 GLM-5.3: GLM-5.3 card says same base as GLM-5.2. The GLM-5 architecture report / official GLM-5 card gives 744B main parameters and 40B active; 753/754B Hub tensor counts include auxiliary MTP and accounting differences. Plot uses published main architecture count, excluding auxiliary prediction heads. Sources: https://huggingface.co/zai-org/GLM-5 ; https://huggingface.co/zai-org/GLM-5.2 ; https://arxiv.org/abs/2602.15763 . Public config files for GLM-5, GLM-5.2 and GLM-5.3-BF16 were compared on 27 October 2026: parameter-bearing architecture fields match. See qa/slide8-models/glm-architecture-check.json. Latest official Z.ai model list rechecked 27 October 2026: GLM-5.3 remains current flagship; Flash is a separate smaller newly trained model, also shown. Source: https://huggingface.co/zai-org/GLM-5.3 GLM-5.3 Flash: Current GLM-5.3 family member verified 27 October 2026. Official card gives 320B total, 18B active, native multimodal support and a newly trained base; it is not merely a precision conversion of the 744B GLM-5.3 backbone. Released weights and documented inference integrations are available. Added as another datacenter-scale example. Source: https://huggingface.co/zai-org/GLM-5.3-Flash Kimi K3.1 Pro: Latest Kimi release verified 27 October 2026. Official blog calls K3.1 Pro a post-trained version of K3 and a 3T-class model. The approximate 2.8T position is inferred from its published K3 backbone, not a newly released K3.1 checkpoint tensor count. K3.1 Pro and Highspeed are available in apps and API; weights are promised by 4 November and are not yet downloadable as of this check. Dashed dot and visible Weights Pending label distinguish intended datacenter capacity from a currently runnable local model. Do not describe current API access as local availability. K3 reference: https://huggingface.co/moonshotai/Kimi-K3 . Source: https://www.kimi.com/en/blog/kimi-k3-1-pro Laguna XS.2: Poolside's Laguna M.1/XS.2 technical report gives 33.4B total parameters and 3B activated per token. The visible name is Laguna XS.2, as requested, not the later Laguna XS 2.1. Apache 2.0 open weights; released official INT4 and NVFP4 variants. The official INT4 repository is approximately 23.9 GB because it mixes INT4 and INT8 experts and other tensors; do not use the ideal 16.7 GB four-bit arithmetic as its actual footprint. Place it at the 32 GB end of the 24-32 GB consumer category with constrained practical context, not a guarantee of full 256K context or a fit on a 24 GB GPU. File source: https://huggingface.co/poolside/Laguna-XS.2-INT4/tree/main . Base card independently describes local deployment on a Mac with 36 GB RAM. MoE active parameters affect per-token computation; all expert weights still count toward resident model memory. Sources checked 27 October 2026. Technical report: https://arxiv.org/abs/2605.27605 Official model: https://huggingface.co/poolside/Laguna-XS.2 Official INT4: https://huggingface.co/poolside/Laguna-XS.2-INT4 Layout update: Bolt moved right into workstation/server. The workstation column is wider to accommodate six models; numeric marker heights remain calculated from the same log parameter scale.

Gemma 4 Family

DenseSame path for every tokenMixture of ExpertsRouter selects experts

Total Parameters 30B 20B 10B 0

E2B5.1B totalDense
2.3B effective

E4B8B totalDense
4.5B effective

12B11.95B totalDense

26B-A4B25.2B totalMoE
3.8B active

31B30.7B totalDense

ECore size, excluding large embedding lookup tables

AParameters selected by the router for each token

9

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. Gemma 4 is a family, not one model. The vertical scale is linear and represents total parameter count: a 5.1B bar is roughly one sixth the height of a 30.7B bar. Blue and green describe architecture. Blue models are dense, meaning that every token follows the same network path. Green marks the mixture-of-experts model, where a router selects only some experts for each token. Effective and active sound similar, but they describe different mechanisms. In E2B and E4B, E means effective. These dense models contain large per-layer embedding tables. The tables must be stored, but each token only performs quick lookups in them. Google therefore describes the computational core as 2.3B or 4.5B effective parameters, while the complete stored models contain 5.1B or 8B parameters. In 26B-A4B, A means active. The complete mixture-of-experts model contains 25.2B parameters. For each token, its router selects eight experts plus one shared expert from 128 experts in each MoE layer, so about 3.8B parameters participate in the computation. All 25.2B parameters still have to fit in memory for fast inference. The 12B and 31B variants are dense models with 11.95B and 30.7B parameters. All five are multimodal. The E2B and E4B models support 128K context; the 12B, 26B-A4B and 31B models support 256K. Parameter count is only one dimension: quantization, context size and architecture determine the actual memory requirement and speed. Source: Google Gemma 4 model card, https://ai.google.dev/gemma/docs/core/model_card_4 . Transition: compare that range with the much smaller Qwen 3.6 family.

Memory Provisioning

RecommendedFully Served by VRAM

Sizing Criteria

Model Size
Quantization
KV Cache
Context Size
Operational Memory

10

The recommended target is to keep the complete inference workload in GPU memory. This avoids moving model data between storage, system RAM and VRAM while generating tokens. Start with the model size. Quantization determines how many bytes the stored weights require, plus scales and metadata. The KV cache needs additional memory and grows with the context size. Its exact size also depends on the model architecture, cache precision and batch size. Operational memory covers the inference engine, temporary compute buffers and memory allocator overhead. Leave headroom rather than planning exactly to the advertised VRAM capacity. On systems with unified memory, such as Apple silicon, CPU and GPU share one memory pool, but bandwidth, allocation limits and the operating system still reduce the amount available to the model. Transition: quantization changes the largest part of this calculation, the weight memory.

Quantization Principle

0:00.0 / 0:06.4

Original Range0–100

0100

18

47

83

÷ 10, then round

Quantized Range0–10

182

475

838

Reconstructed× 10

20Δ +2

50Δ +3

80Δ −3

11

Presenter controls: use Next to pause at the three AI DevCraft teaching checkpoints: highlight the original values, animate their mapping to 2/5/8, then reveal reconstruction and quantization error. Previous moves back through these checkpoints. Double arrows skip slides. Start with a deliberately simple example. The original values can occupy many positions between 0 and 100. We choose a scale of 10, divide each value by that scale and round to the nearest integer. That maps 18 to 2, 47 to 5 and 83 to 8. We now store one of only eleven integer values instead of a much finer original number. To reconstruct approximate original values, multiply the stored values by the same scale of 10. The results are 20, 50 and 80. Compared with 18, 47 and 83, the signed differences are +2, +3 and −3. This difference is quantization error: information discarded during rounding cannot be recovered exactly. Real model quantization can use different scales and zero points for tensors, channels or groups. Some methods also calibrate against data or treat important weights differently. The animation explains the basic idea, not a specific file format. Transition: translate the number of bits into model weight memory.

Choosing a Quantization Level

Qualityhigher fidelity

Memory and File Sizesmallerlarger

Local Sweet Spot

emergency fit

tight memory

default start

conservative

near reference

reference size

Practical RuleStart at Q4. Move up if quality breaks.
Move down only when the model does not fit.

12

Read this as a practical direction, not a universal benchmark curve. Moving from Q2 toward Q8 or BF16 usually increases file size and memory use while preserving more of the reference model's behavior. The exact quality difference depends on the model, the quantization algorithm, the dataset used for calibration, the runtime and the task. Labels such as Q4 or Q5 are families of formats rather than one standard method. A good local starting point is often a well-supported Q4 quantization. Move to Q5 or Q6 when the model fits comfortably and your evaluation finds meaningful quality loss. Move below Q4 only when memory capacity prevents the model from running, because aggressive quantization can damage reasoning and coding reliability. Q8 often stays close to the reference behavior but saves much less memory than Q4. BF16 represents the high-precision reference here. Always qualify the exact model file, not the label alone. Transition: video codecs show how compression methods can improve across generations.

Video Compression Over Time

The same mountain video frame — 1993MPEG-1≈ 600 MB

13

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. The repeated image is deliberate. We compare how much data is needed for roughly the same perceived result, rather than showing a sequence of better pictures. The file sizes form an illustrative one-minute 1080p example. They use a simple bitrate scale of 80 megabits per second for MPEG-1, 40 for MPEG-2, 20 for H.264, 10 for HEVC, 8 for AV1 and 5 for VVC. File size equals bitrate multiplied by sixty seconds and divided by eight. That gives about 600, 300, 150, 75, 60 and 37.5 megabytes. These are not measurements from one universal benchmark. MPEG-1 was designed for Video CD-era CIF or SIF resolution, so its 1080p figure is a conceptual extrapolation. The later values make the documented generational improvements easy to see. H.264 can deliver comparable quality at roughly half the bitrate of MPEG-2. HEVC targets another reduction of about half. AV1 arrived in 2018 as an open, royalty-free codec and practical tests show substantial savings over H.264 and VP9. VVC targets about half the bitrate of HEVC. Actual sizes vary with the content, encoder implementation, profile, settings and quality target. Newer codecs can also require more computation. This is the analogy to model quantization: better algorithms can preserve more useful quality within a smaller representation. It does not prove that every newer quantization method or every quantized model is better. We still have to test the exact artifact on our tasks. Sources: MPEG-1 Video, published in 1993, https://mpeg.chiariglione.org/standards/mpeg-1/video.html; ITU-T H.262, July 1995, https://www.itu.int/rec/T-REC-H.262-199507-S/en; ITU-T H.264, May 2003, https://www.itu.int/rec/T-REC-H.264-200305-P; ITU HEVC announcement, January 2013, https://www.itu.int/net/pressoffice/press_releases/2013/01.aspx; AOMedia AV1 specification release, March 2018, https://aomedia.org/press%20releases/the-alliance-for-open-media-kickstarts-video-innovation-era-with-av1-release/; Meta AV1 comparison, April 2018, https://engineering.fb.com/2018/04/10/video-engineering/av1-beats-x264-and-libvpx-vp9-in-practical-use-case/; Fraunhofer HHI VVC overview, finalized July 2020, https://www.hhi.fraunhofer.de/en/departments/vca/technologies-and-solutions/h266-vvc/vvc-overview.html. Transition: next, see how inference uses compute and memory during prefill and decoding. Illustrative file size for one minute of the same 1080p source at comparable visual quality.

Prefill and Decoding without KV Cache

0:00.0 / 0:18.5

Prefill

Compute-Bound

Decode

Recomputes the Entire Context

User Input
Tool Response
Thinking

Pleasebuildatestedcomponent

withourexistingsignalstore

andaddthebackendintegration

Model Layers

Predicted Token

returnthefinalresult

returnthefinal

Inputreturnthefinal

14

Presenter controls: Next plays to the next teaching checkpoint and pauses after each generated token. Previous returns to the preceding checkpoint. Double arrows skip slides. These are the original AI DevCraft workshop checkpoints. This first animation deliberately assumes that no KV cache exists. During prefill, the complete input passes through every transformer layer and the model predicts the first new token. The generated token then joins the input. Without cached keys and values, the next prediction requires another forward pass over the complete sequence. The animation moves the full, growing sequence through one layer at a time. Only after the complete sequence has passed through all layers can the model produce the next token. The model parameters do not move into the model with every step. They remain loaded in memory and the computation reads them again for each pass. Reprocessing the whole context makes uncached generation increasingly expensive as the sequence grows. Transition: the KV cache avoids most of that repeated work.

KV Cache During Inference

0:00.0 / 0:12.5

Prefill

Builds the Cache

Decode

Reuses the Cache

User Input
Tool Response
Thinking

KV Cache

Pleasebuildatestedcomponent

withourexistingsignalstore

andaddthebackendintegration

Model Layers

Predicted Token

returnthefinalresult

returnthefinal

Inputreturnthefinal

15

Presenter controls: Next plays to the next teaching checkpoint and pauses after each generated token. Previous returns to the preceding checkpoint. Double arrows skip slides. These are the original AI DevCraft workshop checkpoints. KV means key-value. The border is a conceptual aid: the cache does not store these token labels. During prefill, every transformer layer computes key and value tensors for the input tokens and stores them in the KV cache. During decode, the model adds key and value tensors for each new token. It reuses the cached tensors instead of recomputing the complete preceding sequence for every prediction. That reuse makes autoregressive decoding practical. The cache still grows with the token count, the number of layers, the attention shape, the batch size and the cache precision. It can consume a substantial amount of VRAM at long context sizes. Transition: apply these components to two concrete Gemma 4 memory plans.

Gemma 4 26B-A4B Memory Provisioning

Memory in VRAM / Unified Memory (GiB)

061218243036

16.72 GiB

13.27 GiB Model

1.45 GiB KV

2 GiB Runtime

64K

17.97 GiB

13.27 GiB Model

2.70 GiB KV

2 GiB Runtime

128K

20.47 GiB

13.27 GiB Model

5.20 GiB KV

2 GiB Runtime

256K

Context Size

16

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. This is a planning estimate for the quantized Gemma 4 26B-A4B artifact shown in the reference. The weight file remains constant at 13.27 GiB for every context size. A4B means that about 3.8 billion of roughly 25.2 billion total parameters are active for a token. That can reduce compute, but it does not turn the model into a 4B memory footprint. All expert weights still have to reside in memory. The F16 KV cache grows from 1.45 GiB at 64K to 2.70 GiB at 128K and 5.20 GiB at 256K. The runtime estimate is a range of one to two GiB. The totals shown here use the conservative two GiB endpoint: 16.72, 17.97 and 20.47 GiB. These numbers plan capacity; the actual allocation depends on the runtime, backend, batch settings and model file. Read the startup logs before treating the fit as guaranteed. Transition: compare this mixture-of-experts model with the dense Qwen 3.8 27B. All 25.2B parameters must fit in memory. Totals use a 2 GiB runtime reserve. IT QAT UD-Q4_K_XL by unsloth · 3.8B active / 25.2B total · one inference slot · F16 KV cache The two memory charts share the same 0-36 GiB scale. The source memory calculations are unchanged.

Qwen 3.8 27B Memory Provisioning

Memory in VRAM / Unified Memory (GiB)

061218243036

22.39 GiB

16.39 GiB Model

4.00 GiB KV

2 GiB Runtime

64K

26.39 GiB

16.39 GiB Model

8.00 GiB KV

2 GiB Runtime

128K

34.39 GiB

16.39 GiB Model

16.00 GiB KV

2 GiB Runtime

256K

Context Size

17

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. This planning estimate uses the 17.6 GB decimal UD-Q4_K_XL artifact published by unsloth, which is 16.39 GiB. Qwen 3.8 27B has 64 layers, with 16 full-attention layers, four KV heads and a head dimension of 256. With F16 keys and values, the growing KV cache is 4 GiB at 64K, 8 GiB at 128K and 16 GiB at the native 256K context. Adding a two GiB runtime reserve produces planning totals of 22.39, 26.39 and 34.39 GiB. The vision projector is not included. Actual llama.cpp allocation, state checkpoints and batch settings can require more headroom. Sources: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-Q4_K_XL.gguf ; https://huggingface.co/Qwen/Qwen3.8-27B ; https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/config.json . Transition: hardware capacity is only one part of the decision. The next chapter asks how we control quality. Hybrid architecture: 16 of 64 layers grow the KV cache. Totals use a 2 GiB runtime reserve. UD-Q4_K_XL by unsloth · 27B dense · 16 KV layers · one inference slot · F16 KV cache The two memory charts share the same 0-36 GiB scale. The source memory calculations are unchanged.

Quality vs. Data Sovereignty

Data Sovereignty Quality

Local Models

High Quality

Data Sovereignty

Frontier Models

Peak Quality

No Full Data Sovereignty

18

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. Treat these as two qualitative positions, not measured data points. Local models can deliver high-quality software work while keeping model execution and data inside infrastructure the organization controls. Frontier models still provide peak capability, but closed provider-hosted models do not provide full data sovereignty under this definition. Contracts, regional hosting and technical controls can improve privacy and governance, but the organization does not control the complete inference stack. The bubbles deliberately avoid precise coordinates because individual models, tasks and deployment choices vary. Treat speed as a separate question. Local inference speed mainly depends on hardware capacity and budget. With sufficiently powerful accelerators and a comparable model configuration, a local system can match the latency or throughput of cloud inference. Transition: use the Porsche analogy to ask whether every software task needs peak model capability. Speed is a hardware budget question. Powerful local hardware can match cloud inference speed. Use the shared arrows: reveal Local, then Frontier. Previous reverses; leaving resets the reveals.

A high-performance sports car caught in traffic and roadworks

The Porsche Test

Do We Need Peak Capability for Every Trip?

19

Benchmark philosophy

Qualification Instead of Ranking

Qualification threshold

Model A

Model B

Model C

Qualification criteria

✓Works with coding agents

✓Handles long-running tasks

✓Supports the required context

20

Full-Stack Capstone Task

Angular

Feature component
UI component
SignalStore
Unit test

Spring

JPA entities
Mapper
Controller
Database migration

Playwright

End-to-end test

21

The capstone is deliberately broad and delivered as one prompt. On the Angular side, the model must implement the feature according to existing architecture, separate feature and UI responsibilities, use a SignalStore, and add tests. On the Spring side, it must add the JPA model, mapper, controller and database migration. Finally, it must add a Playwright end-to-end test. This is difficult because success requires repository discovery, architectural consistency, cross-boundary changes and verification—not because one file is exotic. Logo sources: https://angular.dev/press-kit, https://simpleicons.org/?q=spring, https://playwright.dev/img/playwright-logo.svg. In this benchmark, the concrete feature is a German vocabulary trainer added to the existing holidays application. Use this slide to explain the scope before showing the benchmark components, validation and review loop.

Benchmark Components

Security: NVIDIA OpenShell
Implementation: Pi coding agent
Inference engine: llama.cpp
Hardware: GX10
Tracing: Langfuse
Versioning: Git worktrees

22

Use case: software development. The task adds a German vocabulary trainer to an existing holidays application, following its quiz architecture. The same five words are seeded for Lübeck and Vienna. The user opens an accessible Vocabulary Test link and answers questions; counts update on blur. This is one substantial full-stack feature, not a general intelligence benchmark. Use the diagram from left to right: fixed task and Git baseline, a dedicated worktree per sample, the Pi agent executing in OpenShell, and model inference on the GX10 through https://inference.local/v1. The configured OpenShell vm driver uses libkrun. PostgreSQL and Chromium run inside the sandbox; no host Docker socket is shared. The harness pins and checks model identity around sessions. Each model's concrete quantization is part of the tested setup, not normalized away. Current macOS pilot has limitations: Landlock and inner cgroup PID enforcement are unavailable on this driver. Do not promise production-grade hardening. Policy-denied runs are inconclusive infrastructure/policy outcomes, not model-quality failures. Official OpenShell source: https://github.com/NVIDIA/OpenShell Sources: /Users/rainerh/programming/local-models-for-coding-benchmark-langfuse/README.md; openshell/README.md; src/fresh-review-loop.ts. Current report: /Users/rainerh/programming/local-models-for-coding-benchmark/results/index.html. Audit records: results/outer-assessments.json. Snapshot inspected 13 October 2026. Component list: click Next to reveal one row at a time (six steps); Previous hides the last revealed row. No animated transition. Logos use the official project marks; GX10 uses the ASUS manufacturer wordmark. Sources, checked 13 October 2026: openshell: https://raw.githubusercontent.com/NVIDIA/OpenShell/main/docs/brand/assets/openshell-mark.svg pi: https://pi.dev/press-kit (embedded primary logo SVG) llama: https://raw.githubusercontent.com/ggml-org/llama.brand/master/icon/icon-light.svg langfuse: https://langfuse.com/brand-assets/icon/color/langfuse-icon.svg git: https://raw.githubusercontent.com/git/git-scm.com/gh-pages/static/images/logos/downloads/Git-Icon-1788C.svg asus: https://www.asus.com/networking-iot-servers/desktop-ai-supercomputer/desktop-ai-supercomputer/asus-ascent-gx10/ (embedded ASUS wordmark)

Worker, Verifier and Reviewer

🤖WRITE

Worker

Implements the task
Edits the worktree

EXECUTE

Validation

Builds · unit tests
Backend · Playwright

🤖READ ONLY

Reviewer

Does the patch meet
the requirements?

Done

Save result

🤖READ ONLY

Verifier

Explains failures

23

Worker writes; validation executes; verifier diagnoses failed checks; reviewer judges whether the successful patch meets the task. Validation includes frontend dev and production builds, frontend tests, backend tests and Playwright. Worker-triggered validation can help development, but the harness runs its authoritative checks separately. The verifier and reviewer receive fresh, read-only sessions. Their model is the same candidate in this cohort; separate roles are not independent external judges. A rejected review or diagnosis goes to a fresh worker session with a bounded handoff and the same sample's worktree. Each review round allows three worker attempts in total: the initial attempt plus at most two repairs. There are at most three review rounds. Per-session generated-output budgets: worker 250,000; verifier 50,000; reviewer 50,000 tokens. Time watchdogs are emergency limits. Completion can also mean exhausted budget or failed checks: all such outcomes are saved. External audit is outside this loop and is shown separately in the report. Sources: /Users/rainerh/programming/local-models-for-coding-benchmark-langfuse/README.md; openshell/README.md; src/fresh-review-loop.ts. Current report: /Users/rainerh/programming/local-models-for-coding-benchmark/results/index.html. Audit records: results/outer-assessments.json. Snapshot inspected 13 October 2026. Flow animation: the full diagram is visible immediately. Next/click/right arrow moves a pulse along one connection: Worker → Validation → (fail) Verifier → Worker → Validation → (pass) Reviewer → (decline) Worker → Validation → (pass) Reviewer → (accept) Done. Nine clicks complete the sequence. Previous rewinds one transition; leaving and re-entering resets it. Reduced-motion mode uses destination highlighting without movement. Stopping rules (configured for these benchmark runs): +- If deterministic validation still fails after the third worker attempt within one review round, stop the run as unsuccessful (repair_attempts_exhausted). The verifier diagnoses the failed checks; it does not itself produce the deterministic pass/fail verdict. This is three total attempts, not three retries after the initial attempt. +- If the reviewer declines in the third review round, stop the run as unsuccessful (review_rounds_exhausted). An earlier decline starts a new round with its worker-attempt counter reset to one. +- The maximum is therefore nine worker attempts across three review rounds, provided validation eventually passes within each round. Exhausting repairs in any round stops the run immediately. +- Keep the two outcomes distinct in the report: a review-exhausted run can still have passing validation. Infrastructure or policy interruptions and inconclusive runs are recorded separately; do not describe them as model failures. Diagram colors: blue boxes are the three AI-agent roles; the warm-colored Validation box runs deterministic checks. Green Done represents reviewer acceptance in the illustrated successful path. Limit sources: src/fresh-review-loop.ts (nextAfterVerification / nextAfterReview), src/openshell-executor.ts (initial position and control flow), and results/qwen38-token-v1-20261003-a1-live-backend-r1/config.env (MAX_REPAIR_ATTEMPTS=3, MAX_REVIEW_ROUNDS=3). Presenter reminder: the worker, verifier and reviewer in this cohort use the same candidate model. Separate sessions and roles do not create an independent external judge. The external audit is a separate evidence stage.

What The Selected Runs Show

Configuration	Validation	Reviewer
Qwen3.8 27BQ4_K_XL · 17.6 GB · Aug 2026 · 2:58	5/5	5/5
Qwen3.6 27BQ4_K_XL · 17.6 GB · Apr 2026 · 3:05	5/5	5/5
Qwen3-Coder 30B-A3BQ4_K_XL · 17.7 GB · Jul 2025 · 0:34	0/5	0/5
Gemma 4 26B-A4B QATQ4_K_XL · 14.2 GB · Jun 2026 · 0:46	2/4	1/41 incomplete
Gemma 4 31B QATQ4_K_XL · 17.3 GB · Jun 2026 · 1:44	1/3	1/3
Muse Glimmer 30BQ4_K_XL · 15.9 GB · Aug 2026 · 3:00	2/5	2/5

Configuration	Validation	Reviewer
Ternary Bonsai 2 27BPTQ1_0 · 5.9 GB · Sep 2026 · 4:06	4/4	4/4
Devstral Small 2 24BQ4_K_XL · 14.5 GB · Dec 2025 · 5:23	1/2	1/2
DeepSeek V4 FlashIQ1_M · 86.9 GB · Apr 2026 · 1:59	3/4	3/4
GLM-5.3 FlashIQ1_M · 97.6 GB · Aug 2026 · 1:27	4/4	4/4
Nemotron 3.5 Bolt 74BIQ2 · 22.0 GB · Oct 2026 · 1:34	2/2	1/21 incomplete
Nemotron 3.5 Lightning 30BQ4_0 · 18.9 GB · Aug 2026 · —	—	—Inconclusive

43 Runs · 12 Configurations

24

Snapshot checked 27 October 2026. Slide 24 now shows all twelve model configurations found in the current same-baseline, session-output-token-v1 benchmark cohort. Forty selected runs across ten configurations come from the saved report (last modified 23 October); direct artifact checks add three Gemma 4 31B QAT runs; Lightning is listed but its final inconclusive run is not scored. Total: 43 scored runs. Table order is for presentation, not a ranking. Fractions use all included runs for the exact configuration as denominator. Validation means all five recorded automated checks passed (frontend development build, frontend production build, frontend tests, backend tests, Playwright). Reviewer means accepted by the same candidate model in a fresh session. Reviewer not reached or incomplete is not acceptance. Gemma 26B and Bolt each have one incomplete reviewer, labelled on the slide. Reviewer acceptance is not independent qualification. Infrastructure exclusions already applied to the original report are preserved. New Lightning A1, A2 and A5 are excluded because their structured assistant message_end records contain direct 502/503 inference-service errors. Lightning A4 has no direct transport error, but the final campaign decision excludes it as inconclusive after three incomplete verifier sessions. Gemma 31B A2/A3/A5 have no direct inference-service errors; A2 passed and was accepted, A3/A5 failed validation. Missing terminal artifacts and in-progress attempts are not counted as model failures. No benchmark runs were launched or modified. The external-audit column and its verdicts are omitted from slide 24 at the user's request. The source benchmark report is unchanged. This is a view of validation and reviewer outcomes only. Source report: /Users/rainerh/programming/local-models-for-coding-benchmark/results/index.html Infrastructure exclusions: /Users/rainerh/programming/local-models-for-coding-benchmark/results/infrastructure-audit-20261023.md Exact included prefixes, result paths, configuration IDs, counts and additional infrastructure exclusions: assets/ijs/selected-runs-data.json Qwen3.8 27B (unsloth/Qwen3.8-27B-GGUF:Q4_K_XL): 5/5 validation passed; 5/5 reviewer accepted Included: qwen38-token-v1-20261003-a1-live-backend-r1, qwen38-token-v1-20261003-a2-live-backend-r1, qwen38-token-v1-20261003-a3-live-backend-r1, qwen38-token-v1-20261003-a4-live-backend-r1, qwen38-token-v1-20261003-a5-live-backend-r1 Qwen3.6 27B (unsloth/Qwen3.6-27B-GGUF:Q4_K_XL): 5/5 validation passed; 5/5 reviewer accepted Included: qwen36-token-v1-20261006-a2-live-backend-r1, qwen36-token-v1-20261006-a4-live-backend-r1, qwen36-token-v1-20261006-a5-live-backend-r1, qwen36-token-v1-20261006-a6-live-backend-r1, qwen36-token-v1-20261006-a7-live-backend-r1 Qwen3-Coder 30B-A3B (unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF:Q4_K_XL): 0/5 validation passed; 0/5 reviewer accepted Included: qwen3-coder-token-v1-20261006-a1-live-backend-r1, qwen3-coder-token-v1-20261006-a2-live-backend-r1, qwen3-coder-token-v1-20261006-a3-live-backend-r1, qwen3-coder-token-v1-20261006-a4-live-backend-r1, qwen3-coder-token-v1-20261006-a5-live-backend-r1 Gemma 4 26B-A4B QAT (unsloth/gemma-4-26B-A4B-it-qat-GGUF:Q4_K_XL): 2/4 validation passed; 1/4 reviewer accepted; 1 reviewer incomplete Included: gemma4-26b-a4b-token-v1-20261004-a1-live-backend-r1, gemma4-26b-a4b-token-v1-20261004-a2-live-backend-r1, gemma4-26b-a4b-token-v1-20261004-a3-live-backend-r1, gemma4-26b-a4b-token-v1-20261004-a5-live-backend-r1 Gemma 4 31B QAT (unsloth/gemma-4-31B-it-qat-GGUF:Q4_K_XL): 1/3 validation passed; 1/3 reviewer accepted Included: gemma4-31b-qat-token-v1-20261013-a2-live-backend-r1, gemma4-31b-qat-token-v1-20261013-a3-live-backend-r1, gemma4-31b-qat-token-v1-20261013-a5-live-backend-r1 Muse Glimmer 30B (unsloth/Muse-Glimmer-30B-GGUF:Q4_K_XL): 2/5 validation passed; 2/5 reviewer accepted Included: muse-glimmer-token-v1-20261004-a1-live-backend-r1, muse-glimmer-token-v1-20261004-a2-live-backend-r1, muse-glimmer-token-v1-20261004-a3-live-backend-r1, muse-glimmer-token-v1-20261004-a4-live-backend-r1, muse-glimmer-token-v1-20261004-a5-live-backend-r1 Ternary Bonsai 2 27B (prism-ml/Ternary-Bonsai-2-27B-gguf:PTQ1_0): 4/4 validation passed; 4/4 reviewer accepted Included: bonsai-token-v1-20261006-a1-live-backend-r1, bonsai-token-v1-20261006-a3-live-backend-r1, bonsai-token-v1-20261006-a5-live-backend-r1, bonsai-token-v1-20261006-a9-live-backend-r1 Devstral Small 2 24B (unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF:Q4_K_XL): 1/2 validation passed; 1/2 reviewer accepted Included: devstral-small2-token-v1-20261011-a2-live-backend-r1, devstral-small2-token-v1-20261011-a4-live-backend-r1 DeepSeek V4 Flash (unsloth/DeepSeek-V4-Flash-GGUF:IQ1_M): 3/4 validation passed; 3/4 reviewer accepted Included: deepseek-v4-flash-iq1m-token-v1-20261015-a2-live-backend-r1, deepseek-v4-flash-iq1m-token-v1-20261015-a3-live-backend-r1, deepseek-v4-flash-iq1m-token-v1-20261015-a4-live-backend-r1, deepseek-v4-flash-iq1m-token-v1-20261015-a5-live-backend-r1 GLM-5.3 Flash (unsloth/GLM-5.3-Flash-GGUF:IQ1_M): 4/4 validation passed; 4/4 reviewer accepted Included: glm53-iq1m-token-v1-20261018-a1-live-backend-r1, glm53-iq1m-token-v1-20261018-a2-live-backend-r1, glm53-iq1m-token-v1-20261018-a3-live-backend-r1, glm53-iq1m-token-v1-20261018-a5-live-backend-r1 Nemotron 3.5 Bolt 74B (nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF): 2/2 validation passed; 1/2 reviewer accepted; 1 reviewer incomplete Included: nemotron35-bolt-token-v1-20261018-a2-live-backend-r1, nemotron35-bolt-token-v1-20261018-a3-live-backend-r1 Nemotron 3.5 Lightning 30B (ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF): Inconclusive; excluded from scored totals Excluded as inconclusive: nemotron35-lightning-token-v1-20261023-a4-live-backend-r1 Metadata checked 27 October 2026. Decimal GB of main GGUF weights only, summing shards. Excludes separate vision projectors, MTP/draft models, KV cache and runtime overhead. Dates are underlying model release months; Gemma QAT uses June 2026 QAT release, not April base-model release. Qwen3.8 27B: Q4_K_XL · 17.6 GB · Aug 2026 Release: https://github.com/QwenLM/Qwen3.8 GGUF file sizes: https://huggingface.co/api/models/unsloth/Qwen3.8-27B-GGUF?blobs=true Files: Qwen3.8-27B-UD-Q4_K_XL.gguf (17559178144 bytes) Qwen3.6 27B: Q4_K_XL · 17.6 GB · Apr 2026 Release: https://qwen.ai/blog?id=qwen3.6-27b&lang=en GGUF file sizes: https://huggingface.co/api/models/unsloth/Qwen3.6-27B-GGUF?blobs=true Files: Qwen3.6-27B-UD-Q4_K_XL.gguf (17612564704 bytes) Qwen3-Coder 30B-A3B: Q4_K_XL · 17.7 GB · Jul 2025 Release: https://github.com/QwenLM/Qwen3-Coder GGUF file sizes: https://huggingface.co/api/models/unsloth/Qwen3-Coder-30B-A3B-Instruct-GGUF?blobs=true Files: Qwen3-Coder-30B-A3B-Instruct-UD-Q4_K_XL.gguf (17665334432 bytes) Gemma 4 26B-A4B QAT: Q4_K_XL · 14.2 GB · Jun 2026 Release: https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4-builders/ GGUF file sizes: https://huggingface.co/api/models/unsloth/gemma-4-26B-A4B-it-qat-GGUF?blobs=true Files: gemma-4-26B-A4B-it-qat-UD-Q4_K_XL.gguf (14249047104 bytes) Gemma 4 31B QAT: Q4_K_XL · 17.3 GB · Jun 2026 Release: https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4-builders/ GGUF file sizes: https://huggingface.co/api/models/unsloth/gemma-4-31B-it-qat-GGUF?blobs=true Files: gemma-4-31B-it-qat-UD-Q4_K_XL.gguf (17287670048 bytes) Muse Glimmer 30B: Q4_K_XL · 15.9 GB · Aug 2026 Release: https://research.meta.ai/blog/introducing-muse-glimmer-open-agentic-model GGUF file sizes: https://huggingface.co/api/models/unsloth/Muse-Glimmer-30B-GGUF?blobs=true Files: Muse-Glimmer-30B-UD-Q4_K_XL.gguf (15878222368 bytes) Ternary Bonsai 2 27B: PTQ1_0 · 5.9 GB · Sep 2026 Release: https://prismml.com/news/prismml-launches-bonsai-2-27b GGUF file sizes: https://huggingface.co/api/models/prism-ml/Ternary-Bonsai-2-27B-gguf?blobs=true Files: Ternary-Bonsai-2-27B-PTQ1_0.gguf (5946648928 bytes) Devstral Small 2 24B: Q4_K_XL · 14.5 GB · Dec 2025 Release: https://mistral.ai/news/devstral-2-vibe-cli/ GGUF file sizes: https://huggingface.co/api/models/unsloth/Devstral-Small-2-24B-Instruct-2512-GGUF?blobs=true Files: Devstral-Small-2-24B-Instruct-2512-UD-Q4_K_XL.gguf (14506151072 bytes) DeepSeek V4 Flash: IQ1_M · 86.9 GB · Apr 2026 Release: https://deepseek.com/en/news/v4-preview/ GGUF file sizes: https://huggingface.co/api/models/unsloth/DeepSeek-V4-Flash-GGUF?blobs=true Files: UD-IQ1_M/DeepSeek-V4-Flash-UD-IQ1_M-00001-of-00003.gguf (5256864 bytes), UD-IQ1_M/DeepSeek-V4-Flash-UD-IQ1_M-00002-of-00003.gguf (49971881120 bytes), UD-IQ1_M/DeepSeek-V4-Flash-UD-IQ1_M-00003-of-00003.gguf (36924175168 bytes) GLM-5.3 Flash: IQ1_M · 97.6 GB · Aug 2026 Release: https://autoclaw.z.ai/models/ GGUF file sizes: https://huggingface.co/api/models/unsloth/GLM-5.3-Flash-GGUF?blobs=true Files: UD-IQ1_M/GLM-5.3-Flash-UD-IQ1_M-00001-of-00003.gguf (9429984 bytes), UD-IQ1_M/GLM-5.3-Flash-UD-IQ1_M-00002-of-00003.gguf (49996246592 bytes), UD-IQ1_M/GLM-5.3-Flash-UD-IQ1_M-00003-of-00003.gguf (47573668096 bytes) Nemotron 3.5 Bolt 74B: IQ2 · 22.0 GB · Oct 2026 Release: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF GGUF file sizes: https://huggingface.co/api/models/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF?blobs=true Files: NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB.gguf (22018326528 bytes) Nemotron 3.5 Lightning 30B: Q4_0 · 18.9 GB · Aug 2026 Release: https://blogs.nvidia.com/blog/nemotron-lightning-switchyard-rtx-dgx/ GGUF file sizes: https://huggingface.co/api/models/ggml-org/NVIDIA-Nemotron-3.5-Lightning-30B-A3B-GGUF?blobs=true Files: NVIDIA-Nemotron-3.5-Lightning-30B-A3B-Q4_0.gguf (18898091584 bytes) Lightning correction: final campaign decision at 22:00Z classifies A4 INCONCLUSIVE after three incomplete verifier sessions. It is excluded from scored totals; em dashes are not a failure. Source: /Users/rainerh/programming/local-models-for-coding-benchmark/results/campaigns/lightning-readiness-20261023.md Run times: Median of agent-result.json duration_ms across the included scored runs for each configuration, rounded to the nearest minute and shown as hours:minutes. Full workflow elapsed time includes model generation, tool execution, validation, repairs and reviewer steps where reached. Failed runs may finish earlier. Inconclusive Lightning is not assigned a duration. Qwen3.8 27B: 2:58; recorded durations in milliseconds: [12404539, 10539915, 16484930, 6542340, 10651918] Qwen3.6 27B: 3:05; recorded durations in milliseconds: [15081223, 11106332, 3401857, 5117736, 13567476] Qwen3-Coder 30B-A3B: 0:34; recorded durations in milliseconds: [2086247, 2034579, 2913952, 2068667, 1781091] Gemma 4 26B-A4B QAT: 0:46; recorded durations in milliseconds: [1495303, 801932, 5919252, 4049135] Gemma 4 31B QAT: 1:44; recorded durations in milliseconds: [2093067, 6540759, 6268379] Muse Glimmer 30B: 3:00; recorded durations in milliseconds: [9557874, 6006248, 12918488, 11847213, 10822578] Ternary Bonsai 2 27B: 4:06; recorded durations in milliseconds: [17854992, 11648766, 10149993, 23293784] Devstral Small 2 24B: 5:23; recorded durations in milliseconds: [2957176, 35828501] DeepSeek V4 Flash: 1:59; recorded durations in milliseconds: [7948944, 2797101, 6294333, 16078104] GLM-5.3 Flash: 1:27; recorded durations in milliseconds: [4790451, 5088067, 8490719, 5305288] Nemotron 3.5 Bolt 74B: 1:34; recorded durations in milliseconds: [6747507, 4536265] Nemotron 3.5 Lightning 30B: —; recorded durations in milliseconds: []

A Green Build Is One Check

Validation

Does the application build?

Do the tests pass?

Requirements And Audit

Does the feature match the task?

Does independent evidence agree?

Reviewer acceptance and external audit can disagree.

25

The surrounding system

Engineering Around the Model

Models make mistakes. The surrounding system must detect them and remain in control.

Bound each model turn and run verification through external tools, so inner loops cannot block the workflow.
A well-structured codebase gives the model reliable patterns to follow.
Application-specific instructions and skills cover Angular, Spring, Playwright, and project conventions.
24/7 local inference makes loop / graph engineering with independent review and repeated validation practical.

26

A Bounded Model Session

Bounded model session

ModelGemma 4

Tool handlerExecutes the call

Tools

Validationng build./gradlew test

READANSWER

WRITEANSWER

VALIDATE

0:00.0 / 0:13.0

27

Local, Hosted Or Hybrid?

Local Inference

Control over the inference boundary.

Owned capacity and local operations.

Hosted Or Hybrid

Access to other model capabilities.

Explicit rules for data and routing.

The workload, data boundary and measured outcome decide.

28

How Far Can You Get?

Local coding is useful when the complete system clears the task's acceptance bar.

01

One representative task.

02

One recorded configuration.

03

Repeated, independently checked outcomes.

29

pi-kit

For Local Models

Pi + AG-UI

Meat Proxy Detector
Self-Learning
Background Agents

pi-kit.dev

pi-kit desktop UI with a coding conversation, local Gemma model controls and an integrated Artifacts browser

30

Introduce pi-kit as our desktop coding harness built on the Pi coding agent, with a focus on working with local models. It also supports hosted providers. The interface uses AG-UI and the Angular port of CopilotKit for tool interfaces, input requests and MCP Apps. Verified from the public website on 26 October 2026: local GGUF model setup through Hugging Face and llama-swap; durable background agents; session branching; Git worktrees and review; Artifacts browser; skill drafting and verification. Self-learning creates reusable instructions, not new model weights. Proposed skills are reviewed before installation; replay verification is optional. The product screenshot is captured from the homepage's real pi-kit desktop UI figure. It contains a seeded demo conversation and a separate local demo project, as identified by its original caption; it is not a benchmark result. Sources: https://pi-kit.dev/ ; https://pi-kit.dev/docs/learning ; https://pi-kit.dev/docs/settings . The website describes a beta desktop client. Rainer explicitly supplied the feature name Meat Proxy Detector on 26 October 2026. That name is included as user-provided product information; the inspected public pages do not describe its behavior. The screenshot and pi-kit.dev text are clickable links. This slide sits immediately before the existing Questions closing slide.

Q&A

Your questions about local models

Your Feedback

How did you like the talk?

QR code to the talk feedback form — soverius.ai/feedback/ijs-munich-2026-local-models

31

Five minutes for questions. The Soverius AI feedback form was duplicated from DevFest Venice at the user's request on 27 October 2026. Same questions: required star rating (1-5), How did you like the talk in general?; optional free text, What did you like? What could we improve? Anything else? Description and newsletter opt-in match the Venice template. Form has no time limits. No unrelated Venice talk or resource links are attached. Feedback URL: https://soverius.ai/feedback/ijs-munich-2026-local-models Form ID: 7e99439c-bb67-46de-ad1e-314b7ad65fec Template form: 57b94e9a-f942-4be5-8a66-e62440171442 Layout reused from /Users/rainerh/Documents/Soverius/Venice-2026/agentic-frontends/templates/feedback.html and its feedback CSS. Kept the same composition with a large clickable QR, adapted the short copy to this local-model talk, and did not promise slides or repository resources that have not been published for this session. pi-kit stays immediately before the closing Q&A/feedback slide. This is Soverius AI feedback, with no organiser template.