W-JAX 2026 · Munich

5 November 2026

Java-Entwicklung mit lokalen AI-ModellenWie weit kommt man?

Laptop with a Java icon on screen and a glowing local model mounted on its exposed motherboard.

Rainer Hahnekamp Soverius AI

1

Soverius AI — We build production-ready AI systems

Why local?

Reasons to Run Models Locally

€

Cost Control

No provider charge per token.

Privacy

Keep code and context inside.

Independence

Reduce provider dependency.

3

Cost Schema: Local vs. Frontier

4

Three presenter-controlled steps through the shared arrows. Entry shows the axes and legend; there is no autoplay. 1. Draw only the first quarter of both curves: Local and a low, flat Frontier subscription. Within the included usage budget, the subscription is cheaper in this illustrative comparison. 2. Extend both curves. Frontier reaches the end of its included allowance, then API Token Costs rise and eventually cross above Local. The shaded area marks the region where the illustrative frontier cost exceeds local cost. 3. Reveal Automatic Loops with a rightward arrow toward greater token usage. Loops can be used from the beginning; they do not start at a particular point on this chart. As soon as they are used, repeated model calls consume more tokens and make exceeding the subscription allowance more likely. The horizontal axis is Tokens, not time. The arrow indicates consumption direction and is not tied to the subscription/API bend. Starting a loop does not itself trigger API billing; provider-plan and tool rules determine how extra usage is charged. Previous reverses one teaching step. Double arrows skip slides; returning starts at step 0. Reduced-motion mode shows the same states without line-drawing motion. The local curve starts higher because infrastructure must be purchased, then rises gently with energy and operating costs. The comparison assumes a fixed accounting period, model/input-output/cache mix and workload within local capacity. It is schematic, with no measured break-even price or token count. Capacity upgrades, model quality, utilisation and engineering effort affect real costs. Plan/usage reference: https://support.claude.com/en/articles/12429409-manage-usage-credits-for-paid-claude-plans (inspected 26 October 2026).

Prerequisites

Find the Smallest Model That Can Do the Job
Review the Generated Code
Engineer the Loop

5

Frame the talk around three prerequisites: a model small enough for the available hardware and capable enough for the task, a developer who reads the generated code, and a loop that can find and repair failures. Larger models such as GLM-5.3 Flash, GLM-5.3, and Kimi K3.1 Pro illustrate the high end. The question for this talk is how far down the hardware and model-size scale we can go. The consumer-GPU category gives us a concrete starting point. First determine which model and context fit, then qualify that configuration on work like yours. The benchmark covers specific configurations on our setup. It does not establish a universal parameter cutoff. Read the patch as well as the test results. The developer still needs to decide whether the change implements the intent and belongs in this codebase. Model review and automated tests supply evidence for that decision. The loop is part of what makes a model usable. Bound worker turns, run builds and tests outside the model, diagnose failures in a fresh session, and pass that evidence to a fresh worker for repair. Review the patch in a separate session after validation passes. Later in the benchmark, use Qwen3.8 27B A3 as an example. In that run, the workflow recorded two worker attempts and one verifier session before the patch passed validation and was accepted by the same-model reviewer. That shows recovery within this run. A separate single-pass baseline would be needed to quantify the loop’s contribution. A well-structured codebase gives the model reliable patterns. Supply application-specific instructions and skills for Angular, Spring, Playwright, and project conventions. Local inference makes repeated repair and review practical without a provider token charge for those calls. Hardware, energy, engineering and hosted tools still have costs. Run evidence: results/qwen38-token-v1-20261003-a3-live-backend-r1/agent-result.json in the local-models-for-coding-benchmark repository.

Typical Required Components

MODEL

Gemma
Qwen
Muse Glimmer

← runs

INFERENCE ENGINE

llama.cpp
vLLM
ONNX Runtime

← calls

AGENT

Claude Code
Google ADK
Mastra
LangGraph

6

Build the diagram from the model outward. First, the model predicts tokens. Next, the inference engine loads and runs the weights. Finally, the agent calls the engine and manages the application workflow and tools. Gemma, Qwen and Meta’s Muse Glimmer are example models and model families. llama.cpp, vLLM and ONNX Runtime are example inference engines. The ONNX browser example uses ONNX Runtime Web through Transformers.js. Claude Code is a ready-made coding agent; Google ADK, Mastra and LangGraph are frameworks for building agents. These are examples of each role, not a claim that every combination connects without configuration. Simple prompting applications can call the inference engine directly without an agent framework. Logo sources: adk.png: https://raw.githubusercontent.com/google/adk-python/main/assets/agent-development-kit.png vllm.png: https://raw.githubusercontent.com/vllm-project/vllm/main/docs/assets/logos/vllm-logo-only-light.png langgraph.svg: https://raw.githubusercontent.com/langchain-ai/langgraph/main/.github/images/logo-light.svg (standalone symbol) qwen.jpeg: https://cdn-avatars.huggingface.co/v1/production/uploads/6215ca5692c0ecfba9186921/hrRM50-6XcdWgg2AKpENG.jpeg mastra.svg: https://mastra.ai (official header symbol, static first frame) claude: https://commons.wikimedia.org/wiki/File:Claude_AI_symbol.svg llama.cpp: https://github.com/ggml-org/llama.brand gemma: https://github.com/google-deepmind/gemma/blob/main/docs/images/icon.svg onnx-runtime.svg: https://onnxruntime.ai/images/logos/onnxruntime/ORT_icon_for_light_bg.svg (official icon for light backgrounds; https://onnxruntime.ai/docs/reference/citing.html)

CPU and GPU Inference

CPU1× ComputeIntel Core i9-14900K

Hard Disk or SSD≈0.1–7 GB/sBarraCuda HDD
870 EVO / 990 PRO SSDSwap · No Compute

System RAM≈90–100 GB/sDual-Channel DDR5-5600

PossibleOften too slow

GPU≈30–100× ComputeRTX 4070 Super – RTX 5090

VRAM≈500–1,800 GB/sGDDR6X – GDDR7

PracticalUsually 5–10× faster

You can serve a model larger than 1 TB from disk, but do not expect a response anytime soon.

7

Start with the simple answer. A model can fit on a hard disk or SSD, but storage performs no model computation. The storage bandwidth range combines an ordinary hard disk at roughly 0.1 to 0.2 GB/s, a SATA SSD around 0.5 GB/s and a fast consumer NVMe SSD around 7 GB/s. llama.cpp can memory-map the model file and ask the operating system to load the required pages into system memory. If the model is larger than the available RAM, those pages must be loaded and discarded repeatedly. That swapping can make inference extremely slow. CPU plus system RAM can execute the model without a GPU. The 90 to 100 GB/s figure represents a normal dual-channel DDR5 desktop system, and the CPU serves as the 1-times compute baseline. GPU plus VRAM is the preferred path because both components are faster. Current consumer GPUs suitable for local inference span roughly 500 to 1,800 GB/s of memory bandwidth, while their parallel matrix throughput can be tens of times higher than a desktop CPU. Do not multiply those two ratios. Each phase is limited by its slower resource. As a rule of thumb, full GPU inference is often five to ten times faster than CPU-only inference for token generation. Prompt processing can show a larger difference. The precise speed depends on the model, quantization, hardware and runtime. Unified-memory systems combine RAM and VRAM into one pool, but the same principle remains: memory bandwidth and GPU compute determine the useful speed. Reference hardware behind the displayed bandwidths: - CPU and system RAM: Intel Core i9-14900K with dual-channel DDR5-5600, 89.6 GB/s, rounded to about 90 GB/s. - GPU lower endpoint: NVIDIA GeForce RTX 4070 SUPER, 504 GB/s. - GPU upper endpoint: NVIDIA GeForce RTX 5090, 1,792 GB/s. - Hard disk: Seagate BarraCuda ST2000DM008, up to 220 MB/s. - SATA SSD: Samsung 870 EVO, up to 560 MB/s. - NVMe SSD: Samsung 990 PRO, up to 7,450 MB/s. Transition: now look at three concrete hardware classes for local inference. Sources: https://www.seagate.com/files/www-content/product-content/barracuda-fam/barracuda-new/en-us/docs/100817550L.pdf ; https://semiconductor.samsung.com/news-events/news/samsung-introduces-latest-in-its-worlds-best-selling-consumer-sata-ssd-series-the-870-evo/ ; https://download.semiconductor.samsung.com/resources/brochure/990_PRO_Series_Brouchure_Web_Version_1.0.pdf ; https://www.intel.com/content/www/us/en/products/sku/236773/intel-core-i9-processor-14900k-36m-cache-up-to-6-00-ghz/specifications.html ; https://www.nvidia.com/en-us/geforce/graphics-cards/40-series/rtx-4070-family/ ; https://www.nvidia.com/en-sg/geforce/graphics-cards/50-series/rtx-5090/ ; https://github.com/ggml-org/llama.cpp

Local Inference Hardware

Consumer

RTX PRO 4000

24 GB VRAM≈ €2,500 per GPU

Mac Studio

36-512 GB Unifiedfrom €2,999

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. These examples show practical local-inference options rather than a complete buying guide. The consumer column deliberately contains two different approaches. NVIDIA's RTX PRO 4000 Blackwell is a discrete GPU with 24 GB of GDDR7 VRAM and 672 GB/s of memory bandwidth. Twenty-four gigabytes is a useful starting point for quantized coding models, although the usable model size also depends on the context and KV cache. The RTX PRO name belongs to NVIDIA's professional range; consumer here describes the size of the local setup, not NVIDIA's market category. Austrian retail listings put the card at roughly €2,500. This buys the GPU, not a complete computer. A Mac Studio is the integrated alternative. Apple's current configurations use M5 Max or M5 Ultra and range from 36 GB to 512 GB of unified memory. The 512 GB figure is a configurable maximum, not the base configuration. M5 Max configurations span 36 to 128 GB. M5 Ultra configurations span 96 to 512 GB. Apple starts the M5 Max system at €2,999 and the M5 Ultra system at €6,599 in Austria. Apple says the 512 GB option will become available at the end of October 2026, and the final price of that option is not yet shown. Unified memory is shared by CPU and GPU, so it is not directly equivalent to dedicated VRAM, but the larger configurations can keep very large quantized models in memory. The wide range is important: Mac Studio is one product family, not one fixed capacity. The middle class is the ASUS Ascent GX10. Its NVIDIA GB10 Grace Blackwell Superchip uses 128 GB of coherent unified LPDDR5x memory. This gives a single developer far more model capacity than a 24 GB graphics card, although its 273 GB/s memory bandwidth is lower than the bandwidth of high-end discrete GPUs. Current Austrian retail listings start near €4,850 for the 1 TB system. ASUS originally announced a €3,499 recommended price, so availability and street pricing matter. The enterprise class uses RTX PRO 6000 Blackwell GPUs. Each GPU provides 96 GB of GDDR7 ECC memory. Austrian retail listings for the NVIDIA Workstation Edition currently start around €17,700 per card. The three cards in the picture would therefore represent more than €50,000 in GPUs alone. A workstation can contain several cards. Multiple GPUs add capacity and throughput, but one model does not automatically see them as one simple memory pool. The inference runtime must split or replicate the model, and interconnect bandwidth matters. All prices on the slide are rounded snapshots checked on 7 September 2026 and can change. Transition: move from physical machines to the model sizes and deployment classes they can support. Sources: https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-4000/ ; https://geizhals.at/nvidia-rtx-pro-4000-blackwell-900-5g147-2570-000-a3488409.html ; https://www.apple.com/at/newsroom/2026/08/apple-introduces-new-mac-studio-with-m5-max-and-m5-ultra/ ; https://www.apple.com/at/mac-studio/specs/ ; https://www.asus.com/networking-iot-servers/desktop-ai-supercomputer/ultra-small-ai-supercomputers/asus-ascent-gx10/ ; https://www.idealo.at/preisvergleich/OffersOfProduct/208466423_-ascent-gx10-asus.html ; https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000-family/ ; https://geizhals.at/nvidia-rtx-pro-6000-blackwell-workstation-edition-900-5g144-2500-000-a3476784.html Image sources: RTX PRO 4000 and RTX PRO 6000 images supplied by the presenter; https://www.apple.com/v/mac-studio/specs/b/images/specs/hero__d0yd117u6qy6_large_2x.jpg ; https://dlcdnwebimgs.asus.com/gain/c66958f8-57de-4563-b0db-bd317bfaf1f3/w800 Example configurations - not endorsements or product recommendations.

Ambitious Consumer

ASUS Ascent GX10

128 GB Unified Memory ≈ €4,850 per system

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. These examples show practical local-inference options rather than a complete buying guide. The consumer column deliberately contains two different approaches. NVIDIA's RTX PRO 4000 Blackwell is a discrete GPU with 24 GB of GDDR7 VRAM and 672 GB/s of memory bandwidth. Twenty-four gigabytes is a useful starting point for quantized coding models, although the usable model size also depends on the context and KV cache. The RTX PRO name belongs to NVIDIA's professional range; consumer here describes the size of the local setup, not NVIDIA's market category. Austrian retail listings put the card at roughly €2,500. This buys the GPU, not a complete computer. A Mac Studio is the integrated alternative. Apple's current configurations use M5 Max or M5 Ultra and range from 36 GB to 512 GB of unified memory. The 512 GB figure is a configurable maximum, not the base configuration. M5 Max configurations span 36 to 128 GB. M5 Ultra configurations span 96 to 512 GB. Apple starts the M5 Max system at €2,999 and the M5 Ultra system at €6,599 in Austria. Apple says the 512 GB option will become available at the end of October 2026, and the final price of that option is not yet shown. Unified memory is shared by CPU and GPU, so it is not directly equivalent to dedicated VRAM, but the larger configurations can keep very large quantized models in memory. The wide range is important: Mac Studio is one product family, not one fixed capacity. The middle class is the ASUS Ascent GX10. Its NVIDIA GB10 Grace Blackwell Superchip uses 128 GB of coherent unified LPDDR5x memory. This gives a single developer far more model capacity than a 24 GB graphics card, although its 273 GB/s memory bandwidth is lower than the bandwidth of high-end discrete GPUs. Current Austrian retail listings start near €4,850 for the 1 TB system. ASUS originally announced a €3,499 recommended price, so availability and street pricing matter. The enterprise class uses RTX PRO 6000 Blackwell GPUs. Each GPU provides 96 GB of GDDR7 ECC memory. Austrian retail listings for the NVIDIA Workstation Edition currently start around €17,700 per card. The three cards in the picture would therefore represent more than €50,000 in GPUs alone. A workstation can contain several cards. Multiple GPUs add capacity and throughput, but one model does not automatically see them as one simple memory pool. The inference runtime must split or replicate the model, and interconnect bandwidth matters. All prices on the slide are rounded snapshots checked on 7 September 2026 and can change. Transition: move from physical machines to the model sizes and deployment classes they can support. Sources: https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-4000/ ; https://geizhals.at/nvidia-rtx-pro-4000-blackwell-900-5g147-2570-000-a3488409.html ; https://www.apple.com/at/newsroom/2026/08/apple-introduces-new-mac-studio-with-m5-max-and-m5-ultra/ ; https://www.apple.com/at/mac-studio/specs/ ; https://www.asus.com/networking-iot-servers/desktop-ai-supercomputer/ultra-small-ai-supercomputers/asus-ascent-gx10/ ; https://www.idealo.at/preisvergleich/OffersOfProduct/208466423_-ascent-gx10-asus.html ; https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000-family/ ; https://geizhals.at/nvidia-rtx-pro-6000-blackwell-workstation-edition-900-5g144-2500-000-a3476784.html Image sources: RTX PRO 4000 and RTX PRO 6000 images supplied by the presenter; https://www.apple.com/v/mac-studio/specs/b/images/specs/hero__d0yd117u6qy6_large_2x.jpg ; https://dlcdnwebimgs.asus.com/gain/c66958f8-57de-4563-b0db-bd317bfaf1f3/w800 Example configurations - not endorsements or product recommendations.

Enterprise

$NVIDIA RTX PRO 6000 Blackwell graphics card$

RTX PRO 6000 Blackwell

96 GB VRAM per GPU ≈ €17,700 each

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. These examples show practical local-inference options rather than a complete buying guide. The consumer column deliberately contains two different approaches. NVIDIA's RTX PRO 4000 Blackwell is a discrete GPU with 24 GB of GDDR7 VRAM and 672 GB/s of memory bandwidth. Twenty-four gigabytes is a useful starting point for quantized coding models, although the usable model size also depends on the context and KV cache. The RTX PRO name belongs to NVIDIA's professional range; consumer here describes the size of the local setup, not NVIDIA's market category. Austrian retail listings put the card at roughly €2,500. This buys the GPU, not a complete computer. A Mac Studio is the integrated alternative. Apple's current configurations use M5 Max or M5 Ultra and range from 36 GB to 512 GB of unified memory. The 512 GB figure is a configurable maximum, not the base configuration. M5 Max configurations span 36 to 128 GB. M5 Ultra configurations span 96 to 512 GB. Apple starts the M5 Max system at €2,999 and the M5 Ultra system at €6,599 in Austria. Apple says the 512 GB option will become available at the end of October 2026, and the final price of that option is not yet shown. Unified memory is shared by CPU and GPU, so it is not directly equivalent to dedicated VRAM, but the larger configurations can keep very large quantized models in memory. The wide range is important: Mac Studio is one product family, not one fixed capacity. The middle class is the ASUS Ascent GX10. Its NVIDIA GB10 Grace Blackwell Superchip uses 128 GB of coherent unified LPDDR5x memory. This gives a single developer far more model capacity than a 24 GB graphics card, although its 273 GB/s memory bandwidth is lower than the bandwidth of high-end discrete GPUs. Current Austrian retail listings start near €4,850 for the 1 TB system. ASUS originally announced a €3,499 recommended price, so availability and street pricing matter. The enterprise class uses RTX PRO 6000 Blackwell GPUs. Each GPU provides 96 GB of GDDR7 ECC memory. Austrian retail listings for the NVIDIA Workstation Edition currently start around €17,700 per card. The three cards in the picture would therefore represent more than €50,000 in GPUs alone. A workstation can contain several cards. Multiple GPUs add capacity and throughput, but one model does not automatically see them as one simple memory pool. The inference runtime must split or replicate the model, and interconnect bandwidth matters. All prices on the slide are rounded snapshots checked on 7 September 2026 and can change. Transition: move from physical machines to the model sizes and deployment classes they can support. Sources: https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-4000/ ; https://geizhals.at/nvidia-rtx-pro-4000-blackwell-900-5g147-2570-000-a3488409.html ; https://www.apple.com/at/newsroom/2026/08/apple-introduces-new-mac-studio-with-m5-max-and-m5-ultra/ ; https://www.apple.com/at/mac-studio/specs/ ; https://www.asus.com/networking-iot-servers/desktop-ai-supercomputer/ultra-small-ai-supercomputers/asus-ascent-gx10/ ; https://www.idealo.at/preisvergleich/OffersOfProduct/208466423_-ascent-gx10-asus.html ; https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000-family/ ; https://geizhals.at/nvidia-rtx-pro-6000-blackwell-workstation-edition-900-5g144-2500-000-a3476784.html Image sources: RTX PRO 4000 and RTX PRO 6000 images supplied by the presenter; https://www.apple.com/v/mac-studio/specs/b/images/specs/hero__d0yd117u6qy6_large_2x.jpg ; https://dlcdnwebimgs.asus.com/gain/c66958f8-57de-4563-b0db-bd317bfaf1f3/w800 Example configurations - not endorsements or product recommendations.

8

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. These examples show practical local-inference options rather than a complete buying guide. The consumer column deliberately contains two different approaches. NVIDIA's RTX PRO 4000 Blackwell is a discrete GPU with 24 GB of GDDR7 VRAM and 672 GB/s of memory bandwidth. Twenty-four gigabytes is a useful starting point for quantized coding models, although the usable model size also depends on the context and KV cache. The RTX PRO name belongs to NVIDIA's professional range; consumer here describes the size of the local setup, not NVIDIA's market category. Austrian retail listings put the card at roughly €2,500. This buys the GPU, not a complete computer. A Mac Studio is the integrated alternative. Apple's current configurations use M5 Max or M5 Ultra and range from 36 GB to 512 GB of unified memory. The 512 GB figure is a configurable maximum, not the base configuration. M5 Max configurations span 36 to 128 GB. M5 Ultra configurations span 96 to 512 GB. Apple starts the M5 Max system at €2,999 and the M5 Ultra system at €6,599 in Austria. Apple says the 512 GB option will become available at the end of October 2026, and the final price of that option is not yet shown. Unified memory is shared by CPU and GPU, so it is not directly equivalent to dedicated VRAM, but the larger configurations can keep very large quantized models in memory. The wide range is important: Mac Studio is one product family, not one fixed capacity. The middle class is the ASUS Ascent GX10. Its NVIDIA GB10 Grace Blackwell Superchip uses 128 GB of coherent unified LPDDR5x memory. This gives a single developer far more model capacity than a 24 GB graphics card, although its 273 GB/s memory bandwidth is lower than the bandwidth of high-end discrete GPUs. Current Austrian retail listings start near €4,850 for the 1 TB system. ASUS originally announced a €3,499 recommended price, so availability and street pricing matter. The enterprise class uses RTX PRO 6000 Blackwell GPUs. Each GPU provides 96 GB of GDDR7 ECC memory. Austrian retail listings for the NVIDIA Workstation Edition currently start around €17,700 per card. The three cards in the picture would therefore represent more than €50,000 in GPUs alone. A workstation can contain several cards. Multiple GPUs add capacity and throughput, but one model does not automatically see them as one simple memory pool. The inference runtime must split or replicate the model, and interconnect bandwidth matters. All prices on the slide are rounded snapshots checked on 7 September 2026 and can change. Transition: move from physical machines to the model sizes and deployment classes they can support. Sources: https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-4000/ ; https://geizhals.at/nvidia-rtx-pro-4000-blackwell-900-5g147-2570-000-a3488409.html ; https://www.apple.com/at/newsroom/2026/08/apple-introduces-new-mac-studio-with-m5-max-and-m5-ultra/ ; https://www.apple.com/at/mac-studio/specs/ ; https://www.asus.com/networking-iot-servers/desktop-ai-supercomputer/ultra-small-ai-supercomputers/asus-ascent-gx10/ ; https://www.idealo.at/preisvergleich/OffersOfProduct/208466423_-ascent-gx10-asus.html ; https://www.nvidia.com/en-us/products/workstations/professional-desktop-gpus/rtx-pro-6000-family/ ; https://geizhals.at/nvidia-rtx-pro-6000-blackwell-workstation-edition-900-5g144-2500-000-a3476784.html Image sources: RTX PRO 4000 and RTX PRO 6000 images supplied by the presenter; https://www.apple.com/v/mac-studio/specs/b/images/specs/hero__d0yd117u6qy6_large_2x.jpg ; https://dlcdnwebimgs.asus.com/gain/c66958f8-57de-4563-b0db-bd317bfaf1f3/w800 Example configurations - not endorsements or product recommendations.

Model Size by Deployment Class

9

Return to the first prerequisite: how small can the model be and still do the work? The larger models establish the upper end. Point to the consumer-GPU column and ask which configurations fit there and which are worth qualifying on real coding tasks. Published closed frontier estimates, 27 October 2026. All open-model marker heights are calculated from log10(total main-model parameters in billions), with 1B at the bottom and 10T at the top. Main-model embeddings and lookup tables are included; separately reported auxiliary prediction heads are excluded. The Gemini Nano marker is a 2-4B range. Label offsets are relative to their marker, and x positions within a category simply keep similarly sized models distinct. There are no label connector lines. Visible model labels contain only names, parameter counts and essential availability status. Published closed-frontier estimates, checked 27 October 2026 and included at the user's explicit request. Visible labels contain approximate total parameters, the word Estimate and clickable AI BENCHY attribution. The publisher calls these low-confidence catalog estimates carried forward from earlier model families. They are not vendor disclosures or measured checkpoint sizes. The panel is unscaled; row position does not imply parameter count. No new estimate is calculated here. GPT-6.1 Sol: approximately 2T total parameters. AI BENCHY calls this a low-confidence catalog estimate carried forward from GPT-6 Sol. OpenAI has not disclosed the count; the linked announcement establishes identity and availability, not the parameter figure. Published estimate: https://aibenchy.com/model/openai-gpt-6-1-sol-low/ Claude Opus 5.5: approximately 5T total parameters. AI BENCHY calls this a low-confidence catalog estimate carried forward from the prior Opus 5 family. Anthropic has not disclosed the count; the linked announcement does not establish it. Published estimate: https://aibenchy.com/model/anthropic-claude-opus-5-5-low/ Gemini 4 Argon: approximately 1.2T total parameters. AI BENCHY calls this a provisional low-confidence catalog estimate carried forward from the Gemini Pro family. Google has not disclosed the count and describes Argon as a new model rather than a fine-tune of a prior model. The inherited estimate should not be interpreted as proof of a shared architecture. Published estimate: https://aibenchy.com/model/google-gemini-4-argon-low/ The three values above are the publisher's estimates, not our calculations. Only total parameters are shown. We do not adopt AI BENCHY's separate active-parameter or architecture guesses. Epoch continues to list parameters for these models as Unknown: https://epoch.ai/models/gpt-6-1-sol https://epoch.ai/models/claude-opus-5-5 https://epoch.ai/models/gemini-4-argon Grok 4.7: 2.1T, visibly labelled Musk Claim and linked to the original public statement of 28 July 2026. This is a pre-release founder statement, not an independently verified parameter estimate for the released checkpoint. Original rendered post inspected directly in the preceding source audit. Release announcement and developer guide do not specify a numeric count. Primary claim: https://x.com/elonmusk/status/2082123925283041545 Release: https://x.ai/news/grok-4-7 Developer guide: https://docs.x.ai/developers/grok-4-7 The old 5T / 10T / 6T / 3T benchmark-derived figures remain removed: they were our unvalidated extrapolations. The anpaure/frontier-model-sizes snapshot covers older versions and is not the source for the currently displayed estimates. Epoch's Grok 4.7 page currently displays 2.1e9, conflicting with the original founder's 2.1T claim; it is not numeric corroboration. Source provenance and limitations are retained under frontier_evidence in model-deployment-data.json. Selected memory configurations are documented here, rather than on the visible plot: Bolt four-bit capacity example; Gemma/Qwen Q4 and Super VL NVFP4 ~4-bit; Kolibri official FP8 ~8-bit; original Apertus and Soofi BF16 ~16-bit. These are example released/configured formats, not architectural requirements or a claim that other quantizations do not exist. Actual memory also includes quantization scales, higher-precision tensors, runtime, caches, vision components, optional draft models and the operating system. Hardware categories are representative capacity examples at the selected formats, not benchmarked guarantees for every device or full native context. Soofi retains Closed Beta; Kimi K3.1 Pro retains Weights Pending. Gemma 4 E2B: 5.1B includes per-layer embeddings; effective parameter count is 2.3B. Embedded deployment, not a claim of Chrome Prompt API support. Source: https://huggingface.co/google/gemma-4-E2B Gemini Nano: Chrome describes hardware-selected variants such as 2B and 4B; draw a range, not an exact midpoint model. Source: https://developer.chrome.com/docs/ai/understand-built-in-model-management Gemma 4 26B A4B: Replaces Gemma 4 12B on 27 October 2026. Google publishes 25.2B total language-model parameters / 3.8B active for the 26B-A4B version, plus a separate ~550M vision encoder. The main-model count determines the plotted height. Official QAT Q4_0 GGUF is available: 14.4 GB model plus 1.19 GB vision projector, about 15.6 GB combined before context/runtime. Consumer 24-32 GB remains a capacity example, with practical context constrained by remaining memory. Quantization source: https://huggingface.co/google/gemma-4-26B-A4B-it-qat-q4_0-gguf/tree/main . Source: https://huggingface.co/google/gemma-4-26B-A4B-it Qwen3.8: 27B language-model parameters. Four-bit weights approximately 13.5 GB. The 24-32 GB category is a capacity example with reduced practical context, not a promise of full 262K context. Source: https://huggingface.co/Qwen/Qwen3.8-27B Nemotron 3.5 Bolt: 74B total, 9B active. Moved to the workstation/server category at the user's request. The previous 2-bit IQ2 example is replaced by a four-bit capacity estimate: approximately 37 GB of raw weights before scales, cache, runtime, vision and optional draft model. This is an illustrative 64-192 GB workstation capacity placement, not a measured runtime. NVIDIA also publishes a highly compressed nominal 24 GB IQ2 variant with a documented roughly 28 GB runtime recipe for 32 GB or larger GPUs; that special configuration is not the deployment example shown here. Source: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Bolt-74B-A9B-24GB-GGUF Soofi S: Model architecture section reports 31.6B total, despite 30B family shorthand. Original checkpoint is gated closed beta; official files approximately 63.2 GB. BF16 capacity planning therefore belongs in the workstation/server group, with more than 64 GB once overhead is included. No unverified 4-bit distribution assumed. Source: https://huggingface.co/Soofi-Project/Soofi-S-Base Kolibri 1: 78,103,074,560 total parameters, 3,457,573,120 active. Official FP8 footprint about 78 GB. Provider minimum includes H200/B200/B300 or multiple 80 GB accelerators. This is not a claim of verified 128 GB GB10 support. Source: https://huggingface.co/Aleph-Alpha/Kolibri-1 Nemotron 3.5 Super VL: Provider reports 120B total, 12B active. Official deployment includes DGX Spark. Mixed-precision NVFP4, not all tensors at four bits. Source: https://huggingface.co/nvidia/NVIDIA-Nemotron-3.5-Super-VL-120B-A12B-NVFP4 Qwen3.8 Flash Next: Main model: 125B language backbone plus 51B n-gram embeddings = 176B. Optional 4B MTP adds 4B to checkpoint accounting (180B including MTP). Plot excludes auxiliary prediction heads consistently with the main-model convention. Four-bit weight estimate about 88 GB before runtime; practical deployment needs shortened/quantized context. Source: https://huggingface.co/Qwen/Qwen3.8-Flash-Next Apertus 1.5: Swiss AI Initiative (ETH Zurich, EPFL and CSCS), Switzerland. Current 1.5 release of the 70B family, with text/image/audio input and up to 262,144 context tokens. The official Hub reports approximately 72B BF16/F32 stored parameters after expanding input embeddings for multimodality; plot the rounded stored main-model count (72B), including embeddings, rather than treating the 70B family name as an exact count. No separate auxiliary prediction model is assumed. BF16 weights alone are about 144 GB, requiring the upper end of the 64-192 GB category or multiple accelerators; context and runtime add memory. Original weights require acceptance of published access conditions; this is not Soofi-style closed beta. Main-model source and parameter metadata checked 27 October 2026. Source: https://huggingface.co/swiss-ai/Apertus-v1.5-70B DeepSeek V4.1 Flash-1017: Latest released update verified 27 October 2026: Flash-1017, a minor update to V4.1 Flash. Base card specifies 552B backbone and a separate 196B Engram conditional-memory table. The plotted approximate main-model-plus-lookup count is 748B = 552+196; excluding auxiliary DSpark/MTP and separate vision components. This matches the chart convention used for Qwen n-gram embeddings. Do not plot only 552B while calling it total main model including embeddings. Official repository has weights and MIT license; full FP8/mixed distribution has ~763B stored tensors including auxiliary components. Sources: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash ; https://api-docs.deepseek.com/news/news261017/ . Source: https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash-1017 GLM-5.3: GLM-5.3 card says same base as GLM-5.2. The GLM-5 architecture report / official GLM-5 card gives 744B main parameters and 40B active; 753/754B Hub tensor counts include auxiliary MTP and accounting differences. Plot uses published main architecture count, excluding auxiliary prediction heads. Sources: https://huggingface.co/zai-org/GLM-5 ; https://huggingface.co/zai-org/GLM-5.2 ; https://arxiv.org/abs/2602.15763 . Public config files for GLM-5, GLM-5.2 and GLM-5.3-BF16 were compared on 27 October 2026: parameter-bearing architecture fields match. See qa/slide8-models/glm-architecture-check.json. Latest official Z.ai model list rechecked 27 October 2026: GLM-5.3 remains current flagship; Flash is a separate smaller newly trained model, also shown. Source: https://huggingface.co/zai-org/GLM-5.3 GLM-5.3 Flash: Current GLM-5.3 family member verified 27 October 2026. Official card gives 320B total, 18B active, native multimodal support and a newly trained base; it is not merely a precision conversion of the 744B GLM-5.3 backbone. Released weights and documented inference integrations are available. Added as another datacenter-scale example. Source: https://huggingface.co/zai-org/GLM-5.3-Flash Mistral Large 4 (Le Chonk): Mistral reports approximately 1T total parameters and 52B active including embeddings and output layers. Open weights were released on 29 October 2026, and Mistral also publishes an NVFP4 checkpoint. The chart places the released model in the multi-accelerator Datacenter category using its reported total parameter count. This placement is a deployment example. Model card: https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B Release: https://mistral.ai/news/mistral-large-4-oss/ NVFP4 checkpoint: https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVFP4 Kimi K3.1 Pro: Latest Kimi release verified 27 October 2026. Official blog calls K3.1 Pro a post-trained version of K3 and a 3T-class model. The approximate 2.8T position is inferred from its published K3 backbone, not a newly released K3.1 checkpoint tensor count. K3.1 Pro and Highspeed are available in apps and API; weights are promised by 4 November and are not yet downloadable as of this check. Dashed dot and visible Weights Pending label distinguish intended datacenter capacity from a currently runnable local model. Do not describe current API access as local availability. K3 reference: https://huggingface.co/moonshotai/Kimi-K3 . Source: https://www.kimi.com/en/blog/kimi-k3-1-pro Laguna XS.2: Poolside's Laguna M.1/XS.2 technical report gives 33.4B total parameters and 3B activated per token. The visible name is Laguna XS.2, as requested, not the later Laguna XS 2.1. Apache 2.0 open weights; released official INT4 and NVFP4 variants. The official INT4 repository is approximately 23.9 GB because it mixes INT4 and INT8 experts and other tensors; do not use the ideal 16.7 GB four-bit arithmetic as its actual footprint. Place it at the 32 GB end of the 24-32 GB consumer category with constrained practical context, not a guarantee of full 256K context or a fit on a 24 GB GPU. File source: https://huggingface.co/poolside/Laguna-XS.2-INT4/tree/main . Base card independently describes local deployment on a Mac with 36 GB RAM. MoE active parameters affect per-token computation; all expert weights still count toward resident model memory. Sources checked 27 October 2026. Technical report: https://arxiv.org/abs/2605.27605 Official model: https://huggingface.co/poolside/Laguna-XS.2 Official INT4: https://huggingface.co/poolside/Laguna-XS.2-INT4 Layout update: Bolt moved right into workstation/server. The workstation column is wider to accommodate six models; numeric marker heights remain calculated from the same log parameter scale.

Gemma 4 Family

DenseSame path for every tokenMixture of ExpertsRouter selects experts

Total Parameters 30B 20B 10B 0

E2B5.1B totalDense
2.3B effective

E4B8B totalDense
4.5B effective

12B11.95B totalDense

26B-A4B25.2B totalMoE
3.8B active

31B30.7B totalDense

ECore size, excluding large embedding lookup tables

AParameters selected by the router for each token

10

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. Gemma 4 is a family, not one model. The vertical scale is linear and represents total parameter count: a 5.1B bar is roughly one sixth the height of a 30.7B bar. Blue and green describe architecture. Blue models are dense, meaning that every token follows the same network path. Green marks the mixture-of-experts model, where a router selects only some experts for each token. Effective and active sound similar, but they describe different mechanisms. In E2B and E4B, E means effective. These dense models contain large per-layer embedding tables. The tables must be stored, but each token only performs quick lookups in them. Google therefore describes the computational core as 2.3B or 4.5B effective parameters, while the complete stored models contain 5.1B or 8B parameters. In 26B-A4B, A means active. The complete mixture-of-experts model contains 25.2B parameters. For each token, its router selects eight experts plus one shared expert from 128 experts in each MoE layer, so about 3.8B parameters participate in the computation. All 25.2B parameters still have to fit in memory for fast inference. The 12B and 31B variants are dense models with 11.95B and 30.7B parameters. All five are multimodal. The E2B and E4B models support 128K context; the 12B, 26B-A4B and 31B models support 256K. Parameter count is only one dimension: quantization, context size and architecture determine the actual memory requirement and speed. Source: Google Gemma 4 model card, https://ai.google.dev/gemma/docs/core/model_card_4 . Transition: compare that range with the much smaller Qwen 3.6 family.

Memory Provisioning

RecommendedFully Served by VRAM

Sizing Criteria

Model Size
Quantization
KV Cache
Context Size
Operational Memory

11

The recommended target is to keep the complete inference workload in GPU memory. This avoids moving model data between storage, system RAM and VRAM while generating tokens. Start with the model size. Quantization determines how many bytes the stored weights require, plus scales and metadata. The KV cache needs additional memory and grows with the context size. Its exact size also depends on the model architecture, cache precision and batch size. Operational memory covers the inference engine, temporary compute buffers and memory allocator overhead. Leave headroom rather than planning exactly to the advertised VRAM capacity. On systems with unified memory, such as Apple silicon, CPU and GPU share one memory pool, but bandwidth, allocation limits and the operating system still reduce the amount available to the model. Transition: quantization changes the largest part of this calculation, the weight memory.

Quantization Principle

0:00.0 / 0:06.4

Original Range0–100

0100

18

47

83

÷ 10, then round

Quantized Range0–10

182

475

838

Reconstructed× 10

20Δ +2

50Δ +3

80Δ −3

12

Presenter controls: use Next to pause at the three AI DevCraft teaching checkpoints: highlight the original values, animate their mapping to 2/5/8, then reveal reconstruction and quantization error. Previous moves back through these checkpoints. Double arrows skip slides. Start with a deliberately simple example. The original values can occupy many positions between 0 and 100. We choose a scale of 10, divide each value by that scale and round to the nearest integer. That maps 18 to 2, 47 to 5 and 83 to 8. We now store one of only eleven integer values instead of a much finer original number. To reconstruct approximate original values, multiply the stored values by the same scale of 10. The results are 20, 50 and 80. Compared with 18, 47 and 83, the signed differences are +2, +3 and −3. This difference is quantization error: information discarded during rounding cannot be recovered exactly. Real model quantization can use different scales and zero points for tensors, channels or groups. Some methods also calibrate against data or treat important weights differently. The animation explains the basic idea, not a specific file format. Transition: translate the number of bits into model weight memory.

Choosing a Quantization Level

Qualityhigher fidelity

Memory and File Sizesmallerlarger

Local Sweet Spot

emergency fit

tight memory

default start

conservative

near reference

reference size

Practical RuleStart at Q4. Move up if quality breaks.
Move down only when the model does not fit.

13

Read this as a practical direction, not a universal benchmark curve. Moving from Q2 toward Q8 or BF16 usually increases file size and memory use while preserving more of the reference model's behavior. The exact quality difference depends on the model, the quantization algorithm, the dataset used for calibration, the runtime and the task. Labels such as Q4 or Q5 are families of formats rather than one standard method. A good local starting point is often a well-supported Q4 quantization. Move to Q5 or Q6 when the model fits comfortably and your evaluation finds meaningful quality loss. Move below Q4 only when memory capacity prevents the model from running, because aggressive quantization can damage reasoning and coding reliability. Q8 often stays close to the reference behavior but saves much less memory than Q4. BF16 represents the high-precision reference here. Always qualify the exact model file, not the label alone. Transition: video codecs show how compression methods can improve across generations.

Video Compression Over Time

The same mountain video frame — 1993MPEG-1≈ 600 MB

14

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. The repeated image is deliberate. We compare how much data is needed for roughly the same perceived result, rather than showing a sequence of better pictures. The file sizes form an illustrative one-minute 1080p example. They use a simple bitrate scale of 80 megabits per second for MPEG-1, 40 for MPEG-2, 20 for H.264, 10 for HEVC, 8 for AV1 and 5 for VVC. File size equals bitrate multiplied by sixty seconds and divided by eight. That gives about 600, 300, 150, 75, 60 and 37.5 megabytes. These are not measurements from one universal benchmark. MPEG-1 was designed for Video CD-era CIF or SIF resolution, so its 1080p figure is a conceptual extrapolation. The later values make the documented generational improvements easy to see. H.264 can deliver comparable quality at roughly half the bitrate of MPEG-2. HEVC targets another reduction of about half. AV1 arrived in 2018 as an open, royalty-free codec and practical tests show substantial savings over H.264 and VP9. VVC targets about half the bitrate of HEVC. Actual sizes vary with the content, encoder implementation, profile, settings and quality target. Newer codecs can also require more computation. This is the analogy to model quantization: better algorithms can preserve more useful quality within a smaller representation. It does not prove that every newer quantization method or every quantized model is better. We still have to test the exact artifact on our tasks. Sources: MPEG-1 Video, published in 1993, https://mpeg.chiariglione.org/standards/mpeg-1/video.html; ITU-T H.262, July 1995, https://www.itu.int/rec/T-REC-H.262-199507-S/en; ITU-T H.264, May 2003, https://www.itu.int/rec/T-REC-H.264-200305-P; ITU HEVC announcement, January 2013, https://www.itu.int/net/pressoffice/press_releases/2013/01.aspx; AOMedia AV1 specification release, March 2018, https://aomedia.org/press%20releases/the-alliance-for-open-media-kickstarts-video-innovation-era-with-av1-release/; Meta AV1 comparison, April 2018, https://engineering.fb.com/2018/04/10/video-engineering/av1-beats-x264-and-libvpx-vp9-in-practical-use-case/; Fraunhofer HHI VVC overview, finalized July 2020, https://www.hhi.fraunhofer.de/en/departments/vca/technologies-and-solutions/h266-vvc/vvc-overview.html. Transition: next, see how inference uses compute and memory during prefill and decoding. Illustrative file size for one minute of the same 1080p source at comparable visual quality.

Prefill and Decoding without KV Cache

0:00.0 / 0:18.5

Prefill

Compute-Bound

Decode

Recomputes the Entire Context

User Input
Tool Response
Thinking

Pleasebuildatestedcomponent

withourexistingsignalstore

andaddthebackendintegration

Model Layers

Predicted Token

returnthefinalresult

returnthefinal

Inputreturnthefinal

15

Presenter controls: Next plays to the next teaching checkpoint and pauses after each generated token. Previous returns to the preceding checkpoint. Double arrows skip slides. These are the original AI DevCraft workshop checkpoints. This first animation deliberately assumes that no KV cache exists. During prefill, the complete input passes through every transformer layer and the model predicts the first new token. The generated token then joins the input. Without cached keys and values, the next prediction requires another forward pass over the complete sequence. The animation moves the full, growing sequence through one layer at a time. Only after the complete sequence has passed through all layers can the model produce the next token. The model parameters do not move into the model with every step. They remain loaded in memory and the computation reads them again for each pass. Reprocessing the whole context makes uncached generation increasingly expensive as the sequence grows. Transition: the KV cache avoids most of that repeated work.

KV Cache During Inference

0:00.0 / 0:12.5

Prefill

Builds the Cache

Decode

Reuses the Cache

User Input
Tool Response
Thinking

KV Cache

Pleasebuildatestedcomponent

withourexistingsignalstore

andaddthebackendintegration

Model Layers

Predicted Token

returnthefinalresult

returnthefinal

Inputreturnthefinal

16

Presenter controls: Next plays to the next teaching checkpoint and pauses after each generated token. Previous returns to the preceding checkpoint. Double arrows skip slides. These are the original AI DevCraft workshop checkpoints. KV means key-value. The border is a conceptual aid: the cache does not store these token labels. During prefill, every transformer layer computes key and value tensors for the input tokens and stores them in the KV cache. During decode, the model adds key and value tensors for each new token. It reuses the cached tensors instead of recomputing the complete preceding sequence for every prediction. That reuse makes autoregressive decoding practical. The cache still grows with the token count, the number of layers, the attention shape, the batch size and the cache precision. It can consume a substantial amount of VRAM at long context sizes. Transition: apply these components to two concrete Gemma 4 memory plans.

Gemma 4 26B-A4B Memory Provisioning

Memory in VRAM / Unified Memory (GiB)

061218243036

16.72 GiB

13.27 GiB Model

1.45 GiB KV

2 GiB Runtime

64K

17.97 GiB

13.27 GiB Model

2.70 GiB KV

2 GiB Runtime

128K

20.47 GiB

13.27 GiB Model

5.20 GiB KV

2 GiB Runtime

256K

Context Size

17

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. This is a planning estimate for the quantized Gemma 4 26B-A4B artifact shown in the reference. The weight file remains constant at 13.27 GiB for every context size. A4B means that about 3.8 billion of roughly 25.2 billion total parameters are active for a token. That can reduce compute, but it does not turn the model into a 4B memory footprint. All expert weights still have to reside in memory. The F16 KV cache grows from 1.45 GiB at 64K to 2.70 GiB at 128K and 5.20 GiB at 256K. The runtime estimate is a range of one to two GiB. The totals shown here use the conservative two GiB endpoint: 16.72, 17.97 and 20.47 GiB. These numbers plan capacity; the actual allocation depends on the runtime, backend, batch settings and model file. Read the startup logs before treating the fit as guaranteed. Transition: compare this mixture-of-experts model with the dense Qwen 3.8 27B. All 25.2B parameters must fit in memory. Totals use a 2 GiB runtime reserve. IT QAT UD-Q4_K_XL by unsloth · 3.8B active / 25.2B total · one inference slot · F16 KV cache The two memory charts share the same 0-36 GiB scale. The source memory calculations are unchanged.

Qwen 3.8 27B Memory Provisioning

Memory in VRAM / Unified Memory (GiB)

061218243036

22.39 GiB

16.39 GiB Model

4.00 GiB KV

2 GiB Runtime

64K

26.39 GiB

16.39 GiB Model

8.00 GiB KV

2 GiB Runtime

128K

34.39 GiB

16.39 GiB Model

16.00 GiB KV

2 GiB Runtime

256K

Context Size

18

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. This planning estimate uses the 17.6 GB decimal UD-Q4_K_XL artifact published by unsloth, which is 16.39 GiB. Qwen 3.8 27B has 64 layers, with 16 full-attention layers, four KV heads and a head dimension of 256. With F16 keys and values, the growing KV cache is 4 GiB at 64K, 8 GiB at 128K and 16 GiB at the native 256K context. Adding a two GiB runtime reserve produces planning totals of 22.39, 26.39 and 34.39 GiB. The vision projector is not included. Actual llama.cpp allocation, state checkpoints and batch settings can require more headroom. Sources: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF/blob/main/Qwen3.8-27B-UD-Q4_K_XL.gguf ; https://huggingface.co/Qwen/Qwen3.8-27B ; https://huggingface.co/Qwen/Qwen3.8-27B/blob/main/config.json . Transition: hardware capacity is only one part of the decision. The next chapter asks how we control quality. Hybrid architecture: 16 of 64 layers grow the KV cache. Totals use a 2 GiB runtime reserve. UD-Q4_K_XL by unsloth · 27B dense · 16 KV layers · one inference slot · F16 KV cache The two memory charts share the same 0-36 GiB scale. The source memory calculations are unchanged.

Quality vs. Data Sovereignty

Data Sovereignty Quality

Local Models

High Quality

Data Sovereignty

Frontier Models

Peak Quality

No Full Data Sovereignty

19

Restored from the 11 September 2026 Graz deck. Model and hardware examples, prices and planning estimates retain that dated source context. Treat these as two qualitative positions, not measured data points. Local models can deliver high-quality software work while keeping model execution and data inside infrastructure the organization controls. Frontier models still provide peak capability, but closed provider-hosted models do not provide full data sovereignty under this definition. Contracts, regional hosting and technical controls can improve privacy and governance, but the organization does not control the complete inference stack. The bubbles deliberately avoid precise coordinates because individual models, tasks and deployment choices vary. Treat speed as a separate question. Local inference speed mainly depends on hardware capacity and budget. With sufficiently powerful accelerators and a comparable model configuration, a local system can match the latency or throughput of cloud inference. Transition: use the Porsche analogy to ask whether every software task needs peak model capability. Speed is a hardware budget question. Powerful local hardware can match cloud inference speed. Use the shared arrows: reveal Local, then Frontier. Previous reverses; leaving resets the reveals.

A high-performance sports car caught in traffic and roadworks

The Porsche Test

Do We Need Peak Capability for Every Trip?

20

Benchmark philosophy

Qualification Instead of Ranking

Qualification threshold

Model A

Model B

Model C

Qualification criteria

✓Works with coding agents

✓Handles long-running tasks

✓Supports the required context

21

Full-Stack Capstone Task

Angular

Feature component
UI component
SignalStore
Unit test

The capstone is deliberately broad and delivered as one prompt. On the Angular side, the model must implement the feature according to existing architecture, separate feature and UI responsibilities, use a SignalStore, and add tests. On the Spring side, it must add the JPA model, mapper, controller and database migration. Finally, it must add a Playwright end-to-end test. This is difficult because success requires repository discovery, architectural consistency, cross-boundary changes and verification—not because one file is exotic. Logo sources: https://angular.dev/press-kit, https://simpleicons.org/?q=spring, https://playwright.dev/img/playwright-logo.svg. In this benchmark, the concrete feature is a German vocabulary trainer added to the existing holidays application. Use this slide to explain the scope before showing the benchmark components, validation and review loop.

Spring

JPA entities
Mapper
Controller
Database migration

The capstone is deliberately broad and delivered as one prompt. On the Angular side, the model must implement the feature according to existing architecture, separate feature and UI responsibilities, use a SignalStore, and add tests. On the Spring side, it must add the JPA model, mapper, controller and database migration. Finally, it must add a Playwright end-to-end test. This is difficult because success requires repository discovery, architectural consistency, cross-boundary changes and verification—not because one file is exotic. Logo sources: https://angular.dev/press-kit, https://simpleicons.org/?q=spring, https://playwright.dev/img/playwright-logo.svg. In this benchmark, the concrete feature is a German vocabulary trainer added to the existing holidays application. Use this slide to explain the scope before showing the benchmark components, validation and review loop.

Playwright

End-to-end test

The capstone is deliberately broad and delivered as one prompt. On the Angular side, the model must implement the feature according to existing architecture, separate feature and UI responsibilities, use a SignalStore, and add tests. On the Spring side, it must add the JPA model, mapper, controller and database migration. Finally, it must add a Playwright end-to-end test. This is difficult because success requires repository discovery, architectural consistency, cross-boundary changes and verification—not because one file is exotic. Logo sources: https://angular.dev/press-kit, https://simpleicons.org/?q=spring, https://playwright.dev/img/playwright-logo.svg. In this benchmark, the concrete feature is a German vocabulary trainer added to the existing holidays application. Use this slide to explain the scope before showing the benchmark components, validation and review loop.

22

The capstone is deliberately broad and delivered as one prompt. On the Angular side, the model must implement the feature according to existing architecture, separate feature and UI responsibilities, use a SignalStore, and add tests. On the Spring side, it must add the JPA model, mapper, controller and database migration. Finally, it must add a Playwright end-to-end test. This is difficult because success requires repository discovery, architectural consistency, cross-boundary changes and verification—not because one file is exotic. Logo sources: https://angular.dev/press-kit, https://simpleicons.org/?q=spring, https://playwright.dev/img/playwright-logo.svg. In this benchmark, the concrete feature is a German vocabulary trainer added to the existing holidays application. Use this slide to explain the scope before showing the benchmark components, validation and review loop.

Benchmark Components

Security: NVIDIA OpenShell
Implementation: Pi coding agent
Inference engine: llama.cpp
Hardware: GX10
Tracing: Langfuse
Versioning: Git worktrees

23

Use case: software development. The task adds a German vocabulary trainer to an existing holidays application, following its quiz architecture. The same five words are seeded for Lübeck and Vienna. The user opens an accessible Vocabulary Test link and answers questions; counts update on blur. This is one substantial full-stack feature, not a general intelligence benchmark. Use the diagram from left to right: fixed task and Git baseline, a dedicated worktree per sample, the Pi agent executing in OpenShell, and model inference on the GX10 through https://inference.local/v1. The configured OpenShell vm driver uses libkrun. PostgreSQL and Chromium run inside the sandbox; no host Docker socket is shared. The harness pins and checks model identity around sessions. Each model's concrete quantization is part of the tested setup, not normalized away. Current macOS pilot has limitations: Landlock and inner cgroup PID enforcement are unavailable on this driver. Do not promise production-grade hardening. Policy-denied runs are inconclusive infrastructure/policy outcomes, not model-quality failures. Official OpenShell source: https://github.com/NVIDIA/OpenShell Sources: local-models-for-coding-benchmark-langfuse/README.md; openshell/README.md; src/fresh-review-loop.ts. Current report: local-models-for-coding-benchmark/results/index.html. Audit records: results/outer-assessments.json. Snapshot inspected 13 October 2026. Component list: click Next to reveal one row at a time (six steps); Previous hides the last revealed row. No animated transition. Logos use the official project marks; GX10 uses the ASUS manufacturer wordmark. Sources, checked 13 October 2026: openshell: https://raw.githubusercontent.com/NVIDIA/OpenShell/main/docs/brand/assets/openshell-mark.svg pi: https://pi.dev/press-kit (embedded primary logo SVG) llama: https://raw.githubusercontent.com/ggml-org/llama.brand/master/icon/icon-light.svg langfuse: https://langfuse.com/brand-assets/icon/color/langfuse-icon.svg git: https://raw.githubusercontent.com/git/git-scm.com/gh-pages/static/images/logos/downloads/Git-Icon-1788C.svg asus: https://www.asus.com/networking-iot-servers/desktop-ai-supercomputer/desktop-ai-supercomputer/asus-ascent-gx10/ (embedded ASUS wordmark)

Worker, Verifier and Reviewer

🤖WRITE

Worker

Implements the task
Edits the worktree

EXECUTE

Validation

Builds · unit tests
Backend · Playwright

🤖READ ONLY

Reviewer

Does the patch meet
the requirements?

Done

Save result

🤖READ ONLY

Verifier

Explains failures

24

Worker writes; validation executes; verifier diagnoses failed checks; reviewer judges whether the successful patch meets the task. Validation includes frontend dev and production builds, frontend tests, backend tests and Playwright. Worker-triggered validation can help development, but the harness runs its authoritative checks separately. The verifier and reviewer receive fresh, read-only sessions. Their model is the same candidate in this cohort; separate roles are not independent external judges. A rejected review or diagnosis goes to a fresh worker session with a bounded handoff and the same sample's worktree. Each review round allows three worker attempts in total: the initial attempt plus at most two repairs. There are at most three review rounds. Per-session generated-output budgets: worker 250,000; verifier 50,000; reviewer 50,000 tokens. Time watchdogs are emergency limits. Completion can also mean exhausted budget or failed checks: all such outcomes are saved. External audit is outside this loop and is shown separately in the report. Sources: local-models-for-coding-benchmark-langfuse/README.md; openshell/README.md; src/fresh-review-loop.ts. Current report: local-models-for-coding-benchmark/results/index.html. Audit records: results/outer-assessments.json. Snapshot inspected 13 October 2026. Flow animation: the full diagram is visible immediately. Next/click/right arrow moves a pulse along one connection: Worker → Validation → (fail) Verifier → Worker → Validation → (pass) Reviewer → (decline) Worker → Validation → (pass) Reviewer → (accept) Done. Nine clicks complete the sequence. Previous rewinds one transition; leaving and re-entering resets it. Reduced-motion mode uses destination highlighting without movement. Stopping rules (configured for these benchmark runs): +- If deterministic validation still fails after the third worker attempt within one review round, stop the run as unsuccessful (repair_attempts_exhausted). The verifier diagnoses the failed checks; it does not itself produce the deterministic pass/fail verdict. This is three total attempts, not three retries after the initial attempt. +- If the reviewer declines in the third review round, stop the run as unsuccessful (review_rounds_exhausted). An earlier decline starts a new round with its worker-attempt counter reset to one. +- The maximum is therefore nine worker attempts across three review rounds, provided validation eventually passes within each round. Exhausting repairs in any round stops the run immediately. +- Keep the two outcomes distinct in the report: a review-exhausted run can still have passing validation. Infrastructure or policy interruptions and inconclusive runs are recorded separately; do not describe them as model failures. Diagram colors: blue boxes are the three AI-agent roles; the warm-colored Validation box runs deterministic checks. Green Done represents reviewer acceptance in the illustrated successful path. Limit sources: src/fresh-review-loop.ts (nextAfterVerification / nextAfterReview), src/openshell-executor.ts (initial position and control flow), and results/qwen38-token-v1-20261003-a1-live-backend-r1/config.env (MAX_REPAIR_ATTEMPTS=3, MAX_REVIEW_ROUNDS=3). Presenter reminder: the worker, verifier and reviewer in this cohort use the same candidate model. Separate sessions and roles do not create an independent external judge. The external audit is a separate evidence stage.

Benchmark Results

Spring Boot + Angular · Measured on ASUS GX10

Model	Weights	Quantization	Time (h:mm)	Eff. Tokens/s	Validation	Review
Nemotron 3.5 Bolt 74B	22.0 GB	IQ2	2:39	29.8	3/5	1/52 incomplete
Ternary Bonsai 2 27B	5.9 GB	PTQ1_0	4:06	16.4*	4/4	4/4
GLM-5.3 Flash	97.6 GB	IQ1_M	1:28	13.4	5/5	5/5
Muse Glimmer 30B	15.9 GB	Q4_K_XL	3:00	12.1	2/5	2/5
Qwen3.8 27B	17.6 GB	Q4_K_XL	2:58	8.6	5/5	5/5
Gemma 4 26B-A4B QAT	14.2 GB	Q4_K_XL	0:46	52.5	2/4	1/41 incomplete
Gemma 4 31B QAT	17.3 GB	Q4_K_XL	1:44	9.6	1/3	1/3
DeepSeek V4 Flash	86.9 GB	IQ1_M	1:59	15.0	3/4	3/4
Qwen3.6 27B	17.6 GB	Q4_K_XL	3:05	9.3	5/5	5/5
Devstral Small 2 24B	14.5 GB	Q4_K_XL	5:23	9.3*	1/2	1/2
Qwen3-Coder 30B-A3B	17.7 GB	Q4_K_XL	0:34	36.9	0/5	0/5

47 runs · Medians · * Partial timing

25

Snapshot as of 1 November 2026. The cohort includes 44 runs from the benchmark report and three separately documented Gemma 4 31B QAT runs. All use the same Spring Boot and Angular benchmark, the session-output-token-v1 protocol, and baseline 31599ef4e9cb5423117b37a6ec426773d6222526. The task and its checks cover the full stack. Validation requires passing all five automated checks. Review acceptance means the same model accepted the result in a fresh session. Each denominator includes every selected run for that configuration. Only completed reviews with an acceptance count toward the review total. Lightning had no eligible runs. Time is the median wall-clock duration, including tools and checks. Failed runs may finish earlier. Throughput is the median effective output tokens per second measured on the ASUS GX10. Client-observed model time includes waiting, prompt processing, and context compaction. Tool execution time is excluded. An asterisk indicates at least one run with partial timing. Source report: local-models-for-coding-benchmark/results/index.html Added runs: glm53-iq1m-token-v1-20261028-s9-live-backend-r1, nemotron35-bolt-token-v1-20261029-s14-live-backend-r1, nemotron35-bolt-token-v1-20261029-s16-live-backend-r1, nemotron35-bolt-token-v1-20261029-s6-live-backend-r1. The exact run selection and hardware specifications are in assets/ijs/w-jax-benchmark-data.json.

Tokens/s on Other Hardware

Model

GX10 Measurement × Memory Bandwidth ÷ 273 GB/s

Device	Memory	Bandwidth	Eff. Tokens/s
ASUS GX10	128 GB	273 GB/s	—
NVIDIA DGX Spark	128 GB	273 GB/s	—
MacBook Pro M5 Max	128 GB	614 GB/s	—
Mac Studio M5 Ultra	256 GB	1,200 GB/s	—
GeForce RTX 5090	32 GB	1,792 GB/s	—
RTX PRO 6000 Blackwell	96 GB	1,792 GB/s	—

Blue = measured on GX10 · ≈ = bandwidth estimate

Estimates assume model weights, KV cache, and runtime buffers all fit in available GPU memory.

26

The estimator uses the benchmark report’s hardware profiles and formula. It multiplies effective output tokens/s measured on the GX10 by the target’s advertised memory bandwidth, then divides by 273 GB/s. It assumes one active request, the same quantization and context length, full GPU residency, and equal relative bandwidth utilization. The estimate excludes prefill compute, CUDA or Metal kernel differences, thermals, and competing workloads. The memory check rules out configurations whose weights alone exceed advertised capacity. Actual fit also depends on the KV cache, runtime buffers, and operating system. Devices with equal bandwidth can still differ in real throughput. An asterisk indicates at least one run with partial timing. The default example is Qwen3.8 27B. In our benchmark, its Q4_K_XL weights occupied 17.6 GB and the selected configuration passed validation and same-model review in all five runs. Its median rate was 8.6 effective output tokens/s on the ASUS GX10. Use it as an example of a smaller configuration that passed our task. Fit on a consumer GPU depends on the context and available memory. Device names link to manufacturer specifications. Use the model selector in the HTML deck to compare other models.

pi-kit

For Local Models

Pi + AG-UI

Meat Proxy Detector
Self-Learning
Background Agents

pi-kit.dev

pi-kit desktop UI with a coding conversation, local Gemma model controls and an integrated Artifacts browser

27

Introduce pi-kit as our desktop coding harness built on the Pi coding agent, with a focus on working with local models. It also supports hosted providers. The interface uses AG-UI and the Angular port of CopilotKit for tool interfaces, input requests and MCP Apps. Verified from the public website on 26 October 2026: local GGUF model setup through Hugging Face and llama-swap; durable background agents; session branching; Git worktrees and review; Artifacts browser; skill drafting and verification. Self-learning creates reusable instructions, not new model weights. Proposed skills are reviewed before installation; replay verification is optional. The product screenshot is captured from the homepage's real pi-kit desktop UI figure. It contains a seeded demo conversation and a separate local demo project, as identified by its original caption; it is not a benchmark result. Sources: https://pi-kit.dev/ ; https://pi-kit.dev/docs/learning ; https://pi-kit.dev/docs/settings . The website describes a beta desktop client. Rainer explicitly supplied the feature name Meat Proxy Detector on 26 October 2026. That name is included as user-provided product information; the inspected public pages do not describe its behavior. The screenshot and pi-kit.dev text are clickable links. This slide sits immediately before the existing Questions closing slide.

Questions?

Share Your Feedback

soverius.ai/feedback/w-jax-2026-local-models

Once a model reliably handles the task, more speed is mainly a question of cost.

28

Java-Entwicklung mit lokalen AI-ModellenWie weit kommt man?

Reasons to Run Models Locally

Cost Control

Privacy

Independence

Find the Smallest Model That Can Do the Job

Review the Generated Code

Engineer the Loop

Typical Required Components

CPU and GPU Inference

Local Inference Hardware

RTX PRO 4000

Mac Studio

ASUS Ascent GX10

RTX PRO 6000 Blackwell

Model Size by Deployment Class

Gemma 4 Family

Memory Provisioning

Sizing Criteria

Quantization Principle

Choosing a Quantization Level

Video Compression Over Time

Prefill and Decoding without KV Cache

Prefill

Decode

KV Cache During Inference

Prefill

Decode

Gemma 4 26B-A4B Memory Provisioning

Qwen 3.8 27B Memory Provisioning

Quality vs. Data Sovereignty

High Quality

Peak Quality

Do We Need Peak Capability for Every Trip?

Qualification Instead of Ranking

Full-Stack Capstone Task

Angular

Spring

Playwright

Worker

Validation

Reviewer

Done

Verifier

For Local Models

Share Your Feedback