AI Workstation & GPU Performance Report (2026)

• CUDA + cuBLAS (default): All major frameworks use this. Best throughput for models that fully fit in VRAM. Zero additional configuration in Ollama or LM Studio.
• TensorRT-LLM: NVIDIA's production inference engine. Requires model compilation; delivers up to 2× the throughput of vanilla llama.cpp for continuous batching workloads.
• FlashAttention-2: Reduces attention memory complexity from O(n²) to O(n). Enabled by default in vLLM and llama.cpp — provides 1.5–2× speedup on long contexts.
• Multi-GPU NVLink: Pro cards (A100, H100, RTX 6000 Ada) support NVLink for full bandwidth pooling. Consumer cards do not — PCIE bandwidth becomes the bottleneck with 2× GPUs.
• GPU offload layers (-ngl): In llama.cpp, set -ngl 99 to send all transformer layers to GPU. If VRAM is too small, reduce layers incrementally; each layer in VRAM improves speed.
• FP8 quantisation (Hopper GPUs): H100/H200 support FP8 native precision — 2× throughput over BF16 with <1% quality loss. Not available on consumer RTX cards.

• MLX framework (Apple-native): 20–30% faster than llama.cpp with Ollama for inference on Apple Silicon. Install via pip install mlx-lm. Best choice for Apple hardware in 2025.
• Metal Performance Shaders (MPS): Used by PyTorch on Apple Silicon. Not as optimised as MLX for LLM inference but works for fine-tuning and training experiments.
• Speculative decoding (MLX): Use a small draft model (e.g. 1B) to propose tokens, verified by the main model. Can nearly double tokens/sec on large models with no quality change.
• llama.cpp Metal backend: Used by Ollama. Slower than native MLX but has broader model format support. Set LLAMA_METAL=1 at compile time.
• Unified memory — use it all: Unlike a GPU, macOS can dynamically share RAM between GPU and CPU. Run models that are 70–80% of total RAM; leave ~20% for macOS and applications.
• Context length and memory: Increasing context window (e.g. 8K → 128K) consumes significant KV-cache RAM. At 128K context a 70B model can use >60 GB extra RAM. Set --ctx-size conservatively.

• ROCm (HIP): AMD's CUDA-equivalent. Now supported in llama.cpp, vLLM, and PyTorch. Requires ROCm 6.0+ on compatible RX 6000/7000 series GPUs. Performance is within ~10% of CUDA on inference.
• Vulkan backend (llama.cpp): Cross-platform GPU acceleration using Vulkan compute shaders. Often 10–20% faster than ROCm on discrete GPUs for pure inference. Use -DGGML_VULKAN=ON at build time.
• Ryzen AI Max+ 395 — BIOS RAM allocation: By default, the NPU/GPU shares 32 GB. Open BIOS, set iGPU VRAM allocation to 96 or 128 GB for full performance. Without this, large models page to slow system RAM.
• hipBLAS (ROCm BLAS): Replaces cuBLAS for AMD systems. Install via ROCm SDK and set HIP_VISIBLE_DEVICES=0. Used automatically by torch and llama.cpp ROCm builds.
• AMDGPU target: Set HSA_OVERRIDE_GFX_VERSION=11.0.0 for RDNA3 cards (RX 7000 series) if ROCm doesn't detect the GPU correctly — common issue on consumer ARC/RDNA setups.
• Concurrent requests: AMD's ROCm handles batching less efficiently than CUDA for high concurrency. For single-user inference, the gap is small (<10%). For serving multiple users, NVIDIA holds a clear advantage.

• AVX-512 (Intel Icelake+, AMD Zen4): llama.cpp auto-detects and uses AVX-512 VNNI instructions for 4-bit matrix multiply. ~1.4× faster than AVX2. Use a compiled binary or set CMAKE_CXX_FLAGS=-march=native.
• Thread count: Set threads to equal physical cores (not hyperthreads). Hyperthreading adds latency on memory-bound tasks. E.g. for a 16P + 8E core i9-13900K, use -t 16 (P-cores only).
• NUMA pinning: On multi-socket systems, pin to one NUMA domain. Cross-socket memory access halves effective bandwidth. Use numactl --cpunodebind=0 --membind=0.
• Dual-channel / quad-channel RAM: Each additional memory channel doubles bandwidth. DDR5-6000 quad-channel gives ~190 GB/s vs ~48 GB/s for DDR4 single-channel — a 4× inference speedup.
• mmap and mlock: llama.cpp by default mmaps model files. Use --mlock to pin the model in RAM and prevent page faults during inference. Requires sufficient free RAM and root/CAP_IPC_LOCK.

• Ollama: Easiest setup; wraps llama.cpp with automatic hardware detection (CUDA, Metal, ROCm). Good for development and single-user. Not optimised for high-throughput multi-user serving.
• LM Studio: GUI-based; good for non-technical users. Uses llama.cpp backend. GPU offload via "GPU Layers" slider. Match layers to VRAM — rule of thumb: each 7B layer ~= 100 MB VRAM.
• llama.cpp (direct): Maximum control. Supports every acceleration path. Use -ngl 99 -t 8 --ctx-size 4096 --mlock as a good starting configuration.
• vLLM: Best for production multi-user serving on NVIDIA. PagedAttention enables efficient KV-cache sharing. Not supported on Apple Silicon. Requires CUDA 11.8+.
• ExLlamaV2: Fastest CUDA inference for quantised models. Uses EXL2 format (superior to GGUF on NVIDIA). Can exceed TensorRT-LLM speed for chat workloads at Q4–Q5.

• KV-cache size scales quadratically: A 7B model at 4K context uses ~500 MB KV-cache; at 128K context it uses ~16 GB. Reduce --ctx-size to free memory for the model itself.
• KV quantisation (--cache-type-k q8_0): Quantising the KV cache from FP16 to Q8 or Q4 halves its memory footprint with minor quality impact. Supported in llama.cpp ≥ b2300.
• Grouped-query attention (GQA) models: Llama 3, Mistral, and Qwen2+ use GQA — their KV cache is inherently smaller than older models. Prefer GQA models when context length matters.
• Flash Attention 2: Required for efficient long-context inference. Enabled automatically in vLLM and supported in llama.cpp with --flash-attn.

Best for: NVIDIA CUDA users wanting a dual-purpose gaming and AI workstation. Widest consumer GPU support, easiest driver management, and the best GUI tool ecosystem.

• CUDA works natively. Ollama, LM Studio, ExLlamaV2, and llama.cpp all run without WSL2 on NVIDIA GPUs. CUDA driver installation is handled automatically by the NVIDIA App.
• WSL2 is near-mandatory for serious Python-based AI work. Tools like vLLM, Axolotl, Unsloth, and BitsAndBytes are Linux-first. Install WSL2 (Ubuntu 22.04 LTS) and do Python AI development inside it. NVIDIA GPU passthrough to WSL2 delivers near-native CUDA performance with minimal overhead.
• Claude Code requires WSL2. It is not natively supported on PowerShell or Command Prompt. After enabling WSL2, install Node.js inside Ubuntu and run npm install -g @anthropic-ai/claude-code. This one step also resolves most Python ML compatibility issues.
• AMD discrete GPU ROCm is not available on Windows. AMD users should use the Vulkan backend in llama.cpp/Ollama (which works well for inference) as an alternative. For PyTorch training with AMD discrete GPUs, Linux is required.
• DirectML accelerates ONNX Runtime models (Phi-3, Mistral, Llama via ONNX) on any Windows GPU including Intel Arc. Slower than CUDA but hardware-agnostic — useful when CUDA is not an option.
• Recommended setup: NVIDIA drivers natively → WSL2 + Ubuntu 22.04 → Ollama and LM Studio on native Windows for daily chat → Claude Code and Python tools (vLLM, transformers, axolotl) inside WSL2.
• AI coding tool overhead. Windows NTFS filesystem operations are measurably slower than macOS APFS or Linux ext4. AI coding agents (Claude Code, Aider, Cursor) perform frequent file reads, diffs, and writes during each interaction. On Windows, this translates to longer response latencies and higher token consumption per task — the agent spends more time waiting on I/O and generates more retries. In practice, the same coding task can cost 10–30% more tokens on Windows compared to macOS or Linux. For heavy agentic coding use, WSL2 with files stored inside the Linux filesystem (not /mnt/c/) mitigates this, but native macOS or Linux remains faster.

Best for: Developers who want a polished Unix environment, large unified-memory model capacity, low power draw, and the best Claude Code experience without Linux complexity.

• MLX is the optimal inference path. Apple's open-source MLX framework is 20–30% faster than llama.cpp Metal for inference on Apple Silicon. Install via pip install mlx-lm, then run models directly with mlx_lm.generate. Ollama uses Metal/llama.cpp internally and is slightly slower but simpler to set up.
• Unified memory is the key architectural advantage. CPU and GPU share one high-bandwidth pool — up to 512 GB on Mac Studio M3 Ultra. A 70B model (40 GB Q4) loads without any VRAM/host memory boundary or PCIe transfer overhead.
• Claude Code works natively. macOS is a first-class Claude Code platform. zsh, Homebrew, and all standard Unix utilities work out of the box. No WSL2 or compatibility layer needed.
• No CUDA, EXL2, GPTQ/AWQ, vLLM, or BitsAndBytes. Axolotl and Unsloth for fine-tuning are not supported. MLX includes its own LoRA fine-tuning path (mlx_lm.lora) for Apple Silicon.
• Supported quantization formats: GGUF (all variants via Ollama/llama.cpp) and MLX native formats (int4, int8, fp16, bf16). EXL2, GPTQ, AWQ, and BitsAndBytes are not available.
• RAM is soldered and non-upgradeable. Choose your memory configuration at purchase. The 128 GB step (M4 Max MacBook Pro or Mac Studio) is the threshold for 70B Q4 models; 192–512 GB (Mac Studio M3 Ultra) enables 120B+ models.
• Power efficiency. An M3 Ultra running 70B at 18 t/s draws ~150 W. An equivalent-speed NVIDIA A100 draws 300–400 W. For always-on home servers this is a meaningful difference over time.

Best for: Fine-tuning, production multi-user serving, AMD discrete GPU acceleration, multi-GPU setups, and developers who need the full toolchain without compatibility workarounds.

• The reference platform for all AI/ML work. vLLM, Axolotl, Unsloth, TensorRT-LLM, and BitsAndBytes are developed and tested on Linux first. Some capabilities are Linux-only or substantially faster on Linux.
• ROCm works properly only on Linux. AMD ROCm 6.0+ enables full GPU acceleration for llama.cpp, vLLM, and PyTorch on RX 6000/7000 series GPUs. On Windows, ROCm support for discrete GPUs is not available — use Vulkan there instead.
• Claude Code and Aider are native. The full agentic experience — file editing, bash commands, multi-file reasoning, git integration — works without any wrapper or workaround.
• Fine-tuning requires Linux. Axolotl (most widely used LoRA/QLoRA pipeline), Unsloth (2× faster LoRA with less VRAM), and BitsAndBytes (required for QLoRA) all require Linux + CUDA. Attempting native Windows fine-tuning consistently produces environment and CUDA library compatibility errors.
• Docker + NVIDIA Container Toolkit enables fully reproducible GPU-accelerated environments. Deploy vLLM in one command: docker run --gpus all vllm/vllm-openai:latest --model meta-llama/Llama-3.1-70B-Instruct — the standard production pattern.
• Driver management is the main friction point. NVIDIA drivers require matching kernel headers and DKMS modules. Before upgrading the kernel, confirm the driver package is compatible. Use apt-mark hold linux-image-generic to pin a stable kernel once the driver is working.
• Full quantization access: GGUF, EXL2, GPTQ, AWQ, BitsAndBytes (4-bit/8-bit), and all llama.cpp-supported formats. Linux + CUDA gives access to every inference and training optimization path simultaneously.

• Model library: ollama pull llama3.1:70b downloads, quantizes, and runs in one command. 200+ curated models available.
• GPU: Auto-detects CUDA (NVIDIA), Metal (Apple), and ROCm/Vulkan (AMD). No configuration needed for most setups.
• API compatibility: Exposes /v1/chat/completions endpoint, so any OpenAI-compatible client (Open WebUI, Continue.dev, Cursor) connects immediately.
• Limitation: Not optimised for high-throughput multi-user serving. Single-user development and local apps are the sweet spot.

• Best for beginners: No command line required. Browse models from Hugging Face directly in the app, download with one click, adjust GPU layers via slider.
• GPU offload: Drag the "GPU Layers" slider to send as many transformer layers as fit into VRAM. Rule of thumb: each 7B layer ≈ 100 MB VRAM.
• Local server mode: Enables a persistent server that other apps (Open WebUI, Continue.dev) can connect to.
• Limitation: Closed source; slightly behind llama.cpp direct on performance. Does not support vLLM-style batching.

• Full offline operation: Designed to work with no internet connection. Model files are stored locally; no data leaves the machine.
• Extension ecosystem: Supports plugins for additional functionality such as web search, document retrieval, and remote model endpoints.
• API server: Exposes an OpenAI-compatible API on localhost:1337, allowing integration with any client that speaks the OpenAI format.
• Good alternative to LM Studio for users who prefer open-source software with the same ease of use.

• Air-gap friendly: The strongest choice for environments where internet connectivity must be fully severed after setup.
• LocalDocs feature: Index local documents (PDF, Word, text) and ask questions about them with retrieval-augmented generation (RAG).
• CPU-first design: Works well on CPU-only systems. GPU acceleration via Vulkan is available but less optimised than Ollama/LM Studio.

• Single executable: Download one file and run it — no Python environment or package manager required.
• Advanced sampling: Includes mirostat, tail-free sampling, and Typical P sampling not found in all frontends — particularly useful for creative/story generation.
• SillyTavern/Agnai integration: The community-standard backend for character-based AI chat interfaces.

• Maximum control: Exposes every optimization flag: -ngl (GPU layers), --ctx-size, --cache-type-k, --flash-attn, -t (threads), --mlock, and more.
• Supports all GPU backends in one codebase: CUDA, Metal, ROCm (HIPrt), Vulkan, OpenCL, and CPU-only (AVX2, AVX-512).
• GGUF quantization server: Includes llama-quantize to convert and quantize models locally without uploading to a cloud service.
• Best choice when you are benchmarking, scripting, or need a parameter not exposed by higher-level tools.

• PagedAttention: Manages KV-cache in variable-size pages, dramatically reducing memory fragmentation and enabling 2–5× higher throughput than llama.cpp for concurrent users.
• Continuous batching: Processes multiple requests simultaneously without waiting for a full batch, reducing latency under load.
• OpenAI-compatible API: Drop-in replacement for OpenAI endpoints — connect any client without code changes.
• Requires Linux + CUDA. Not supported on macOS or Windows without WSL2. The standard choice for production NVIDIA deployments.

• EXL2 format: Per-row mixed-precision quantization. A 4-bit EXL2 model typically outperforms a GGUF Q4_K_M on the same hardware in quality benchmarks.
• Speed: Consistently outperforms TensorRT-LLM on chat workloads at Q4–Q5. Best choice for maximum CUDA inference speed with quantized models.
• Speculative decoding support: Use a small draft model to propose tokens — can yield near 2× speed on certain workloads.
• Tabby API server: ExLlamaV2 is the backend for TabbyAPI, a high-performance OpenAI-compatible server used by many frontends.

• Full agentic loop: Can read project files, propose and apply edits, run tests, and iterate on errors without constant user prompting. Most effective on large codebases.
• Git integration: Understands git history, creates commits, and works across branches. Best used in a git-tracked repository.
• Windows note: Requires WSL2 (Ubuntu 22.04+). Running inside WSL2 gives the same experience as Linux. File edits across the /mnt/c mount work but are slower than editing files inside the WSL2 filesystem.
• Model selection: Defaults to Claude Sonnet; can be configured to use Claude Opus for complex reasoning tasks at higher cost.

• Inline completion: Suggests single lines or entire function bodies as you type with low latency. Works in virtually all languages.
• Copilot Chat: Ask questions about the codebase, get explanations, generate tests, and fix errors directly in the IDE sidebar.
• Copilot Workspace / Agent: Multi-step task completion — describe a feature and Copilot proposes a plan, creates files, and opens PRs (enterprise tier).
• Model choice: GitHub now lets users select the underlying model (GPT-4o, Claude Sonnet, Gemini 2.0) per session.

• Composer mode: Describe a change in natural language; Cursor proposes and applies diffs across multiple files simultaneously — the main differentiator from GitHub Copilot.
• Codebase indexing: Indexes your entire repository for semantic search, giving the model accurate context even in large codebases.
• Model selection: Supports Claude 3.5/3.7 Sonnet, GPT-4o, and o3 — swappable per-task. Can also point to a local Ollama endpoint for completions.
• Tab / Ghost text: A distinctive predictive editing feature that anticipates your next edit based on recent changes — faster than standard autocomplete.

• Local model support: Connect to Ollama, LM Studio, or any OpenAI-compatible API. Run Qwen2.5-Coder 32B locally for code completion with full privacy.
• Configurable via JSON: config.json specifies models for chat, completion, and embedding independently — mix local and cloud models.
• Tab autocomplete: Uses a separate fast local model (e.g. Qwen2.5-Coder 1.5B) for low-latency line completions while using a larger model for chat.
• Best choice for developers who want GitHub Copilot-style features but with local models and no subscription.

• Cascade agent: Handles multi-file edits, runs terminal commands, reads error output, and iterates — similar to Claude Code but embedded in the IDE.
• Generous free tier: Unlimited code completions and a meaningful number of agent interactions per month at no cost — competitive with Cursor.
• Codeium extension also available as a VS Code/JetBrains plugin if you don't want to switch editors, providing completions without the full Windsurf agent.

• Git-first workflow: Every change is committed with a descriptive message, making it easy to review, revert, or cherry-pick AI edits.
• Local model support: Connect to an Ollama endpoint and use Qwen2.5-Coder 32B or DeepSeek-Coder V2 for fully offline coding assistance.
• Benchmark leading: Consistently at the top of the SWE-Bench leaderboard for open-source coding agents, particularly in architect + editor mode (uses two models cooperatively).
• Works well alongside Claude Code — use Aider for git-structured file edits and Claude Code for broader exploration and command execution.

• One-command install: docker run -d -p 3000:80 --add-host=host.docker.internal:host-gateway ghcr.io/open-webui/open-webui:main — connects to a running Ollama instance automatically.
• RAG support: Upload PDFs, web URLs, or YouTube transcripts; the model retrieves relevant chunks during conversation.
• Multi-model conversations: Chat with multiple models side by side and compare responses.
• User management: Supports multiple user accounts with role-based access — suitable for sharing a local model server within a team.

• Multi-provider in one UI: Switch between ChatGPT, Claude, Gemini, and a local Ollama model within the same conversation history.
• Branching conversations: Fork any message to explore alternative responses without losing the original thread.
• Best choice when you want to compare cloud and local model outputs or need team access to multiple API providers from a single shared interface.

• Desktop-native: Installs like a normal application; no Docker or server setup needed. Connect to Ollama by pointing it to http://localhost:11434.
• Privacy-first defaults: Conversation history stored locally by default. Suitable for sensitive work where cloud storage of chat history is undesirable.

• Model comparison view: Query multiple models simultaneously and view responses side by side in the same window.
• Knowledge library: Attach PDF, Word, or text files to a conversation with built-in RAG. No separate setup required.
• Good choice for users who want a more polished interface than Open WebUI without the Docker setup complexity.

• YAML configuration: Specify model, dataset, LoRA rank, batch size, learning rate, and all training parameters in a single config.yml. Run with axolotl train config.yml.
• Broad format support: Accepts ShareGPT, Alpaca, instruction, and completion formats out of the box. Custom tokenization templates for any chat format.
• QLoRA support: Load the base model in 4-bit BitsAndBytes precision while training LoRA adapters in FP16 — fine-tune a 70B model on a single A100 80GB.
• DeepSpeed integration: Enables multi-GPU and CPU memory offloading for training models that don't fit in VRAM alone.

• 2× faster than HF PEFT at the same quality. Fine-tune Llama 3.1 8B on a 24 GB GPU in under 2 hours on a typical instruction dataset.
• 70% less VRAM. Fine-tune a 7B model on 6 GB VRAM; a 13B on 12 GB. Enables fine-tuning on consumer RTX 3060/4060 GPUs that previously could not run training at all.
• Colab-native: Unsloth provides free Google Colab notebooks for every major model — fine-tune without owning any GPU hardware.
• Export to GGUF: Built-in export to GGUF quantization formats (Q4_K_M, Q5_K_M, Q8_0) immediately after training.

• Web UI (LlamaBoard): Configure and launch fine-tuning jobs from a browser without any command line or Python knowledge.
• Advanced alignment methods: Supports RLHF, DPO, ORPO, and reward model training — not just supervised fine-tuning. The most comprehensive fine-tuning toolkit available.
• Multi-GPU and distributed training via DeepSpeed and FSDP with minimal configuration.

• Native Apple Silicon acceleration: Uses Metal for all matrix operations — no overhead from cross-platform abstraction layers.
• LoRA and QLoRA supported on most Llama, Mistral, Phi, and Qwen architecture models. Uses MLX native int4 quantization for the base model during QLoRA.
• Limitation: Dataset format support is narrower than Axolotl. Works best for instruction fine-tuning on simple JSONL datasets. Complex alignment training (DPO, RLHF) is not yet fully supported.
• Run with: mlx_lm.lora --model <path> --train --data ./data --iters 1000 --batch-size 4