Benchmarking Local LLMs: Throughput, TTFT, and Speculative Decoding in LM Studio

Running Large Language Models locally has evolved dramatically with modern quantization techniques, Mixture-of-Experts (MoE) architectures, and Multi-Token Prediction (MTP) / Speculative Decoding.

In this benchmark, we evaluate the performance of several popular local LLMs running via LM Studio on a consumer-grade NVIDIA GeForce RTX 3060 (12 GB VRAM). We measure generation throughput (tokens/sec), Time to First Token (TTFT), memory usage, and the real-world efficiency of speculative decoding draft token acceptance.

---

🎯 The Benchmark Methodology

To ensure reproducible and standardized evaluations, all models were prompted with a complex coding task requiring input validation, dynamic programming optimization, and clean formatting:

Benchmark Prompt:
“Write a Python function called fibonacci(n) that returns the n-th Fibonacci number. Optimize it using memoization or dynamic programming to handle large values of n. Include basic error handling for negative integers or non-integer inputs, and provide a brief example of how to call the function.”

Test Parameters

• Sampling Temperature: 0.2
• Max Tokens: 1024
• Runs per Model: 2 measured runs (+ 1 warmup run)
• Execution Mode: Sequential with cooldown pauses between models to ensure clean GPU VRAM loading

---

📊 Comprehensive Benchmark Leaderboard

Here is the summary of results across all evaluated models:

Model ID	Model Name	Architecture / Quant	Context Window	KV Cache	Decode Speed	Total Speed	TTFT (ms)	Total Tokens	Draft Acceptance Rate	Status
qwen3.5-4b-mtp	Qwen 3.5 4b	Q4_K_M + MTP	65536	FP16	81.27 tok/s	80.21 tok/s	113.3 ms	758	91.4% (733/802)	✅ OK
gemma-4-e4b-it	Gemma 4 e4b Instruct	Q4_K_M	65536	FP16	69.12 tok/s	68.46 tok/s	137.5 ms	1066	N/A (Standard)	✅ OK
qwen3.5-9b-mtp	Qwen 3.5 9b	Q4_K_M + MTP	65536	FP16	69.00 tok/s	64.54 tok/s	122.1 ms	203	93.1% (149/160)	✅ OK
ornith-1.5-9b	Ornith 1.5 9b	Q4_K_M + MTP	65536	FP16	56.99 tok/s	56.57 tok/s	135.9 ms	1091	95.6% (1011/1057)	✅ OK
gemma-4-12b-it	Gemma 4 12b	Q4_K_M	32768	FP16	36.99 tok/s	36.56 tok/s	319.3 ms	1099	N/A (Standard)	✅ OK
qwen3.6-35b-a3b-mtp	Qwen 3.6 35b A3b MTP	Q4_K_S (MoE ~3B active) + MTP	32768	Q8_0	20.01 tok/s	19.93 tok/s	206.3 ms	1091	96.1% (1110/1155)	✅ OK
qwen3.6-27b-mtp	Qwen 3.6 27b MTP	UD_IQ2_XXS + MTP	8192	Q4_0	N/A	N/A	N/A	-	-	❌ Spilled to RAM

---

⚡ Speculative Decoding & Multi-Token Prediction (MTP)

Speculative decoding leverages a smaller draft model or specialized prediction heads (MTP) to draft multiple prospective tokens in parallel, which the main model verifies in a single forward pass.

When the acceptance rate is high, this drastically increases throughput without degrading generation quality.

Model	Architecture	Total Draft Tokens	Accepted Tokens	Rejected Tokens	Acceptance Rate
qwen3.6-35b-a3b-mtp	MoE (35B total, ~3B active)	1,155	1,110	45	96.10%
ornith-1.5-9b	Dense 9B	1,057	1,011	46	95.65%
qwen3.5-9b-mtp	Dense 9B	160	149	11	93.12%
qwen3.5-4b-mtp	Dense 4B	802	733	69	91.40%

Key Takeaway on Speculative Decoding

• Models equipped with MTP consistently achieved > 91% acceptance rate, with the 35B MoE model reaching an impressive 96.1%.
• This enables compact and MoE architectures to deliver near-interactive speeds while preserving reasoning capability.

---

🔍 Detailed Model Breakdown

1. Qwen 3.5 4B MTP (Q4_K_M)

• Decode Throughput: 81.27 tok/sec
• Time to First Token (TTFT): 113.3 ms

• Context Window: 65,536

  KV Cache: FP16 (Unquantized)

• Speculative Acceptance: 91.40% (733 / 802)
• Observations: Extremely fast response time and excellent throughput. Very lightweight, making it ideal for low-latency coding assistance.

2. Gemma 4 e4b Instruct (Q4_K_M)

• Decode Throughput: 69.12 tok/sec
• Time to First Token (TTFT): 137.5 ms

• Context Window: 65,536

  KV Cache: FP16 (Unquantized)

• Observations: Solid throughput for a standard (non-speculative) 4B-class model. Generated structured explanations and thorough unit test examples.

3. Qwen 3.5 9B MTP (Q4_K_M)

• Decode Throughput: 69.00 tok/sec
• Time to First Token (TTFT): 122.1 ms

• Context Window: 65,536

  KV Cache: FP16 (Unquantized)

• Speculative Acceptance: 93.12% (149 / 160)
• Observations: Outstanding balance of code quality and speed. Delivered concise, optimized dynamic programming solutions in under 2 seconds.

4. Ornith 1.5 9B MTP (Q4_K_M)

• Decode Throughput: 56.99 tok/sec
• Time to First Token (TTFT): 135.9 ms

• Context Window: 65,536

  KV Cache: FP16 (Unquantized)

• Speculative Acceptance: 95.65% (1,011 / 1,057)
• Observations: Extensive step-by-step reasoning tokens (1024 tokens generated) while maintaining a high 57 tok/s speed thanks to speculative decoding.

5. Gemma 4 12B (Q4_K_M)

• Decode Throughput: 36.99 tok/sec
• Time to First Token (TTFT): 319.3 ms

• Context Window: 32,768

  KV Cache: FP16 (Unquantized)

• Observations: Higher parameter count delivers detailed architectural reasoning, but at the cost of lower throughput (~37 tok/s).

6. Qwen 3.6 35B A3B MTP (Q4_K_S MoE)

• Decode Throughput: 20.01 tok/sec
• Time to First Token (TTFT): 206.3 ms

• Context Window: 32,768

  KV Cache: Q8_0

• Speculative Acceptance: 96.10% (1,110 / 1,155)
• Observations: Because only ~3B parameters are active per token, this 35B MoE fits in 12GB VRAM and maintains a steady 20 tok/s with a massive 32k context window and Q8_0 KV cache.

---

⚠️ The 12GB VRAM Ceiling: Why Dense 27B Spills to RAM

During testing of qwen3.6-27b-mtp (UD_IQ2_XXS), the model experienced significant slowdowns due to system RAM offloading.

Even after lowering the context window to 8,192 and quantizing the KV cache to Q4_0, the combined memory footprint exceeded the 12 GB physical VRAM capacity:

Component	VRAM Required
Model Weights (27B @ UD_IQ2_XXS)	~7.8 GB
MTP / Draft Heads	~1.2 GB
KV Cache @ 8k Context (Q4_0)	~1.1 GB
CUDA Buffers & LM Studio Runtime Overhead	~2.2 GB
Total Memory Required	~12.3 – 12.8 GB
Available GPU VRAM (RTX 3060)	12.0 GB (12,288 MB)

Because the memory requirement exceeds 12 GB, LM Studio offloads several layers to DDR system RAM over PCIe, reducing throughput from GPU memory bandwidth (~360 GB/s) down to system RAM speeds (~30–50 GB/s).

---

💡 Practical Recommendations for 12GB GPUs

1. Best Daily Driver for Coding / Speed: Qwen 3.5 9B MTP (Q4_K_M) or Qwen 3.5 4B MTP — 69–81 tok/s with sub-130ms TTFT.
2. Best Large Model Intelligence without RAM Spillover: Qwen 3.6 35B A3B MTP (Q4_K_S MoE) — full 32k context, Q8_0 KV cache, and steady 20 tok/s.
3. Reasoning-Heavy Workloads: Ornith 1.5 9B MTP — comprehensive reasoning traces with 57 tok/s throughput.
4. Avoid Dense 27B+ on 12GB VRAM: Stick to MoE architectures (like A3B) or 12B/9B dense models to avoid PCIe bus bottlenecks.

---

Benchmarked with the automated LLM Benchmark Suite.

---

🛠️ The Python Benchmark Script

Below is the complete Python benchmarking script used for this evaluation. It connects to any OpenAI-compatible server (LM Studio, vLLM, Ollama, etc.), measures TTFT with microsecond precision, tracks speculative decoding draft tokens, executes models sequentially with cooldowns, and generates markdown reports.

#!/usr/bin/env python3
"""
LLM Benchmark Suite for LM Studio and OpenAI-Compatible APIs
Measures TTFT, Generation tok/sec, Total Tokens, Draft Token Acceptance Rate, and generates Markdown reports.
Executes benchmarks sequentially one model at a time with automatic load handling.
"""

import argparse
import datetime
import json
import os
import re
import statistics
import sys
import time
from dataclasses import asdict, dataclass, field
from pathlib import Path
from typing import Any, Dict, List, Optional, Tuple

try:
    import requests
except ImportError:
    print("Error: 'requests' library is required. Install with: pip install requests", file=sys.stderr)
    sys.exit(1)

try:
    from rich.console import Console
    from rich.panel import Panel
    from rich.table import Table
    from rich import box
    RICH_AVAILABLE = True
    console = Console()
except ImportError:
    RICH_AVAILABLE = False
    console = None


DEFAULT_PROMPT = (
    "Write a Python function called fibonacci(n) that returns the n-th Fibonacci number. "
    "Optimize it using memoization or dynamic programming to handle large values of n. "
    "Include basic error handling for negative integers or non-integer inputs, "
    "and provide a brief example of how to call the function."
)

LOCAL_CONFIG_PATH = Path(__file__).resolve().parent / "opencode.jsonc"
HCM_CONFIG_PATH = Path(__file__).resolve().parent.parent / "hcm" / "opencode.jsonc"
USER_CONFIG_PATH = Path.home() / ".config" / "opencode" / "opencode.jsonc"

if LOCAL_CONFIG_PATH.exists():
    DEFAULT_CONFIG_PATH = LOCAL_CONFIG_PATH
elif HCM_CONFIG_PATH.exists():
    DEFAULT_CONFIG_PATH = HCM_CONFIG_PATH
else:
    DEFAULT_CONFIG_PATH = USER_CONFIG_PATH

DEFAULT_OUTPUT_PATH = Path(__file__).resolve().parent / "benchmark_results.md"


def strip_jsonc_comments(text: str) -> str:
    """Strip single-line and multi-line comments from JSONC without affecting strings."""
    result = []
    in_string = False
    escape = False
    in_single_comment = False
    in_multi_comment = False
    i = 0
    while i < len(text):
        c = text[i]
        next_c = text[i + 1] if i + 1 < len(text) else ""
        if in_single_comment:
            if c == "\n":
                in_single_comment = False
                result.append(c)
        elif in_multi_comment:
            if c == "*" and next_c == "/":
                in_multi_comment = False
                i += 1
        elif in_string:
            result.append(c)
            if escape:
                escape = False
            elif c == "\\":
                escape = True
            elif c == '"':
                in_string = False
        else:
            if c == "/" and next_c == "/":
                in_single_comment = True
                i += 1
            elif c == "/" and next_c == "*":
                in_multi_comment = True
                i += 1
            else:
                result.append(c)
                if c == '"':
                    in_string = True
        i += 1
    return "".join(result)


@dataclass
class ModelConfigInfo:
    model_id: str
    display_name: str
    context_limit: int = 65536
    output_limit: int = 16384
    kv_cache: str = "FP16 (Unquantized)"


def load_opencode_config(config_path: Path) -> Tuple[Optional[str], Dict[str, ModelConfigInfo]]:
    """Load base URL and detailed models metadata from opencode.jsonc config file."""
    if not config_path.exists():
        return None, {}
    try:
        with open(config_path, "r", encoding="utf-8") as f:
            raw = f.read()
        cleaned = strip_jsonc_comments(raw)
        data = json.loads(cleaned)
        provider = data.get("provider", {})
        lmstudio = provider.get("lmstudio", {})
        base_url = lmstudio.get("options", {}).get("baseURL")
        models_cfg = lmstudio.get("models", {})
        models = {}
        for k, v in models_cfg.items():
            if isinstance(v, dict):
                display_name = v.get("name", k)
                limit = v.get("limit", {})
                context_limit = limit.get("context", 65536)
                output_limit = limit.get("output", 16384)
            else:
                display_name = k
                context_limit = 65536
                output_limit = 16384
            models[k] = ModelConfigInfo(
                model_id=k,
                display_name=display_name,
                context_limit=context_limit,
                output_limit=output_limit,
                kv_cache="FP16 (Unquantized)",
            )
        return base_url, models
    except Exception as e:
        if RICH_AVAILABLE:
            console.print(f"[yellow]Warning: Could not parse {config_path}: {e}[/yellow]")
        else:
            print(f"Warning: Could not parse {config_path}: {e}", file=sys.stderr)
        return None, {}


def fetch_server_models(base_url: str) -> List[str]:
    """Fetch list of active/available models from the OpenAI-compatible /models endpoint."""
    url = f"{base_url.rstrip('/')}/models"
    try:
        resp = requests.get(url, timeout=5)
        resp.raise_for_status()
        data = resp.json()
        models = [m["id"] for m in data.get("data", []) if "id" in m]
        return models
    except Exception:
        return []


@dataclass
class SingleRunResult:
    run_idx: int
    ttft_ms: float
    decode_time_s: float
    total_time_s: float
    prompt_tokens: int
    completion_tokens: int
    total_tokens: int
    reasoning_tokens: int
    draft_tokens_total: int
    draft_tokens_accepted: int
    draft_tokens_rejected: int
    decode_tok_per_sec: float
    total_tok_per_sec: float
    draft_acceptance_rate: Optional[float]
    response_text: str
    error: Optional[str] = None


@dataclass
class ModelBenchmarkSummary:
    model_id: str
    model_name: str
    context_window: int = 65536
    kv_cache: str = "FP16 (Unquantized)"
    runs: List[SingleRunResult] = field(default_factory=list)
    successful_runs: int = 0
    failed_runs: int = 0
    avg_ttft_ms: float = 0.0
    min_ttft_ms: float = 0.0
    max_ttft_ms: float = 0.0
    std_ttft_ms: float = 0.0
    avg_decode_tok_per_sec: float = 0.0
    min_decode_tok_per_sec: float = 0.0
    max_decode_tok_per_sec: float = 0.0
    std_decode_tok_per_sec: float = 0.0
    avg_total_tok_per_sec: float = 0.0
    avg_prompt_tokens: float = 0.0
    avg_completion_tokens: float = 0.0
    avg_reasoning_tokens: float = 0.0
    avg_total_tokens: float = 0.0
    avg_total_time_s: float = 0.0
    total_draft_tokens: int = 0
    accepted_draft_tokens: int = 0
    rejected_draft_tokens: int = 0
    avg_draft_acceptance_rate: Optional[float] = None
    sample_response: str = ""


def run_single_inference(
    base_url: str,
    model_id: str,
    prompt: str,
    temperature: float,
    max_tokens: int,
    run_idx: int,
    stream_output: bool = False,
    timeout_s: int = 300,
    retry_on_loading: bool = True,
) -> SingleRunResult:
    """Execute a single streaming inference and capture performance metrics."""
    endpoint = f"{base_url.rstrip('/')}/chat/completions"
    payload = {
        "model": model_id,
        "messages": [{"role": "user", "content": prompt}],
        "temperature": temperature,
        "max_tokens": max_tokens,
        "stream": True,
        "stream_options": {"include_usage": True},
    }

    start_time = time.perf_counter()
    first_token_time: Optional[float] = None
    accumulated_content: List[str] = []
    usage_data: Dict[str, Any] = {}
    stats_data: Dict[str, Any] = {}

    try:
        resp = requests.post(endpoint, json=payload, stream=True, timeout=timeout_s)
        
        if resp.status_code != 200 and retry_on_loading:
            time.sleep(3.0)
            start_time = time.perf_counter()
            resp = requests.post(endpoint, json=payload, stream=True, timeout=timeout_s)

        if resp.status_code != 200:
            err_msg = f"HTTP {resp.status_code}: {resp.text[:200]}"
            try:
                err_json = resp.json()
                if "error" in err_json and "message" in err_json["error"]:
                    err_msg = f"HTTP {resp.status_code}: {err_json['error']['message']}"
            except Exception:
                pass
            raise RuntimeError(err_msg)

        for line in resp.iter_lines():
            if not line:
                continue
            decoded_line = line.decode("utf-8")
            if not decoded_line.startswith("data: "):
                continue
            data_str = decoded_line[6:].strip()
            if data_str == "[DONE]":
                break
            try:
                chunk = json.loads(data_str)
                choices = chunk.get("choices", [])
                if choices:
                    delta = choices[0].get("delta", {})
                    content = delta.get("content") or delta.get("reasoning_content") or ""
                    if content and first_token_time is None:
                        first_token_time = time.perf_counter()
                    if content:
                        accumulated_content.append(content)
                        if stream_output and RICH_AVAILABLE:
                            console.print(content, end="")
                        elif stream_output:
                            sys.stdout.write(content)
                            sys.stdout.flush()
                if "usage" in chunk and chunk["usage"]:
                    usage_data = chunk["usage"]
                if "stats" in chunk and chunk["stats"]:
                    stats_data = chunk["stats"]
            except json.JSONDecodeError:
                continue

        end_time = time.perf_counter()
        if stream_output:
            print()

        total_time_s = end_time - start_time
        if first_token_time is None:
            first_token_time = end_time
        ttft_ms = (first_token_time - start_time) * 1000.0
        decode_time_s = max(0.0001, end_time - first_token_time)

        response_text = "".join(accumulated_content)
        prompt_tokens = usage_data.get("prompt_tokens", 0)
        completion_tokens = usage_data.get("completion_tokens", len(accumulated_content))
        total_tokens = usage_data.get("total_tokens", prompt_tokens + completion_tokens)
        reasoning_tokens = usage_data.get("completion_tokens_details", {}).get("reasoning_tokens", 0)

        draft_total = stats_data.get("total_draft_tokens_count", 0)
        draft_accepted = stats_data.get("accepted_draft_tokens_count", 0)
        draft_rejected = stats_data.get("rejected_draft_tokens_count", 0)

        acceptance_rate = (draft_accepted / draft_total * 100.0) if draft_total > 0 else None
        decode_tok_per_sec = completion_tokens / decode_time_s if decode_time_s > 0 else 0.0
        total_tok_per_sec = completion_tokens / total_time_s if total_time_s > 0 else 0.0

        return SingleRunResult(
            run_idx=run_idx,
            ttft_ms=ttft_ms,
            decode_time_s=decode_time_s,
            total_time_s=total_time_s,
            prompt_tokens=prompt_tokens,
            completion_tokens=completion_tokens,
            total_tokens=total_tokens,
            reasoning_tokens=reasoning_tokens,
            draft_tokens_total=draft_total,
            draft_tokens_accepted=draft_accepted,
            draft_tokens_rejected=draft_rejected,
            decode_tok_per_sec=decode_tok_per_sec,
            total_tok_per_sec=total_tok_per_sec,
            draft_acceptance_rate=acceptance_rate,
            response_text=response_text,
        )

    except Exception as e:
        end_time = time.perf_counter()
        return SingleRunResult(
            run_idx=run_idx,
            ttft_ms=0.0,
            decode_time_s=0.0,
            total_time_s=end_time - start_time,
            prompt_tokens=0,
            completion_tokens=0,
            total_tokens=0,
            reasoning_tokens=0,
            draft_tokens_total=0,
            draft_tokens_accepted=0,
            draft_tokens_rejected=0,
            decode_tok_per_sec=0.0,
            total_tok_per_sec=0.0,
            draft_acceptance_rate=None,
            response_text="",
            error=str(e),
        )


def benchmark_model(
    base_url: str,
    model_id: str,
    model_name: str,
    prompt: str,
    num_runs: int,
    warmup_runs: int,
    temperature: float,
    max_tokens: int,
    show_stream: bool = False,
    cooldown_s: float = 3.0,
    context_window: int = 65536,
    kv_cache: str = "FP16 (Unquantized)",
) -> ModelBenchmarkSummary:
    """Benchmark a model across warmup and measured iterations sequentially."""
    summary = ModelBenchmarkSummary(
        model_id=model_id,
        model_name=model_name,
        context_window=context_window,
        kv_cache=kv_cache,
    )

    # Warmup runs
    for w in range(warmup_runs):
        run_single_inference(
            base_url=base_url,
            model_id=model_id,
            prompt=prompt,
            temperature=temperature,
            max_tokens=max_tokens,
            run_idx=-(w + 1),
            stream_output=False,
        )
        time.sleep(cooldown_s)

    # Measured runs
    for r in range(1, num_runs + 1):
        res = run_single_inference(
            base_url=base_url,
            model_id=model_id,
            prompt=prompt,
            temperature=temperature,
            max_tokens=max_tokens,
            run_idx=r,
            stream_output=show_stream,
        )
        summary.runs.append(res)
        if res.error:
            summary.failed_runs += 1
        else:
            summary.successful_runs += 1
        if r < num_runs:
            time.sleep(cooldown_s)

    success_runs = [r for r in summary.runs if not r.error and r.completion_tokens > 0]
    if success_runs:
        summary.sample_response = success_runs[0].response_text
        ttfts = [r.ttft_ms for r in success_runs]
        decode_speeds = [r.decode_tok_per_sec for r in success_runs]
        total_speeds = [r.total_tok_per_sec for r in success_runs]
        prompt_tokens_list = [r.prompt_tokens for r in success_runs]
        completion_tokens_list = [r.completion_tokens for r in success_runs]
        reasoning_tokens_list = [r.reasoning_tokens for r in success_runs]
        total_tokens_list = [r.total_tokens for r in success_runs]
        total_times = [r.total_time_s for r in success_runs]

        summary.avg_ttft_ms = statistics.mean(ttfts)
        summary.min_ttft_ms = min(ttfts)
        summary.max_ttft_ms = max(ttfts)
        summary.std_ttft_ms = statistics.stdev(ttfts) if len(ttfts) > 1 else 0.0

        summary.avg_decode_tok_per_sec = statistics.mean(decode_speeds)
        summary.min_decode_tok_per_sec = min(decode_speeds)
        summary.max_decode_tok_per_sec = max(decode_speeds)
        summary.std_decode_tok_per_sec = statistics.stdev(decode_speeds) if len(decode_speeds) > 1 else 0.0

        summary.avg_total_tok_per_sec = statistics.mean(total_speeds)
        summary.avg_prompt_tokens = statistics.mean(prompt_tokens_list)
        summary.avg_completion_tokens = statistics.mean(completion_tokens_list)
        summary.avg_reasoning_tokens = statistics.mean(reasoning_tokens_list)
        summary.avg_total_tokens = statistics.mean(total_tokens_list)
        summary.avg_total_time_s = statistics.mean(total_times)

        summary.total_draft_tokens = sum(r.draft_tokens_total for r in success_runs)
        summary.accepted_draft_tokens = sum(r.draft_tokens_accepted for r in success_runs)
        summary.rejected_draft_tokens = sum(r.draft_tokens_rejected for r in success_runs)
        if summary.total_draft_tokens > 0:
            summary.avg_draft_acceptance_rate = (
                summary.accepted_draft_tokens / summary.total_draft_tokens * 100.0
            )

    return summary

How to Run

# Benchmark all models in opencode.jsonc sequentially
python3 benchmark.py

# Benchmark specific model(s)
python3 benchmark.py -m qwen3.5-9b-mtp gemma-4-12b-it

# Benchmark with custom context window & KV cache flag
python3 benchmark.py -m qwen3.6-35b-a3b-mtp --context-window 32768 --kv-cache Q8_0