Using Nekko Agent

Local inference benchmarks

View as Markdown llms.txt

How fast Nekko Agent's engine runs a model compared with Ollama and LM Studio, on the same machine and the same model file, measured the same way. vLLM is supported by the harness but not measured here (it has no native Windows build).

The numbers are from one machine. They are not a promise about yours; run the harness (below) to get your own.

#Results, 2026-09-30

Model: gemma-4-12B-it-Q4_K_M.gguf (lmstudio-community), the same file for every server. Machine: NVIDIA RTX 5090 32 GB, AMD Ryzen 9 9950X3D, Windows 11. Median of 3 runs; 32k context; everything on the GPU.

Six chats visited in turn, three rounds each, more chats than the server has slots (what a person with several agent chats open does):

#Reading the results honestly

• Where Nekko Agent is clearly ahead: edits that repeat their input (about 4.5x the other servers, from n-gram speculative decoding), plain decode (about a third faster), coming back to a chat that lost its slot (about 40% faster than Ollama), and parallel requests (Ollama serves one at a time by default).
• Where it is not: the first turn of a brand-new long conversation is slower than Ollama's (1.7 s against 1.3 s), and prefill is level with everyone. The first-turn gap was 2.3 s before this work; the remaining difference is under investigation.
• Noise: run-to-run variation was about 10% on prefill and larger on the four-request number (the same configuration measured 256, 283 and 343 tokens/s across runs). Differences smaller than that are not claims.
• "Nekko Agent engine" here is llama-server from the pinned llama.cpp build started with exactly the flags the engine now passes, measured without the engine's router in front of it. The router (Rust, in the engine daemon) adds about 0.1 ms per request: 27.39 ms against 27.29 ms straight to the model server, median of 30 one-token requests.

#What changed to get there

Measured flag by flag on the same model and build (b11011):

• --spec-default (n-gram speculative decoding from the conversation, no draft model): code edit 117 to 506-534 tokens/s, decode 127 to 154-155.
• --no-cache-idle-slots: llama.cpp's default copies every idle slot to its RAM prompt cache when a new request arrives. Keeping them in place cut the first turn of a new chat from 2.3 s to 1.7 s and a revisit of an evicted chat from 1.7 s to 0.8 s. The RAM cache still catches what a slot really loses (turning it off entirely was worse on both).
• --cache-reuse 256: reuse cached chunks around an edit in the middle of a prompt.
• Vision models keep it. With Gemma's projector loaded as well, the code edit ran at 633 tokens/s and decode at 152; llama.cpp only drops --cache-reuse for multimodal models.
• A draft model is opt-in, not automatic. Gemma 4 E4B as the draft for the 12B raised the code edit to 624 tokens/s but halved ordinary decode (155 to 74) because only 55% of its guesses were accepted. A draft helps only when it is much smaller than its target.
• Parallel slots and a shared KV pool were already llama.cpp's default (-np auto with a unified cache), so nothing changed there.

Each flag is passed only when the installed engine build supports it (the engine reads the binary's --help once), so an older engine keeps working without them.

#Reproducing

The harness is crates/nekko-bench. It measures; it does not start servers.

cargo build --release -p nekko-bench

Start each server with the same GGUF, then list them in a config file:

{
  "machine": "RTX 5090 32 GB, Windows 11",
  "model": "gemma-4-12B-it Q4_K_M",
  "repeats": 3,
  "targets": [
    { "name": "Nekko Agent engine", "baseUrl": "http://127.0.0.1:18095/v1", "model": "gemma" },
    { "name": "Ollama", "baseUrl": "http://127.0.0.1:11434/v1", "model": "bench-gemma" },
    { "name": "LM Studio", "baseUrl": "http://127.0.0.1:1234/v1", "model": "bench-gemma" }
  ]
}

target/release/nekko-bench bench.json --out results.json --markdown results.md

--only decode,agent runs a subset. The workloads: decode, prefill, agent (six turns over an 8k-token context), rotate (six chats, three rounds), code-edit, concurrent (four at once).

How each server was started for the results above:

• Nekko Agent engine: llama-server --model <gguf> --alias gemma --port 18095 --ctx-size 32768 --n-gpu-layers 999 --cache-reuse 256 --spec-default --no-cache-idle-slots
• llama.cpp as before: the same without the last three flags.
• Ollama: a Modelfile with FROM <gguf>, PARAMETER num_ctx 32768, PARAMETER num_gpu 999, then ollama create bench-gemma -f Modelfile.
• LM Studio: lms server start and lms load google/gemma-4-12b --context-length 32768 --gpu max --identifier bench-gemma.

The harness counts tokens from each server's usage report when it sends one, and times the first streamed piece of text or reasoning as the first token.