opencoti-llamafile

Self-contained single-file inference engine from the opencoti project — a Mozilla-Ocho llamafile 0.10.3 base carrying the opencoti patch series: advanced KV residency/quantization (PolyKV/TurboQuant/TCQ), rolling-KV window with host-RAM spill, DCA long-context extension, MTP speculative decode, sparse attention, RYS layer duplication, and a runtime introspection/control API.

Start with USAGE.md — it explains exactly how this engine diverges from upstream llamafile, every added feature, its flags, defaults, limitations, and which features compose.

This repo hosts the packaged release artifacts (they exceed GitHub's 2 GiB release-asset cap) plus the full patch series under patches/ and the project documentation under docs/. Release notes and per-release SHA256SUMS live under releases/ and are mirrored on the corresponding GitHub release at mann1x/opencoti.

The repo root and dso/ always hold the LATEST cut only. Superseded cuts are archived whole — binaries, manifests, their dso/ set, release notes and SHA256SUMS — under releases/archived/ in one self-contained directory per release tag (e.g. releases/archived/llamafile-v0.10.3+opencoti.c4/).

Supported / target model families

This engine is not model-agnostic in what it optimizes. Upstream llama.cpp GGUF support is unchanged (any GGUF that loads upstream loads here), but the opencoti feature set is built, tuned, and validated on two families:

Family	Role	Models	What's tuned for them
Gemma-4	PRIMARY target	26B-A4B-128e MoE ("A4B"), dense 12B / 31B, elastic E-series (E2B/E4B)	head_dim 256/512 kernels, iSWA dual-cache (rolling-KV, SharedKVPool, DCA wiring), MTP via gemma4-assistant drafters, fused MoE
Qwen	SECONDARY / verification	Qwen3.5 / 3.6 dense + MoE + hybrid (gated-delta-net), Qwen2.5-14B-1M	head_dim 128 kernels, NextN self-spec MTP (fused multi-step draft), DCA long-context on 1M-class models

Other architectures run with upstream behavior; opencoti features either fall back safely or are unvalidated on them — see the model-families section at the top of USAGE.md before relying on an opencoti feature elsewhere.

Artifacts

Since c6 the whole family is ALL-COMPRESSED: every GPU payload is built with nvcc --compress-mode=size (~7.6× smaller; the driver decompresses lazily at module load — measured on 3090 Linux, 5080 native Windows and 5080 WSL2 with no load-time or throughput drawback).

Artifact	Platform	GPU
opencoti-llamafile-<ver>-<tag>-x86_64.llamafile (~640 MB)	x86_64 + aarch64 (fat APE; Linux/macOS/BSD)	CUDA 13.3 embedded (compressed), x86_64-linux (sm_75/80/86/89/90/120f)
opencoti-llamafile-<ver>-<tag>-aarch64.llamafile (~361 MB)	same fat APE	CUDA 13.3 embedded (compressed), sbsa/aarch64-linux (sm_110f DGX Spark GB10 / Jetson Thor, sm_121a)
opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe (~55 MB)	same fat APE, incl. Windows	none embedded — CPU out of the box, GPU via side-load (below); smallest download
opencoti-llamafile-<ver>-<tag>-win-x86_64-gpu.llamafile.exe (~620 MB)	same fat APE, incl. Windows	CUDA 13.3 .dll embedded (compressed) — Windows GPU out of the box, sm_75→120f
opencoti-llamafile-<ver>-<tag>-universal.llamafile.exe (~1.2 GB)	ONE file for mixed Linux+Windows fleets	BOTH the Linux .so and Windows .dll embedded (compressed); each OS picks its own from /zip
dso/<ver>-<tag>/ggml-cuda-x86_64.so	side-load DSO for the bare APE on x86_64-linux	CUDA 13.3 (compressed), x86_64-linux (sm_75/80/86/89/90/120f)
dso/<ver>-<tag>/ggml-cuda-sbsa-aarch64.so	side-load DSO for the bare APE on aarch64-linux	CUDA 13.3 (compressed), sbsa/aarch64-linux (sm_110f, sm_121a)
dso/<ver>-<tag>/ggml-cuda-win-x86_64.dll	side-load DSO for the bare .exe on Windows x86_64	CUDA 13.3 (compressed), win-x86_64 (sm_75/80/86/89/90/120f)

The APE host binary is identical across variants and runs natively on both x86_64 and aarch64; the variants differ ONLY in which compressed GPU payloads are embedded. Windows refuses to run executables larger than 4 GiB — compression is what makes the -win-x86_64-gpu and -universal .exes possible. With no matching DSO available, inference falls back to CPU.

Release layout: only the latest cut's artifacts live at the repo root (and its DSOs under dso/<ver>-<tag>/). Older cuts are moved whole — artifacts, MANIFESTs, DSOs, notes, SHA256SUMS — to releases/archived/, one folder per release. Release notes for the current cut are under releases/.

Usage

chmod +x opencoti-llamafile-*.llamafile
# server mode
sh ./opencoti-llamafile-*.llamafile --server --port 8080 \
    -m <model>.gguf -ngl 99 --flash-attn on
# without --server it starts the interactive chat CLI

On the first GPU run the embedded CUDA DSO self-extracts to ~/.llamafile/v/<ver>/. --version prints the opencoti version string; GET /props exposes an opencoti introspection section at runtime (see USAGE.md §introspection).

-win variant / GPU side-load

• Windows, zero-setup GPU: grab opencoti-llamafile-<ver>-<tag>-win-x86_64-gpu.llamafile.exe (or the -universal one) — the compressed CUDA DLL is embedded; no side-load step, no CUDA Toolkit, only the NVIDIA driver.

• Windows, minimal download: run opencoti-llamafile-<ver>-<tag>-win-x86_64.llamafile.exe directly (CPU inference works out of the box). For GPU, download the published dso/<ver>-<tag>/ggml-cuda-win-x86_64.dll and place it at the side-load path — no CUDA Toolkit or MSVC install needed (only the NVIDIA driver):

  mkdir "%USERPROFILE%\.llamafile\v\<ver>"   rem e.g. 0.10.3
  copy ggml-cuda-win-x86_64.dll "%USERPROFILE%\.llamafile\v\<ver>\ggml-cuda.dll"
  

  then run with --gpu nvidia -ngl 99. Covers sm_75→120f (Turing → Blackwell). Verified on RTX 5080.

• Linux x86_64 (bare APE + GPU without the 4.8 GB download’s re-download): drop the published side-load DSO where the engine looks before extracting an embedded one:

  mkdir -p ~/.llamafile/v/<ver>/   # e.g. 0.10.3
  cp ggml-cuda-x86_64.so ~/.llamafile/v/<ver>/ggml-cuda.so
  

  Verify against releases/<tag>/SHA256SUMS first. The -win file is still a normal APE — sh ./…​.exe runs fine on Linux/macOS/BSD.

Verification

Each artifact ships with a MANIFEST.json recording the artifact sha256, versionString, gitCommit, the full patch list, and the sha256 of every embedded backend DSO; releases/<tag>/SHA256SUMS is the committed contract (CI verifies this repo's LFS blobs against it on every release tag). Verify the embedded DSO without running:

unzip -p opencoti-llamafile-*.llamafile ggml-cuda.so | sha256sum

License

Apache-2.0 (llamafile) + MIT (llama.cpp and bundled projects). opencoti patches and packaging are MIT.

PolyKV multi-session stress benchmark (package_courier)

An agentic multi-session benchmark: LangGraph courier agents (tool-calling, 10-step delivery chains, 100 packages) run against ONE engine at --parallel 32, ctx 163840, q8_0 KV, with a tps-floor scheduler that admits new sessions only while every live session keeps ≥ floor tok/s (PolyKV /polykv/tps SSE telemetry drives admission). Every run below was measured on a single NVIDIA RTX 6000 Blackwell (96 GB) — one GPU, one engine process, no tensor/pipeline split; the whole model and all 32 slots' KV are resident, so the numbers are pure scheduler/serving behaviour rather than an offload artifact. Each worker episode is a compiled langgraph.StateGraph (capacity_gate → agent ⇄ tools, with the capacity-gate node reading the engine's /polykv/capacity), so the harness doubles as the reference LangGraph integration for this engine. The full harness is published in this repo at bench/package_courier/ (work server + LangGraph agent app + launcher + HTML report generator — see its README and the usage guide at docs/features/package_courier_harness.md); reports under bench/reports/ (per-run HTML + raw stats).

The opencoti_langgraph Python package the harness builds on — OpencotiChatModel (LangChain BaseChatModel with PolyKV pool/session attach), PolykvClient/AsyncPolykvClient, pool tools, and the capacity-gate / compaction LangGraph nodes — is published at integrations/langgraph/ (pip install ./integrations/langgraph, needs langchain-core>=1.4.9

• langgraph>=1.2.9; see its README).

fan-out is the time-weighted median of concurrently running workers over the run — the number of agents actually in flight, not the scheduler's target and not the engine's attached-session count (which is cumulative and saturates at slots). Two token rates, and the distinction matters:

• gen tok/s — generated tokens ÷ wall. This is what the tps-floor scheduler gates on, so it is what sets fan-out. It is not a model-speed comparison: at 36:1–74:1 prompt-to-completion it largely reflects how terse a model is.
• work tok/s — (prompt tokens actually processed, i.e. net of prompt-cache hits) + generated, ÷ wall. This is the real work rate. Note the cache-hit column: each turn resends the whole conversation and the engine serves 25–76 % of it from KV cache, so counting raw prompt length would inflate this several-fold.

model (MTP)	floor	score	wall	delivery mean	fan-out (med / peak)	gen tok/s	work tok/s	cache hit
Qwen3.6-27B-Omnimerge-v4 Q6_K (NextN self-spec, dense)	15	99/100	2171 s	146 s	7 / 8	79.9	813	74 %
″	5	99/100	2509 s	467 s	20 / 22	70.4	800	70 %
″	1	98/100	2803 s	818 s	32 / 32 (cap)	62.9	794	67 %
Gemma-4 26B-A4B-it Q4_K_M (assistant-MTP, MoE)	15	100/100	864 s	51 s	6 / 8	64.5	2912	40 %
″	5	99/100	1090 s	145 s	14 / 16	50.8	2682	30 %
″	1	95/100	1340 s	346 s	32 / 32 (cap)	40.4	2296	25 %
″ (cap 64)	25	100/100	773 s	31 s	4 / 6	72.0	3010	45 %
″ (cap 64)	20	100/100	823 s	38 s	5 / 6	67.6	2862	44 %
Gemma-4 26B-A4B-it F16 (assistant-MTP, cap 64)	25	100/100	1181 s	35 s	3 / 5	47.1	1502	58 %
″	20	100/100	1268 s	44 s	4 / 5	43.9	1548	54 %
″	15	100/100	1369 s	59 s	5 / 6	40.7	1571	49 %
″	5	100/100	1712 s	174 s	10 / 14	32.5	1567	36 %
″	1	100/100	2542 s	1282 s	64 / 64 (cap)	21.9	735	56 %
Qwen3.6-35B-A3B UD-Q6_K (NextN self-spec, MoE)	25	100/100	912 s	55 s	6 / 7	110.1	1609	75 %
″	20	100/100	890 s	64 s	7 / 9	112.5	1600	76 %
″	15	100/100	914 s	84 s	10 / 12	109.9	1609	75 %
″	5	100/100	1041 s	225 s	24 / 27	96.5	1567	72 %
″	1	100/100	1574 s	786 s	64 / 64 (cap)	63.7	1264	65 %

Reading these numbers

Work rate is set by active parameters. The A4B pair runs at 2912 work tok/s against 813 for the dense 27B at floor 15 — 3.6×, which is what a 4B-active MoE against a 27B-dense should look like. Weight precision costs on top of that: the same A4B at F16 does 1571, i.e. 1.85× behind Q4_K_M, a bandwidth-bound MoE reading 4× the bytes per active expert.

Terseness is a separate axis from speed, and gen tok/s conflates them. The dense 27B posts the higher generation rate at floor 15 (79.9 vs 64.5) while doing 3.6× less work per second — because it is a reasoning model that emits <think> blocks and produces 173 k tokens to the A4B's 55.6 k for the same 100 deliveries. Wall time follows the product of both axes: A4B finishes 2.5× sooner (864 s vs 2171 s) by being both faster and terser. If you take one thing from this table: do not rank engines by generation tok/s on an agentic workload — it rewards verbosity.

The cache-hit column is why rows are not freely comparable. Every turn resends the full conversation; the engine matches the common prefix in KV and only prefills the delta. A model that needs more turns per package re-sends a longer already-cached prefix, so its hit rate climbs and its measured work rate falls — not because the engine got slower, but because there was less left to do. That is why the 35B-A3B's 1609 cannot be read as "slower than the A4B's 2912": it ran at 75 % cache hit against 40 %. Compare rows freely only where the cache column is close (e.g. A4B Q4_K_M vs F16, 40 % vs 49 %).

None of this is a clean engine benchmark — it is a scheduler-and-serving measurement over a real agentic workload. For isolated decode/prefill throughput use docs/evaluations/mtp.md.

How fan-out is set. workers ≈ gen tok/s ÷ floor — the scheduler admits until the generation rate the engine can deliver is exactly divided into per-session shares of floor tok/s. That rate is not a constant: it rises as the floor rises (A4B-Q4: 40.4 → 72.0 tok/s from floor 1 to 25) at identical total work, so the engine does the same job faster with fewer concurrent sequences — consistent with per-token batch interference and KV pressure, though we have not isolated which dominates. Practically: a higher floor buys both a faster per-session experience and a shorter wall time. You only pay in parallelism.

The A4B + assistant-MTP pair finishes ~2.5× faster end-to-end than the dense 27B at floor 15 (864 s vs 2171 s) while generating 3.1× fewer tokens for the same 100 deliveries (55.6 k vs 173 k) — a terser agent on a faster engine.

All losses across all runs are agent-side (turn-budget wander on hard packages) — zero tool errors, zero LLM errors, engines clean throughout. The ab-pooled vs ab-naive pair isolates the SharedKVPool win at identical workload.

The A4B F16 ladder answers "what does weight precision cost under agentic load": same engine, same workload, F16 weights are ~1.55× slower end-to-end than Q4_K_M at every floor (1.9× at floor 1, where contention compounds it) — a bandwidth-bound MoE. At comfortable floors quality is saturated at Q4 (both 100/100), but at max fan-out precision shows: F16 floor 1 finished zero-loss at the full 64-worker cap grinding at ~1.1 tok/s median, while Q4_K_M at floor 1 loses 3-5 first-wave packages to turn-budget wander (95/100 original; a controlled re-run on the current StateGraph agent, same seed/cap, scored 97/100 with the identical loss signature — confirming the loss mode is model-precision-driven under cold-start contention, not a harness artifact).

The 35B-A3B ladder (worker cap raised to 64, floors extended to 20/25) is the cleanest full ladder: 100/100 at every floor including max fan-out, with the scheduler holding median per-turn tps on the floor (5→5.6, 15→17, 20→23, 25→28) as the fleet shrinks 64→6. It also locates the fan-out knee: wall time is flat (~890-1040 s) from floor 5 up, but floor 1 costs 1574 s — 64 workers oversubscribing 32 slots just queues (per-session tps 2.8, deliveries waiting 786 s mean). Sweet spot for this engine: floor 20-25 — same wall as floor 15 with the best per-package latency, at 6-7 concurrent workers each getting 23-28 tok/s. Note how little concurrency it takes to saturate: throughput is already at its maximum by floor 15, and everything above ~10 workers on this engine buys queueing, not work.

---

Computing resources kindly provided by Vodafone