Loke-60000/pika-9b-exp

Pika 9B (experimental)

Qwen3.5-9B with a 22x smaller KV cache. Pika replaces the 8 full-attention layers of Qwen3.5-9B with DeepSeek-V4-style attention: one compressed latent per position, a short sliding window, caches shared across layers, and a trained top-k indexer that lets each query read only the positions that matter. The 24 Gated DeltaNet layers are Qwen's, untouched.

This is an experiment. The model is unstable and may give worse output than Qwen3.5-9B. Load it with trust_remote_code=True (see Usage): the attention code that makes it work ships in this repo.

Benchmarks

What changed inside

Only the 8 full-attention layers (3, 7, 11, ..., 31). Each one now:

• keeps one 512-dim latent per position that serves as both key and value, instead of 4 key-value heads of 256 dims each;
• attends to a 128-token window of its own latents plus a compressed cache shared by its group of layers (pooled 2:1 in the first half of the network, 1:1 in the second);
• uses a trained indexer that picks the 512 most relevant compressed positions per query, so attention cost stays flat as the context grows;
• stores its cache in fp8 (latents) and fp4 (index keys), and has per-head attention sinks.

One layer (27) keeps its own cache instead of sharing its group's: without this anchor layer, the converted model could not retrieve anything past the window.

Context	Original attention cache	Pika attention cache
32k	1.1 GB	49 MB
128k	4.3 GB	196 MB
262k	8.6 GB	391 MB

Both models also carry 26.7 MB of fixed DeltaNet state, at any length.

Usage

The attention code ships in this repo, so plain transformers loads it with trust_remote_code:

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Loke-60000/pika-9b-exp", trust_remote_code=True,
                                             dtype="bfloat16", device_map="cuda")
tokenizer = AutoTokenizer.from_pretrained("Loke-60000/pika-9b-exp")

messages = [{"role": "user", "content": "Hello!"}]
text = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
ids = tokenizer(text, return_tensors="pt", add_special_tokens=False).input_ids.cuda()
out = model.generate(ids, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(out[0, ids.size(1):], skip_special_tokens=True))

For very long prompts, feed the prompt in chunks with a cache instead of one forward pass, so the activations stay small and only the compressed cache grows:

import torch
from transformers import DynamicCache

cache, chunk = DynamicCache(config=model.config), 4096
with torch.no_grad():
    for start in range(0, ids.size(1), chunk):
        part = ids[:, start : start + chunk]
        positions = torch.arange(start, start + part.size(1), device=ids.device)[None]
        logits = model(part, position_ids=positions, past_key_values=cache, use_cache=True,
                       logits_to_keep=1).logits
    # then decode token by token from `logits`, passing the same cache and the next position

License

Apache-2.0, as is the base model, Qwen/Qwen3.5-9B. Pika is a derivative of Qwen3.5-9B: the attention layers were rebuilt and retrained, the rest of the weights are Qwen's. See LICENSE.