README.md

metadata

language:
  - en
license: apache-2.0
tags:
  - ssm
  - state-space-model
  - causal-lm
  - rabbit
  - rtaforge
  - proof-of-concept
base_model: RtaForge/Anvaya-Rabbit-2.7B

Anvaya-Rabbit 2.7B — v0.1 Alpha

The architecture, training protocol, and infrastructure are the story.

Rabbit is the first model in the Anvaya series — a proof of concept demonstrating that a fully custom State-Space Model (SSM) can be trained from scratch, on a single consumer-grade GPU, with no dependence on attention or transformer building blocks.

This is not a production model. It is the opening move in a deliberate curriculum: Rabbit → Raccoon → Polar Bear. The benchmarks below are a baseline, not a claim.

Architecture

• Type: Ṛta-SSM v7.2.2, Fortress Unbroken — recurrent SSM, no attention
• Parameters: ~2.7B (post-subsumination)
• Layers: 64
• d_model / d_state: 2560
• Vocabulary: 50,280 (GPT-NeoX tokenizer)
• Precision: bfloat16
• Training seq_len: 64

Weights

This repository contains the base pretrained checkpoint (base/Anvaya-Rabbit-2.7B-0.1-alpha-base.pt) and the SFT imprint checkpoint (imprint/Anvaya-Rabbit-2.7B-0.1-alpha-imprint.pt).

Load the imprint weights (base + SFT overlay, recommended for inference):

from white_rabbit.rabbit_model import create_rabbit_model
from transformers import AutoTokenizer
import torch

model = create_rabbit_model(
    vocab_size=50280,
    durga_variant="fu-64",  # 64-layer Fortress Unbroken backbone
)
sd = torch.load("imprint/Anvaya-Rabbit-2.7B-0.1-alpha-imprint.pt", map_location="cpu")
model.load_state_dict(sd, strict=False)
model.eval()

tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")

Requires: rtaforge-substrates (private repository — contact guha@rtaforge.in for access). This model uses a custom SSM architecture not compatible with standard HuggingFace AutoModel.

Training infrastructure: Rta-Forge/polaris-revival — patched ROCm 7.2 runtime restoring native HIP dispatch on gfx803 (RX 560X), with fused SSM recurrence kernels. MIT licensed.

Training

Two proprietary components make this training regime possible:

• Subsuminator — migrates learned weights across architectures without retraining from scratch, enabling efficient curriculum transfer.
• Gurukul — a constitutional Sisya/Guru proposal-validation loop. Sisya proposes weight deltas; Guru validates them against constitutional constraints before applying. Strong learning signals extracted from limited data and compute.

Together they are why Rabbit trained in 7 days on a single consumer GPU.

1,500 accepted Gurukul proposals across 6 phases on a single AceCloud L4 (24GB VRAM). ~7 days effective training time (total elapsed higher due to crash recovery and VRAM leak debugging).

Phase	Proposals	Dataset	Focus
0	125	CAMEL Physics	Physical reasoning
1	125	CAMEL Chemistry	Chemical reasoning
2	125	CAMEL Biology	Biological reasoning
3	250	Raccoon Phase 1	General reasoning
4	500	Rabbit E2 Phase 4	Extended curriculum
5	375	Raccoon Phase 3 (consolidation re-run)	Pattern consolidation

Final checkpoint: Step 1,500. seq_len=64, batch_size=3, optimizer=Lion, lr=1e-5.

SFT imprint applied using surface-only gate-layer fine-tuning (65 examples, 3 epochs), trained with the Anvaya Gurukul protocol.

Evaluation Results (Step 1,500)

Internal — Scale-Invariant Metrics

Evaluated using Top-K accuracy and Mean Reciprocal Rank vs. a randomly initialised baseline of identical architecture. 50 samples per corpus, seq_len=64.

Metric	Random Init	Trained (Step 1,500)	Gain
Top-1 Accuracy (aggregate)	0.24%	1.90%	~8×
Top-10 Accuracy (aggregate)	0.24%	35.84%	~149×
MRR (aggregate)	0.0026	0.1724	~66×
MRR — Deep Math	0.0084	0.186	22×
Top-10 — Biology	~1.3%	~12%	~10×
Top-10 — Chemistry	~1.3%	~13%	~10×

These gains are measured against a randomly initialised model of identical architecture — they reflect what the training curriculum taught, not absolute capability.

Commercial Benchmarks (lm-eval harness)

Standard academic benchmarks are not yet meaningful here. Rabbit was deliberately trained at seq_len=64 as a pure architecture proof. Standard lm-eval prompts (few-shot examples + question) run 150–400 tokens — well beyond Rabbit's training context. Raccoon (seq_len=512) removes this constraint entirely.

Benchmark	Score	Notes
HellaSwag	25.89%	Prompt exceeds training seq_len
ARC-Challenge	26.71%	Prompt exceeds training seq_len
MMLU	26.89%	Prompt exceeds training seq_len
WinoGrande	48.62%	Prompt exceeds training seq_len
TruthfulQA MC1	21.91%	Prompt exceeds training seq_len

What Comes Next

Model	Params	seq_len	Status
Rabbit	~2.7B	64	✅ This model — v0.1 Alpha
Raccoon	~6.1B	512	In training — reasoning curriculum (math ×2, logic ×2)
Polar Bear	~13B	512	Planned — STEM + AEVA anti-hallucination layer

The delta between Rabbit and Raccoon is the story. One epoch → two epochs, seq_len 64 → 512, 2.7B → 6.1B. Same pipeline, same hardware philosophy. Give us more resources and watch what happens.