README.md

---
language:
- en
license: apache-2.0
tags:
- ssm
- state-space-model
- causal-lm
- rabbit
- rtaforge
- proof-of-concept
base_model: RtaForge/Anvaya-Rabbit-2.7B
---

# Anvaya-Rabbit 2.7B — v0.1 Alpha

> **The architecture, training protocol, and infrastructure are the story.**

Rabbit is the first model in the Anvaya series — a proof of concept demonstrating
that a fully custom State-Space Model (SSM) can be trained from scratch, on a
single consumer-grade GPU, with no dependence on attention or transformer
building blocks.

This is not a production model. It is the opening move in a deliberate curriculum:
**Rabbit → Raccoon → Polar Bear.** The benchmarks below are a baseline, not a claim.

## Architecture

- **Type**: Ṛta-SSM v7.2.2, Fortress Unbroken — recurrent SSM, no attention
- **Parameters**: ~2.7B (post-subsumination)
- **Layers**: 64
- **d_model / d_state**: 2560
- **Vocabulary**: 50,280 (GPT-NeoX tokenizer)
- **Precision**: bfloat16
- **Training seq_len**: 64

## Weights

This repository contains the base pretrained checkpoint
(`base/Anvaya-Rabbit-2.7B-0.1-alpha-base.pt`) and the SFT imprint checkpoint
(`imprint/Anvaya-Rabbit-2.7B-0.1-alpha-imprint.pt`).

Load the imprint weights (base + SFT overlay, recommended for inference):

```python
from white_rabbit.rabbit_model import create_rabbit_model
from transformers import AutoTokenizer
import torch

model = create_rabbit_model(
    vocab_size=50280,
    durga_variant="fu-64",  # 64-layer Fortress Unbroken backbone
)
sd = torch.load("imprint/Anvaya-Rabbit-2.7B-0.1-alpha-imprint.pt", map_location="cpu")
model.load_state_dict(sd, strict=False)
model.eval()

tokenizer = AutoTokenizer.from_pretrained("EleutherAI/gpt-neox-20b")
```

> **Requires**: `rtaforge-substrates` (private repository — contact
> guha@rtaforge.in for access). This model uses a custom SSM architecture
> not compatible with standard HuggingFace `AutoModel`.

**Training infrastructure**: [`Rta-Forge/polaris-revival`](https://github.com/Rta-Forge/polaris-revival) —
patched ROCm 7.2 runtime restoring native HIP dispatch on gfx803 (RX 560X), with
fused SSM recurrence kernels. MIT licensed.

## Training

Two proprietary components make this training regime possible:

- **Subsuminator** — migrates learned weights across architectures without
  retraining from scratch, enabling efficient curriculum transfer.
- **Gurukul** — a constitutional Sisya/Guru proposal-validation loop. Sisya
  proposes weight deltas; Guru validates them against constitutional constraints
  before applying. Strong learning signals extracted from limited data and compute.

Together they are why Rabbit trained in 7 days on a single consumer GPU.

**1,500 accepted Gurukul proposals across 6 phases on a single AceCloud L4 (24GB VRAM).
~7 days effective training time (total elapsed higher due to crash recovery and VRAM
leak debugging).**

| Phase | Proposals | Dataset | Focus |
|-------|-----------|---------|-------|
| 0 | 125 | CAMEL Physics | Physical reasoning |
| 1 | 125 | CAMEL Chemistry | Chemical reasoning |
| 2 | 125 | CAMEL Biology | Biological reasoning |
| 3 | 250 | Raccoon Phase 1 | General reasoning |
| 4 | 500 | Rabbit E2 Phase 4 | Extended curriculum |
| 5 | 375 | Raccoon Phase 3 (consolidation re-run) | Pattern consolidation |

**Final checkpoint: Step 1,500.** seq_len=64, batch_size=3, optimizer=Lion, lr=1e-5.

SFT imprint applied using surface-only gate-layer fine-tuning (65 examples, 3 epochs),
trained with the Anvaya Gurukul protocol.

## Evaluation Results (Step 1,500)

### Internal — Scale-Invariant Metrics

Evaluated using Top-K accuracy and Mean Reciprocal Rank vs. a randomly initialised
baseline of identical architecture. 50 samples per corpus, seq_len=64.

| Metric | Random Init | Trained (Step 1,500) | Gain |
|--------|-------------|----------------------|------|
| Top-1 Accuracy (aggregate) | 0.24% | **1.90%** | **~8×** |
| Top-10 Accuracy (aggregate) | 0.24% | **35.84%** | **~149×** |
| MRR (aggregate) | 0.0026 | **0.1724** | **~66×** |
| MRR — Deep Math | 0.0084 | **0.186** | **22×** |
| Top-10 — Biology | ~1.3% | **~12%** | **~10×** |
| Top-10 — Chemistry | ~1.3% | **~13%** | **~10×** |

These gains are measured against a randomly initialised model of identical
architecture — they reflect what the training curriculum taught, not absolute
capability.

### Commercial Benchmarks (lm-eval harness)

> **Standard academic benchmarks are not yet meaningful here.** Rabbit was
> deliberately trained at seq_len=64 as a pure architecture proof. Standard
> lm-eval prompts (few-shot examples + question) run 150–400 tokens — well
> beyond Rabbit's training context. Raccoon (seq_len=512) removes this
> constraint entirely.

| Benchmark | Score | Notes |
|-----------|-------|-------|
| HellaSwag | 25.89% | Prompt exceeds training seq_len |
| ARC-Challenge | 26.71% | Prompt exceeds training seq_len |
| MMLU | 26.89% | Prompt exceeds training seq_len |
| WinoGrande | 48.62% | Prompt exceeds training seq_len |
| TruthfulQA MC1 | 21.91% | Prompt exceeds training seq_len |

## What Comes Next

| Model | Params | seq_len | Status |
|-------|--------|---------|--------|
| **Rabbit** | ~2.7B | 64 | ✅ This model — v0.1 Alpha |
| **Raccoon** | ~6.1B | 512 | In training — reasoning curriculum (math ×2, logic ×2) |
| **Polar Bear** | ~13B | 512 | Planned — STEM + AEVA anti-hallucination layer |

The delta between Rabbit and Raccoon is the story. One epoch → two epochs,
seq_len 64 → 512, 2.7B → 6.1B. Same pipeline, same hardware philosophy.
**Give us more resources and watch what happens.**