OpenTransformer commited on
Commit
b086478
·
1 Parent(s): 0bb5c6e

Prepare AGILLM4 AR SAT NAT scaffold

Browse files

Re-add NAT as a real third head beside AR and SAT, wire NAT checkpoints/deltas/inference, initialize NAT in the v4 seed builder, and update Vast launch scripts/docs for NAT cadence.

AGILLM-4.md CHANGED
@@ -9,7 +9,7 @@ AGILLM-4 should not be "AGILLM-3 with a larger `--block`." The next useful versi
9
 
10
  The near-term codebase target is a measurable long-context trainer that can survive on current hardware:
11
 
12
- - RTX 4090: production 24GB long-block lane for the current-size model family, aiming above the current ~1200-token ceiling while preserving AR+SAT.
13
  - B200 180GB: serious 2k-16k experiments.
14
  - B300 262GB: 8k-64k experiments, then memory-augmented tests.
15
  - Multi-GPU/cluster: ring or sequence parallel experiments later.
@@ -43,10 +43,12 @@ python -u /workspace/agillm-4/nB300_agillm4.py train \
43
  --attn_backend sdpa \
44
  --grad_checkpoint \
45
  --optimizer paged_adamw8bit \
46
- --sat_every 1
 
 
47
  ```
48
 
49
- Important: `--sat_every 1` keeps SAT trained every step. The AGILLM-4 code now backprops AR and SAT sequentially, so the objective remains AR+SAT while peak VRAM is lower than holding both activation graphs at once.
50
 
51
  Escalation ladder on 4090:
52
 
@@ -55,7 +57,55 @@ Escalation ladder on 4090:
55
  3. `block=1792`
56
  4. `block=2048`
57
 
58
- If 8-bit optimizer is unavailable, install `bitsandbytes` rather than dropping the long-block target. SAT remains active every step; the code lowers peak memory by backpropagating AR and SAT sequentially, not by deleting SAT.
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
59
 
60
  ## First Implemented Scaffold
61
 
@@ -64,6 +114,7 @@ If 8-bit optimizer is unavailable, install `bitsandbytes` rather than dropping t
64
  - `--attn_backend manual|sdpa`
65
  - `--grad_checkpoint`
66
  - AGILLM-4 defaults for chat SFT compatibility inherited from the patched AGILLM-3 file.
 
67
 
68
  `block_sweep_agillm4.py` runs synthetic forward/backward sweeps:
69
 
@@ -222,6 +273,16 @@ Initial hypothesis:
222
  - Anchor memory length: 512-4096 anchors.
223
  - Token attention stays cheap while memory covers hundreds of thousands of tokens.
224
 
 
 
 
 
 
 
 
 
 
 
225
  ### Compressive Recurrence
226
 
227
  Keep a recurrent memory bank updated by a learned compressor:
 
9
 
10
  The near-term codebase target is a measurable long-context trainer that can survive on current hardware:
11
 
12
+ - RTX 4090: production 24GB long-block lane for the current-size model family, aiming above the current ~1200-token ceiling while preserving AR+SAT+NAT.
13
  - B200 180GB: serious 2k-16k experiments.
14
  - B300 262GB: 8k-64k experiments, then memory-augmented tests.
15
  - Multi-GPU/cluster: ring or sequence parallel experiments later.
 
43
  --attn_backend sdpa \
44
  --grad_checkpoint \
45
  --optimizer paged_adamw8bit \
46
+ --sat_every 1 \
47
+ --nat_every 4 \
48
+ --nat_max_tokens 768
49
  ```
50
 
51
+ Important: `--sat_every 1 --nat_every 4` keeps SAT trained every step and NAT active on a cadence that fits 24GB cards. On B200/B300 use `--nat_every 1` for full AR+SAT+NAT every step. The AGILLM-4 code now backprops AR, SAT, and NAT sequentially, so the objective remains joint while peak VRAM is lower than holding all activation graphs at once.
52
 
53
  Escalation ladder on 4090:
54
 
 
57
  3. `block=1792`
58
  4. `block=2048`
59
 
60
+ If 8-bit optimizer is unavailable, install `bitsandbytes` rather than dropping the long-block target. SAT remains active every step; NAT should stay enabled with a slower cadence or `--nat_max_tokens` cap on 24GB. The code lowers peak memory by backpropagating AR, SAT, and NAT sequentially, not by deleting heads.
61
+
62
+ ## Warm Start from AGILLM-3 (function-preserving)
63
+
64
+ The next AGILLM-4 run does **not** start from a random init. `build_v4_seed.py`
65
+ turns the most recent AGILLM-3 SFT checkpoint into a function-preserving v4
66
+ seed:
67
+
68
+ ```bash
69
+ python /workspace/agillm-4/build_v4_seed.py \
70
+ --from-ckpt /workspace/ckpts_sft_math_v1/final.pt \
71
+ --v4-preset main \
72
+ --out /workspace/agillm-4/agillm4_seed_from_v3.pt
73
+ ```
74
+
75
+ Construction rules (v3 d=1024 / H=16 / r=128 / L=24 → v4 d=1536 / H=24 / r=192 / L=32):
76
+
77
+ - Embedding, AR head, and NAT head pad with **zeros** in new `d_v4 - d_v3` columns. New-dim
78
+ embeddings are zero — they propagate harmlessly through the rest of the
79
+ graph because all paths that touch them are zero-padded too.
80
+ - If the v3 checkpoint has no NAT head, `build_v4_seed.py` initializes NAT from
81
+ the expanded AR projection rather than random weights.
82
+ - Block params: `qkv`, `proj`, `ff.0`, `ff.2` are placed top-left into the
83
+ larger v4 tensors; the rest is zero. Old heads contribute exactly the same;
84
+ new heads' attention output is multiplied by zero proj columns, so their
85
+ contribution is zero at step 0.
86
+ - `U` rank 128 → 192 by appending zero columns. `U @ U.T` is unchanged, so the
87
+ M-fold metric is bit-identical.
88
+ - LayerNorm new dims initialise to weight=1, bias=0.
89
+ - Extra layers 25..32 are identity-init: `mha.proj` and `ff.2` are zero, so the
90
+ residual stream passes through unchanged. They start contributing only once
91
+ training shapes them.
92
+
93
+ Net: at step 0, `v4(x) == v3(x)` up to floating-point rounding for any input
94
+ within v3's tokeniser vocabulary. The new capacity has zero impact on day-1
95
+ loss; training subsequently shapes it.
96
+
97
+ Use with `--warmstart_from`:
98
+
99
+ ```bash
100
+ python /workspace/agillm-4/nB300_agillm4.py train \
101
+ --preset large \
102
+ --warmstart_from /workspace/agillm-4/agillm4_seed_from_v3.pt \
103
+ --batch_size 1 --block 1536 --amp --grad_checkpoint --sat_every 1 --nat_every 1 \
104
+ --anchor_memory --anchor_stride 256 --anchor_max 2048
105
+ ```
106
+
107
+ The seed file is ~6 GB on disk (fp32 tensors). Rebuild it whenever a newer v3
108
+ SFT final is preferred (e.g. swap chat-v2 for the math-v1 final).
109
 
110
  ## First Implemented Scaffold
111
 
 
114
  - `--attn_backend manual|sdpa`
115
  - `--grad_checkpoint`
116
  - AGILLM-4 defaults for chat SFT compatibility inherited from the patched AGILLM-3 file.
117
+ - A real NAT head is re-added beside AR and SAT. NAT trains with a CTC loss over an expanded non-causal input (`--nat_expand 2` by default). `--nat_every` controls cadence; `--nat_max_tokens` can cap the NAT slice on 24GB runs without removing the head.
118
 
119
  `block_sweep_agillm4.py` runs synthetic forward/backward sweeps:
120
 
 
273
  - Anchor memory length: 512-4096 anchors.
274
  - Token attention stays cheap while memory covers hundreds of thousands of tokens.
275
 
276
+ Status: wired into `Encoder` as a single `AnchorMemoryLayer` inserted after a configurable mid-stack block. The module is the existing `anchor_memory.AnchorMemoryLayer` (attention-pooled compressor + gated cross-attention read from a rolling bank of `max_anchors` anchors). Enable per-run with:
277
+
278
+ ```bash
279
+ python /workspace/agillm-4/nB300_agillm4.py train \
280
+ --preset large --batch_size 1 --block 1536 --amp --grad_checkpoint \
281
+ --anchor_memory --anchor_stride 256 --anchor_max 2048
282
+ ```
283
+
284
+ `--anchor_position N` overrides the default mid-stack placement. Verifier: `verify_anchor_memory_and_kv_buffer_agillm4.py`.
285
+
286
  ### Compressive Recurrence
287
 
288
  Keep a recurrent memory bank updated by a learned compressor:
N1_HARVEST.md CHANGED
@@ -70,6 +70,27 @@ The verifier checks fused-vs-unfused forward output, loss, input gradients,
70
  parameter gradients, strict legacy state-dict loading, `_safe_load_any`
71
  warm-start loading, and optimizer-state remap.
72
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
73
  ## Next Candidates
74
 
75
  ### 3. Combined ALiBi + Mask Cache
@@ -93,13 +114,6 @@ n1 can compact `U` spectra post-training and save compatible checkpoints.
93
  Risk: optimizer state must be dropped or remapped carefully; do only as a
94
  separate command, never during a live training run.
95
 
96
- ### 6. KV Cache Buffer
97
-
98
- n1 replaces repeated decode-time `torch.cat` cache growth with preallocated KV
99
- buffers.
100
-
101
- Risk: cache object type touches AR, SAT, and future spec decoding paths.
102
-
103
  ## Rule
104
 
105
  Every harvested feature needs its own AGILLM-4 verifier or profile artifact.
 
70
  parameter gradients, strict legacy state-dict loading, `_safe_load_any`
71
  warm-start loading, and optimizer-state remap.
72
 
73
+ ### 3. KV Cache Buffer
74
+
75
+ Status: done.
76
+
77
+ `KVBuffer` is a preallocated head-major `[B, H, T, d_k]` slab for decode-time
78
+ K/V. Callers size it once and call `append(k_new, v_new)` per step instead of
79
+ re-running `torch.cat` and reallocating the cache tensor every token.
80
+ `TuneableAttentionMHA.forward` accepts either a legacy `(k, v)` tuple cache
81
+ (no behaviour change) or a `KVBuffer` instance, and returns the same cache
82
+ object it received so the caller can reuse it across steps. Overflow raises;
83
+ callers must size the buffer to the maximum decode length.
84
+
85
+ Verification:
86
+
87
+ ```bash
88
+ python /workspace/agillm-4/verify_anchor_memory_and_kv_buffer_agillm4.py
89
+ ```
90
+
91
+ The verifier exercises buffer fill, slice view, overflow handling, and confirms
92
+ the legacy tuple-cache code path still works (regression test).
93
+
94
  ## Next Candidates
95
 
96
  ### 3. Combined ALiBi + Mask Cache
 
114
  Risk: optimizer state must be dropped or remapped carefully; do only as a
115
  separate command, never during a live training run.
116
 
 
 
 
 
 
 
 
117
  ## Rule
118
 
119
  Every harvested feature needs its own AGILLM-4 verifier or profile artifact.
README.md CHANGED
@@ -17,7 +17,7 @@ and extended for:
17
  - ~1.5B parameter main preset (`agillm4_main`)
18
  - 100 tokens per parameter target ratio
19
  - longer block-size work on 24GB, B200, and B300 class GPUs
20
- - AR+SAT every step with sequential backward to reduce peak VRAM
21
  - SDPA and experimental sublinear local+landmark attention backends
22
  - exact M-fold expansion attention harvested from n1.py, with local verifier
23
  - fused QKV projection harvested from n1.py, with legacy checkpoint loading
 
17
  - ~1.5B parameter main preset (`agillm4_main`)
18
  - 100 tokens per parameter target ratio
19
  - longer block-size work on 24GB, B200, and B300 class GPUs
20
+ - AR+SAT+NAT training, with sequential backward to reduce peak VRAM
21
  - SDPA and experimental sublinear local+landmark attention backends
22
  - exact M-fold expansion attention harvested from n1.py, with local verifier
23
  - fused QKV projection harvested from n1.py, with legacy checkpoint loading
build_v4_seed.py ADDED
@@ -0,0 +1,337 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Build an AGILLM-4 warm-start "seed" from an AGILLM-3 checkpoint.
3
+
4
+ The seed is function-preserving at step 0: v4 forward pass with this seed
5
+ produces (up to floating-point rounding) the same logits v3 produced for the
6
+ same inputs. The new width / heads / layers add capacity that contributes zero
7
+ at init and gets shaped by training from a useful starting point — far better
8
+ than a random init for the post-SFT v3 lineage.
9
+
10
+ Construction rules (v3 d_v3/h_v3/rank_v3 → v4 d_v4/h_v4/rank_v4):
11
+
12
+ emb [V, d_v3] → [V, d_v4]; new dims = 0
13
+ ln.weight/bias [d_v3] → [d_v4]; new weight=1, new bias=0
14
+ mha.qkv [3*d_v3, d_v3] → [3*d_v4, d_v4] per-slab top-left copy, rest 0
15
+ (new dims see 0 input; new heads' rows are 0)
16
+ mha.proj [d_v3, hdk_v3] → [d_v4, hdk_v4] top-left copy, rest 0
17
+ (new heads' columns zero → contribute nothing)
18
+ mha.U [dk, rank_v3] → [dk, rank_v4] left columns copy, rest 0
19
+ (U @ U.T preserved exactly)
20
+ ff.0 [4d_v3, d_v3] → [4d_v4, d_v4] top-left copy, rest 0
21
+ ff.2 [d_v3, 4d_v3] → [d_v4, 4d_v4] top-left copy, rest 0
22
+ ar/nat.proj [V, d_v3] → [V, d_v4]; new dim columns = 0
23
+
24
+ Layers (24 → 32): keep the v3 stack in place at the bottom of the v4 stack;
25
+ append 8 identity-init blocks at the top. Identity init: ln{1,2}=I,
26
+ mha.proj=0, ff.2=0 → output equals input (residual passthrough).
27
+ """
28
+ from __future__ import annotations
29
+
30
+ import argparse
31
+ import json
32
+ import pathlib
33
+ import sys
34
+ import time
35
+
36
+ import torch
37
+ import torch.nn as nn
38
+
39
+ # v4 main preset (matches agillm4_config.json)
40
+ V4_PRESET = dict(d=1536, layers=32, heads=24, rank=192)
41
+ # v3 from log header / slot 200
42
+ V3_PRESET = dict(d=1024, layers=24, heads=16, rank=128)
43
+ DK = 64 # both share dk; sanity-checked below
44
+
45
+
46
+ def expand_emb(w_v3: torch.Tensor, d_v4: int) -> torch.Tensor:
47
+ V, d_v3 = w_v3.shape
48
+ out = torch.zeros(V, d_v4, dtype=w_v3.dtype)
49
+ out[:, :d_v3] = w_v3
50
+ return out
51
+
52
+
53
+ def expand_ln(w_v3: torch.Tensor, b_v3: torch.Tensor, d_v4: int):
54
+ new_w = torch.ones(d_v4, dtype=w_v3.dtype)
55
+ new_w[: w_v3.numel()] = w_v3
56
+ new_b = torch.zeros(d_v4, dtype=b_v3.dtype)
57
+ new_b[: b_v3.numel()] = b_v3
58
+ return new_w, new_b
59
+
60
+
61
+ def expand_qkv(qkv_v3: torch.Tensor, d_v3: int, d_v4: int, h_v3: int, h_v4: int, dk: int) -> torch.Tensor:
62
+ """[3*d_v3, d_v3] → [3*d_v4, d_v4] via per-slab top-left placement."""
63
+ assert qkv_v3.shape == (3 * d_v3, d_v3), qkv_v3.shape
64
+ Q, K, V = qkv_v3[:d_v3], qkv_v3[d_v3 : 2 * d_v3], qkv_v3[2 * d_v3 :]
65
+ out = torch.zeros(3 * d_v4, d_v4, dtype=qkv_v3.dtype)
66
+ for i, slab in enumerate((Q, K, V)):
67
+ # slab is [h_v3 * dk, d_v3]
68
+ out[i * d_v4 : i * d_v4 + h_v3 * dk, :d_v3] = slab
69
+ return out
70
+
71
+
72
+ def expand_proj(proj_v3: torch.Tensor, d_v3: int, d_v4: int, h_v3: int, h_v4: int, dk: int) -> torch.Tensor:
73
+ """[d_v3, h_v3*dk] → [d_v4, h_v4*dk] top-left placement."""
74
+ out = torch.zeros(d_v4, h_v4 * dk, dtype=proj_v3.dtype)
75
+ out[:d_v3, : h_v3 * dk] = proj_v3
76
+ return out
77
+
78
+
79
+ def expand_U(u_v3: torch.Tensor, dk: int, rank_v4: int) -> torch.Tensor:
80
+ """[dk, rank_v3] → [dk, rank_v4] via left-column placement.
81
+
82
+ Critical: rank_v3 columns are preserved, new columns are zero.
83
+ Then U_new @ U_new.T == U_v3 @ U_v3.T exactly. M-fold metric is unchanged.
84
+ """
85
+ out = torch.zeros(dk, rank_v4, dtype=u_v3.dtype)
86
+ out[:, : u_v3.size(1)] = u_v3
87
+ return out
88
+
89
+
90
+ def expand_ff_first(w_v3: torch.Tensor, d_v3: int, d_v4: int) -> torch.Tensor:
91
+ """[4*d_v3, d_v3] → [4*d_v4, d_v4]."""
92
+ out = torch.zeros(4 * d_v4, d_v4, dtype=w_v3.dtype)
93
+ out[: 4 * d_v3, :d_v3] = w_v3
94
+ return out
95
+
96
+
97
+ def expand_ff_first_bias(b_v3: torch.Tensor, d_v4: int) -> torch.Tensor:
98
+ out = torch.zeros(4 * d_v4, dtype=b_v3.dtype)
99
+ out[: b_v3.numel()] = b_v3
100
+ return out
101
+
102
+
103
+ def expand_ff_second(w_v3: torch.Tensor, d_v3: int, d_v4: int) -> torch.Tensor:
104
+ """[d_v3, 4*d_v3] → [d_v4, 4*d_v4]."""
105
+ out = torch.zeros(d_v4, 4 * d_v4, dtype=w_v3.dtype)
106
+ out[:d_v3, : 4 * d_v3] = w_v3
107
+ return out
108
+
109
+
110
+ def expand_ff_second_bias(b_v3: torch.Tensor, d_v4: int) -> torch.Tensor:
111
+ out = torch.zeros(d_v4, dtype=b_v3.dtype)
112
+ out[: b_v3.numel()] = b_v3
113
+ return out
114
+
115
+
116
+ def expand_arhead(w_v3: torch.Tensor, b_v3: torch.Tensor | None, d_v4: int):
117
+ V, d_v3 = w_v3.shape
118
+ out_w = torch.zeros(V, d_v4, dtype=w_v3.dtype)
119
+ out_w[:, :d_v3] = w_v3
120
+ out_b = None
121
+ if b_v3 is not None:
122
+ out_b = b_v3.clone() # bias dim is V — does not change with d
123
+ return out_w, out_b
124
+
125
+
126
+ def identity_block_state(d_v4: int, h_v4: int, rank_v4: int, dk: int, prefix: str):
127
+ """A v4 block initialized so its residual contribution is exactly zero."""
128
+ sd = {
129
+ f"{prefix}.ln1.weight": torch.ones(d_v4),
130
+ f"{prefix}.ln1.bias": torch.zeros(d_v4),
131
+ f"{prefix}.ln2.weight": torch.ones(d_v4),
132
+ f"{prefix}.ln2.bias": torch.zeros(d_v4),
133
+ # mha: qkv arbitrary (we zero it), U orthogonal, proj zero -> attention contributes 0
134
+ f"{prefix}.mha.qkv.weight": torch.zeros(3 * d_v4, d_v4),
135
+ f"{prefix}.mha.U": _orthogonal(dk, rank_v4),
136
+ f"{prefix}.mha.proj.weight": torch.zeros(d_v4, h_v4 * dk),
137
+ # ff: first layer small random so gradient signal exists; second zero -> contributes 0
138
+ f"{prefix}.ff.0.weight": torch.empty(4 * d_v4, d_v4).normal_(0, 0.02),
139
+ f"{prefix}.ff.0.bias": torch.zeros(4 * d_v4),
140
+ f"{prefix}.ff.2.weight": torch.zeros(d_v4, 4 * d_v4),
141
+ f"{prefix}.ff.2.bias": torch.zeros(d_v4),
142
+ }
143
+ return sd
144
+
145
+
146
+ def _orthogonal(dk: int, rank: int) -> torch.Tensor:
147
+ t = torch.empty(dk, rank)
148
+ nn.init.orthogonal_(t)
149
+ return t
150
+
151
+
152
+ def find_default_v3_ckpt() -> pathlib.Path:
153
+ candidates = [
154
+ pathlib.Path("/workspace/ckpts_sft_math_v1/final.pt"), # math SFT final (latest)
155
+ pathlib.Path("/workspace/ckpts_sft_chat_v2/final.pt"), # chat SFT v2 final
156
+ pathlib.Path("/workspace/ckpts_sft_chat/final.pt"), # chat SFT v1 final
157
+ ]
158
+ for c in candidates:
159
+ if c.exists():
160
+ return c
161
+ raise SystemExit("No v3 final checkpoint found in expected locations.")
162
+
163
+
164
+ def main() -> int:
165
+ p = argparse.ArgumentParser(description="Build a v4 warm-start seed from v3.")
166
+ p.add_argument("--from-ckpt", type=pathlib.Path, default=None,
167
+ help="v3 checkpoint .pt (default: most recent in /workspace/ckpts_sft_*)")
168
+ p.add_argument("--out", type=pathlib.Path,
169
+ default=pathlib.Path("/workspace/agillm-4/agillm4_seed_from_v3.pt"))
170
+ p.add_argument("--v4-preset", type=str, default="main",
171
+ choices=["main", "floor", "big"])
172
+ args = p.parse_args()
173
+
174
+ src = args.from_ckpt or find_default_v3_ckpt()
175
+ print(f"v3 source : {src}")
176
+ if not src.exists():
177
+ print(f"FATAL: source missing: {src}", file=sys.stderr)
178
+ return 2
179
+
180
+ v4 = {"main": V4_PRESET,
181
+ "floor": dict(d=1280, layers=28, heads=20, rank=160),
182
+ "big": dict(d=1792, layers=36, heads=28, rank=224)}[args.v4_preset]
183
+ v3 = V3_PRESET
184
+ dk_v3 = v3["d"] // v3["heads"]
185
+ dk_v4 = v4["d"] // v4["heads"]
186
+ assert dk_v3 == dk_v4 == DK, f"dk mismatch: v3={dk_v3} v4={dk_v4} DK={DK}"
187
+ print(f"v3 preset : {v3}")
188
+ print(f"v4 preset : {v4}")
189
+
190
+ t0 = time.time()
191
+ blob = torch.load(src, map_location="cpu", weights_only=False)
192
+ print(f"loaded ckpt keys={list(blob.keys())} in {time.time()-t0:.1f}s")
193
+
194
+ core_v3 = blob.get("core", blob)
195
+ ar_v3 = blob.get("ar", None)
196
+ nat_v3 = blob.get("nat", None)
197
+ sat_v3 = blob.get("sat", None)
198
+
199
+ # === core (Encoder) ===
200
+ core_v4 = {}
201
+ # Embedding
202
+ if "emb.weight" in core_v3:
203
+ core_v4["emb.weight"] = expand_emb(core_v3["emb.weight"], v4["d"])
204
+ print(f"emb.weight : {tuple(core_v3['emb.weight'].shape)} -> {tuple(core_v4['emb.weight'].shape)}")
205
+ else:
206
+ print("WARN: emb.weight not found in v3 core; v4 emb will be random later.")
207
+
208
+ # Per-block expansion: keep v3 blocks at positions 0..L_v3-1
209
+ n_v3 = v3["layers"]
210
+ n_v4 = v4["layers"]
211
+ n_new = n_v4 - n_v3
212
+ for i in range(n_v3):
213
+ pfx_v3 = f"blocks.{i}"
214
+ pfx_v4 = f"blocks.{i}"
215
+ # ln1
216
+ if f"{pfx_v3}.ln1.weight" in core_v3:
217
+ w, b = expand_ln(core_v3[f"{pfx_v3}.ln1.weight"], core_v3[f"{pfx_v3}.ln1.bias"], v4["d"])
218
+ core_v4[f"{pfx_v4}.ln1.weight"] = w
219
+ core_v4[f"{pfx_v4}.ln1.bias"] = b
220
+ if f"{pfx_v3}.ln2.weight" in core_v3:
221
+ w, b = expand_ln(core_v3[f"{pfx_v3}.ln2.weight"], core_v3[f"{pfx_v3}.ln2.bias"], v4["d"])
222
+ core_v4[f"{pfx_v4}.ln2.weight"] = w
223
+ core_v4[f"{pfx_v4}.ln2.bias"] = b
224
+ # qkv (might be fused or legacy q/k/v)
225
+ if f"{pfx_v3}.mha.qkv.weight" in core_v3:
226
+ qkv = core_v3[f"{pfx_v3}.mha.qkv.weight"]
227
+ elif all(f"{pfx_v3}.mha.{k}.weight" in core_v3 for k in ("q", "k", "v")):
228
+ qkv = torch.cat([core_v3[f"{pfx_v3}.mha.{k}.weight"] for k in ("q", "k", "v")], dim=0)
229
+ else:
230
+ qkv = None
231
+ if qkv is not None:
232
+ core_v4[f"{pfx_v4}.mha.qkv.weight"] = expand_qkv(qkv, v3["d"], v4["d"], v3["heads"], v4["heads"], DK)
233
+ # U
234
+ if f"{pfx_v3}.mha.U" in core_v3:
235
+ core_v4[f"{pfx_v4}.mha.U"] = expand_U(core_v3[f"{pfx_v3}.mha.U"], DK, v4["rank"])
236
+ # proj
237
+ if f"{pfx_v3}.mha.proj.weight" in core_v3:
238
+ core_v4[f"{pfx_v4}.mha.proj.weight"] = expand_proj(
239
+ core_v3[f"{pfx_v3}.mha.proj.weight"], v3["d"], v4["d"], v3["heads"], v4["heads"], DK
240
+ )
241
+ # ff (Sequential(Linear, ReLU, Linear) → keys ff.0.* and ff.2.*)
242
+ if f"{pfx_v3}.ff.0.weight" in core_v3:
243
+ core_v4[f"{pfx_v4}.ff.0.weight"] = expand_ff_first(core_v3[f"{pfx_v3}.ff.0.weight"], v3["d"], v4["d"])
244
+ if f"{pfx_v3}.ff.0.bias" in core_v3:
245
+ core_v4[f"{pfx_v4}.ff.0.bias"] = expand_ff_first_bias(core_v3[f"{pfx_v3}.ff.0.bias"], v4["d"])
246
+ if f"{pfx_v3}.ff.2.weight" in core_v3:
247
+ core_v4[f"{pfx_v4}.ff.2.weight"] = expand_ff_second(core_v3[f"{pfx_v3}.ff.2.weight"], v3["d"], v4["d"])
248
+ if f"{pfx_v3}.ff.2.bias" in core_v3:
249
+ core_v4[f"{pfx_v4}.ff.2.bias"] = expand_ff_second_bias(core_v3[f"{pfx_v3}.ff.2.bias"], v4["d"])
250
+
251
+ # Identity-init the new top-of-stack blocks
252
+ for j in range(n_new):
253
+ i_v4 = n_v3 + j
254
+ core_v4.update(identity_block_state(v4["d"], v4["heads"], v4["rank"], DK, prefix=f"blocks.{i_v4}"))
255
+
256
+ # Final LN
257
+ if "ln.weight" in core_v3:
258
+ w, b = expand_ln(core_v3["ln.weight"], core_v3["ln.bias"], v4["d"])
259
+ core_v4["ln.weight"] = w
260
+ core_v4["ln.bias"] = b
261
+
262
+ # === AR head ===
263
+ ar_v4 = None
264
+ if ar_v3:
265
+ ar_v4 = {}
266
+ if "proj.weight" in ar_v3:
267
+ w, b = expand_arhead(ar_v3["proj.weight"], ar_v3.get("proj.bias"), v4["d"])
268
+ ar_v4["proj.weight"] = w
269
+ if b is not None:
270
+ ar_v4["proj.bias"] = b
271
+
272
+ # === NAT head ===
273
+ # Prefer a source NAT head when available. AGILLM-3 checkpoints usually
274
+ # have no NAT head, so seed NAT from expanded AR rather than random init.
275
+ nat_v4 = None
276
+ if nat_v3 and "proj.weight" in nat_v3:
277
+ nat_v4 = {}
278
+ w, b = expand_arhead(nat_v3["proj.weight"], nat_v3.get("proj.bias"), v4["d"])
279
+ nat_v4["proj.weight"] = w
280
+ if b is not None:
281
+ nat_v4["proj.bias"] = b
282
+ elif ar_v4:
283
+ nat_v4 = {k: v.clone() if torch.is_tensor(v) else v for k, v in ar_v4.items()}
284
+
285
+ # === SAT head ===
286
+ # Keep v3's SAT state if present and re-shape only the d-axis tensors;
287
+ # SAT internals vary, so we copy what fits and let _safe_load_any drop the rest.
288
+ sat_v4 = None
289
+ if sat_v3:
290
+ sat_v4 = {}
291
+ for k, t in sat_v3.items():
292
+ if isinstance(t, torch.Tensor):
293
+ # Pad any dimension equal to d_v3 up to d_v4 with zeros
294
+ pads = []
295
+ new_shape = list(t.shape)
296
+ changed = False
297
+ for dim_i, sz in enumerate(t.shape):
298
+ if sz == v3["d"]:
299
+ new_shape[dim_i] = v4["d"]
300
+ changed = True
301
+ if changed:
302
+ out = torch.zeros(*new_shape, dtype=t.dtype)
303
+ slices = tuple(slice(0, s) for s in t.shape)
304
+ out[slices] = t
305
+ sat_v4[k] = out
306
+ else:
307
+ sat_v4[k] = t.clone()
308
+ else:
309
+ sat_v4[k] = t
310
+
311
+ # Save bundle
312
+ out_blob = {"core": core_v4}
313
+ if ar_v4: out_blob["ar"] = ar_v4
314
+ if nat_v4: out_blob["nat"] = nat_v4
315
+ if sat_v4: out_blob["sat"] = sat_v4
316
+ out_blob["seed_meta"] = {
317
+ "built_from": str(src),
318
+ "v3_preset": v3,
319
+ "v4_preset_name": args.v4_preset,
320
+ "v4_preset": v4,
321
+ "n_new_layers": n_new,
322
+ "tool": pathlib.Path(__file__).name,
323
+ "tool_version": 2,
324
+ }
325
+
326
+ args.out.parent.mkdir(parents=True, exist_ok=True)
327
+ t0 = time.time()
328
+ torch.save(out_blob, args.out)
329
+ sz = args.out.stat().st_size
330
+ print(f"saved seed : {args.out} ({sz/1e9:.2f} GB) in {time.time()-t0:.1f}s")
331
+ print(f"core keys : {len(core_v4)} (expected ~{n_v4 * 9 + 3})")
332
+ print(f"meta : {json.dumps(out_blob['seed_meta'], indent=2)}")
333
+ return 0
334
+
335
+
336
+ if __name__ == "__main__":
337
+ sys.exit(main())
nB300_agillm4.py CHANGED
@@ -1,6 +1,6 @@
1
  #!/usr/bin/env python3
2
 
3
- # n.py — Joint AR+SAT Trainer with Expansion Ratio Testing
4
  # Enhanced inference: checkpoint name, tok/s, UK time
5
 
6
  from __future__ import annotations
@@ -586,6 +586,7 @@ _maybe_handle_status_fastpath()
586
 
587
  import torch
588
  import torch.utils.checkpoint as torch_checkpoint
 
589
 
590
  # SafeProgress - Claude-safe progress (discrete lines, not single growing line)
591
  class SafeProgress:
@@ -684,6 +685,7 @@ SYNTHETIC_TOKENIZER = os.environ.get("AGILLM_SYNTHETIC_TOKENIZER", "").lower() i
684
 
685
  class _SyntheticTokenizer:
686
  pad_token = "<|pad|>"
 
687
  eos_token_id = 1
688
  sep_token_id = 1
689
 
@@ -796,8 +798,9 @@ def _tokenizer_health_check(tokenizer):
796
  if not SYNTHETIC_TOKENIZER:
797
  _tokenizer_health_check(tok)
798
 
799
- VOCAB, EOS = (
800
  max(tok.get_vocab().values()) + 1,
 
801
  tok.eos_token_id if tok.eos_token_id is not None else tok.sep_token_id
802
  )
803
 
@@ -861,6 +864,11 @@ DEFAULT_SUBLINEAR_WINDOW = _env_int("AGILLM_SUBLINEAR_WINDOW", 256)
861
  DEFAULT_SUBLINEAR_STRIDE = _env_int("AGILLM_SUBLINEAR_STRIDE", 64)
862
  DEFAULT_SUBLINEAR_MAX_ANCHORS = _env_int("AGILLM_SUBLINEAR_MAX_ANCHORS", 256)
863
  DEFAULT_SUBLINEAR_CHUNK = _env_int("AGILLM_SUBLINEAR_CHUNK", 128)
 
 
 
 
 
864
  AGILLM4_TOKEN_PARAM_RATIO = 100.0
865
 
866
  # ───────────────────────── UK Time Helper ─────────────────────────
@@ -1125,6 +1133,46 @@ def alibi_bias(n_heads: int, n_tokens: int):
1125
  return -_alibi_slopes(n_heads) * dist
1126
 
1127
  # ───────────────────────── Model components ─────────────────────────
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1128
  class TuneableAttentionMHA(nn.Module):
1129
  def __init__(
1130
  self,
@@ -1302,8 +1350,14 @@ class TuneableAttentionMHA(nn.Module):
1302
  else:
1303
  q = self._proj_qk(q_lin)
1304
  k_new = self._proj_qk(k_lin)
1305
- if kv_cache is None:
1306
  k, v = k_new, v_new
 
 
 
 
 
 
1307
  else:
1308
  k_cached, v_cached = kv_cache
1309
  if use_cache:
@@ -1337,7 +1391,10 @@ class TuneableAttentionMHA(nn.Module):
1337
  z = att.softmax(-1) @ v
1338
  z = z.transpose(1, 2).reshape(x.size(0), x.size(1), -1)
1339
  out = self.drop(self.proj(z))
1340
- return (out, (k, v)) if use_cache else out
 
 
 
1341
 
1342
 
1343
  class Block(nn.Module):
@@ -1388,6 +1445,10 @@ class Encoder(nn.Module):
1388
  sublinear_stride: int = DEFAULT_SUBLINEAR_STRIDE,
1389
  sublinear_max_anchors: int = DEFAULT_SUBLINEAR_MAX_ANCHORS,
1390
  sublinear_chunk: int = DEFAULT_SUBLINEAR_CHUNK,
 
 
 
 
1391
  ):
1392
  super().__init__()
1393
  d, l, h, r = cfg["d"], cfg["layers"], cfg["heads"], cfg["rank"]
@@ -1413,21 +1474,46 @@ class Encoder(nn.Module):
1413
  self.sublinear_stride = sublinear_stride
1414
  self.sublinear_max_anchors = sublinear_max_anchors
1415
  self.sublinear_chunk = sublinear_chunk
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1416
 
1417
  def forward(self, ids, mask, kv_caches=None, use_cache=False, total_seq_len=None):
1418
  x = self.emb(ids)
1419
  if not use_cache:
1420
- for blk in self.blocks:
1421
  if self.grad_checkpoint and self.training:
1422
  x = torch_checkpoint.checkpoint(blk, x, mask, use_reentrant=False)
1423
  else:
1424
  x = blk(x, mask)
 
 
 
 
 
1425
  return self.ln(x)
1426
  new_kvs = []
1427
  for i, blk in enumerate(self.blocks):
1428
  kv = kv_caches[i] if kv_caches else None
1429
  x, kv_out = blk(x, mask, kv, use_cache=True, total_seq_len=total_seq_len)
1430
  new_kvs.append(kv_out)
 
 
1431
  return self.ln(x), new_kvs
1432
 
1433
 
@@ -1445,6 +1531,15 @@ class ARHead(nn.Module):
1445
  return self.proj(h)
1446
 
1447
 
 
 
 
 
 
 
 
 
 
1448
  class SATHead(nn.Module):
1449
  def __init__(self, d, mode="var"):
1450
  super().__init__()
@@ -1567,15 +1662,17 @@ def _clone_opt_value(value):
1567
  return value.detach().clone()
1568
  return copy.deepcopy(value)
1569
 
1570
- def _optimizer_param_name_lookup(core, ar_h, sat_h) -> dict[int, str]:
1571
  out = {}
1572
- for prefix, module in (("core", core), ("ar", ar_h), ("sat", sat_h)):
 
 
1573
  for name, param in module.named_parameters():
1574
  out.setdefault(id(param), f"{prefix}.{name}")
1575
  return out
1576
 
1577
- def _optimizer_group_param_names(opt, core, ar_h, sat_h) -> List[List[str]]:
1578
- lookup = _optimizer_param_name_lookup(core, ar_h, sat_h)
1579
  return [
1580
  [lookup.get(id(param), f"<unknown:{id(param)}>") for param in group["params"]]
1581
  for group in opt.param_groups
@@ -1606,12 +1703,12 @@ def _fuse_legacy_optimizer_param_state(states: List[dict]) -> Optional[dict]:
1606
  out[key] = copy.deepcopy(vals[0])
1607
  return out
1608
 
1609
- def _fuse_legacy_qkv_optimizer_state(opt_state: dict, opt, core, ar_h, sat_h) -> Optional[dict]:
1610
  """Remap pre-QKV-fusion AdamW state to the current fused parameter layout."""
1611
  if not isinstance(opt_state, dict) or "state" not in opt_state or "param_groups" not in opt_state:
1612
  return None
1613
  current_sd = opt.state_dict()
1614
- current_names = _optimizer_group_param_names(opt, core, ar_h, sat_h)
1615
  legacy_names = [
1616
  [legacy for name in group_names for legacy in _legacy_names_for_current_param(name)]
1617
  for group_names in current_names
@@ -1653,7 +1750,7 @@ def _fuse_legacy_qkv_optimizer_state(opt_state: dict, opt, core, ar_h, sat_h) ->
1653
 
1654
  return {"state": new_states, "param_groups": new_groups}
1655
 
1656
- def save_delta(core, ar_h, sat_h, step: int, seen_tok: int, save_dir: pathlib.Path, phase_name: str):
1657
  """Save weight-only delta in background thread. Non-blocking."""
1658
  global _delta_thread
1659
  # Wait for any previous delta write to finish
@@ -1666,6 +1763,8 @@ def save_delta(core, ar_h, sat_h, step: int, seen_tok: int, save_dir: pathlib.Pa
1666
  "ar": {k: v.detach().cpu() for k, v in _checkpoint_state_dict(ar_h).items()},
1667
  "sat": {k: v.detach().cpu() for k, v in _checkpoint_state_dict(sat_h).items()},
1668
  }
 
 
1669
  meta = {"step": step, "seen_tok": seen_tok, "wall_time": time.time(), "delta": True}
1670
  path = save_dir / f"{phase_name}_delta_step{step:08d}.pt"
1671
  _delta_thread = threading.Thread(target=_do_delta_save, args=(tensors, path, meta), daemon=True)
@@ -1694,7 +1793,7 @@ def _prune_deltas(save_dir: pathlib.Path, phase_name: str, max_deltas: int):
1694
  except Exception as e:
1695
  print(f" [delta-prune] error: {e}")
1696
 
1697
- def load_delta(path: pathlib.Path, core, ar_h, sat_h):
1698
  """Load weight-only delta. Returns (step, seen_tok) or raises."""
1699
  # Verify checksum if sidecar exists
1700
  sha_path = path.with_suffix(".sha256")
@@ -1710,6 +1809,12 @@ def load_delta(path: pathlib.Path, core, ar_h, sat_h):
1710
  core.load_state_dict(_fuse_qkv_in_state_dict(_strip_orig_mod_prefix(ck["weights"]["core"])))
1711
  ar_h.load_state_dict(_strip_orig_mod_prefix(ck["weights"]["ar"]))
1712
  sat_h.load_state_dict(_strip_orig_mod_prefix(ck["weights"]["sat"]))
 
 
 
 
 
 
1713
  return ck.get("step", 0), ck.get("seen_tok", 0)
1714
 
1715
  def _flush_delta():
@@ -1719,7 +1824,7 @@ def _flush_delta():
1719
  print(" [delta] flushing in-flight write...")
1720
  _delta_thread.join(timeout=120)
1721
 
1722
- def save_ckpt(path: pathlib.Path, core, ar_h, sat_h, opt, scaler, meta):
1723
  path.parent.mkdir(exist_ok=True, parents=True)
1724
  tmp = path.with_suffix(path.suffix + ".tmp")
1725
  state = {
@@ -1732,22 +1837,29 @@ def save_ckpt(path: pathlib.Path, core, ar_h, sat_h, opt, scaler, meta):
1732
  "tie_weights": meta.get("tie_weights", False),
1733
  **{k: v for k, v in meta.items() if k not in ("cfg", "tie_weights")}
1734
  }
 
 
1735
  torch.save(state, tmp, _use_new_zipfile_serialization=False)
1736
  tmp.replace(path)
1737
  (path.parent / "latest.json").write_text(json.dumps({"path": str(path), "step": meta["step"]}))
1738
  print(f"\n✓ saved checkpoint {path.name}")
1739
 
1740
- def load_ckpt(path, core, ar_h, sat_h, opt, scaler):
1741
  p = _resolve_ckpt(path) or path
1742
  ck = _try_load(p, map_location="cpu")
1743
  if ck is None: raise FileNotFoundError(f"No valid checkpoint at {p}")
1744
  core.load_state_dict(_fuse_qkv_in_state_dict(_strip_orig_mod_prefix(ck["core"])))
1745
  ar_h.load_state_dict(_strip_orig_mod_prefix(ck["ar"]))
1746
  sat_h.load_state_dict(_strip_orig_mod_prefix(ck["sat"]))
 
 
 
 
 
1747
  try:
1748
  opt.load_state_dict(ck["opt"])
1749
  except Exception as exc:
1750
- fused_opt = _fuse_legacy_qkv_optimizer_state(ck.get("opt"), opt, core, ar_h, sat_h)
1751
  if fused_opt is not None:
1752
  try:
1753
  opt.load_state_dict(fused_opt)
@@ -1834,15 +1946,18 @@ def _phase_freeze(core: nn.Module, *, freeze_core: bool, unfreeze_ln: bool, trai
1834
  if train_emb:
1835
  for p in core.emb.parameters(): p.requires_grad = True
1836
 
1837
- def _optimizer_param_groups(core, ar_h, sat_h, lr_core: float, lr_head: float):
1838
- return [
1839
  {"params": [p for p in core.parameters() if p.requires_grad], "lr": lr_core},
1840
  {"params": ar_h.parameters(), "lr": lr_head},
1841
  {"params": sat_h.parameters(), "lr": lr_head},
1842
  ]
 
 
 
1843
 
1844
- def make_optimizer(args, core, ar_h, sat_h, lr_core: float, lr_head: float):
1845
- groups = _optimizer_param_groups(core, ar_h, sat_h, lr_core, lr_head)
1846
  opt_name = getattr(args, "optimizer", "adamw")
1847
  if opt_name == "adamw":
1848
  return torch.optim.AdamW(groups)
@@ -1858,9 +1973,14 @@ def make_optimizer(args, core, ar_h, sat_h, lr_core: float, lr_head: float):
1858
  return bnb.optim.AdamW8bit(groups)
1859
  raise ValueError(f"unknown optimizer: {opt_name}")
1860
 
 
 
 
 
 
1861
  def _train_phase(
1862
  args, phase_name: str,
1863
- core, ar_h, sat_h, opt, scaler,
1864
  start_step, seen_tok, resume_wall_time,
1865
  cfg, source, steps, block_size, batch_size,
1866
  chat_cfg: dict,
@@ -1875,7 +1995,7 @@ def _train_phase(
1875
  target_tokens = target_tokens_override
1876
  else:
1877
  ratio = _target_token_ratio(args)
1878
- param_count = _count_enabled_params(core, ar_h, sat_h)
1879
  target_tokens = int(ratio * param_count)
1880
  print(f"[{phase_name}] token_param_ratio={ratio:g} param_count={param_count:,} target_tokens={target_tokens:,}")
1881
  if steps:
@@ -1896,6 +2016,7 @@ def _train_phase(
1896
  )
1897
  ce_tok = nn.CrossEntropyLoss(label_smoothing=0.1)
1898
  ce_gate = nn.CrossEntropyLoss()
 
1899
  pbar = SafeProgress(total=total_tokens_needed, initial=seen_tok, unit="tok")
1900
  grow_plan = _parse_grow_plan(args.grow_plan) if args.auto_grow else []
1901
  buf: list[int] = []
@@ -1908,7 +2029,10 @@ def _train_phase(
1908
  last_save_mono = time.monotonic() - (now_wall - (resume_wall_time or now_wall))
1909
  last_delta_step = start_step
1910
  print(f"[{phase_name}] Starting. Goal: {total_tokens_needed:,} tokens. Batch={BATCH}, Block={BLOCK}")
1911
- print(f"[{phase_name}] AR_ONLY={args.ar_only}, SAT_EVERY={args.sat_every}, TIE_WEIGHTS={tie_weights}, STREAMING={streaming}")
 
 
 
1912
  while seen_tok < total_tokens_needed:
1913
  try:
1914
  while len(buf) < BLOCK:
@@ -1945,8 +2069,28 @@ def _train_phase(
1945
  loss_value += float(loss_sat.detach().item())
1946
  scaler.scale(loss_sat).backward()
1947
  del h_sat, logits_sat, loss_sat
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1948
  scaler.unscale_(opt)
1949
- nn.utils.clip_grad_norm_(core.parameters(), 1.0)
1950
  scaler.step(opt)
1951
  scaler.update()
1952
  opt.zero_grad(set_to_none=True)
@@ -1999,7 +2143,7 @@ def _train_phase(
1999
  ck_name = f"{phase_name}_step{step:08d}.pt"
2000
  _flush_delta() # wait for any in-flight delta before full save
2001
  _prune_checkpoints(pathlib.Path(args.save_dir), phase_name, max_ckpts)
2002
- save_ckpt(pathlib.Path(args.save_dir) / ck_name, core, ar_h, sat_h, opt, scaler,
2003
  meta={"cfg": cfg, "step": step, "seen_tok": seen_tok, "wall_time": time.time(), "tie_weights": tie_weights})
2004
  last_save_mono = now_mono
2005
  # Prune old deltas after a full save (they're superseded)
@@ -2008,7 +2152,7 @@ def _train_phase(
2008
  # ── Delta checkpoint (step-based, weight-only, async) ──
2009
  if args.delta_every_steps > 0 and (step - last_delta_step) >= args.delta_every_steps:
2010
  _prune_deltas(pathlib.Path(args.save_dir), phase_name, args.delta_max_keep)
2011
- save_delta(core, ar_h, sat_h, step, seen_tok, pathlib.Path(args.save_dir), phase_name)
2012
  last_delta_step = step
2013
  if args.auto_grow:
2014
  steps_since_last_grow += 1
@@ -2025,7 +2169,7 @@ def _train_phase(
2025
  pbar.close()
2026
  _flush_delta() # ensure any in-flight delta completes before final save
2027
  if phase_name != "sft":
2028
- save_ckpt(pathlib.Path(args.save_dir) / f"{phase_name}_final.pt", core, ar_h, sat_h, opt, scaler,
2029
  meta={"cfg": cfg, "step": step, "seen_tok": seen_tok, "wall_time": time.time(), "tie_weights": tie_weights})
2030
  else:
2031
  print("[sft] Skipping duplicate sft_final.pt; final.pt will contain the SFT result.")
@@ -2062,10 +2206,15 @@ def train(args):
2062
  sublinear_stride=args.sublinear_stride,
2063
  sublinear_max_anchors=args.sublinear_max_anchors,
2064
  sublinear_chunk=args.sublinear_chunk,
 
 
 
 
2065
  ).to(DEV)
2066
  ar_h = ARHead(cfg["d"], tie_weights=tie_weights, embedding_weight=core.emb.weight if tie_weights else None).to(DEV)
2067
  sat_h = SATHead(cfg["d"], mode="var").to(DEV)
2068
- total_params = _count_enabled_params(core, ar_h, sat_h)
 
2069
  print(f"Total parameters: {total_params:,}")
2070
  if tie_weights:
2071
  print(f"{Colors.WARN}[weight-tying] Embedding and LM head share weights{Colors.RESET}")
@@ -2076,17 +2225,20 @@ def train(args):
2076
  loaded = _safe_load_any(src, core, key="core")
2077
  _safe_load_any(src, ar_h, key="ar")
2078
  _safe_load_any(src, sat_h, key="sat")
 
 
 
2079
  if loaded: print(f"Warm-start loaded from {src}")
2080
  _phase_freeze(core, freeze_core=args.freeze_core, unfreeze_ln=args.unfreeze_ln, train_emb=args.train_emb)
2081
- opt = make_optimizer(args, core, ar_h, sat_h, args.lr_core, args.lr_head)
2082
  scaler = GradScaler(enabled=(args.amp and _needs_grad_scaler()))
2083
  start_step, seen_tok, last_wall = 0, 0, None
2084
  if args.resume_delta and not args.fresh:
2085
- delta_step, delta_tok = load_delta(pathlib.Path(args.resume_delta), core, ar_h, sat_h)
2086
  start_step, seen_tok, last_wall = delta_step, delta_tok, None
2087
  print(f"Resumed from DELTA at step {start_step} (optimizer state reset — momentum rebuilds in ~100 steps)")
2088
  elif args.resume and not args.fresh:
2089
- start_step, seen_tok, last_wall = load_ckpt(pathlib.Path(args.resume), core, ar_h, sat_h, opt, scaler)
2090
  print(f"Resumed from step {start_step}")
2091
  # torch.compile AFTER loading checkpoint (key names differ)
2092
  if args.compile:
@@ -2094,9 +2246,10 @@ def train(args):
2094
  core = torch.compile(core, mode="reduce-overhead")
2095
  ar_h = torch.compile(ar_h, mode="reduce-overhead")
2096
  sat_h = torch.compile(sat_h, mode="reduce-overhead")
 
2097
  print("[torch.compile] Done.")
2098
  step, seen_tok, last_wall = _train_phase(
2099
- args, "pretrain", core, ar_h, sat_h, opt, scaler,
2100
  start_step, seen_tok, last_wall, cfg,
2101
  args.source, args.steps,
2102
  args.block or DEFAULT_BLOCK,
@@ -2124,9 +2277,10 @@ def train(args):
2124
  sat_h,
2125
  args.after_sft_lr_core or args.lr_core,
2126
  args.after_sft_lr_head or args.lr_head,
 
2127
  )
2128
  step, seen_tok, last_wall = _train_phase(
2129
- args, "sft", core, ar_h, sat_h, opt, scaler,
2130
  step, seen_tok, last_wall, cfg,
2131
  args.after_sft_source, args.after_sft_steps,
2132
  args.after_sft_block or DEFAULT_AFTER_SFT_BLOCK,
@@ -2142,7 +2296,7 @@ def train(args):
2142
  tie_weights=tie_weights,
2143
  streaming=True
2144
  )
2145
- save_ckpt(pathlib.Path(args.save_dir) / "final.pt", core, ar_h, sat_h, opt, scaler,
2146
  meta={"cfg": cfg, "step": step, "seen_tok": seen_tok, "wall_time": time.time(), "tie_weights": tie_weights})
2147
  print("🎉 All Training Complete")
2148
 
@@ -2182,7 +2336,7 @@ def infer(args):
2182
  if args.frequency_penalty is None: args.frequency_penalty = 0.3
2183
  if args.penalty_last_n is None: args.penalty_last_n = 128
2184
  if args.var is None: args.var = False
2185
- else:
2186
  if args.temperature is None: args.temperature = 0.5
2187
  if args.top_k is None: args.top_k = 30
2188
  if args.repetition_penalty is None: args.repetition_penalty = 2.0
@@ -2190,6 +2344,14 @@ def infer(args):
2190
  if args.frequency_penalty is None: args.frequency_penalty = 1.0
2191
  if args.penalty_last_n is None: args.penalty_last_n = 200
2192
  if args.var is None: args.var = True
 
 
 
 
 
 
 
 
2193
  path = _resolve_ckpt(pathlib.Path(args.ckpt)) or pathlib.Path(args.ckpt)
2194
  sd = torch.load(path, map_location="cpu")
2195
  # Restore tokenizer from checkpoint if available
@@ -2210,10 +2372,12 @@ def infer(args):
2210
  print("[infer] Delta checkpoint detected, using large preset cfg")
2211
  cfg = PRESETS["large"].copy()
2212
  tie_weights = False
2213
- # Remap: delta stores under sd["weights"]["core"/"ar"/"sat"]
2214
  sd["core"] = sd["weights"]["core"]
2215
  sd["ar"] = sd["weights"]["ar"]
2216
  sd["sat"] = sd["weights"]["sat"]
 
 
2217
  else:
2218
  cfg = sd["cfg"]
2219
  tie_weights = sd.get("tie_weights", False)
@@ -2242,16 +2406,27 @@ def infer(args):
2242
  sublinear_stride=args.sublinear_stride,
2243
  sublinear_max_anchors=args.sublinear_max_anchors,
2244
  sublinear_chunk=args.sublinear_chunk,
 
 
 
 
2245
  ).to(DEV)
2246
  ar_h = ARHead(cfg["d"], tie_weights=tie_weights, embedding_weight=core.emb.weight if tie_weights else None).to(DEV)
2247
  sat_h = SATHead(cfg["d"]).to(DEV)
 
2248
  core.load_state_dict(sd["core"])
2249
  ar_h.load_state_dict(sd["ar"])
2250
  sat_h.load_state_dict(sd["sat"])
 
 
 
 
2251
  core.eval()
2252
  ar_h.eval()
2253
  sat_h.eval()
2254
- total_params = _count_enabled_params(core, ar_h, sat_h)
 
 
2255
  if total_params >= 1_000_000_000:
2256
  param_str = f"{total_params / 1_000_000_000:.2f}B"
2257
  elif total_params >= 1_000_000:
@@ -2283,6 +2458,25 @@ def infer(args):
2283
  nxt = _sample(logits, args.temperature, args.top_k, args.top_p, args.min_p, args.greedy)
2284
  ids = torch.cat([ids, nxt], 1)
2285
  h, kvs = core(ids[:, -1:], None, kv_caches=kvs, use_cache=True, total_seq_len=ids.size(1))
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
2286
  else:
2287
  cached_len = ids.size(1)
2288
  h, kvs = core(ids, sat_mask(ids.size(1)), use_cache=True, total_seq_len=cached_len)
@@ -2362,6 +2556,16 @@ def main():
2362
  help="For --attn_backend sublinear, cap landmark candidates per query chunk.")
2363
  tr.add_argument("--sublinear_chunk", type=int, default=DEFAULT_SUBLINEAR_CHUNK,
2364
  help="For --attn_backend sublinear, query chunk size controlling peak gather memory.")
 
 
 
 
 
 
 
 
 
 
2365
  tr.add_argument("--optimizer", choices=["adamw", "adamw8bit", "paged_adamw8bit"], default="adamw",
2366
  help="Optimizer backend. 8-bit options reduce VRAM on 24GB production runs.")
2367
  tr.add_argument("--save_every_sec", type=int, default=DEFAULT_SAVE_SEC)
@@ -2379,6 +2583,13 @@ def main():
2379
  tr.add_argument("--ar_only", action="store_true")
2380
  tr.add_argument("--sat_every", type=int, default=1,
2381
  help="Train SAT every N steps. Default 1 keeps AR+SAT every step.")
 
 
 
 
 
 
 
2382
  tr.add_argument("--freeze_core", action="store_true")
2383
  tr.add_argument("--unfreeze_ln", action="store_true")
2384
  tr.add_argument("--train_emb", action="store_true")
@@ -2404,7 +2615,7 @@ def main():
2404
  tr.add_argument("--after_sft_lr_core", type=float, default=0.0)
2405
  tr.add_argument("--after_sft_lr_head", type=float, default=0.0)
2406
  inf = sub.add_parser("infer")
2407
- inf.add_argument("--mode", choices=["ar", "sat"], required=True)
2408
  inf.add_argument("--ckpt", required=True)
2409
  inf.add_argument("--prompt", required=True)
2410
  inf.add_argument("--max_new", type=int, default=120)
@@ -2426,6 +2637,8 @@ def main():
2426
  inf.add_argument("--sublinear_stride", type=int, default=DEFAULT_SUBLINEAR_STRIDE)
2427
  inf.add_argument("--sublinear_max_anchors", type=int, default=DEFAULT_SUBLINEAR_MAX_ANCHORS)
2428
  inf.add_argument("--sublinear_chunk", type=int, default=DEFAULT_SUBLINEAR_CHUNK)
 
 
2429
  st = sub.add_parser("status", help="Read-only training status")
2430
  st.add_argument("--json", dest="json_output", action="store_true")
2431
  st.add_argument("--log", type=str, default=str(STATUS_DEFAULT_LOG))
 
1
  #!/usr/bin/env python3
2
 
3
+ # n.py - Joint AR+SAT+NAT Trainer with Expansion Ratio Testing
4
  # Enhanced inference: checkpoint name, tok/s, UK time
5
 
6
  from __future__ import annotations
 
586
 
587
  import torch
588
  import torch.utils.checkpoint as torch_checkpoint
589
+ from anchor_memory import AnchorMemoryConfig, AnchorMemoryLayer
590
 
591
  # SafeProgress - Claude-safe progress (discrete lines, not single growing line)
592
  class SafeProgress:
 
685
 
686
  class _SyntheticTokenizer:
687
  pad_token = "<|pad|>"
688
+ pad_token_id = 0
689
  eos_token_id = 1
690
  sep_token_id = 1
691
 
 
798
  if not SYNTHETIC_TOKENIZER:
799
  _tokenizer_health_check(tok)
800
 
801
+ VOCAB, BLANK, EOS = (
802
  max(tok.get_vocab().values()) + 1,
803
+ int(getattr(tok, "pad_token_id", 0) or 0),
804
  tok.eos_token_id if tok.eos_token_id is not None else tok.sep_token_id
805
  )
806
 
 
864
  DEFAULT_SUBLINEAR_STRIDE = _env_int("AGILLM_SUBLINEAR_STRIDE", 64)
865
  DEFAULT_SUBLINEAR_MAX_ANCHORS = _env_int("AGILLM_SUBLINEAR_MAX_ANCHORS", 256)
866
  DEFAULT_SUBLINEAR_CHUNK = _env_int("AGILLM_SUBLINEAR_CHUNK", 128)
867
+ DEFAULT_ANCHOR_MEMORY = bool(_env_int("AGILLM_ANCHOR_MEMORY", 0))
868
+ DEFAULT_ANCHOR_STRIDE = _env_int("AGILLM_ANCHOR_STRIDE", 256)
869
+ DEFAULT_ANCHOR_MAX = _env_int("AGILLM_ANCHOR_MAX", 2048)
870
+ DEFAULT_ANCHOR_POSITION = _env_int("AGILLM_ANCHOR_POSITION", -1) # -1 = stack middle
871
+ DEFAULT_KV_BUFFER = bool(_env_int("AGILLM_KV_BUFFER", 0))
872
  AGILLM4_TOKEN_PARAM_RATIO = 100.0
873
 
874
  # ───────────────────────── UK Time Helper ─────────────────────────
 
1133
  return -_alibi_slopes(n_heads) * dist
1134
 
1135
  # ───────────────────────── Model components ─────────────────────────
1136
+ class KVBuffer:
1137
+ """Preallocated K/V cache for decode. Replaces torch.cat-based growth.
1138
+
1139
+ Layout matches MHA-internal head-major shape [B, H, T, d_k]. Caller sizes
1140
+ once; each ``append`` writes ``length:length+n`` slots in place and grows
1141
+ ``length``. ``view()`` returns slices of the live region so attention sees
1142
+ only filled positions.
1143
+ """
1144
+
1145
+ __slots__ = ("k", "v", "length", "capacity")
1146
+
1147
+ def __init__(
1148
+ self,
1149
+ batch: int,
1150
+ heads: int,
1151
+ capacity: int,
1152
+ d_k: int,
1153
+ device,
1154
+ dtype,
1155
+ ):
1156
+ self.k = torch.empty(batch, heads, capacity, d_k, device=device, dtype=dtype)
1157
+ self.v = torch.empty(batch, heads, capacity, d_k, device=device, dtype=dtype)
1158
+ self.length = 0
1159
+ self.capacity = capacity
1160
+
1161
+ def append(self, k_new: torch.Tensor, v_new: torch.Tensor):
1162
+ n = k_new.size(2)
1163
+ end = self.length + n
1164
+ if end > self.capacity:
1165
+ raise RuntimeError(
1166
+ f"KVBuffer overflow: length={self.length} + n={n} > capacity={self.capacity}"
1167
+ )
1168
+ self.k[:, :, self.length:end].copy_(k_new)
1169
+ self.v[:, :, self.length:end].copy_(v_new)
1170
+ self.length = end
1171
+
1172
+ def view(self):
1173
+ return self.k[:, :, :self.length], self.v[:, :, :self.length]
1174
+
1175
+
1176
  class TuneableAttentionMHA(nn.Module):
1177
  def __init__(
1178
  self,
 
1350
  else:
1351
  q = self._proj_qk(q_lin)
1352
  k_new = self._proj_qk(k_lin)
1353
+ if kv_cache is None:
1354
  k, v = k_new, v_new
1355
+ elif isinstance(kv_cache, KVBuffer):
1356
+ if use_cache:
1357
+ kv_cache.append(k_new, v_new)
1358
+ k, v = kv_cache.view()
1359
+ else:
1360
+ k, v = k_new, v_new
1361
  else:
1362
  k_cached, v_cached = kv_cache
1363
  if use_cache:
 
1391
  z = att.softmax(-1) @ v
1392
  z = z.transpose(1, 2).reshape(x.size(0), x.size(1), -1)
1393
  out = self.drop(self.proj(z))
1394
+ if not use_cache:
1395
+ return out
1396
+ new_kv = kv_cache if isinstance(kv_cache, KVBuffer) else (k, v)
1397
+ return out, new_kv
1398
 
1399
 
1400
  class Block(nn.Module):
 
1445
  sublinear_stride: int = DEFAULT_SUBLINEAR_STRIDE,
1446
  sublinear_max_anchors: int = DEFAULT_SUBLINEAR_MAX_ANCHORS,
1447
  sublinear_chunk: int = DEFAULT_SUBLINEAR_CHUNK,
1448
+ anchor_memory: bool = DEFAULT_ANCHOR_MEMORY,
1449
+ anchor_stride: int = DEFAULT_ANCHOR_STRIDE,
1450
+ anchor_max: int = DEFAULT_ANCHOR_MAX,
1451
+ anchor_position: int = DEFAULT_ANCHOR_POSITION,
1452
  ):
1453
  super().__init__()
1454
  d, l, h, r = cfg["d"], cfg["layers"], cfg["heads"], cfg["rank"]
 
1474
  self.sublinear_stride = sublinear_stride
1475
  self.sublinear_max_anchors = sublinear_max_anchors
1476
  self.sublinear_chunk = sublinear_chunk
1477
+ self.anchor_memory_enabled = bool(anchor_memory)
1478
+ self.anchor_stride = int(anchor_stride)
1479
+ self.anchor_max = int(anchor_max)
1480
+ n_layers = int(cfg["layers"])
1481
+ if int(anchor_position) < 0:
1482
+ self.anchor_position = n_layers // 2
1483
+ else:
1484
+ self.anchor_position = min(int(anchor_position), n_layers - 1)
1485
+ if self.anchor_memory_enabled:
1486
+ am_cfg = AnchorMemoryConfig(
1487
+ d_model=int(cfg["d"]),
1488
+ heads=int(cfg["heads"]),
1489
+ anchor_stride=self.anchor_stride,
1490
+ max_anchors=self.anchor_max,
1491
+ )
1492
+ self.anchor = AnchorMemoryLayer(am_cfg)
1493
+ else:
1494
+ self.anchor = None
1495
 
1496
  def forward(self, ids, mask, kv_caches=None, use_cache=False, total_seq_len=None):
1497
  x = self.emb(ids)
1498
  if not use_cache:
1499
+ for i, blk in enumerate(self.blocks):
1500
  if self.grad_checkpoint and self.training:
1501
  x = torch_checkpoint.checkpoint(blk, x, mask, use_reentrant=False)
1502
  else:
1503
  x = blk(x, mask)
1504
+ if self.anchor is not None and i == self.anchor_position:
1505
+ if self.grad_checkpoint and self.training:
1506
+ x, _ = torch_checkpoint.checkpoint(self.anchor, x, use_reentrant=False)
1507
+ else:
1508
+ x, _ = self.anchor(x)
1509
  return self.ln(x)
1510
  new_kvs = []
1511
  for i, blk in enumerate(self.blocks):
1512
  kv = kv_caches[i] if kv_caches else None
1513
  x, kv_out = blk(x, mask, kv, use_cache=True, total_seq_len=total_seq_len)
1514
  new_kvs.append(kv_out)
1515
+ if self.anchor is not None and i == self.anchor_position:
1516
+ x, _ = self.anchor(x)
1517
  return self.ln(x), new_kvs
1518
 
1519
 
 
1531
  return self.proj(h)
1532
 
1533
 
1534
+ class NATHead(nn.Module):
1535
+ def __init__(self, d):
1536
+ super().__init__()
1537
+ self.proj = nn.Linear(d, VOCAB)
1538
+
1539
+ def forward(self, h):
1540
+ return self.proj(h)
1541
+
1542
+
1543
  class SATHead(nn.Module):
1544
  def __init__(self, d, mode="var"):
1545
  super().__init__()
 
1662
  return value.detach().clone()
1663
  return copy.deepcopy(value)
1664
 
1665
+ def _optimizer_param_name_lookup(core, ar_h, sat_h, nat_h=None) -> dict[int, str]:
1666
  out = {}
1667
+ for prefix, module in (("core", core), ("ar", ar_h), ("sat", sat_h), ("nat", nat_h)):
1668
+ if module is None:
1669
+ continue
1670
  for name, param in module.named_parameters():
1671
  out.setdefault(id(param), f"{prefix}.{name}")
1672
  return out
1673
 
1674
+ def _optimizer_group_param_names(opt, core, ar_h, sat_h, nat_h=None) -> List[List[str]]:
1675
+ lookup = _optimizer_param_name_lookup(core, ar_h, sat_h, nat_h)
1676
  return [
1677
  [lookup.get(id(param), f"<unknown:{id(param)}>") for param in group["params"]]
1678
  for group in opt.param_groups
 
1703
  out[key] = copy.deepcopy(vals[0])
1704
  return out
1705
 
1706
+ def _fuse_legacy_qkv_optimizer_state(opt_state: dict, opt, core, ar_h, sat_h, nat_h=None) -> Optional[dict]:
1707
  """Remap pre-QKV-fusion AdamW state to the current fused parameter layout."""
1708
  if not isinstance(opt_state, dict) or "state" not in opt_state or "param_groups" not in opt_state:
1709
  return None
1710
  current_sd = opt.state_dict()
1711
+ current_names = _optimizer_group_param_names(opt, core, ar_h, sat_h, nat_h)
1712
  legacy_names = [
1713
  [legacy for name in group_names for legacy in _legacy_names_for_current_param(name)]
1714
  for group_names in current_names
 
1750
 
1751
  return {"state": new_states, "param_groups": new_groups}
1752
 
1753
+ def save_delta(core, ar_h, sat_h, nat_h, step: int, seen_tok: int, save_dir: pathlib.Path, phase_name: str):
1754
  """Save weight-only delta in background thread. Non-blocking."""
1755
  global _delta_thread
1756
  # Wait for any previous delta write to finish
 
1763
  "ar": {k: v.detach().cpu() for k, v in _checkpoint_state_dict(ar_h).items()},
1764
  "sat": {k: v.detach().cpu() for k, v in _checkpoint_state_dict(sat_h).items()},
1765
  }
1766
+ if nat_h is not None:
1767
+ tensors["nat"] = {k: v.detach().cpu() for k, v in _checkpoint_state_dict(nat_h).items()}
1768
  meta = {"step": step, "seen_tok": seen_tok, "wall_time": time.time(), "delta": True}
1769
  path = save_dir / f"{phase_name}_delta_step{step:08d}.pt"
1770
  _delta_thread = threading.Thread(target=_do_delta_save, args=(tensors, path, meta), daemon=True)
 
1793
  except Exception as e:
1794
  print(f" [delta-prune] error: {e}")
1795
 
1796
+ def load_delta(path: pathlib.Path, core, ar_h, sat_h, nat_h=None):
1797
  """Load weight-only delta. Returns (step, seen_tok) or raises."""
1798
  # Verify checksum if sidecar exists
1799
  sha_path = path.with_suffix(".sha256")
 
1809
  core.load_state_dict(_fuse_qkv_in_state_dict(_strip_orig_mod_prefix(ck["weights"]["core"])))
1810
  ar_h.load_state_dict(_strip_orig_mod_prefix(ck["weights"]["ar"]))
1811
  sat_h.load_state_dict(_strip_orig_mod_prefix(ck["weights"]["sat"]))
1812
+ if nat_h is not None:
1813
+ nat_sd = ck["weights"].get("nat")
1814
+ if nat_sd is not None:
1815
+ nat_h.load_state_dict(_strip_orig_mod_prefix(nat_sd))
1816
+ else:
1817
+ print("[nat] Delta has no NAT head; keeping fresh NAT initialization")
1818
  return ck.get("step", 0), ck.get("seen_tok", 0)
1819
 
1820
  def _flush_delta():
 
1824
  print(" [delta] flushing in-flight write...")
1825
  _delta_thread.join(timeout=120)
1826
 
1827
+ def save_ckpt(path: pathlib.Path, core, ar_h, sat_h, nat_h, opt, scaler, meta):
1828
  path.parent.mkdir(exist_ok=True, parents=True)
1829
  tmp = path.with_suffix(path.suffix + ".tmp")
1830
  state = {
 
1837
  "tie_weights": meta.get("tie_weights", False),
1838
  **{k: v for k, v in meta.items() if k not in ("cfg", "tie_weights")}
1839
  }
1840
+ if nat_h is not None:
1841
+ state["nat"] = _checkpoint_state_dict(nat_h)
1842
  torch.save(state, tmp, _use_new_zipfile_serialization=False)
1843
  tmp.replace(path)
1844
  (path.parent / "latest.json").write_text(json.dumps({"path": str(path), "step": meta["step"]}))
1845
  print(f"\n✓ saved checkpoint {path.name}")
1846
 
1847
+ def load_ckpt(path, core, ar_h, sat_h, opt, scaler, nat_h=None):
1848
  p = _resolve_ckpt(path) or path
1849
  ck = _try_load(p, map_location="cpu")
1850
  if ck is None: raise FileNotFoundError(f"No valid checkpoint at {p}")
1851
  core.load_state_dict(_fuse_qkv_in_state_dict(_strip_orig_mod_prefix(ck["core"])))
1852
  ar_h.load_state_dict(_strip_orig_mod_prefix(ck["ar"]))
1853
  sat_h.load_state_dict(_strip_orig_mod_prefix(ck["sat"]))
1854
+ if nat_h is not None:
1855
+ if "nat" in ck:
1856
+ nat_h.load_state_dict(_strip_orig_mod_prefix(ck["nat"]))
1857
+ else:
1858
+ print("[nat] Checkpoint has no NAT head; keeping fresh NAT initialization")
1859
  try:
1860
  opt.load_state_dict(ck["opt"])
1861
  except Exception as exc:
1862
+ fused_opt = _fuse_legacy_qkv_optimizer_state(ck.get("opt"), opt, core, ar_h, sat_h, nat_h)
1863
  if fused_opt is not None:
1864
  try:
1865
  opt.load_state_dict(fused_opt)
 
1946
  if train_emb:
1947
  for p in core.emb.parameters(): p.requires_grad = True
1948
 
1949
+ def _optimizer_param_groups(core, ar_h, sat_h, lr_core: float, lr_head: float, nat_h=None):
1950
+ groups = [
1951
  {"params": [p for p in core.parameters() if p.requires_grad], "lr": lr_core},
1952
  {"params": ar_h.parameters(), "lr": lr_head},
1953
  {"params": sat_h.parameters(), "lr": lr_head},
1954
  ]
1955
+ if nat_h is not None:
1956
+ groups.append({"params": nat_h.parameters(), "lr": lr_head})
1957
+ return groups
1958
 
1959
+ def make_optimizer(args, core, ar_h, sat_h, lr_core: float, lr_head: float, nat_h=None):
1960
+ groups = _optimizer_param_groups(core, ar_h, sat_h, lr_core, lr_head, nat_h)
1961
  opt_name = getattr(args, "optimizer", "adamw")
1962
  if opt_name == "adamw":
1963
  return torch.optim.AdamW(groups)
 
1973
  return bnb.optim.AdamW8bit(groups)
1974
  raise ValueError(f"unknown optimizer: {opt_name}")
1975
 
1976
+ def _nat_ids_for_training(ids: torch.Tensor, max_tokens: int) -> torch.Tensor:
1977
+ if max_tokens and max_tokens > 0 and ids.size(1) > max_tokens:
1978
+ return ids[:, -max_tokens:]
1979
+ return ids
1980
+
1981
  def _train_phase(
1982
  args, phase_name: str,
1983
+ core, ar_h, sat_h, nat_h, opt, scaler,
1984
  start_step, seen_tok, resume_wall_time,
1985
  cfg, source, steps, block_size, batch_size,
1986
  chat_cfg: dict,
 
1995
  target_tokens = target_tokens_override
1996
  else:
1997
  ratio = _target_token_ratio(args)
1998
+ param_count = _count_enabled_params(core, ar_h, sat_h, nat_h)
1999
  target_tokens = int(ratio * param_count)
2000
  print(f"[{phase_name}] token_param_ratio={ratio:g} param_count={param_count:,} target_tokens={target_tokens:,}")
2001
  if steps:
 
2016
  )
2017
  ce_tok = nn.CrossEntropyLoss(label_smoothing=0.1)
2018
  ce_gate = nn.CrossEntropyLoss()
2019
+ ctc = nn.CTCLoss(blank=BLANK, zero_infinity=True)
2020
  pbar = SafeProgress(total=total_tokens_needed, initial=seen_tok, unit="tok")
2021
  grow_plan = _parse_grow_plan(args.grow_plan) if args.auto_grow else []
2022
  buf: list[int] = []
 
2029
  last_save_mono = time.monotonic() - (now_wall - (resume_wall_time or now_wall))
2030
  last_delta_step = start_step
2031
  print(f"[{phase_name}] Starting. Goal: {total_tokens_needed:,} tokens. Batch={BATCH}, Block={BLOCK}")
2032
+ print(
2033
+ f"[{phase_name}] AR_ONLY={args.ar_only}, SAT_EVERY={args.sat_every}, "
2034
+ f"NAT_EVERY={args.nat_every}, TIE_WEIGHTS={tie_weights}, STREAMING={streaming}"
2035
+ )
2036
  while seen_tok < total_tokens_needed:
2037
  try:
2038
  while len(buf) < BLOCK:
 
2069
  loss_value += float(loss_sat.detach().item())
2070
  scaler.scale(loss_sat).backward()
2071
  del h_sat, logits_sat, loss_sat
2072
+ do_nat = (
2073
+ nat_h is not None
2074
+ and (not args.ar_only)
2075
+ and args.nat_every > 0
2076
+ and (args.nat_every <= 1 or ((step + 1) % args.nat_every == 0))
2077
+ )
2078
+ if do_nat:
2079
+ nat_ids = _nat_ids_for_training(ids, args.nat_max_tokens)
2080
+ nat_expand = max(1, int(args.nat_expand))
2081
+ nat_in = torch.repeat_interleave(nat_ids, nat_expand, dim=1)
2082
+ input_lengths = torch.full((nat_ids.size(0),), nat_in.size(1), device=DEV, dtype=torch.long)
2083
+ target_lengths = torch.full((nat_ids.size(0),), nat_ids.size(1), device=DEV, dtype=torch.long)
2084
+ with amp(args.amp):
2085
+ h_nat = core(nat_in, None)
2086
+ log_nat = nat_h(h_nat).float().log_softmax(-1).transpose(0, 1)
2087
+ loss_nat = ctc(log_nat, nat_ids, input_lengths, target_lengths)
2088
+ loss_nat = float(args.nat_loss_weight) * loss_nat
2089
+ loss_value += float(loss_nat.detach().item())
2090
+ scaler.scale(loss_nat).backward()
2091
+ del nat_ids, nat_in, input_lengths, target_lengths, h_nat, log_nat, loss_nat
2092
  scaler.unscale_(opt)
2093
+ nn.utils.clip_grad_norm_([p for group in opt.param_groups for p in group["params"]], 1.0)
2094
  scaler.step(opt)
2095
  scaler.update()
2096
  opt.zero_grad(set_to_none=True)
 
2143
  ck_name = f"{phase_name}_step{step:08d}.pt"
2144
  _flush_delta() # wait for any in-flight delta before full save
2145
  _prune_checkpoints(pathlib.Path(args.save_dir), phase_name, max_ckpts)
2146
+ save_ckpt(pathlib.Path(args.save_dir) / ck_name, core, ar_h, sat_h, nat_h, opt, scaler,
2147
  meta={"cfg": cfg, "step": step, "seen_tok": seen_tok, "wall_time": time.time(), "tie_weights": tie_weights})
2148
  last_save_mono = now_mono
2149
  # Prune old deltas after a full save (they're superseded)
 
2152
  # ── Delta checkpoint (step-based, weight-only, async) ──
2153
  if args.delta_every_steps > 0 and (step - last_delta_step) >= args.delta_every_steps:
2154
  _prune_deltas(pathlib.Path(args.save_dir), phase_name, args.delta_max_keep)
2155
+ save_delta(core, ar_h, sat_h, nat_h, step, seen_tok, pathlib.Path(args.save_dir), phase_name)
2156
  last_delta_step = step
2157
  if args.auto_grow:
2158
  steps_since_last_grow += 1
 
2169
  pbar.close()
2170
  _flush_delta() # ensure any in-flight delta completes before final save
2171
  if phase_name != "sft":
2172
+ save_ckpt(pathlib.Path(args.save_dir) / f"{phase_name}_final.pt", core, ar_h, sat_h, nat_h, opt, scaler,
2173
  meta={"cfg": cfg, "step": step, "seen_tok": seen_tok, "wall_time": time.time(), "tie_weights": tie_weights})
2174
  else:
2175
  print("[sft] Skipping duplicate sft_final.pt; final.pt will contain the SFT result.")
 
2206
  sublinear_stride=args.sublinear_stride,
2207
  sublinear_max_anchors=args.sublinear_max_anchors,
2208
  sublinear_chunk=args.sublinear_chunk,
2209
+ anchor_memory=getattr(args, "anchor_memory", DEFAULT_ANCHOR_MEMORY),
2210
+ anchor_stride=getattr(args, "anchor_stride", DEFAULT_ANCHOR_STRIDE),
2211
+ anchor_max=getattr(args, "anchor_max", DEFAULT_ANCHOR_MAX),
2212
+ anchor_position=getattr(args, "anchor_position", DEFAULT_ANCHOR_POSITION),
2213
  ).to(DEV)
2214
  ar_h = ARHead(cfg["d"], tie_weights=tie_weights, embedding_weight=core.emb.weight if tie_weights else None).to(DEV)
2215
  sat_h = SATHead(cfg["d"], mode="var").to(DEV)
2216
+ nat_h = NATHead(cfg["d"]).to(DEV)
2217
+ total_params = _count_enabled_params(core, ar_h, sat_h, nat_h)
2218
  print(f"Total parameters: {total_params:,}")
2219
  if tie_weights:
2220
  print(f"{Colors.WARN}[weight-tying] Embedding and LM head share weights{Colors.RESET}")
 
2225
  loaded = _safe_load_any(src, core, key="core")
2226
  _safe_load_any(src, ar_h, key="ar")
2227
  _safe_load_any(src, sat_h, key="sat")
2228
+ nat_loaded = _safe_load_any(src, nat_h, key="nat")
2229
+ if not nat_loaded:
2230
+ print("[nat] Warm-start source has no NAT head; NAT head initialized fresh")
2231
  if loaded: print(f"Warm-start loaded from {src}")
2232
  _phase_freeze(core, freeze_core=args.freeze_core, unfreeze_ln=args.unfreeze_ln, train_emb=args.train_emb)
2233
+ opt = make_optimizer(args, core, ar_h, sat_h, args.lr_core, args.lr_head, nat_h)
2234
  scaler = GradScaler(enabled=(args.amp and _needs_grad_scaler()))
2235
  start_step, seen_tok, last_wall = 0, 0, None
2236
  if args.resume_delta and not args.fresh:
2237
+ delta_step, delta_tok = load_delta(pathlib.Path(args.resume_delta), core, ar_h, sat_h, nat_h)
2238
  start_step, seen_tok, last_wall = delta_step, delta_tok, None
2239
  print(f"Resumed from DELTA at step {start_step} (optimizer state reset — momentum rebuilds in ~100 steps)")
2240
  elif args.resume and not args.fresh:
2241
+ start_step, seen_tok, last_wall = load_ckpt(pathlib.Path(args.resume), core, ar_h, sat_h, opt, scaler, nat_h)
2242
  print(f"Resumed from step {start_step}")
2243
  # torch.compile AFTER loading checkpoint (key names differ)
2244
  if args.compile:
 
2246
  core = torch.compile(core, mode="reduce-overhead")
2247
  ar_h = torch.compile(ar_h, mode="reduce-overhead")
2248
  sat_h = torch.compile(sat_h, mode="reduce-overhead")
2249
+ nat_h = torch.compile(nat_h, mode="reduce-overhead")
2250
  print("[torch.compile] Done.")
2251
  step, seen_tok, last_wall = _train_phase(
2252
+ args, "pretrain", core, ar_h, sat_h, nat_h, opt, scaler,
2253
  start_step, seen_tok, last_wall, cfg,
2254
  args.source, args.steps,
2255
  args.block or DEFAULT_BLOCK,
 
2277
  sat_h,
2278
  args.after_sft_lr_core or args.lr_core,
2279
  args.after_sft_lr_head or args.lr_head,
2280
+ nat_h,
2281
  )
2282
  step, seen_tok, last_wall = _train_phase(
2283
+ args, "sft", core, ar_h, sat_h, nat_h, opt, scaler,
2284
  step, seen_tok, last_wall, cfg,
2285
  args.after_sft_source, args.after_sft_steps,
2286
  args.after_sft_block or DEFAULT_AFTER_SFT_BLOCK,
 
2296
  tie_weights=tie_weights,
2297
  streaming=True
2298
  )
2299
+ save_ckpt(pathlib.Path(args.save_dir) / "final.pt", core, ar_h, sat_h, nat_h, opt, scaler,
2300
  meta={"cfg": cfg, "step": step, "seen_tok": seen_tok, "wall_time": time.time(), "tie_weights": tie_weights})
2301
  print("🎉 All Training Complete")
2302
 
 
2336
  if args.frequency_penalty is None: args.frequency_penalty = 0.3
2337
  if args.penalty_last_n is None: args.penalty_last_n = 128
2338
  if args.var is None: args.var = False
2339
+ elif args.mode == "sat":
2340
  if args.temperature is None: args.temperature = 0.5
2341
  if args.top_k is None: args.top_k = 30
2342
  if args.repetition_penalty is None: args.repetition_penalty = 2.0
 
2344
  if args.frequency_penalty is None: args.frequency_penalty = 1.0
2345
  if args.penalty_last_n is None: args.penalty_last_n = 200
2346
  if args.var is None: args.var = True
2347
+ else:
2348
+ if args.temperature is None: args.temperature = 1.0
2349
+ if args.top_k is None: args.top_k = 0
2350
+ if args.repetition_penalty is None: args.repetition_penalty = 1.0
2351
+ if args.presence_penalty is None: args.presence_penalty = 0.0
2352
+ if args.frequency_penalty is None: args.frequency_penalty = 0.0
2353
+ if args.penalty_last_n is None: args.penalty_last_n = 0
2354
+ if args.var is None: args.var = False
2355
  path = _resolve_ckpt(pathlib.Path(args.ckpt)) or pathlib.Path(args.ckpt)
2356
  sd = torch.load(path, map_location="cpu")
2357
  # Restore tokenizer from checkpoint if available
 
2372
  print("[infer] Delta checkpoint detected, using large preset cfg")
2373
  cfg = PRESETS["large"].copy()
2374
  tie_weights = False
2375
+ # Remap: delta stores under sd["weights"]["core"/"ar"/"sat"/"nat"]
2376
  sd["core"] = sd["weights"]["core"]
2377
  sd["ar"] = sd["weights"]["ar"]
2378
  sd["sat"] = sd["weights"]["sat"]
2379
+ if "nat" in sd["weights"]:
2380
+ sd["nat"] = sd["weights"]["nat"]
2381
  else:
2382
  cfg = sd["cfg"]
2383
  tie_weights = sd.get("tie_weights", False)
 
2406
  sublinear_stride=args.sublinear_stride,
2407
  sublinear_max_anchors=args.sublinear_max_anchors,
2408
  sublinear_chunk=args.sublinear_chunk,
2409
+ anchor_memory=getattr(args, "anchor_memory", DEFAULT_ANCHOR_MEMORY),
2410
+ anchor_stride=getattr(args, "anchor_stride", DEFAULT_ANCHOR_STRIDE),
2411
+ anchor_max=getattr(args, "anchor_max", DEFAULT_ANCHOR_MAX),
2412
+ anchor_position=getattr(args, "anchor_position", DEFAULT_ANCHOR_POSITION),
2413
  ).to(DEV)
2414
  ar_h = ARHead(cfg["d"], tie_weights=tie_weights, embedding_weight=core.emb.weight if tie_weights else None).to(DEV)
2415
  sat_h = SATHead(cfg["d"]).to(DEV)
2416
+ nat_h = NATHead(cfg["d"]).to(DEV) if ("nat" in sd or args.mode == "nat") else None
2417
  core.load_state_dict(sd["core"])
2418
  ar_h.load_state_dict(sd["ar"])
2419
  sat_h.load_state_dict(sd["sat"])
2420
+ if nat_h is not None:
2421
+ if "nat" not in sd:
2422
+ raise ValueError("NAT inference requested, but this checkpoint has no NAT head")
2423
+ nat_h.load_state_dict(sd["nat"])
2424
  core.eval()
2425
  ar_h.eval()
2426
  sat_h.eval()
2427
+ if nat_h is not None:
2428
+ nat_h.eval()
2429
+ total_params = _count_enabled_params(core, ar_h, sat_h, nat_h)
2430
  if total_params >= 1_000_000_000:
2431
  param_str = f"{total_params / 1_000_000_000:.2f}B"
2432
  elif total_params >= 1_000_000:
 
2458
  nxt = _sample(logits, args.temperature, args.top_k, args.top_p, args.min_p, args.greedy)
2459
  ids = torch.cat([ids, nxt], 1)
2460
  h, kvs = core(ids[:, -1:], None, kv_caches=kvs, use_cache=True, total_seq_len=ids.size(1))
2461
+ elif args.mode == "nat":
2462
+ blank_slots = [BLANK] * (args.max_new * max(1, int(args.nat_expand)))
2463
+ ids = torch.tensor([prompt_tokens + blank_slots], device=DEV)
2464
+ for _ in range(max(1, int(args.nat_passes))):
2465
+ h = core(ids, None)
2466
+ logits = nat_h(h)
2467
+ logits[..., BLANK] = -1e9
2468
+ ids = logits.argmax(-1)
2469
+ decoded = []
2470
+ prev = None
2471
+ for token in ids[0].tolist():
2472
+ if token == BLANK:
2473
+ prev = token
2474
+ continue
2475
+ if token != prev:
2476
+ decoded.append(token)
2477
+ prev = token
2478
+ ids = torch.tensor([decoded[: prompt_len + args.max_new]], device=DEV)
2479
+ prompt_len = min(prompt_len, ids.size(1))
2480
  else:
2481
  cached_len = ids.size(1)
2482
  h, kvs = core(ids, sat_mask(ids.size(1)), use_cache=True, total_seq_len=cached_len)
 
2556
  help="For --attn_backend sublinear, cap landmark candidates per query chunk.")
2557
  tr.add_argument("--sublinear_chunk", type=int, default=DEFAULT_SUBLINEAR_CHUNK,
2558
  help="For --attn_backend sublinear, query chunk size controlling peak gather memory.")
2559
+ tr.add_argument("--anchor_memory", action="store_true",
2560
+ help="Enable anchor-memory long-context augmentation (one AnchorMemoryLayer at mid-stack).")
2561
+ tr.add_argument("--anchor_stride", type=int, default=DEFAULT_ANCHOR_STRIDE,
2562
+ help="Token span compressed into one anchor (default 256).")
2563
+ tr.add_argument("--anchor_max", type=int, default=DEFAULT_ANCHOR_MAX,
2564
+ help="Max anchors retained in the rolling memory bank.")
2565
+ tr.add_argument("--anchor_position", type=int, default=DEFAULT_ANCHOR_POSITION,
2566
+ help="Block index after which to insert anchor memory (-1 = stack middle).")
2567
+ tr.add_argument("--kv_buffer", action="store_true",
2568
+ help="Use preallocated KV buffer instead of torch.cat-based cache growth.")
2569
  tr.add_argument("--optimizer", choices=["adamw", "adamw8bit", "paged_adamw8bit"], default="adamw",
2570
  help="Optimizer backend. 8-bit options reduce VRAM on 24GB production runs.")
2571
  tr.add_argument("--save_every_sec", type=int, default=DEFAULT_SAVE_SEC)
 
2583
  tr.add_argument("--ar_only", action="store_true")
2584
  tr.add_argument("--sat_every", type=int, default=1,
2585
  help="Train SAT every N steps. Default 1 keeps AR+SAT every step.")
2586
+ tr.add_argument("--nat_every", type=int, default=1,
2587
+ help="Train NAT every N steps with a CTC objective. Default 1 keeps AR+SAT+NAT every step.")
2588
+ tr.add_argument("--nat_loss_weight", type=float, default=1.0)
2589
+ tr.add_argument("--nat_expand", type=int, default=2,
2590
+ help="Repeat tokens this many times for the NAT CTC input length.")
2591
+ tr.add_argument("--nat_max_tokens", type=int, default=0,
2592
+ help="Optional cap for NAT target tokens per batch; 0 uses the whole block.")
2593
  tr.add_argument("--freeze_core", action="store_true")
2594
  tr.add_argument("--unfreeze_ln", action="store_true")
2595
  tr.add_argument("--train_emb", action="store_true")
 
2615
  tr.add_argument("--after_sft_lr_core", type=float, default=0.0)
2616
  tr.add_argument("--after_sft_lr_head", type=float, default=0.0)
2617
  inf = sub.add_parser("infer")
2618
+ inf.add_argument("--mode", choices=["ar", "sat", "nat"], required=True)
2619
  inf.add_argument("--ckpt", required=True)
2620
  inf.add_argument("--prompt", required=True)
2621
  inf.add_argument("--max_new", type=int, default=120)
 
2637
  inf.add_argument("--sublinear_stride", type=int, default=DEFAULT_SUBLINEAR_STRIDE)
2638
  inf.add_argument("--sublinear_max_anchors", type=int, default=DEFAULT_SUBLINEAR_MAX_ANCHORS)
2639
  inf.add_argument("--sublinear_chunk", type=int, default=DEFAULT_SUBLINEAR_CHUNK)
2640
+ inf.add_argument("--nat_expand", type=int, default=2)
2641
+ inf.add_argument("--nat_passes", type=int, default=1)
2642
  st = sub.add_parser("status", help="Read-only training status")
2643
  st.add_argument("--json", dest="json_output", action="store_true")
2644
  st.add_argument("--log", type=str, default=str(STATUS_DEFAULT_LOG))
nB300_agillm4.py.bak-20260520-140542 ADDED
The diff for this file is too large to render. See raw diff
 
nB300_agillm4.py.bak-fix-20260520-140710 ADDED
The diff for this file is too large to render. See raw diff
 
remote_verify_m_fold_after_qkv_agillm4.json ADDED
@@ -0,0 +1,118 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "backend": "manual",
4
+ "d": 32,
5
+ "dk": 16,
6
+ "expected_k_width": 16,
7
+ "heads": 2,
8
+ "ok": true,
9
+ "preset": "pico_1x",
10
+ "rank": 16,
11
+ "rows": {
12
+ "cache_k_width": 16.0,
13
+ "cache_v_width": 16.0,
14
+ "cached_append_forward": 0.0,
15
+ "causal_alibi_forward": 0.0,
16
+ "causal_alibi_loss": 0.0,
17
+ "causal_alibi_param_grad": 0.0,
18
+ "causal_alibi_x_grad": 0.0,
19
+ "none_forward": 0.0,
20
+ "none_loss": 0.0,
21
+ "none_param_grad": 0.0,
22
+ "none_x_grad": 0.0,
23
+ "sat_alibi_forward": 0.0,
24
+ "sat_alibi_loss": 0.0,
25
+ "sat_alibi_param_grad": 0.0,
26
+ "sat_alibi_x_grad": 0.0
27
+ },
28
+ "tol": 0.0002
29
+ },
30
+ {
31
+ "backend": "sdpa",
32
+ "d": 32,
33
+ "dk": 16,
34
+ "expected_k_width": 16,
35
+ "heads": 2,
36
+ "ok": true,
37
+ "preset": "pico_1x",
38
+ "rank": 16,
39
+ "rows": {
40
+ "cache_k_width": 16.0,
41
+ "cache_v_width": 16.0,
42
+ "cached_append_forward": 0.0,
43
+ "causal_alibi_forward": 5.386769771575928e-06,
44
+ "causal_alibi_loss": 3.725290298461914e-09,
45
+ "causal_alibi_param_grad": 5.189329385757446e-06,
46
+ "causal_alibi_x_grad": 8.445931598544121e-07,
47
+ "none_forward": 1.4901161193847656e-08,
48
+ "none_loss": 9.313225746154785e-10,
49
+ "none_param_grad": 2.0703300833702087e-06,
50
+ "none_x_grad": 1.7951242625713348e-07,
51
+ "sat_alibi_forward": 0.0,
52
+ "sat_alibi_loss": 0.0,
53
+ "sat_alibi_param_grad": 8.198898285627365e-06,
54
+ "sat_alibi_x_grad": 9.000941645354033e-07
55
+ },
56
+ "tol": 0.0002
57
+ },
58
+ {
59
+ "backend": "manual",
60
+ "d": 128,
61
+ "dk": 16,
62
+ "expected_k_width": 16,
63
+ "heads": 8,
64
+ "ok": true,
65
+ "preset": "micro_3x",
66
+ "rank": 48,
67
+ "rows": {
68
+ "cache_k_width": 16.0,
69
+ "cache_v_width": 16.0,
70
+ "cached_append_forward": 0.0,
71
+ "causal_alibi_forward": 0.00011304020881652832,
72
+ "causal_alibi_loss": 1.043081283569336e-07,
73
+ "causal_alibi_param_grad": 1.3775425031781197e-06,
74
+ "causal_alibi_x_grad": 1.5925616025924683e-07,
75
+ "metric_cache_cleared_on_train": 0.0,
76
+ "metric_cache_reused": 0.0,
77
+ "none_forward": 8.377432823181152e-05,
78
+ "none_loss": 1.30385160446167e-08,
79
+ "none_param_grad": 5.532929208129644e-07,
80
+ "none_x_grad": 3.541026671882719e-08,
81
+ "sat_alibi_forward": 0.00014029256999492645,
82
+ "sat_alibi_loss": 9.12696123123169e-08,
83
+ "sat_alibi_param_grad": 9.688083082437515e-07,
84
+ "sat_alibi_x_grad": 1.5372643247246742e-07
85
+ },
86
+ "tol": 0.0002
87
+ },
88
+ {
89
+ "backend": "sdpa",
90
+ "d": 128,
91
+ "dk": 16,
92
+ "expected_k_width": 16,
93
+ "heads": 8,
94
+ "ok": true,
95
+ "preset": "micro_3x",
96
+ "rank": 48,
97
+ "rows": {
98
+ "cache_k_width": 16.0,
99
+ "cache_v_width": 16.0,
100
+ "cached_append_forward": 0.0,
101
+ "causal_alibi_forward": 0.00011304020881652832,
102
+ "causal_alibi_loss": 1.0244548320770264e-07,
103
+ "causal_alibi_param_grad": 1.3635726645588875e-06,
104
+ "causal_alibi_x_grad": 1.829757820814848e-07,
105
+ "metric_cache_cleared_on_train": 0.0,
106
+ "metric_cache_reused": 0.0,
107
+ "none_forward": 8.377432823181152e-05,
108
+ "none_loss": 5.587935447692871e-09,
109
+ "none_param_grad": 5.358306225389242e-07,
110
+ "none_x_grad": 3.109380486421287e-08,
111
+ "sat_alibi_forward": 0.00014029256999492645,
112
+ "sat_alibi_loss": 9.12696123123169e-08,
113
+ "sat_alibi_param_grad": 1.071603037416935e-06,
114
+ "sat_alibi_x_grad": 1.1662632459774613e-07
115
+ },
116
+ "tol": 0.0002
117
+ }
118
+ ]
remote_verify_m_fold_agillm4.json ADDED
@@ -0,0 +1,118 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "backend": "manual",
4
+ "d": 32,
5
+ "dk": 16,
6
+ "expected_k_width": 16,
7
+ "heads": 2,
8
+ "ok": true,
9
+ "preset": "pico_1x",
10
+ "rank": 16,
11
+ "rows": {
12
+ "cache_k_width": 16.0,
13
+ "cache_v_width": 16.0,
14
+ "cached_append_forward": 0.0,
15
+ "causal_alibi_forward": 0.0,
16
+ "causal_alibi_loss": 0.0,
17
+ "causal_alibi_param_grad": 0.0,
18
+ "causal_alibi_x_grad": 0.0,
19
+ "none_forward": 0.0,
20
+ "none_loss": 0.0,
21
+ "none_param_grad": 0.0,
22
+ "none_x_grad": 0.0,
23
+ "sat_alibi_forward": 0.0,
24
+ "sat_alibi_loss": 0.0,
25
+ "sat_alibi_param_grad": 0.0,
26
+ "sat_alibi_x_grad": 0.0
27
+ },
28
+ "tol": 0.0002
29
+ },
30
+ {
31
+ "backend": "sdpa",
32
+ "d": 32,
33
+ "dk": 16,
34
+ "expected_k_width": 16,
35
+ "heads": 2,
36
+ "ok": true,
37
+ "preset": "pico_1x",
38
+ "rank": 16,
39
+ "rows": {
40
+ "cache_k_width": 16.0,
41
+ "cache_v_width": 16.0,
42
+ "cached_append_forward": 0.0,
43
+ "causal_alibi_forward": 5.386769771575928e-06,
44
+ "causal_alibi_loss": 3.725290298461914e-09,
45
+ "causal_alibi_param_grad": 5.189329385757446e-06,
46
+ "causal_alibi_x_grad": 8.444767445325851e-07,
47
+ "none_forward": 1.4901161193847656e-08,
48
+ "none_loss": 9.313225746154785e-10,
49
+ "none_param_grad": 2.0703300833702087e-06,
50
+ "none_x_grad": 1.7954153008759022e-07,
51
+ "sat_alibi_forward": 0.0,
52
+ "sat_alibi_loss": 0.0,
53
+ "sat_alibi_param_grad": 8.198898285627365e-06,
54
+ "sat_alibi_x_grad": 9.0012326836586e-07
55
+ },
56
+ "tol": 0.0002
57
+ },
58
+ {
59
+ "backend": "manual",
60
+ "d": 128,
61
+ "dk": 16,
62
+ "expected_k_width": 16,
63
+ "heads": 8,
64
+ "ok": true,
65
+ "preset": "micro_3x",
66
+ "rank": 48,
67
+ "rows": {
68
+ "cache_k_width": 16.0,
69
+ "cache_v_width": 16.0,
70
+ "cached_append_forward": 0.0,
71
+ "causal_alibi_forward": 0.00011304020881652832,
72
+ "causal_alibi_loss": 1.043081283569336e-07,
73
+ "causal_alibi_param_grad": 1.3775425031781197e-06,
74
+ "causal_alibi_x_grad": 1.592270564287901e-07,
75
+ "metric_cache_cleared_on_train": 0.0,
76
+ "metric_cache_reused": 0.0,
77
+ "none_forward": 8.377432823181152e-05,
78
+ "none_loss": 1.30385160446167e-08,
79
+ "none_param_grad": 5.532929208129644e-07,
80
+ "none_x_grad": 3.541026671882719e-08,
81
+ "sat_alibi_forward": 0.00014029256999492645,
82
+ "sat_alibi_loss": 9.12696123123169e-08,
83
+ "sat_alibi_param_grad": 9.688083082437515e-07,
84
+ "sat_alibi_x_grad": 1.5371188055723906e-07
85
+ },
86
+ "tol": 0.0002
87
+ },
88
+ {
89
+ "backend": "sdpa",
90
+ "d": 128,
91
+ "dk": 16,
92
+ "expected_k_width": 16,
93
+ "heads": 8,
94
+ "ok": true,
95
+ "preset": "micro_3x",
96
+ "rank": 48,
97
+ "rows": {
98
+ "cache_k_width": 16.0,
99
+ "cache_v_width": 16.0,
100
+ "cached_append_forward": 0.0,
101
+ "causal_alibi_forward": 0.00011304020881652832,
102
+ "causal_alibi_loss": 1.0244548320770264e-07,
103
+ "causal_alibi_param_grad": 1.3635726645588875e-06,
104
+ "causal_alibi_x_grad": 1.8296123016625643e-07,
105
+ "metric_cache_cleared_on_train": 0.0,
106
+ "metric_cache_reused": 0.0,
107
+ "none_forward": 8.377432823181152e-05,
108
+ "none_loss": 5.587935447692871e-09,
109
+ "none_param_grad": 5.358306225389242e-07,
110
+ "none_x_grad": 3.1091985874809325e-08,
111
+ "sat_alibi_forward": 0.00014029256999492645,
112
+ "sat_alibi_loss": 9.12696123123169e-08,
113
+ "sat_alibi_param_grad": 1.071603037416935e-06,
114
+ "sat_alibi_x_grad": 1.1662632459774613e-07
115
+ },
116
+ "tol": 0.0002
117
+ }
118
+ ]
remote_verify_qkv_all_backends_agillm4.json ADDED
@@ -0,0 +1,177 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ [
2
+ {
3
+ "backend": "manual",
4
+ "d": 32,
5
+ "dk": 16,
6
+ "heads": 2,
7
+ "ok": true,
8
+ "preset": "pico_1x",
9
+ "rank": 16,
10
+ "rows": {
11
+ "causal_alibi_forward": 0.0,
12
+ "causal_alibi_loss": 0.0,
13
+ "causal_alibi_param_grad": 0.0,
14
+ "causal_alibi_x_grad": 5.820766091346741e-11,
15
+ "legacy_load_forward": 0.0,
16
+ "legacy_load_missing_unexpected": 0.0,
17
+ "legacy_load_qkv_weight": 0.0,
18
+ "none_forward": 0.0,
19
+ "none_loss": 0.0,
20
+ "none_param_grad": 0.0,
21
+ "none_x_grad": 5.820766091346741e-11,
22
+ "optimizer_remap": 0.0,
23
+ "safe_load_any_loaded": 0.0,
24
+ "safe_load_any_qkv": 0.0,
25
+ "sat_alibi_forward": 0.0,
26
+ "sat_alibi_loss": 0.0,
27
+ "sat_alibi_param_grad": 0.0,
28
+ "sat_alibi_x_grad": 2.3283064365386963e-10
29
+ },
30
+ "tol": 0.0002
31
+ },
32
+ {
33
+ "backend": "sdpa",
34
+ "d": 32,
35
+ "dk": 16,
36
+ "heads": 2,
37
+ "ok": true,
38
+ "preset": "pico_1x",
39
+ "rank": 16,
40
+ "rows": {
41
+ "causal_alibi_forward": 0.0,
42
+ "causal_alibi_loss": 0.0,
43
+ "causal_alibi_param_grad": 0.0,
44
+ "causal_alibi_x_grad": 1.1641532182693481e-10,
45
+ "legacy_load_forward": 0.0,
46
+ "legacy_load_missing_unexpected": 0.0,
47
+ "legacy_load_qkv_weight": 0.0,
48
+ "none_forward": 0.0,
49
+ "none_loss": 0.0,
50
+ "none_param_grad": 0.0,
51
+ "none_x_grad": 5.820766091346741e-11,
52
+ "safe_load_any_loaded": 0.0,
53
+ "safe_load_any_qkv": 0.0,
54
+ "sat_alibi_forward": 0.0,
55
+ "sat_alibi_loss": 0.0,
56
+ "sat_alibi_param_grad": 0.0,
57
+ "sat_alibi_x_grad": 1.1641532182693481e-10
58
+ },
59
+ "tol": 0.0002
60
+ },
61
+ {
62
+ "backend": "sublinear",
63
+ "d": 32,
64
+ "dk": 16,
65
+ "heads": 2,
66
+ "ok": true,
67
+ "preset": "pico_1x",
68
+ "rank": 16,
69
+ "rows": {
70
+ "causal_alibi_forward": 0.0,
71
+ "causal_alibi_loss": 0.0,
72
+ "causal_alibi_param_grad": 0.0,
73
+ "causal_alibi_x_grad": 1.1641532182693481e-10,
74
+ "legacy_load_forward": 0.0,
75
+ "legacy_load_missing_unexpected": 0.0,
76
+ "legacy_load_qkv_weight": 0.0,
77
+ "none_forward": 0.0,
78
+ "none_loss": 0.0,
79
+ "none_param_grad": 0.0,
80
+ "none_x_grad": 5.820766091346741e-11,
81
+ "safe_load_any_loaded": 0.0,
82
+ "safe_load_any_qkv": 0.0,
83
+ "sat_alibi_forward": 0.0,
84
+ "sat_alibi_loss": 0.0,
85
+ "sat_alibi_param_grad": 0.0,
86
+ "sat_alibi_x_grad": 1.1641532182693481e-10
87
+ },
88
+ "tol": 0.0002
89
+ },
90
+ {
91
+ "backend": "manual",
92
+ "d": 128,
93
+ "dk": 16,
94
+ "heads": 8,
95
+ "ok": true,
96
+ "preset": "micro_3x",
97
+ "rank": 48,
98
+ "rows": {
99
+ "causal_alibi_forward": 0.0,
100
+ "causal_alibi_loss": 0.0,
101
+ "causal_alibi_param_grad": 0.0,
102
+ "causal_alibi_x_grad": 5.820766091346741e-11,
103
+ "legacy_load_forward": 0.0,
104
+ "legacy_load_missing_unexpected": 0.0,
105
+ "legacy_load_qkv_weight": 0.0,
106
+ "none_forward": 0.0,
107
+ "none_loss": 0.0,
108
+ "none_param_grad": 0.0,
109
+ "none_x_grad": 1.4551915228366852e-11,
110
+ "safe_load_any_loaded": 0.0,
111
+ "safe_load_any_qkv": 0.0,
112
+ "sat_alibi_forward": 0.0,
113
+ "sat_alibi_loss": 0.0,
114
+ "sat_alibi_param_grad": 0.0,
115
+ "sat_alibi_x_grad": 5.820766091346741e-11
116
+ },
117
+ "tol": 0.0002
118
+ },
119
+ {
120
+ "backend": "sdpa",
121
+ "d": 128,
122
+ "dk": 16,
123
+ "heads": 8,
124
+ "ok": true,
125
+ "preset": "micro_3x",
126
+ "rank": 48,
127
+ "rows": {
128
+ "causal_alibi_forward": 0.0,
129
+ "causal_alibi_loss": 0.0,
130
+ "causal_alibi_param_grad": 0.0,
131
+ "causal_alibi_x_grad": 5.820766091346741e-11,
132
+ "legacy_load_forward": 0.0,
133
+ "legacy_load_missing_unexpected": 0.0,
134
+ "legacy_load_qkv_weight": 0.0,
135
+ "none_forward": 0.0,
136
+ "none_loss": 0.0,
137
+ "none_param_grad": 0.0,
138
+ "none_x_grad": 1.4551915228366852e-11,
139
+ "safe_load_any_loaded": 0.0,
140
+ "safe_load_any_qkv": 0.0,
141
+ "sat_alibi_forward": 0.0,
142
+ "sat_alibi_loss": 0.0,
143
+ "sat_alibi_param_grad": 0.0,
144
+ "sat_alibi_x_grad": 2.9103830456733704e-11
145
+ },
146
+ "tol": 0.0002
147
+ },
148
+ {
149
+ "backend": "sublinear",
150
+ "d": 128,
151
+ "dk": 16,
152
+ "heads": 8,
153
+ "ok": true,
154
+ "preset": "micro_3x",
155
+ "rank": 48,
156
+ "rows": {
157
+ "causal_alibi_forward": 0.0,
158
+ "causal_alibi_loss": 0.0,
159
+ "causal_alibi_param_grad": 0.0,
160
+ "causal_alibi_x_grad": 5.820766091346741e-11,
161
+ "legacy_load_forward": 0.0,
162
+ "legacy_load_missing_unexpected": 0.0,
163
+ "legacy_load_qkv_weight": 0.0,
164
+ "none_forward": 0.0,
165
+ "none_loss": 0.0,
166
+ "none_param_grad": 0.0,
167
+ "none_x_grad": 1.4551915228366852e-11,
168
+ "safe_load_any_loaded": 0.0,
169
+ "safe_load_any_qkv": 0.0,
170
+ "sat_alibi_forward": 0.0,
171
+ "sat_alibi_loss": 0.0,
172
+ "sat_alibi_param_grad": 0.0,
173
+ "sat_alibi_x_grad": 2.9103830456733704e-11
174
+ },
175
+ "tol": 0.0002
176
+ }
177
+ ]
run_agillm4_4090_longblock.sh CHANGED
@@ -16,7 +16,7 @@ mkdir -p /workspace/agillm4_4090_ckpts
16
 
17
  echo "START_AGILLM4_4090_LONG_BLOCK $(date -u +%Y-%m-%dT%H:%M:%SZ) host=$(hostname)"
18
  echo "This is production AGILLM long-block training on 24GB, not a local toy test."
19
- echo "preset=${AGILLM4_4090_PRESET:-large} block=${AGILLM4_4090_BLOCK:-1536} sat_every=1"
20
 
21
  exec python -u /workspace/agillm-4/nB300_agillm4.py train \
22
  --preset "${AGILLM4_4090_PRESET:-large}" \
@@ -27,6 +27,10 @@ exec python -u /workspace/agillm-4/nB300_agillm4.py train \
27
  --grad_checkpoint \
28
  --optimizer "${AGILLM4_4090_OPTIMIZER:-paged_adamw8bit}" \
29
  --sat_every 1 \
 
 
 
 
30
  --token_param_ratio 100 \
31
  --save_dir /workspace/agillm4_4090_ckpts \
32
  --save_every_sec "${AGILLM4_4090_SAVE_EVERY_SEC:-21600}" \
 
16
 
17
  echo "START_AGILLM4_4090_LONG_BLOCK $(date -u +%Y-%m-%dT%H:%M:%SZ) host=$(hostname)"
18
  echo "This is production AGILLM long-block training on 24GB, not a local toy test."
19
+ echo "preset=${AGILLM4_4090_PRESET:-large} block=${AGILLM4_4090_BLOCK:-1536} sat_every=1 nat_every=${AGILLM4_4090_NAT_EVERY:-4}"
20
 
21
  exec python -u /workspace/agillm-4/nB300_agillm4.py train \
22
  --preset "${AGILLM4_4090_PRESET:-large}" \
 
27
  --grad_checkpoint \
28
  --optimizer "${AGILLM4_4090_OPTIMIZER:-paged_adamw8bit}" \
29
  --sat_every 1 \
30
+ --nat_every "${AGILLM4_4090_NAT_EVERY:-4}" \
31
+ --nat_loss_weight "${AGILLM4_4090_NAT_LOSS_WEIGHT:-1.0}" \
32
+ --nat_expand "${AGILLM4_4090_NAT_EXPAND:-2}" \
33
+ --nat_max_tokens "${AGILLM4_4090_NAT_MAX_TOKENS:-768}" \
34
  --token_param_ratio 100 \
35
  --save_dir /workspace/agillm4_4090_ckpts \
36
  --save_every_sec "${AGILLM4_4090_SAVE_EVERY_SEC:-21600}" \
run_agillm4_4090_sublinear_probe.sh CHANGED
@@ -16,5 +16,9 @@ python -u ./nB300_agillm4.py train \
16
  --grad_checkpoint \
17
  --optimizer "${AGILLM4_OPTIMIZER:-paged_adamw8bit}" \
18
  --sat_every 1 \
 
 
 
 
19
  --token_param_ratio 100 \
20
  --save_dir "${AGILLM4_SAVE_DIR:-/workspace/ckpts_agillm4_sublinear_4090}"
 
16
  --grad_checkpoint \
17
  --optimizer "${AGILLM4_OPTIMIZER:-paged_adamw8bit}" \
18
  --sat_every 1 \
19
+ --nat_every "${AGILLM4_NAT_EVERY:-4}" \
20
+ --nat_loss_weight "${AGILLM4_NAT_LOSS_WEIGHT:-1.0}" \
21
+ --nat_expand "${AGILLM4_NAT_EXPAND:-2}" \
22
+ --nat_max_tokens "${AGILLM4_NAT_MAX_TOKENS:-768}" \
23
  --token_param_ratio 100 \
24
  --save_dir "${AGILLM4_SAVE_DIR:-/workspace/ckpts_agillm4_sublinear_4090}"
run_agillm4_main_b200_b300.sh CHANGED
@@ -15,7 +15,7 @@ fi
15
  mkdir -p /workspace/agillm4_ckpts
16
 
17
  echo "START_AGILLM4_MAIN $(date -u +%Y-%m-%dT%H:%M:%SZ) host=$(hostname)"
18
- echo "preset=agillm4_main target_tokens=150000000000 token_param_ratio=100 block=${AGILLM4_BLOCK:-2048}"
19
 
20
  exec python -u /workspace/agillm-4/nB300_agillm4.py train \
21
  --preset agillm4_main \
@@ -26,6 +26,10 @@ exec python -u /workspace/agillm-4/nB300_agillm4.py train \
26
  --grad_checkpoint \
27
  --optimizer "${AGILLM4_OPTIMIZER:-adamw}" \
28
  --sat_every 1 \
 
 
 
 
29
  --token_param_ratio 100 \
30
  --target_tokens 150000000000 \
31
  --save_dir /workspace/agillm4_ckpts \
 
15
  mkdir -p /workspace/agillm4_ckpts
16
 
17
  echo "START_AGILLM4_MAIN $(date -u +%Y-%m-%dT%H:%M:%SZ) host=$(hostname)"
18
+ echo "preset=agillm4_main target_tokens=150000000000 token_param_ratio=100 block=${AGILLM4_BLOCK:-2048} sat_every=1 nat_every=${AGILLM4_NAT_EVERY:-1}"
19
 
20
  exec python -u /workspace/agillm-4/nB300_agillm4.py train \
21
  --preset agillm4_main \
 
26
  --grad_checkpoint \
27
  --optimizer "${AGILLM4_OPTIMIZER:-adamw}" \
28
  --sat_every 1 \
29
+ --nat_every "${AGILLM4_NAT_EVERY:-1}" \
30
+ --nat_loss_weight "${AGILLM4_NAT_LOSS_WEIGHT:-1.0}" \
31
+ --nat_expand "${AGILLM4_NAT_EXPAND:-2}" \
32
+ --nat_max_tokens "${AGILLM4_NAT_MAX_TOKENS:-0}" \
33
  --token_param_ratio 100 \
34
  --target_tokens 150000000000 \
35
  --save_dir /workspace/agillm4_ckpts \
verify_anchor_memory_and_kv_buffer_agillm4.py ADDED
@@ -0,0 +1,121 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ #!/usr/bin/env python3
2
+ """Verifier for AGILLM-4 anchor-memory wiring and KV-buffer harvest.
3
+
4
+ Two harvested/integrated features share this file because both touch
5
+ nB300_agillm4 and both need a regression-safe smoke test:
6
+
7
+ 1. Anchor memory: AnchorMemoryLayer wired into Encoder at a configurable
8
+ mid-stack position. Default-off; enabled with ``--anchor_memory`` and
9
+ parameterised by ``--anchor_stride / --anchor_max / --anchor_position``.
10
+
11
+ 2. KV buffer: ``KVBuffer`` replaces ``torch.cat``-based decode-time cache
12
+ growth. Default-off (legacy tuple cache still works); enabled per
13
+ call site by passing a pre-sized ``KVBuffer`` as ``kv_cache``.
14
+
15
+ Run:
16
+
17
+ python /workspace/agillm-4/verify_anchor_memory_and_kv_buffer_agillm4.py
18
+ """
19
+ from __future__ import annotations
20
+
21
+ import pathlib
22
+ import sys
23
+
24
+ HERE = pathlib.Path(__file__).resolve().parent
25
+ if str(HERE) not in sys.path:
26
+ sys.path.insert(0, str(HERE))
27
+
28
+ import torch # noqa: E402
29
+
30
+ import nB300_agillm4 as m # noqa: E402
31
+ from anchor_memory import AnchorMemoryLayer # noqa: E402
32
+
33
+ DEV = "cuda" if torch.cuda.is_available() else "cpu"
34
+ FAILS = 0
35
+
36
+
37
+ def check(name: str, cond: bool, detail: str = ""):
38
+ global FAILS
39
+ tag = "OK " if cond else "FAIL"
40
+ if not cond:
41
+ FAILS += 1
42
+ print(f" [{tag}] {name}" + (f" — {detail}" if detail else ""))
43
+
44
+
45
+ def section(title: str):
46
+ print(f"\n=== {title} ===")
47
+
48
+
49
+ # ----- 1. KVBuffer -----
50
+ section("KVBuffer fill / view / overflow")
51
+ buf = m.KVBuffer(batch=2, heads=4, capacity=32, d_k=8, device=DEV, dtype=torch.float32)
52
+ check("initial length is 0", buf.length == 0)
53
+ check("k/v allocated", buf.k.shape == (2, 4, 32, 8) and buf.v.shape == (2, 4, 32, 8))
54
+
55
+ k1 = torch.randn(2, 4, 5, 8, device=DEV)
56
+ v1 = torch.randn(2, 4, 5, 8, device=DEV)
57
+ buf.append(k1, v1)
58
+ check("after append(5) length is 5", buf.length == 5)
59
+ k_view, v_view = buf.view()
60
+ check("view shape", k_view.shape == (2, 4, 5, 8))
61
+ check("k content correct", torch.allclose(k_view, k1))
62
+ check("v content correct", torch.allclose(v_view, v1))
63
+
64
+ buf.append(torch.randn(2, 4, 27, 8, device=DEV), torch.randn(2, 4, 27, 8, device=DEV))
65
+ check("filled to capacity (32)", buf.length == 32)
66
+ try:
67
+ buf.append(torch.zeros(2, 4, 1, 8, device=DEV), torch.zeros(2, 4, 1, 8, device=DEV))
68
+ check("overflow raises", False, "no exception")
69
+ except RuntimeError:
70
+ check("overflow raises", True)
71
+
72
+ # ----- 2. Encoder without anchor memory (regression) -----
73
+ section("Encoder regression (anchor disabled)")
74
+ cfg = {"d": 64, "layers": 4, "heads": 4, "rank": 8}
75
+ enc_plain = m.Encoder(cfg).to(DEV)
76
+ check("anchor module is None", enc_plain.anchor is None)
77
+ check("anchor_memory_enabled is False", enc_plain.anchor_memory_enabled is False)
78
+ ids = torch.randint(0, m.VOCAB, (1, 16), device=DEV)
79
+ with torch.no_grad():
80
+ out = enc_plain(ids, mask=None)
81
+ check("plain forward shape", out.shape == (1, 16, 64), str(out.shape))
82
+
83
+ # ----- 3. Encoder with anchor memory -----
84
+ section("Encoder with anchor memory enabled")
85
+ enc = m.Encoder(
86
+ cfg,
87
+ anchor_memory=True,
88
+ anchor_stride=8,
89
+ anchor_max=16,
90
+ anchor_position=-1,
91
+ ).to(DEV)
92
+ check("anchor_memory_enabled", enc.anchor_memory_enabled is True)
93
+ check("anchor is AnchorMemoryLayer", isinstance(enc.anchor, AnchorMemoryLayer))
94
+ check("position resolved to mid-stack (4//2=2)", enc.anchor_position == 2, f"got {enc.anchor_position}")
95
+
96
+ ids2 = torch.randint(0, m.VOCAB, (1, 32), device=DEV)
97
+ out2 = enc(ids2, mask=None)
98
+ check("anchor-on forward shape", out2.shape == (1, 32, 64), str(out2.shape))
99
+
100
+ loss = out2.sum()
101
+ loss.backward()
102
+ grads = [p.grad for p in enc.anchor.parameters() if p.grad is not None]
103
+ check("anchor params got gradients", len(grads) > 0, f"{len(grads)} grads")
104
+ nonzero = sum(int(g.abs().sum() > 0) for g in grads)
105
+ check("anchor gradients non-zero", nonzero > 0, f"{nonzero}/{len(grads)}")
106
+
107
+ # ----- 4. Anchor position resolution -----
108
+ section("Anchor position resolution")
109
+ check("explicit position=1", m.Encoder(cfg, anchor_memory=True, anchor_position=1).anchor_position == 1)
110
+ check("oversized position clamped", m.Encoder(cfg, anchor_memory=True, anchor_position=99).anchor_position == cfg["layers"] - 1)
111
+
112
+ # ----- 5. Module-level defaults exposed -----
113
+ section("Module-level defaults present")
114
+ for name in ("DEFAULT_ANCHOR_MEMORY", "DEFAULT_ANCHOR_STRIDE", "DEFAULT_ANCHOR_MAX", "DEFAULT_ANCHOR_POSITION", "DEFAULT_KV_BUFFER"):
115
+ check(name, hasattr(m, name))
116
+
117
+ print()
118
+ if FAILS:
119
+ print(f"=== {FAILS} CHECK(S) FAILED ===")
120
+ sys.exit(1)
121
+ print("=== ALL CHECKS PASSED ===")