Commit ·
b086478
1
Parent(s): 0bb5c6e
Prepare AGILLM4 AR SAT NAT scaffold
Browse filesRe-add NAT as a real third head beside AR and SAT, wire NAT checkpoints/deltas/inference, initialize NAT in the v4 seed builder, and update Vast launch scripts/docs for NAT cadence.
- AGILLM-4.md +65 -4
- N1_HARVEST.md +21 -7
- README.md +1 -1
- build_v4_seed.py +337 -0
- nB300_agillm4.py +251 -38
- nB300_agillm4.py.bak-20260520-140542 +0 -0
- nB300_agillm4.py.bak-fix-20260520-140710 +0 -0
- remote_verify_m_fold_after_qkv_agillm4.json +118 -0
- remote_verify_m_fold_agillm4.json +118 -0
- remote_verify_qkv_all_backends_agillm4.json +177 -0
- run_agillm4_4090_longblock.sh +5 -1
- run_agillm4_4090_sublinear_probe.sh +4 -0
- run_agillm4_main_b200_b300.sh +5 -1
- verify_anchor_memory_and_kv_buffer_agillm4.py +121 -0
AGILLM-4.md
CHANGED
|
@@ -9,7 +9,7 @@ AGILLM-4 should not be "AGILLM-3 with a larger `--block`." The next useful versi
|
|
| 9 |
|
| 10 |
The near-term codebase target is a measurable long-context trainer that can survive on current hardware:
|
| 11 |
|
| 12 |
-
- RTX 4090: production 24GB long-block lane for the current-size model family, aiming above the current ~1200-token ceiling while preserving AR+SAT.
|
| 13 |
- B200 180GB: serious 2k-16k experiments.
|
| 14 |
- B300 262GB: 8k-64k experiments, then memory-augmented tests.
|
| 15 |
- Multi-GPU/cluster: ring or sequence parallel experiments later.
|
|
@@ -43,10 +43,12 @@ python -u /workspace/agillm-4/nB300_agillm4.py train \
|
|
| 43 |
--attn_backend sdpa \
|
| 44 |
--grad_checkpoint \
|
| 45 |
--optimizer paged_adamw8bit \
|
| 46 |
-
--sat_every 1
|
|
|
|
|
|
|
| 47 |
```
|
| 48 |
|
| 49 |
-
Important: `--sat_every 1` keeps SAT trained every step. The AGILLM-4 code now backprops AR and
|
| 50 |
|
| 51 |
Escalation ladder on 4090:
|
| 52 |
|
|
@@ -55,7 +57,55 @@ Escalation ladder on 4090:
|
|
| 55 |
3. `block=1792`
|
| 56 |
4. `block=2048`
|
| 57 |
|
| 58 |
-
If 8-bit optimizer is unavailable, install `bitsandbytes` rather than dropping the long-block target. SAT remains active every step;
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 59 |
|
| 60 |
## First Implemented Scaffold
|
| 61 |
|
|
@@ -64,6 +114,7 @@ If 8-bit optimizer is unavailable, install `bitsandbytes` rather than dropping t
|
|
| 64 |
- `--attn_backend manual|sdpa`
|
| 65 |
- `--grad_checkpoint`
|
| 66 |
- AGILLM-4 defaults for chat SFT compatibility inherited from the patched AGILLM-3 file.
|
|
|
|
| 67 |
|
| 68 |
`block_sweep_agillm4.py` runs synthetic forward/backward sweeps:
|
| 69 |
|
|
@@ -222,6 +273,16 @@ Initial hypothesis:
|
|
| 222 |
- Anchor memory length: 512-4096 anchors.
|
| 223 |
- Token attention stays cheap while memory covers hundreds of thousands of tokens.
|
| 224 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 225 |
### Compressive Recurrence
|
| 226 |
|
| 227 |
Keep a recurrent memory bank updated by a learned compressor:
|
|
|
|
| 9 |
|
| 10 |
The near-term codebase target is a measurable long-context trainer that can survive on current hardware:
|
| 11 |
|
| 12 |
+
- RTX 4090: production 24GB long-block lane for the current-size model family, aiming above the current ~1200-token ceiling while preserving AR+SAT+NAT.
|
| 13 |
- B200 180GB: serious 2k-16k experiments.
|
| 14 |
- B300 262GB: 8k-64k experiments, then memory-augmented tests.
|
| 15 |
- Multi-GPU/cluster: ring or sequence parallel experiments later.
|
|
|
|
| 43 |
--attn_backend sdpa \
|
| 44 |
--grad_checkpoint \
|
| 45 |
--optimizer paged_adamw8bit \
|
| 46 |
+
--sat_every 1 \
|
| 47 |
+
--nat_every 4 \
|
| 48 |
+
--nat_max_tokens 768
|
| 49 |
```
|
| 50 |
|
| 51 |
+
Important: `--sat_every 1 --nat_every 4` keeps SAT trained every step and NAT active on a cadence that fits 24GB cards. On B200/B300 use `--nat_every 1` for full AR+SAT+NAT every step. The AGILLM-4 code now backprops AR, SAT, and NAT sequentially, so the objective remains joint while peak VRAM is lower than holding all activation graphs at once.
|
| 52 |
|
| 53 |
Escalation ladder on 4090:
|
| 54 |
|
|
|
|
| 57 |
3. `block=1792`
|
| 58 |
4. `block=2048`
|
| 59 |
|
| 60 |
+
If 8-bit optimizer is unavailable, install `bitsandbytes` rather than dropping the long-block target. SAT remains active every step; NAT should stay enabled with a slower cadence or `--nat_max_tokens` cap on 24GB. The code lowers peak memory by backpropagating AR, SAT, and NAT sequentially, not by deleting heads.
|
| 61 |
+
|
| 62 |
+
## Warm Start from AGILLM-3 (function-preserving)
|
| 63 |
+
|
| 64 |
+
The next AGILLM-4 run does **not** start from a random init. `build_v4_seed.py`
|
| 65 |
+
turns the most recent AGILLM-3 SFT checkpoint into a function-preserving v4
|
| 66 |
+
seed:
|
| 67 |
+
|
| 68 |
+
```bash
|
| 69 |
+
python /workspace/agillm-4/build_v4_seed.py \
|
| 70 |
+
--from-ckpt /workspace/ckpts_sft_math_v1/final.pt \
|
| 71 |
+
--v4-preset main \
|
| 72 |
+
--out /workspace/agillm-4/agillm4_seed_from_v3.pt
|
| 73 |
+
```
|
| 74 |
+
|
| 75 |
+
Construction rules (v3 d=1024 / H=16 / r=128 / L=24 → v4 d=1536 / H=24 / r=192 / L=32):
|
| 76 |
+
|
| 77 |
+
- Embedding, AR head, and NAT head pad with **zeros** in new `d_v4 - d_v3` columns. New-dim
|
| 78 |
+
embeddings are zero — they propagate harmlessly through the rest of the
|
| 79 |
+
graph because all paths that touch them are zero-padded too.
|
| 80 |
+
- If the v3 checkpoint has no NAT head, `build_v4_seed.py` initializes NAT from
|
| 81 |
+
the expanded AR projection rather than random weights.
|
| 82 |
+
- Block params: `qkv`, `proj`, `ff.0`, `ff.2` are placed top-left into the
|
| 83 |
+
larger v4 tensors; the rest is zero. Old heads contribute exactly the same;
|
| 84 |
+
new heads' attention output is multiplied by zero proj columns, so their
|
| 85 |
+
contribution is zero at step 0.
|
| 86 |
+
- `U` rank 128 → 192 by appending zero columns. `U @ U.T` is unchanged, so the
|
| 87 |
+
M-fold metric is bit-identical.
|
| 88 |
+
- LayerNorm new dims initialise to weight=1, bias=0.
|
| 89 |
+
- Extra layers 25..32 are identity-init: `mha.proj` and `ff.2` are zero, so the
|
| 90 |
+
residual stream passes through unchanged. They start contributing only once
|
| 91 |
+
training shapes them.
|
| 92 |
+
|
| 93 |
+
Net: at step 0, `v4(x) == v3(x)` up to floating-point rounding for any input
|
| 94 |
+
within v3's tokeniser vocabulary. The new capacity has zero impact on day-1
|
| 95 |
+
loss; training subsequently shapes it.
|
| 96 |
+
|
| 97 |
+
Use with `--warmstart_from`:
|
| 98 |
+
|
| 99 |
+
```bash
|
| 100 |
+
python /workspace/agillm-4/nB300_agillm4.py train \
|
| 101 |
+
--preset large \
|
| 102 |
+
--warmstart_from /workspace/agillm-4/agillm4_seed_from_v3.pt \
|
| 103 |
+
--batch_size 1 --block 1536 --amp --grad_checkpoint --sat_every 1 --nat_every 1 \
|
| 104 |
+
--anchor_memory --anchor_stride 256 --anchor_max 2048
|
| 105 |
+
```
|
| 106 |
+
|
| 107 |
+
The seed file is ~6 GB on disk (fp32 tensors). Rebuild it whenever a newer v3
|
| 108 |
+
SFT final is preferred (e.g. swap chat-v2 for the math-v1 final).
|
| 109 |
|
| 110 |
## First Implemented Scaffold
|
| 111 |
|
|
|
|
| 114 |
- `--attn_backend manual|sdpa`
|
| 115 |
- `--grad_checkpoint`
|
| 116 |
- AGILLM-4 defaults for chat SFT compatibility inherited from the patched AGILLM-3 file.
|
| 117 |
+
- A real NAT head is re-added beside AR and SAT. NAT trains with a CTC loss over an expanded non-causal input (`--nat_expand 2` by default). `--nat_every` controls cadence; `--nat_max_tokens` can cap the NAT slice on 24GB runs without removing the head.
|
| 118 |
|
| 119 |
`block_sweep_agillm4.py` runs synthetic forward/backward sweeps:
|
| 120 |
|
|
|
|
| 273 |
- Anchor memory length: 512-4096 anchors.
|
| 274 |
- Token attention stays cheap while memory covers hundreds of thousands of tokens.
|
| 275 |
|
| 276 |
+
Status: wired into `Encoder` as a single `AnchorMemoryLayer` inserted after a configurable mid-stack block. The module is the existing `anchor_memory.AnchorMemoryLayer` (attention-pooled compressor + gated cross-attention read from a rolling bank of `max_anchors` anchors). Enable per-run with:
|
| 277 |
+
|
| 278 |
+
```bash
|
| 279 |
+
python /workspace/agillm-4/nB300_agillm4.py train \
|
| 280 |
+
--preset large --batch_size 1 --block 1536 --amp --grad_checkpoint \
|
| 281 |
+
--anchor_memory --anchor_stride 256 --anchor_max 2048
|
| 282 |
+
```
|
| 283 |
+
|
| 284 |
+
`--anchor_position N` overrides the default mid-stack placement. Verifier: `verify_anchor_memory_and_kv_buffer_agillm4.py`.
|
| 285 |
+
|
| 286 |
### Compressive Recurrence
|
| 287 |
|
| 288 |
Keep a recurrent memory bank updated by a learned compressor:
|
N1_HARVEST.md
CHANGED
|
@@ -70,6 +70,27 @@ The verifier checks fused-vs-unfused forward output, loss, input gradients,
|
|
| 70 |
parameter gradients, strict legacy state-dict loading, `_safe_load_any`
|
| 71 |
warm-start loading, and optimizer-state remap.
|
| 72 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 73 |
## Next Candidates
|
| 74 |
|
| 75 |
### 3. Combined ALiBi + Mask Cache
|
|
@@ -93,13 +114,6 @@ n1 can compact `U` spectra post-training and save compatible checkpoints.
|
|
| 93 |
Risk: optimizer state must be dropped or remapped carefully; do only as a
|
| 94 |
separate command, never during a live training run.
|
| 95 |
|
| 96 |
-
### 6. KV Cache Buffer
|
| 97 |
-
|
| 98 |
-
n1 replaces repeated decode-time `torch.cat` cache growth with preallocated KV
|
| 99 |
-
buffers.
|
| 100 |
-
|
| 101 |
-
Risk: cache object type touches AR, SAT, and future spec decoding paths.
|
| 102 |
-
|
| 103 |
## Rule
|
| 104 |
|
| 105 |
Every harvested feature needs its own AGILLM-4 verifier or profile artifact.
|
|
|
|
| 70 |
parameter gradients, strict legacy state-dict loading, `_safe_load_any`
|
| 71 |
warm-start loading, and optimizer-state remap.
|
| 72 |
|
| 73 |
+
### 3. KV Cache Buffer
|
| 74 |
+
|
| 75 |
+
Status: done.
|
| 76 |
+
|
| 77 |
+
`KVBuffer` is a preallocated head-major `[B, H, T, d_k]` slab for decode-time
|
| 78 |
+
K/V. Callers size it once and call `append(k_new, v_new)` per step instead of
|
| 79 |
+
re-running `torch.cat` and reallocating the cache tensor every token.
|
| 80 |
+
`TuneableAttentionMHA.forward` accepts either a legacy `(k, v)` tuple cache
|
| 81 |
+
(no behaviour change) or a `KVBuffer` instance, and returns the same cache
|
| 82 |
+
object it received so the caller can reuse it across steps. Overflow raises;
|
| 83 |
+
callers must size the buffer to the maximum decode length.
|
| 84 |
+
|
| 85 |
+
Verification:
|
| 86 |
+
|
| 87 |
+
```bash
|
| 88 |
+
python /workspace/agillm-4/verify_anchor_memory_and_kv_buffer_agillm4.py
|
| 89 |
+
```
|
| 90 |
+
|
| 91 |
+
The verifier exercises buffer fill, slice view, overflow handling, and confirms
|
| 92 |
+
the legacy tuple-cache code path still works (regression test).
|
| 93 |
+
|
| 94 |
## Next Candidates
|
| 95 |
|
| 96 |
### 3. Combined ALiBi + Mask Cache
|
|
|
|
| 114 |
Risk: optimizer state must be dropped or remapped carefully; do only as a
|
| 115 |
separate command, never during a live training run.
|
| 116 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
## Rule
|
| 118 |
|
| 119 |
Every harvested feature needs its own AGILLM-4 verifier or profile artifact.
|
README.md
CHANGED
|
@@ -17,7 +17,7 @@ and extended for:
|
|
| 17 |
- ~1.5B parameter main preset (`agillm4_main`)
|
| 18 |
- 100 tokens per parameter target ratio
|
| 19 |
- longer block-size work on 24GB, B200, and B300 class GPUs
|
| 20 |
-
- AR+SAT
|
| 21 |
- SDPA and experimental sublinear local+landmark attention backends
|
| 22 |
- exact M-fold expansion attention harvested from n1.py, with local verifier
|
| 23 |
- fused QKV projection harvested from n1.py, with legacy checkpoint loading
|
|
|
|
| 17 |
- ~1.5B parameter main preset (`agillm4_main`)
|
| 18 |
- 100 tokens per parameter target ratio
|
| 19 |
- longer block-size work on 24GB, B200, and B300 class GPUs
|
| 20 |
+
- AR+SAT+NAT training, with sequential backward to reduce peak VRAM
|
| 21 |
- SDPA and experimental sublinear local+landmark attention backends
|
| 22 |
- exact M-fold expansion attention harvested from n1.py, with local verifier
|
| 23 |
- fused QKV projection harvested from n1.py, with legacy checkpoint loading
|
build_v4_seed.py
ADDED
|
@@ -0,0 +1,337 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Build an AGILLM-4 warm-start "seed" from an AGILLM-3 checkpoint.
|
| 3 |
+
|
| 4 |
+
The seed is function-preserving at step 0: v4 forward pass with this seed
|
| 5 |
+
produces (up to floating-point rounding) the same logits v3 produced for the
|
| 6 |
+
same inputs. The new width / heads / layers add capacity that contributes zero
|
| 7 |
+
at init and gets shaped by training from a useful starting point — far better
|
| 8 |
+
than a random init for the post-SFT v3 lineage.
|
| 9 |
+
|
| 10 |
+
Construction rules (v3 d_v3/h_v3/rank_v3 → v4 d_v4/h_v4/rank_v4):
|
| 11 |
+
|
| 12 |
+
emb [V, d_v3] → [V, d_v4]; new dims = 0
|
| 13 |
+
ln.weight/bias [d_v3] → [d_v4]; new weight=1, new bias=0
|
| 14 |
+
mha.qkv [3*d_v3, d_v3] → [3*d_v4, d_v4] per-slab top-left copy, rest 0
|
| 15 |
+
(new dims see 0 input; new heads' rows are 0)
|
| 16 |
+
mha.proj [d_v3, hdk_v3] → [d_v4, hdk_v4] top-left copy, rest 0
|
| 17 |
+
(new heads' columns zero → contribute nothing)
|
| 18 |
+
mha.U [dk, rank_v3] → [dk, rank_v4] left columns copy, rest 0
|
| 19 |
+
(U @ U.T preserved exactly)
|
| 20 |
+
ff.0 [4d_v3, d_v3] → [4d_v4, d_v4] top-left copy, rest 0
|
| 21 |
+
ff.2 [d_v3, 4d_v3] → [d_v4, 4d_v4] top-left copy, rest 0
|
| 22 |
+
ar/nat.proj [V, d_v3] → [V, d_v4]; new dim columns = 0
|
| 23 |
+
|
| 24 |
+
Layers (24 → 32): keep the v3 stack in place at the bottom of the v4 stack;
|
| 25 |
+
append 8 identity-init blocks at the top. Identity init: ln{1,2}=I,
|
| 26 |
+
mha.proj=0, ff.2=0 → output equals input (residual passthrough).
|
| 27 |
+
"""
|
| 28 |
+
from __future__ import annotations
|
| 29 |
+
|
| 30 |
+
import argparse
|
| 31 |
+
import json
|
| 32 |
+
import pathlib
|
| 33 |
+
import sys
|
| 34 |
+
import time
|
| 35 |
+
|
| 36 |
+
import torch
|
| 37 |
+
import torch.nn as nn
|
| 38 |
+
|
| 39 |
+
# v4 main preset (matches agillm4_config.json)
|
| 40 |
+
V4_PRESET = dict(d=1536, layers=32, heads=24, rank=192)
|
| 41 |
+
# v3 from log header / slot 200
|
| 42 |
+
V3_PRESET = dict(d=1024, layers=24, heads=16, rank=128)
|
| 43 |
+
DK = 64 # both share dk; sanity-checked below
|
| 44 |
+
|
| 45 |
+
|
| 46 |
+
def expand_emb(w_v3: torch.Tensor, d_v4: int) -> torch.Tensor:
|
| 47 |
+
V, d_v3 = w_v3.shape
|
| 48 |
+
out = torch.zeros(V, d_v4, dtype=w_v3.dtype)
|
| 49 |
+
out[:, :d_v3] = w_v3
|
| 50 |
+
return out
|
| 51 |
+
|
| 52 |
+
|
| 53 |
+
def expand_ln(w_v3: torch.Tensor, b_v3: torch.Tensor, d_v4: int):
|
| 54 |
+
new_w = torch.ones(d_v4, dtype=w_v3.dtype)
|
| 55 |
+
new_w[: w_v3.numel()] = w_v3
|
| 56 |
+
new_b = torch.zeros(d_v4, dtype=b_v3.dtype)
|
| 57 |
+
new_b[: b_v3.numel()] = b_v3
|
| 58 |
+
return new_w, new_b
|
| 59 |
+
|
| 60 |
+
|
| 61 |
+
def expand_qkv(qkv_v3: torch.Tensor, d_v3: int, d_v4: int, h_v3: int, h_v4: int, dk: int) -> torch.Tensor:
|
| 62 |
+
"""[3*d_v3, d_v3] → [3*d_v4, d_v4] via per-slab top-left placement."""
|
| 63 |
+
assert qkv_v3.shape == (3 * d_v3, d_v3), qkv_v3.shape
|
| 64 |
+
Q, K, V = qkv_v3[:d_v3], qkv_v3[d_v3 : 2 * d_v3], qkv_v3[2 * d_v3 :]
|
| 65 |
+
out = torch.zeros(3 * d_v4, d_v4, dtype=qkv_v3.dtype)
|
| 66 |
+
for i, slab in enumerate((Q, K, V)):
|
| 67 |
+
# slab is [h_v3 * dk, d_v3]
|
| 68 |
+
out[i * d_v4 : i * d_v4 + h_v3 * dk, :d_v3] = slab
|
| 69 |
+
return out
|
| 70 |
+
|
| 71 |
+
|
| 72 |
+
def expand_proj(proj_v3: torch.Tensor, d_v3: int, d_v4: int, h_v3: int, h_v4: int, dk: int) -> torch.Tensor:
|
| 73 |
+
"""[d_v3, h_v3*dk] → [d_v4, h_v4*dk] top-left placement."""
|
| 74 |
+
out = torch.zeros(d_v4, h_v4 * dk, dtype=proj_v3.dtype)
|
| 75 |
+
out[:d_v3, : h_v3 * dk] = proj_v3
|
| 76 |
+
return out
|
| 77 |
+
|
| 78 |
+
|
| 79 |
+
def expand_U(u_v3: torch.Tensor, dk: int, rank_v4: int) -> torch.Tensor:
|
| 80 |
+
"""[dk, rank_v3] → [dk, rank_v4] via left-column placement.
|
| 81 |
+
|
| 82 |
+
Critical: rank_v3 columns are preserved, new columns are zero.
|
| 83 |
+
Then U_new @ U_new.T == U_v3 @ U_v3.T exactly. M-fold metric is unchanged.
|
| 84 |
+
"""
|
| 85 |
+
out = torch.zeros(dk, rank_v4, dtype=u_v3.dtype)
|
| 86 |
+
out[:, : u_v3.size(1)] = u_v3
|
| 87 |
+
return out
|
| 88 |
+
|
| 89 |
+
|
| 90 |
+
def expand_ff_first(w_v3: torch.Tensor, d_v3: int, d_v4: int) -> torch.Tensor:
|
| 91 |
+
"""[4*d_v3, d_v3] → [4*d_v4, d_v4]."""
|
| 92 |
+
out = torch.zeros(4 * d_v4, d_v4, dtype=w_v3.dtype)
|
| 93 |
+
out[: 4 * d_v3, :d_v3] = w_v3
|
| 94 |
+
return out
|
| 95 |
+
|
| 96 |
+
|
| 97 |
+
def expand_ff_first_bias(b_v3: torch.Tensor, d_v4: int) -> torch.Tensor:
|
| 98 |
+
out = torch.zeros(4 * d_v4, dtype=b_v3.dtype)
|
| 99 |
+
out[: b_v3.numel()] = b_v3
|
| 100 |
+
return out
|
| 101 |
+
|
| 102 |
+
|
| 103 |
+
def expand_ff_second(w_v3: torch.Tensor, d_v3: int, d_v4: int) -> torch.Tensor:
|
| 104 |
+
"""[d_v3, 4*d_v3] → [d_v4, 4*d_v4]."""
|
| 105 |
+
out = torch.zeros(d_v4, 4 * d_v4, dtype=w_v3.dtype)
|
| 106 |
+
out[:d_v3, : 4 * d_v3] = w_v3
|
| 107 |
+
return out
|
| 108 |
+
|
| 109 |
+
|
| 110 |
+
def expand_ff_second_bias(b_v3: torch.Tensor, d_v4: int) -> torch.Tensor:
|
| 111 |
+
out = torch.zeros(d_v4, dtype=b_v3.dtype)
|
| 112 |
+
out[: b_v3.numel()] = b_v3
|
| 113 |
+
return out
|
| 114 |
+
|
| 115 |
+
|
| 116 |
+
def expand_arhead(w_v3: torch.Tensor, b_v3: torch.Tensor | None, d_v4: int):
|
| 117 |
+
V, d_v3 = w_v3.shape
|
| 118 |
+
out_w = torch.zeros(V, d_v4, dtype=w_v3.dtype)
|
| 119 |
+
out_w[:, :d_v3] = w_v3
|
| 120 |
+
out_b = None
|
| 121 |
+
if b_v3 is not None:
|
| 122 |
+
out_b = b_v3.clone() # bias dim is V — does not change with d
|
| 123 |
+
return out_w, out_b
|
| 124 |
+
|
| 125 |
+
|
| 126 |
+
def identity_block_state(d_v4: int, h_v4: int, rank_v4: int, dk: int, prefix: str):
|
| 127 |
+
"""A v4 block initialized so its residual contribution is exactly zero."""
|
| 128 |
+
sd = {
|
| 129 |
+
f"{prefix}.ln1.weight": torch.ones(d_v4),
|
| 130 |
+
f"{prefix}.ln1.bias": torch.zeros(d_v4),
|
| 131 |
+
f"{prefix}.ln2.weight": torch.ones(d_v4),
|
| 132 |
+
f"{prefix}.ln2.bias": torch.zeros(d_v4),
|
| 133 |
+
# mha: qkv arbitrary (we zero it), U orthogonal, proj zero -> attention contributes 0
|
| 134 |
+
f"{prefix}.mha.qkv.weight": torch.zeros(3 * d_v4, d_v4),
|
| 135 |
+
f"{prefix}.mha.U": _orthogonal(dk, rank_v4),
|
| 136 |
+
f"{prefix}.mha.proj.weight": torch.zeros(d_v4, h_v4 * dk),
|
| 137 |
+
# ff: first layer small random so gradient signal exists; second zero -> contributes 0
|
| 138 |
+
f"{prefix}.ff.0.weight": torch.empty(4 * d_v4, d_v4).normal_(0, 0.02),
|
| 139 |
+
f"{prefix}.ff.0.bias": torch.zeros(4 * d_v4),
|
| 140 |
+
f"{prefix}.ff.2.weight": torch.zeros(d_v4, 4 * d_v4),
|
| 141 |
+
f"{prefix}.ff.2.bias": torch.zeros(d_v4),
|
| 142 |
+
}
|
| 143 |
+
return sd
|
| 144 |
+
|
| 145 |
+
|
| 146 |
+
def _orthogonal(dk: int, rank: int) -> torch.Tensor:
|
| 147 |
+
t = torch.empty(dk, rank)
|
| 148 |
+
nn.init.orthogonal_(t)
|
| 149 |
+
return t
|
| 150 |
+
|
| 151 |
+
|
| 152 |
+
def find_default_v3_ckpt() -> pathlib.Path:
|
| 153 |
+
candidates = [
|
| 154 |
+
pathlib.Path("/workspace/ckpts_sft_math_v1/final.pt"), # math SFT final (latest)
|
| 155 |
+
pathlib.Path("/workspace/ckpts_sft_chat_v2/final.pt"), # chat SFT v2 final
|
| 156 |
+
pathlib.Path("/workspace/ckpts_sft_chat/final.pt"), # chat SFT v1 final
|
| 157 |
+
]
|
| 158 |
+
for c in candidates:
|
| 159 |
+
if c.exists():
|
| 160 |
+
return c
|
| 161 |
+
raise SystemExit("No v3 final checkpoint found in expected locations.")
|
| 162 |
+
|
| 163 |
+
|
| 164 |
+
def main() -> int:
|
| 165 |
+
p = argparse.ArgumentParser(description="Build a v4 warm-start seed from v3.")
|
| 166 |
+
p.add_argument("--from-ckpt", type=pathlib.Path, default=None,
|
| 167 |
+
help="v3 checkpoint .pt (default: most recent in /workspace/ckpts_sft_*)")
|
| 168 |
+
p.add_argument("--out", type=pathlib.Path,
|
| 169 |
+
default=pathlib.Path("/workspace/agillm-4/agillm4_seed_from_v3.pt"))
|
| 170 |
+
p.add_argument("--v4-preset", type=str, default="main",
|
| 171 |
+
choices=["main", "floor", "big"])
|
| 172 |
+
args = p.parse_args()
|
| 173 |
+
|
| 174 |
+
src = args.from_ckpt or find_default_v3_ckpt()
|
| 175 |
+
print(f"v3 source : {src}")
|
| 176 |
+
if not src.exists():
|
| 177 |
+
print(f"FATAL: source missing: {src}", file=sys.stderr)
|
| 178 |
+
return 2
|
| 179 |
+
|
| 180 |
+
v4 = {"main": V4_PRESET,
|
| 181 |
+
"floor": dict(d=1280, layers=28, heads=20, rank=160),
|
| 182 |
+
"big": dict(d=1792, layers=36, heads=28, rank=224)}[args.v4_preset]
|
| 183 |
+
v3 = V3_PRESET
|
| 184 |
+
dk_v3 = v3["d"] // v3["heads"]
|
| 185 |
+
dk_v4 = v4["d"] // v4["heads"]
|
| 186 |
+
assert dk_v3 == dk_v4 == DK, f"dk mismatch: v3={dk_v3} v4={dk_v4} DK={DK}"
|
| 187 |
+
print(f"v3 preset : {v3}")
|
| 188 |
+
print(f"v4 preset : {v4}")
|
| 189 |
+
|
| 190 |
+
t0 = time.time()
|
| 191 |
+
blob = torch.load(src, map_location="cpu", weights_only=False)
|
| 192 |
+
print(f"loaded ckpt keys={list(blob.keys())} in {time.time()-t0:.1f}s")
|
| 193 |
+
|
| 194 |
+
core_v3 = blob.get("core", blob)
|
| 195 |
+
ar_v3 = blob.get("ar", None)
|
| 196 |
+
nat_v3 = blob.get("nat", None)
|
| 197 |
+
sat_v3 = blob.get("sat", None)
|
| 198 |
+
|
| 199 |
+
# === core (Encoder) ===
|
| 200 |
+
core_v4 = {}
|
| 201 |
+
# Embedding
|
| 202 |
+
if "emb.weight" in core_v3:
|
| 203 |
+
core_v4["emb.weight"] = expand_emb(core_v3["emb.weight"], v4["d"])
|
| 204 |
+
print(f"emb.weight : {tuple(core_v3['emb.weight'].shape)} -> {tuple(core_v4['emb.weight'].shape)}")
|
| 205 |
+
else:
|
| 206 |
+
print("WARN: emb.weight not found in v3 core; v4 emb will be random later.")
|
| 207 |
+
|
| 208 |
+
# Per-block expansion: keep v3 blocks at positions 0..L_v3-1
|
| 209 |
+
n_v3 = v3["layers"]
|
| 210 |
+
n_v4 = v4["layers"]
|
| 211 |
+
n_new = n_v4 - n_v3
|
| 212 |
+
for i in range(n_v3):
|
| 213 |
+
pfx_v3 = f"blocks.{i}"
|
| 214 |
+
pfx_v4 = f"blocks.{i}"
|
| 215 |
+
# ln1
|
| 216 |
+
if f"{pfx_v3}.ln1.weight" in core_v3:
|
| 217 |
+
w, b = expand_ln(core_v3[f"{pfx_v3}.ln1.weight"], core_v3[f"{pfx_v3}.ln1.bias"], v4["d"])
|
| 218 |
+
core_v4[f"{pfx_v4}.ln1.weight"] = w
|
| 219 |
+
core_v4[f"{pfx_v4}.ln1.bias"] = b
|
| 220 |
+
if f"{pfx_v3}.ln2.weight" in core_v3:
|
| 221 |
+
w, b = expand_ln(core_v3[f"{pfx_v3}.ln2.weight"], core_v3[f"{pfx_v3}.ln2.bias"], v4["d"])
|
| 222 |
+
core_v4[f"{pfx_v4}.ln2.weight"] = w
|
| 223 |
+
core_v4[f"{pfx_v4}.ln2.bias"] = b
|
| 224 |
+
# qkv (might be fused or legacy q/k/v)
|
| 225 |
+
if f"{pfx_v3}.mha.qkv.weight" in core_v3:
|
| 226 |
+
qkv = core_v3[f"{pfx_v3}.mha.qkv.weight"]
|
| 227 |
+
elif all(f"{pfx_v3}.mha.{k}.weight" in core_v3 for k in ("q", "k", "v")):
|
| 228 |
+
qkv = torch.cat([core_v3[f"{pfx_v3}.mha.{k}.weight"] for k in ("q", "k", "v")], dim=0)
|
| 229 |
+
else:
|
| 230 |
+
qkv = None
|
| 231 |
+
if qkv is not None:
|
| 232 |
+
core_v4[f"{pfx_v4}.mha.qkv.weight"] = expand_qkv(qkv, v3["d"], v4["d"], v3["heads"], v4["heads"], DK)
|
| 233 |
+
# U
|
| 234 |
+
if f"{pfx_v3}.mha.U" in core_v3:
|
| 235 |
+
core_v4[f"{pfx_v4}.mha.U"] = expand_U(core_v3[f"{pfx_v3}.mha.U"], DK, v4["rank"])
|
| 236 |
+
# proj
|
| 237 |
+
if f"{pfx_v3}.mha.proj.weight" in core_v3:
|
| 238 |
+
core_v4[f"{pfx_v4}.mha.proj.weight"] = expand_proj(
|
| 239 |
+
core_v3[f"{pfx_v3}.mha.proj.weight"], v3["d"], v4["d"], v3["heads"], v4["heads"], DK
|
| 240 |
+
)
|
| 241 |
+
# ff (Sequential(Linear, ReLU, Linear) → keys ff.0.* and ff.2.*)
|
| 242 |
+
if f"{pfx_v3}.ff.0.weight" in core_v3:
|
| 243 |
+
core_v4[f"{pfx_v4}.ff.0.weight"] = expand_ff_first(core_v3[f"{pfx_v3}.ff.0.weight"], v3["d"], v4["d"])
|
| 244 |
+
if f"{pfx_v3}.ff.0.bias" in core_v3:
|
| 245 |
+
core_v4[f"{pfx_v4}.ff.0.bias"] = expand_ff_first_bias(core_v3[f"{pfx_v3}.ff.0.bias"], v4["d"])
|
| 246 |
+
if f"{pfx_v3}.ff.2.weight" in core_v3:
|
| 247 |
+
core_v4[f"{pfx_v4}.ff.2.weight"] = expand_ff_second(core_v3[f"{pfx_v3}.ff.2.weight"], v3["d"], v4["d"])
|
| 248 |
+
if f"{pfx_v3}.ff.2.bias" in core_v3:
|
| 249 |
+
core_v4[f"{pfx_v4}.ff.2.bias"] = expand_ff_second_bias(core_v3[f"{pfx_v3}.ff.2.bias"], v4["d"])
|
| 250 |
+
|
| 251 |
+
# Identity-init the new top-of-stack blocks
|
| 252 |
+
for j in range(n_new):
|
| 253 |
+
i_v4 = n_v3 + j
|
| 254 |
+
core_v4.update(identity_block_state(v4["d"], v4["heads"], v4["rank"], DK, prefix=f"blocks.{i_v4}"))
|
| 255 |
+
|
| 256 |
+
# Final LN
|
| 257 |
+
if "ln.weight" in core_v3:
|
| 258 |
+
w, b = expand_ln(core_v3["ln.weight"], core_v3["ln.bias"], v4["d"])
|
| 259 |
+
core_v4["ln.weight"] = w
|
| 260 |
+
core_v4["ln.bias"] = b
|
| 261 |
+
|
| 262 |
+
# === AR head ===
|
| 263 |
+
ar_v4 = None
|
| 264 |
+
if ar_v3:
|
| 265 |
+
ar_v4 = {}
|
| 266 |
+
if "proj.weight" in ar_v3:
|
| 267 |
+
w, b = expand_arhead(ar_v3["proj.weight"], ar_v3.get("proj.bias"), v4["d"])
|
| 268 |
+
ar_v4["proj.weight"] = w
|
| 269 |
+
if b is not None:
|
| 270 |
+
ar_v4["proj.bias"] = b
|
| 271 |
+
|
| 272 |
+
# === NAT head ===
|
| 273 |
+
# Prefer a source NAT head when available. AGILLM-3 checkpoints usually
|
| 274 |
+
# have no NAT head, so seed NAT from expanded AR rather than random init.
|
| 275 |
+
nat_v4 = None
|
| 276 |
+
if nat_v3 and "proj.weight" in nat_v3:
|
| 277 |
+
nat_v4 = {}
|
| 278 |
+
w, b = expand_arhead(nat_v3["proj.weight"], nat_v3.get("proj.bias"), v4["d"])
|
| 279 |
+
nat_v4["proj.weight"] = w
|
| 280 |
+
if b is not None:
|
| 281 |
+
nat_v4["proj.bias"] = b
|
| 282 |
+
elif ar_v4:
|
| 283 |
+
nat_v4 = {k: v.clone() if torch.is_tensor(v) else v for k, v in ar_v4.items()}
|
| 284 |
+
|
| 285 |
+
# === SAT head ===
|
| 286 |
+
# Keep v3's SAT state if present and re-shape only the d-axis tensors;
|
| 287 |
+
# SAT internals vary, so we copy what fits and let _safe_load_any drop the rest.
|
| 288 |
+
sat_v4 = None
|
| 289 |
+
if sat_v3:
|
| 290 |
+
sat_v4 = {}
|
| 291 |
+
for k, t in sat_v3.items():
|
| 292 |
+
if isinstance(t, torch.Tensor):
|
| 293 |
+
# Pad any dimension equal to d_v3 up to d_v4 with zeros
|
| 294 |
+
pads = []
|
| 295 |
+
new_shape = list(t.shape)
|
| 296 |
+
changed = False
|
| 297 |
+
for dim_i, sz in enumerate(t.shape):
|
| 298 |
+
if sz == v3["d"]:
|
| 299 |
+
new_shape[dim_i] = v4["d"]
|
| 300 |
+
changed = True
|
| 301 |
+
if changed:
|
| 302 |
+
out = torch.zeros(*new_shape, dtype=t.dtype)
|
| 303 |
+
slices = tuple(slice(0, s) for s in t.shape)
|
| 304 |
+
out[slices] = t
|
| 305 |
+
sat_v4[k] = out
|
| 306 |
+
else:
|
| 307 |
+
sat_v4[k] = t.clone()
|
| 308 |
+
else:
|
| 309 |
+
sat_v4[k] = t
|
| 310 |
+
|
| 311 |
+
# Save bundle
|
| 312 |
+
out_blob = {"core": core_v4}
|
| 313 |
+
if ar_v4: out_blob["ar"] = ar_v4
|
| 314 |
+
if nat_v4: out_blob["nat"] = nat_v4
|
| 315 |
+
if sat_v4: out_blob["sat"] = sat_v4
|
| 316 |
+
out_blob["seed_meta"] = {
|
| 317 |
+
"built_from": str(src),
|
| 318 |
+
"v3_preset": v3,
|
| 319 |
+
"v4_preset_name": args.v4_preset,
|
| 320 |
+
"v4_preset": v4,
|
| 321 |
+
"n_new_layers": n_new,
|
| 322 |
+
"tool": pathlib.Path(__file__).name,
|
| 323 |
+
"tool_version": 2,
|
| 324 |
+
}
|
| 325 |
+
|
| 326 |
+
args.out.parent.mkdir(parents=True, exist_ok=True)
|
| 327 |
+
t0 = time.time()
|
| 328 |
+
torch.save(out_blob, args.out)
|
| 329 |
+
sz = args.out.stat().st_size
|
| 330 |
+
print(f"saved seed : {args.out} ({sz/1e9:.2f} GB) in {time.time()-t0:.1f}s")
|
| 331 |
+
print(f"core keys : {len(core_v4)} (expected ~{n_v4 * 9 + 3})")
|
| 332 |
+
print(f"meta : {json.dumps(out_blob['seed_meta'], indent=2)}")
|
| 333 |
+
return 0
|
| 334 |
+
|
| 335 |
+
|
| 336 |
+
if __name__ == "__main__":
|
| 337 |
+
sys.exit(main())
|
nB300_agillm4.py
CHANGED
|
@@ -1,6 +1,6 @@
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
|
| 3 |
-
# n.py
|
| 4 |
# Enhanced inference: checkpoint name, tok/s, UK time
|
| 5 |
|
| 6 |
from __future__ import annotations
|
|
@@ -586,6 +586,7 @@ _maybe_handle_status_fastpath()
|
|
| 586 |
|
| 587 |
import torch
|
| 588 |
import torch.utils.checkpoint as torch_checkpoint
|
|
|
|
| 589 |
|
| 590 |
# SafeProgress - Claude-safe progress (discrete lines, not single growing line)
|
| 591 |
class SafeProgress:
|
|
@@ -684,6 +685,7 @@ SYNTHETIC_TOKENIZER = os.environ.get("AGILLM_SYNTHETIC_TOKENIZER", "").lower() i
|
|
| 684 |
|
| 685 |
class _SyntheticTokenizer:
|
| 686 |
pad_token = "<|pad|>"
|
|
|
|
| 687 |
eos_token_id = 1
|
| 688 |
sep_token_id = 1
|
| 689 |
|
|
@@ -796,8 +798,9 @@ def _tokenizer_health_check(tokenizer):
|
|
| 796 |
if not SYNTHETIC_TOKENIZER:
|
| 797 |
_tokenizer_health_check(tok)
|
| 798 |
|
| 799 |
-
VOCAB, EOS = (
|
| 800 |
max(tok.get_vocab().values()) + 1,
|
|
|
|
| 801 |
tok.eos_token_id if tok.eos_token_id is not None else tok.sep_token_id
|
| 802 |
)
|
| 803 |
|
|
@@ -861,6 +864,11 @@ DEFAULT_SUBLINEAR_WINDOW = _env_int("AGILLM_SUBLINEAR_WINDOW", 256)
|
|
| 861 |
DEFAULT_SUBLINEAR_STRIDE = _env_int("AGILLM_SUBLINEAR_STRIDE", 64)
|
| 862 |
DEFAULT_SUBLINEAR_MAX_ANCHORS = _env_int("AGILLM_SUBLINEAR_MAX_ANCHORS", 256)
|
| 863 |
DEFAULT_SUBLINEAR_CHUNK = _env_int("AGILLM_SUBLINEAR_CHUNK", 128)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 864 |
AGILLM4_TOKEN_PARAM_RATIO = 100.0
|
| 865 |
|
| 866 |
# ───────────────────────── UK Time Helper ─────────────────────────
|
|
@@ -1125,6 +1133,46 @@ def alibi_bias(n_heads: int, n_tokens: int):
|
|
| 1125 |
return -_alibi_slopes(n_heads) * dist
|
| 1126 |
|
| 1127 |
# ───────────────────────── Model components ─────────────────────────
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1128 |
class TuneableAttentionMHA(nn.Module):
|
| 1129 |
def __init__(
|
| 1130 |
self,
|
|
@@ -1302,8 +1350,14 @@ class TuneableAttentionMHA(nn.Module):
|
|
| 1302 |
else:
|
| 1303 |
q = self._proj_qk(q_lin)
|
| 1304 |
k_new = self._proj_qk(k_lin)
|
| 1305 |
-
if kv_cache is None:
|
| 1306 |
k, v = k_new, v_new
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1307 |
else:
|
| 1308 |
k_cached, v_cached = kv_cache
|
| 1309 |
if use_cache:
|
|
@@ -1337,7 +1391,10 @@ class TuneableAttentionMHA(nn.Module):
|
|
| 1337 |
z = att.softmax(-1) @ v
|
| 1338 |
z = z.transpose(1, 2).reshape(x.size(0), x.size(1), -1)
|
| 1339 |
out = self.drop(self.proj(z))
|
| 1340 |
-
|
|
|
|
|
|
|
|
|
|
| 1341 |
|
| 1342 |
|
| 1343 |
class Block(nn.Module):
|
|
@@ -1388,6 +1445,10 @@ class Encoder(nn.Module):
|
|
| 1388 |
sublinear_stride: int = DEFAULT_SUBLINEAR_STRIDE,
|
| 1389 |
sublinear_max_anchors: int = DEFAULT_SUBLINEAR_MAX_ANCHORS,
|
| 1390 |
sublinear_chunk: int = DEFAULT_SUBLINEAR_CHUNK,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1391 |
):
|
| 1392 |
super().__init__()
|
| 1393 |
d, l, h, r = cfg["d"], cfg["layers"], cfg["heads"], cfg["rank"]
|
|
@@ -1413,21 +1474,46 @@ class Encoder(nn.Module):
|
|
| 1413 |
self.sublinear_stride = sublinear_stride
|
| 1414 |
self.sublinear_max_anchors = sublinear_max_anchors
|
| 1415 |
self.sublinear_chunk = sublinear_chunk
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1416 |
|
| 1417 |
def forward(self, ids, mask, kv_caches=None, use_cache=False, total_seq_len=None):
|
| 1418 |
x = self.emb(ids)
|
| 1419 |
if not use_cache:
|
| 1420 |
-
for blk in self.blocks:
|
| 1421 |
if self.grad_checkpoint and self.training:
|
| 1422 |
x = torch_checkpoint.checkpoint(blk, x, mask, use_reentrant=False)
|
| 1423 |
else:
|
| 1424 |
x = blk(x, mask)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1425 |
return self.ln(x)
|
| 1426 |
new_kvs = []
|
| 1427 |
for i, blk in enumerate(self.blocks):
|
| 1428 |
kv = kv_caches[i] if kv_caches else None
|
| 1429 |
x, kv_out = blk(x, mask, kv, use_cache=True, total_seq_len=total_seq_len)
|
| 1430 |
new_kvs.append(kv_out)
|
|
|
|
|
|
|
| 1431 |
return self.ln(x), new_kvs
|
| 1432 |
|
| 1433 |
|
|
@@ -1445,6 +1531,15 @@ class ARHead(nn.Module):
|
|
| 1445 |
return self.proj(h)
|
| 1446 |
|
| 1447 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1448 |
class SATHead(nn.Module):
|
| 1449 |
def __init__(self, d, mode="var"):
|
| 1450 |
super().__init__()
|
|
@@ -1567,15 +1662,17 @@ def _clone_opt_value(value):
|
|
| 1567 |
return value.detach().clone()
|
| 1568 |
return copy.deepcopy(value)
|
| 1569 |
|
| 1570 |
-
def _optimizer_param_name_lookup(core, ar_h, sat_h) -> dict[int, str]:
|
| 1571 |
out = {}
|
| 1572 |
-
for prefix, module in (("core", core), ("ar", ar_h), ("sat", sat_h)):
|
|
|
|
|
|
|
| 1573 |
for name, param in module.named_parameters():
|
| 1574 |
out.setdefault(id(param), f"{prefix}.{name}")
|
| 1575 |
return out
|
| 1576 |
|
| 1577 |
-
def _optimizer_group_param_names(opt, core, ar_h, sat_h) -> List[List[str]]:
|
| 1578 |
-
lookup = _optimizer_param_name_lookup(core, ar_h, sat_h)
|
| 1579 |
return [
|
| 1580 |
[lookup.get(id(param), f"<unknown:{id(param)}>") for param in group["params"]]
|
| 1581 |
for group in opt.param_groups
|
|
@@ -1606,12 +1703,12 @@ def _fuse_legacy_optimizer_param_state(states: List[dict]) -> Optional[dict]:
|
|
| 1606 |
out[key] = copy.deepcopy(vals[0])
|
| 1607 |
return out
|
| 1608 |
|
| 1609 |
-
def _fuse_legacy_qkv_optimizer_state(opt_state: dict, opt, core, ar_h, sat_h) -> Optional[dict]:
|
| 1610 |
"""Remap pre-QKV-fusion AdamW state to the current fused parameter layout."""
|
| 1611 |
if not isinstance(opt_state, dict) or "state" not in opt_state or "param_groups" not in opt_state:
|
| 1612 |
return None
|
| 1613 |
current_sd = opt.state_dict()
|
| 1614 |
-
current_names = _optimizer_group_param_names(opt, core, ar_h, sat_h)
|
| 1615 |
legacy_names = [
|
| 1616 |
[legacy for name in group_names for legacy in _legacy_names_for_current_param(name)]
|
| 1617 |
for group_names in current_names
|
|
@@ -1653,7 +1750,7 @@ def _fuse_legacy_qkv_optimizer_state(opt_state: dict, opt, core, ar_h, sat_h) ->
|
|
| 1653 |
|
| 1654 |
return {"state": new_states, "param_groups": new_groups}
|
| 1655 |
|
| 1656 |
-
def save_delta(core, ar_h, sat_h, step: int, seen_tok: int, save_dir: pathlib.Path, phase_name: str):
|
| 1657 |
"""Save weight-only delta in background thread. Non-blocking."""
|
| 1658 |
global _delta_thread
|
| 1659 |
# Wait for any previous delta write to finish
|
|
@@ -1666,6 +1763,8 @@ def save_delta(core, ar_h, sat_h, step: int, seen_tok: int, save_dir: pathlib.Pa
|
|
| 1666 |
"ar": {k: v.detach().cpu() for k, v in _checkpoint_state_dict(ar_h).items()},
|
| 1667 |
"sat": {k: v.detach().cpu() for k, v in _checkpoint_state_dict(sat_h).items()},
|
| 1668 |
}
|
|
|
|
|
|
|
| 1669 |
meta = {"step": step, "seen_tok": seen_tok, "wall_time": time.time(), "delta": True}
|
| 1670 |
path = save_dir / f"{phase_name}_delta_step{step:08d}.pt"
|
| 1671 |
_delta_thread = threading.Thread(target=_do_delta_save, args=(tensors, path, meta), daemon=True)
|
|
@@ -1694,7 +1793,7 @@ def _prune_deltas(save_dir: pathlib.Path, phase_name: str, max_deltas: int):
|
|
| 1694 |
except Exception as e:
|
| 1695 |
print(f" [delta-prune] error: {e}")
|
| 1696 |
|
| 1697 |
-
def load_delta(path: pathlib.Path, core, ar_h, sat_h):
|
| 1698 |
"""Load weight-only delta. Returns (step, seen_tok) or raises."""
|
| 1699 |
# Verify checksum if sidecar exists
|
| 1700 |
sha_path = path.with_suffix(".sha256")
|
|
@@ -1710,6 +1809,12 @@ def load_delta(path: pathlib.Path, core, ar_h, sat_h):
|
|
| 1710 |
core.load_state_dict(_fuse_qkv_in_state_dict(_strip_orig_mod_prefix(ck["weights"]["core"])))
|
| 1711 |
ar_h.load_state_dict(_strip_orig_mod_prefix(ck["weights"]["ar"]))
|
| 1712 |
sat_h.load_state_dict(_strip_orig_mod_prefix(ck["weights"]["sat"]))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1713 |
return ck.get("step", 0), ck.get("seen_tok", 0)
|
| 1714 |
|
| 1715 |
def _flush_delta():
|
|
@@ -1719,7 +1824,7 @@ def _flush_delta():
|
|
| 1719 |
print(" [delta] flushing in-flight write...")
|
| 1720 |
_delta_thread.join(timeout=120)
|
| 1721 |
|
| 1722 |
-
def save_ckpt(path: pathlib.Path, core, ar_h, sat_h, opt, scaler, meta):
|
| 1723 |
path.parent.mkdir(exist_ok=True, parents=True)
|
| 1724 |
tmp = path.with_suffix(path.suffix + ".tmp")
|
| 1725 |
state = {
|
|
@@ -1732,22 +1837,29 @@ def save_ckpt(path: pathlib.Path, core, ar_h, sat_h, opt, scaler, meta):
|
|
| 1732 |
"tie_weights": meta.get("tie_weights", False),
|
| 1733 |
**{k: v for k, v in meta.items() if k not in ("cfg", "tie_weights")}
|
| 1734 |
}
|
|
|
|
|
|
|
| 1735 |
torch.save(state, tmp, _use_new_zipfile_serialization=False)
|
| 1736 |
tmp.replace(path)
|
| 1737 |
(path.parent / "latest.json").write_text(json.dumps({"path": str(path), "step": meta["step"]}))
|
| 1738 |
print(f"\n✓ saved checkpoint {path.name}")
|
| 1739 |
|
| 1740 |
-
def load_ckpt(path, core, ar_h, sat_h, opt, scaler):
|
| 1741 |
p = _resolve_ckpt(path) or path
|
| 1742 |
ck = _try_load(p, map_location="cpu")
|
| 1743 |
if ck is None: raise FileNotFoundError(f"No valid checkpoint at {p}")
|
| 1744 |
core.load_state_dict(_fuse_qkv_in_state_dict(_strip_orig_mod_prefix(ck["core"])))
|
| 1745 |
ar_h.load_state_dict(_strip_orig_mod_prefix(ck["ar"]))
|
| 1746 |
sat_h.load_state_dict(_strip_orig_mod_prefix(ck["sat"]))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1747 |
try:
|
| 1748 |
opt.load_state_dict(ck["opt"])
|
| 1749 |
except Exception as exc:
|
| 1750 |
-
fused_opt = _fuse_legacy_qkv_optimizer_state(ck.get("opt"), opt, core, ar_h, sat_h)
|
| 1751 |
if fused_opt is not None:
|
| 1752 |
try:
|
| 1753 |
opt.load_state_dict(fused_opt)
|
|
@@ -1834,15 +1946,18 @@ def _phase_freeze(core: nn.Module, *, freeze_core: bool, unfreeze_ln: bool, trai
|
|
| 1834 |
if train_emb:
|
| 1835 |
for p in core.emb.parameters(): p.requires_grad = True
|
| 1836 |
|
| 1837 |
-
def _optimizer_param_groups(core, ar_h, sat_h, lr_core: float, lr_head: float):
|
| 1838 |
-
|
| 1839 |
{"params": [p for p in core.parameters() if p.requires_grad], "lr": lr_core},
|
| 1840 |
{"params": ar_h.parameters(), "lr": lr_head},
|
| 1841 |
{"params": sat_h.parameters(), "lr": lr_head},
|
| 1842 |
]
|
|
|
|
|
|
|
|
|
|
| 1843 |
|
| 1844 |
-
def make_optimizer(args, core, ar_h, sat_h, lr_core: float, lr_head: float):
|
| 1845 |
-
groups = _optimizer_param_groups(core, ar_h, sat_h, lr_core, lr_head)
|
| 1846 |
opt_name = getattr(args, "optimizer", "adamw")
|
| 1847 |
if opt_name == "adamw":
|
| 1848 |
return torch.optim.AdamW(groups)
|
|
@@ -1858,9 +1973,14 @@ def make_optimizer(args, core, ar_h, sat_h, lr_core: float, lr_head: float):
|
|
| 1858 |
return bnb.optim.AdamW8bit(groups)
|
| 1859 |
raise ValueError(f"unknown optimizer: {opt_name}")
|
| 1860 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1861 |
def _train_phase(
|
| 1862 |
args, phase_name: str,
|
| 1863 |
-
core, ar_h, sat_h, opt, scaler,
|
| 1864 |
start_step, seen_tok, resume_wall_time,
|
| 1865 |
cfg, source, steps, block_size, batch_size,
|
| 1866 |
chat_cfg: dict,
|
|
@@ -1875,7 +1995,7 @@ def _train_phase(
|
|
| 1875 |
target_tokens = target_tokens_override
|
| 1876 |
else:
|
| 1877 |
ratio = _target_token_ratio(args)
|
| 1878 |
-
param_count = _count_enabled_params(core, ar_h, sat_h)
|
| 1879 |
target_tokens = int(ratio * param_count)
|
| 1880 |
print(f"[{phase_name}] token_param_ratio={ratio:g} param_count={param_count:,} target_tokens={target_tokens:,}")
|
| 1881 |
if steps:
|
|
@@ -1896,6 +2016,7 @@ def _train_phase(
|
|
| 1896 |
)
|
| 1897 |
ce_tok = nn.CrossEntropyLoss(label_smoothing=0.1)
|
| 1898 |
ce_gate = nn.CrossEntropyLoss()
|
|
|
|
| 1899 |
pbar = SafeProgress(total=total_tokens_needed, initial=seen_tok, unit="tok")
|
| 1900 |
grow_plan = _parse_grow_plan(args.grow_plan) if args.auto_grow else []
|
| 1901 |
buf: list[int] = []
|
|
@@ -1908,7 +2029,10 @@ def _train_phase(
|
|
| 1908 |
last_save_mono = time.monotonic() - (now_wall - (resume_wall_time or now_wall))
|
| 1909 |
last_delta_step = start_step
|
| 1910 |
print(f"[{phase_name}] Starting. Goal: {total_tokens_needed:,} tokens. Batch={BATCH}, Block={BLOCK}")
|
| 1911 |
-
print(
|
|
|
|
|
|
|
|
|
|
| 1912 |
while seen_tok < total_tokens_needed:
|
| 1913 |
try:
|
| 1914 |
while len(buf) < BLOCK:
|
|
@@ -1945,8 +2069,28 @@ def _train_phase(
|
|
| 1945 |
loss_value += float(loss_sat.detach().item())
|
| 1946 |
scaler.scale(loss_sat).backward()
|
| 1947 |
del h_sat, logits_sat, loss_sat
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1948 |
scaler.unscale_(opt)
|
| 1949 |
-
nn.utils.clip_grad_norm_(
|
| 1950 |
scaler.step(opt)
|
| 1951 |
scaler.update()
|
| 1952 |
opt.zero_grad(set_to_none=True)
|
|
@@ -1999,7 +2143,7 @@ def _train_phase(
|
|
| 1999 |
ck_name = f"{phase_name}_step{step:08d}.pt"
|
| 2000 |
_flush_delta() # wait for any in-flight delta before full save
|
| 2001 |
_prune_checkpoints(pathlib.Path(args.save_dir), phase_name, max_ckpts)
|
| 2002 |
-
save_ckpt(pathlib.Path(args.save_dir) / ck_name, core, ar_h, sat_h, opt, scaler,
|
| 2003 |
meta={"cfg": cfg, "step": step, "seen_tok": seen_tok, "wall_time": time.time(), "tie_weights": tie_weights})
|
| 2004 |
last_save_mono = now_mono
|
| 2005 |
# Prune old deltas after a full save (they're superseded)
|
|
@@ -2008,7 +2152,7 @@ def _train_phase(
|
|
| 2008 |
# ── Delta checkpoint (step-based, weight-only, async) ──
|
| 2009 |
if args.delta_every_steps > 0 and (step - last_delta_step) >= args.delta_every_steps:
|
| 2010 |
_prune_deltas(pathlib.Path(args.save_dir), phase_name, args.delta_max_keep)
|
| 2011 |
-
save_delta(core, ar_h, sat_h, step, seen_tok, pathlib.Path(args.save_dir), phase_name)
|
| 2012 |
last_delta_step = step
|
| 2013 |
if args.auto_grow:
|
| 2014 |
steps_since_last_grow += 1
|
|
@@ -2025,7 +2169,7 @@ def _train_phase(
|
|
| 2025 |
pbar.close()
|
| 2026 |
_flush_delta() # ensure any in-flight delta completes before final save
|
| 2027 |
if phase_name != "sft":
|
| 2028 |
-
save_ckpt(pathlib.Path(args.save_dir) / f"{phase_name}_final.pt", core, ar_h, sat_h, opt, scaler,
|
| 2029 |
meta={"cfg": cfg, "step": step, "seen_tok": seen_tok, "wall_time": time.time(), "tie_weights": tie_weights})
|
| 2030 |
else:
|
| 2031 |
print("[sft] Skipping duplicate sft_final.pt; final.pt will contain the SFT result.")
|
|
@@ -2062,10 +2206,15 @@ def train(args):
|
|
| 2062 |
sublinear_stride=args.sublinear_stride,
|
| 2063 |
sublinear_max_anchors=args.sublinear_max_anchors,
|
| 2064 |
sublinear_chunk=args.sublinear_chunk,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2065 |
).to(DEV)
|
| 2066 |
ar_h = ARHead(cfg["d"], tie_weights=tie_weights, embedding_weight=core.emb.weight if tie_weights else None).to(DEV)
|
| 2067 |
sat_h = SATHead(cfg["d"], mode="var").to(DEV)
|
| 2068 |
-
|
|
|
|
| 2069 |
print(f"Total parameters: {total_params:,}")
|
| 2070 |
if tie_weights:
|
| 2071 |
print(f"{Colors.WARN}[weight-tying] Embedding and LM head share weights{Colors.RESET}")
|
|
@@ -2076,17 +2225,20 @@ def train(args):
|
|
| 2076 |
loaded = _safe_load_any(src, core, key="core")
|
| 2077 |
_safe_load_any(src, ar_h, key="ar")
|
| 2078 |
_safe_load_any(src, sat_h, key="sat")
|
|
|
|
|
|
|
|
|
|
| 2079 |
if loaded: print(f"Warm-start loaded from {src}")
|
| 2080 |
_phase_freeze(core, freeze_core=args.freeze_core, unfreeze_ln=args.unfreeze_ln, train_emb=args.train_emb)
|
| 2081 |
-
opt = make_optimizer(args, core, ar_h, sat_h, args.lr_core, args.lr_head)
|
| 2082 |
scaler = GradScaler(enabled=(args.amp and _needs_grad_scaler()))
|
| 2083 |
start_step, seen_tok, last_wall = 0, 0, None
|
| 2084 |
if args.resume_delta and not args.fresh:
|
| 2085 |
-
delta_step, delta_tok = load_delta(pathlib.Path(args.resume_delta), core, ar_h, sat_h)
|
| 2086 |
start_step, seen_tok, last_wall = delta_step, delta_tok, None
|
| 2087 |
print(f"Resumed from DELTA at step {start_step} (optimizer state reset — momentum rebuilds in ~100 steps)")
|
| 2088 |
elif args.resume and not args.fresh:
|
| 2089 |
-
start_step, seen_tok, last_wall = load_ckpt(pathlib.Path(args.resume), core, ar_h, sat_h, opt, scaler)
|
| 2090 |
print(f"Resumed from step {start_step}")
|
| 2091 |
# torch.compile AFTER loading checkpoint (key names differ)
|
| 2092 |
if args.compile:
|
|
@@ -2094,9 +2246,10 @@ def train(args):
|
|
| 2094 |
core = torch.compile(core, mode="reduce-overhead")
|
| 2095 |
ar_h = torch.compile(ar_h, mode="reduce-overhead")
|
| 2096 |
sat_h = torch.compile(sat_h, mode="reduce-overhead")
|
|
|
|
| 2097 |
print("[torch.compile] Done.")
|
| 2098 |
step, seen_tok, last_wall = _train_phase(
|
| 2099 |
-
args, "pretrain", core, ar_h, sat_h, opt, scaler,
|
| 2100 |
start_step, seen_tok, last_wall, cfg,
|
| 2101 |
args.source, args.steps,
|
| 2102 |
args.block or DEFAULT_BLOCK,
|
|
@@ -2124,9 +2277,10 @@ def train(args):
|
|
| 2124 |
sat_h,
|
| 2125 |
args.after_sft_lr_core or args.lr_core,
|
| 2126 |
args.after_sft_lr_head or args.lr_head,
|
|
|
|
| 2127 |
)
|
| 2128 |
step, seen_tok, last_wall = _train_phase(
|
| 2129 |
-
args, "sft", core, ar_h, sat_h, opt, scaler,
|
| 2130 |
step, seen_tok, last_wall, cfg,
|
| 2131 |
args.after_sft_source, args.after_sft_steps,
|
| 2132 |
args.after_sft_block or DEFAULT_AFTER_SFT_BLOCK,
|
|
@@ -2142,7 +2296,7 @@ def train(args):
|
|
| 2142 |
tie_weights=tie_weights,
|
| 2143 |
streaming=True
|
| 2144 |
)
|
| 2145 |
-
save_ckpt(pathlib.Path(args.save_dir) / "final.pt", core, ar_h, sat_h, opt, scaler,
|
| 2146 |
meta={"cfg": cfg, "step": step, "seen_tok": seen_tok, "wall_time": time.time(), "tie_weights": tie_weights})
|
| 2147 |
print("🎉 All Training Complete")
|
| 2148 |
|
|
@@ -2182,7 +2336,7 @@ def infer(args):
|
|
| 2182 |
if args.frequency_penalty is None: args.frequency_penalty = 0.3
|
| 2183 |
if args.penalty_last_n is None: args.penalty_last_n = 128
|
| 2184 |
if args.var is None: args.var = False
|
| 2185 |
-
|
| 2186 |
if args.temperature is None: args.temperature = 0.5
|
| 2187 |
if args.top_k is None: args.top_k = 30
|
| 2188 |
if args.repetition_penalty is None: args.repetition_penalty = 2.0
|
|
@@ -2190,6 +2344,14 @@ def infer(args):
|
|
| 2190 |
if args.frequency_penalty is None: args.frequency_penalty = 1.0
|
| 2191 |
if args.penalty_last_n is None: args.penalty_last_n = 200
|
| 2192 |
if args.var is None: args.var = True
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2193 |
path = _resolve_ckpt(pathlib.Path(args.ckpt)) or pathlib.Path(args.ckpt)
|
| 2194 |
sd = torch.load(path, map_location="cpu")
|
| 2195 |
# Restore tokenizer from checkpoint if available
|
|
@@ -2210,10 +2372,12 @@ def infer(args):
|
|
| 2210 |
print("[infer] Delta checkpoint detected, using large preset cfg")
|
| 2211 |
cfg = PRESETS["large"].copy()
|
| 2212 |
tie_weights = False
|
| 2213 |
-
# Remap: delta stores under sd["weights"]["core"/"ar"/"sat"]
|
| 2214 |
sd["core"] = sd["weights"]["core"]
|
| 2215 |
sd["ar"] = sd["weights"]["ar"]
|
| 2216 |
sd["sat"] = sd["weights"]["sat"]
|
|
|
|
|
|
|
| 2217 |
else:
|
| 2218 |
cfg = sd["cfg"]
|
| 2219 |
tie_weights = sd.get("tie_weights", False)
|
|
@@ -2242,16 +2406,27 @@ def infer(args):
|
|
| 2242 |
sublinear_stride=args.sublinear_stride,
|
| 2243 |
sublinear_max_anchors=args.sublinear_max_anchors,
|
| 2244 |
sublinear_chunk=args.sublinear_chunk,
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2245 |
).to(DEV)
|
| 2246 |
ar_h = ARHead(cfg["d"], tie_weights=tie_weights, embedding_weight=core.emb.weight if tie_weights else None).to(DEV)
|
| 2247 |
sat_h = SATHead(cfg["d"]).to(DEV)
|
|
|
|
| 2248 |
core.load_state_dict(sd["core"])
|
| 2249 |
ar_h.load_state_dict(sd["ar"])
|
| 2250 |
sat_h.load_state_dict(sd["sat"])
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2251 |
core.eval()
|
| 2252 |
ar_h.eval()
|
| 2253 |
sat_h.eval()
|
| 2254 |
-
|
|
|
|
|
|
|
| 2255 |
if total_params >= 1_000_000_000:
|
| 2256 |
param_str = f"{total_params / 1_000_000_000:.2f}B"
|
| 2257 |
elif total_params >= 1_000_000:
|
|
@@ -2283,6 +2458,25 @@ def infer(args):
|
|
| 2283 |
nxt = _sample(logits, args.temperature, args.top_k, args.top_p, args.min_p, args.greedy)
|
| 2284 |
ids = torch.cat([ids, nxt], 1)
|
| 2285 |
h, kvs = core(ids[:, -1:], None, kv_caches=kvs, use_cache=True, total_seq_len=ids.size(1))
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2286 |
else:
|
| 2287 |
cached_len = ids.size(1)
|
| 2288 |
h, kvs = core(ids, sat_mask(ids.size(1)), use_cache=True, total_seq_len=cached_len)
|
|
@@ -2362,6 +2556,16 @@ def main():
|
|
| 2362 |
help="For --attn_backend sublinear, cap landmark candidates per query chunk.")
|
| 2363 |
tr.add_argument("--sublinear_chunk", type=int, default=DEFAULT_SUBLINEAR_CHUNK,
|
| 2364 |
help="For --attn_backend sublinear, query chunk size controlling peak gather memory.")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2365 |
tr.add_argument("--optimizer", choices=["adamw", "adamw8bit", "paged_adamw8bit"], default="adamw",
|
| 2366 |
help="Optimizer backend. 8-bit options reduce VRAM on 24GB production runs.")
|
| 2367 |
tr.add_argument("--save_every_sec", type=int, default=DEFAULT_SAVE_SEC)
|
|
@@ -2379,6 +2583,13 @@ def main():
|
|
| 2379 |
tr.add_argument("--ar_only", action="store_true")
|
| 2380 |
tr.add_argument("--sat_every", type=int, default=1,
|
| 2381 |
help="Train SAT every N steps. Default 1 keeps AR+SAT every step.")
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 2382 |
tr.add_argument("--freeze_core", action="store_true")
|
| 2383 |
tr.add_argument("--unfreeze_ln", action="store_true")
|
| 2384 |
tr.add_argument("--train_emb", action="store_true")
|
|
@@ -2404,7 +2615,7 @@ def main():
|
|
| 2404 |
tr.add_argument("--after_sft_lr_core", type=float, default=0.0)
|
| 2405 |
tr.add_argument("--after_sft_lr_head", type=float, default=0.0)
|
| 2406 |
inf = sub.add_parser("infer")
|
| 2407 |
-
inf.add_argument("--mode", choices=["ar", "sat"], required=True)
|
| 2408 |
inf.add_argument("--ckpt", required=True)
|
| 2409 |
inf.add_argument("--prompt", required=True)
|
| 2410 |
inf.add_argument("--max_new", type=int, default=120)
|
|
@@ -2426,6 +2637,8 @@ def main():
|
|
| 2426 |
inf.add_argument("--sublinear_stride", type=int, default=DEFAULT_SUBLINEAR_STRIDE)
|
| 2427 |
inf.add_argument("--sublinear_max_anchors", type=int, default=DEFAULT_SUBLINEAR_MAX_ANCHORS)
|
| 2428 |
inf.add_argument("--sublinear_chunk", type=int, default=DEFAULT_SUBLINEAR_CHUNK)
|
|
|
|
|
|
|
| 2429 |
st = sub.add_parser("status", help="Read-only training status")
|
| 2430 |
st.add_argument("--json", dest="json_output", action="store_true")
|
| 2431 |
st.add_argument("--log", type=str, default=str(STATUS_DEFAULT_LOG))
|
|
|
|
| 1 |
#!/usr/bin/env python3
|
| 2 |
|
| 3 |
+
# n.py - Joint AR+SAT+NAT Trainer with Expansion Ratio Testing
|
| 4 |
# Enhanced inference: checkpoint name, tok/s, UK time
|
| 5 |
|
| 6 |
from __future__ import annotations
|
|
|
|
| 586 |
|
| 587 |
import torch
|
| 588 |
import torch.utils.checkpoint as torch_checkpoint
|
| 589 |
+
from anchor_memory import AnchorMemoryConfig, AnchorMemoryLayer
|
| 590 |
|
| 591 |
# SafeProgress - Claude-safe progress (discrete lines, not single growing line)
|
| 592 |
class SafeProgress:
|
|
|
|
| 685 |
|
| 686 |
class _SyntheticTokenizer:
|
| 687 |
pad_token = "<|pad|>"
|
| 688 |
+
pad_token_id = 0
|
| 689 |
eos_token_id = 1
|
| 690 |
sep_token_id = 1
|
| 691 |
|
|
|
|
| 798 |
if not SYNTHETIC_TOKENIZER:
|
| 799 |
_tokenizer_health_check(tok)
|
| 800 |
|
| 801 |
+
VOCAB, BLANK, EOS = (
|
| 802 |
max(tok.get_vocab().values()) + 1,
|
| 803 |
+
int(getattr(tok, "pad_token_id", 0) or 0),
|
| 804 |
tok.eos_token_id if tok.eos_token_id is not None else tok.sep_token_id
|
| 805 |
)
|
| 806 |
|
|
|
|
| 864 |
DEFAULT_SUBLINEAR_STRIDE = _env_int("AGILLM_SUBLINEAR_STRIDE", 64)
|
| 865 |
DEFAULT_SUBLINEAR_MAX_ANCHORS = _env_int("AGILLM_SUBLINEAR_MAX_ANCHORS", 256)
|
| 866 |
DEFAULT_SUBLINEAR_CHUNK = _env_int("AGILLM_SUBLINEAR_CHUNK", 128)
|
| 867 |
+
DEFAULT_ANCHOR_MEMORY = bool(_env_int("AGILLM_ANCHOR_MEMORY", 0))
|
| 868 |
+
DEFAULT_ANCHOR_STRIDE = _env_int("AGILLM_ANCHOR_STRIDE", 256)
|
| 869 |
+
DEFAULT_ANCHOR_MAX = _env_int("AGILLM_ANCHOR_MAX", 2048)
|
| 870 |
+
DEFAULT_ANCHOR_POSITION = _env_int("AGILLM_ANCHOR_POSITION", -1) # -1 = stack middle
|
| 871 |
+
DEFAULT_KV_BUFFER = bool(_env_int("AGILLM_KV_BUFFER", 0))
|
| 872 |
AGILLM4_TOKEN_PARAM_RATIO = 100.0
|
| 873 |
|
| 874 |
# ───────────────────────── UK Time Helper ─────────────────────────
|
|
|
|
| 1133 |
return -_alibi_slopes(n_heads) * dist
|
| 1134 |
|
| 1135 |
# ───────────────────────── Model components ─────────────────────────
|
| 1136 |
+
class KVBuffer:
|
| 1137 |
+
"""Preallocated K/V cache for decode. Replaces torch.cat-based growth.
|
| 1138 |
+
|
| 1139 |
+
Layout matches MHA-internal head-major shape [B, H, T, d_k]. Caller sizes
|
| 1140 |
+
once; each ``append`` writes ``length:length+n`` slots in place and grows
|
| 1141 |
+
``length``. ``view()`` returns slices of the live region so attention sees
|
| 1142 |
+
only filled positions.
|
| 1143 |
+
"""
|
| 1144 |
+
|
| 1145 |
+
__slots__ = ("k", "v", "length", "capacity")
|
| 1146 |
+
|
| 1147 |
+
def __init__(
|
| 1148 |
+
self,
|
| 1149 |
+
batch: int,
|
| 1150 |
+
heads: int,
|
| 1151 |
+
capacity: int,
|
| 1152 |
+
d_k: int,
|
| 1153 |
+
device,
|
| 1154 |
+
dtype,
|
| 1155 |
+
):
|
| 1156 |
+
self.k = torch.empty(batch, heads, capacity, d_k, device=device, dtype=dtype)
|
| 1157 |
+
self.v = torch.empty(batch, heads, capacity, d_k, device=device, dtype=dtype)
|
| 1158 |
+
self.length = 0
|
| 1159 |
+
self.capacity = capacity
|
| 1160 |
+
|
| 1161 |
+
def append(self, k_new: torch.Tensor, v_new: torch.Tensor):
|
| 1162 |
+
n = k_new.size(2)
|
| 1163 |
+
end = self.length + n
|
| 1164 |
+
if end > self.capacity:
|
| 1165 |
+
raise RuntimeError(
|
| 1166 |
+
f"KVBuffer overflow: length={self.length} + n={n} > capacity={self.capacity}"
|
| 1167 |
+
)
|
| 1168 |
+
self.k[:, :, self.length:end].copy_(k_new)
|
| 1169 |
+
self.v[:, :, self.length:end].copy_(v_new)
|
| 1170 |
+
self.length = end
|
| 1171 |
+
|
| 1172 |
+
def view(self):
|
| 1173 |
+
return self.k[:, :, :self.length], self.v[:, :, :self.length]
|
| 1174 |
+
|
| 1175 |
+
|
| 1176 |
class TuneableAttentionMHA(nn.Module):
|
| 1177 |
def __init__(
|
| 1178 |
self,
|
|
|
|
| 1350 |
else:
|
| 1351 |
q = self._proj_qk(q_lin)
|
| 1352 |
k_new = self._proj_qk(k_lin)
|
| 1353 |
+
if kv_cache is None:
|
| 1354 |
k, v = k_new, v_new
|
| 1355 |
+
elif isinstance(kv_cache, KVBuffer):
|
| 1356 |
+
if use_cache:
|
| 1357 |
+
kv_cache.append(k_new, v_new)
|
| 1358 |
+
k, v = kv_cache.view()
|
| 1359 |
+
else:
|
| 1360 |
+
k, v = k_new, v_new
|
| 1361 |
else:
|
| 1362 |
k_cached, v_cached = kv_cache
|
| 1363 |
if use_cache:
|
|
|
|
| 1391 |
z = att.softmax(-1) @ v
|
| 1392 |
z = z.transpose(1, 2).reshape(x.size(0), x.size(1), -1)
|
| 1393 |
out = self.drop(self.proj(z))
|
| 1394 |
+
if not use_cache:
|
| 1395 |
+
return out
|
| 1396 |
+
new_kv = kv_cache if isinstance(kv_cache, KVBuffer) else (k, v)
|
| 1397 |
+
return out, new_kv
|
| 1398 |
|
| 1399 |
|
| 1400 |
class Block(nn.Module):
|
|
|
|
| 1445 |
sublinear_stride: int = DEFAULT_SUBLINEAR_STRIDE,
|
| 1446 |
sublinear_max_anchors: int = DEFAULT_SUBLINEAR_MAX_ANCHORS,
|
| 1447 |
sublinear_chunk: int = DEFAULT_SUBLINEAR_CHUNK,
|
| 1448 |
+
anchor_memory: bool = DEFAULT_ANCHOR_MEMORY,
|
| 1449 |
+
anchor_stride: int = DEFAULT_ANCHOR_STRIDE,
|
| 1450 |
+
anchor_max: int = DEFAULT_ANCHOR_MAX,
|
| 1451 |
+
anchor_position: int = DEFAULT_ANCHOR_POSITION,
|
| 1452 |
):
|
| 1453 |
super().__init__()
|
| 1454 |
d, l, h, r = cfg["d"], cfg["layers"], cfg["heads"], cfg["rank"]
|
|
|
|
| 1474 |
self.sublinear_stride = sublinear_stride
|
| 1475 |
self.sublinear_max_anchors = sublinear_max_anchors
|
| 1476 |
self.sublinear_chunk = sublinear_chunk
|
| 1477 |
+
self.anchor_memory_enabled = bool(anchor_memory)
|
| 1478 |
+
self.anchor_stride = int(anchor_stride)
|
| 1479 |
+
self.anchor_max = int(anchor_max)
|
| 1480 |
+
n_layers = int(cfg["layers"])
|
| 1481 |
+
if int(anchor_position) < 0:
|
| 1482 |
+
self.anchor_position = n_layers // 2
|
| 1483 |
+
else:
|
| 1484 |
+
self.anchor_position = min(int(anchor_position), n_layers - 1)
|
| 1485 |
+
if self.anchor_memory_enabled:
|
| 1486 |
+
am_cfg = AnchorMemoryConfig(
|
| 1487 |
+
d_model=int(cfg["d"]),
|
| 1488 |
+
heads=int(cfg["heads"]),
|
| 1489 |
+
anchor_stride=self.anchor_stride,
|
| 1490 |
+
max_anchors=self.anchor_max,
|
| 1491 |
+
)
|
| 1492 |
+
self.anchor = AnchorMemoryLayer(am_cfg)
|
| 1493 |
+
else:
|
| 1494 |
+
self.anchor = None
|
| 1495 |
|
| 1496 |
def forward(self, ids, mask, kv_caches=None, use_cache=False, total_seq_len=None):
|
| 1497 |
x = self.emb(ids)
|
| 1498 |
if not use_cache:
|
| 1499 |
+
for i, blk in enumerate(self.blocks):
|
| 1500 |
if self.grad_checkpoint and self.training:
|
| 1501 |
x = torch_checkpoint.checkpoint(blk, x, mask, use_reentrant=False)
|
| 1502 |
else:
|
| 1503 |
x = blk(x, mask)
|
| 1504 |
+
if self.anchor is not None and i == self.anchor_position:
|
| 1505 |
+
if self.grad_checkpoint and self.training:
|
| 1506 |
+
x, _ = torch_checkpoint.checkpoint(self.anchor, x, use_reentrant=False)
|
| 1507 |
+
else:
|
| 1508 |
+
x, _ = self.anchor(x)
|
| 1509 |
return self.ln(x)
|
| 1510 |
new_kvs = []
|
| 1511 |
for i, blk in enumerate(self.blocks):
|
| 1512 |
kv = kv_caches[i] if kv_caches else None
|
| 1513 |
x, kv_out = blk(x, mask, kv, use_cache=True, total_seq_len=total_seq_len)
|
| 1514 |
new_kvs.append(kv_out)
|
| 1515 |
+
if self.anchor is not None and i == self.anchor_position:
|
| 1516 |
+
x, _ = self.anchor(x)
|
| 1517 |
return self.ln(x), new_kvs
|
| 1518 |
|
| 1519 |
|
|
|
|
| 1531 |
return self.proj(h)
|
| 1532 |
|
| 1533 |
|
| 1534 |
+
class NATHead(nn.Module):
|
| 1535 |
+
def __init__(self, d):
|
| 1536 |
+
super().__init__()
|
| 1537 |
+
self.proj = nn.Linear(d, VOCAB)
|
| 1538 |
+
|
| 1539 |
+
def forward(self, h):
|
| 1540 |
+
return self.proj(h)
|
| 1541 |
+
|
| 1542 |
+
|
| 1543 |
class SATHead(nn.Module):
|
| 1544 |
def __init__(self, d, mode="var"):
|
| 1545 |
super().__init__()
|
|
|
|
| 1662 |
return value.detach().clone()
|
| 1663 |
return copy.deepcopy(value)
|
| 1664 |
|
| 1665 |
+
def _optimizer_param_name_lookup(core, ar_h, sat_h, nat_h=None) -> dict[int, str]:
|
| 1666 |
out = {}
|
| 1667 |
+
for prefix, module in (("core", core), ("ar", ar_h), ("sat", sat_h), ("nat", nat_h)):
|
| 1668 |
+
if module is None:
|
| 1669 |
+
continue
|
| 1670 |
for name, param in module.named_parameters():
|
| 1671 |
out.setdefault(id(param), f"{prefix}.{name}")
|
| 1672 |
return out
|
| 1673 |
|
| 1674 |
+
def _optimizer_group_param_names(opt, core, ar_h, sat_h, nat_h=None) -> List[List[str]]:
|
| 1675 |
+
lookup = _optimizer_param_name_lookup(core, ar_h, sat_h, nat_h)
|
| 1676 |
return [
|
| 1677 |
[lookup.get(id(param), f"<unknown:{id(param)}>") for param in group["params"]]
|
| 1678 |
for group in opt.param_groups
|
|
|
|
| 1703 |
out[key] = copy.deepcopy(vals[0])
|
| 1704 |
return out
|
| 1705 |
|
| 1706 |
+
def _fuse_legacy_qkv_optimizer_state(opt_state: dict, opt, core, ar_h, sat_h, nat_h=None) -> Optional[dict]:
|
| 1707 |
"""Remap pre-QKV-fusion AdamW state to the current fused parameter layout."""
|
| 1708 |
if not isinstance(opt_state, dict) or "state" not in opt_state or "param_groups" not in opt_state:
|
| 1709 |
return None
|
| 1710 |
current_sd = opt.state_dict()
|
| 1711 |
+
current_names = _optimizer_group_param_names(opt, core, ar_h, sat_h, nat_h)
|
| 1712 |
legacy_names = [
|
| 1713 |
[legacy for name in group_names for legacy in _legacy_names_for_current_param(name)]
|
| 1714 |
for group_names in current_names
|
|
|
|
| 1750 |
|
| 1751 |
return {"state": new_states, "param_groups": new_groups}
|
| 1752 |
|
| 1753 |
+
def save_delta(core, ar_h, sat_h, nat_h, step: int, seen_tok: int, save_dir: pathlib.Path, phase_name: str):
|
| 1754 |
"""Save weight-only delta in background thread. Non-blocking."""
|
| 1755 |
global _delta_thread
|
| 1756 |
# Wait for any previous delta write to finish
|
|
|
|
| 1763 |
"ar": {k: v.detach().cpu() for k, v in _checkpoint_state_dict(ar_h).items()},
|
| 1764 |
"sat": {k: v.detach().cpu() for k, v in _checkpoint_state_dict(sat_h).items()},
|
| 1765 |
}
|
| 1766 |
+
if nat_h is not None:
|
| 1767 |
+
tensors["nat"] = {k: v.detach().cpu() for k, v in _checkpoint_state_dict(nat_h).items()}
|
| 1768 |
meta = {"step": step, "seen_tok": seen_tok, "wall_time": time.time(), "delta": True}
|
| 1769 |
path = save_dir / f"{phase_name}_delta_step{step:08d}.pt"
|
| 1770 |
_delta_thread = threading.Thread(target=_do_delta_save, args=(tensors, path, meta), daemon=True)
|
|
|
|
| 1793 |
except Exception as e:
|
| 1794 |
print(f" [delta-prune] error: {e}")
|
| 1795 |
|
| 1796 |
+
def load_delta(path: pathlib.Path, core, ar_h, sat_h, nat_h=None):
|
| 1797 |
"""Load weight-only delta. Returns (step, seen_tok) or raises."""
|
| 1798 |
# Verify checksum if sidecar exists
|
| 1799 |
sha_path = path.with_suffix(".sha256")
|
|
|
|
| 1809 |
core.load_state_dict(_fuse_qkv_in_state_dict(_strip_orig_mod_prefix(ck["weights"]["core"])))
|
| 1810 |
ar_h.load_state_dict(_strip_orig_mod_prefix(ck["weights"]["ar"]))
|
| 1811 |
sat_h.load_state_dict(_strip_orig_mod_prefix(ck["weights"]["sat"]))
|
| 1812 |
+
if nat_h is not None:
|
| 1813 |
+
nat_sd = ck["weights"].get("nat")
|
| 1814 |
+
if nat_sd is not None:
|
| 1815 |
+
nat_h.load_state_dict(_strip_orig_mod_prefix(nat_sd))
|
| 1816 |
+
else:
|
| 1817 |
+
print("[nat] Delta has no NAT head; keeping fresh NAT initialization")
|
| 1818 |
return ck.get("step", 0), ck.get("seen_tok", 0)
|
| 1819 |
|
| 1820 |
def _flush_delta():
|
|
|
|
| 1824 |
print(" [delta] flushing in-flight write...")
|
| 1825 |
_delta_thread.join(timeout=120)
|
| 1826 |
|
| 1827 |
+
def save_ckpt(path: pathlib.Path, core, ar_h, sat_h, nat_h, opt, scaler, meta):
|
| 1828 |
path.parent.mkdir(exist_ok=True, parents=True)
|
| 1829 |
tmp = path.with_suffix(path.suffix + ".tmp")
|
| 1830 |
state = {
|
|
|
|
| 1837 |
"tie_weights": meta.get("tie_weights", False),
|
| 1838 |
**{k: v for k, v in meta.items() if k not in ("cfg", "tie_weights")}
|
| 1839 |
}
|
| 1840 |
+
if nat_h is not None:
|
| 1841 |
+
state["nat"] = _checkpoint_state_dict(nat_h)
|
| 1842 |
torch.save(state, tmp, _use_new_zipfile_serialization=False)
|
| 1843 |
tmp.replace(path)
|
| 1844 |
(path.parent / "latest.json").write_text(json.dumps({"path": str(path), "step": meta["step"]}))
|
| 1845 |
print(f"\n✓ saved checkpoint {path.name}")
|
| 1846 |
|
| 1847 |
+
def load_ckpt(path, core, ar_h, sat_h, opt, scaler, nat_h=None):
|
| 1848 |
p = _resolve_ckpt(path) or path
|
| 1849 |
ck = _try_load(p, map_location="cpu")
|
| 1850 |
if ck is None: raise FileNotFoundError(f"No valid checkpoint at {p}")
|
| 1851 |
core.load_state_dict(_fuse_qkv_in_state_dict(_strip_orig_mod_prefix(ck["core"])))
|
| 1852 |
ar_h.load_state_dict(_strip_orig_mod_prefix(ck["ar"]))
|
| 1853 |
sat_h.load_state_dict(_strip_orig_mod_prefix(ck["sat"]))
|
| 1854 |
+
if nat_h is not None:
|
| 1855 |
+
if "nat" in ck:
|
| 1856 |
+
nat_h.load_state_dict(_strip_orig_mod_prefix(ck["nat"]))
|
| 1857 |
+
else:
|
| 1858 |
+
print("[nat] Checkpoint has no NAT head; keeping fresh NAT initialization")
|
| 1859 |
try:
|
| 1860 |
opt.load_state_dict(ck["opt"])
|
| 1861 |
except Exception as exc:
|
| 1862 |
+
fused_opt = _fuse_legacy_qkv_optimizer_state(ck.get("opt"), opt, core, ar_h, sat_h, nat_h)
|
| 1863 |
if fused_opt is not None:
|
| 1864 |
try:
|
| 1865 |
opt.load_state_dict(fused_opt)
|
|
|
|
| 1946 |
if train_emb:
|
| 1947 |
for p in core.emb.parameters(): p.requires_grad = True
|
| 1948 |
|
| 1949 |
+
def _optimizer_param_groups(core, ar_h, sat_h, lr_core: float, lr_head: float, nat_h=None):
|
| 1950 |
+
groups = [
|
| 1951 |
{"params": [p for p in core.parameters() if p.requires_grad], "lr": lr_core},
|
| 1952 |
{"params": ar_h.parameters(), "lr": lr_head},
|
| 1953 |
{"params": sat_h.parameters(), "lr": lr_head},
|
| 1954 |
]
|
| 1955 |
+
if nat_h is not None:
|
| 1956 |
+
groups.append({"params": nat_h.parameters(), "lr": lr_head})
|
| 1957 |
+
return groups
|
| 1958 |
|
| 1959 |
+
def make_optimizer(args, core, ar_h, sat_h, lr_core: float, lr_head: float, nat_h=None):
|
| 1960 |
+
groups = _optimizer_param_groups(core, ar_h, sat_h, lr_core, lr_head, nat_h)
|
| 1961 |
opt_name = getattr(args, "optimizer", "adamw")
|
| 1962 |
if opt_name == "adamw":
|
| 1963 |
return torch.optim.AdamW(groups)
|
|
|
|
| 1973 |
return bnb.optim.AdamW8bit(groups)
|
| 1974 |
raise ValueError(f"unknown optimizer: {opt_name}")
|
| 1975 |
|
| 1976 |
+
def _nat_ids_for_training(ids: torch.Tensor, max_tokens: int) -> torch.Tensor:
|
| 1977 |
+
if max_tokens and max_tokens > 0 and ids.size(1) > max_tokens:
|
| 1978 |
+
return ids[:, -max_tokens:]
|
| 1979 |
+
return ids
|
| 1980 |
+
|
| 1981 |
def _train_phase(
|
| 1982 |
args, phase_name: str,
|
| 1983 |
+
core, ar_h, sat_h, nat_h, opt, scaler,
|
| 1984 |
start_step, seen_tok, resume_wall_time,
|
| 1985 |
cfg, source, steps, block_size, batch_size,
|
| 1986 |
chat_cfg: dict,
|
|
|
|
| 1995 |
target_tokens = target_tokens_override
|
| 1996 |
else:
|
| 1997 |
ratio = _target_token_ratio(args)
|
| 1998 |
+
param_count = _count_enabled_params(core, ar_h, sat_h, nat_h)
|
| 1999 |
target_tokens = int(ratio * param_count)
|
| 2000 |
print(f"[{phase_name}] token_param_ratio={ratio:g} param_count={param_count:,} target_tokens={target_tokens:,}")
|
| 2001 |
if steps:
|
|
|
|
| 2016 |
)
|
| 2017 |
ce_tok = nn.CrossEntropyLoss(label_smoothing=0.1)
|
| 2018 |
ce_gate = nn.CrossEntropyLoss()
|
| 2019 |
+
ctc = nn.CTCLoss(blank=BLANK, zero_infinity=True)
|
| 2020 |
pbar = SafeProgress(total=total_tokens_needed, initial=seen_tok, unit="tok")
|
| 2021 |
grow_plan = _parse_grow_plan(args.grow_plan) if args.auto_grow else []
|
| 2022 |
buf: list[int] = []
|
|
|
|
| 2029 |
last_save_mono = time.monotonic() - (now_wall - (resume_wall_time or now_wall))
|
| 2030 |
last_delta_step = start_step
|
| 2031 |
print(f"[{phase_name}] Starting. Goal: {total_tokens_needed:,} tokens. Batch={BATCH}, Block={BLOCK}")
|
| 2032 |
+
print(
|
| 2033 |
+
f"[{phase_name}] AR_ONLY={args.ar_only}, SAT_EVERY={args.sat_every}, "
|
| 2034 |
+
f"NAT_EVERY={args.nat_every}, TIE_WEIGHTS={tie_weights}, STREAMING={streaming}"
|
| 2035 |
+
)
|
| 2036 |
while seen_tok < total_tokens_needed:
|
| 2037 |
try:
|
| 2038 |
while len(buf) < BLOCK:
|
|
|
|
| 2069 |
loss_value += float(loss_sat.detach().item())
|
| 2070 |
scaler.scale(loss_sat).backward()
|
| 2071 |
del h_sat, logits_sat, loss_sat
|
| 2072 |
+
do_nat = (
|
| 2073 |
+
nat_h is not None
|
| 2074 |
+
and (not args.ar_only)
|
| 2075 |
+
and args.nat_every > 0
|
| 2076 |
+
and (args.nat_every <= 1 or ((step + 1) % args.nat_every == 0))
|
| 2077 |
+
)
|
| 2078 |
+
if do_nat:
|
| 2079 |
+
nat_ids = _nat_ids_for_training(ids, args.nat_max_tokens)
|
| 2080 |
+
nat_expand = max(1, int(args.nat_expand))
|
| 2081 |
+
nat_in = torch.repeat_interleave(nat_ids, nat_expand, dim=1)
|
| 2082 |
+
input_lengths = torch.full((nat_ids.size(0),), nat_in.size(1), device=DEV, dtype=torch.long)
|
| 2083 |
+
target_lengths = torch.full((nat_ids.size(0),), nat_ids.size(1), device=DEV, dtype=torch.long)
|
| 2084 |
+
with amp(args.amp):
|
| 2085 |
+
h_nat = core(nat_in, None)
|
| 2086 |
+
log_nat = nat_h(h_nat).float().log_softmax(-1).transpose(0, 1)
|
| 2087 |
+
loss_nat = ctc(log_nat, nat_ids, input_lengths, target_lengths)
|
| 2088 |
+
loss_nat = float(args.nat_loss_weight) * loss_nat
|
| 2089 |
+
loss_value += float(loss_nat.detach().item())
|
| 2090 |
+
scaler.scale(loss_nat).backward()
|
| 2091 |
+
del nat_ids, nat_in, input_lengths, target_lengths, h_nat, log_nat, loss_nat
|
| 2092 |
scaler.unscale_(opt)
|
| 2093 |
+
nn.utils.clip_grad_norm_([p for group in opt.param_groups for p in group["params"]], 1.0)
|
| 2094 |
scaler.step(opt)
|
| 2095 |
scaler.update()
|
| 2096 |
opt.zero_grad(set_to_none=True)
|
|
|
|
| 2143 |
ck_name = f"{phase_name}_step{step:08d}.pt"
|
| 2144 |
_flush_delta() # wait for any in-flight delta before full save
|
| 2145 |
_prune_checkpoints(pathlib.Path(args.save_dir), phase_name, max_ckpts)
|
| 2146 |
+
save_ckpt(pathlib.Path(args.save_dir) / ck_name, core, ar_h, sat_h, nat_h, opt, scaler,
|
| 2147 |
meta={"cfg": cfg, "step": step, "seen_tok": seen_tok, "wall_time": time.time(), "tie_weights": tie_weights})
|
| 2148 |
last_save_mono = now_mono
|
| 2149 |
# Prune old deltas after a full save (they're superseded)
|
|
|
|
| 2152 |
# ── Delta checkpoint (step-based, weight-only, async) ──
|
| 2153 |
if args.delta_every_steps > 0 and (step - last_delta_step) >= args.delta_every_steps:
|
| 2154 |
_prune_deltas(pathlib.Path(args.save_dir), phase_name, args.delta_max_keep)
|
| 2155 |
+
save_delta(core, ar_h, sat_h, nat_h, step, seen_tok, pathlib.Path(args.save_dir), phase_name)
|
| 2156 |
last_delta_step = step
|
| 2157 |
if args.auto_grow:
|
| 2158 |
steps_since_last_grow += 1
|
|
|
|
| 2169 |
pbar.close()
|
| 2170 |
_flush_delta() # ensure any in-flight delta completes before final save
|
| 2171 |
if phase_name != "sft":
|
| 2172 |
+
save_ckpt(pathlib.Path(args.save_dir) / f"{phase_name}_final.pt", core, ar_h, sat_h, nat_h, opt, scaler,
|
| 2173 |
meta={"cfg": cfg, "step": step, "seen_tok": seen_tok, "wall_time": time.time(), "tie_weights": tie_weights})
|
| 2174 |
else:
|
| 2175 |
print("[sft] Skipping duplicate sft_final.pt; final.pt will contain the SFT result.")
|
|
|
|
| 2206 |
sublinear_stride=args.sublinear_stride,
|
| 2207 |
sublinear_max_anchors=args.sublinear_max_anchors,
|
| 2208 |
sublinear_chunk=args.sublinear_chunk,
|
| 2209 |
+
anchor_memory=getattr(args, "anchor_memory", DEFAULT_ANCHOR_MEMORY),
|
| 2210 |
+
anchor_stride=getattr(args, "anchor_stride", DEFAULT_ANCHOR_STRIDE),
|
| 2211 |
+
anchor_max=getattr(args, "anchor_max", DEFAULT_ANCHOR_MAX),
|
| 2212 |
+
anchor_position=getattr(args, "anchor_position", DEFAULT_ANCHOR_POSITION),
|
| 2213 |
).to(DEV)
|
| 2214 |
ar_h = ARHead(cfg["d"], tie_weights=tie_weights, embedding_weight=core.emb.weight if tie_weights else None).to(DEV)
|
| 2215 |
sat_h = SATHead(cfg["d"], mode="var").to(DEV)
|
| 2216 |
+
nat_h = NATHead(cfg["d"]).to(DEV)
|
| 2217 |
+
total_params = _count_enabled_params(core, ar_h, sat_h, nat_h)
|
| 2218 |
print(f"Total parameters: {total_params:,}")
|
| 2219 |
if tie_weights:
|
| 2220 |
print(f"{Colors.WARN}[weight-tying] Embedding and LM head share weights{Colors.RESET}")
|
|
|
|
| 2225 |
loaded = _safe_load_any(src, core, key="core")
|
| 2226 |
_safe_load_any(src, ar_h, key="ar")
|
| 2227 |
_safe_load_any(src, sat_h, key="sat")
|
| 2228 |
+
nat_loaded = _safe_load_any(src, nat_h, key="nat")
|
| 2229 |
+
if not nat_loaded:
|
| 2230 |
+
print("[nat] Warm-start source has no NAT head; NAT head initialized fresh")
|
| 2231 |
if loaded: print(f"Warm-start loaded from {src}")
|
| 2232 |
_phase_freeze(core, freeze_core=args.freeze_core, unfreeze_ln=args.unfreeze_ln, train_emb=args.train_emb)
|
| 2233 |
+
opt = make_optimizer(args, core, ar_h, sat_h, args.lr_core, args.lr_head, nat_h)
|
| 2234 |
scaler = GradScaler(enabled=(args.amp and _needs_grad_scaler()))
|
| 2235 |
start_step, seen_tok, last_wall = 0, 0, None
|
| 2236 |
if args.resume_delta and not args.fresh:
|
| 2237 |
+
delta_step, delta_tok = load_delta(pathlib.Path(args.resume_delta), core, ar_h, sat_h, nat_h)
|
| 2238 |
start_step, seen_tok, last_wall = delta_step, delta_tok, None
|
| 2239 |
print(f"Resumed from DELTA at step {start_step} (optimizer state reset — momentum rebuilds in ~100 steps)")
|
| 2240 |
elif args.resume and not args.fresh:
|
| 2241 |
+
start_step, seen_tok, last_wall = load_ckpt(pathlib.Path(args.resume), core, ar_h, sat_h, opt, scaler, nat_h)
|
| 2242 |
print(f"Resumed from step {start_step}")
|
| 2243 |
# torch.compile AFTER loading checkpoint (key names differ)
|
| 2244 |
if args.compile:
|
|
|
|
| 2246 |
core = torch.compile(core, mode="reduce-overhead")
|
| 2247 |
ar_h = torch.compile(ar_h, mode="reduce-overhead")
|
| 2248 |
sat_h = torch.compile(sat_h, mode="reduce-overhead")
|
| 2249 |
+
nat_h = torch.compile(nat_h, mode="reduce-overhead")
|
| 2250 |
print("[torch.compile] Done.")
|
| 2251 |
step, seen_tok, last_wall = _train_phase(
|
| 2252 |
+
args, "pretrain", core, ar_h, sat_h, nat_h, opt, scaler,
|
| 2253 |
start_step, seen_tok, last_wall, cfg,
|
| 2254 |
args.source, args.steps,
|
| 2255 |
args.block or DEFAULT_BLOCK,
|
|
|
|
| 2277 |
sat_h,
|
| 2278 |
args.after_sft_lr_core or args.lr_core,
|
| 2279 |
args.after_sft_lr_head or args.lr_head,
|
| 2280 |
+
nat_h,
|
| 2281 |
)
|
| 2282 |
step, seen_tok, last_wall = _train_phase(
|
| 2283 |
+
args, "sft", core, ar_h, sat_h, nat_h, opt, scaler,
|
| 2284 |
step, seen_tok, last_wall, cfg,
|
| 2285 |
args.after_sft_source, args.after_sft_steps,
|
| 2286 |
args.after_sft_block or DEFAULT_AFTER_SFT_BLOCK,
|
|
|
|
| 2296 |
tie_weights=tie_weights,
|
| 2297 |
streaming=True
|
| 2298 |
)
|
| 2299 |
+
save_ckpt(pathlib.Path(args.save_dir) / "final.pt", core, ar_h, sat_h, nat_h, opt, scaler,
|
| 2300 |
meta={"cfg": cfg, "step": step, "seen_tok": seen_tok, "wall_time": time.time(), "tie_weights": tie_weights})
|
| 2301 |
print("🎉 All Training Complete")
|
| 2302 |
|
|
|
|
| 2336 |
if args.frequency_penalty is None: args.frequency_penalty = 0.3
|
| 2337 |
if args.penalty_last_n is None: args.penalty_last_n = 128
|
| 2338 |
if args.var is None: args.var = False
|
| 2339 |
+
elif args.mode == "sat":
|
| 2340 |
if args.temperature is None: args.temperature = 0.5
|
| 2341 |
if args.top_k is None: args.top_k = 30
|
| 2342 |
if args.repetition_penalty is None: args.repetition_penalty = 2.0
|
|
|
|
| 2344 |
if args.frequency_penalty is None: args.frequency_penalty = 1.0
|
| 2345 |
if args.penalty_last_n is None: args.penalty_last_n = 200
|
| 2346 |
if args.var is None: args.var = True
|
| 2347 |
+
else:
|
| 2348 |
+
if args.temperature is None: args.temperature = 1.0
|
| 2349 |
+
if args.top_k is None: args.top_k = 0
|
| 2350 |
+
if args.repetition_penalty is None: args.repetition_penalty = 1.0
|
| 2351 |
+
if args.presence_penalty is None: args.presence_penalty = 0.0
|
| 2352 |
+
if args.frequency_penalty is None: args.frequency_penalty = 0.0
|
| 2353 |
+
if args.penalty_last_n is None: args.penalty_last_n = 0
|
| 2354 |
+
if args.var is None: args.var = False
|
| 2355 |
path = _resolve_ckpt(pathlib.Path(args.ckpt)) or pathlib.Path(args.ckpt)
|
| 2356 |
sd = torch.load(path, map_location="cpu")
|
| 2357 |
# Restore tokenizer from checkpoint if available
|
|
|
|
| 2372 |
print("[infer] Delta checkpoint detected, using large preset cfg")
|
| 2373 |
cfg = PRESETS["large"].copy()
|
| 2374 |
tie_weights = False
|
| 2375 |
+
# Remap: delta stores under sd["weights"]["core"/"ar"/"sat"/"nat"]
|
| 2376 |
sd["core"] = sd["weights"]["core"]
|
| 2377 |
sd["ar"] = sd["weights"]["ar"]
|
| 2378 |
sd["sat"] = sd["weights"]["sat"]
|
| 2379 |
+
if "nat" in sd["weights"]:
|
| 2380 |
+
sd["nat"] = sd["weights"]["nat"]
|
| 2381 |
else:
|
| 2382 |
cfg = sd["cfg"]
|
| 2383 |
tie_weights = sd.get("tie_weights", False)
|
|
|
|
| 2406 |
sublinear_stride=args.sublinear_stride,
|
| 2407 |
sublinear_max_anchors=args.sublinear_max_anchors,
|
| 2408 |
sublinear_chunk=args.sublinear_chunk,
|
| 2409 |
+
anchor_memory=getattr(args, "anchor_memory", DEFAULT_ANCHOR_MEMORY),
|
| 2410 |
+
anchor_stride=getattr(args, "anchor_stride", DEFAULT_ANCHOR_STRIDE),
|
| 2411 |
+
anchor_max=getattr(args, "anchor_max", DEFAULT_ANCHOR_MAX),
|
| 2412 |
+
anchor_position=getattr(args, "anchor_position", DEFAULT_ANCHOR_POSITION),
|
| 2413 |
).to(DEV)
|
| 2414 |
ar_h = ARHead(cfg["d"], tie_weights=tie_weights, embedding_weight=core.emb.weight if tie_weights else None).to(DEV)
|
| 2415 |
sat_h = SATHead(cfg["d"]).to(DEV)
|
| 2416 |
+
nat_h = NATHead(cfg["d"]).to(DEV) if ("nat" in sd or args.mode == "nat") else None
|
| 2417 |
core.load_state_dict(sd["core"])
|
| 2418 |
ar_h.load_state_dict(sd["ar"])
|
| 2419 |
sat_h.load_state_dict(sd["sat"])
|
| 2420 |
+
if nat_h is not None:
|
| 2421 |
+
if "nat" not in sd:
|
| 2422 |
+
raise ValueError("NAT inference requested, but this checkpoint has no NAT head")
|
| 2423 |
+
nat_h.load_state_dict(sd["nat"])
|
| 2424 |
core.eval()
|
| 2425 |
ar_h.eval()
|
| 2426 |
sat_h.eval()
|
| 2427 |
+
if nat_h is not None:
|
| 2428 |
+
nat_h.eval()
|
| 2429 |
+
total_params = _count_enabled_params(core, ar_h, sat_h, nat_h)
|
| 2430 |
if total_params >= 1_000_000_000:
|
| 2431 |
param_str = f"{total_params / 1_000_000_000:.2f}B"
|
| 2432 |
elif total_params >= 1_000_000:
|
|
|
|
| 2458 |
nxt = _sample(logits, args.temperature, args.top_k, args.top_p, args.min_p, args.greedy)
|
| 2459 |
ids = torch.cat([ids, nxt], 1)
|
| 2460 |
h, kvs = core(ids[:, -1:], None, kv_caches=kvs, use_cache=True, total_seq_len=ids.size(1))
|
| 2461 |
+
elif args.mode == "nat":
|
| 2462 |
+
blank_slots = [BLANK] * (args.max_new * max(1, int(args.nat_expand)))
|
| 2463 |
+
ids = torch.tensor([prompt_tokens + blank_slots], device=DEV)
|
| 2464 |
+
for _ in range(max(1, int(args.nat_passes))):
|
| 2465 |
+
h = core(ids, None)
|
| 2466 |
+
logits = nat_h(h)
|
| 2467 |
+
logits[..., BLANK] = -1e9
|
| 2468 |
+
ids = logits.argmax(-1)
|
| 2469 |
+
decoded = []
|
| 2470 |
+
prev = None
|
| 2471 |
+
for token in ids[0].tolist():
|
| 2472 |
+
if token == BLANK:
|
| 2473 |
+
prev = token
|
| 2474 |
+
continue
|
| 2475 |
+
if token != prev:
|
| 2476 |
+
decoded.append(token)
|
| 2477 |
+
prev = token
|
| 2478 |
+
ids = torch.tensor([decoded[: prompt_len + args.max_new]], device=DEV)
|
| 2479 |
+
prompt_len = min(prompt_len, ids.size(1))
|
| 2480 |
else:
|
| 2481 |
cached_len = ids.size(1)
|
| 2482 |
h, kvs = core(ids, sat_mask(ids.size(1)), use_cache=True, total_seq_len=cached_len)
|
|
|
|
| 2556 |
help="For --attn_backend sublinear, cap landmark candidates per query chunk.")
|
| 2557 |
tr.add_argument("--sublinear_chunk", type=int, default=DEFAULT_SUBLINEAR_CHUNK,
|
| 2558 |
help="For --attn_backend sublinear, query chunk size controlling peak gather memory.")
|
| 2559 |
+
tr.add_argument("--anchor_memory", action="store_true",
|
| 2560 |
+
help="Enable anchor-memory long-context augmentation (one AnchorMemoryLayer at mid-stack).")
|
| 2561 |
+
tr.add_argument("--anchor_stride", type=int, default=DEFAULT_ANCHOR_STRIDE,
|
| 2562 |
+
help="Token span compressed into one anchor (default 256).")
|
| 2563 |
+
tr.add_argument("--anchor_max", type=int, default=DEFAULT_ANCHOR_MAX,
|
| 2564 |
+
help="Max anchors retained in the rolling memory bank.")
|
| 2565 |
+
tr.add_argument("--anchor_position", type=int, default=DEFAULT_ANCHOR_POSITION,
|
| 2566 |
+
help="Block index after which to insert anchor memory (-1 = stack middle).")
|
| 2567 |
+
tr.add_argument("--kv_buffer", action="store_true",
|
| 2568 |
+
help="Use preallocated KV buffer instead of torch.cat-based cache growth.")
|
| 2569 |
tr.add_argument("--optimizer", choices=["adamw", "adamw8bit", "paged_adamw8bit"], default="adamw",
|
| 2570 |
help="Optimizer backend. 8-bit options reduce VRAM on 24GB production runs.")
|
| 2571 |
tr.add_argument("--save_every_sec", type=int, default=DEFAULT_SAVE_SEC)
|
|
|
|
| 2583 |
tr.add_argument("--ar_only", action="store_true")
|
| 2584 |
tr.add_argument("--sat_every", type=int, default=1,
|
| 2585 |
help="Train SAT every N steps. Default 1 keeps AR+SAT every step.")
|
| 2586 |
+
tr.add_argument("--nat_every", type=int, default=1,
|
| 2587 |
+
help="Train NAT every N steps with a CTC objective. Default 1 keeps AR+SAT+NAT every step.")
|
| 2588 |
+
tr.add_argument("--nat_loss_weight", type=float, default=1.0)
|
| 2589 |
+
tr.add_argument("--nat_expand", type=int, default=2,
|
| 2590 |
+
help="Repeat tokens this many times for the NAT CTC input length.")
|
| 2591 |
+
tr.add_argument("--nat_max_tokens", type=int, default=0,
|
| 2592 |
+
help="Optional cap for NAT target tokens per batch; 0 uses the whole block.")
|
| 2593 |
tr.add_argument("--freeze_core", action="store_true")
|
| 2594 |
tr.add_argument("--unfreeze_ln", action="store_true")
|
| 2595 |
tr.add_argument("--train_emb", action="store_true")
|
|
|
|
| 2615 |
tr.add_argument("--after_sft_lr_core", type=float, default=0.0)
|
| 2616 |
tr.add_argument("--after_sft_lr_head", type=float, default=0.0)
|
| 2617 |
inf = sub.add_parser("infer")
|
| 2618 |
+
inf.add_argument("--mode", choices=["ar", "sat", "nat"], required=True)
|
| 2619 |
inf.add_argument("--ckpt", required=True)
|
| 2620 |
inf.add_argument("--prompt", required=True)
|
| 2621 |
inf.add_argument("--max_new", type=int, default=120)
|
|
|
|
| 2637 |
inf.add_argument("--sublinear_stride", type=int, default=DEFAULT_SUBLINEAR_STRIDE)
|
| 2638 |
inf.add_argument("--sublinear_max_anchors", type=int, default=DEFAULT_SUBLINEAR_MAX_ANCHORS)
|
| 2639 |
inf.add_argument("--sublinear_chunk", type=int, default=DEFAULT_SUBLINEAR_CHUNK)
|
| 2640 |
+
inf.add_argument("--nat_expand", type=int, default=2)
|
| 2641 |
+
inf.add_argument("--nat_passes", type=int, default=1)
|
| 2642 |
st = sub.add_parser("status", help="Read-only training status")
|
| 2643 |
st.add_argument("--json", dest="json_output", action="store_true")
|
| 2644 |
st.add_argument("--log", type=str, default=str(STATUS_DEFAULT_LOG))
|
nB300_agillm4.py.bak-20260520-140542
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
nB300_agillm4.py.bak-fix-20260520-140710
ADDED
|
The diff for this file is too large to render.
See raw diff
|
|
|
remote_verify_m_fold_after_qkv_agillm4.json
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"backend": "manual",
|
| 4 |
+
"d": 32,
|
| 5 |
+
"dk": 16,
|
| 6 |
+
"expected_k_width": 16,
|
| 7 |
+
"heads": 2,
|
| 8 |
+
"ok": true,
|
| 9 |
+
"preset": "pico_1x",
|
| 10 |
+
"rank": 16,
|
| 11 |
+
"rows": {
|
| 12 |
+
"cache_k_width": 16.0,
|
| 13 |
+
"cache_v_width": 16.0,
|
| 14 |
+
"cached_append_forward": 0.0,
|
| 15 |
+
"causal_alibi_forward": 0.0,
|
| 16 |
+
"causal_alibi_loss": 0.0,
|
| 17 |
+
"causal_alibi_param_grad": 0.0,
|
| 18 |
+
"causal_alibi_x_grad": 0.0,
|
| 19 |
+
"none_forward": 0.0,
|
| 20 |
+
"none_loss": 0.0,
|
| 21 |
+
"none_param_grad": 0.0,
|
| 22 |
+
"none_x_grad": 0.0,
|
| 23 |
+
"sat_alibi_forward": 0.0,
|
| 24 |
+
"sat_alibi_loss": 0.0,
|
| 25 |
+
"sat_alibi_param_grad": 0.0,
|
| 26 |
+
"sat_alibi_x_grad": 0.0
|
| 27 |
+
},
|
| 28 |
+
"tol": 0.0002
|
| 29 |
+
},
|
| 30 |
+
{
|
| 31 |
+
"backend": "sdpa",
|
| 32 |
+
"d": 32,
|
| 33 |
+
"dk": 16,
|
| 34 |
+
"expected_k_width": 16,
|
| 35 |
+
"heads": 2,
|
| 36 |
+
"ok": true,
|
| 37 |
+
"preset": "pico_1x",
|
| 38 |
+
"rank": 16,
|
| 39 |
+
"rows": {
|
| 40 |
+
"cache_k_width": 16.0,
|
| 41 |
+
"cache_v_width": 16.0,
|
| 42 |
+
"cached_append_forward": 0.0,
|
| 43 |
+
"causal_alibi_forward": 5.386769771575928e-06,
|
| 44 |
+
"causal_alibi_loss": 3.725290298461914e-09,
|
| 45 |
+
"causal_alibi_param_grad": 5.189329385757446e-06,
|
| 46 |
+
"causal_alibi_x_grad": 8.445931598544121e-07,
|
| 47 |
+
"none_forward": 1.4901161193847656e-08,
|
| 48 |
+
"none_loss": 9.313225746154785e-10,
|
| 49 |
+
"none_param_grad": 2.0703300833702087e-06,
|
| 50 |
+
"none_x_grad": 1.7951242625713348e-07,
|
| 51 |
+
"sat_alibi_forward": 0.0,
|
| 52 |
+
"sat_alibi_loss": 0.0,
|
| 53 |
+
"sat_alibi_param_grad": 8.198898285627365e-06,
|
| 54 |
+
"sat_alibi_x_grad": 9.000941645354033e-07
|
| 55 |
+
},
|
| 56 |
+
"tol": 0.0002
|
| 57 |
+
},
|
| 58 |
+
{
|
| 59 |
+
"backend": "manual",
|
| 60 |
+
"d": 128,
|
| 61 |
+
"dk": 16,
|
| 62 |
+
"expected_k_width": 16,
|
| 63 |
+
"heads": 8,
|
| 64 |
+
"ok": true,
|
| 65 |
+
"preset": "micro_3x",
|
| 66 |
+
"rank": 48,
|
| 67 |
+
"rows": {
|
| 68 |
+
"cache_k_width": 16.0,
|
| 69 |
+
"cache_v_width": 16.0,
|
| 70 |
+
"cached_append_forward": 0.0,
|
| 71 |
+
"causal_alibi_forward": 0.00011304020881652832,
|
| 72 |
+
"causal_alibi_loss": 1.043081283569336e-07,
|
| 73 |
+
"causal_alibi_param_grad": 1.3775425031781197e-06,
|
| 74 |
+
"causal_alibi_x_grad": 1.5925616025924683e-07,
|
| 75 |
+
"metric_cache_cleared_on_train": 0.0,
|
| 76 |
+
"metric_cache_reused": 0.0,
|
| 77 |
+
"none_forward": 8.377432823181152e-05,
|
| 78 |
+
"none_loss": 1.30385160446167e-08,
|
| 79 |
+
"none_param_grad": 5.532929208129644e-07,
|
| 80 |
+
"none_x_grad": 3.541026671882719e-08,
|
| 81 |
+
"sat_alibi_forward": 0.00014029256999492645,
|
| 82 |
+
"sat_alibi_loss": 9.12696123123169e-08,
|
| 83 |
+
"sat_alibi_param_grad": 9.688083082437515e-07,
|
| 84 |
+
"sat_alibi_x_grad": 1.5372643247246742e-07
|
| 85 |
+
},
|
| 86 |
+
"tol": 0.0002
|
| 87 |
+
},
|
| 88 |
+
{
|
| 89 |
+
"backend": "sdpa",
|
| 90 |
+
"d": 128,
|
| 91 |
+
"dk": 16,
|
| 92 |
+
"expected_k_width": 16,
|
| 93 |
+
"heads": 8,
|
| 94 |
+
"ok": true,
|
| 95 |
+
"preset": "micro_3x",
|
| 96 |
+
"rank": 48,
|
| 97 |
+
"rows": {
|
| 98 |
+
"cache_k_width": 16.0,
|
| 99 |
+
"cache_v_width": 16.0,
|
| 100 |
+
"cached_append_forward": 0.0,
|
| 101 |
+
"causal_alibi_forward": 0.00011304020881652832,
|
| 102 |
+
"causal_alibi_loss": 1.0244548320770264e-07,
|
| 103 |
+
"causal_alibi_param_grad": 1.3635726645588875e-06,
|
| 104 |
+
"causal_alibi_x_grad": 1.829757820814848e-07,
|
| 105 |
+
"metric_cache_cleared_on_train": 0.0,
|
| 106 |
+
"metric_cache_reused": 0.0,
|
| 107 |
+
"none_forward": 8.377432823181152e-05,
|
| 108 |
+
"none_loss": 5.587935447692871e-09,
|
| 109 |
+
"none_param_grad": 5.358306225389242e-07,
|
| 110 |
+
"none_x_grad": 3.109380486421287e-08,
|
| 111 |
+
"sat_alibi_forward": 0.00014029256999492645,
|
| 112 |
+
"sat_alibi_loss": 9.12696123123169e-08,
|
| 113 |
+
"sat_alibi_param_grad": 1.071603037416935e-06,
|
| 114 |
+
"sat_alibi_x_grad": 1.1662632459774613e-07
|
| 115 |
+
},
|
| 116 |
+
"tol": 0.0002
|
| 117 |
+
}
|
| 118 |
+
]
|
remote_verify_m_fold_agillm4.json
ADDED
|
@@ -0,0 +1,118 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"backend": "manual",
|
| 4 |
+
"d": 32,
|
| 5 |
+
"dk": 16,
|
| 6 |
+
"expected_k_width": 16,
|
| 7 |
+
"heads": 2,
|
| 8 |
+
"ok": true,
|
| 9 |
+
"preset": "pico_1x",
|
| 10 |
+
"rank": 16,
|
| 11 |
+
"rows": {
|
| 12 |
+
"cache_k_width": 16.0,
|
| 13 |
+
"cache_v_width": 16.0,
|
| 14 |
+
"cached_append_forward": 0.0,
|
| 15 |
+
"causal_alibi_forward": 0.0,
|
| 16 |
+
"causal_alibi_loss": 0.0,
|
| 17 |
+
"causal_alibi_param_grad": 0.0,
|
| 18 |
+
"causal_alibi_x_grad": 0.0,
|
| 19 |
+
"none_forward": 0.0,
|
| 20 |
+
"none_loss": 0.0,
|
| 21 |
+
"none_param_grad": 0.0,
|
| 22 |
+
"none_x_grad": 0.0,
|
| 23 |
+
"sat_alibi_forward": 0.0,
|
| 24 |
+
"sat_alibi_loss": 0.0,
|
| 25 |
+
"sat_alibi_param_grad": 0.0,
|
| 26 |
+
"sat_alibi_x_grad": 0.0
|
| 27 |
+
},
|
| 28 |
+
"tol": 0.0002
|
| 29 |
+
},
|
| 30 |
+
{
|
| 31 |
+
"backend": "sdpa",
|
| 32 |
+
"d": 32,
|
| 33 |
+
"dk": 16,
|
| 34 |
+
"expected_k_width": 16,
|
| 35 |
+
"heads": 2,
|
| 36 |
+
"ok": true,
|
| 37 |
+
"preset": "pico_1x",
|
| 38 |
+
"rank": 16,
|
| 39 |
+
"rows": {
|
| 40 |
+
"cache_k_width": 16.0,
|
| 41 |
+
"cache_v_width": 16.0,
|
| 42 |
+
"cached_append_forward": 0.0,
|
| 43 |
+
"causal_alibi_forward": 5.386769771575928e-06,
|
| 44 |
+
"causal_alibi_loss": 3.725290298461914e-09,
|
| 45 |
+
"causal_alibi_param_grad": 5.189329385757446e-06,
|
| 46 |
+
"causal_alibi_x_grad": 8.444767445325851e-07,
|
| 47 |
+
"none_forward": 1.4901161193847656e-08,
|
| 48 |
+
"none_loss": 9.313225746154785e-10,
|
| 49 |
+
"none_param_grad": 2.0703300833702087e-06,
|
| 50 |
+
"none_x_grad": 1.7954153008759022e-07,
|
| 51 |
+
"sat_alibi_forward": 0.0,
|
| 52 |
+
"sat_alibi_loss": 0.0,
|
| 53 |
+
"sat_alibi_param_grad": 8.198898285627365e-06,
|
| 54 |
+
"sat_alibi_x_grad": 9.0012326836586e-07
|
| 55 |
+
},
|
| 56 |
+
"tol": 0.0002
|
| 57 |
+
},
|
| 58 |
+
{
|
| 59 |
+
"backend": "manual",
|
| 60 |
+
"d": 128,
|
| 61 |
+
"dk": 16,
|
| 62 |
+
"expected_k_width": 16,
|
| 63 |
+
"heads": 8,
|
| 64 |
+
"ok": true,
|
| 65 |
+
"preset": "micro_3x",
|
| 66 |
+
"rank": 48,
|
| 67 |
+
"rows": {
|
| 68 |
+
"cache_k_width": 16.0,
|
| 69 |
+
"cache_v_width": 16.0,
|
| 70 |
+
"cached_append_forward": 0.0,
|
| 71 |
+
"causal_alibi_forward": 0.00011304020881652832,
|
| 72 |
+
"causal_alibi_loss": 1.043081283569336e-07,
|
| 73 |
+
"causal_alibi_param_grad": 1.3775425031781197e-06,
|
| 74 |
+
"causal_alibi_x_grad": 1.592270564287901e-07,
|
| 75 |
+
"metric_cache_cleared_on_train": 0.0,
|
| 76 |
+
"metric_cache_reused": 0.0,
|
| 77 |
+
"none_forward": 8.377432823181152e-05,
|
| 78 |
+
"none_loss": 1.30385160446167e-08,
|
| 79 |
+
"none_param_grad": 5.532929208129644e-07,
|
| 80 |
+
"none_x_grad": 3.541026671882719e-08,
|
| 81 |
+
"sat_alibi_forward": 0.00014029256999492645,
|
| 82 |
+
"sat_alibi_loss": 9.12696123123169e-08,
|
| 83 |
+
"sat_alibi_param_grad": 9.688083082437515e-07,
|
| 84 |
+
"sat_alibi_x_grad": 1.5371188055723906e-07
|
| 85 |
+
},
|
| 86 |
+
"tol": 0.0002
|
| 87 |
+
},
|
| 88 |
+
{
|
| 89 |
+
"backend": "sdpa",
|
| 90 |
+
"d": 128,
|
| 91 |
+
"dk": 16,
|
| 92 |
+
"expected_k_width": 16,
|
| 93 |
+
"heads": 8,
|
| 94 |
+
"ok": true,
|
| 95 |
+
"preset": "micro_3x",
|
| 96 |
+
"rank": 48,
|
| 97 |
+
"rows": {
|
| 98 |
+
"cache_k_width": 16.0,
|
| 99 |
+
"cache_v_width": 16.0,
|
| 100 |
+
"cached_append_forward": 0.0,
|
| 101 |
+
"causal_alibi_forward": 0.00011304020881652832,
|
| 102 |
+
"causal_alibi_loss": 1.0244548320770264e-07,
|
| 103 |
+
"causal_alibi_param_grad": 1.3635726645588875e-06,
|
| 104 |
+
"causal_alibi_x_grad": 1.8296123016625643e-07,
|
| 105 |
+
"metric_cache_cleared_on_train": 0.0,
|
| 106 |
+
"metric_cache_reused": 0.0,
|
| 107 |
+
"none_forward": 8.377432823181152e-05,
|
| 108 |
+
"none_loss": 5.587935447692871e-09,
|
| 109 |
+
"none_param_grad": 5.358306225389242e-07,
|
| 110 |
+
"none_x_grad": 3.1091985874809325e-08,
|
| 111 |
+
"sat_alibi_forward": 0.00014029256999492645,
|
| 112 |
+
"sat_alibi_loss": 9.12696123123169e-08,
|
| 113 |
+
"sat_alibi_param_grad": 1.071603037416935e-06,
|
| 114 |
+
"sat_alibi_x_grad": 1.1662632459774613e-07
|
| 115 |
+
},
|
| 116 |
+
"tol": 0.0002
|
| 117 |
+
}
|
| 118 |
+
]
|
remote_verify_qkv_all_backends_agillm4.json
ADDED
|
@@ -0,0 +1,177 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
[
|
| 2 |
+
{
|
| 3 |
+
"backend": "manual",
|
| 4 |
+
"d": 32,
|
| 5 |
+
"dk": 16,
|
| 6 |
+
"heads": 2,
|
| 7 |
+
"ok": true,
|
| 8 |
+
"preset": "pico_1x",
|
| 9 |
+
"rank": 16,
|
| 10 |
+
"rows": {
|
| 11 |
+
"causal_alibi_forward": 0.0,
|
| 12 |
+
"causal_alibi_loss": 0.0,
|
| 13 |
+
"causal_alibi_param_grad": 0.0,
|
| 14 |
+
"causal_alibi_x_grad": 5.820766091346741e-11,
|
| 15 |
+
"legacy_load_forward": 0.0,
|
| 16 |
+
"legacy_load_missing_unexpected": 0.0,
|
| 17 |
+
"legacy_load_qkv_weight": 0.0,
|
| 18 |
+
"none_forward": 0.0,
|
| 19 |
+
"none_loss": 0.0,
|
| 20 |
+
"none_param_grad": 0.0,
|
| 21 |
+
"none_x_grad": 5.820766091346741e-11,
|
| 22 |
+
"optimizer_remap": 0.0,
|
| 23 |
+
"safe_load_any_loaded": 0.0,
|
| 24 |
+
"safe_load_any_qkv": 0.0,
|
| 25 |
+
"sat_alibi_forward": 0.0,
|
| 26 |
+
"sat_alibi_loss": 0.0,
|
| 27 |
+
"sat_alibi_param_grad": 0.0,
|
| 28 |
+
"sat_alibi_x_grad": 2.3283064365386963e-10
|
| 29 |
+
},
|
| 30 |
+
"tol": 0.0002
|
| 31 |
+
},
|
| 32 |
+
{
|
| 33 |
+
"backend": "sdpa",
|
| 34 |
+
"d": 32,
|
| 35 |
+
"dk": 16,
|
| 36 |
+
"heads": 2,
|
| 37 |
+
"ok": true,
|
| 38 |
+
"preset": "pico_1x",
|
| 39 |
+
"rank": 16,
|
| 40 |
+
"rows": {
|
| 41 |
+
"causal_alibi_forward": 0.0,
|
| 42 |
+
"causal_alibi_loss": 0.0,
|
| 43 |
+
"causal_alibi_param_grad": 0.0,
|
| 44 |
+
"causal_alibi_x_grad": 1.1641532182693481e-10,
|
| 45 |
+
"legacy_load_forward": 0.0,
|
| 46 |
+
"legacy_load_missing_unexpected": 0.0,
|
| 47 |
+
"legacy_load_qkv_weight": 0.0,
|
| 48 |
+
"none_forward": 0.0,
|
| 49 |
+
"none_loss": 0.0,
|
| 50 |
+
"none_param_grad": 0.0,
|
| 51 |
+
"none_x_grad": 5.820766091346741e-11,
|
| 52 |
+
"safe_load_any_loaded": 0.0,
|
| 53 |
+
"safe_load_any_qkv": 0.0,
|
| 54 |
+
"sat_alibi_forward": 0.0,
|
| 55 |
+
"sat_alibi_loss": 0.0,
|
| 56 |
+
"sat_alibi_param_grad": 0.0,
|
| 57 |
+
"sat_alibi_x_grad": 1.1641532182693481e-10
|
| 58 |
+
},
|
| 59 |
+
"tol": 0.0002
|
| 60 |
+
},
|
| 61 |
+
{
|
| 62 |
+
"backend": "sublinear",
|
| 63 |
+
"d": 32,
|
| 64 |
+
"dk": 16,
|
| 65 |
+
"heads": 2,
|
| 66 |
+
"ok": true,
|
| 67 |
+
"preset": "pico_1x",
|
| 68 |
+
"rank": 16,
|
| 69 |
+
"rows": {
|
| 70 |
+
"causal_alibi_forward": 0.0,
|
| 71 |
+
"causal_alibi_loss": 0.0,
|
| 72 |
+
"causal_alibi_param_grad": 0.0,
|
| 73 |
+
"causal_alibi_x_grad": 1.1641532182693481e-10,
|
| 74 |
+
"legacy_load_forward": 0.0,
|
| 75 |
+
"legacy_load_missing_unexpected": 0.0,
|
| 76 |
+
"legacy_load_qkv_weight": 0.0,
|
| 77 |
+
"none_forward": 0.0,
|
| 78 |
+
"none_loss": 0.0,
|
| 79 |
+
"none_param_grad": 0.0,
|
| 80 |
+
"none_x_grad": 5.820766091346741e-11,
|
| 81 |
+
"safe_load_any_loaded": 0.0,
|
| 82 |
+
"safe_load_any_qkv": 0.0,
|
| 83 |
+
"sat_alibi_forward": 0.0,
|
| 84 |
+
"sat_alibi_loss": 0.0,
|
| 85 |
+
"sat_alibi_param_grad": 0.0,
|
| 86 |
+
"sat_alibi_x_grad": 1.1641532182693481e-10
|
| 87 |
+
},
|
| 88 |
+
"tol": 0.0002
|
| 89 |
+
},
|
| 90 |
+
{
|
| 91 |
+
"backend": "manual",
|
| 92 |
+
"d": 128,
|
| 93 |
+
"dk": 16,
|
| 94 |
+
"heads": 8,
|
| 95 |
+
"ok": true,
|
| 96 |
+
"preset": "micro_3x",
|
| 97 |
+
"rank": 48,
|
| 98 |
+
"rows": {
|
| 99 |
+
"causal_alibi_forward": 0.0,
|
| 100 |
+
"causal_alibi_loss": 0.0,
|
| 101 |
+
"causal_alibi_param_grad": 0.0,
|
| 102 |
+
"causal_alibi_x_grad": 5.820766091346741e-11,
|
| 103 |
+
"legacy_load_forward": 0.0,
|
| 104 |
+
"legacy_load_missing_unexpected": 0.0,
|
| 105 |
+
"legacy_load_qkv_weight": 0.0,
|
| 106 |
+
"none_forward": 0.0,
|
| 107 |
+
"none_loss": 0.0,
|
| 108 |
+
"none_param_grad": 0.0,
|
| 109 |
+
"none_x_grad": 1.4551915228366852e-11,
|
| 110 |
+
"safe_load_any_loaded": 0.0,
|
| 111 |
+
"safe_load_any_qkv": 0.0,
|
| 112 |
+
"sat_alibi_forward": 0.0,
|
| 113 |
+
"sat_alibi_loss": 0.0,
|
| 114 |
+
"sat_alibi_param_grad": 0.0,
|
| 115 |
+
"sat_alibi_x_grad": 5.820766091346741e-11
|
| 116 |
+
},
|
| 117 |
+
"tol": 0.0002
|
| 118 |
+
},
|
| 119 |
+
{
|
| 120 |
+
"backend": "sdpa",
|
| 121 |
+
"d": 128,
|
| 122 |
+
"dk": 16,
|
| 123 |
+
"heads": 8,
|
| 124 |
+
"ok": true,
|
| 125 |
+
"preset": "micro_3x",
|
| 126 |
+
"rank": 48,
|
| 127 |
+
"rows": {
|
| 128 |
+
"causal_alibi_forward": 0.0,
|
| 129 |
+
"causal_alibi_loss": 0.0,
|
| 130 |
+
"causal_alibi_param_grad": 0.0,
|
| 131 |
+
"causal_alibi_x_grad": 5.820766091346741e-11,
|
| 132 |
+
"legacy_load_forward": 0.0,
|
| 133 |
+
"legacy_load_missing_unexpected": 0.0,
|
| 134 |
+
"legacy_load_qkv_weight": 0.0,
|
| 135 |
+
"none_forward": 0.0,
|
| 136 |
+
"none_loss": 0.0,
|
| 137 |
+
"none_param_grad": 0.0,
|
| 138 |
+
"none_x_grad": 1.4551915228366852e-11,
|
| 139 |
+
"safe_load_any_loaded": 0.0,
|
| 140 |
+
"safe_load_any_qkv": 0.0,
|
| 141 |
+
"sat_alibi_forward": 0.0,
|
| 142 |
+
"sat_alibi_loss": 0.0,
|
| 143 |
+
"sat_alibi_param_grad": 0.0,
|
| 144 |
+
"sat_alibi_x_grad": 2.9103830456733704e-11
|
| 145 |
+
},
|
| 146 |
+
"tol": 0.0002
|
| 147 |
+
},
|
| 148 |
+
{
|
| 149 |
+
"backend": "sublinear",
|
| 150 |
+
"d": 128,
|
| 151 |
+
"dk": 16,
|
| 152 |
+
"heads": 8,
|
| 153 |
+
"ok": true,
|
| 154 |
+
"preset": "micro_3x",
|
| 155 |
+
"rank": 48,
|
| 156 |
+
"rows": {
|
| 157 |
+
"causal_alibi_forward": 0.0,
|
| 158 |
+
"causal_alibi_loss": 0.0,
|
| 159 |
+
"causal_alibi_param_grad": 0.0,
|
| 160 |
+
"causal_alibi_x_grad": 5.820766091346741e-11,
|
| 161 |
+
"legacy_load_forward": 0.0,
|
| 162 |
+
"legacy_load_missing_unexpected": 0.0,
|
| 163 |
+
"legacy_load_qkv_weight": 0.0,
|
| 164 |
+
"none_forward": 0.0,
|
| 165 |
+
"none_loss": 0.0,
|
| 166 |
+
"none_param_grad": 0.0,
|
| 167 |
+
"none_x_grad": 1.4551915228366852e-11,
|
| 168 |
+
"safe_load_any_loaded": 0.0,
|
| 169 |
+
"safe_load_any_qkv": 0.0,
|
| 170 |
+
"sat_alibi_forward": 0.0,
|
| 171 |
+
"sat_alibi_loss": 0.0,
|
| 172 |
+
"sat_alibi_param_grad": 0.0,
|
| 173 |
+
"sat_alibi_x_grad": 2.9103830456733704e-11
|
| 174 |
+
},
|
| 175 |
+
"tol": 0.0002
|
| 176 |
+
}
|
| 177 |
+
]
|
run_agillm4_4090_longblock.sh
CHANGED
|
@@ -16,7 +16,7 @@ mkdir -p /workspace/agillm4_4090_ckpts
|
|
| 16 |
|
| 17 |
echo "START_AGILLM4_4090_LONG_BLOCK $(date -u +%Y-%m-%dT%H:%M:%SZ) host=$(hostname)"
|
| 18 |
echo "This is production AGILLM long-block training on 24GB, not a local toy test."
|
| 19 |
-
echo "preset=${AGILLM4_4090_PRESET:-large} block=${AGILLM4_4090_BLOCK:-1536} sat_every=1"
|
| 20 |
|
| 21 |
exec python -u /workspace/agillm-4/nB300_agillm4.py train \
|
| 22 |
--preset "${AGILLM4_4090_PRESET:-large}" \
|
|
@@ -27,6 +27,10 @@ exec python -u /workspace/agillm-4/nB300_agillm4.py train \
|
|
| 27 |
--grad_checkpoint \
|
| 28 |
--optimizer "${AGILLM4_4090_OPTIMIZER:-paged_adamw8bit}" \
|
| 29 |
--sat_every 1 \
|
|
|
|
|
|
|
|
|
|
|
|
|
| 30 |
--token_param_ratio 100 \
|
| 31 |
--save_dir /workspace/agillm4_4090_ckpts \
|
| 32 |
--save_every_sec "${AGILLM4_4090_SAVE_EVERY_SEC:-21600}" \
|
|
|
|
| 16 |
|
| 17 |
echo "START_AGILLM4_4090_LONG_BLOCK $(date -u +%Y-%m-%dT%H:%M:%SZ) host=$(hostname)"
|
| 18 |
echo "This is production AGILLM long-block training on 24GB, not a local toy test."
|
| 19 |
+
echo "preset=${AGILLM4_4090_PRESET:-large} block=${AGILLM4_4090_BLOCK:-1536} sat_every=1 nat_every=${AGILLM4_4090_NAT_EVERY:-4}"
|
| 20 |
|
| 21 |
exec python -u /workspace/agillm-4/nB300_agillm4.py train \
|
| 22 |
--preset "${AGILLM4_4090_PRESET:-large}" \
|
|
|
|
| 27 |
--grad_checkpoint \
|
| 28 |
--optimizer "${AGILLM4_4090_OPTIMIZER:-paged_adamw8bit}" \
|
| 29 |
--sat_every 1 \
|
| 30 |
+
--nat_every "${AGILLM4_4090_NAT_EVERY:-4}" \
|
| 31 |
+
--nat_loss_weight "${AGILLM4_4090_NAT_LOSS_WEIGHT:-1.0}" \
|
| 32 |
+
--nat_expand "${AGILLM4_4090_NAT_EXPAND:-2}" \
|
| 33 |
+
--nat_max_tokens "${AGILLM4_4090_NAT_MAX_TOKENS:-768}" \
|
| 34 |
--token_param_ratio 100 \
|
| 35 |
--save_dir /workspace/agillm4_4090_ckpts \
|
| 36 |
--save_every_sec "${AGILLM4_4090_SAVE_EVERY_SEC:-21600}" \
|
run_agillm4_4090_sublinear_probe.sh
CHANGED
|
@@ -16,5 +16,9 @@ python -u ./nB300_agillm4.py train \
|
|
| 16 |
--grad_checkpoint \
|
| 17 |
--optimizer "${AGILLM4_OPTIMIZER:-paged_adamw8bit}" \
|
| 18 |
--sat_every 1 \
|
|
|
|
|
|
|
|
|
|
|
|
|
| 19 |
--token_param_ratio 100 \
|
| 20 |
--save_dir "${AGILLM4_SAVE_DIR:-/workspace/ckpts_agillm4_sublinear_4090}"
|
|
|
|
| 16 |
--grad_checkpoint \
|
| 17 |
--optimizer "${AGILLM4_OPTIMIZER:-paged_adamw8bit}" \
|
| 18 |
--sat_every 1 \
|
| 19 |
+
--nat_every "${AGILLM4_NAT_EVERY:-4}" \
|
| 20 |
+
--nat_loss_weight "${AGILLM4_NAT_LOSS_WEIGHT:-1.0}" \
|
| 21 |
+
--nat_expand "${AGILLM4_NAT_EXPAND:-2}" \
|
| 22 |
+
--nat_max_tokens "${AGILLM4_NAT_MAX_TOKENS:-768}" \
|
| 23 |
--token_param_ratio 100 \
|
| 24 |
--save_dir "${AGILLM4_SAVE_DIR:-/workspace/ckpts_agillm4_sublinear_4090}"
|
run_agillm4_main_b200_b300.sh
CHANGED
|
@@ -15,7 +15,7 @@ fi
|
|
| 15 |
mkdir -p /workspace/agillm4_ckpts
|
| 16 |
|
| 17 |
echo "START_AGILLM4_MAIN $(date -u +%Y-%m-%dT%H:%M:%SZ) host=$(hostname)"
|
| 18 |
-
echo "preset=agillm4_main target_tokens=150000000000 token_param_ratio=100 block=${AGILLM4_BLOCK:-2048}"
|
| 19 |
|
| 20 |
exec python -u /workspace/agillm-4/nB300_agillm4.py train \
|
| 21 |
--preset agillm4_main \
|
|
@@ -26,6 +26,10 @@ exec python -u /workspace/agillm-4/nB300_agillm4.py train \
|
|
| 26 |
--grad_checkpoint \
|
| 27 |
--optimizer "${AGILLM4_OPTIMIZER:-adamw}" \
|
| 28 |
--sat_every 1 \
|
|
|
|
|
|
|
|
|
|
|
|
|
| 29 |
--token_param_ratio 100 \
|
| 30 |
--target_tokens 150000000000 \
|
| 31 |
--save_dir /workspace/agillm4_ckpts \
|
|
|
|
| 15 |
mkdir -p /workspace/agillm4_ckpts
|
| 16 |
|
| 17 |
echo "START_AGILLM4_MAIN $(date -u +%Y-%m-%dT%H:%M:%SZ) host=$(hostname)"
|
| 18 |
+
echo "preset=agillm4_main target_tokens=150000000000 token_param_ratio=100 block=${AGILLM4_BLOCK:-2048} sat_every=1 nat_every=${AGILLM4_NAT_EVERY:-1}"
|
| 19 |
|
| 20 |
exec python -u /workspace/agillm-4/nB300_agillm4.py train \
|
| 21 |
--preset agillm4_main \
|
|
|
|
| 26 |
--grad_checkpoint \
|
| 27 |
--optimizer "${AGILLM4_OPTIMIZER:-adamw}" \
|
| 28 |
--sat_every 1 \
|
| 29 |
+
--nat_every "${AGILLM4_NAT_EVERY:-1}" \
|
| 30 |
+
--nat_loss_weight "${AGILLM4_NAT_LOSS_WEIGHT:-1.0}" \
|
| 31 |
+
--nat_expand "${AGILLM4_NAT_EXPAND:-2}" \
|
| 32 |
+
--nat_max_tokens "${AGILLM4_NAT_MAX_TOKENS:-0}" \
|
| 33 |
--token_param_ratio 100 \
|
| 34 |
--target_tokens 150000000000 \
|
| 35 |
--save_dir /workspace/agillm4_ckpts \
|
verify_anchor_memory_and_kv_buffer_agillm4.py
ADDED
|
@@ -0,0 +1,121 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
#!/usr/bin/env python3
|
| 2 |
+
"""Verifier for AGILLM-4 anchor-memory wiring and KV-buffer harvest.
|
| 3 |
+
|
| 4 |
+
Two harvested/integrated features share this file because both touch
|
| 5 |
+
nB300_agillm4 and both need a regression-safe smoke test:
|
| 6 |
+
|
| 7 |
+
1. Anchor memory: AnchorMemoryLayer wired into Encoder at a configurable
|
| 8 |
+
mid-stack position. Default-off; enabled with ``--anchor_memory`` and
|
| 9 |
+
parameterised by ``--anchor_stride / --anchor_max / --anchor_position``.
|
| 10 |
+
|
| 11 |
+
2. KV buffer: ``KVBuffer`` replaces ``torch.cat``-based decode-time cache
|
| 12 |
+
growth. Default-off (legacy tuple cache still works); enabled per
|
| 13 |
+
call site by passing a pre-sized ``KVBuffer`` as ``kv_cache``.
|
| 14 |
+
|
| 15 |
+
Run:
|
| 16 |
+
|
| 17 |
+
python /workspace/agillm-4/verify_anchor_memory_and_kv_buffer_agillm4.py
|
| 18 |
+
"""
|
| 19 |
+
from __future__ import annotations
|
| 20 |
+
|
| 21 |
+
import pathlib
|
| 22 |
+
import sys
|
| 23 |
+
|
| 24 |
+
HERE = pathlib.Path(__file__).resolve().parent
|
| 25 |
+
if str(HERE) not in sys.path:
|
| 26 |
+
sys.path.insert(0, str(HERE))
|
| 27 |
+
|
| 28 |
+
import torch # noqa: E402
|
| 29 |
+
|
| 30 |
+
import nB300_agillm4 as m # noqa: E402
|
| 31 |
+
from anchor_memory import AnchorMemoryLayer # noqa: E402
|
| 32 |
+
|
| 33 |
+
DEV = "cuda" if torch.cuda.is_available() else "cpu"
|
| 34 |
+
FAILS = 0
|
| 35 |
+
|
| 36 |
+
|
| 37 |
+
def check(name: str, cond: bool, detail: str = ""):
|
| 38 |
+
global FAILS
|
| 39 |
+
tag = "OK " if cond else "FAIL"
|
| 40 |
+
if not cond:
|
| 41 |
+
FAILS += 1
|
| 42 |
+
print(f" [{tag}] {name}" + (f" — {detail}" if detail else ""))
|
| 43 |
+
|
| 44 |
+
|
| 45 |
+
def section(title: str):
|
| 46 |
+
print(f"\n=== {title} ===")
|
| 47 |
+
|
| 48 |
+
|
| 49 |
+
# ----- 1. KVBuffer -----
|
| 50 |
+
section("KVBuffer fill / view / overflow")
|
| 51 |
+
buf = m.KVBuffer(batch=2, heads=4, capacity=32, d_k=8, device=DEV, dtype=torch.float32)
|
| 52 |
+
check("initial length is 0", buf.length == 0)
|
| 53 |
+
check("k/v allocated", buf.k.shape == (2, 4, 32, 8) and buf.v.shape == (2, 4, 32, 8))
|
| 54 |
+
|
| 55 |
+
k1 = torch.randn(2, 4, 5, 8, device=DEV)
|
| 56 |
+
v1 = torch.randn(2, 4, 5, 8, device=DEV)
|
| 57 |
+
buf.append(k1, v1)
|
| 58 |
+
check("after append(5) length is 5", buf.length == 5)
|
| 59 |
+
k_view, v_view = buf.view()
|
| 60 |
+
check("view shape", k_view.shape == (2, 4, 5, 8))
|
| 61 |
+
check("k content correct", torch.allclose(k_view, k1))
|
| 62 |
+
check("v content correct", torch.allclose(v_view, v1))
|
| 63 |
+
|
| 64 |
+
buf.append(torch.randn(2, 4, 27, 8, device=DEV), torch.randn(2, 4, 27, 8, device=DEV))
|
| 65 |
+
check("filled to capacity (32)", buf.length == 32)
|
| 66 |
+
try:
|
| 67 |
+
buf.append(torch.zeros(2, 4, 1, 8, device=DEV), torch.zeros(2, 4, 1, 8, device=DEV))
|
| 68 |
+
check("overflow raises", False, "no exception")
|
| 69 |
+
except RuntimeError:
|
| 70 |
+
check("overflow raises", True)
|
| 71 |
+
|
| 72 |
+
# ----- 2. Encoder without anchor memory (regression) -----
|
| 73 |
+
section("Encoder regression (anchor disabled)")
|
| 74 |
+
cfg = {"d": 64, "layers": 4, "heads": 4, "rank": 8}
|
| 75 |
+
enc_plain = m.Encoder(cfg).to(DEV)
|
| 76 |
+
check("anchor module is None", enc_plain.anchor is None)
|
| 77 |
+
check("anchor_memory_enabled is False", enc_plain.anchor_memory_enabled is False)
|
| 78 |
+
ids = torch.randint(0, m.VOCAB, (1, 16), device=DEV)
|
| 79 |
+
with torch.no_grad():
|
| 80 |
+
out = enc_plain(ids, mask=None)
|
| 81 |
+
check("plain forward shape", out.shape == (1, 16, 64), str(out.shape))
|
| 82 |
+
|
| 83 |
+
# ----- 3. Encoder with anchor memory -----
|
| 84 |
+
section("Encoder with anchor memory enabled")
|
| 85 |
+
enc = m.Encoder(
|
| 86 |
+
cfg,
|
| 87 |
+
anchor_memory=True,
|
| 88 |
+
anchor_stride=8,
|
| 89 |
+
anchor_max=16,
|
| 90 |
+
anchor_position=-1,
|
| 91 |
+
).to(DEV)
|
| 92 |
+
check("anchor_memory_enabled", enc.anchor_memory_enabled is True)
|
| 93 |
+
check("anchor is AnchorMemoryLayer", isinstance(enc.anchor, AnchorMemoryLayer))
|
| 94 |
+
check("position resolved to mid-stack (4//2=2)", enc.anchor_position == 2, f"got {enc.anchor_position}")
|
| 95 |
+
|
| 96 |
+
ids2 = torch.randint(0, m.VOCAB, (1, 32), device=DEV)
|
| 97 |
+
out2 = enc(ids2, mask=None)
|
| 98 |
+
check("anchor-on forward shape", out2.shape == (1, 32, 64), str(out2.shape))
|
| 99 |
+
|
| 100 |
+
loss = out2.sum()
|
| 101 |
+
loss.backward()
|
| 102 |
+
grads = [p.grad for p in enc.anchor.parameters() if p.grad is not None]
|
| 103 |
+
check("anchor params got gradients", len(grads) > 0, f"{len(grads)} grads")
|
| 104 |
+
nonzero = sum(int(g.abs().sum() > 0) for g in grads)
|
| 105 |
+
check("anchor gradients non-zero", nonzero > 0, f"{nonzero}/{len(grads)}")
|
| 106 |
+
|
| 107 |
+
# ----- 4. Anchor position resolution -----
|
| 108 |
+
section("Anchor position resolution")
|
| 109 |
+
check("explicit position=1", m.Encoder(cfg, anchor_memory=True, anchor_position=1).anchor_position == 1)
|
| 110 |
+
check("oversized position clamped", m.Encoder(cfg, anchor_memory=True, anchor_position=99).anchor_position == cfg["layers"] - 1)
|
| 111 |
+
|
| 112 |
+
# ----- 5. Module-level defaults exposed -----
|
| 113 |
+
section("Module-level defaults present")
|
| 114 |
+
for name in ("DEFAULT_ANCHOR_MEMORY", "DEFAULT_ANCHOR_STRIDE", "DEFAULT_ANCHOR_MAX", "DEFAULT_ANCHOR_POSITION", "DEFAULT_KV_BUFFER"):
|
| 115 |
+
check(name, hasattr(m, name))
|
| 116 |
+
|
| 117 |
+
print()
|
| 118 |
+
if FAILS:
|
| 119 |
+
print(f"=== {FAILS} CHECK(S) FAILED ===")
|
| 120 |
+
sys.exit(1)
|
| 121 |
+
print("=== ALL CHECKS PASSED ===")
|