Ngseo/basevla-maniskill-aug

BaseVLA — ManiSkill Franka (Qwen2.5-VL-3B), vision-only, with augmentation

Plain behaviour cloning baseline: no V-JEPA, no latent alignment, no decorrelation. Just L1(action) on ManiSkill's merged Franka set, trained without proprioception and with domain augmentation on.

Checkpoint at step 40,000. The run was stopped early at ~44k steps (of a planned 100k) because the loss had flattened — 0.0487 at 30k to 0.0474 at 42k.

Data

maniskill-franka-merged from Ngseo/stage1-data, single camera (observation.images.anchor, 256×256 h264), 30 fps, action action.ee_delta_pose (7-D).

The raw set needs filtering. 17.1% of its action.ee_delta_pose rows are NaN, and they are not spread evenly — whole tasks were never populated:

task	frames	NaN (ee_delta)	NaN (joint_delta)	NaN (joint_pos)
PushT-v1	168,934	100%	100%	100%
PokeCube-v1	68,545	69.7%	32.5%	100%
PlaceSphere-v1	118,223	12.1%	2.6%	0.8%
PegInsertionSide-v1	150,055	6.2%	0.7%	0.7%
other 7 tasks	946,166	0.4–2.1%

The ~1% floor is the last frame of each episode (no next action) and is normal. PushT has no usable action under any of the three control modes.

Training on it unfiltered silently produces a broken model: NaNs get zeroed by nan_to_num, so the policy learns "output nothing" on 17% of frames, and the normalisation statistics come out as NaN.

This run therefore drops episodes whose action column is NaN in more than 5% of rows (--max_nan_frac 0.05):

9,216 / 12,257 episodes kept  →  1,218,188 frames (83.3% of the original)

Architecture

Qwen2.5-VL-3B + LoRA ─ forward_full_hidden ─┬─ Head A → z_a ─┐
                                            └─ Head B → z_b ─┴─ concat → ResNetActionHead → 30×7

L = L1(action)        ← that is the whole objective

LoRA	r=32
Heads	dual AttentiveLatentHead, proj 4096, 8 queries, depth 2
Inputs	1 RGB frame @224 + task string — no proprioceptive state
Output	30-step action chunk (1.0 s @ 30 fps)
Optimiser	AdamW, lr 5e-5, batch 32, bf16
Augmentation	ColorJitter/SharpnessJitter ×2 + DomainRandomization p=0.7

--no_state is used, so the state token degenerates to a constant learned bias (state_proj(0)) and no proprioception reaches the model. ManiSkill's merged set has no observation.state column to begin with.

Training curve

step	loss
1,000	0.0962
10,000	0.0601
20,000	0.0524
30,000	0.0487
40,000	0.0476

Not evaluated in simulation or on a robot. The number above is a training-set L1 on normalised actions. No ManiSkill success rate is reported.

Contents

checkpoint.pt holds policy (full VLM incl. LoRA), latent_head, free_latent_head, action_head, optimizer, agg_stats (normalisation statistics) and args (the exact CLI configuration, including the NaN filter threshold).

import torch
ck = torch.load("checkpoint.pt", map_location="cpu", weights_only=False)
ck["step"]        # 40000
ck["args"]        # full training config
ck["agg_stats"]   # action normalisation used at train time