robocurve/gr00t-n1.7-yam-molmoact2

GR00T N1.7: bimanual YAM fine-tune on MolmoAct2 data

Full fine-tune (native Isaac-GR00T recipe: action head + projector trainable, VLM backbone frozen) of nvidia/GR00T-N1.7-3B on the AllenAI MolmoAct2-BimanualYAM dataset for the I2RT YAM bimanual arm platform. The GR00T analog of robocurve/pi05-yam-molmoact2; weights in this repo are the full consolidated checkpoint (no adapters).

Intended use & safety

• Intended use: research and evaluation on I2RT YAM bimanual arms (2× 6-dof arms + grippers, three cameras) for the trained task families: block manipulation, box packing, cable charging.
• Out of scope: any other embodiment or rig without fine-tuning (VLA policies do not zero-shot transfer across embodiments); unattended operation; operation near people without a hardware e-stop and enforced workspace/torque limits.
• Validation status: open-loop MSE on one held-out episode repo (see Losses & evaluation); no closed-loop sim or real-robot success rates reported. Users are responsible for safe integration (guardrails, e-stop, workspace limits) before deployment.

Training

Data	124 source repos (block / box / charging), ~5,145 episodes, ~11.35M frames; no filtering (full dataset release); LeRobot v3 → v2 converted; AV1 videos transcoded to H.264 (torchcodec cannot reliably decode AV1). Held-out val: the separate episode repo allenai/19012026-block-13 (repo-level holdout)
Embodiment	NEW_EMBODIMENT with a custom bimanual config; state & action left_arm → [0:6], left_gripper → [6:7], right_arm → [7:13], right_gripper → [13:14] (absolute joints)
Image preprocessing	shortest-edge-256 resize + 0.95 fractional crop, aspect-preserving (from this repo's processor_config.json; homogeneous first-party camera geometry, no letterbox needed); three views (base_view, left_wrist_view, right_wrist_view)
Schedule	10,000 steps @ global batch 256 × grad-accum 2 (effective 512), lr 1e-4, warmup 1%, episode_sampling_rate=1.0; DeepSpeed ZeRO-2, multi-GPU (B200-class; exact count unknown, plan targeted 4×)
Checkpoint selection	final milestone (step 10,000); the run's eval_results.json records val MSE for this checkpoint only, so no argmin over the curve was possible from artifacts
Headline curve	open-loop action MSE at step 10,000: 0.00279 (earlier-step values unrecorded in available artifacts)

Config chosen by a 4-arm tuning sweep (baseline / lr×4 / batch×2 / full-sampling): the lr 4e-4 arm failed to train (loss stuck ≈1.24), consistent with N1.7 LR-instability observed independently on the SO-101 project at 6e-4.

Losses & evaluation

• Training loss: GR00T N1.7's flow-matching action objective (DiT action head predicts the velocity field for noised action chunks conditioned on frozen VLM features + robot state; MSE against the interpolation target). Trainable: action head + projector; frozen: LLM and vision encoder.
• Eval regime: open-loop MSE. Predicted vs ground-truth action trajectories on the fully held-out episode repo allenai/19012026-block-13, computed by the training pipeline's out-of-band val sidecar (upstream Isaac-GR00T's HF-Trainer eval path is non-functional for sharded datasets). Trajectory/sample counts: unknown (reconstructed from artifacts; per-trajectory eval plots for 10 trajectories exist on the training volume). Normalization parity and eval seeding: unknown (sidecar implementation lives in the private training repo).
• Comparison: π0.5 on the same held-out repo and metric: robocurve/pi05-yam-molmoact2 reports 0.00206 (see that card and robocurve/pi05-yam-replication (private): openpi training + eval-MSE protocol).
• Scope: open-loop MSE is an action-prediction proxy, not a closed-loop success rate; compounding-error effects are not captured.

Provenance

Trained by	aris @ Robocurve, 2026-07
Training code	robocurve/gr00t-n17-yam-replication (private): Modal pipeline covering data prep (v3→v2 + AV1→H.264), training, val-monitor sidecar, plan
Framework	Isaac-GR00T (N1.7), HF Trainer + DeepSpeed ZeRO-2; exact commit and library versions unknown (not recorded in checkpoint artifacts)
Compute provider	Modal; B200-class multi-GPU (plan targeted 4×; exact count unknown)
Wall-clock	unknown (not recorded in available artifacts)
Total compute	unknown (not recorded)
Cost	unknown (not recorded)
Experiment tracking	wandb config present in checkpoint (wandb_config.json); run not publicly linked
Card authorship	written by the team (jeqcho's publishing session), reconstructed from run artifacts on the training volume (eval_results.json, tune_results.json, checkpoint contents, repo README); not first-hand

Usage

Requires the Isaac-GR00T environment, and a Hugging Face login with access to the gated nvidia/Cosmos-Reason2-2B (the N1.7 backbone loads its config from there even though this fine-tune is public).

from gr00t.policy.gr00t_policy import Gr00tPolicy

policy = Gr00tPolicy(
    embodiment_tag="new_embodiment",
    model_path="robocurve/gr00t-n1.7-yam-molmoact2",
    device="cuda",
)
actions, info = policy.get_action(observation)  # dict of (1, 16, D) arrays per action key

• Observations: cameras base_view (overhead/top), left_wrist_view, right_wrist_view (source keys observation.images.top/left/right); state layout left_arm → [0:6], left_gripper → [6:7], right_arm → [7:13], right_gripper → [13:14] in the source data's YAM joint conventions.
• Actions: same 14-dim layout, absolute joint targets; action chunk = 16 steps (experiment_cfg/config.yaml delta_indices 0..15; padded internally to the processor's 40 cap). Values are normalized with the per-dataset statistics shipped in this repo (statistics.json, experiment_cfg/); the policy wrapper applies them.
• Serving: for Inspect Robots evals on real YAM arms, use the gr00t policy and scripts/serve_gr00t_act.py shim from robocurve/inspect-robots-yam (v0.13.0+), which validates this checkpoint's layout and statistics before serving. Isaac-GR00T's own gr00t/eval/run_gr00t_server.py also works but speaks a different, incompatible protocol.

Data provenance & caveats

• Training data is the AllenAI MolmoAct2-BimanualYAM dataset (first-party AllenAI collection; see the dataset card for its license). Videos were transcoded AV1 → H.264 for training.
• Three task families only (block, box, charging); expect limited transfer beyond them.
• Eval limits: open-loop MSE on a single held-out repo; no closed-loop or real-robot results.
• Known training-run facts from the tuning sweep are in this card's Training section; the full incident log lives in the (private) training repo.

Versioning & contact

• main is stable (single published checkpoint, step 10,000). Pin the revision hash for exact reproduction. Superseding checkpoints will set new_version here.
• Issues: HF Discussions on this repo.