SO-101 brush painting

Let your agent handle the setup

Point your coding agent at this bucket and give it this prompt:

Set up and run the SO-101 painting simulation from https://huggingface.co/buckets/mishig/so101-painting on my computer. Read the README and learning report, download the files, verify their checksums, and use uv for all Python environment and package management. Start with attempt 04's settings to paint the Eiffel Tower and Seine. Record the new attempt, inspect its actual camera image, and explain any proposed changes before testing more attempts. Preserve every recording and use the unchanged camera evaluator to compare results.

Download the public bucket

uvx --from huggingface-hub hf buckets sync hf://buckets/mishig/so101-painting ./so101-painting
cd so101-painting
uv sync --locked
uv run --locked python verify_manifest.py

The release includes a MANIFEST.sha256 listing each file's checksum. The verification command detects missing or changed files; it does not run the simulator.

About the demo

A local MuJoCo simulation of an SO-101 arm painting the Eiffel Tower and the Seine. Each attempt starts with clean paper and keeps its own recording, parameters, camera images, and reports so you can inspect changes between runs.

Recorded learning trials

Four Paris attempts are included. Watch the 30-second comparison, compare the camera results, or read what changed and why. Every original recording and parameter file is retained in results/attempt-01 through results/attempt-04. The fixed RGB geometry/evenness score improved from 83.37 to 96.26 across this batch. It is a diagnostic metric, not an aesthetic rating or a promise that future attempts will improve.

To continue from the best settings, point your agent to this README and learning-report.md, then use results/attempt-04/parameters.json as the starting point for a new attempt. Ask it to inspect the actual camera image, state its proposed adjustment, preserve each recording, compare all results with the unchanged evaluator, and retain the best result if a trial regresses.

Run a new recorded attempt

Install uv and FFmpeg, then run these commands from this folder. On macOS, FFmpeg can be installed with brew install ffmpeg; on Debian/Ubuntu, use sudo apt install ffmpeg.

uv sync --locked
uv run --locked python scripts/run_attempt.py \
  --run-dir results/my-attempt \
  --parameters results/attempt-04/parameters.json

The runner opens a fresh simulator, records the painting, finalizes the video, and shuts down its own simulator. It uses mjpython from the uv environment for the macOS viewer and the environment's Python for headless runs and other systems. No physical robot is connected.

Use a new result directory for every attempt. The runner preserves existing recordings and refuses to attach to an occupied port. To use another port, append --port 8880. Add --headless to omit the viewer or --fast to advance the simulation faster than wall time. Headless Linux still needs a working MuJoCo rendering backend; with an EGL-capable graphics driver, set MUJOCO_GL=egl before running the command.

If the native viewer fails during startup, retry with --headless and a new --run-dir. Check server.log for the startup error. The controller sends no painting commands until the runner confirms that its own server is ready.

The pinned Python dependencies are in pyproject.toml and uv.lock. Manage the environment with uv; the runner reuses the interpreter and MuJoCo launcher from that environment.

Create the short video

uv run python scripts/edit_video.py \
  --run-dir results/my-attempt \
  --output results/my-attempt/preview.mp4

The edit is at most 30 seconds, with native 1080p text, recorded activity captions, and a visible playback-speed label. Actions use one uniform whole-number speed; the ending plays at normal speed. If the recording is too short for the ending hold, that hold is labeled. The enlarged canvas is rectified from the same live front-camera frames, so the brush can occlude it naturally. The editor never substitutes the finished paint texture during the action.

Recreate the four-attempt comparison with uv run python scripts/edit_comparison.py --selected 4 --output learning-runs.mp4. The comparison uses each complete action sequence with its own speed label, then shows the four final camera views. The included learning-video-edit.json and each attempt's preview-edit.json preserve the original edit timing.

Keep and compare attempts

Copy an earlier parameters.json to a new file, change the desired values, and pass that file to a new run. Parameters include brush contact depth, stroke speed, motion easing, settling frames, and touch/lift timing. Every run copies its parameter file into its result directory before starting.

Each attempt retains:

An interrupted or failed attempt can contain only the files produced before it stopped. Available recordings are retained. The short editor labels attempts without a final camera completion marker as “Run ended.” A completed run means the scripted execution finished; it does not certify that the picture is artistically better than another attempt.

How the robot paints

The Paris picture is a scripted stroke plan in scripts/paris_design.py. Four cyan fiducials are detected in the front RGB image to locate the paper and calculate a planar homography. The controller uses joint-angle feedback and the robot's kinematic model to convert planned strokes into joint targets. It checks camera images after strokes.

The controller also reads simulated paint-load state to decide when to refill, simulated brush color to confirm the palette load, and contact counters to detect a missed paper contact. This is not camera-only control. A physical implementation would need corresponding sensing or estimates for these quantities.

MuJoCo simulates the arm, palette, paper contact, and a spring-loaded brush. Paint transfers when the brush contacts the paper. A separate 2D pigment model approximates paint loading, spreading, mixing, and drying, and updates the visible canvas texture. It is a simplified wet-paint model, not a 3D fluid simulation.

Parameter changes between attempts are explicit experiments. There is no self-training or reinforcement-learning loop, and improvement is not guaranteed to be monotonic. Keep the failed and successful attempts to compare the effects of each change.

Source and licenses

The SO-101 robot model and meshes originate from TheRobotStudio/SO-ARM100, under Apache License 2.0. The scene adds the brush, palette, paper, and simulation-specific contact settings. See LICENSE.

The video captions use the bundled, unmodified Inter font under the SIL Open Font License 1.1. Its license and pinned source/hash metadata are in assets/fonts. No proprietary system fonts are required.

Everything runs and saves locally. These scripts do not upload attempts or videos to Hugging Face or another service.