I built a small robot-learning model from scratch. It takes a camera image, an instruction, and the robot's joint state, and outputs motor commands to pick up a cube and place it at a target. It sat at 0% success for days. Bigger heads, more layers, different losses: none of it moved the number.

The fix was never the model. It was what the model could see. Twice, the thing that unblocked me was fixing the observation, not the architecture. Getting from 0% to a working policy turned out to be a debugging story rather than a modelling one, and that story is the most useful thing in this repo. (The exact number, 68% and later 93%, comes with a twist from the ablations further down.)

So VIRENA is really three things:

1. A working VLA in about 2,000 lines that trains on a laptop (frozen CLIP plus a tiny trained head, no GPU needed).
2. An ablation cookbook that measures what each trick is actually worth, with confidence intervals.
3. A failure-diagnosis toolkit that tells you whether your policy is failing because of perception, control, or data. That is the thing I had to work out by hand, now packaged so you don't have to.

It's small enough to read in an afternoon, it runs on a Mac without a GPU, and it's built to teach you how these systems actually work and how they fail. If you're learning how a VLA fits together, or you want a compact base to hack your own ideas onto, that's who it's for.

The task

PickCube-v1 in the ManiSkill3 simulator. A Panda arm has to pick up a red cube and place it at a random 3D goal, within 2.5 cm, then hold still. The scene is randomized every episode. The model only ever gets a 128x128 camera image, the arm's joint angles, and the goal coordinate.

_{A trained rollout (external view). The policy itself only sees the small wrist image.}

The 30-second mental model

Only two small pieces get trained: the AttentionPool and the MLP head, about 950k parameters between them. Both CLIP towers stay frozen. That's why it trains in a couple of hours on an M2 and doesn't need a GPU. The full walk-through lives in docs/ARCHITECTURE.md, written for engineers who know PyTorch but not robotics. Every term is defined the first time it shows up.

From 0% to a working policy

Every failure below was an observability problem. No model change fixed them. Changing what the model could observe did.

Finding 1: the camera was blind at the worst possible moment. The first setup used a fixed external camera. At the instant of grasping, the gripper itself covers the cube from that viewpoint. Watch the cube (the little red patch) disappear as the arm closes in:

_{Before: fixed external camera. Cube clearly visible at the start, then swallowed by the gripper right when precision matters most. Success: 30%.}

The symptom was easy to read once I looked for it: the hand-to-cube distance would improve, then flatten out at a hard floor around 3.3 cm, and it was the same floor for every architecture I tried. A floor that no model change can break is the fingerprint of a perception problem, not a learning one. You can't learn to close the last 3 cm to something you can't see.

The fix was a wrist-mounted camera (an "eye in hand") that keeps pointing at the cube through the grasp. Same cube, same grasp, but now the model can actually see what it's doing:

_{After: wrist camera. The cube stays centered and visible all the way into the grasp. Success: 68%. Same everything else, +38 points from the camera alone.}

Same expert, same joint data, only the image source changed. The floor dropped to 1.5 cm and grasping started working. The ablation below puts a number on it: keeping everything else identical and swapping the wrist camera back for the external one drops success from 68% to 30%.

Finding 2: the goal was literally invisible. Even with grasping solved, placement was impossible. In PickCube the goal is a hidden marker at a random 3D point, and it isn't rendered in any camera. The model was being asked to move the cube to a location it had no way to perceive. The fix is goal-conditioning: feed the goal coordinate in as an explicit 3D vector. That coordinate is the task specification, not privileged internal state.

The takeaway, and the reason the two tools below exist: when a robot policy fails, first ask whether it can even see what it needs. Do that before you touch the model.

The ablation cookbook: what each trick is worth

Every design choice is a single flag in config.py, and scripts/ablation.py trains a fresh model for each variant, evaluates them all on the same 100 seeded scenes, and reports a Wilson 95% confidence interval. Turning off one component at a time turns "I think this helped" into a number:

Reading it honestly, including two results I did not expect:

• Goal-conditioning is the backbone (-67). Drop the goal vector and it collapses to 1%. That is Finding 2 quantified: with the goal invisible in the image, the model has nowhere to place the cube.
• The wrist camera is worth +38. base_camera is Finding 1 quantified: identical everything, occluded viewpoint, and success falls from 68% to 30%.
• Normalizing inputs matters (goal -14, proprio -7), the usual "keep your inputs on a sane scale" lesson.
• Action chunking actually hurts here (+25 to remove it). This is the surprise. I added chunking because ACT does, but for PickCube at this scale, plain single-step closed-loop prediction (chunk=1) scores 93%, well above the chunked full at 68%. Chunking commits to stale actions; re-deciding every step wins. It also reframes an earlier inference-only result where temporal ensembling looked worth +25 points: ensembling was partly repairing the damage chunking did to the chunked model, but plain single-step beats both.
• Two things I built make no measurable difference here: patch tokens vs the pooled CLIP embedding, and proprio history. Both land exactly at 68%. Good to know before I talk myself into believing they were essential.

The failure-diagnosis toolkit: perception, control, or data?

diagnose/ packages the debugging method into one command that returns a verdict of PERCEPTION, CONTROL, or DATA, with evidence. It combines a linear probe (can the frozen features even predict the actions?) with rollout signatures (distance floors, grasped-then-dropped, wandering).

$ python scripts/diagnose.py --checkpoint checkpoints/best.pt --seed 5

Perception probe   dim   probe R2
                    dx      0.754
                    dy      0.885     (features localise the cube fine)
                    dz      0.807
[rollout seed=5]  tcp_to_obj: min=0.028 (step 9) -> final=0.257
                  signatures: reached_then_left, grasped_then_lost

VERDICT: CONTROL  (confidence: high)
  - features encode the reach well, so perception is not the bottleneck
  - the target WAS grasped then lost, so it is clearly localizable
  -> Fix CONTROL: the failure is holding the grasp, not seeing it. If closed-loop
     is already on, the residual is fine-grained precision (edge-case data or resolution).

Point it at a policy and it tells you whether to fix the camera, the controller, or the data, with evidence instead of guesswork.

Quick start

One conda environment does everything (data collection, training, evaluation, diagnosis). Installing pinocchio from conda-forge first is the trick that lets ManiSkill's IK and the training stack coexist; a plain pip install of both hits a numpy conflict. On macOS the scripts set KMP_DUPLICATE_LIB_OK=TRUE for you so you don't hit the duplicate-OpenMP abort.

conda create -n virena python=3.12 && conda activate virena
conda install -c conda-forge pinocchio
pip install "mani-skill==3.0.1" torch torchvision transformers Pillow tqdm imageio

Try it without training

A pretrained single-step checkpoint (the 93% model) ships in the repo at pretrained/pickcube.pt, so after the install above you can run everything immediately, no data collection or training required:

python scripts/record_demo.py --checkpoint pretrained/pickcube.pt   # make the demo GIF
python scripts/eval_sim.py    --checkpoint pretrained/pickcube.pt --episodes 100
python scripts/diagnose.py    --checkpoint pretrained/pickcube.pt --seed 5

Train it yourself

# 1. Collect demonstrations (scripted expert, wrist camera, successes only)
python -m data_collection.collect --num-episodes 1200 --output-dir data/raw --success-only
python -m data_collection.build_dataset --episodes-dir data/raw/episodes --output data/dataset.jsonl --success-only

# 2. Train (single-step is the default; writes checkpoints/best.pt)
python scripts/train.py --epochs 50

# 3. Evaluate and diagnose
python scripts/eval_sim.py --episodes 100
python scripts/diagnose.py --checkpoint checkpoints/best.pt --seed 5

Optional: for fast model-only work (the perception probe, dataloader checks, training) you can use a lightweight pip venv with just requirements-model.txt and skip installing the simulator. It's a speed convenience, not a requirement.

Project layout

config.py                 every design toggle in one dataclass (saved into each checkpoint)
data_collection/          scripted expert -> sim rollouts -> JSONL dataset
model/
  vision/                 frozen CLIP patch tokens plus a trainable AttentionPool
  language/               frozen CLIP text encoder
  action/                 the trainable MLP head (action chunking)
  vla.py                  fuses the four input streams
  vla_policy.py           inference: temporal ensembling plus the gripper latch
dataset/                  chunks, masks, episode-level split, normalization stats
diagnose/                 the perception/control/data toolkit (probe, rollout, verdict, tasks)
scripts/                  train, eval_sim, ablation, diagnose
docs/ARCHITECTURE.md      the full guided explanation

Built on ManiSkill3 and OpenAI CLIP. Action chunking and temporal ensembling come from ACT (Zhao et al. 2023).

License

MIT. Do what you like with it; a link back is appreciated.