Robot Learning — Imitation Learning on the SO-101 Arm

A study on deploying modern imitation-learning policies, ACT, π0.5, and SmolVLA, on a physical SO-101 robotic arm. I collected teleoperated demonstrations, fine-tuned each policy, and evaluated the resulting policies on the real arm.

ACT is trained from scratch on a separate 50-demonstration single-object task. π0.5 carries the quantitative study: 143 individually logged rollouts on a harder, language-conditioned multitask setup, reported a controlled ablation that isolates where placement fails and why. SmolVLA is trained on the same 192 demonstrations as π0.5, but with its vision backbone frozen, and is included as a qualitative comparison point rather than a measured one.

Task setup

The workspace holds two placement targets — a larger cream plate and a smaller white bowl — and a set of coloured, differently-shaped objects. The policy is given a language instruction (e.g. “pick blue star and place on the cream coloured plate”) and must localize the correct object, grasp it, and place it on the correct target.

Two tasks of deliberately different difficulty:

• Easy (ACT): a single object, picked from a fixed location and placed on the plate at a fixed location.
• Hard (π0.5, SmolVLA): three objects (blue star, red circular block, purple cube), picked from varied locations, placed on either the plate or the bowl depending on the instruction. This demands object recognition, language grounding to the commanded object and target, and correct placement.

Results at a glance

ACT — trained from scratch on 50 demonstrations of the easy single-object task (purple cube → cream plate, both at fixed positions). Succeeds on 9 of 10 rollouts, establishing that the pipeline works end to end: teleoperated collection → training → deployment on the arm.

π0.5 — LoRA fine-tune of pi05_base on the 192-demonstration language-conditioned multitask set. Places on the cream plate in 58–70% of rollouts and generalizes to objects never seen in training, but never places on the white bowl (0/20): placement follows a memorized location rather than the visible target, which a relocation ablation confirms. Grasping proves geometry-dependent — the cylindrical red block is picked in only 3 of 23 rollouts despite an equal share of the training data.

SmolVLA — the same 192 demonstrations, but fine-tuned with the vision backbone frozen and only the action expert updated. Performs pick-and-place at times, with weak object-level language grounding — it will grasp an object other than the one named. Included as a qualitative comparison point; not yet quantified.

Scoring for the hard task: each rollout scores 0.5 for a successful pick + 0.5 for a successful place (max 1.0), logged individually with object, target, position and orientation. The quantitative results in the π0.5 section below come from the evaluation log, 143 recorded π0.5 rollouts and every figure is computed from it. ACT and SmolVLA are not part of this log: the ACT figure comes from a separate 10-rollout evaluation of the easy task, and SmolVLA has not been quantified yet.

π0.5 — detailed evaluation

Training data is balanced across both objects and targets: 3 objects × 2 targets × 32 demonstrations = 192 episodes / 51,801 recorded timesteps, i.e. 96 plate and 96 bowl episodes, and 64 episodes per object.

Fair comparison. Some conditions were evaluated at two object start positions (and two orientations); others at one. The tables below therefore report the common slice — object position 1, orientation 1 — so every row describes the same starting configuration. Full counts across all positions and orientations follow.

In-distribution tasks  (objects and targets both seen in training)

Generalization tasks  (objects absent from training; target seen in training)

Full record — all positions and orientations

Plate-relocation ablation  Result: 0/10

Test. Move the cream plate to a location unseen in training, issue the in-distribution instruction “pick purple cube and place on the cream coloured plate”, and re-run 10 episodes. The logic:

• If placement follows the relocated plate → the policy grounds the target visually.
• If placement still goes to the training location → placement relies on a position prior, with no visual grounding of the target.

Result: 0/10 placements on the relocated plate, with picking unaffected (8/10). Against the identical task with the plate at its training location — 7/10 pick, 6/10 place. The only thing that changed is the plate’s position, and placement went to zero. The arm continued to travel to the plate’s training location and released there. This resolves the ambiguity above: placement is driven by a learned position prior, with no visual grounding of the target. The plate-placement success in the in-distribution evaluation is therefore attributable to the plate sitting where it sat during training, not to the policy locating it.

Fix (future work): randomize target positions during data collection (including bowl-in-varied-locations demonstrations), forcing the policy to ground the target visually instead of memorizing its coordinates.

Grasping is geometry-dependent  3/23 picks

The red circular block is not short of data: it received 64 demonstrations, identical to the other two objects (32 per target). Yet it is grasped in only 3 of 23 rollouts (13%) — against 63% for the purple cube and 70% for the blue star. The policy approaches the object, so localization is not the problem; it fails to close a successful grasp.

Working hypothesis: a smooth cylindrical body offers fewer stable contact points and demands tighter gripper alignment than a flat-faced cube or a star with protrusions. So, a harder geometry might need more and higher quality data.

Generalization to novel objects

Two objects absent from the training set — a green object and a yellow object, in colours that appear in neither the demonstrations nor the instructions — were evaluated on the cream plate.

Both are picked and placed, at rates comparable to the in-distribution objects: green 6/10 pick and 6/10 place, yellow 5/10 and 5/10, against 6/10 place for the in-distribution purple cube at the same object position. Every successful grasp also produced a successful placement. So unseen object colour and shape are handled: the policy’s generalization gap is not about object appearance — it is specifically about target position, as the ablation above shows.

ACT — baseline  9/10

On the easy single-object, fixed-location task, ACT — trained from scratch on 50 demonstrations, on Apple Silicon, for 29k steps — succeeds on 9 of 10 rollouts — a clean baseline confirming the pipeline (data collection → training → on-arm deployment) and isolating the hard task’s difficulty as the object of study.

SmolVLA — qualitative  Eval pending

SmolVLA performs pick-and-place sometimes, but grounds the language instruction to the object poorly — asked for the blue star, it will grasp a different object in the scene.

The fine-tuning was done with freeze_vision_encoder = true and train_expert_only = true, so the SmolVLM2 backbone was held fixed and only the action expert was updated. The perception and language representations therefore never adapted to these particular objects — the policy learned how to move from the demonstrations, but not what this scene’s objects look like.

Next steps: (1) a quantified eval under the same 0.5/0.5 scoring, measuring the commanded-object-picked rate; (2) a retrain with the vision encoder unfrozen, which turns the explanation above into a verified test.

Method notes

• Data: teleoperated demonstrations collected with lerobot-record, published as the LeRobot dataset parn31/transformed_picknplace_multitask — 192 episodes / 51,801 timesteps, balanced across objects and targets.
• π0.5: fine-tuned with the openpi codebase. SmolVLA & ACT: trained with lerobot.
• Scoring: per-rollout, 0.5 (pick) + 0.5 (place); every episode logged with object, target, position/orientation, and outcome.

π0.5 fine-tuning setup

ACT and SmolVLA training setup

Both multitask policies were trained on the same 192 demonstrations. The two dataset entries differ only in storage format — the SmolVLA copy was converted to the LeRobot v3.0 layout — so the two runs see identical underlying data. (π0.5 additionally applies a delta-joint-action transform at load time, as listed in its setup above.)

Practical notes

Lessons from bringing imitation learning onto real hardware: camera resolution must be matched to the inference compute budget; teleoperation speed should be kept consistent with the target control frequency, or the policy sees a distribution shift and can stall at the initial states of a rollout; and demonstration coverage must be balanced across object positions.

Clips

Cite this page

If the results or the failure analysis on this page are useful in your own work, you are very welcome to cite it.

@misc{parnika2026so101,
  author       = {{Parnika}},
  title        = {Robot Learning --- Imitation Learning on the {SO-101} Arm},
  year         = {2026},
  howpublished = {\url{https://parnika31.github.io/robot-learning/}},
  note         = {Accessed: DD Month YYYY}
}

The double braces around {{Parnika}} are intentional: I have a single given name and no surname, and the braces stop BibTeX styles from splitting it or inventing an initial. Please cite it as Parnika, not Parnika, P. My ORCID iD is 0009-0008-3877-2416, which identifies me unambiguously regardless of how a given system formats the name.

Comments and feedback

Questions, corrections and suggestions are welcome — especially if you have run similar evaluations and reached different conclusions.