Few-shot temporal steering for robot action models
One temporal descriptor. One feature per task. Frozen model weights.
A shared descriptor of feature dynamics links offline feature selection with online decisions about when and how to steer.
Paper overview
Improving pretrained robot action models at test time from only a few rollouts remains challenging. Activation steering offers a lightweight approach by modifying feature activations while keeping model weights fixed. However, effective steering requires identifying which features to modify and determining the timing and strength of modification.
We introduce SteerAct, a few-shot temporal steering framework that uses a shared temporal descriptor of sparse-feature dynamics for offline feature selection and online control. A pretrained sparse autoencoder provides candidate features, and our temporal descriptor summarizes the distribution of local activation-order patterns for each feature, enabling comparisons across rollouts without requiring timestep alignment. Offline, we select a single feature per task whose temporal descriptor banks best separate successful and failed rollouts, and retain its success and failure reference banks. Online, the same descriptor guides intervention timing and the steering coefficient through comparisons with these references.
Using only 24 outcome-labeled rollouts per task, SteerAct improves task success by approximately 10% relative to unsteered models in evaluations spanning π0.5, Fast-WAM, RoboTwin 2.0, DexJoCo, and real-world manipulation, without updating action-model weights. Code and Demo can be found at: Link.
Keywords: Vision–language–action models, World–action models, Activation steering, Few-shot adaptation.
Method
A shared sparse autoencoder (SAE) provides features and steering directions. The descriptor summarizes local activation-order patterns, comparing histories without timestep alignment.
Build success and failure reference banks for each candidate feature. Select the feature whose temporal patterns best separate the outcomes, accounting for variation within each bank. Retain its decoder direction and reference banks for deployment.
The Support Gate checks whether the current history supports comparison; the Failure Gate checks whether it is closer to failure. When both gates are open, a bounded search chooses a signed coefficient whose locally predicted descriptor is closest to the success bank.
Real robot results
Three physical tasks with a frozen π0.5 backbone. Rates use 30 trials per task and method; each video pair shows an example failure and success.
Simulation results
Selected examples from nine RoboTwin 2.0 gripper tasks and nine DexJoCo dexterous-hand tasks, evaluated with π0.5 and Fast-WAM. Rates use 200 held-out rollouts per task and method.
Benchmark
How it works
Follow the feature history, two gates, and steering coefficient alongside the robot videos. 3× speed
Loading the rollout footage.
Unsteered timestep 0100 / 1200
Loading the rollout footage.
SteerAct timestep 0100 / 1200
Shared timeline
Both rollouts
Feature curves and the shaded reference band are illustrative, not measured activation distributions. The method compares temporal descriptors with fixed success and failure banks; the shaded band is not the bank itself.