HARP-VLAHuman-Robot Aligned Representation Learning
for Vision-Language-Action Model

Xiang Zhu^1,2,*Puzhen Yuan^1,*Yichen Liu^1,*Jianyu Chen^1,2,†

¹ Institute for Interdisciplinary Information Sciences, Tsinghua University
² Shanghai Qi Zhi Institute

* Equal contribution † Corresponding author

Align vision. Align action. Learn from human videos.

HARP reduces the human-robot visual gap with a robot-only adapter and aligns latent actions across embodiments. — Paired demonstrations bridge embodiments. Unpaired videos provide scalable dynamics supervision.

Abstract

Human videos offer a rich source of manipulation knowledge, but visual and action differences make that knowledge difficult to transfer to robots. HARP jointly aligns human–robot visual representations and latent actions. A lightweight robot-only adapter anchors robot features to human semantics, while self- and cross-prediction learn shared motion representations from paired and unpaired videos. Manipulation-centric auxiliary cues and a source-relative pair-discriminative objective improve alignment while preserving distinctions between demonstrations. The aligned encoder and latent action model then supervise VLA pretraining, followed by a lightweight action head for robot control.

Experiments show stronger cross-embodiment alignment and downstream manipulation performance, reaching 4.481 average sequence length on CALVIN ABC→D and 76.3% real-world success—a 7.1 percentage-point improvement over the strongest evaluated baseline.

Method Overview

01 / Method

A shared representation for humans and robots

A three-stage framework connects cross-embodiment alignment to executable robot actions.

Align

Jointly learn visual representations and latent actions.

Pretrain

Learn a policy from aligned latent-action supervision.

Act

Ground the policy in continuous robot action chunks.

Paired and unpaired video transitions enter a frozen vision backbone with robot-only adapters. Self- and cross-prediction, shared-cue losses and SRPD alignment train a latent action model. — Human features remain anchored to the frozen encoder. A robot-only adapter aligns robot features, while the latent action model learns to predict future representations within and across embodiments. The source-relative term improves alignment over the original robot features; the pair-discriminative term preserves meaningful correspondence.

Stage 2 uses aligned vision features and latent-action labels to pretrain the VLA. Stage 3 adds proprioception and a real-action head. — The learned vision encoder and latent-action labels provide a common interface for human and robot video pretraining. A continuous action head grounds the resulting policy in ten-step robot action chunks.

02 / Representation analysis

Closer across embodiments. Distinct across tasks.

Alignment is measured on the same held-out paired demonstrations, with identical sampling, pooling and similarity metrics.

UMAP projections from the paper show improved alignment of human circles and robot triangles in visual and latent-action spaces. — Visual and latent-action representations before and after alignment. Original UMAP visualization from Figure 4; colored pairs highlight corresponding demonstrations.

Cross-embodiment retrieval

Average Recall@1 rises from 43.55% to 78.50%. HARP improves both geometric alignment and discrimination between paired demonstrations.

Average retrieval R@1: Unadapted 43.55%, HR 45.16%, HARP-HR 53.23%, HARP-L2 61.83%, HARP-SR 74.74%, HARP-SRPD 78.50%. — Average of human-to-robot and robot-to-human Recall@1. Higher is better. Table 1.

View full results · Table 1 · Bidirectional retrieval

Table 1 · Bidirectional retrieval
Method	Human → robot	Robot → human	Average R@1
Unadapted	44.09	43.01	43.55
HR	45.16	45.16	45.16
HARP-HR	46.24	60.22	53.23
HARP-L2	70.97	52.69	61.83
HARP-SR	84.95	64.52	74.74
HARP-SRPD	87.10	69.89	78.50

03 / Ablations & scaling

Complementary signals make alignment stronger

Self-prediction, cross-prediction, unpaired videos and manipulation-centric auxiliary cues each contribute to the learned representation.

Full HARP leads all ablated variants in visual retrieval, latent-action retrieval and code-agreement gap. — Controlled Stage-1 ablations. Removing one source of supervision degrades correspondence. Table 2.

View full results · Table 2 · Component ablations

Table 2 · Component ablations
Variant	Visual R@1	Latent R@1	Code-agreement gap
Full HARP	78.50	36.29	30.67
Self-only	73.93	26.62	22.24
Cross-only	75.10	28.03	25.16
Paired-only	75.16	32.06	30.04
Without auxiliary cues	73.37	34.07	26.71

Strong performance with less paired supervision

With 5% of the paired data, HARP retains 72.95% visual Recall@1 and a CALVIN average length of 4.214. Unpaired data and optimization budget are held fixed.

Paired data scaling at 5%, 25% and 100% shows CALVIN average length 4.214, 4.354 and 4.481. — Paired supervision scales from approximately 1.2M to 24M processed frames. Table 3.

View full results · Table 3 · Paired-data scaling

Table 3 · Paired-data scaling
Paired data	Visual R@1	Latent R@1	RLBench	CALVIN
100% (~24M frames)	78.50	36.29	46.59	4.481
25% (~6M frames)	77.36	28.43	42.10	4.354
5% (~1.2M frames)	72.95	24.40	40.70	4.214

04 / Visual representation transfer

Better features for robot policy learning

With frozen visual encoders and a shared downstream policy setup, HARP-SRPD achieves 46.59% average success across 18 RLBench tasks, compared with 37.56% for the unadapted encoder.

RLBench average success increases from 37.56% with unadapted features to 46.59% with HARP-SRPD. — Same policy architecture, data, action space and optimization budget. Table 4.

View full results · Table 4 · Frozen-encoder RLBench results

Table 4 · Frozen-encoder RLBench results
Method	Success rate (%)
Unadapted	37.56
HR	39.70
HR-Style	38.22
HARP-HR	35.11
HARP-HR-Style	40.07
HARP-L2	40.78
HARP-SR	43.41
HARP-SRPD	46.59

05 / Policy evaluation

CALVIN · Generalization across environments

Long-horizon manipulation in the ABC→D setting.

4.481Average sequence length

75.9%Five-task success rate

10Actions per prediction

CALVIN sequence success rates for eight evaluated models. HARP-VLA achieves 99.8, 96.7, 91.3, 84.4 and 75.9 percent for one through five consecutive tasks. — Success rate for completing 1–5 consecutive tasks. System-level comparisons across evaluated policies. Table 6.

View full results · Table 6 · CALVIN ABC→D

Table 6 · CALVIN ABC→D
Model	1 task	2 tasks	3 tasks	4 tasks	5 tasks	Average length
π₀	92.3	82.4	72.1	62.2	53.7	3.627
π₀.₅	94.4	86.0	76.4	69.7	61.0	3.875
OpenVLA	91.3	77.8	62.0	52.1	43.5	3.270
UniVLA	95.4	85.5	75.4	66.9	56.5	3.800
OpenVLA-OFT	94.2	86.4	78.0	70.4	62.7	3.917
HARP-VLA (L2)	95.8	89.7	81.3	72.8	64.8	4.044
HARP-VLA (w/o freezing)	98.8	93.9	86.1	77.7	68.5	4.250
HARP-VLA	99.8	96.7	91.3	84.4	75.9	4.481

Five sequences, five steps each

Read each row from left to right: five consecutive successful subtasks in each recorded sequence.

Swipe to view all five steps →

Sequence 1 · Step 1Lift pink block slider

Sequence 1 · Step 2Place in slider

Sequence 1 · Step 3Open drawer

Sequence 1 · Step 4Rotate red block right

Sequence 1 · Step 5Lift red block table

Sequence 2 · Step 1Open drawer

Sequence 2 · Step 2Lift pink block table

Sequence 2 · Step 3Place in slider

Sequence 2 · Step 4Turn on lightbulb

Sequence 2 · Step 5Rotate blue block left

Sequence 3 · Step 1Push blue block left

Sequence 3 · Step 2Close drawer

Sequence 3 · Step 3Lift red block slider

Sequence 3 · Step 4Place in slider

Sequence 3 · Step 5Turn off lightbulb

Sequence 4 · Step 1Push pink block right

Sequence 4 · Step 2Lift pink block table

Sequence 4 · Step 3Place in slider

Sequence 4 · Step 4Push into drawer

Sequence 4 · Step 5Close drawer

Sequence 5 · Step 1Open drawer

Sequence 5 · Step 2Push red block right

Sequence 5 · Step 3Move slider left

Sequence 5 · Step 4Lift pink block slider

Sequence 5 · Step 5Place in slider

Representative recorded rollouts. The gallery is separate from the aggregate benchmark evaluation.

06 / Real-world manipulation

From human videos to dexterous robot skills

Xarm7 with a Robotera Xhand, evaluated on four tabletop manipulation tasks with 60 trials per task.

76.3%Average success rate

+7.1 ppOver the strongest evaluated baseline

Real-world success for pi0.5, OpenVLA-OFT and HARP-VLA on Pick, Push, Press and Flip. HARP-VLA scores 76.7%, 81.7%, 85.0% and 61.7%. — Three reference policies shown for readability; all eight evaluated models are available below. Table 5.

View full results · Table 5 · Real-world success rates (%)

Table 5 · Real-world success rates (%)
Model	Pick	Push	Press	Flip	Average
π₀	58.3	75.0	56.7	35.0	56.3
π₀.₅	71.7	83.3	68.3	53.3	69.2
OpenVLA	0.0	23.3	18.3	0.0	10.4
UniVLA	38.3	61.7	31.7	21.7	38.4
OpenVLA-OFT	51.7	71.7	76.7	43.3	60.9
HARP-VLA (L2)	70.0	71.7	81.7	56.7	70.0
HARP-VLA (w/o freezing)	76.7	80.0	78.3	58.3	73.3
HARP-VLA	76.7	81.7	85.0	61.7	76.3

Backbones, pretraining data and visual inputs differ across systems; these are system-level comparisons.

Real-world demonstrations

Three demonstrations per task. Switch between synchronized third-person and wrist-camera views.

Pick and Place

Bamboo · 01

Orange · 02

Yellow box · 03

Push Box

Red box · 01

Green box · 02

Red box · 03

Press Button

Button · 01

Button · 02

Button · 03

Flip Cup

Shaker cup · 01

Blue cup · 02

Blue cup · 03

Looking ahead

HARP shows how a small paired bridge, combined with unpaired video dynamics, can improve learning from human demonstrations. Current limitations include the cost of paired data, sensitivity to trajectory tracking and viewpoint differences, and evaluation on a single real-world tabletop platform. Future directions include more embodiments, longer-horizon tasks and bimanual manipulation.

Citation

@article{zhu2026harp,
  title={HARP-VLA: Human-Robot Aligned Representation Learning for Vision-Language-Action Model},
  author={Zhu, Xiang and Yuan, Puzhen and Liu, Yichen and Chen, Jianyu},
  journal={arXiv preprint arXiv:2605.31234},
  year={2026}
}