Video overview
4 min 49 sec · English narration
Training-freeSuccess rate
Abstract
Embodied navigation requires an agent to interpret visual observations, accumulate spatial knowledge, and execute actions over long horizons. Existing training-free methods benefit from multimodal large language models, but their task-specific pipelines often lack explicit mechanisms to reconcile proposals with spatial evidence, task progress, and execution failures.
HarnessVLN introduces an Agent Harness that coordinates perception, retrieval, grounding, navigation, recovery, and termination through a unified tool interface. Hierarchical event memory tracks progress and execution history, while a persistent Spatiotemporal Graph organizes reusable evidence for target verification and recovery.
Method
Framework overview
The multimodal language model proposes operations. The Agent Harness assembles context, validates proposals, dispatches tools, and integrates execution feedback.
Hierarchical event memory
Working, progress, and reflection memory represent the current task context, completed subgoals, and execution failures.
Spatiotemporal graph
Places, entities, navigable relations, evidence provenance, and failure annotations persist across subgoals.
Validated execution
Operation proposals are validated against available evidence, geometric feasibility, task consistency, and prior failures.
Method
Unified tool interface
Eight tools provide a common interface for perception, retrieval, grounding, navigation, recovery, and termination.
01observePerception
Capture current RGB-D and pose.
02observe_panoramaPerception
Capture predefined headings.
03retrieve_memoryRetrieval
Recall task-relevant evidence.
04ground_targetGrounding
Map semantics to an image region.
05query_depthPerception
Estimate target depth and uncertainty.
06navigate_toNavigation
Execute a validated spatial target.
07backtrackRecovery
Return to a verified visited location.
08request_stopTermination
Verify completion before stopping.
Evaluation
Quantitative results
Training-free SR60.8%
R2R val-unseen
| Method | R2R | RxR | ||||
|---|---|---|---|---|---|---|
| OSR ↑ | SR ↑ | SPL ↑ | SR ↑ | SPL ↑ | nDTW ↑ | |
| CA-Nav | 48.0 | 25.3 | 10.8 | 19.0 | 6.0 | 13.5 |
| GC-VLN | 41.8 | 33.6 | 16.3 | 33.8 | 13.8 | — |
| HSGM | 58.7 | 47.9 | 32.8 | 41.8 | 25.1 | 54.9 |
| AgenticNav-GPT-5.5 | 65.0 | 55.0 | 48.4 | — | — | — |
| HarnessVLN-GPT-5.5Ours | 72.7 | 60.8 | 43.5 | 53.9 | 38.0 | 54.8 |
All metrics are percentages. The page shows selected training-free baselines; see the paper for the complete comparison.
Training-free SR76.0%
HM3D-v2
| Method | HM3D-v2 | HM3D-OVON | ||
|---|---|---|---|---|
| SR ↑ | SPL ↑ | SR ↑ | SPL ↑ | |
| InstructNav | 58.0 | 20.9 | — | — |
| VLFM | 63.6 | 32.5 | 38.5 | 22.2 |
| STEGNav | 69.4 | 28.2 | — | — |
| DRIVE-Nav | 72.4 | 41.3 | 50.2 | 32.6 |
| MSGNav | 74.4 | 33.4 | 48.3 | 27.0 |
| HarnessVLNOurs | 76.0 | 37.9 | 59.3 | 36.6 |
All listed methods are training-free. See the paper for training-based comparisons and full protocol details.
Component ablation
Effect of individual components
Cumulative ablations evaluate the contribution of event memory, graph retrieval, and stop validation on fixed 100-episode subsets.
R2R success rate (%) · fixed 100-episode subset
Qualitative results
Simulation demonstrations
Recorded trajectories show egocentric observations, spatial maps, and task-state updates on four navigation benchmarks.
R2R01 / 04 · 30 s
Instruction following with stair traversal
The agent ascends the stairs and enters the first doorway on the right.
RxR02 / 04 · 31 s
Multi-stage instruction following
The agent follows an instruction involving an archway, a change of direction, and a dining-table landmark.
HM3D-v203 / 04 · 45 s
Object-goal navigation: bed
An extended search trajectory culminating in successful navigation to a bed.
HM3D-OVON04 / 04 · 30 s
Open-vocabulary navigation: picture
The agent explores the scene and navigates to a picture specified as the target.
Selected successful episodes from the September 9 demo set. Playback is temporally condensed; the HM3D-v2 clip also uses uniform frame subsampling. These examples illustrate behavior and do not constitute an aggregate evaluation.
Robot experiments
Real-world deployment
HarnessVLN is deployed on a 1.74 m AgiBot A3U humanoid equipped with stereo vision, 3D LiDAR, onboard NVIDIA Thor compute, and a local path planner.
01 / 03Combined navigation task
Stop at the entrance signboard, then continue to the vending machine through one shared Harness protocol.
BibTeX
@misc{harnessvln2026,
title={HarnessVLN: Unifying Training-Free Embodied Navigation through an Agent Harness},
author={Anonymous},
note={Manuscript under review},
year={2026}
}