RoboPilot: Generalizable Dynamic Robotic Manipulation with Dual-thinking Modes

Xinyi Liu1,2, Mohammadreza Fani Sani1, Zewei Zhou3, Julius Wirbel2, Bahram Zarrin1, Roberto Galeazzi2
1Microsoft, 2Technical University of Denmark, 3University of California, Los Angeles
Paper arXiv Code

Abstract

Despite rapid progress in autonomous robotics, executing complex or long-horizon tasks remains a fundamental challenge. Most current approaches follow an open-loop paradigm with limited reasoning and no feedback, resulting in poor robustness to environmental changes and severe error accumulation.

We present RoboPilot, a dual-thinking closed-loop framework for robotic manipulation that supports adaptive reasoning for complex tasks in real-world dynamic environments. RoboPilot leverages primitive actions for structured task planning and flexible action generation, while introducing feedback to enable replanning from dynamic changes and execution errors. Chain-of-Thought reasoning further enhances high-level task planning and guides low-level action generation. The system dynamically switches between fast and slow thinking to balance efficiency and accuracy.

To systematically evaluate the robustness of RoboPilot in diverse robot manipulation scenarios, we introduce RoboPilot-Bench, a benchmark spanning 21 tasks across 10 categories, including infeasible-task recognition and failure recovery. Experiments show that RoboPilot outperforms state-of-the-art baselines by 25.9% in task success rate, and the real-world deployment on an industrial robot further demonstrates its robustness in real-world settings.

Framework

RoboPilot Main Architecture

RoboPilot: The ModeSelector processes language instruction and visual information to select fast- or slow-thinking mode. The action generation module then orchestrates action primitives to solve the task, guided by the Chain-of-Thought reasoning module (only in slow-thinking mode). The execution monitor validates generated actions and tracks environment changes, providing closed-loop feedback for replanning.

Benchmark

To evaluate the performance of dynamic manipulation with a focus on robustness, we propose RoboPilot-Bench. This benchmark is composed of: (i) the Canonical Manipulation Suite, (ii) the Robustness Evaluation Suite, which is new, extended task suite with categories targeting robustness evaluation such as infeasible task recognition and error recovery tasks.

Canonical Manipulation Suite

13 tasks across 5 groups: Simple Manipulation, Spatial Allocation, Stable Stacking, Perceptual Matching and Spatial Reasoning.

Robustness Evaluation Suite

8 tasks across 5 groups: Conditional Reasoning, Sequential Planning, Feasibility Recognition, Linguistic Robustness and Error Recovery.

Results

RoboPilot Main Architecture

Qualitative Results: (a) Simple sequential reasoning task (Fast-Thinking Mode). (b) A spatial reasoning task (Slow-Thinking Mode). (c) An error recovery long-horizon task (Slow-Thinking Mode).

Spatial Allocation

Error Recovery

Spatial Reasoning & Perceptual Matching

Adaptibility to Dynamic Environment

BibTeX

@article{xinyi2025robopilot,
  author    = {Xinyi Liu, Mohammadreza Fani Sani, Zewei Zhou, Julius Wirbel, Bahram Zarrin, Roberto Galeazzi},
  title     = {RoboPilot: Generalizable Dynamic Robotic Manipulation with Dual-thinking Modes},
  journal   = {arXiv},
  year      = {2025},
}