Human-guided robot programming and learning for smart manufacturing : a mixed reality-assisted digital twin approach

Abstract

Traditional robot programming methods often lack intuitiveness and adaptability, failing to meet the growing demands for flexibility and efficiency in the mass person­alization paradigm. In addition, insufficient attention has been paid to exploring the role of human interaction in robot skill acquisition and the methodologies that support it. To address these challenges, this study presents a systematic approach built on mixed reality-assisted digital twin (MR-assisted DT) technology, enabling non-expert users to guide robot programming and skill learning effectively. Targeting human-in-the-loop (HITL) robotic assembly applications, the research tackles three key objectives: First, it develops a skill-based low-code robot programming system leveraging MR-assisted DT to enhance the intuitiveness and efficiency of human-robot interaction. Second, it proposes a scene-centric object manipulation approach enabling task intent expression through object’s DT manipulation, achieved via zero-shot segmentation in the MR environment. Lastly, it presents a large language model (LLM)–enhanced hybrid robot skill imitation approach combining human demonstrations with autonomous learning to boost the efficiency and generalization. This work provides an innovative solution from fundamental human-robot interac­tion (HRI) and control, to enhanced robot perception and skill execution, for HITL robotic assembly in complex and dynamic environments. Experimental results show significant improvements in the intuitiveness and efficiency of human-guided robot programming across perception, interaction, and execution dimensions. This thesis hopes to adopt an apprentice-like approach, enabling seamless robot programming through natural interactions and sustained skill imitations.