Man-machine cooperation method and system based on multi-modal behavior online prediction

Abstract

The invention discloses a man-machine cooperation method and system based on multi-modal behavior online prediction. The method comprises the steps of obtaining video data; determining visual semantic deep features and human body posture features corresponding to human body behaviors of operators according to the video data, wherein the visual semantic deep features are used for reflecting time-space semantic information of the human body behaviors in a time-sequence visual mode; determining a target human body behavior intention corresponding to the operator according to the visual semantic deep features and the human body posture features; and determining an execution operation and a moving path corresponding to the mobile cooperative robot according to the target human body behavior intention. The problems that in the prior art, a manual assembling mode needs to consume a large amount of assembling time, and the manual assembling mode is difficult to adapt to a development mode that the life cycle is gradually shortened and product innovation is increasingly accelerated in an industrial technical system are solved. 本发明公开了一种基于多模态行为在线预测的人机协作方法和系统，所述方法包括：获取视频数据；根据所述视频数据，确定与作业人员的人体行为所对应的视觉语义深层特征和人体姿态特征；所述视觉语义深层特征用于反映所述人体行为在时序性视觉模式下的时空间语义信息；根据所述视觉语义深层特征和所述人体姿态特征，确定所述作业人员对应的目标人体行为意图；根据所述目标人体行为意图确定移动式协作机器人对应的执行操作和移动路径。解决了现有技术中手工装配模式需要耗费大量的装配时间，难以适应工业技术体系中生命周期逐渐缩短、产品创新日益加快的发展模式的问题。