Please use this identifier to cite or link to this item:
http://hdl.handle.net/10397/92596
Title: | Toward proactive human-robot collaborative assembly : a multimodal transfer-learning-enabled action prediction approach | Authors: | Li, S Zheng, P Fan, J Wang, L |
Issue Date: | Aug-2022 | Source: | IEEE transactions on industrial electronics, Aug. 2022, v. 69, no. 8, p. 8579-8588 | Abstract: | Human-robot collaborative assembly (HRCA) is vital for achieving high-level flexible automation for mass personalization in today's smart factories. However, existing works in both industry and academia mainly focus on the adaptive robot planning, while seldom consider human operator's intentions in advance. Hence, it hinders the HRCA transition toward a proactive manner. To overcome the bottleneck, this article proposes a multimodal transfer-learning-enabled action prediction approach, serving as the prerequisite to ensure the proactive HRCA. First, a multimodal intelligence-based action recognition approach is proposed to predict ongoing human actions by leveraging the visual stream and skeleton stream with short-time input frames. Second, a transfer-learning-enabled model is adapted to transfer learnt knowledge from daily activities to industrial assembly operations rapidly for online operator intention analysis. Third, a dynamic decision-making mechanism, including robotic decision and motion control, is described to allow mobile robots to assist operators in a proactive manner. Finally, an aircraft bracket assembly task is demonstrated in the laboratory environment, and the comparative study result shows that the proposed approach outperforms other state-of-the-art ones for efficient action prediction. | Keywords: | Action recognition Human-robot collaboration Multimodal intelligence Transfer learning |
Publisher: | Institute of Electrical and Electronics Engineers | Journal: | IEEE transactions on industrial electronics | ISSN: | 0278-0046 | EISSN: | 1557-9948 | DOI: | 10.1109/TIE.2021.3105977 | Rights: | © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. The following publication S. Li, P. Zheng, J. Fan and L. Wang, "Toward Proactive Human–Robot Collaborative Assembly: A Multimodal Transfer-Learning-Enabled Action Prediction Approach," in IEEE Transactions on Industrial Electronics, vol. 69, no. 8, pp. 8579-8588, Aug. 2022 is available at https://dx.doi.org/10.1109/TIE.2021.3105977. |
Appears in Collections: | Journal/Magazine Article |
Files in This Item:
File | Description | Size | Format | |
---|---|---|---|---|
Li_Towards_Proactive_Human.pdf | Pre-Published version | 3.98 MB | Adobe PDF | View/Open |
Page views
75
Last Week
0
0
Last month
Citations as of May 5, 2024
Downloads
454
Citations as of May 5, 2024
SCOPUSTM
Citations
53
Citations as of Apr 26, 2024
WEB OF SCIENCETM
Citations
48
Citations as of May 2, 2024
Google ScholarTM
Check
Altmetric
Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.