Egocentric human-object interaction detection : a new benchmark and method

Abstract

Egocentric human-object interaction (Ego-HOI) detection is crucial for intelligent agents to comprehend and assist human activities from a first-person perspective. However, progress has been hindered by the lack of dedicated benchmarks and methods robust to severe egocentric challenges like hand-object occlusion. This work bridges this gap through three key contributions. Firstly, we introduce Ego-HOIBench, a pioneering benchmark dataset derived from HOI4D for real-world Ego-HOI detection, comprising over 27K real images with explicit, fine-grained <hand, verb, object> triplet annotations. Secondly, we propose Hand Geometry and Interactivity Refinement (HGIR), a novel plug-and-play module that captures the structural geometry of hands to learn occlusion-robust, pose-aware interaction representations. Thirdly, comprehensive experiments demonstrate that HGIR significantly enhances Ego-HOI detection performance across various methods, achieving state-of-the-art results and laying a solid foundation for future research in egocentric vision. Project page:https://dengkunyuan.github.io/EgoHOIBench/