Active Multi-Modal Approach for Enhanced User Recognition in Social Robots

The field of Human-Robot Interaction (HRI) is swiftly expanding, driven by notable advancements in artificial intelligence (AI). Humanoid robots, now capable of being equipped with advanced AI models, are being considered for a wide array of applications due to their ability to perform complex tasks and interact with humans in both natural and intelligent ways. In the domain of social robotics, the capability to selectively interact with authorized users is crucial for ensuring security and providing personalized user experiences. Unimodal user recognition methods, such as audio-based user recognition, are commonly used in social robots. However, these methods can be susceptible to ambient noise and might exhibit reduced accuracy. Although face recognition modality is often employed to enhance accuracy, the majority of audio-visual person recognition methods are trained on datasets with only a single user in the frame.This paper introduces a method for audio-visual user recognition with multiple users in the frame, utilizing an additional sound localization modality. The proposed method is evaluated using a dataset created from interactions between the social robot Pepper and multiple users. The results demonstrated that the proposed method significantly outperformed unimodal user recognition methods.