Hearing Heartbeat from Voice: Towards Next Generation Voice-User Interfaces with Cardiac Sensing Functions

Voice user interfaces (VUIs) have been adopted in many IoT and mobile devices in daily life. VUIs provide a good user experience with lower-cost hardware (i.e., microphone) and higher throughput (compared with keyboard and touchscreen). Currently, identity authentication and receiving commands are the two most common interactions through VUIs, leaving physiological information in the voice unexploited. Recognizing this untapped potential, we propose VocalHR to extend VUIs beyond voice commands to heart activity sensing without additional hardware. VocalHR is built upon the voice-heart modulation effect, which is rooted in the cardiac activities' impacts on the behavior of the vocal organ during voice production. VocalHR captures voice features of cardiac activity in multiple voice organs and proposes a deep learning pipeline to transform features into cardiac activities. As this is the first study exploring voice-based heart activity sensing, we conducted extensive experiments on 43 demographically diverse subjects to verify the intrinsic link between voice and heart activities. On average, VocalHR can achieve less than 11.1% normalized sensing error on the heart event timing. Our further evaluation shows VocalHR is robust to different microphone specifications and varying speech rates.