AR in hand: Egocentric palm pose tracking and gesture recognition for augmented reality applications

Wearable devices such as Microsoft Hololens and Google glass are highly popular in recent years. As traditional input hardware is dificult to use on such platforms, vision-based hand pose tracking and gesture control techniques are more suitable alternatives. This demo shows the possibility to interact with 3D contents with bare hands on wearable de-vices by two Augmented Reality applications, including vir-tual teapot manipulation and fountain animation in hand. Technically, we use a head-mounted depth camera to cap-ture the RGB-D images from egocentric view, and adopt the random forest to regress for the palm pose and classify the hand gesture simultaneously via a spatial-voting framework. The predicted pose and gesture are used to render the 3D virtual objects, which are overlaid onto the hand region in input RGB images with camera calibration parameters for seamless virtual and real scene synthesis.