Continuous Micro Finger Writing for AR/VR Interaction Using Vibration on a Commodity Smartwatch
Abstract:
Input is a significant problem for wearable devices, particularly for head-mounted virtual and augmented reality systems. Contemporary AR/VR systems use in-air gestures or handheld controllers for interactivity. However, mid-air handwriting provides a natural, subtle, and easy-to-use way to input commands and text. In this paper, we propose and investigate ViFin, a new technique for input commands and text entry, which tracks continuous micro finger-level writing with a commodity smartwatch through vibrations. Inspired by the recurrent neural aligner and transfer learning, ViFin recognizes continuous finger writing and works across different users and achieves an accuracy of 90\% and 91\% for recognizing numbers and letters, respectively. We quantify the accuracy of our approach through real-time system experiments in different arm positions, writing speeds, and smartwatch position displacement. Finally, a real-time writing system with two specific applications using AR smartglasses is implemented and assessed.
Committee:
- Brad Campbell (Chair)
- John Stankovic (Advisor)
- Madhur Behl
- Tariq Iqbal