University of Pittsburgh researchers have developed wearable technology based on Mobile Acoustic Field (MAF) to allow recognition of hand-to-face movements (gestures on or close to the face). This device, using bone conduction earphones and a lightweight deep neural network combined with signal processing, can detect hand gestures with a high degree of accuracy. MAF is a novel approach to hand-to-face gesture interactions which could revolutionize various industries.
Description
Sound, projected through bone conduction earphones, can produce surface acoustic waves (SAW) around the head. SAW travel through the facial region, dispersing in the air forming leaky surface acoustic waves (LSAW) forming a MAF that envelops the user’s face. This novel technology leverages the MAF to detect and recognize different hand gestures on or near the face.
Applications
- Interactive smart technology including virtual and extended reality
- Enhanced gaming experience or military uses for remote operation
- Face-touching awareness and prevention
Advantages
Conventional hand-to-face gesture recognition systems rely on Inertial Measurement Unit (IMU) sensors, mmWave sensors, cameras, and LIDAR sensors. These systems have several shortcomings, including only being accurate for gestures involving direct contact with the users’ face, inefficiency in low-lighting environments, or requiring large amounts of power to operate. Additionally, privacy concerns and high costs have limited the rollout of this technology.
MAF overcomes these challenges using commercially available bone conduction earphones in conjunction with a mobile device. This lightweight wearable solution could allow users to interact with the device seamlessly without restriction of movement, effortlessly enjoying hand-to-face gesture interactions without the need for additional equipment.
Invention Readiness
A prototype system has been developed. The system uses bone conduction earphones to emit a single tone in the ultrasound frequency band with a microphone attached to detect LSAW along the face. As part of an in-depth evaluation involving 22 participants, it was confirmed hand gestures generate a new signal by impacting how SAW propagate. In laboratory testing, it was found that MAF can recognize 10 hand gestures with up to 92% accuracy across 10 different gestures (4 on face, 6 over the face). The MAF was robust, and observations revealed facial hydration did not impact on sensitivity. Motion did impact sensitivity on the right side of the face while preserving sensitivity on the left where the microphone was positioned. Work is now required to optimize this device including recognition of micro-gestures.
IP Status
Patent Pending
