Listening to Your Fingertips

The future of precise hearing could be at your fingertips.Carnegie Mellon University researchers have developed a technique that turns a person's fingertips into tiny microphones to better sense what someone is doing.Daehwa Kim, a Ph.D. student in the School of Computer Science's Human-Computer Interaction Institute (HCII), said the new technique, called SoundBubble, helps isolate and amplify the often subtle sounds of fingers as they tap, trace or rap various surfaces, or use different hand tools. That information could, in turn, give computers important contextual hints about what users are doing and how the computer might help them."It's a way to sense human manipulation using sound," Kim said.SCS researchers have developed a technique that turns a person's fingertips into tiny microphones that give computers important contextual hints about what users are doing and how technology might help them.If a person is using artificial intelligence to guide them in assembling furniture, for example, SoundBubble could help the AI sense when the user has finished using a tool or has completed a step before it moves on to the next one. Or people using SoundBubble could input data or control a device by tapping or swiping over a table, a wall or even another body part.Using microphones to sense what people are doing with their hands isn't new. Researchers including HCII Associate Professor Chris Harrison, who leads the Future Interfaces Group where SoundBubble was developed, have used rings, wristbands and watches to get microphones as close to the hand as possible and minimize the background noises that obscure finger sounds.With SoundBubble, however, Kim and Harrison entirely eliminated the need to position microphones on a person's hands or arms. Instead, they used the microphone array built into augmented reality headsets or smart glasses. A camera in the headset locates the finger or fingers that need to be monitored, and SoundBubble creates virtual microphones on those fingers using a signal processing technique known as beamforming.Beamforming leverages the interference patterns created by sound waves as they interact with each other to amplify or cancel sound. Waveforms in phase with each other amplify sound, while those out of phase can cancel each other out. SoundBubble processes signals from multiple microphones to manipulate this effect so sounds from the fingertips are amplified, while sounds from elsewhere are minimized.Beamforming is a tried-and-true technique, originally used to detect the sound of enemy aircraft and later applied to radio frequency and microwave radiation. It can be used both in receiving and transmitting signals."It's only recently that people have started to think about having beamforming arrays on wearables, such as smart glasses," Harrison said.Kim said the team tested SoundBubble in noisy environments, including cafés and places where loud music was playing. In each case, the user could clearly hear sounds from SoundBubble with little interference from the environment."Our bubble is actually a focal point in space where only sounds from that area are amplified," Kim added.In addition to contextual sensing, SoundBubble could also be used to control a computer by tapping or drawing on a work surface. Kim also sees potential for it in prototyping. Designers testing a new game controller, for instance, might adjust the position of buttons and switches by easily moving ridged stickers on a mock controller.The number and size of SoundBubbles can be adjusted as necessary for each application, as all of them use the same sound inputs but are created through software.Kim and Harrison presented their work at last month's Association for Computing Machinery Conference on Human Factors in Computing Systems (CHI 2026) in Barcelona. For more on their research, read their paper.

Raccontata dacs.cmu.edu