July 2, 2018
Purdue research: Forget the cell phone screen, a forearm is all you'll need to read
Postdoctoral student Yang Jiao communicates words to Jaeong Jung, an undergraduate student, using phoneme signals transmitted to the haptic device on his forearm. (Purdue University photo/Brian Huchel)
Download image
WEST LAFAYETTE, Ind. — Communication could step beyond reading a cellular phone screen with a new technique by Purdue College of Engineering researchers to learn and read messages through a person’s sense of touch.
Hong Tan, a professor of electrical and computer engineering, was the lead professor on the haptic research that developed a method to receive messages by learning to interpret signals such as a buzzing sensation and others through the skin on the forearm.
The research results were presented Friday (June 15) at the Proceedings of EuroHaptics 2018 conference in Pisa, Italy.
The yearlong project was done with researchers from the Massachusetts Institute of Technology and Facebook Inc.
Tan, founder and director of Purdue’s Haptic Interface Research Laboratory, said while the research lends itself to being used by hearing-impaired and visually impaired users, the study is also being looked at on general terms for any number of possible uses.
“We are collaborating with Facebook through the company’s Sponsored Academic Research Agreement. Facebook is interested in developing new platforms for communication and the haptic research we are doing has been promising,” she said.
“I’m excited about this … imagine a future where you’re able to wear a sleeve that discreetly sends messages to you – through your skin – in times when it may be inconvenient to look at a text message,” Tan said. “I’m really hoping this takes off as a general idea for a new way to communicate.
“When that happens, the hearing-impaired, the visually-impaired, everyone can benefit.”
In the study, subjects used a material cuff encircling the forearm from the wrist to below the elbow. The instrument, wrapped around the test subject’s non-dominant arm, featured 24 tactors that, when stimulated, emitted a vibration against the skin, changing quality and position in the process.
Tan said the 39 phonemes (units of sound in a language that distinguish one word from another) in the English language were mapped using signals from specific tactors. The sounds of consonants such as K, P and T were stationary sensations on different areas of the arm while vowels were indicated by stimulations that moved up, down or around the forearm.
For video on vowel sensations, go to https://www.youtube.com/watch?v=CYfqcdnvMyE&feature=youtu.be.
“We used anything that can help you establish the mapping and recognize and memorize it,” Tan said. “This is based on a better understanding of how to transmit information through the sense of touch in a more intuitive and effective manner.”
Twelve subjects learned haptic symbols through the phoneme-based method at a schedule of 100 words in 100 minutes for the research, while 12 others learned using a word-based system with the haptic signals on their arm.
Research results showed the phonemes worked better than a word-based approach by providing a more consistent path for user learning in a shorter period of time. Performance levels varied greatly among the test subjects for each method, but with the phoneme-based approach at least half could perform at 80 percent accuracy while two subjects reached 90 percent accuracy.
Tan said using phonemes was more efficient compared to letters, noting there are less phonemes in a word compared with the number of letters.
She said the reasoning behind the project is the thought that there are many ways to encode speech into feelings. The project goal is to demonstrate one way that actually works.
“For this research, the learning progress is one of the key things,” Tan said. “With this, not only do we have a system that works, but we’re able to train people within hours rather than months or even years.”
For the study, a concentrated, intense training regimen was developed where the test subjects worked for about 10 minutes a day. They were then tested on their progress.
“It is more efficient than if they sit here for three hours to study,” Tan said. “You can’t keep good concentration for that long.”
“People who are hearing-impaired may be motivated to spend additional time to training themselves, but the general public probably doesn’t have the patience,” she added.
The research was funded by Facebook Inc.
Writer: Brian L. Huchel, 765-494-2084, bhuchel@purdue.edu
Source: Hong Tan, 765-494-6416, hongtan@purdue.edu
Note to Journalists: Purdue researchers presented “A Comparative Study of Phoneme- and Word-Based Learning of English Words Presented to the Skin” on Friday (June 15) and a hands-on demonstration of the system on Saturday (June 16) at the Proceedings of EuroHaptics 2018 conference in Pisa, Italy. Video of the haptic sleeve research is available at https://goo.gl/mpZvLy
ABSTRACT
A Comparative Study of Phoneme- and Word-Based Learning of English Words Presented to the Skin
Yang Jiao1, Frederico M. Severgnini1, Juan S. Martinez1, Jaehong Jung1, Hong Z. Tan1,
Charlotte M. Reed2, E. Courtenay Wilson2, Frances Lau3, Ali Israr3, Robert Turcott3, Keith Klumb3 and Freddy Abnousi3
1 Haptic Interface Research Lab, Purdue University
2 Research Lab of Electronics, Massachusetts Institute of Technology
3 Facebook Inc.
Past research has demonstrated that speech communication on the skin is entirely achievable. However, there is still no definitive conclusion on the best training method that minimizes the time it takes for users to reach a prescribed performance level with a speech communication device. The present study reports the design and testing of two learning approaches with a system that translates English phonemes to haptic stimulation patterns (haptic symbols). With the phoneme-based learning approach, users learned the haptic symbols associated with the phonemes before attempting to acquire words made up of the phonemes. With the word-based approach, users learned words on day one. Two experiments were conducted with the two learning approaches, each employing twelve participants who spent 100 minutes each learning 100 English words made up of 39 phonemes. Results in terms of the total number of words learned show that performance levels vary greatly among the individuals tested (with the best learners in both methods achieving word-recognition scores >90 percent-correct on a 100-word vocabulary), both approaches are feasible for successful acquisition of word through the skin, and the phoneme-based approach provides a more consistent path for learning across users in a shorter period of time.
