Hands-on with augmented reality in remote classrooms

WEST LAFAYETTE, Ind. — During the COVID-19 pandemic, remote education has forced teachers to use every tool in their toolbox. But augmented reality, which allows both visual and hands-on instruction from any setting, remains out of reach for most educators.

A team led by Purdue University has built an app platform called MetaAR, enabling students and teachers to collaborate easily using augmented reality.

“Augmented reality is overlaying digital content onto the physical world,” said Karthik Ramani, Purdue’s Donald W. Feddersen Distinguished Professor of Mechanical Engineering. “It can be delivered on many different devices, from a phone and a tablet, to a head-mounted display. For example, you can hold your smartphone camera up to your living room and rearrange the furniture in 3D. Or you can play a video game, and the characters show up on your kitchen table.”

MetaAR is a platform that allows educators to create their own augmented reality apps and collaborate with students remotely. (Purdue University image/Convergence Design Lab) Download image

Ramani believes augmented reality (AR) is still in its infancy.

“We’ve seen many applications in entertainment and advertising, which basically show you some sort of action projected onto real-world settings,” he said. “But the key thing is for people to become part of the action. Augmented reality can see what you see. It can sense your hands and see how you are manipulating objects, which means AR can actually help you. It can learn from everyone, and everyone can learn from it. That’s why it’s ideal for education.”

The main roadblock, however, is authoring content. In order to use AR today, entertainment studios and big companies employ 3D artists, computer programmers and many other highly specialized software developers. Existing commercial platforms like Magic Leap and Microsoft HoloLens have not achieved mainstream availability. The cost and difficulty make it impossible for most educators to teach in AR. That’s why Ramani and his team have created MetaAR, an authoring platform that allows educators to create their own AR apps. It also allows students and teachers to collaborate remotely in AR.

“We want to make the creation of AR content as simple as creating a PowerPoint file and presenting it yourself,” Ramani said. “Using our technology platform, any instructor can easily train on MetaAR to create their own kits and lesson plans and start collaborating in a virtual classroom.”

For its test case, Ramani’s team built a simple STEM education kit: a small model of a city designed to teach the basics of circuit boards. Wooden pieces, like streets and buildings, had conductive material, which lit up streetlights if students assembled the “circuit” correctly. Using their MetaAR software, the researchers duplicated the pieces in 3D and formulated a simple lesson to teach students how to build the city. In the classroom, students looked at their desks through tablets, which showed them how to orient the physical pieces. It also alerted the students when they placed the physical pieces incorrectly.

MetaAR’s real magic, however, is its collaborative platform. If students had a question, they used video and images of their work to ask for help. Other students chimed in if they had a better solution and demonstrated those solutions using the virtual pieces. Instructors then integrated the students’ suggestions immediately and updated the AR content instantaneously.

“All of this happens in the cloud,” Ramani said, “which means that the teacher no longer has to physically look over the shoulder of 40 different students. Both the teacher and the students can be located anywhere and get the same hands-on education. The teacher can be present during the session, or students can learn on their own with the lesson the teacher has already authored. Students can learn at their own pace. And teachers can incorporate feedback, in real time, to improve the lab.”

Ramani was surprised at how quickly the students took to collaborating by AR. “It seems like this generation was born with this technology,” he said. “They gave us very good feedback of what works and what doesn’t work, which helped us make our software even better.”

MetaAR will be presented virtually on Tuesday (April 28) at the 2020 CHI Conference on Human Factors in Computing Systems, one of the leading conferences on human-computer interaction. Several other members of Ramani’s lab also will present advances in augmented reality, including StoryMakAR, which uses AR to help children create storytelling experiences. Other presentations are a study using AR “ghosts” to teach factory workers to perform machine tasks, and Vipo, a spatial-visual platform to program the interactions of mobile robots. Co-authors on this work are Ana Villanueva, Zhengzhe Zhu, Ziyi Liu and Thomas Redick from Purdue University and Kylie Peppler from the University of California, Irvine.

Ramani’s lab is working to bring the potential of augmented and virtual reality into many different real-world applications. “The strength of augmented reality is being able to capture an action in the real world and deliver that to people in the virtual world,” he said. “Imagine assembling something in your home. AR can show you exactly how to put it together and give you feedback on whether you’re doing it correctly, which is something paper manuals and even YouTube videos cannot do. If you’re in a factory, AR can show you exactly how to operate a piece of equipment and adapt to your actions. At the high end, doctors and nurses can train together to do robotic surgery. We can extend the capabilities of humans, using technology.”

As remote education becomes increasingly prevalent, Ramani believes that AR can serve as a vital tool, especially for STEM education.

“The earliest way we learn things is by seeing and doing,” he said. “Now we are able to scale that up and deliver that interactive experience to anyone, anywhere. Even if it’s virtual, we want hands-on education to be hands-on.”

The Purdue Research Foundation Office of Technology Commercialization has applied for several patents for technologies emerging from Convergence Design Lab.

This work is supported by a $2.5 million grant by the National Science Foundation's Future of Work at the Human-Technology Frontier program, one of 10 new Big Ideas for Future Investment. This work is also supported by a $1 million Phase I award from the Convergence Accelerator, a new capability within the NSF to accelerate user-inspired convergence research to reality in areas of national importance. Many corporations and other organizations are a part of the Convergence Design Lab efforts. 

About Purdue University

Purdue University is a top public research institution developing practical solutions to today’s toughest challenges. Ranked the No. 6 Most Innovative University in the United States by U.S. News & World Report, Purdue delivers world-changing research and out-of-this-world discovery. Committed to hands-on and online, real-world learning, Purdue offers a transformative education to all. Committed to affordability and accessibility, Purdue has frozen tuition and most fees at 2012-13 levels, enabling more students than ever to graduate debt-free. See how Purdue never stops in the persistent pursuit of the next giant leap at purdue.edu.

Media contact: Kayla Wiles, wiles5@purdue.edu. Working remotely but will provide immediate response.

Writer: Jared Pike

Source: Karthik Ramani, ramani@purdue.edu

Note to Journalists: A video showing a demonstration of MetaAR is available on YouTube. The video was produced by Jared Pike, communications specialist for Purdue’s School of Mechanical Engineering, using footage from the Convergence Design Lab. Photos of the MetaAR interface are available in a Google Drive folder at https://purdue.university/3eHiWiD.

ABSTRACT

Meta-AR-App: An Authoring Platform for Collaborative Augmented Reality in STEM Classrooms

Ana Villanueva, Zhengzhe Zhu, Ziyi Liu, Thomas Redick, Kylie Peppler, Karthik Ramani
Purdue University, West Lafayette, IN

Augmented Reality (AR) has become a valuable tool for education and training processes. Meanwhile, cloud-based technologies can foster collaboration and other interaction modalities to enhance learning. We combine the cloud capabilities with AR technologies to present Meta-AR-App, an authoring platform for collaborative AR, which enables authoring between instructors and students. Additionally, we introduce a new application of an established collaboration process, the pull-based development model, to enable sharing and retrieving of AR learning content. We customize this model and create two modalities of interaction for the classroom: local (student to student) and global (instructor to class) pull. Based on observations from our user studies, we organize a four-quadrant classroom model which implements our system: Work, Design, Collaboration, and Technology. Further, our system enables an iterative improvement workflow of the class content and enables synergistic collaboration that empowers students to be active agents in the learning process.