September 1, 2009

Signals tells students how they're doing even before the test

Signals screen
Download photo
caption below

More than 11,000 students enrolled in 500-plus introductory "gateway" course sections will know by as early as the second week of classes whether their effort is putting them on a path to success or not. All they need to do is glance at the traffic signal-like red, yellow and green lights they'll see when they log into the course Web site on their computer or, later this year, on their cell phone.

Those signals - for which the system is named - are followed by messages from their instructor giving suggestions on how they can change their academic behavior to improve their grades by taking steps such as attending help sessions or reading additional materials.

Signals cell phone screen
Download photo
caption below

Tim Delworth, a continuing lecturer in mathematics, says Signals had a successful pilot in his classes.

"I give Signals an 'A,'" he says. "The student response has been very strong. Before, no one would e-mail me and say, 'I'm at 58 percent and I want to get to 72 percent, what do I need to do?' But the students who get a red light almost all contact me immediately to ask how to raise their grades.

"The information was there before, but it wasn't in front of them and it wasn't in a form that was as clear to them as a red or yellow warning signal."

To the students the red, yellow and green Signals message is intuitive and simple, but behind the uncomplicated interface lies a sophisticated data mining and analytics algorithm that checks more than 20 data points, focusing more on the student's effort than just their grades.

For example, the system might look at whether students have read online course materials assigned by the professor, done practice assignments, attended tutorial sessions after class or participated in online class discussions. This effort, combined with the student's quiz and test grades, allows the system to predict how well a student will do in the course by comparing the student's effort with work done by previous students.  

This information is delivered beginning in the second week of classes, often before the students have taken any tests.

In addition to the on-screen traffic lights, students also receive messages from the faculty member offering tips on improving performance or simply encouraging them to keep up the good work. The messages are delivered via the course Web site, through e-mail, or can be sent by text message or automated voicemail.

Signals was piloted in the 2006-07 and 2007-08 academic years with nearly 2,000 students, where a double-blind study found that 67 percent of students receiving a yellow or red warning improved their effort and grade. For students who received a red light only, 78 percent improved their grade and effort during the mid-term period.

Examples of the Signals system using fictional students can be found at https://www.itap.purdue.edu/tlt/signals/demo/

Ashley Lambrosa, a junior biochemistry major, says she was a top student in high school but was surprised by the difficulty of her college courses.

"I didn't know what I was getting myself into," she says. "College is not like high school. It moves fast, and if you're not careful your grades can slip. Once I started getting the Signals messages my grades improved. Without that I probably wouldn't have gone to the help sessions."

Lambrosa, like many students, found that the tips and suggestions offered by the Signals program helped her with other classes as well.

"Now I go to every help session my professors offer," she says.

Delworth says the early intervention benefits students before it's too late.

"Maybe they're only spending 15 minutes a day on their homework for this course when they need to spend an hour a day on it," he says. "They'll find out by the second week that their effort isn't enough for them to be successful, or they'll get a message saying 'Good job, keep up the good work.'"

Isaiah Johnson, a sophomore liberal arts major, says he was surprised when he saw a red light in his freshman math class.

"I didn't know I was doing that bad in that class, but I read the messages and they told me what to do. I had a D in the class but ended up with a B. Now I know how to study and prepare for my classes."

Delworth says education often has programs for students who excel and those who struggle, but rarely has programs for students in the middle.

"That's one of the nice things about Signals, because it's most effective for students in the broad middle," he says. "It takes students making D's and C's and helps them get C's and B's instead."

Kimberly Arnold, educational assessment and evaluation specialist for Information Technology at Purdue, says the program is most useful in the so-called STEM disciplines of science, technology, engineering and mathematics.

"Our early warning system is especially important for STEM fields. These courses have the most difficulties with retention," Arnold says. "Also, the structure of these courses tend to have good predictors, such as standardized test scores, or previous experience in this subject, and the courses are more likely to have links to outside resources or walk-in tutorial hours."

Arnold says the system appears to have a lasting effect.

"We found that the semester after the students aren't involved in the pilot study that they were still seeking resources and outside assistance at a rate higher than their peers."

The early warning system will be used primarily in freshman and sophomore courses.

"The predictive power of the model is most useful for freshmen," Arnold says. "When you get to upper-level courses the model loses its power for some reason, most likely because they've set their academic behaviors by that point."

Signals is based on the research of John Campbell, associate vice president for Purdue's Rosen Center for Advanced Computing. Campbell spent much of 2007 as a "scholar in residence" for EDUCAUSE, a national organization focused on the use of technology in higher education. His charge was to see whether classroom technologies helped students.

Campbell found that the online course management systems used by most classes at most universities contained unused data about how the students worked.

By data mining these systems and comparing the effort of individual students with how comparable students had done in the class before, the system could predict how successful a student would be. However, he also found that it was possible to use the technology to change students' behavior to make them more successful by making them aware of their status and the opportunities for improvement that were available.

"College is the first opportunity to be independent for most of these students," Campbell says. "What we should be doing is not expecting them to climb that mountain all alone, but instead we should offer a lot of scaffolding early in their college career that fades away as they become more experienced."

Diana Oblinger, president and CEO of EDUCAUSE, says, “Academic analytics combines the data colleges and universities have about students with statistical techniques and predictive modeling to help identify which students may need assistance to achieve their learning objectives. The goal is to improve student success."

Analytics, sometimes called business intelligence in the corporate world, allows businesses to mine data from their IT systems to learn more about customer behavior. Major retailers often use analytics to anticipate customer demand based on trends in prior purchasing.

According to Oblinger, similar techniques can be adapted for academic purposes.

"Understanding academic performance is full of complexities," she says. "Academic analytics is one tool that promises to help colleges and universities do even more to ensure student learning."

Arnold says that although now Signals is used to predict student success, upcoming versions will offer more specific recommendations based on the experience of fellow students.

"When you shop on Amazon, their site tells you that other people who bought the book you just purchased also bought this book, or Netflix suggests that you may like a certain movie based on the viewing habits of other people that are similar to yours," Arnold says. "We're building this capability into Signals. The system will say 'Students who had difficulty with this topic found this exercise helpful.'

"This is how online retailers use the ability of predictive analytics to encourage consumers to make more purchases, but we're using the same data mining and analytic techniques to help students become independent learners and achieve more."

Gerry McCartney, vice president for information technology and chief information officer at Purdue, says the Signals early warning system, like other classroom technologies in use at Purdue, is a direct outcome of the university's strategic plan.

"One of the key goals of the strategic plan, which is led by our president, France Córdova, is to enable students to be more successful," he says. "Nobody wants to see students failing. It is very expensive for the student, for the institution and for society. After years of testing we know that using these technologies mean fewer students fail, and that's a huge win for everyone."

Writer: Steve Tally, 765-494-9809. tally@purdue.edu

Sources: Tim Delworth, 765-494-0548, delworth@math.purdue.edu

Kimberly Arnold, 765-496-1088, kimarnold@purdue.edu

John Campbell, 765-494-1289, john-campbell@purdue.edu

Diana Oblinger, 919-306-4191, doblinger@educause.edu

Gerry McCartney, 765-496-2270, mccart@purdue.edu

Purdue News Service: (765) 494-2096; purduenews@purdue.edu

IMAGE CAPTION:
Purdue University's Signals student success program offers freshmen Boilermakers an academic dashboard where they can see at a glance how they are doing in certain courses. Students also receive automated messages from their instructors giving them tips on improving their grades or simply encouraging them to keep up the good work. (Purdue University image/Alex Kingman)

A publication-quality image is available at https://www.purdue.edu/uns/images/+2009/signals-screen.jpg 

IMAGE CAPTION:
Purdue University's Signals student success program lets beginning Boilermakers know how they are doing in certain classes, even on a cell phone. Students also receive e-mail or text messages from their instructors with tips on improving their grades in the classes. (Purdue University image/Alex Kingman)

A publication-quality image is available at https://www.purdue.edu/uns/images/+2009/signals-cellphone.jpg

Notes to Journalists: To arrange an interview with students Ashley Lambrosa or Isaiah Johnson, please contact Steve Tally at tally@purdue.edu. Images of the student "dashboard" of classes for both the computer and iPhone are available.

To the News Service home page