Purdue researchers uncover blind spots at the intersection of AI and neuroscience

Purdue University researchers are doing work at the intersection of artificial intelligence and neuroscience. In this photo, a research participant is wearing an EEG cap with electrodes. (Image provided)

Findings debunk dozens of prominent published papers claiming to read minds with EEG

WEST LAFAYETTE, Ind. – Is it possible to read a person’s mind by analyzing the electric signals from the brain? The answer may be much more complex than most people think.

Purdue University researchers – working at the intersection of artificial intelligence and neuroscience – say a prominent dataset used to try to answer this question is confounded, and therefore many eye-popping findings that were based on this dataset and received high-profile recognition are false after all.

The Purdue team performed extensive tests over more than one year on the dataset, which looked at the brain activity of individuals taking part in a study where they looked at a series of images. Each individual wore a cap with dozens of electrodes while they viewed the images.

The Purdue team’s work is published in IEEE Transactions on Pattern Analysis and Machine Intelligence. The team received funding from the National Science Foundation.

“This measurement technique, known as electroencephalography or EEG, can provide information about brain activity that could, in principle, be used to read minds,” said Jeffrey Mark Siskind, professor of electrical and computer engineering in Purdue’s College of Engineering. “The problem is that they used EEG in a way that the dataset itself was contaminated. The study was conducted without randomizing the order of images, so the researchers were able to tell what image was being seen just by reading the timing and order information contained in EEG, instead of solving the real problem of decoding visual perception from the brain waves.”

The Purdue researchers originally began questioning the dataset when they could not obtain similar outcomes from their own tests. That’s when they started analyzing the previous results and determined that a lack of randomization contaminated the dataset.

“This is one of the challenges of working in cross-disciplinary research areas,” said Hari Bharadwaj, an assistant professor with a joint appointment in Purdue’s College of Engineering and College of Health and Human Sciences. “Important scientific questions often demand cross-disciplinary work. The catch is that, sometimes, researchers trained in one field are not aware of the common pitfalls that can occur when applying their ideas to another. In this case, the prior work seems to have suffered from a disconnect between AI/machine-learning scientists, and pitfalls that are well-known to neuroscientists.”

The Purdue team reviewed publications that used the dataset for tasks such as object classification, transfer learning and generation of images depicting human perception and thought using brain-derived representations measured through electroencephalograms (EEGs)

“The question of whether someone can read another person’s mind through electric brain activity is very valid,” said Ronnie Wilbur, a professor with a joint appointment in Purdue’s College of Health and Human Sciences and College of Liberal Arts. “Our research shows that a better approach is needed.”

Siskind is a well-known Purdue innovator and has worked on multiple patented technologies with the Purdue Research Foundation Office of Technology Commercialization. For more information on licensing and other opportunities with Purdue technologies, contact OTC at otcip@prf.org.

About Purdue Research Foundation Office of Technology Commercialization

The Purdue Research Foundation Office of Technology Commercialization operates one of the most comprehensive technology transfer programs among leading research universities in the U.S. Services provided by this office support the economic development initiatives of Purdue University and benefit the university's academic activities through commercializing, licensing and protecting Purdue intellectual property. The office recently moved into the Convergence Center for Innovation and Collaboration in Discovery Park District, adjacent to the Purdue campus. In fiscal year 2020, the office reported 148 deals finalized with 225 technologies signed, 408 disclosures received and 180 issued U.S. patents. The office is managed by the Purdue Research Foundation, which received the 2019 Innovation and Economic Prosperity Universities Award for Place from the Association of Public and Land-grant Universities. In 2020, IPWatchdog Institute ranked Purdue third nationally in startup creation and in the top 20 for patents. The Purdue Research Foundation is a private, nonprofit foundation created to advance the mission of Purdue University. Contact otcip@prf.org for more information.     

About Purdue University 

Purdue University is a top public research institution developing practical solutions to today’s toughest challenges. Ranked the No. 5 Most Innovative University in the United States by U.S. News & World Report, Purdue delivers world-changing research and out-of-this-world discovery. Committed to hands-on and online, real-world learning, Purdue offers a transformative education to all. Committed to affordability and accessibility, Purdue has frozen tuition and most fees at 2012-13 levels, enabling more students than ever to graduate debt-free. See how Purdue never stops in the persistent pursuit of the next giant leap at purdue.edu.

Writer: Chris Adam, cladam@prf.org 

Sources: Jeffrey Siskind, qobi@purdue.edu

Hari Bharadwaj, hbharadw@purdue.edu

Ronnie Wilbur, wilbur@purdue.edu

ABSTRACT

The Perils and Pitfalls of Block Design for EEG Classification Experiments

Ren Li, Jared S. Johansen, Hamad Ahmed, Thomas V. Ilyevsky, Ronnie B. Wilbur, Hari M. Bharadwaj and Jeffrey Mark Siskind

A recent paper claims to classify brain processing evoked in subjects watching ImageNet stimuli as measured with EEG and to employ a representation derived from this processing to construct a novel object classifier. That paper, together with a series of subsequent papers, claims to achieve successful results on a wide variety of computer-vision tasks, including object classification, transfer learning, and generation of images depicting human perception and thought using brain-derived representations measured through EEG. Our novel experiments and analyses demonstrate that their results crucially depend on the block design that they employ, where all stimuli of a given class are presented together, and fail with a rapid-event design, where stimuli of different classes are randomly intermixed. The block design leads to classification of arbitrary brain states based on block-level temporal correlations that are known to exist in all EEG data, rather than stimulus-related activity. Because every trial in their test sets comes from the same block as many trials in the corresponding training sets, their block design thus leads to classifying arbitrary temporal artifacts of the data instead of stimulus-related activity. This invalidates all subsequent analyses performed on this data in multiple published papers and calls into question all of the reported results. We further show that a novel object classifier constructed with a random codebook performs as well as or better than a novel object classifier constructed with the representation extracted from EEG data, suggesting that the performance of their classifier constructed with a representation extracted from EEG data does not benefit from the brain-derived representation. Together, our results illustrate the far-reaching implications of the temporal autocorrelations that exist in all neuroimaging data for classification experiments. Further, our results calibrate the underlying difficulty of the tasks involved and caution against overly optimistic, but incorrect, claims to the contrary.