The challenge of out-of-distribution detection in motor imagery BCIs

Abstract Objective Machine Learning classifiers used in Brain-Computer Interfaces make classifications based on the distribution of examples on which they were trained. When exposed to EEG from unfamiliar classes they can only make blind guesses. Instead of allowing such guesses, these Out-of-Distribution (OOD) samples should be detected and rejected to improve the robustness of BCIs against unfamiliar cognitive states. Approach We study OOD detection in Motor Imagery BCIs by training a model on some classes and observing whether an unfamiliar movement class can be detected based on increased uncertainty. We tested seven different OOD detection methods and one more method that has been claimed to boost the quality of OOD detection. Main results For many BCI users, the uncertainty for the familiar in-distribution classes can still be higher than for the out-of-distribution classes, due to the high intrinsic variability inherent in EEG signals. As a result, many OOD detection methods that have shown good performance in other machine learning domains prove to be ineffective at identifying unfamiliar motor imagery patterns in BCIs. However, we also found that OOD detection performance is correlated with on-task performance, and that Deep Ensemble models and MC-Dropout models were able to achieve on-task AUROC > 0.9, and OOD detection ability up to 0.7. This shows that for models and subjects where task performance is high, rejecting unfamiliar cognitive states becomes feasible. Significance Our research demonstrates a Leave-One-Class-Out OOD detection setup as an experimental paradigm for studying whether models are robust against unfamiliar motor imagery actions that were not seen in the training data. This is the first benchmark in Motor Imagery BCI that evaluates this type of OOD data. Our results demonstrate how to improve the overall safety and reliability of BCIs by preventing erroneous actions