An open-set learning framework for EEG-based sleep stage classification.

OBJECTIVE: Existing sleep stage classification methods are typically developed under the closed-set assumption, where all sleep categories are predefined and known, thereby neglecting the presence of unknown sleep patterns in real-world electroencephalogram (EEG) analysis. This study aims to address this limitation by enabling the recognition of both known and unknown sleep stages. APPROACH: We propose a two-stage open-set learning framework for EEG-based sleep stage classification. In the first stage, an adaptive Gaussian method (AGM) is employed to model each known sleep category and compute Gaussian scores, while class-specific quantile-based adaptive thresholds are introduced to distinguish known and unknown samples. In the second stage, a trainable decision boundary OpenMax (TDBOM) module is designed to refine classification by optimizing decision boundaries during training. Samples with significant boundary deviations are suppressed, whereas normal samples are further calibrated using OpenMax to obtain reliable probability estimates. MAIN RESULTS: Experimental results on three public EEG datasets demonstrate that the proposed method achieves strong classification performance for known sleep stages while effectively detecting unknown categories, leading to a significant improvement in open-set recognition (OSR) performance. SIGNIFICANCE: This work provides a practical and robust solution for EEG-based sleep stage classification under open-set conditions, facilitating more reliable sleep analysis and supporting potential applications in realworld clinical monitoring and sleep disorder assessment.