Reliable and calibrated diabetic retinopathy grading from Gaussian-filtered fundus images using attention and gated state-space feature mixing

Diabetic retinopathy (DR) is a leading cause of preventable vision loss, and automated severity grading is essential for scalable screening; however, existing deep-learning approaches are limited by inadequate long-range contextual modeling, severe class imbalance, and poorly calibrated predictions. To address these challenges, we propose ABSS-Net, a compact (5.47 M parameters) Attention-Based State Space Network that integrates an ImageNet–Noisy-Student pretrained EfficientNet-B0 multi-scale feature pyramid with hierarchical Convolutional Block Attention Modules (CBAM) and stacked gated State Space (SSM) blocks for efficient global feature mixing. A dual classification–reliability head is optimized using a hybrid objective combining class-weighted focal loss ( \(\gamma = 2.0\) ) and a Brier-style calibration penalty ( \(\lambda = 0.1\) ). Two experimental protocols on the Diabetic Retinopathy \(224\times 224\) Gaussian-filtered dataset (3,662 images, five ICDR classes) are reported. Under an 80/20 stratified development and test split, ABSS-Net achieves an accuracy of \(76.40\%\) , macro-F1 of 0.6107, QWK of 0.8563, AUC of 0.9235, and ECE of 0.0507. Additionally, statistical stability is evaluated using stratified 5-fold cross-validation, yielding \(70.26 \pm 4.20\%\) accuracy, \(0.5206 \pm 0.0553\) macro-F1, \(0.7933 \pm 0.0418\) QWK, \(0.8862 \pm 0.0168\) AUC, and \(0.1778 \pm 0.0439\) ECE, demonstrating that the architecture remains statistically stable across partitions. The self-reflective reliability head independently achieves high accuracy for high-confidence predictions ( \(c \ge 0.76\) ; \(94.27\%\) ) and lower accuracy for low-confidence predictions ( \(46.26\%\) ). A prediction–rejection analysis (AURC \(= 0.1050\) ) quantitatively supports a clinical-deferral workflow at \(81.00\%\) accuracy and \(70\%\) coverage. A Messidor-2 zero-shot external evaluation reveals a substantial domain shift, which is analyzed and identified as a direction for future work. The contribution of each architectural component and loss term is validated through ablation studies. ABSS-Net, leveraging local attention, long-range state-space modeling, and explicit calibration learning, delivers accurate, reliable, and explainable DR grading. This is further supported by Grad-CAM visualizations that align with clinically relevant retinal regions, supporting further investigation towards tele-ophthalmology screening, contingent on the external-domain validation discussed below.