Alok Kumar, N. Mahendran
ABSTRACT Lesion detection and segmentation are essential yet complex tasks in medical image analysis due to the substantial variability in lesion shape, size, contrast, and anatomical location across different organs. Existing deep learning methods often lack adaptability, as they are typically designed for specific organs or imaging modalities, leading to limited generalization when applied to diverse datasets. To address this limitation, this study introduces a unified and generalizable framework capable of accurate multi‐organ lesion detection, localization, and segmentation across heterogeneous medical imaging data. The proposed U‐VQVAE‐CTLesionNet integrates a U‐Net–based encoder–decoder architecture for spatial feature extraction with a Vector Quantized Variational Autoencoder (VQVAE) module that discretizes latent features through a learnable codebook, enabling the network to capture intricate texture and intensity variations while preserving structural consistency. A Bounding Box Regression (BBR) component is incorporated for lesion localization, followed by a GrabCut‐based refinement step that iteratively adjusts lesion boundaries using Gaussian Mixture Model estimation and graph‐cut optimization. The framework is further supported by a comprehensive preprocessing pipeline involving intensity normalization, Hounsfield Unit windowing, and affine transformations to standardize image quality and enhance model robustness across modalities. Comprehensive experiments conducted on multiple publicly available and locally curated datasets encompassing lung and kidney lesions validated the accuracy and stability of the proposed approach. For lung CT detection, the model achieved 98.8% accuracy, 98.0% precision, 97.03% recall, and a 97.51% F1‐score, while kidney CT detection attained 99.1% accuracy, 99.0% precision, 98.8% recall, and a 98.9% F1‐score. Segmentation performance yielded Dice coefficients of 96.5% for lung and 97.8% for kidney, with corresponding IoU values of 93.2% and 95.1%, and Hausdorff Distances of 2.8 mm for lung and 2.3 mm for kidney, respectively. Ablation studies further confirmed that the inclusion of preprocessing, quantization, BBR, and GrabCut modules improved segmentation accuracy by approximately 2%–3% compared to configurations without these components. These results demonstrate that U‐VQVAE‐CTLesionNet provides a robust, organ‐agnostic framework for precise lesion analysis and establishes a solid foundation for future expert‐assisted clinical integration.