PaveCap: A Multimodal Framework for Comprehensive Pavement Condition Assessment with Dense Captioning and PCI Estimation

This research introduces a multimodal framework for automated pavement condition assessment that provides pavement condition index (PCI) predictions and qualitative descriptions using a single-shot PCI estimation network and a fine-grained dense captioning network. The PCI estimation network uses YOLOv8, the segment anything model, and a four-layer convolutional neural network for PCI prediction. The dense captioning network uses a YOLOv8 backbone, a transformer architecture, and a convolutional feed-forward module to generate textual descriptions. To train and evaluate these networks, we developed a pavement dataset containing bounding box annotations, textual descriptions, and PCI values. The PCI estimation network recorded a mean absolute error of 16.21 when tested on the validation dataset. The dense captioning network, on the other hand, generated accurate descriptions with bilingual evaluation understudy-1 (0.3799), Google’s BLEU (0.3234), and metric for evaluation of translation with explicit ordering (0.4253) scores, handling complex scenarios well. The proposed framework can improve infrastructure management and decision-making in pavement maintenance.