Enhancing Vision Transformer with Vision language Model Text Embeddings for Robust Air Pollution Classification

Far from arguing the importance of ecological awareness in our current era it is however necessary to highlight the pressing global environmental challenges that are faced by our society. While air pollution is only a sector of concern it is essentially one of the most critical factors influencing human health and environmental sustainability. Under this premise, monitoring air quality is necessary. While the Air quality index has mostly been measured using Internet of Things (IoT) sensors, detecting visible air pollution has garnered interest due to its accessibility. However, the existing works based on vision only methods (CNNs, Vits) have shown limitations in capturing generalized correlations that are essential for a robust air pollution detection system. The proposed solution investigates the capability of semantic information to broaden the scope of features learned by the model. A Vision language Model (VLM) based text encoder with the objective of introducing knowledge anchors across any datasets. The model generates language tokens to guide a Vision transformer. The proposal also investigates tuning mechanisms for the VLM and image filtering for the input data. The key innovation targets a cross modal integration of vision transformers with a vision language model to create a few shots classification model for air pollution classification. The aim is to produce a model with flexible data integration and capable of leveraging visual and semantic correlations. The research demonstrated an improved generalization across a broad dataset variance. The model outperforms baseline CNN in accuracy when it comes to cross data implementation.