An intelligent multi-modal Q&A system for agriculture combining APGM, PBTCS, and RAG

This study addresses the growing need for efficient large-scale modeling techniques in agricultural production by introducing an intelligent multimodal question-answering system tailored to fruit tree cultivation. At its core is a lightweight multimodal prompt-generation model (APGM), which integrates a CLIP-based visual encoder with a Transformer-based text encoder and employs dual-path cross-attention and dynamic weight fusion for effective cross-modal representation. To further enhance system performance, we construct a domain-specific knowledge base with retrieval-augmented generation (RAG) and propose a Problem-Based Tiered Control Strategy (PBTCS) that allocates language models of different scales according to task complexity, balancing accuracy with computational cost. Experimental results show that the integrated system—combining APGM, RAG, and PBTCS—substantially outperforms general-purpose large models in pest and disease diagnosis, achieving 85.5% accuracy, a 25% increase in inference speed, and over a 50% reduction in GPU memory usage. Ablation studies confirm the synergy between APGM and RAG, with F1 scores improving by up to 18.8% on medium and high-difficulty queries and memory usage decreasing from 31.3 GB to 15.3 GB. This work provides a lightweight, cost-effective, and domain-optimized solution for agricultural question-answering systems.