Retrieval augmented generation for smart calorie estimation in complex food scenarios

Accurate food recognition and calorie estimation are critical for managing diet-related health issues such as obesity and diabetes. Traditional food logging methods rely on manual input, leading to inaccurate nutritional records. Although recent advances in computer vision and deep learning offer automated solutions, existing models struggle with generalizability due to homogeneous datasets and limited representation of complex cuisines like Indian food. This paper introduces a dataset containing over 15,000 images of 56 popular Indian food items. Curated from diverse sources, including social media and real-world photography, the dataset aims to capture the complexity of Indian meals, where multiple food items often appear together in a single image. This ensures greater lighting, presentation, and image quality variability compared to existing data sets. We evaluated the data set with various YOLO-based models, including YOLOv5 through YOLOv12, and enhanced the backbone with omniscale feature learning from OSNet, improving detection accuracy. In addition, we integrate a Retrieval-Augmented-Generation (RAG) module with YOLO, which refines food identification by associating fine-grained food categories with nutritional information, ingredients, and recipes. Our approach demonstrates improved performance in recognizing complex meals. It addresses key challenges in food recognition, offering a scalable solution for accurate calorie estimation, especially for culturally diverse cuisines like Indian food.