A multi-layer big data platform for intelligent monitoring and diagnostics of agrocoenosis parameters based on hybrid machine learning methods

A multi-tier big data platform for intelligent monitoring and diagnostics of agrocenoses parameters has been developed using hybrid machine learning methods. The relevance of this platform is determined by the gap between the growing volumes of sensor, satellite, and aerial imagery data and the performance of single-machine instruments. It is based on a fourtier architecture (device – edge node – distributed core – application circuit) based on Apache Kafka 3.7, Apache Spark 3.5, and TimescaleDB 2.14. The database was collected over 182 days on 2,850 hectares in the Republic of Tatarstan (1,240 sensor nodes, 28 edge gateways); the volume of raw data is 4.2 TB, 12.8 million records. Six models for yield forecasting, anomaly detection, and crop diagnostics were compared. The best result was achieved by CatBoost with five-fold cross-validation: R² = 0.89, RMSE = 2.4 c/ha, MAE = 1.7 c/ha for spring wheat; in the diagnostic task of 14 parameters, Accuracy = 0.898 and AUC-ROC = 0.943 were achieved at a latency of 75 s. Scaling the Spark cluster from 1 to 8 nodes reduced the processing time from 47.3 to 7.1 min (speedup 6.67; efficiency 83.4%). The author’s integral index of the IDC takes into account the accuracy, completeness, and timeliness of the response; its final value was 0.82, which is 0.04–0.17 higher than the values of the review papers.