Full automation or just a copilot? A study of large language models on spare parts demand forecasting with OpenAI o1-mini

Large language models (LLMs) promise process automation but entail reliability concerns. We investigate the capability of LLMs in the context of operations management, specifically demand forecasting of spare parts. For this, we develop a self-reflective simulation framework with an LLM producing automatic demand forecasts, using simulated and real-world data. This includes generating and debugging a Python script, verifying the results, and iteratively improving the outcomes. Our results demonstrate a high variability of responses, especially for stock-keeping units with irregular demand. Averaging over several repetitions can improve performance and result in forecasts that are comparable in accuracy to common benchmarks. Providing more detailed instructions in the prompt can improve forecasting accuracy. Training a decision tree to determine which method to choose for which kind of stock keeping unit results in lower forecasting errors compared to always using the same method or deciding on the method based on classifications from previous research.