Anchoring bias in large language models: an experimental study

Large Language Models (LLMs) like GPT-4 and Gemini have significantly advanced artificial intelligence by enabling machines to generate and comprehend human-like text. Despite their impressive capabilities, LLMs are not without limitations. They exhibit various biases. Although much research has explored demographic biases, cognitive biases in LLM have not been equally studied. This study delves into anchoring bias, a cognitive bias in which initial information disproportionately influences judgment. Using an experimental data set, we examine how anchoring bias manifests in LLM and verify the effectiveness of various mitigation strategies. Our findings highlight the sensitivity of LLM responses to biased prompts. At the same time, our experiments show that to mitigate anchoring bias, one needs to collect information from comprehensive angles to prevent the LLMs from being anchored to individual pieces of information, as simple algorithms such as Chain-of-Thought, Thoughts of Principles, Ignoring Anchor Hints, and Reflection are insufficient.