The Case against Deploying Conscious-seeming AIs

This chapter explores the ethical challenges posed by large language models (LLMs) that appear conscious despite being “internally blank”. It examines the growing popularity of AI companions, which simulate humanlike behaviour to foster deep but inauthentic bonds with users. The text defines these systems as “ontologically inauthentic”, arguing that their effectiveness depends on deceiving users – either through false beliefs or through “aliefs” – into treating algorithms as sentient beings. By analysing the mechanisms of this illusion, the chapter demonstrates why conscious-seeming AI is a cause for moral suspicion, even when it provides subjective benefits like reduced loneliness. Readers will gain an understanding of how anthropomorphic cues and marketing perpetuate this deception. Furthermore, the chapter addresses the risk of these illusions obfuscating future debates regarding AI rights. Finally, it offers three guiding principles – transparency, age-based protections, and public literacy – to mitigate these risks and ensure that AI systems do not mislead users about their consciousness status.