Moral AI in medical decision-making

Building on the framework for moral artificial intelligence (AI) proposed by Schaich Borg, Sinnott-Armstrong and Conitzer (SSC), we discuss what would be required for AI as a moral decision-making aid in the context of medical decision-making. SCC outlines a five-step approach that centres on how to best handle the training data for AI: survey people’s moral views, use preference elicitation methods to ascertain the weights of different considerations, idealise preferences to avoid the problem of bias based on ignorance, aggregate individual preferences to group judgements, and model moral decision-making. While their framework is plausible and implementable, we argue that it rests on three problematic assumptions about the three pairs of similar but distinct concepts. The aim of this article is to use the SSC framework as a starting point for developing an account of moral AI that preserves the strengths of their model while adding further features suitable for assisting moral decision-making. In order to outline a more comprehensive model of moral AI, we emphasise three conceptual distinctions: (1) preferences versus reasons, (2) rankings versus deliberation and (3) predictions versus judgements. The resulting approach focuses on the latter concepts of these pairs and suggests a version of moral AI that retains the virtues of the SSC approach while avoiding some of the potential pitfalls.