Determinants of LLM-assisted Decision-Making
Decision-making is a fundamental capability in everyday life. Large Language Models (LLMs) provide multifaceted support in enhancing human decision-making processes. However, understanding the influencing factors of LLM-assisted decision-making is crucial for enabling individuals to utilize LLM-provided advantages and minimize associated risks in order to make more informed and better decisions. This study presents the results of a comprehensive literature analysis, providing a structural overview and detailed analysis of determinants impacting decision-making with LLM support. In particular, we explore the effects of technological aspects of LLMs, including transparency and prompt engineering, psychological factors such as emotions and decision-making styles, as well as decision-specific determinants such as task difficulty and accountability. In addition, the impact of the determinants on the decision-making process is illustrated via multiple application scenarios. Drawing from our analysis, we develop a dependency framework that systematizes possible interactions in terms of reciprocal interdependencies between these determinants. Our research reveals that, due to the multifaceted interactions with various determinants, factors such as trust in or reliance on LLMs, the user’s mental model, and the characteristics of information processing are identified as significant aspects influencing LLM-assisted decision-making processes. Our findings can be seen as crucial for improving decision quality in human-AI collaboration, empowering both users and organizations, and designing more effective LLM interfaces.
Large Language Models (LLMs) offer versatile assistance in decision-making processes. For instance, their ability to process and summarize extensive text data [114] enables decision-makers to comprehend key insights swiftly. Moreover, LLMs are adept at idea generation [59] and are capable of generating different solutions [200], enhancing the creation of various alternatives in decision-making. They can also identify patterns [86] and analyze historical data [151], thus potentially providing support in the analysis of decision situations and evaluation of alternatives. Additionally, LLMs demonstrate the capability to adopt personas of various characters and engage in social interactions with each other [144] and can simulate debates featuring arXiv:2402.17385v1 [cs.AI] 27 Feb 2024 different opinions [173]. Through the ability to incorporate diverse perspectives and simulate discussions, LLMs empower decision-makers to systematically explore a multitude of scenarios and potential outcomes. Moreover, LLMs exhibit a high degree of rationality in decision-making tasks [27], implying that LLMs hold the potential to enhance human decision-making processes by providing reasoned outputs.
Nevertheless, the increased capabilities of LLMs are associated with heightened risks [85]. Undesirable behaviors exhibited by LLMs encompass, for instance, generating nonfactual or untruthful information (hallucinations) [7], reiterating a user’s presented viewpoints (sycophancy) [148], providing false rationalizations that diverge from the true reasons behind the LLMS’ outputs (unfaithful reasoning) [180], and employing deception because LLMs have rationalized that it can advance a particular objective (strategic deception) [145].
We present a structural overview and detailed analysis of technological, psychological, and decision specific factors determining LLM-assisted decision-making as result of an integrative literature review.
Drawing from this analysis, we develop a dependency framework systematizing the potential interactions and interdependencies between these determinants.
Furthermore, we demonstrate the utility of our work by illustrating its application in the context of multiple exemplary scenarios
For instance, being aware of how psychological factors interact with the technological aspects of LLMs allows users to comprehend how their expectations, experiences, trust in AI, and biases collectively shape decisions. This awareness facilitates more thoughtful and informed decision-making processes.
Simon [169] proposed a decision process consisting of the following three phases:
Intelligence: In this initial stage, the decision maker recognizes the problem and the need to make a decision and gathers information about the problem situation.
Design: The design phase involves systematically structuring the problem, establishing specific criteria, and identifying a range of alternatives aimed at resolving the issue at hand.
Choice: In the choice phase, the decision-maker selects the most optimal alternative that aligns with the defined criteria and subsequently makes the final decision.
Human-AI decision-making, also referred to as AI-assisted decision-making, involves scenarios where an AI model supports the user in making a final judgment or decision, frequently viewed as a kind of collaboration between humans and AI systems [26]. Ultimately, the human decision-maker makes the final decision [172]. In AI-assisted human decision-making, the following cycle is typically repeated: (1) receiving input from the environment, (2) the AI suggesting a (possibly erroneous) action, (3) the human making a decision based on the AI’s input, and (4) the environment providing feedback, which the person learns when to trust the AI’s recommendation [8].
Karmaker and Teler [158] propose a taxonomy to categorize LLM prompts for complex tasks based on the following four dimensions:
Turn: This dimension refers to the number of turns applied while prompting an LLM.
Expression: Depending on how the task and its sub-tasks are articulated, prompts can be categorized as either question-style or instruction-style.
Role: This dimension classifies prompts based on whether a specific system role is defined in the LLM system prior to presenting the actual prompt. Prompts may have either a defined or undefined system role.
Level of Details: Prompts are categorized based on the presence or absence of specific elements of the goal task definition in the instruction.
In the Design Phase of decision-making, LLMs can provide assistance in structuring the problem by identifying key components, relationships, and dependencies within the problem statement. Moreover, LLMs may support the decision-maker in defining specific and relevant criteria for evaluating potential alternatives, for instance, by analyzing existing industry standards, expert opinions, and relevant literature. LLMs can contribute to identifying alternatives aimed at resolving the problem by generating ideas through processing vast amounts of textual data. Hence, they are capable of proposing solutions and alternatives that decision-makers might not have considered otherwise.
In the Choice Phase of decision making, LLMs can support the decision-maker by evaluating and comparing different options based on defined criteria. They are capable to process vast amounts of textual data to assess how each option aligns with the defined criteria, enabling decision-makers to make data-driven choices. Based on the provided data, LLMs can simulate different scenarios by including various choices. This can help decision-makers understand the potential outcomes of each choice and enables them to select options with favorable consequences. As LLMs analyze historical data to assess potential risks associated with each option, LLMs can aid decision-makers in making informed decisions that consider potential challenges
Challenges associated with decisions made prior to the implementation of an LLM, include, for example, Unfathomable Datasets, Fine-Tuning, and Tokenizer-Reliance. Unfathomable Datasets refer to the issue that the size of the pre-training datasets currently in use is so large that it becomes nearly impossible for individuals to validate the quality of the documents they contain [89]. For instance, the dataset of LLMs contains numerous near-duplicates, which negatively impacts the models’ performance [107]. An additional obstacle involves Fine-Tuning required for integrating up-to-date item information, which, in turn, demands substantial computational resources and incurs time costs [112]. Challenges related to Tokenizers include computational overhead, dealing with new words, and low interpretability on the user side [89]. Despite their capabilities, LLMs are susceptible to errors, particularly if they have been trained on biased or incomplete data. Given their continuous learning from internet texts, neglecting to thoroughly verify and validate LLMs’ responses may lead to incorrect or incomplete decisions [30].
Behavioral challenges of LLMs that emerge during deployment include Prompt Brittleness, Misaligned Behavior and Outdated Knowledge. Prompt Brittleness [143] refers to the phenomenon where even modifications in wording can significantly impact the overall accuracy [205]. Misaligned Behavior points to the fact that outputs generated by LLMs often do not align well with human values or intentions, resulting in unintended or adverse consequences [53, 157]. Moreover, the knowledge incorporated into LLMs might become outdated or inappropriate as time progresses [196]. Another limitation of LLMs is that the high complexity and scaling of LLMs pose challenges in terms of explainability [54, 66].
Concerning the determinant Trust in/ and Reliance on LLMs, only one of the mentioned sub-determinants Adequate Reliance, Under-Reliance or Over-Reliance, can have an impact on the decision-making process. Figure 5 also shows that both the user’s mental model regarding LLMs and the decision problem influence LLM-assisted decision-making. In information processing, Intuitive Thinking always influences the decision-making process, whereas Deliberate Thinking may not necessarily occur. As can be inferred from Figure 5, either Positive or Negative Feelings and Mood have an impact on LLMassisted decision-making. Regarding Metacognitions, Monitoring and/ or Controlling can occur. Of the sub-determinants, controlling and monitoring, either one or more can influence the decision-making process. Furthermore, Figure 5 illustrates that only one of the three Decision-Making Styles, i.e., Optimizing, Satisficing or Minimizing, can exert an impact on decision processes assisted by LLMs.
impact of explanations on mental models can be analyzed through two key processes: maintenance and building. In the mental model maintenance process, individuals tend to uphold or strengthen their existing beliefs. When faced with new information, they interpret or integrate it in a way that aligns with their current understanding. This implies that well-crafted explanations can help users reinforce their existing mental models and ensure they are aligned with the AI system’s decision rationale. On the other hand, the mental model building process occurs when individuals undergo substantial restructuring or create entirely new mental models in response to novel or contradictory information
Incomplete or inaccurate mental models can lead to inappropriate reliance and trust [172], resulting in over- or under-reliance [141].
an accurate mental model of prompt engineering enables users to formulate queries and prompts in a manner that aligns with the LLM’s capabilities, enhancing the likelihood of obtaining precise and relevant responses. By utilizing appropriate prompting techniques users can structure prompts to extract key information essential for the specific decision-making task.
Meta-reasoning focuses on "the processes that monitor and control reasoning, problem solving, and decision making" [177, p. 275]. These metacognitive processes act as the "top manager" of cognitive functions, responsible for regulating functions, such as setting goals, selecting among reasoning strategies, and making decisions [48]. Monitoring is defined as the subjective assessment of the quality of performance in a cognitive task. In contrast, metacognitive Control involves initiating, terminating, or modifying the effort allocated to a cognitive task ([2]. Monitoring can manifest in various forms of judgments that individuals spontaneously exhibit before, during, or after cognitive processing. These judgments might include assessing whether a task is feasible, evaluating progress, or estimating the likelihood of success of a particular choice [1].
The Affect-as-Information Hypothesis suggests that emotions and mood act as sources of information [32]. The depth of processing is influenced by the presence of positive and negative emotions, which activate either heuristic or systematic information processing in decision-making [147]. Consequently, individuals in a positive mood perceive their environment as benevolent, leading them to process information in a global and heuristic manner. Conversely, those in a negative mood view their environment as problematic, prompting them to process information analytically and diagnostically.
Feeling of Rightness (FOR) can impact subsequent behaviors and predict the likelihood of changing answers later [187]. When the FOR is weak, it triggers analytical problem-solving and extended deliberation. In contrast, a strong FOR indicates that further reflection and reconsideration of the answer are unnecessary [2], likely resulting in over-reliance on AI. The Feeling-of-Error (FOE) can signal to individuals a possible failure in their mental processes.
Decision-Making Styles and Information Processing. Minimizers, who prioritize minimizing resources in decision-making, are likely to employ fast information processing. Consequently, they tend to rely on fast, intuitive judgments and heuristics for swift decision-making. Satisficers, who seek options that fulfill particular criteria, might use a combination of fast and deliberate information processing, probably influenced by the complexity of the decision. For simpler decisions, satisficers might use fast processing, relying on heuristics and readily available information to quickly identify satisfactory options. In contrast, for more complex decisions, satisficers may shift to slow processing, taking the time to gather and analyze detailed information to ensure the chosen option meets their criteria adequately. Maximizers, who strive for the the best possible outcome, are inclined to predominantly engage in slow information processing. They are likely to invest considerable time and effort in thouroughly researching, evaluating, and considering all available information to ensure they make the most optimal decision.
LLM 5.6.3
Deriving Implications for LLM-assisted Decision-Making Decision-Making Styles and Information Processing. The decision-making styles of minimizers, satisficers, and maximizers are likely to influence their interactions with and utilization of LLMs in their decision-making processes. Minimizers, who prefer quick and efficient decision-making, might readily accept the first LLM suggestion that meets their requirements and do not further analyze other LLM generated recommendations. The approach of satisficers may vary depending on the complexity of the decision. For simple decisions, they might use LLMs similarly to minimizers, seeking quick answers. However, for more complex decisions, they would expect the LLM to provide more detailed and carefully analyzed information. Since satisficers search for options that meet specific criteria, they would utilize LLMs to filter and present options that align with these criteria. Maximizers, who aim for the best possible outcome, involve themselves in thorough research and evaluation. Consequently, they would use LLMs for in-depth information gathering. This likely leads to extended interactions with the LLM, as they probe various aspects of a decision, compare options, and weigh the pros and cons.