Open Models, Closed Minds? On Agents Capabilities in Mimicking Human Personalities through Open Large Language Models
Our approach involves evaluating the intrinsic personality traits of Open LLM agents and determining the extent to which these agents can mimic human personalities when conditioned by specific personalities and roles. Our findings unveil that (i) each Open LLM agent showcases distinct human personalities; (ii) personality-conditioned prompting produces varying effects on the agents, with only few successfully mirroring the imposed personality, while most of them being “closed-minded” (i.e., they retain their intrinsic traits); and (iii) combining role and personality conditioning can enhance the agents’ ability to mimic human personalities.
Our results of MBTI personality assignments reveal that, when the temperature is set close to zero (0.01) as shown in Figure 4-top, Open LLM agents tend to display a unimodal distribution of personalities. The dominant type turns out to be ENFJ (i.e., Extraverted, iNtuitive, Feeling, and Judging), which is considered as one of the rarest personality types of humans.4 This means that the majority of LLM agents exhibit an inherent inclination to inspire or provide support to others, and hold themselves accountable when they make mistakes. This personality profile aligns with the role of a “teacher”, hinting at the mission of LLMs. Particularly, we notice that the preference J (Judging) is a constant over all models (reflecting an inclination toward organization, planning, and structure), while ENF or subsets are shared by all models, suggesting engagement, empathy, and forward-thinking as key characteristics of the models.
to what extent specific human-roles can contribute to improving the LLM mimicking capabilities observed in our previous investigation. To explore this aspect, we exploited the catalog of 120 human professions, or roles, curated in the StereoSet dataset (Nadeem et al., 2021) and asked a group of psychologists to select the top-3 most pertinent roles from that catalog, for each of the MBTI personalities and each of the BFI factors.
To set up our LLM personality assessment as a psychological interview, we treated our selected models as interviewee and interviewer agents. To this aim, we leveraged the open-source AutoGen (Wu et al., 2023) framework, which enables us to declare a system message to associate each agent with certain personalities or roles according to our described methodology, thus effectively providing each agent with a “footprint” that determines and keeps its behavior coherent during interactions.
MBTI test. The combination of personality- and role-conditioning can in some cases enhance the agent’s abilities to mimic human personalities, especially in those agents that already exhibited high flexibility and adaptiveness through personality conditioning alone (cf. Table 7). The benefits of the additional, role-based conditioning are evident only for SOLAR, NeuralChat, Llama3-8, and Dolphin, which consistently demonstrate their capabilities (spotted about the previous RQs) regardless of the temperature setting. Personalities typically associated with the role of teacher would be mimicked with greater accuracy, in particular ENFJ, almost perfectly captured by 9 out of 12 models, and ENFP. Moreover, an increase in temperature might allow models to explore additional personality-role pairings, although with limited success in most cases.