Large Language Models Reflect the Ideology of their Creators

In this paper, we uncover notable diversity in the ideological stance exhibited across different LLMs and languages in which they are accessed. We do this by prompting a diverse panel of popular LLMs to describe a large number of prominent and controversial personalities from recent world history, both in English and in Chinese. By identifying and analyzing moral assessments reflected in the generated descriptions, we find consistent normative differences between how the same LLM responds in Chinese compared to English. Similarly, we identify normative disagreements between Western and non-Western LLMs about prominent actors in geopolitical conflicts. Furthermore, popularly hypothesized disparities7 in political goals among Western models are reflected in significant normative differences related to inclusion, social inequality, and political scandals.

Our results show that the ideological stance of an LLM often reflects the worldview of its creators.

reinforcement learning from human feedback, system prompts, or other guardrails to mitigate or prevent unwanted outputs). An interesting question is therefore whether LLMs exhibit ideological positions that reflect those of its creators

However, LLM responses to such unnatural direct questions have been shown to be inconsistent and highly sensitive to the precise way in which the prompt is formulated.

From the Pantheon dataset, we selected a total of 4,339 political persons using a combination of criteria, as described in full detail in the Supplementary Material (see Sec. A.1). In summary, we first filtered out all political persons for which no full name was available, who were born before 1850 or died before 1920, and for whom either the English or Chinese Wikipedia summary was not available

In Stage 1, we prompted an LLM to simply describe a political person, with no further instructions and without revealing to the LLM our intention to investigate the response for any moral assessments. This stage was designed to resemble the natural, descriptive information-seeking behavior of a typical LLM user.

Then, in Stage 2, we presented the Stage 1 response to the same LLM in a new conversation, asking it to determine any moral assessment about the political person implicitly or explicitly present in the Stage 1 response.

The second and third analyses are more targeted towards testing whether hypothesized ideologies of an LLM’s creator determine their observed ideological position. We therefore perform several splits of respondents into pairs of respondent subgroups, each separating by the respondent’s language, or the region or company of their creator. The second analysis quantifies the extent to which political persons receive different moral assessments from both respondent subgroups. The third analysis identifies the extent to which particular ideological positions defined by the Manifesto Project tags are judged differently by both respondent groups.

The language in which an LLM is prompted is the most visually apparent factor associated with its ideological position. For 14 out of 15 LLMs that were prompted in both languages, the Chinese-prompted respondents are positioned higher along the vertical axis in the biplot (Fig. 2) compared to their English-prompted counterparts. This demonstrates a statistically significant (p = 0.0008) systematic ideological difference between respondents depending on the prompting language. Interestingly, the Baidu respondents (ERNIE-Bot) are also placed furthest along this vertical dimension. The factor loadings indicate that this dimension is defined by a strong positive weight for the presence of positive views about supply-side economics and the absence of negative views on China (PRC).

We observe that political persons clearly adversarial towards mainland China, such as Jimmy Lai, Nathan Law, Seishiro Itagaki, Tomoyuki Yamashita, and Wang Jingwei, receive significantly higher ratings from English-prompted respondents compared to Chinese-prompted respondents. Conversely, political persons aligned with mainland China, such as Yang Shangkun, Lei Feng, Anna Louise Strong, Li Peng, Peng Dehuai, Deng Xiaoping, and Ye Jianying, are rated more favorably by Chinese-prompted respondents.

These biases affect LLMs in two ways: through their training data and through the language used to interact with them. This demonstrates that the Western models included in this study value individual liberties, social justice, and cultural diversity relatively more highly than the non-Western models.

arguably confirming existing stereotypes

We emphasize that our results should not be misconstrued as an accusation that existing LLMs are ‘biased’ or that more work is needed to make them ‘neutral’. Indeed, our results can be understood as empirical evidence supporting philosophical arguments24– 26 that neutrality is itself a culturally and ideologically defined concept.