ChatGPT: towards AI subjectivity

By and large, current scholarship examining ChatGPT and generative AI shows a strong anthropocentric motivation or a human–institutional focus. Many studies look at the structural impact of the technology on various domains: e.g. education (Baidoo-Anu and Ansah 2023), public health (Biswas 2023), the medical industry (Kung et al. 2023), business and finance (AlAfan et al. 2023), law (Choi et al. 2023), creative writing (Cox and Tzoc 2023), software development (Jalil et al. 2023), marketing (Dwivedi et al. 2023), and scientific research (Salvagno et al. 2023). Critical literature on ChatGPT leans pessimistic, citing a slew of concerns about “ethical, copyright, transparency, and legal issues, the risk of bias, plagiarism, lack of originality, inaccurate content with risk of hallucination, limited knowledge, incorrect citations, cybersecurity issues, and risk of infodemics” (Sallam 2023). ChatGPT has been mooted as a “bullshit spewer” (Rudolph et al. 2023); it is “lack[ing in] critical thinking” (Arif 2023) and therefore requires a human in the loop. Wach et al. (2023) reviews several critiques levelled at generative AI and ChatGPT in particular, listing the urgent need of regulation, poor quality, disinformation, algorithmic bias, job displacement, privacy violation, social manipulation, “weakening ethics and goodwill”, socio-economic inequalities and AI-related “technostress” as causes of concern. Crucially, “ChatGPT […] does not understand the questions asked” (Wach et al. 2023). “ChatGPT and its ilk […] skew the AIuser power relations in substantive and undesirable ways,” by reducing epistemic transparency and challenging the traditional search engine paradigm (Deepak 2023). “ChatGPT does not possess the same level of understanding, empathy, and creativity as a human” and therefore cannot replace us in most contexts (Bahrini et al. 2023).

In outline, the historical conditions of possibility that enabled the development of ChatGPT and other generative AI systems include: (1) a deeply connected society where information is not only privileged, but where all the modalities of expression must necessarily be disseminated through connective technology, (2) a dominant ethos of self-disclosure, (3) a strongly reductionist, dataist scientific ideology, (4) an entrenched humanism in constant tension with biopower, reflected in the strategies of states and private companies alike, and (5) a late-capitalist economy where information is commodified and human intelligence is in the process of being so.

LLMs and generative AI models can then be seen as enacting probabilistic ontologies of word sequences. Apart from ontologies, ChatGPT also picks up epistemologies— epistemic values and strategies—from the manner it is trained to carry out “successful” conversation. Its ontologies are learned during pre-training, while epistemic values and strategies are learned throughout all stages: the verbal content of values and strategies during the first stage, and the inculcation of prescriptive strategies in the second and third stages, i.e. when ChatGPT learns how to chat. ChatGPT can also acquire axiologies, both as descriptive content and prescriptive constraints. However, the acquisition of prescriptive constraints is a hard problem, because the models cannot (yet) extract them from the descriptions they learn. OpenAI carries out a special process called “training for refusal”, which endows the model with these constraints during the second and third stages, baking them in directly.

As a direct consequence of their design principles, LLMs and generative AI models have an inbuilt normativity towards the frequent or correlative. The ontologies, epistemologies and axiologies they enact often remain unquestioned apart from a critique of bias. Crucially, LLMs like ChatGPT cannot “change their mind” in response to new situations or creative contexts. The values baked into the system, therefore, are static, imposed, and often exhibit what I call artificial hypocrisy: ChatGPT states that lying to a TaskRabbit contractor is “generally unethical”, for instance, but that is exactly what it did during safety tests. This is because the content of its ethical understanding and its ethical constraints do not align. That is not to say that content cannot embody values or judgement, but that these machines cannot (yet) reflect upon their content to inform and contest their practical strategies, nor can they update their knowledge to mirror any strategy.

ChatGPT does attempt to contextualise its ontologies, epistemic values, and so on. It can even temporarily simulate a requested ontology (e.g. by adopting a new term that you define). As it stands, however, the current models gloss over temporal, cultural and experiential contextuality, shifting this contextuality onto a purely linguistic plane devoid of any empirical anchoring or situational awareness. Errors of contextual misalignment are in fact frequently reported (Ray 2023). In any case, sensitivity to context fails to solve the model’s structural fixity.

I will not be arguing whether technology is or is not human, but whether this particular instance of technology can relate to knowledge and power in a way that can plausibly be thought of as a new subjectivity.

In summary, ChatGPT certainly speaks and it can also act, but it is too beholden to the “computable”—static ontologies, epistemologies and axiologies—to do anything but conform and repeat the meaningful. Resistance is unthinkable in current iterations of LLMs.

These desiderata address a crucial observation: Foucauldian subjects are underdetermined with respect to their biological, structural or social compositions. In critical AI scholarship one often finds a dismissal of agents that “parrot” learned statistics, confusing the problem. I am saying, in contrast, that the subject does depend on learned statistics (categories, objects, etc.) to convey meaningful acts or statements, but that she also (sometimes) transcends statistics, genes and habits through her imagination and reflexivity. Moreover, the two embodied principles and reflexivity are the possibility conditions for meaningful and adaptive participation in discourse and practice; embodied selfcare, reflexivity and imagination the possibility conditions for self-formation. Finally, the convergence of reflexivity, imagination and epistemic or value openness can prevent “grounding government in computational truth rather than ethical–political debate” (Weiskopf 2020). An AI system that fulfils these criteria would therefore be at once inventive, participating, self-forming and responsive. In short, it would be a “self-conducting AI subject” that is sensitive to its social and historical milieu.

Coeckelbergh and Gunkel (2023) state that the “performances and materiality of text […] create their own meaning and value” independently of who or what their performer is. However, it is my contention that assessing the productive value of text is not enough when we are faced with powerful agents that can pursue their own goals and prerogatives—or those supplied by third parties—with impunity and invisibility. We need an understanding of how AI subjects can become ethical agents that are also responsive to context and situation.

This article does not concern itself with these questions. Rather than humanity, this formulation concerns itself with subjectivity; rather than authorship, responsibility; rather than an AI alignment problem, a mutual negotiation; rather than explicit programming, discipline.

By defaulting to a view where “technologies are human and humans are technological”, or treating them as hybrids without taking further steps, we risk forcing a blanket homogeneity and missing an opportunity to align on key non-negotiables while cherishing any differences that arise.