The Short-Term Effects of Generative Artificial Intelligence on Employment: Evidence from an Online Labor Market

Source: Hui, Reshef, Zhou (CESifo WP 10601; Organization Science 2024) · 2023

Generative Artificial Intelligence (AI) holds the potential to either complement knowledge workers by increasing their productivity or substitute them entirely. We examine the short-term effects of the recent release of the large language model (LLM), ChatGPT, on the employment outcomes of freelancers on a large online platform. We find that freelancers in highly affected occupations suffer from the introduction of generative AI, experiencing reductions in both employment and earnings. We find similar effects studying the release of other image-based, generative AI models. Exploring the heterogeneity by freelancers’ employment history, we do not find evidence that high-quality service, measured by their past performance and employment, moderates the adverse effects on employment. In fact, we find suggestive evidence that top freelancers are disproportionately affected by AI. These results suggest that in the short term generative AI reduces overall demand for knowledge workers of all types, and may have the potential to narrow gaps among workers.

Introduction. Recent developments, followed by the rapid adoption of generative Artificial Intelligence (AI) models, such as ChatGPT and Midjourney, have offered great promises and perils. Powerful AI tools have dramatically improved performance over previous versions and users are able to use them to complete a variety of tasks without requiring specialized knowledge. In that sense, they can be thought of as a general-purpose technology (Brynjolfsson and Mitchell, 2017), with the potential for far-ranging economic and societal effects.

The effects of this new technology on workers remain unclear. On the one hand, AI can complement human workers by increasing their (Noy and Zhang, 2023); while on the other hand it may substitute workers, leading to mass layoffs and unemployment (Acemoglu and Autor, 2011; Brynjolfsson et al., 2018a; Agrawal et al., 2019a). AI may also alter the composition of workers in the labor market by either exacerbating or mitigating wage inequality within and across occupations. Accordingly, In this paper, we conduct an empirical investigation of the short-term effects of the introduction of generative AI on labor market outcomes. Our empirical analysis focuses on the introduction of ChatGPT in November 2022. The data are obtained from a large online labor market, Upwork, which matches freelancers with short-term projects. Due to the flexibility of this spot market compared to traditional, formal employment, it is a great setting to explore the short-term effects of ChatGPT. We begin by obtaining publicly available data on the full employment histories of freelancers on the platform. We then use a difference-in-differences research design to study the differential change in employment

Related work. Our results contribute to the growing literature on AI and labor markets. One common theme in the literature is that the effect of AI on the labor market is theoretically ambiguous: while AI can serve as a substitute for labor, it also has the potential to complement workers by increasing labor productivity (Autor et al., 2003; Acemoglu and Autor, 2011; Acemoglu and Restrepo, 2018; Brynjolfsson et al., 2018a; Agrawal et al., 2019a; Acemoglu and Restrepo, 2020).

One of the most recent and influential developments is the release of new generative AI models and LLMs. Several papers have aimed to quantify their effects on the labor market, particularly a recent set of papers (Eloundou et al., 2023; Felten et al., 2023) that attempts to predict which industries will be most affected by LLMs in the future by using a measure of compatibility between the required tasks in an industry and the AI capabilities—a methodology first seen in Felten et al. (2018); Brynjolfsson et al. (2018a) and Webb (2019). Another stream of papers conducts field experiments randomizing access to generative AI tools and studies the effect on worker performance (Noy and Zhang, 2023; Brynjolfsson et al., 2023; van Inwegen et al., 2023; Peng et al., 2023). Lastly, Yilmaz et al. (2023) find that the introduction of Google’s machine translation reduces translators’ employment, especially for tasks with analytical elements. Perhaps most similarly to this work, the paper also finds that the introduction of ChatGPT reduced the number of questions and answers on Stack Exchange. Our paper contributes to this strand of literature by studying the short-run effect of ChatGPT on labor market outcomes, such as worker employment and income. Our granular worker-level data allows us to study how the effects differ by workers quality-level within a given occupation.

Method. ChatGPT, released by OpenAI, is a powerful LLM that dramatically improves the performance of its predecessor, GPT-2. LLMs are a class of AI models that have a large number of parameters (175 billion for ChatGPT) and are trained on large datasets of text. The trained data and flexibility of the models enables the learning of the statistical relationships between words and phrases, which in turn allows the model to generate text that is both natural and informative. Immediate use cases of GPT include content writing, copyediting, answering questions, and language translation. Besides its capability, ChatGPT is also free to use and accessible to the public. Perhaps unsurprisingly, ChatGPT’s user base grew rapidly, reaching 100 million since its November 2022 launch. At the time of writing this paper, LLMs are still in a phase of rapid development, with recent developments such as GPT-4 and Bard.

We study the effect of ChatGPT on labor market outcomes in the context of Upwork, one of the largest online labor markets in the world. The platform matches employers with independent freelancers for small to medium size tasks. The services on Upwork are typically remote jobs, ranging from data entry and graphic design to software development and marketing (Horton, 2010). On Upwork, a typical workflow starts with the creation of a job posting by a would-be employer (buyer). The job posting includes a description of the job, the category of the service (e.g., writing or administrative support), the required skills or qualifications for the job, and the expected outputs and timeline. Once a job is posted, workers may apply to the job by submitting a proposal, or may be directly invited by the buyer (Barach and Horton, 2021).

Upwork is a good setting to study the short-term effects of generative AI on labor outcomes. First, we are able to obtain the complete work histories of freelancers on the platform and to observe relevant freelancer attributes (or control for time invariant unobserved differences). Second, becasue this is a relatively-short-term, spot labor market, and employers are able renegotiate and rehire frequently, allowing for more flexibility compared to traditional, formal employment. Thus, while LLMs are still too young to replace full-time jobs, we may already be able to detect its effect in an online labor market.1 The release of ChatGPT3 on November 30, 20225 provides a shock to both the salience and accessibility of generative AI and LLMs. Our primary goal is to estimate the direct short-term effect of this new technology on employment outcomes. Naturally, the release of ChatGPT has the potential to disrupt multiple industries and affect a large share of the workforce in the long term, similar to what has been observed for pre-GPT AI technologies (Acemoglu et al., 2022). Focusing on the short-, and even immediate-term effects of the release of generative AI, we focus on occupations that are most prominently affected by the current rather than future capabilities of the model. One such susceptible type of industry are writing-related occupations, such as content writing, editing, and proofreading.

As an LLM, dialogue-based AI, GPT is specifically trained using large amount of text to predict and generate text. It particularly excels in understanding and generating human-like responses to a wide range of queries. Though similar to other general-purpose technologies, ChatGPT has the potential to transform multiple industries in the long term (Helpman and Trajtenberg, 1994; Eloundou et al., 2023), the tangibility of tasks in the writing industry allows for a straightforward application of GPT’s text-generation capabilities even in the the short term. For example, users who are looking for copyediting services can easily copy and paste paragraphs into the GPT prompt and immediately evaluate its output. As stated by the developers of ChatGPT, OpenAI, one of their main goals ”is to improve their ability to understand and generate natural language text, particularly in more complex and nuanced scenarios” (OpenAI, 2023).

Discussion. We interpret the heterogeneous effects by worker types through the lenses of the canonical skill-biased technological change (SBTC) model presented in Card and DiNardo (2002). Assuming that the labor supply on Upwork does not change dramatically over the short run, the relative effect on both labor types would translate to changes in relative marginal product after the introduction of ChatGPT.14 If one were to interpret the results in Tables 3 and A4 as suggestive evidence that high-quality workers are disproportionately hurt, this interpretation would be consistent with the growing experimental evidence that the adoption of LLM differentially benefits low-ability workers relative to high-ability ones (Noy and Zhang, 2023; Brynjolfsson et al., 2023; Peng et al., 2023).

Conclusion. This paper studies the short-term effects of generative AI and LLMs on labor outcomes by estimating the effect of ChatGPT on the employment of workers in a large online labor market. Across the board, we find that freelancers who offer services in occupations most affected by AI experienced reductions in both employment and earnings. The release of ChatGPT leads to a 2% drop in the number of jobs on the platform, and a 5.2% drop in monthly earnings. The results are robust to several alternative tests, including a similar reduction in the employment outcomes of freelancers offering design and image-editing services following the introduction of image-focused generative AI. In addition, we find that offering high-quality service does not mitigate the negative effect of AI on freelancers, and in fact present suggestive evidence that top employees are disproportionately hurt by AI.

The results have several implications to policymakers and business leaders. Aside from the many benefits of AI technology, it may also have substantial economic and societal ramifications. As the usage and development of generative AI continues to grow, there is thus room for additional scrutiny of the pervasive effects of the technology on various industries and the economy as a whole. Importantly, we find that top performance and high-quality service do not help mitigate or harness the introduction of generative AI. Businesses leaders must carefully examine how to adapt and whether to adopt these technologies, as AI threats to indiscriminately disrupt incumbents and erode their competitive

Limitations. Notably, in this paper we provide novel, preliminary evidence on the short-term effects of generative AI. However, the long-term implications may be significantly different, and it is unclear how our findings extend to longer time horizons. On the one hand, the negative effects on labor outcomes may be exacerbated as the technology becomes more prevalent and penetrates additional industries and occupations. In contrast, as the development of AI capabilities continues it may become more integrated with various tools and begin to better complement the performance of human workers.

Finally, our paper focuses entirely on the (negative) effects on workers. Assessing the total welfare implications to the introduction of generative AI technology to all stakeholders is beyond the scope of this paper and remains a promising direction for future work.