Working with AI: Measuring the Occupational Implications of Generative AI

In this work, we take a step toward that goal by analyzing the work activities people do with AI, how successfully and broadly those activities are done, and combine that with data on what occupations do those activities. We analyze a dataset of 200k anonymized and privacy-scrubbed conversations between users and Microsoft Bing Copilot, a publicly available generative AI system. We find the most common work activities people seek AI assistance for involve gathering information and writing, while the most common activities that AI itself is performing are providing information and assistance, writing, teaching, and advising. Combining these activity classifications with measurements of task success and scope of impact, we compute an AI applicability score for each occupation. We find the highest AI applicability scores for knowledge work occupation groups such as computer and mathematical, and office and administrative support, as well as occupations such as sales whose work activities involve providing and communicating information.

First, the user is seeking assistance with a task they are trying to accomplish; we call this the user goal. Analyzing user goals allows us to measure how generative AI is assisting different work activities. In addition, the AI itself performs a task in the conversation, which we call the AI action. Classifying AI actions separately lets us measure which work activities generative AI is performing. To illustrate the distinction, if the user is trying to figure out how to print a document, the user goal is to operate office equipment, while the AI action is to train others to use equipment.

Our user goal vs. AI action distinction, combined with their classification into work activities, relates to a key question in the literature and public discourse around AI: to what extent is AI automating vs. augmenting work activities? The implication is that augmentation will raise wages and automation will lower wages or lead to job loss. However, this question often conflates the capability of a new technology with the downstream business choices made as a result of that technology. For example, if AI makes software developers 50% more productive, companies could raise their ambitions and hire more developers as they are now getting more output per developer, or hire fewer developers because they can get the same amount done with fewer of them. Our data is only about AI usage and we have no data on the downstream impacts of that usage, so we only weigh in on the automation vs. augmentation question by separately measuring the tasks that AI performs and assists.

We find that information gathering, writing, and communicating with others are the most common user goals in Copilot conversations. In addition to being the most common user goals, information gathering and writing activities receive the most positive thumbs feedback and are the most successfully completed tasks. On the AI action side, we see that AI often acts in a service role to the human as a coach, advisor, or teacher that gathers information and explains it to the user. Furthermore, the activities that AI performs are very different from the user goals the AI assists: in 40% of conversations, these sets are disjoint. To measure occupation-level impacts, we use the standard practice of decomposing an occupation into its constituent work activities [4]. The occupations with highest AI applicability scores are knowledge work and communication focused occupations, but we find that all occupational groups have at least some potential for AI impact (unsurprisingly, with much narrower effects on occupations with large physical components). More specifically, we find the major occupation categories with the highest AI applicability scores are Sales; Computer and Mathematical; Office and Administrative Support; Community and Social Service; Arts, Design, Entertainment, Sports, and Media; Business and Financial Operations; and Educational Instruction and Library.