Summaries, Highlights, and Action items: Design, implementation and evaluation of an LLM-powered meeting recap system

“Meetings play a critical infrastructural role in the coordination of work. In recent years, the nature of meetings have been changing with the shift to hybrid and remote work – meetings have moved into computer mediated spaces in new ways that have lead to new problems (e.g. more time spent in less engaging meetings) and new opportunities (e.g. automated transcription/captioning and recap support). Recent advances in large language models (LLMs) for dialog summarization have the potential to improve the experience of meetings by reducing individuals’ meeting load and increasing the clarity and alignment of meeting outputs. Despite this potential, they exhibit significant issues if directly applied to summarize meeting long transcripts. Moreover, prior studies of recap highlight varying recap needs based on user’s context that no one design addresses, highlighting the need for in-context evaluations. To address these gaps, we describe the design, implementation, and in-context evaluation a meeting recap system. We first conceptualize two salient recap representations – important “highlights”, and a structured, “hierarchical” minutes view and provide supporting rationales from cognitive science and discourse theories on perception and recall. We develop a system to operationalize the representations with dialogue summarization as its building blocks. Finally, we evaluate the effectiveness of the system with seven users in the context of their work meetings. Our findings show promise in using LLM-based dialogue summarization for meeting recap and the need for both representations in different contexts. However, we find that LLM-based recap still lacks an understanding of whats personally relevant to participants, can miss important details, and mis-attributions can be detrimental to group dynamics. We identify collaboration opportunities such as a shared recap document that a high quality recap enables. We report on implications for designing AI systems to partner with users to learn and improve from natural interactions to overcome the limitations related to personal relevance and summarization quality. We synthesize these findings as design implications to advance the space of meeting recap in supporting group work in organizations.”