Stance Detection on Social Media with Fine-Tuned Large Language Models

The implementation of prompting strategies represents a significant departure from traditional NLP model training methods. By employing these strategies, LLMs can generate predictions without the extensive fine-tuning typically required, showcasing their versatility across various tasks. This methodological shift has not only simplified the application of LLMs but also expanded their utility, making them particularly effective for tasks that require a deep understanding of language nuances, such as stance detection (Bang et al., 2023; Ouyang et al., 2022).

Fine-tuning tailors LLMs like ChatGPT, LLaMa- 2, and Mistral-7B to specific tasks, significantly enhancing their precision and relevance for contextually aware stance detection on social media platforms (Zhang et al., 2023c). This adaptation to the unique language and style of social media discourse enables these models to outperform traditional methods, demonstrating superior performance in identifying sentiments and viewpoints.

The flexibility of LLMs in social media analysis is highlighted by their use of advanced techniques such as fine-tuning, chain of thought prompting (Chen et al., 2023), and both zero-shot and few-shot learning (Zhang et al., 2023a; Cruickshank and Ng, 2023; Aiyappa et al., 2023). These methods help navigate complex online discourse, enhancing the models’ ability to interpret both explicit and implicit content effectively (Gatto et al., 2023; Lan et al., 2023).

Our study evaluates the enhanced performance of fine-tuned LLMs using Twitter datasets, now known as X, to analyze a wide range of user opinions. We aim to show that fine-tuning significantly improves model understanding of user viewpoints, offering a deeper insight into online discourse. This research underscores the advantages of fine-tuning in NLP, particularly in stance detection, highlighting its superiority over traditional and less-tailored methods.

The SemEval-2016 Dataset (Mohammad et al., 2016) includes diverse societal and political topics, with stances categorized as Favor, Against, or None. This dataset examines a range of targets including political figures such as Donald Trump and Hillary Clinton, as well as broad issues like the Feminist Movement and Climate Change. It offers a total of 2,914 training instances and 1,956 testing instances.

A.1 ChatGPT Fine-tuning Prompts

A.1.1 SemEval-2016 Template For the SemEval-2016 dataset, the following structured prompt was utilized:

Instruction:

Analyze the tweet below in the following context: [topic]. Consider the text, subtext, regional and cultural references, and any implicit meanings to determine the stance expressed in the tweet towards the target. The possible stances are: • FAVOR: The tweet has a positive or supportive attitude towards the target, either explicitly or implicitly. • AGAINST: The tweet opposes or criticizes the target, either explicitly or implicitly. • NONE: The tweet is neutral or doesn’t have a stance towards the target. Tweet: [tweet]

Question:

What is the stance expressed in the tweet towards the target "[target]"? Choose one of the following options: FAVOR, AGAINST, NONE.

Answer:

For this prompt structure, placeholders are utilized: [tweet], [target], and [topic]. • [tweet]: Represents the actual tweet being analyzed. • [target]: Denotes what or whom the tweet’s stance is directed at, whether directly or indirectly. • [topic]: Offers a brief description of the [target]. Specifically for the SemEval-2016 dataset, this description was crafted by us to facilitate the understanding of the tweet’s context. When fine-tuning, these placeholders are substituted with real data, making it easier for the model to understand the context and identify the stance.