DEEM: Dynamic Experienced Expert Modeling for Stance Detection

Recent work has made a preliminary attempt to use large language models (LLMs) to solve the stance detection task, showing promising results. However, considering that stance detection usually requires detailed background knowledge, the vanilla reasoning method may neglect the domain knowledge to make a professional and accurate analysis. Thus, there is still room for improvement of LLMs reasoning, especially in leveraging the generation capability of LLMs to simulate specific experts (i.e., multi-agents) to detect the stance. In this paper, different from existing multi-agent works that require detailed descriptions and use fixed experts, we propose a Dynamic Experienced Expert Modeling (DEEM) method which can leverage the generated experienced experts and let LLMs reason in a semi-parametric way, making the experts more generalizable and reliable.

Inspired by the wisdom of crowds in sociological theory (Minsky, 1988; Piaget, 2013), we intuitively propose designing multiple capable experts to collaborate in order to come up with a comprehensive stance prediction.

In particular, to better gather the potential expertise for stance detection, we first leverage labeled samples from the existing training data to generate diverse experts. Then, we design two heuristic rules, namely occurrence numbers and response accuracy, to filter the experienced experts and construct an expert repository. Finally, instead of using a fully generative approach, we adopt a dynamic retrieval method to identify relevant experienced experts for new sentences, facilitating discussions for the final prediction.

Second, it is well known that LLMs can occasionally

generate hallucinations (Guerreiro et al., 2023__; Ji et al.__, 2023__), thus they could make the responses unreliable and lead to incorrect final predictions. Therefore, experienced experts need to be accurate in analyzing the stance towards the target, thus the second heuristic rule is the total prediction accuracy Acc(°§) of each expert e__mj ∈ E__j : Acc(e__mj ) = P =e__i 1(ˆl__i = l__i__) e__i__∈__E 1(e__mj = e__i__) , (4) whereˆl__i is the predicted label corresponding to the i-th element e__i in Eq. 2__, and l__i is the ground-truth label for sentence s__i__. Finally, we discard the expert e__mj ∈ E__j who have low prediction accuracy as the threshold (e.g., Acc(e__mj ) < 50%).