Is Sarcasm Detection A Step-by-Step Reasoning Process in Large Language Models?

However, human sarcasm understanding is often considered an intuitive and holistic cognitive process, in which various linguistic, contextual, and emotional cues are integrated to form a comprehensive understanding, in a way that does not necessarily follow a step-bystep fashion. To verify the validity of this argument, we introduce a new prompting framework (called SarcasmCue) containing four sub-methods, viz. chain of contradiction (CoC), graph of cues (GoC), bagging of cues (BoC) and tensor of cues (ToC), which elicits LLMs to detect human sarcasm by considering sequential and non-sequential prompting methods. Through a comprehensive empirical comparison on four benchmarks, we highlight three key findings: (1) CoC and GoC show superior performance with more advanced models like GPT-4 and Claude 3.5, with an improvement of 3.5% ↑. (2) ToC significantly outperforms other methods when smaller LLMs are evaluated, boosting the F1 score by 29.7% ↑ over the best baseline.

RQ: Is human sarcasm detection a step-by-step reasoning process?

To answer this question, we propose a theoretical framework, called SarcasmCue, based on the sequential and non-sequential prompting paradigm. It consists of four prompting methods, i.e., chain of contradiction (CoC), graph of cues (GoC), bagging of cues (BoC) and tensor of cues (ToC). Each method has its own focus and advantages. In this work, cue is similar to thought, being a coherent language sequence related to linguistics, context, or emotion that serves as an intermediate indicator for identifying sarcasm, such as rhetorical devices or emotional words. More specifically,

• CoC. It harnesses the quintessential property of sarcasm (namely the contradiction between surface sentiment and true intention). It aims to: (1) identify the surface sentiment by extracting keywords, etc.; (2) deduce the true intention by scrutinizing rhetorical devices, etc.; and (3) determine the inconsistency between them. It is a typical linear structure.

• GoC. Generalizing over CoC, GoC frames the problem of sarcasm detection as a search over a graph and treats various cues as nodes, with the relations across cues represented as edges. Unlike CoC and ToT, it goes beyond following a fixed hierarchy or linear reasoning path. In summary, both CoC and GoC follow the step-bystep reasoning process.

• BoC. BoC is a bagging approach that constructs a pool of diverse cues and randomly sampling multiple cue subsets. LLMs are employed to generate multiple predictions based on these subsets, and such predictions are aggregated to produce the final result. It is a set-based structure.

• ToC. ToC treats each type of cues (namely linguistic, contextual, and emotional cues) as an independent, orthogonal view for sarcasm understanding and constructs a multi-view representation through the tensor product. It allows language models to leverage higher-order interactions among the cues. ToC can be visualized as a 3D volumetric structure. Hence, BoC and ToC are proposed based on the assumption that sarcasm detection is not a step-by-step reasoning process.

• Their correlation. These four methods represent an evolution from linear to nonlinear, and from a single perspective to multiple perspectives, together forming a comprehensive theoretical framework (SarcasmCue). Their design aims to adapt to various sarcasm detection scenarios.

We capture the inherent paradoxical nature of sarcasm, which is the incongruity between the surface sentiment and the true intention, and propose chain of contradiction, a CoT-style paradigm that allows LLMs to decompose the problem of sarcasm detection into intermediate steps and solve each before making decision (Fig. 2 (a)).

Step 1. We first ask LLM to detect the surface sentiment via the following prompt p1: Given the input sentence [X], what is the SURFACE sentiment, as indicated by clues such as keywords, sentimental phrases, emojis? c1 is the output sequence, which can be formulated as c1 ∼ LCoC θ (c1|X , p1).

Step 2. We thus ask LLM to carefully discover the true intention via the following prompt p2: Deduce what the sentence really means, namely the TRUE intention, by carefully checking any rhetorical devices, language style, unusual punctuations, common senses. c2 is the output sequence, which can be formulated as c2 ∼ LCoC θ (c2|X , c1, p2).

Step 3. Let LLM examine the consistency between surface sentiment and true intention and make the final prediction: Based on Step 1 and Step 2, evaluate whether the surface sentiment aligns with the true intention. If they do not match, the sentence is probably ‘Sarcastic’. Otherwise, the sentence is ‘Not Sarcastic’. Return the label only. CoC raises a presumption that the cues are linearly correlated, and detects human sarcasm through step-by-step reasoning. Further details see Algorithm 1 in App. A.