Paper deep dive
Sentiment analysis for software engineering: How far can zero-shot learning (ZSL) go?
Reem Alfayez, Manal Binkhonain
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 94%
Last extracted: 4/18/2026, 1:30:33 AM
Summary
This study evaluates the effectiveness of various Zero-Shot Learning (ZSL) techniques—including embedding-based, NLI-based, TARS-based, and generative-based models—for sentiment analysis in software engineering. The research addresses the challenge of annotated dataset scarcity by comparing ZSL performance against fine-tuned transformer models across seven software engineering datasets. Findings indicate that ZSL, particularly when using expert-curated labels with embedding or generative models, achieves performance comparable to supervised fine-tuned models, though subjectivity in annotation and polar facts remain primary causes of misclassification.
Entities (5)
Relation Signals (3)
Zero-Shot Learning → addresseschallenge → Annotated Dataset Scarcity
confidence 95% · This study explores the potential of ZSL to address the scarcity of annotated datasets in sentiment analysis within software engineering
Subjectivity in Annotation → causes → ZSL Misclassifications
confidence 90% · The error analysis revealed that subjectivity in annotation and polar facts are the main contributors to ZSL misclassifications.
ZSL Techniques → performscomparableto → Fine-tuned Transformer Models
confidence 90% · ZSL techniques... can achieve macro-F1 scores comparable to fine-tuned transformer-based models.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Sentiment analysis in software engineering focuses on understanding emotions expressed in software artifacts. Previous research highlighted the limitations of applying general off-the-shelf sentiment analysis tools within the software engineering domain and indicated the need for specialized tools tailored to various software engineering contexts. The development of such tools heavily relies on supervised machine learning techniques that necessitate annotated datasets. Acquiring such datasets is a substantial challenge, as it requires domain-specific expertise and significant effort. Objective: This study explores the potential of ZSL to address the scarcity of annotated datasets in sentiment analysis within software engineering Method:} We conducted an empirical experiment to evaluate the performance of various ZSL techniques, including embedding-based, NLI-based, TARS-based, and generative-based ZSL techniques. We assessed the performance of these techniques under different labels setups to examine the impact of label configurations. Additionally, we compared the results of the ZSL techniques with state-of-the-art fine-tuned transformer-based models. Finally, we performed an error analysis to identify the primary causes of misclassifications. Results: Our findings demonstrate that ZSL techniques, particularly those combining expert-curated labels with embedding-based or generative-based models, can achieve macro-F1 scores comparable to fine-tuned transformer-based models. The error analysis revealed that subjectivity in annotation and polar facts are the main contributors to ZSL misclassifications. Conclusion: This study demonstrates the potential of ZSL for sentiment analysis in software engineering. ZSL can provide a solution to the challenge of annotated dataset scarcity by reducing reliance on annotated dataset.
Tags
Links
- Source: https://arxiv.org/abs/2604.13826v1
- Canonical: https://arxiv.org/abs/2604.13826v1
Trouble viewing inline? Open PDF directly →
Full Text
75,404 characters extracted from source content.
Expand or collapse full text
Sentiment analysis for software engineering: How far can zero-shot learning (ZSL) go? Reem Alfayez Manal Binkhonain Department of Software Engineering, College of Computer and Information Sciences, King Saud University, P.O. Box 51178, Riyadh 11543, Saudi Arabia reealfayez@ksu.edu.sa Received: date / Accepted: date Abstract Context: Sentiment analysis in software engineering focuses on understanding emotions expressed in software artifacts. Previous re- search highlighted the limitations of applying general off-the-shelf sen- timent analysis tools within the software engineering domain and indicated the need for specialized tools tailored to various software engineering contexts. The development of such tools heavily relies on supervised machine learning techniques that necessitate annotated datasets. Acquiring such datasets is a substantial challenge, as it requires domain-specific expertise and significant effort. Objective: This study explores the potential of zero-shot learning (ZSL) to ad- dress the scarcity of annotated datasets in sentiment analysis within software engineering Method: We conducted an empirical experi- ment to evaluate the performance of various ZSL techniques, including embedding-based, natural language inference (NLI)-based, task-aware representation of sentences (TARS)-based, and generative-based ZSL techniques. We assessed the performance of these techniques under different labels setups to examine the impact of label configurations. Additionally, we compared the results of the ZSL techniques with 1 arXiv:2604.13826v1 [cs.SE] 15 Apr 2026 state-of-the-art fine-tuned transformer-based models. Finally, we per- formed an error analysis to identify the primary causes of misclassifica- tions. Results: Our findings demonstrate that ZSL techniques, par- ticularly those combining expert-curated labels with embedding-based or generative-based models, can achieve macro-F1 scores comparable to fine-tuned transformer-based models. The error analysis revealed that subjectivity in annotation and polar facts are the main contribu- tors to ZSL misclassifications. Conclusion: This study demonstrates the potential of ZSL for sentiment analysis in software engineering. ZSL can provide a solution to the challenge of annotated dataset scarcity by reducing reliance on annotated dataset. Keywords: Sentiment Analysis, Software Engineering, Natural Lan- guage Processing, Zero-shot Learning, Text Classification 1 Introduction Over the years, sentiment analysis has evolved as a powerful tool for ex- tracting subjective information from text data [1]. In software engineering, it provides insights into software development and usage by analyzing app reviews, developer communications, discussions on technical Q&A websites, and more [1, 2, 3, 4, 5, 6, 7, 8]. Utilizing sentiment analysis within the software engineering domain presents significant challenges. General-purpose sentiment analysis tools often demon- strate suboptimal performance when used in software engineering contexts [9, 10]. This limitation has led researchers to develop specialized tools tailored to the unique characteristics of software engineering contexts. Despite these efforts, it has been observed that even sentiment analysis tools that perform well in one software engineering context lack generalizability across different contexts, where they underperform in the new contexts. The context-bound limitation highlights the need for context-specific tools [3, 4, 11]. Developing context-based tools is not an easy feat, as most sentiment analysis tools in software engineering are based on supervised machine learn- ing techniques that frame sentiment analysis tasks as text classification prob- lems. These tools are heavily reliant on annotated datasets for training, which is costly, time-consuming, error-prone, and requires domain-specific expertise to obtain [2, 4, 11, 12]. This reliance on annotated datasets is particularly problematic due to the context-bound limitation that requires the development of separate models and datasets for each specific context [4, 11]. The need to curate anno- tated dataset for training in each context further intensifies the challenge 2 of datasets scarcity and complicates the development of effective sentiment analysis models [2, 4, 11]. Zero-shot learning (ZSL) is a promising approach that has the potential to address the challenge of requiring context-specific training data for classi- fication tasks. A ZSL-based model can classify data without prior exposure to specific labels, where it instead relies on its understanding of relationships between words, phrases, and concepts to make classifications. In the context of ZSL, a model generates predictions for tasks it has not been explicitly trained on by leveraging data from other, related tasks to aid its learning process. The model utilizes knowledge acquired through pre-training on other datasets and transfers relevant information to the new classification task [13, 14]. This study explores the potential of ZSL for sentiment analysis in soft- ware engineering by evaluating embedding-based, natural language infer- ence (NLI)-based, task-aware representation of sentences (TARS)-based, and generative-based ZSL techniques across various contexts. We assess the im- pact of label configurations on model performance and compare the best- performing ZSL models with state-of-the-art fine-tuned transformer models. An error analysis is also conducted to understand misclassification causes. The rest of the paper is organized as follows: Section 2 summarizes ZSL for text classification. Section 3 reviews related studies. Section 4 describes the study setup. Section 5 presents the results, and Section 6 discusses them. Section 7 outlines potential validity threats, and Section 8 concludes the paper. 2 Zero-shot learning (ZSL) text classification ZSL text classification is a natural language processing (NLP) approach that enables models to classify text into unseen classes during training. As op- posed to traditional supervised learning, which requires labeled data for each class, ZSL leverages transfer learning and semantic understanding to pre- dict previously unseen classes. This capability is particularly useful when labeled data is scarce [13, 14]. ZSL text classification can be achieved through four main techniques: embedding-based, NLI-based, TARS-based, and generative-based techniques. Below is a brief description of each tech- nique. 3 2.1 Embedding-based ZSL Embedding-based ZSL text classification uses word embeddings to measure the semantic similarity between input text and potential class labels [15]. While the original approach [15] relied on skip-gram static embeddings, we opted for transformer-based large language models (LLMs) embeddings, as these models generate contextual word embeddings that capture both the syntactic and semantic properties of words, along with their context [13, 14, 16]. Figure 1 illustrates the process of embedding-based ZSL text classifica- tion. Both the input text and potential class labels are passed through a pre-trained LLM to generate embeddings. Classification is performed by cal- culating the cosine similarity between the input text embedding and each class label embedding. The class with the highest similarity score is then selected as the predicted label (i.e., label 1 in the example). LLMCosine similarity Text and label 1 = .96 Text and label 2 = .04 Resulting scores Text embedding Label 1 embedding Resulting embeddings Label 2 embedding Input Set of labels Label 1 Label 2 Text Figure 1: An illustration of embedding-based ZSL 2.2 Natural language inference (NLI)-based ZSL NLI-based ZSL frames text classification as a textual entailment problem, where it determines whether a given text (i.e., premise) logically follows an- other hypothesis. The input text is treated as the NLI premise, and each candidate label forms a hypothesis. The model calculates probabilities for en- tailment and contradiction, which are then converted into label probabilities. The text is classified under the label with the highest entailment probability [14, 17]. As Figure 2 presents, the input text serves as the premise, and potential class labels are hypotheses. The NLI model assesses whether the premise entails or contradicts each hypothesis and assigns a probability to each case. The input text is classified based on the label with the highest entailment probability, in this case, label 1. 4 NLI model Text and label 1 = .96 Text and label 2 = .04 Resulting probabilitiesInput Hypothesis 1: label 1 Hypothesis 2: label 2 Premise: text Figure 2: An illustration of NLI-based ZSL 2.3 Task-aware representation of sentences (TARS)- based ZSL TARS formulates the classification task as a universal binary classification problem, where the model learns to predict whether a given text belongs to a particular label or not. Instead of training separate models for each label, TARS simultaneously evaluates the relevance of the text for all labels by adapting LLM representations through label-conditioned embeddings [18]. As Figure 3 illustrates, the input to TARS consists of the text to be classified and a set of candidate labels. TARS generates embeddings condi- tioned on both the text and each label by appending the label to the text to form queries. These queries are processed by a shared transformer encoder to produce task-specific embeddings that capture the semantic relationships between the text and the labels. A binary prediction of true or false is then performed for each label. The label with the highest true confidence is se- lected as the final classification (i.e., label 1 in the example) [18]. Transformer-based encoder Q1 = .96 Q2 = .04 Resulting true scores Q1: < Label 1, text> Queries Q2: < Label 2, text> Input Set of labels Label 1 Label 2 Text Figure 3: An illustration of TARS-based ZSL 2.4 Generative-based ZSL Transformer-based generative models, such as OpenAI ’s Generative Pre- Trained Transformers (GPTs) 1 , are capable to perform ZSL text classifica- 1 https://openai.com/ 5 tion by generating text in response to provided input [19]. In this approach, as Figure 4 depicts, the model receives a prompt that provides specific instructions on how to classify input text and the input text. The provided instruction is in natural language, and it may not include any demonstrations. The model then generates a response that indicates the most appropriate class based on the provided prompt [14, 19]. Generative model Output Generated label Input Prompt Figure 4: An illustration of generative-based ZSL 3 Related work Many studies have assessed sentiment analysis tools, explored the impact of sentiment on software development practices, and more. Due to space constraints, we focus on summarizing (1) systematic reviews related to sen- timent analysis tools in software engineering and (2) research efforts on the development of such tools. 3.1 Systematic reviews on sentiment analysis tools for software engineering S ́anchez-Gord ́on and Colomo-Palacios [20] conducted a systematic literature review (SLR) on software developers’ emotions. The study highlighted the limited research in this domain and noted that although current approaches are recognized as unreliable, many techniques with the potential to enhance the detection of developers’ emotions remain underutilized or unexplored. Obaidi and Kl ̈under [3] conducted an SLR on software engineering sen- timent analysis tools. The study found that most research relies on using existing tools, support vector machine (SVM) is the most utilized technique, and open-source software (OSS) projects are the main data source. The study highlighted the challenges of training data scarcity, inconsistent tool performance, and the subjectivity of annotated data. The study also noted that sarcasm and irony detection remains a major challenge in sentiment analysis for software engineering. 6 In a follow-up systematic mapping review (SMR), Obaidi et al. [21] expanded the previous analysis to include recent sentiment analysis studies in software engineering. The study confirmed that research still predominantly applies existing sentiment analysis tools rather than developing new ones, OSS data remains the most used soure, SVM and gradient boosting tree (GBT) were identified as the most common supervised learning techniques, and that fine-tuned transformer models outperform other approaches. The study also highlighted the unreliability of general sentiment analysis tools and the need for customization to specific software engineering contexts. Lin et al. [12] conducted an SLR that identified sentiment analysis tools currently in use and raised concerns about their application in unintended domains without proper validation. The review summarized comparisons of these tools, identified publicly available datasets, and highlighted challenges in software engineering sentiment analysis, such as identifying neutral senti- ment. The study emphasized that model quality depends on training dataset quality and noted the considerable effort needed to train supervised machine learning models. 3.2 Sentiment analysis tools for software engineering Several sentiment analysis tools have been developed for software engineering utilizing a variety of NLP and machine learning approaches. Heuristic-based methods were employed by Islam et al. [22] and Islam and Zibran [23]. SVM were utilized by Calefato et al. [8], Islam et al. [24], Murgia et al. [25], and Cagnoni et al. [26]. Ensemble approaches were adopted by Uddin et al. [27], Ahmed et al. [28], and Ding et al. [29]. Aiming to leverage transformer-based models, Biswas et al. [30] intro- duced BERT4SentiSE, a BERT-based sentiment classifier fine-tuned on Stack Overflow (SO) posts. Zhang et al. [1] assessed the performance of BERT, RoBERTa, XLNet, and ALBERT pre-trained transformer models across var- ious software engineering contexts, and they found that fine-tuning these models outperforms state-of-the-art sentiment analysis tools. Similarly, Ba- tra et al. [31] evaluated fine-tuned BERT, ensemble BERT models, and compressed BERT for sentiment analysis on SO posts, GitHub commit com- ments, and Jira issue comments. The study found that compressed BERT and ensemble BERT produced better similar results, with the compressed version recommended for resource conservation. Bleyl et al. [32] developed a fine-tuned BERT model for detecting emotions in SO posts. Additionally, Sun et al. [33] introduced EASTER, a sentiment analysis tool that integrates RoBERTa as the embedding layer in TextCNN. EASTER was evaluated on app reviews, Jira issue comments, and SO posts. 7 Shafikuzzaman et al. [34] evaluated the performance of twelve pretrained language models, including fine-tuned models, SentiStrength-SE, and Sen- tiCR [28] on the Gerrit, GitHub, Google Play, Jira, and SO posts datasets. The study found that model performance varied across datasets, with fine- tuned models performing better on larger datasets. To better understand models’ behavior, the study used Shapley Additive Explanations (SHAP) to conduct error analysis. Leveraging generative-based models, Zhang et al. [2] explored the potential of Llama 2-Chat; Vicuna; and WizardLM, re- ferred to as larger language models (bLLMs), for sentiment analysis. The study assessed their performance using ZSL and few-shot learning (FSL) on sentiment analysis tasks across five datasets: Gerrit, GitHub, Google Play, Jira, and SO. Due to cost concerns, the study was conducted on a strati- fied representative sample drawn from 10% of each dataset that represents the test set. The findings indicated no significant performance difference between ZSL and FSL and that FSL did not necessarily outperform ZSL. The analysis was extended by fine-tuning BERT, RoBERTa, ALBERT, XL- Net, and DistilBERT, referred to as smaller language models (sLLMs). The comparison of the sLLMs to the bLLMs revealed that bLLMs outperformed sLLMs on imbalanced datasets or those with limited training data, while fine-tuned sLLMs performed better when ample training data and balanced distributions were available. 4 Study setup This section describes the setup of the study, including the goal and research questions (RQs), datasets, selected LLMs, label curation and configuration, performance measures, statistical analysis, and implementation details. 4.1 Goal and research questions (RQs) The goal of this study is defined using the Goal-Question-Metric (GQM) template [35], as follows: Assessing the performance of ZSL in sentiment classification within the context of API reviews, code review comments, pull requests and commit comments, developer messages, mobile app reviews, issue comments, and posts on technical question-and-answer websites in the software engineering domain. To achieve this goal, we formulated the following RQs: • RQ1: Which ZSL technique is most effective for sentiment classifica- tion, and among the techniques that evaluate multiple models, which model demonstrates the best performance? 8 • RQ2: Do different label configurations have an impact on the perfor- mance of ZSL-based sentiment classification? • RQ3: How does the performance of ZSL-based models compare with that of the state-of-the-art fine-tuned transformer-based models in sen- timent classification? • RQ4: What factors contribute to the misclassification of sentiment labels in ZSL-based models, and how do these compare with those shared with the state-of-the-art fine-tuned transformer-based models? 4.2 Datasets To address the RQs, we utilized seven publicly available datasets commonly used for sentiment analysis in software engineering. Table 1 summarizes these datasets, and a description of each is provided below. API reviews: The dataset, curated by Uddin and Khomh [7], contains 4,522 sentences from 1,338 SO posts across 71 threads tagged with 18 Java API-related keywords. The dataset includes 890 positive, 496 negative, and 3,136 neutral sentences. Gerrit: The dataset, curated by Ahmed et al. [28], includes 1,600 code review comments mined from the code review repositories of 20 open-source projects, with 398 labeled as negative and 1,202 as non-negative. GitHub: The dataset, curated by Novielli et al. [6], consists of 7,122 GitHub pull request and commit comments, labeled as 2,013 positive, 2,087 negative, and 3,022 neutral. Gitter: The dataset, curated by Sajadi et al. [5], contains 400 developer messages from 10 Gitter communities. Messages were annotated for six basic emotions (i.e., anger, love, fear, joy, sadness, and surprise) and subcategories based on Shaver’s emotion taxonomy [36]. Following the approach of [4], we mapped love and joy to positive, and anger and sadness to negative, resulting in 201 messages with 127 positive and 74 negative. Google Play: The dataset, curated by Lin et al. [4], contains 341 Android app reviews from Google Play, with 186 labeled as positive, 130 as negative, and 25 as neutral. Jira: The dataset, curated by Lin et al. [4], contains 926 sentences from Jira issue comments, with 636 labeled as negative and 290 as positive. SO: The dataset, curated by Calefato et al. [8], includes 4,423 SO posts, with 1,527 positive, 1,202 negative, and 1,694 neutral. 9 Table 1: Summary of utilized datasets Dataset NameTotalPolarity distribution API reviews4,522Positive (890), negative (496), and neutral (3,136) Gerrit1,600Negative (398) and non-negative (1,202) GitHub7,122Positive (2,013), negative (2,087), and neutral (3,022) Gitter201Positive (127) and negative (74) Google Play341Positive (186), negative (130), and neutral (25) Jira926Positive (290) and negative (636) SO4,423Positive (1,527), negative (1,202), and neutral (1,694) 4.3 Large language model (LLM) selection To address the aforementioned research questions (RQs), we selected the models below. Table 2 summarizes these models, with unique identifier for easy reference throughout the study. Our selection was guided by four key considerations. First, we prior- itized reproducibility and accessibility by including publicly available and widely adopted pretrained models such as BERT, RoBERTa, and ALBERT, which are well-established benchmarks in ZSL and transfer learning research [2, 16]. Second, to ensure architectural diversity, we selected transformer vari- ants trained with distinct pretraining objectives, including masked language modeling, permutation modeling, and next-sentence prediction. Third, we sought domain variation by incorporating both generic and domain-specific models. For instance, BERTOverflow for technical Q&A, RoBERTa-base- go emotions for emotions, and Twitter-RoBERTa for social media sentiment analysis. Finally, to capture a broad availability spectrum, we included both paid and unpaid models. • Embedding-based ZSL:We selected the following unpaid, generic- embeddings: BERT-base-uncased 2 , RoBERTa-base 3 , DistilBERT- base-uncased 4 , ALBERT-base-v2 5 , XLNet-base-cased: 6 , and All- MiniLM-L12-v2 7 . Besides the aforementioned unpaid, generic models, we utilized the following unpaid, domain-specific models: BERTOverflow 8 ,RoBERTa- 2 https://huggingface.co/google-bert/bert-base-uncased 3 https://huggingface.co/FacebookAI/roberta-base 4 https://huggingface.co/distilbert/distilbert-base-uncased 5 https://huggingface.co/albert/albert-base-v2 6 https://huggingface.co/xlnet/xlnet-base-cased 7 https://huggingface.co/sentence-transformers/all-MiniLM-L12-v2 8 https://huggingface.co/jeniya/BERTOverflow 10 base-goemotions 9 , and Twitter-RoBERTa-base-sentiment 10 . We also included three paid OpenAI embeddings 11 : Text-embedding- ada-002, Text-embedding-3-small, and Text-embedding-3-large. • NLI-based ZSL: To evaluate the performance of NLI-based ZSL, we selected the following four models that are specialized for NLI tasks: RoBERTa-large-mnli 12 ,Cross-encoder/nli-deberta-base 13 , BART-large- mnli 14 , and DeBERTa-v3-large-mnli-fever-anli-ling-wanli 15 . • TARS: We used the model implementation from the original paper that introduced the technique [18]. • Generative-based ZSL: For generative-based ZSL, we used GPT- 3.5 Turbo (gpt-3.5-turbo-0125 16 ), as it was the most viable option in terms of efficiency and cost at the time of conducting the study (i.e., May 2024). Table 2: Selected models for each ZSL technique ApproachModelIdentifier Embedding-based ZSLBERT-base-uncasedEM1 RoBERTa-baseEM2 DistilBERT-base-uncasedE M3 ALBERT-base-v2EM4 XLNet-base-casedEM5 All-MiniLM-L12-v2EM6 BERTOverflowE M7 RoBERTa-base-goemotionsEM8 Twitter-RoBERTa-base-sentimentEM9 Text-embedding-ada-002EM10 Text-embedding-3-smallE M11 Text-embedding-3-largeEM12 NLI-based ZSLRoBERTa-large-mnliNM1 Cross-encoder/nli-deberta-baseNM2 BART-large-mnliN M3 DeBERTa-v3-large-mnli-fever-anli-ling-wanliNM4 TARS-based ZSLTARSTM1 Generative-based ZSLGPT-3.5 Turbo (gpt-3.5-turbo-0125)GM1 9 https://huggingface.co/SamLowe/roberta-base-go emotions 10 https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment 11 https://openai.com/index/new-embedding-models-and-api-updates/ 12 https://huggingface.co/FacebookAI/roberta-large-mnli 13 https://huggingface.co/cross-encoder/nli-deberta-base 14 https://huggingface.co/facebook/bart-large-mnli 15 https://huggingface.co/MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli 16 https://platform.openai.com/docs/models#gpt-3-5-turbo 11 4.4 Label curation and configuration To investigate the impact of label configurations on sentiment analysis, we compare three distinct types: the original dataset, expert-curated, and LLM- generated labels. These configurations differ in phrasing, contextual speci- ficity, and descriptive granularity. While the original labels perform well on standard benchmarks, we aim to explore whether enriching them with descriptions and contextual cues can improve model performance in a ZSL setting. Specifically, including contextual information about the dataset in- stance type may help models better interpret the input, while adding senti- ment descriptors can enhance the semantic richness of the labels and improve embedding quality. We include both expert-curated and LLM-generated labels to investigate two approaches to enriching label semantics. Expert-curated labels offer human-level domain insight, with the potential to capture subtle distinctions and contextual relevance that may be overlooked in the original labels. In contrast, LLM-generated labels provide a scalable, automated alternative that reflects the model’s own interpretation of sentiment. Comparing both approaches allows us to assess the trade-offs between human judgment and automated label generation and to examine whether either leads to improved performance in ZSL settings over the original labels. Table 3 summarizes these labels with identifiers and examples. Moreover, our online appendix includes the full set of utilized labels 17 . It is important to note that all alternative labels were derived by mapping to the original label set, with no reannotation of the dataset instances involved. • Original labels: We used the original sentiment labels from the datasets as described in Section 4.2. • Expert-curated labels: The two authors independently created la- bels based on their understanding of the sentiment classes and dataset context. These labels were then reviewed and consolidated in a joint meeting, where two types of disagreements emerged. Phrase disagree- ments, in which one author used “with” while the other used “has” to describe instances. After discussion, the authors decided that using “with” was more appropriate. The other type is content disagreements, where one author used the original set of emotions mapped to senti- ments, while the other did not. The authors decided to retain these labels, as they could improve understanding of the impact of labels. 17 https://osf.io/gzt9r/?view_only=afd4a24f2d724413a5423eac0cdcfa6 12 • LLM-generated labels: We used ChatGPT-3.5 to generate labels us- ing the following prompt: “Generate a list of words that best describe positive sentiment.”. The term “positive” was replaced with “nega- tive” to generate words for negative sentiment, and the results of both were negated to generate neutral labels. The result formed two label configurations: L6 using ChatGPT-suggested words and L7 using a combination of suggested words and corresponding sentiment classes. 4.5 Performance measures To evaluate the performance of the models, we calculated both macro-F1 and micro-F1 scores, which are variations of the F1 score [37], following the approach of previous, related work [2, 6]. Macro-F1 calculates the F1 score for each class independently and averages them, while micro-F1 aggregates the contributions of all classes to compute the average. 4.6 Statistical analysis Merely comparing performance measures is insufficient, as observed differ- ences may arise due to random variability [38]. To assess the significance of these differences, we use the non-parametric Scott-Knott Effect Size Differ- ence (ESD) test, which produces distinct, non-overlapping groups and quan- tifies the magnitude of meaningful median differences [39]. The test is robust against outliers and does not assume homogeneity, normality, or sample size [40], and it has been successfully applied in similar contexts [39]. While we report both macro-F1 and micro-F1 scores for comprehensive evaluation, we base our comparisons on the macro-F1 score, as it better handles imbalanced datasets by giving equal weight to all classes, aligning with previous studies [2, 6, 37]. 4.7 Implementation To implement the empirical assessment, we followed a series of steps for each RQ. For RQ1 and RQ2, we used all datasets, models, and label configurations as described above. For the generative-based ZSL, we used the template: “What is the sentiment of the following app review, which is delimited with triple backticks?” Give your answer as either ‘positive’, ‘negative’, or ‘neu- tral’.” We replaced the term “app review” with the type of each examined dataset and substituted the labels “positive,” “negative,” or “neutral” with 13 Table 3: Summary of label configurations IdentifierCategoryDescriptionExample L1OriginalLabels used as originally described in each dataset, where emotions were mapped to corresponding sentiments (i.e., joy and love to positive; anger and sadness to neg- ative). Positive L2Expert-curatedLabels that use the term sentiment to describe the type of instances in each dataset, where neutral is described using negations of both sentiments. A positive app review L3Expert-curatedLabels describing the type of dataset in- stances with attached sentiment using “with” along with the term “sentiment”, where neutral is described using negations of both sentiments. An app review with posi- tive sentiment L4Expert-curatedLabels describing the type of dataset in- stances with attached sentiment and as- sociated original emotions mapped to the sentiment (i.e., joy and love for posi- tive; anger and sadness for negative) us- ing “with” and the term “sentiment” , where neutral is described using negations of both sentiments and their associated emotions. An app review with pos- itive, joy, or love senti- ments L5Expert-curatedLabels describing the type of dataset in- stances with only the emotions mapped to the sentiment (i.e., joy and love for posi- tive; anger and sadness for negative) us- ing “with”, where neutral is described by negating both the sentiments associated emotions. An app review with joy or love sentiments L6LLM-generatedLabels describing the type of dataset in- stances with only words generated by the LLM using “with”, where neutral is de- scribed by negating both sentiments asso- ciated words. An app review with cheer- fulness, happiness, amuse- ment, satisfaction, bliss, gaiety, glee, jolliness, jovi- ality, joy, delight, enjoy- ment, gladness, jubilation, elation, ecstasy, euphoria, zest, enthusiasm, excite- ment, thrill, zeal, exhila- ration, contentment, plea- sure, and optimism senti- ments L7LLM-generatedLabels describing the type of dataset in- stances with sentiment and words gener- ated by the LLM using “with”, where neu- tral is described by negating both the sen- timents and their associated words. Anappreviewwith positive,cheerfulness, happiness,amusement, satisfaction, bliss, gaiety, glee, jolliness, joviality, joy, delight, enjoyment, gladness, jubilation, ela- tion, ecstasy, euphoria, zest,enthusiasm,ex- citement,thrill,zeal, exhilaration, contentment, pleasure, and optimism sentiments 14 the specific labels being assessed. We used the default parameters provided by the generative model’s API, with the exception of setting the temperature to zero to reduce variability in the outputs. To ensure consistency between the model outputs and the gold labels, we applied simple post-processing rules. For the shorter labels, we checked whether the generated output explicitly mentioned the sentiment name (i.e., positive, negative, neutral) and mapped it to the corresponding label. For the longer labels (i.e., L6 and L7), the model occasionally produced partial matches by omitting certain parts; in these cases, we post-processed the outputs to align them with the original labels. For RQ3, following [2], we partitioned each dataset into training, valida- tion, and test sets in an 8:1:1 ratio with stratified splitting. We fine-tuned state-of-the-art transformer models listed in Table 4. We note that our evaluation does not include earlier SE-specific sentiment analysis tools, such as SentiStrength-SE. This decision is supported by the findings of Zhang et al. [1], who demonstrated that fine-tuned transformer- based models outperform these tools and concluded that such models should be regarded as the state of the art for sentiment analysis in the SE domain. Furthermore, in a more recent work [2], Zhang et al. followed this conclusion by excluding these earlier tools entirely and focusing their evaluation solely on fine-tuned transformer-based models. In line with this direction, we adopt a similar approach and benchmark our results against state-of-the-art trans- former models. We used a learning rate of 2× 10 −5 , 5 epochs, batch size of 32, and a max sequence length of 256 tokens. The model with the highest macro-F1 score on the validation set was evaluated on the test set with original labels (i.e., L1). We compared the performance of fine-tuned models with the best- performing ZSL model-label combinations and the best-performing model for each ZSL technique when paired with L1. For RQ4, we conducted quantitative and qualitative analyses on misclas- sifications from RQ3. The quantitative analysis identified common misclas- sified instances among ZSL-based and fine-tuned models. The qualitative analysis categorized these misclassifications using the framework of Novielli et al. [11]. We independently categorized the commonly misclassified in- stances of the ZSL-based models. Then we compared the results in a joint session and calculated Cohen’s kappa coefficient, which was 0.71, indicating moderate agreement [41]. Disagreements were resolved through discussion, with each author providing justification until consensus was reached. 15 Table 4: Selected models for fine-tuning ModelIdentifier BERT-base-casedFM1 RoBERTa-baseF M2 DistilBERT-base-uncasedFM3 ALBERT-base-v1F M4 XLNet-base-casedF M5 5 Results This section summarizes the results of the study, and our online appendix includes more detailed results for each RQ. 5.1 RQ1: Which ZSL technique is most effective for sentiment classification, and among the techniques that evaluate multiple models, which model demon- strates the best performance? The results of the examined ZSL techniques on each dataset are summarized in Table 5. The “Mac” and “Mic” columns refer to the macro-F1 score and micro-F1 score values, respectively, with the highest values of the macro-F1 and micro-F1 scores for each dataset are bolded. As presented in the table, generative-based ZSL achieved the highest macro-F1 scores across most datasets, with exceptions in the Gitter, Google Play, and Jira datasets. N M4 and NM2 outperformed others on the Git- ter and Google Play datasets, respectively, and E M9 achieved the highest macro-F1 score for the Jira dataset. Within the embedding-based ZSL models, E M9 achieved the highest macro-F1 scores across most datasets (i.e., 5 out of 7). Among the NLI- based models, NM2 can be considered the best performer, where it achieved the highest macro-F1 score in 5 out of 7 datasets. The results of the statistical test confirmed the above findings, where Figure 5 depicts the results. The generative-based ZSL model was ranked as the best performing model among all examined models. The second rank predominantly comprised NLI-based ZSL models and one embedding-based ZSL model (i.e., EM9). The third rank included the remaining NLI-based ZSL model (i.e., NM3), the TARS-based model, and another embedding- based model (i.e., EM11). The rest of the embedding-based models were distributed from the fourth rank to the last rank (i.e., rank 9). The results highlight the dominance of generative-based and NLI-based 16 ZSL techniques, with a few embedding-based models demonstrating compet- itive performance. Table 5: Summary of the performance of ZSL-based models ModelAPIGerritGitHubGitterGoogleJiraSO reviewsPlay MacMicMacMicMacMicMacMicMacMicMacMicMacMic Embedding-based ZSL EM10.370.480.20.250.260.390.290.380.270.360.610.750.330.4 EM20.120.130.430.750.180.30.330.390.190.380.480.70.170.28 EM30.370.480.210.260.40.410.470.520.270.360.590.60.50.5 E M40.080.120.430.750.160.30.290.380.180.380.410.690.140.27 EM50.220.240.50.620.290.310.290.380.270.360.520.580.280.33 EM60.290.340.490.530.420.420.620.620.380.480.670.730.350.36 E M70.260.330.210.250.320.360.840.860.280.440.420.420.30.38 EM80.410.490.430.630.560.570.860.860.540.690.70.750.660.67 E M90.480.50.430.750.60.610.90.90.620.870.960.960.670.69 E M100.340.350.50.510.520.520.820.840.630.750.810.820.570.58 EM110.410.490.550.720.540.540.840.850.570.70.850.860.570.57 E M120.360.40.460.470.480.490.790.810.620.770.80.810.540.56 NLI-based ZSL NM10.410.420.660.710.560.570.890.90.610.850.930.940.630.65 NM20.40.430.690.760.580.580.880.880.710.870.940.950.640.67 NM30.350.350.620.70.50.530.90.90.630.880.910.920.580.62 NM40.430.460.650.690.560.590.910.910.640.890.940.940.630.66 TARS-based ZSL TM10.490.630.640.70.490.520.770.780.590.670.890.90.620.62 Generative-based ZSL GM10.520.580.750.80.730.730.90.910.670.890.880.890.720.73 5.2 RQ2: Do different label configurations have an im- pact on the performance of ZSL-based sentiment classification? The results of varying the utilized labels within each embedding-based model are presented in Table 6, where the highest values for both macro-F1 and micro-F1 scores for each dataset are bolded. We note that, due to space limitations, the table only includes model-label combinations that yielded the highest macro-F1 score for a dataset within each model. We provide the results of all model-label combinations in our online appendix. Two label combinations of EM9 achieved the highest macro-F1 scores in the majority of datasets, with the exception of Google Play, where the combination EM12L3 attained the highest macro-F1 score. Specifically, EM9L1 achieved the highest value on the Jira dataset, while EM9L3 17 123456789 G_M1 E_M9 N_M1N_M2N_M4 E_M11 N_M3 T_M1 E_M8 E_M10E_M12 E_M6E_M3E_M7E_M1E_M5E_M2E_M4 0.00 0.25 0.50 0.75 1.00 Model Macro−F1 score Rank Figure 5: Scott-Knott ESD ranking for ZSL models based on macro-F1 score achieved the highest macro-F1 scores across the remaining five datasets. To identify the overall best embedding-based model-label combination, we conducted a statistical analysis. Figure 6 presents the results, where only the top five ranked combinations are included due to space constraints. As the figure demonstrates, the analysis confirms the superiority of E M9L3, as it was ranked as the top-preformer along with E M9L2. Additionally, EM8L2, EM8L3, EM9L4, and EM9L5 were ranked in second place while EM9L1 was ranked third. Other combinations followed in the re- maining ranks. The analysis revealed that both E M8 and EM9 achieved higher results with varying label combinations compared to others. Table 7 summarizes the results of combining each NLI-based model with label configurations, where the highest macro-F1 and micro-F1 scores for each dataset are bolded. N M2L1 has contributed to the highest macro-F1 score across all NLI-model-label combinations in 3 out of 7 datasets. Moreover, N M4L3 achieved the highest values in 2 datasets. The combinations of NM3L2, NM3L3, NM3L5, NM4L2, and NM4L5 were able to achieve the highest macro-F1 score on a dataset. Figure 7 presents the results of the Scott-Knott ESD test, where only the top 5 ranks are included due to space limitations. The results demonstrate the superiority of the N M4L3 combination, which was ranked first among all NLI-based combinations. Both the combinations of NM4 with L2 and L5 labels were ranked second, along with the NM1L2 and NM2L1 combi- nations. The remaining combinations were distributed across the remaining ranks. 18 Table 6: Summary of the performance of best-performing model-label com- binations of each embedding-based model across each dataset M&LAPIGerritGitHubGitterGoogleJiraSO reviewsPlay MacMicMacMicMacMicMacMicMacMicMacMicMacMic EM1L20.220.250.20.250.230.440.540.550.050.070.720.720.280.38 EM1L30.450.570.20.250.490.50.430.460.040.070.90.920.480.53 EM1L40.220.230.430.750.270.330.290.380.370.470.590.740.320.33 E M1L70.160.210.450.710.150.280.390.630.270.550.240.310.270.39 EM2L20.340.430.450.680.340.370.620.710.280.520.450.490.330.37 E M2L30.160.190.440.740.440.510.650.680.380.470.430.680.320.36 EM2L40.20.220.440.550.40.460.630.640.290.380.650.660.350.39 EM2L50.280.40.470.720.350.460.420.430.240.290.640.680.280.37 EM3L10.370.480.210.260.40.410.470.520.270.360.590.60.50.5 EM3L20.270.260.430.750.590.590.560.570.050.070.740.750.560.56 E M3L50.250.240.440.680.260.340.730.750.520.670.380.410.370.42 EM4L20.370.480.440.720.60.620.860.860.160.160.380.380.440.44 EM4L30.190.180.280.280.170.30.280.370.060.090.410.690.480.53 EM4L40.220.210.20.250.280.340.290.380.580.720.660.670.320.33 EM4L50.090.120.20.250.310.340.610.620.560.70.370.40.290.32 EM5L20.320.650.290.30.270.310.340.370.330.470.560.60.350.39 EM5L30.350.450.360.360.40.410.550.560.290.350.530.610.380.38 EM5L50.230.220.240.270.250.270.470.470.330.380.440.680.250.28 EM5L60.340.560.510.690.270.40.440.450.180.170.490.550.30.39 E M5L70.340.550.490.70.280.390.380.620.180.180.480.550.310.38 EM6L30.360.390.50.560.530.530.740.770.380.560.740.750.440.44 E M6L40.390.480.470.690.430.440.760.780.310.350.730.740.420.44 EM6L70.330.540.460.50.460.460.720.740.510.680.80.820.430.46 EM7L10.260.330.210.250.320.360.840.860.280.440.420.420.30.38 EM7L20.170.240.480.690.20.30.550.660.320.460.440.440.270.35 E M7L30.220.280.430.750.270.440.580.650.230.360.470.470.310.39 EM7L40.270.290.480.640.320.420.430.430.140.150.420.470.290.31 EM7L50.280.440.460.640.310.430.460.460.150.150.460.470.270.34 EM8L20.530.650.610.770.680.690.90.910.620.750.70.740.740.75 EM8L30.520.690.560.780.670.680.910.910.610.740.860.870.780.79 EM9L10.480.50.430.750.60.610.90.90.620.870.960.960.670.69 E M9L20.520.650.740.790.720.710.910.920.630.850.920.930.770.77 EM9L30.550.660.740.820.760.760.910.920.640.850.920.930.790.79 EM9L40.480.540.670.80.70.70.910.920.680.790.930.930.730.73 EM9L60.240.240.430.750.450.530.910.920.640.860.920.930.480.59 EM9L70.240.240.440.750.450.530.910.920.620.860.920.930.480.59 EM10L10.340.350.50.510.520.520.820.840.630.750.810.820.570.58 EM10L20.40.430.50.520.640.640.870.890.580.740.820.830.520.55 EM10L30.390.410.570.60.460.510.870.880.610.790.850.860.60.61 EM10L40.330.330.510.530.450.510.890.890.60.820.890.90.490.57 EM10L50.260.250.420.430.460.510.830.830.610.850.930.930.460.53 EM10L60.30.290.480.540.490.50.890.90.630.810.720.730.480.53 EM11L10.410.490.550.720.540.540.840.850.570.70.850.860.570.57 EM11L20.360.390.520.720.580.590.890.90.640.810.830.840.580.6 E M11L30.370.420.440.450.510.520.90.910.650.860.780.780.580.6 EM11L40.340.360.490.540.550.560.850.860.610.810.870.880.580.6 EM12L20.420.490.520.720.60.60.890.90.620.840.750.750.550.57 EM12L30.080.120.540.550.540.560.880.890.690.840.840.850.650.66 EM12L40.40.450.570.660.580.580.890.90.60.770.810.820.550.58 EM12L50.40.540.540.590.630.630.870.880.620.820.90.910.570.61 EM12L60.340.360.480.540.550.570.880.880.620.820.920.930.550.58 19 12345 E_M9_L2E_M9_L3E_M8_L2E_M8_L3E_M9_L4E_M9_L5E_M9_L1E_M8_L1E_M8_L4 E_M10_L2E_M10_L3E_M11_L1E_M11_L2E_M12_L2E_M12_L4E_M12_L5E_M12_L6E_M11_L3E_M11_L4E_M12_L3E_M12_L7 0.00 0.25 0.50 0.75 1.00 Model Macro−F1 score Rank Figure 6: Scott-Knott ESD ranking for embedding-based model-label com- binations based on macro-F1 score 12345 N_M4_L3N_M1_L2N_M2_L1N_M4_L2N_M4_L5N_M1_L1N_M1_L3N_M2_L2N_M3_L2N_M4_L1N_M2_L3N_M3_L1N_M3_L3N_M4_L4 0.00 0.25 0.50 0.75 1.00 Model Macro−F1 score Rank Figure 7: Scott-Knott ESD ranking for NLI-based model-label combinations based on macro-F1 score 20 Table 7: Summary of the performance of best-performing model-label com- binations of each NLI-based model across each dataset M&LAPIGerritGitHubGitterGoogleJiraSO reviewsPlay MacMicMacMicMacMicMacMicMacMicMacMicMacMic NM1L10.410.420.660.710.560.570.890.90.610.850.930.940.630.65 NM1L20.40.410.680.740.580.590.920.930.680.830.920.930.680.69 NM1L30.470.610.490.490.470.530.920.920.660.830.90.910.730.74 NM1L40.220.220.270.30.460.520.890.90.590.820.90.910.470.56 NM1L50.220.230.260.290.470.520.880.890.620.830.920.930.480.57 N M1L60.090.120.20.250.30.370.690.690.570.790.760.830.320.38 NM1L70.110.130.20.250.330.390.70.70.580.80.80.850.360.43 NM2L10.40.430.690.760.580.580.880.880.710.870.940.950.640.67 NM2L20.470.550.580.760.580.570.880.890.60.750.930.940.670.67 N M2L30.420.620.580.610.630.630.850.870.520.650.910.920.660.66 NM2L40.250.260.520.540.460.490.810.840.520.650.760.760.50.53 NM2L50.290.320.450.450.450.480.780.830.430.620.620.620.480.51 NM2L60.140.130.220.260.30.350.670.70.290.570.520.520.30.36 NM2L70.170.160.240.270.340.390.60.70.350.610.60.60.330.41 NM3L10.350.350.620.70.50.530.90.90.630.880.910.920.580.62 N M3L20.420.440.590.720.550.570.910.920.680.860.940.950.690.69 NM3L30.310.30.230.270.440.510.90.910.630.850.940.950.650.67 NM3L40.240.230.210.260.430.50.910.920.610.850.930.940.460.56 NM3L50.230.230.210.250.430.50.930.930.650.850.930.940.470.57 NM3L60.170.190.20.250.230.320.760.770.480.690.470.480.360.45 N M3L70.180.190.210.250.190.310.750.760.530.770.630.630.310.37 NM4L10.430.460.650.690.560.590.910.910.640.890.940.940.630.66 NM4L20.420.440.620.780.740.730.910.920.650.840.840.850.740.74 NM4L30.480.550.630.650.710.710.920.930.650.850.910.920.80.81 N M4L40.330.330.50.510.570.590.830.860.620.820.780.790.640.65 NM4L50.450.610.650.690.740.730.870.880.60.840.860.870.720.71 N M4L60.130.130.550.720.190.30.670.70.460.680.590.590.240.32 NM4L70.080.120.560.610.20.30.670.680.470.680.670.670.250.33 21 The results of combining TARS model with different label configurations are presented in Table 8, with the highest macro-F1 and micro-F1 score for each dataset are bolded. The results revealed the superiority of L2, which was able to produce the highest macro-F1 score for four datasets. L1 followed closely, achieving the highest scores for three datasets. The other labels failed to produce the highest values for any dataset. Table 8: Summary of the performance of the TARS model-label combinations across each dataset M&LAPIGerritGitHubGitterGoogleJiraSO reviewsPlay MacMicMacMicMacMicMacMicMacMicMacMicMacMic TM1L10.490.630.640.70.490.520.770.780.590.670.890.90.620.62 TM1L20.360.680.550.580.50.540.830.840.60.680.820.840.630.63 T M1L30.330.460.520.590.440.440.620.620.580.650.60.640.360.36 T M1L40.30.390.50.640.380.410.430.460.470.530.50.570.30.3 TM1L50.30.380.480.60.340.380.420.430.220.220.40.50.30.31 TM1L60.30.530.410.410.260.30.340.360.210.230.380.550.250.37 TM1L70.30.640.540.70.30.330.370.370.320.390.40.630.270.33 The results of the statistical test, presented in Figure 8, confirm these findings. The model combinations with L1 and L2 were ranked as the top performers, followed by L3 in second, L4 in third, L5 and L7 in fourth, and L6 in fifth. Table 9 presents the results of combining the generative-based model with the label configurations, with the highest macro-F1 and micro-F1 score for each dataset are bolded. The results demonstrate the superiority of L1, which achieved the highest performance on three datasets. All other labels, except for L3, were able to achieve the highest value on one dataset, while L3 did not produce the highest value on any dataset. Table 9: Summary of the performance of the generative model-label combi- nations across each dataset M&LAPIGerritGitHubGitterGoogleJiraSO reviewsPlay MacMicMacMicMacMicMacMicMacMicMacMicMacMic GM1L10.520.580.750.80.730.730.90.910.670.890.880.890.720.73 G M1L20.50.570.720.770.730.730.880.890.660.890.760.770.760.76 GM1L30.50.570.720.770.640.650.860.880.620.860.820.820.720.73 GM1L40.510.630.750.820.820.820.910.920.620.870.80.80.740.74 GM1L50.380.640.620.80.830.830.90.90.620.860.820.830.620.63 GM1L60.270.240.610.760.40.480.910.910.60.740.890.90.750.75 GM1L70.220.220.690.80.320.380.930.930.60.860.810.820.640.66 The statistical test produced three ranks, as Figure 9 presents. Combining GM1 with L1, L2, or L4 resulted in the top performance. Pairing the model 22 12345 T_M1_L1T_M1_L2T_M1_L3T_M1_L4T_M1_L5T_M1_L7T_M1_L6 0.00 0.25 0.50 0.75 1.00 Model Macro−F1 score Rank Figure 8: Scott-Knott ESD ranking for the TARS model-label combinations based on macro-F1 score with L3 and L5 placed it in the second rank, while L6 and L7 combinations placed the model in the third rank. The results of comparing the performance of all model-label combinations across all ZSL techniques using the Scott-Knott ESD test are presented in Figure 10, with only the top 5 ranks included due to space limitations. The results confirm the superior performance of the E M9 model. Specifically, the model was ranked as the top performer when combined with both L2 and L3. Combining the model with L4 placed it in second place, alongside the generative-based model when combined with both L1 and L4. Additionally, the NLI-based model (i.e., N M4) was ranked second when combined with L3. Other combinations of embedding, NLI, and generative-based models followed in subsequent ranks. Notably, only one TARS-based combination was ranked fifth, with other ZSL technique combinations surpassing it. 23 123 G_M1_L1G_M1_L2G_M1_L4G_M1_L3G_M1_L5G_M1_L6G_M1_L7 0.00 0.25 0.50 0.75 1.00 Model Macro−F1 score Rank Figure 9: Scott-Knott ESD ranking for the generative model-label combina- tions based on macro-F1 score 12345 E_M9_L2E_M9_L3E_M9_L4 N_M4_L3 G_M1_L1G_M1_L4 E_M8_L2E_M8_L3E_M9_L5 N_M4_L2N_M4_L5 G_M1_L2G_M1_L3 N_M1_L1N_M1_L2N_M2_L1N_M2_L2N_M3_L2N_M4_L1 G_M1_L5 E_M9_L1 E_M12_L5 N_M1_L3N_M2_L3N_M3_L1 T_M1_L1 0.00 0.25 0.50 0.75 1.00 Model Macro−F1 score Rank Figure 10: Scott-Knott ESD ranking for model-label combinations based on macro-F1 score 24 5.3 RQ3: How does the performance of ZSL-based models compare with that of the state-of-the-art fine-tuned transformer-based models in sentiment classification? The results of the state-of-the-art fine-tuned transformer-based models and the best-performing model-label combinations for each ZSL-based technique are presented in Table 10, where the highest values of the macro-F1 and micro-F1 scores for each dataset are bolded. As the table depicts, F M2L1 achieved the highest macro scores in 3 out of the 7 datasets, and GM1L4 attained the highest scores in 2 datasets. Furthermore, FM1L1, FM3L1, and EM9L1 each achieved the highest score in one dataset. The results of the statistical analysis are presented in Figure 11. The statistical analysis revealed that F M1L1 , FM4L1 , and EM9L3 are the top performers. The remaining fine-tuned models along with EM9L2 and G M1L4 , were ranked second. NM1L1, NM2L1, and EM9L1 were ranked third. TM1L1 preceded NM4L3 and GM1L1 , with the latter two models ranked last, in the fifth rank. The above results revealed that embedding-based and generative-based ZSL models have competitive performance compared to that of fine-tuned models on the testing set. 5.4 RQ4: What factors contribute to the misclassifi- cation of sentiment labels in ZSL-based models, and how do these compare with those shared with the state-of-the-art fine-tuned transformer-based models? The results of the quantitative error analysis are summarized in Table 11, where the lowest number of misclassified instances for each dataset are bolded. Among the ZSL-based models, E M9L3 yielded the fewest errors across the majority of datasets (i.e., 3 out of 7 datasets), followed by G M1L1 and GM1L4, in 2 out of 7 datasets, and EM9L1 in only one dataset. More- over, common misclassifications across the ZSL-based models included 35 API reviews, 19 GitHub comments, 3 app reviews, 1 Jira comment, and 10 SO posts. When comparing the performance of the fine-tuned models to the ZSL- based models, the fine-tuned models produced fewer misclassifications than 25 Table 10:Summary of the performance of state-of-the-art fine-tuned transformer-based models and the best-performing model-label combinations for each ZSL-based technique M&LAPIGerritGitHubGitterGoogleJiraSO reviewsPlay MacMicMacMicMacMicMacMicMacMicMacMicMacMic Embedding-based ZSL models EM9L10.470.490.430.750.610.620.950.950.60.860.990.990.640.67 EM9L20.560.670.790.840.740.730.950.950.60.860.960.970.770.77 EM9L30.580.670.790.860.790.780.950.950.60.860.960.970.810.81 NLI-based ZSL models NM1L10.420.440.670.730.550.570.90.90.580.830.90.910.650.68 N M2L10.420.450.770.830.580.590.860.860.60.860.910.920.70.72 NM4L30.320.330.530.540.580.60.770.810.560.8.760.760.650.66 TARS-based ZSL models TM1L10.550.650.620.690.510.530.710.710.520.690.870.890.640.63 Generative-based ZSL models GM1L10.570.690.660.690.460.540.660.680.580.830.540.810.760.76 GM1L40.510.680.760.830.770.77110.610.890.830.840.740.73 Fine-tuned models FM1L10.820.890.780.850.920.920.510.670.550.80.960.970.890.89 F M2L10.750.840.820.870.940.940.380.620.430.630.970.980.910.91 FM3L10.80.870.790.840.940.940.380.620.580.830.960.970.880.88 FM4L10.770.860.760.820.920.920.840.860.510.740.960.970.880.88 FM5L10.720.810.780.840.930.930.510.670.530.770.960.970.890.89 26 12345 F_M1_L1F_M4_L1 E_M9_L3 F_M2_L1F_M3_L1F_M5_L1 E_M9_L2 G_M1_L4 N_M1_L1N_M2_L1 E_M9_L1 T_M1_L1 N_M4_L3 G_M1_L1 0.00 0.25 0.50 0.75 1.00 Model Macro−F1 score Rank Figure 11: Scott-Knott ESD ranking for the state-of-the-art fine-tuned transformer-based models and the best-performing model-label combinations for each ZSL-based technique based on macro-F1 score all of the examined ZSL-based models in 4 out of 7 datasets. The fine- tuned models were able to reduce misclassifications on API reviews, Gerrit code review comments, GitHub comments, and SO posts. The common misclassifications by the fine-tuned models included 25 API reviews, 11 Gerrit code review comments, 16 GitHub comments, 1 Gitter developer message, 5 app reviews, 1 Jira comment, and 25 SO posts. Both ZSL-based models and fine-tuned models failed to classify 9 API reviews, 5 GitHub comments, 3 app reviews, 1 Jira comment, and 1 SO post. The analysis of the commonly misclassified instances by the ZSL-based models revealed that among the total 68 misclassified instances, 64.71% were originally annotated as neutral, 22.06% as positive, and 13.24% as negative. The categorization of errors indicated that subjectivity in annotation resulting from different annotators’ perceptions of emotions accounts for 60.29% of the commonly misclassified instances. Polar facts were identi- fied as the second most common source of misclassification, where it rep- resents 22.06% of the errors. Polar facts are those that describe inherently desirable or undesirable situations, expressed in a neutral tone, such as the comment “this doesn’t work”. Politeness errors accounted for 8.82% of the misclassified instances. These errors arose when polite expressions, such as “Thanks!”, led to inconsistent classifications. Misclassifications due to figu- 27 rative language, such as humor, irony, or sarcasm, contributed to 4.41% of the errors. An example is the API review statement: “So initializing high is better than too low”, which was annotated as negative by human annotators but incorrectly classified as positive or neutral by the models. Finally, prag- matic errors, such as statements reporting third-party opinions or emotions, were another source of misclassification, accounting for 4.41% of the errors. In these cases, sentences that humans identify as neutral are misclassified as positive or negative by the models due to the presence of emotion-related words. An example of this is the API review statement: “I know many de- velopers would like to have this”, which was annotated as neutral, but the majority of models classified it as positive. The common misclassified instances among both ZSL-based and fine- tuned models were originally labeled as follows: 57.89% neutral, 26.32% positive, and 15.79% negative. The error categorization revealed that sub- jectivity in annotation was the main contributor, accounting for 73.68% of these instances. Politeness errors accounted for 15.79%, followed by polar facts with 5.26%, and figurative language with 5.26%. Table 11: Summary of missclassifieds instances M&LAPIGerritGitHubGitterGoogleJiraSO reviewsPlay Test set size452160713213593443 ZSL-based models EM9L122940270151148 EM9L214825193153101 EM9L31492315415386 NM1L125443308268143 N M2L125027294357126 NM4L3304742854722151 TM1L11575033361110164 GM1L1138503300618105 G M1L4143271620415118 Common among ZSL-based models3501903110 Fine-tuned models FM1L151245477347 FM2L1742142813241 FM3L160255086351 FM4L162275739352 FM5L185265278350 Common among fine-tuned models25111615125 Common among all9050311 28 6 Discussion The results of RQ1 revealed that generative-based ZSL outperformed other ZSL techniques using original labels, with NLI ranked second. Despite being pre-trained for sentiment analysis in tweets, EM9 performed well on soft- ware engineering sentiment analysis. Although, the paid embedding (i.e., EM10) outperformed most embeddings, it was outperformed by some NLI- based models and E M9. Therefore, we recommend exploring freely available models before investing in a paid one, as the former may yield better results. RQ2 revealed that no single label configuration consistently outperformed others across all models. While the original label L1 performed well with TARS and generative-based ZSL, embedding-based and NLI-based models performed better with expert-curated labels, particularly those incorporating the sentiment term (i.e., L2 and L3). Combining E M9 with L3 yielded the best results, suggesting that including task context in label configurations may enhance performance. Additionally, combining E M9 with L3 yielded the best results across all ZSL techniques. This supports the earlier obser- vation that pre-trained models from other domains are a viable approach for ZSL. In RQ3, when compared to fine-tuned state-of-the-art models, E M9 paired with L3 ranked as the top performer, achieving results comparable to some fine-tuned models, while surpassing others. This finding contra- dicts previous research, which observed that general sentiment analysis tools underperform in software engineering contexts [9, 10], as the finding demon- strates that pre-trained models trained on sentiment analysis in other do- mains can achieve performance similar to or exceeding fine-tuned models without the need for additional training or fine-tuning, thereby addressing the issue of data scarcity. RQ4 revealed that most misclassifications in ZSL-based models occurred in the neutral class, consistent with the observations of [12] and [3] that neutral sentiments are challenging to classify. Subjectivity in annotation and polar facts were identified as the primary causes of these misclassifications. Subjectivity in annotation was also highlighted by [3] as one of the challenges in sentiment analysis. Another observation is that while RQ3 demonstrated that some ZSL- based models performed similarly to fine-tuned models, the latter models were more effective at reducing misclassifications in technical datasets (i.e., API reviews, code reviews, GitHub comments, and SO posts), as evidenced by the results of RQ4. In contrast, ZSL-based models performed better on more conversational datasets, such as developer chat messages, app reviews, and Jira comments. 29 When comparing our results with previous, related studies, our results can be compared with the work of [1], which was performed in RQ3. The other work that can be compared to ours is the work of [2]. Specifically, we compare the performance of our ZSL-based models with their ZSL-based and FSL-based models on the common datasets from both studies. A key difference between our study and [2] is that while we evaluated the models on all test sets in RQ3 and RQ4, [2] selected a stratified representative random sample due to the high cost of running generative models. In our study, combining E M9 with both L2 and L3 resulted in a 0.79 macro-F1 score on the Gerrit dataset, higher than the 0.76 best macro-F1 score achieved by [2] through FSL, where their ZSL achieved 0.75. For GitHub, our highest macro-F1 score is 0.79, achieved by E M9L3, while [2] achieved 0.72 with both FSL and ZSL. [2] achieved a perfect macro- F1 score on the Google Play dataset with FSL and 0.98 with ZSL, whereas our best ZSL model (i.e., G M1L4) achieved only 0.61. EM9L1 achieved an almost perfect macro-F1 score (i.e., 0.99) on the Jira dataset, where [2] achieved 0.91 through FSL and 0.85 through ZSL. Although [2] focused solely on generative-based models, the comparison highlights the superiority of E M9, which achieved higher results on 3 out of the 4 common datasets. This confirms our observation that applying ZSL with a model pre-trained on a similar context can yield competitive performance. 7 Threats to validity Several threats may affect the validity of this study. This section discusses external, construct, internal, and conclusion validity threats, along with cor- responding mitigation strategies, where applicable [42]. 7.1 External validity A potential threat to our study’s external validity is the generalizability of its findings. Although our results are derived from seven sentiment classification datasets representing diverse software engineering contexts, the generalizabil- ity of these findings cannot be claimed beyond these datasets. Another potential external validity threat arises from the models and labels utilized. Since our findings are confined to the models and labels examined, they may not generalize to others that were not included in this study. Consequently, we acknowledge this as a limitation to our study’s external validity. 30 7.2 Construct validity A potential threat to this study’s construct validity lies in the performance measures employed. To address this threat, we report both macro-F1 and micro-F1 scores, with the former being the basis for comparison. This ap- proach aligns with the approach of [2, 6]. Another potential construct validity threat stems from the annotation of the utilized datasets. We rely on datasets annotated in prior work and widely utilized in related studies [1, 2, 4]. While efforts have been made to ensure annotations accuracy, we cannot guarantee the complete absence of errors. Consequently, this limitation is inherited from the datasets. An additional potential construct validity threat is the manual categoriza- tion of misclassified instances, which may introduce subjectivity. To mitigate this threat, we independently categorized the errors then resolved inconsis- tencies through discussion until a consensus was reached. 7.3 Internal validity A potential threat to our study’s internal validity is the possibility of imple- mentation errors. To mitigate this, we used widely accepted and validated model implementations and followed guidelines and tutorials for applying these models in similar contexts, such as [13, 14]. 7.4 Conclusion validity Conclusion validity concerns the appropriate use of statistical tests [42]. Be- sides using common statistical measures, such as percentages, to convey the study’s findings, we only employed Cohen’s Kappa coefficient to measure inter-rater agreement and the non-parametric Scott-Knott ESD test to com- pare the performance of models. The use of Cohen’s Kappa coefficient aligns with its intended purpose and is recommended by empirical standards in software engineering research [43]. The Scott-Knott ESD test was selected for its ability to produce disjoint groups with non-negligible magnitudes of difference, its application in prior studies within similar contexts, and its high tolerance for outliers [39, 40]. 8 Conclusion Aiming to address the challenge of data scarcity within sentiment analysis for software engineering, this study explored the potential of ZSL for sentiment analysis in software engineering. We empirically evaluated the performance of 31 embedding-based, NLI-based, TARS-based, and generative-based ZSL tech- niques across multiple software engineering contexts. We expanded the scope of our analysis by including various label configurations to understand their impact on model performance. We then compared the best-performing ZSL- based models and label combinations with the state-of-the-art fine-tuned transformer-based models. Finally, we conducted an error analysis to better understand the causes of sentiment misclassifications by ZSL-based models. The results demonstrated that ZSL is a viable approach for sentiment analysis within the software engineering domain, achieving performance com- parable to or exceeding state-of-the-art fine-tuned transformer-based models without requiring fine-tuning. Specifically, an embedding-based model, fine- tuned to analyze sentiments within tweets, achieved top performance when paired with expert-generated labels that contextualized the dataset and in- corporated the term “sentiment”. The combination of the model and label was statistically ranked as the top performer, where it either matched or surpassed fine-tuned models. Similarly, the generative-based model when paired with expert-generated labels that elaborated on describing emotions with sentiments was ranked as the second-best performer alongside some of the fine-tuned models. The error analysis revealed that most misclassified instances were originally annotated as neutral sentiment, and the further cat- egorization of the misclassified instances identified subjectivity in annotation and polar facts as the primary causes of misclassification. These findings underscore the potential of ZSL in addressing the chal- lenges of training data shortages for sentiment analysis in software engineer- ing. By eliminating the need for annotated data, ZSL provides a practical solution for sentiment analysis across diverse software engineering contexts. Future work could explore how linguistic variation in label phrasing im- pacts sentiment classification performance. Additionally, investigating the influence of different prompting strategies on model effectiveness in sentiment classification represents a promising avenue for future research. Moreover, a deeper error analysis that leverages explainable artificial intelligence (XAI) and examines differences between developer-oriented and user-generated datasets could provide more nuanced insights into ZSL performance within sentiment analysis in software engineering. 32 Declaration of Generative AI and AI-assisted Technologies in the Writing Process The authors used ChatGPT to improve the readability and language of the manuscript, with full responsibility for the final content. References [1] T. Zhang, B. Xu, F. Thung, S. A. Haryono, D. Lo, L. Jiang, Sentiment analysis for software engineering: How far can pre-trained transformer models go?, in: 2020 IEEE Intealefato2018sentimentrnational Confer- ence on Software Maintenance and Evolution (ICSME), IEEE, 2020, p. 70–80. [2] T. Zhang, I. C. Irsan, F. Thung, D. Lo, Revisiting sentiment analy- sis for software engineering in the era of large language models, ACM Transactions on Software Engineering and Methodology 34 (3) (2025) 1–30. [3] M. Obaidi, J. Kl ̈under, Development and application of sentiment anal- ysis tools in software engineering: A systematic literature review, in: Proceedings of the 25th International Conference on Evaluation and Assessment in Software Engineering, 2021, p. 80–89. [4] B. Lin, F. Zampetti, G. Bavota, M. Di Penta, M. Lanza, R. Oliveto, Sentiment analysis for software engineering: How far can we go?, in: Proceedings of the 40th international conference on software engineering, 2018, p. 94–104. [5] A. Sajadi, K. Damevski, P. Chatterjee, Towards understanding emotions in informal developer interactions: A gitter chat study, in: Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2023, p. 2097– 2101. [6] N. Novielli, F. Calefato, D. Dongiovanni, D. Girardi, F. Lanubile, Can we use se-specific sentiment analysis tools in a cross-platform setting?, in: Proceedings of the 17th International Conference on Mining Software Repositories, 2020, p. 158–168. 33 [7] G. Uddin, F. Khomh, Automatic mining of opinions expressed about apis in stack overflow, IEEE Transactions on Software Engineering 47 (3) (2019) 522–559. [8] F. Calefato, F. Lanubile, F. Maiorano, N. Novielli, Sentiment polarity detection for software development, in: Proceedings of the 40th Inter- national Conference on Software Engineering, 2018, p. 128–128. [9] R. Jongeling, S. Datta, A. Serebrenik, Choosing your weapons: On sen- timent analysis tools for software engineering research, in: 2015 IEEE International Conference on Software Maintenance and Evolution (IC- SME), IEEE, 2015, p. 531–535. [10] P. Tourani, Y. Jiang, B. Adams, Monitoring sentiment in open source mailing lists: exploratory study on the apache ecosystem., in: CASCON, Vol. 14, 2014, p. 34–44. [11] N. Novielli, D. Girardi, F. Lanubile, A benchmark study on sentiment analysis for software engineering research, in: Proceedings of the 15th International Conference on Mining Software Repositories, 2018, p. 364–375. [12] B. Lin, N. Cassee, A. Serebrenik, G. Bavota, N. Novielli, M. Lanza, Opinion mining for software development: a systematic literature re- view, ACM Transactions on Software Engineering and Methodology (TOSEM) 31 (3) (2022) 1–41. [13] L. Tunstall, L. Von Werra, T. Wolf, Natural language processing with transformers, ” O’Reilly Media, Inc.”, 2022. [14] J. Alammar, M. Grootendorst, Hands-On Large Language Models: Lan- guage Understanding and Generation, ” O’Reilly Media, Inc.”, 2024. [15] S. P. Veeranna, J. Nam, E. L. Mencıa, J. F ̈urnkranz, Using semantic similarity for multi-label zero-shot classification of text documents, in: Proceeding of european symposium on artificial neural networks, com- putational intelligence and machine learning. bruges, belgium: Elsevier, 2016, p. 423–428. [16] W. Alhoshan, A. Ferrari, L. Zhao, Zero-shot learning for requirements classification: An exploratory study, Information and Software Technol- ogy 159 (2023) 107202. 34 [17] W. Yin, J. Hay, D. Roth, Benchmarking zero-shot text classifica- tion: Datasets, evaluation and entailment approach, arXiv preprint arXiv:1909.00161 (2019). [18] K. Halder, A. Akbik, J. Krapac, R. Vollgraf, Task-aware representation of sentences for generic text classification, in: Proceedings of the 28th International Conference on Computational Linguistics, 2020, p. 3202– 3213. [19] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., Language models are few-shot learners, Advances in neural information processing systems 33 (2020) 1877–1901. [20] M. S ́anchez-Gord ́on, R. Colomo-Palacios, Taking the emotional pulse of software engineering—a systematic literature review of empirical stud- ies, Information and Software Technology 115 (2019) 23–43. [21] M. Obaidi, L. Nagel, A. Specht, J. Kl ̈under, Sentiment analysis tools in software engineering: A systematic mapping study, Information and software Technology 151 (2022) 107018. [22] M. R. Islam, M. F. Zibran, Sentistrength-se: Exploiting domain speci- ficity for improved sentiment analysis in software engineering text, Jour- nal of Systems and Software 145 (2018) 125–146. [23] M. R. Islam, M. F. Zibran, Deva: sensing emotions in the valence arousal space in software engineering text, in: Proceedings of the 33rd annual ACM symposium on applied computing, 2018, p. 1536–1543. [24] M. R. Islam, M. K. Ahmmed, M. F. Zibran, Marvalous: Machine learn- ing based detection of emotions in the valence-arousal space in software engineering text, in: Proceedings of the 34th ACM/SIGAPP Sympo- sium on Applied Computing, 2019, p. 1786–1793. [25] A. Murgia, M. Ortu, P. Tourani, B. Adams, S. Demeyer, An exploratory qualitative and quantitative analysis of emotions in issue report com- ments of open source systems, Empirical Software Engineering 23 (2018) 521–564. [26] S. Cagnoni, L. Cozzini, G. Lombardo, M. Mordonini, A. Poggi, M. Tomaiuolo, Emotion-based analysis of programming languages on stack overflow, ICT Express 6 (3) (2020) 238–242. 35 [27] G. Uddin, Y.-G. Gu ́eh ́enuc, F. Khomh, C. K. Roy, An empirical study of the effectiveness of an ensemble of stand-alone sentiment detection tools for software engineering datasets, ACM Transactions on Software Engineering and Methodology (TOSEM) 31 (3) (2022) 1–38. [28] T. Ahmed, A. Bosu, A. Iqbal, S. Rahimi, Senticr: A customized sentiment analysis tool for code review interactions, in: 2017 32nd IEEE/ACM International Conference on Automated Software Engineer- ing (ASE), IEEE, 2017, p. 106–111. [29] J. Ding, H. Sun, X. Wang, X. Liu, Entity-level sentiment analysis of issue comments, in: Proceedings of the 3rd International Workshop on Emotion Awareness in Software Engineering, 2018, p. 7–13. [30] E. Biswas, M. E. Karabulut, L. Pollock, K. Vijay-Shanker, Achieving re- liable sentiment analysis in the software engineering domain using bert, in: 2020 IEEE International conference on software maintenance and evolution (ICSME), IEEE, 2020, p. 162–173. [31] H. Batra, N. S. Punn, S. K. Sonbhadra, S. Agarwal, Bert-based sen- timent analysis: A software engineering perspective, in: Database and Expert Systems Applications: 32nd International Conference, DEXA 2021, Virtual Event, September 27–30, 2021, Proceedings, Part I 32, Springer, 2021, p. 138–148. [32] D. Bleyl, E. K. Buxton, Emotion recognition on stackoverflow posts using bert, in: 2022 IEEE International Conference on Big Data (Big Data), IEEE, 2022, p. 5881–5885. [33] K. Sun, X. Shi, H. Gao, H. Kuang, X. Ma, G. Rong, D. Shao, Z. Zhao, H. Zhang, Incorporating pre-trained transformer models into textcnn for sentiment analysis on software engineering texts, in: Proceedings of the 13th Asia-Pacific Symposium on Internetware, 2022, p. 127–136. [34] M. Shafikuzzaman, M. R. Islam, A. C. Rolli, S. Akhter, N. Seliya, An empirical evaluation of the zero-shot, few-shot, and traditional fine- tuning based pretrained language models for sentiment analysis in soft- ware engineering, IEEE Access (2024). [35] V. R. B.-G. Caldiera, H. D. Rombach, Goal question metric paradigm, Encyclopedia of software engineering 1 (528-532) (1994) 6. 36 [36] M. M. Imran, Y. Jain, P. Chatterjee, K. Damevski, Data augmentation for improving emotion recognition in software engineering communica- tion, in: Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering, 2022, p. 1–13. [37] C. D. Manning, P. Raghavan, H. Sch ̈utze, Introduction to information retrieval, Cambridge university press, 2008. [38] I. H. Witten, E. Frank, M. A. Hall, C. J. Pal, M. Data, Practical ma- chine learning tools and techniques, in: Data mining, Vol. 2, Elsevier Amsterdam, The Netherlands, 2005, p. 403–413. [39] C. Tantithamthavorn, S. McIntosh, A. E. Hassan, K. Matsumoto, The impact of automated parameter optimization on defect prediction mod- els, IEEE Transactions on Software Engineering 45 (7) (2018) 683–711. [40] M.-T. Puth, M. Neuh ̈auser, G. D. Ruxton, Effective use of spearman’s and kendall’s correlation coefficients for association between two mea- sured traits, Animal Behaviour 102 (2015) 77–84. [41] M. L. McHugh, Interrater reliability: the kappa statistic, Biochemia medica 22 (3) (2012) 276–282. [42] F. Shull, J. Singer, D. I. Sjøberg, Guide to advanced empirical software engineering, Springer, 2007. [43] P. Ralph, N. b. Ali, S. Baltes, D. Bianculli, J. Diaz, Y. Dittrich, N. Ernst, M. Felderer, R. Feldt, A. Filieri, et al., Empirical standards for software engineering research, arXiv preprint arXiv:2010.03525 (2020). 37