Paper deep dive
Human-Machine Co-Boosted Bug Report Identification with Mutualistic Neural Active Learning
Guoming Long, Shihai Wang, Hui Fang, Tao Chen
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 97%
Last extracted: 4/26/2026, 10:16:40 PM
Summary
The paper introduces Mutualistic Neural Active Learning (MNAL), a cross-project framework designed to automate bug report identification from GitHub repositories. MNAL leverages a neural language model and an active learning paradigm to create a 'mutualistic' relationship between machine learners and human developers. The framework optimizes the labeling process by selecting reports that are both informative for the model and highly readable/identifiable for humans, thereby reducing cognitive fatigue and resource expenditure. Experimental results on a dataset of over 1.2 million reports show significant improvements in readability, identifiability, and model performance compared to state-of-the-art methods.
Entities (7)
Relation Signals (4)
Mutualistic Neural Active Learning → implements → Active Learning
confidence 100% · coupled with active learning to form neural active learning
Mutualistic Neural Active Learning → targets → GitHub
confidence 100% · designed for automated and more effective identification of bug reports from GitHub repositories
Mutualistic Neural Active Learning → uses → Neural Language Model
confidence 100% · MNAL utilizes a neural language model that learns and generalizes reports across different projects
Developer → collaborateswith → Neural Language Model
confidence 90% · the most informative human-labeled reports and their corresponding pseudo-labeled ones are used to update the model while those reports that need to be labeled by developers are more readable
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Bug reports, encompassing a wide range of bug types, are crucial for maintaining software quality. However, the increasing complexity and volume of bug reports pose a significant challenge in sole manual identification and assignment to the appropriate teams for resolution, as dealing with all the reports is time-consuming and resource-intensive. In this paper, we introduce a cross-project framework, dubbed Mutualistic Neural Active Learning (MNAL), designed for automated and more effective identification of bug reports from GitHub repositories boosted by human-machine collaboration. MNAL utilizes a neural language model that learns and generalizes reports across different projects, coupled with active learning to form neural active learning. A distinctive feature of MNAL is the purposely crafted mutualistic relation between the machine learners (neural language model) and human labelers (developers) when enriching the knowledge learned. That is, the most informative human-labeled reports and their corresponding pseudo-labeled ones are used to update the model while those reports that need to be labeled by developers are more readable and identifiable, thereby enhancing the human-machine teaming therein. We evaluate MNAL using a large scale dataset against the SOTA approaches, baselines, and different variants. The results indicate that MNAL achieves up to 95.8% and 196.0% effort reduction in terms of readability and identifiability during human labeling, respectively, while resulting in a better performance in bug report identification. Additionally, our MNAL is model-agnostic since it is capable of improving the model performance with various underlying neural language models. To further verify the efficacy of our approach, we conducted a qualitative case study involving 10 human participants, who rate MNAL as being more effective while saving more time and monetary resources.
Tags
Links
- Source: https://arxiv.org/abs/2604.18862v1
- Canonical: https://arxiv.org/abs/2604.18862v1
Trouble viewing inline? Open PDF directly →
Full Text
210,931 characters extracted from source content.
Expand or collapse full text
Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning GUOMING LONG ∗ , University of Electronic Science and Technology of China, China SHIHAI WANG † , University of Electronic Science and Technology of China, China HUI FANG, Loughborough University, United Kingdom TAO CHEN ‡ , IDEAS Lab, University of Birmingham, United Kingdom Bug reports, encompassing a wide range of bug types, are crucial for maintaining software quality. However, the increasing complexity and volume of bug reports pose a significant challenge in sole manual identification and assignment to the appropriate teams for resolution, as dealing with all the reports is time-consuming and resource-intensive. In this paper, we introduce a cross-project framework, dubbed Mutualistic Neural Active Learning (MNAL), designed for automated and more effective identification of bug reports from GitHub repositories boosted by human-machine collaboration.MNALutilizes a neural language model that learns and generalizes reports across different projects, coupled with active learning to form neural active learning. A distinctive feature ofMNALis the purposely crafted mutualistic relation between the machine learners (neural language model) and human labelers (developers) when enriching the knowledge learned. That is, the most informative human-labeled reports and their corresponding pseudo-labeled ones are used to update the model while those reports that need to be labeled by developers are more readable and identifiable, thereby enhancing the human-machine teaming therein. We evaluateMNALusing a dataset of 1,275,881 reports from over 127,000 software projects against the state-of-the-art approaches, baselines, and different variants. The results indicate that, remarkably,MNALachieves up to 95.8% and 196.0% effort reduction in terms of readability and identifiability during human labeling, respectively, while resulting in a better performance (e.g., F1-score) in bug report identification. Additionally, ourMNALis model-agnostic since it is capable of improving the model performance with various underlying neural language models. To further verify the efficacy of our approach, we conducted a qualitative case study involving 10 human participants, who rateMNALas being more effective while saving more time and monetary resources. The dataset and code are made publicly available at https://github.com/ideas-labo/MNAL. CCS Concepts:• Software and its engineering→ Maintaining software. Additional Key Words and Phrases: Bug Report Analysis, Active Learning, Human-Machine Teaming, Deep Learning, Natural Language Processing, Software Maintenance ∗ Guoming Long is also supervised in the IDEAS Lab. † Shihai Wang is also supervised in the IDEAS Lab. ‡ Corresponding author: Tao Chen, t.chen@bham.ac.uk. Authors’ addresses: Guoming Long, University of Electronic Science and Technology of China, Chengdu, China; Shihai Wang, University of Electronic Science and Technology of China, Chengdu, China; Hui Fang, Loughborough University, Loughborough, United Kingdom, h.fang@lboro.ac.uk; Tao Chen, IDEAS Lab, University of Birmingham, Birmingham, United Kingdom, t.chen@bham.ac.uk. Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from permissions@acm.org. © 2026 Copyright held by the owner/author(s). Publication rights licensed to ACM. ACM 1049-331X/2026/8-ART111 https://doi.org/10.1145/n.n ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. arXiv:2604.18862v1 [cs.SE] 20 Apr 2026 111:2Guoming Long, Shihai Wang, Hui Fang, and Tao Chen ACM Reference Format: Guoming Long, Shihai Wang, Hui Fang, and Tao Chen. 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning. ACM Trans. Softw. Eng. Methodol. 37, 4, Article 111 (August 2026), 56 pages. https://doi.org/10.1145/n.n 1 INTRODUCTION Software bugs can cause devastating consequences for developers and end-users, including crashes, data loss, and performance degradation [51,83,88,89], especially for the modern AI systems [55]. To identify and fix bugs in software systems, modern version control systems, such as GitHub, allow practitioners to formally submit posts, serving as a critical channel for users to report issues, glitches, or malfunctions they have encountered. As such, bug reporting has been playing a pivotal role in the software development lifecycle [109]. However, those submitted reports might not be bug related; they might well be feature requests, documentation updates, or even reports of a false bug alarm simply incurred by an inappropriate usage. In practice, services like GitHub allow the submitter to label the issues. However, this feature is unreliable because: •It has been reported that a considerable amount of submitted GitHub issues are unlabeled due to, e.g., the submitter being a newcomer to the project or lack of experiences [45,87]. In a recent large-scale study over GitHub [87], it has been found that 54.4% of the issues studied have no label at all. •Even if an issue has a label, the quality varies [40,69,109]. For example, a feature request could turn out to be a bug or a bug issue might turn out to be a false alarm. All the above urge the need to automatically predict/verify whether a submitted issue report is bug-related or not, even if it comes with a submitter-provided label. Therefore, the engineering task, namely bug report identification that distinguishes bug related reports from the others, is crucial for ensuring the effectiveness of bug triage [57]. While it can be the most ideal situation to predict multiple types of issues, it is known that those multiple class predictions can be highly inaccurate and hence lack practical significance [56], since the possible number of classes is large. Therefore, here in this work, we focus on a binary classification problem: identifying whether an issue report is indeed bug-related, which is more likely to achieve high accuracy and hence being useful. The outcome would still be significant because often these bug-related reports are the most concerned ones that should have been dealt with before the other types. In practice identifying whether an issue report is indeed bug-related could be time-consuming, and doing so for every issue report submitted is costly [103]. Even with such a simplified scenario, identifying bug related reports is nevertheless not a trivial task. This is because bug reports are often written in a way that is complex, ambiguous, and incomprehensive, making the identification complicated, e.g., Mani et al. [61] show that there could be up to 332 sentences per report on average in which most contents are highly vague. Such a complication significantly slows down the speed and efficiency of a developer to understand and classify the reports. Indeed, Herzig et al. [38] report that two experienced developers can merely identify around 78 reports in terms of whether they are bug related within one full working day. Further, the volume of submitted reports for software projects increases drastically [23,90]. For example, the escalating trend of bug reports is exemplified by the Firefox bugzilla project, where the annual number of bug reports surged from 9962 in 2007 to 15,652 in 2014, marking an increase of 1.5×over the seven-year period [20]. All the above render solely manual report identification expensive and inefficient. Existing work has leveraged machine learning models [22,48,80,107,108], and more recently neural language models [14,21,36,95,106], to comprehensively parse and analyze the information ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:3 from the reports to predict whether they are bug related, hence automating bug report identification. Since those models are data-driven, it is crucial that there are a good amount of labeled samples to train and update them, i.e., previously known cases of bug related and non-bug related reports. It is a common practice to assume a static setting: train a model once and rely on it to identify the newly submitted reports. However, simply assuming that the developers can label a large number of submitted reports beforehand for training the model is unrealistic due to the expensive human analysis process, therefore a considerable amount of the available reports are often unlabeled. Even if there exists a large number of labeled reports, the training over all of them can be resource- and time-consuming, especially when a complex language model is used [67,102]. Of course, a naive approach is to randomly select some labeled reports to update the model (or ask the developers to label them). Yet, this does not guarantee the quality of those samples, hence whether they are positive to the model quality is still questionable. To overcome the lack of timely labeled reports and excessive training overhead, recent efforts have been relying on active learning [29,82,84,86,93]. In a nutshell, active learning is a learning paradigm that allows the model to continually query humans to label a small number of representative new samples for improving model performance, achieving human-machine teaming [15]. Instead of relying solely on pre-labeled datasets, the active learning chooses which reports should be labeled next, typically prioritizing those samples that are more challenging, uncertain, or likely to have a significant impact on improving its performance [29]. The intuition of the above is simple: since a complete manual identification is too labor intensive while the fully automatic bug report identification using learning models relies too heavily on high-quality labeled data and their quantity, why not take an approach that lies between the two extremes and retains the best of both worlds? This fits exactly with the concept behind active learning. However, existing work adopts vanilla active learning procedure to bug report identification, hence limiting its true potential, because 1) they are restricted to a single software project, requiring complete retraining for application on alternative projects which is not resource efficient; 2) when querying the new reports for labeling, they ignore the efforts that the developers need to complete labels, which often results in highly difficult reports and hence negatively impacts the result and time-to-update the model—a typical consequence of cognitive fatigue [81]; 3) since in bug report identification, there are often a much larger number of unlabeled reports than their labeled counterparts [65], existing work has not been able to fully exploit the valuable information from the unlabeled pool of reports given the human-labeled ones. To sum up, using active learning can benefit from mitigating the cost of labor through optimizing the annotation process, making it more efficient and less burdensome by not only improving model accuracy but also reducing the annotators’ labeling effort. While public history data is available for many open-source projects to train general models, training highly accurate models for specific projects can greatly benefit from our method. Using active learning can efficiently train tailored and precise models for a variety of individual projects. To address the challenges and limitations in identifying bug reports from GitHub repositories above, we proposeMutualisticNeuralActive Learning, orMNAL, a cross-project framework that enhances model consolidation through human-machine collaboration. Our approach leverages a neural language model to generalize reports across different projects, paired with active learning that involves human-labeled samples in the training process. This creates a mutualistic relationship between humans (developers who label the data) and machines (the neural language model that generalizes the knowledge from the reports), enriching the learning process. This mutualistic relation mirrors a symbiotic relationship where both parties benefit, as the model selects the most informative reports for labeling, reducing the effort required from developers while improving performance. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:4Guoming Long, Shihai Wang, Hui Fang, and Tao Chen Specifically, the main contributions of this work are as follows: • Neural Active Learning Framework: –We propose a neural active learning framework that facilitates human-machine co-boosted bug report identification and cross-project handling. • Mutualistic Relation in Active Learning: –Model Improvement: The model is enhanced through human-labeled reports and a new method for generating pseudo-labeled reports. –Developer Assistance: With effort-aware uncertainty sampling, developers are only asked to label more readable and identifiable reports, reducing their workload. • Comprehensive Evaluation: – We evaluateMNALon datasets with 1,275,881 reports from over 127,000 GitHub projects, comparing it against five baselines and several variants using six metrics for performance and labeling efforts. –A case study involving 10 human participants provides qualitative evaluation of the human involvement in MNAL. The experimental results are encouraging, demonstrating that MNAL: • Performance and Efficiency: – Achieves significant improvements in readability (98.1%) and identifiability (194.7%) com- pared to random sampling, and 97.2% and 283.6% improvements compared to uncertainty sampling, all statistically significant. – Considerably enhances model performance using pseudo-labeled reports. • Model-Agnostic Benefits: – Improves performance and efficiency across various neural language models, with read- ability and identifiability improvements of 78.6% and 171.5%, respectively, supported by statistically significant results. • Reduction in Labeling Efforts: – Significantly reduces the efforts required for labeling compared to state-of-the-art ac- tive learning approaches, with improvements of up to 95.8% in readability and 196.0% in identifiability. –Human participants report high satisfaction, with qualitative readability improvements of 74.7% and identifiability improvements of 64.8%, and a reduction in time and monetary resources needed for labeling by threefold. The remainder of this paper is organized as follows: Section 2 provides an overview of the background and challenges. Section 3 presents our detailed designs ofMNALand their rationals. Section 4 outlines the specifics of the experiment settings that evaluateMNAL. Sections 5, 6, and 7 discuss the experimental results, any additional aspects related toMNAL, and threats to validity, respectively. Section 8 positions our approach within the related work. Finally, Section 9 presents the conclusion. 2 PRELIMINARIES This section specifies the necessary backgrounds and challenges that motivate this work. 2.1 Bug Report Identification Modern software repositories like GitHub are prevalent, providing the perfect venue for docu- menting and managing information about software. Typically, when users or developers encounter issues with the software, they generate bug reports to document these anomalies [52]. As shown in Figure 1, a typical report contains a summarized title and a description with details about the ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:5 Fig. 1. An example of the submitted reports from GitHub. encountered problem, steps to reproduce the issue, and other relevant information that can assist developers in diagnosing and resolving the problem. However, not all the submitted reports reflect real bugs; some of them might falsely report a nonexistent bug; some may merely be feature requests, documentation updates, or question-asking, which are non-bug related [66]. A crucial step in bug triage is to identify the bug related reports amongst the others, hence developers can focus on investigating those with a high priority, or otherwise any unnecessary delay in analyzing the potential bug in a report might lead to devastating consequences. Yet, identifying bug related reports is not easy, since the report itself can be vague, imprecise, and with very lengthy content, all of which can require a significant amount of effort to understand and also increase the chance of human misjudgment [38]. Further, as software projects grow and attract a larger user base, the number of submitted reports can grow exceptionally, rendering manual identification impractical [63]. Therefore, automatic bug report identification is in high demand, which is considered a necessity for real bugs to be timely and promptly fixed in current software engineering practice [27]. 2.2 Neural Language Model Because the submitted reports from the software repository are written by humans in various ways, they often exhibit characteristics similar to natural language. Therefore, neural language models, such as BERT [17], CodeBERT [26], RoBERTa [54], and RTA [24], can be promising solutions for automatically identifying bug reports [35]. In a nutshell, neural language models are based on neural networks for processing text data. In contrast to classic language models, neural language models typically embed text inputs into a vectorized latent space. This space is then processed using an attention mechanism, allowing the model to focus on specific parts of the input when making predictions. In this way, neural language models are equipped with the ability to better understand the semantics of the text data. Taking BERT—a most conventional pre-trained natural language processing model—as an ex- ample, its strength is the ability to capture contextualized representations of words, taking into ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:6Guoming Long, Shihai Wang, Hui Fang, and Tao Chen account the surrounding words bidirectionally. This bidirectional context helps grasp the nuances and semantics of natural language. In the context of automatic bug report identification, BERT can be leveraged to understand the language pattern associated with software bugs. Mathematically, the representation of a report in BERT can be expressed as푅=푤 1 ,푤 2 , ...,푤 푛 where푤 푛 are the words in a bug report푅. BERT is pre-trained on general natural language corpora (e.g., Wikipedia information) over several phases, including tokenization, embedding, multi-head self-attention with Transformer encoders, and masking words, after which the parameters are updated with data sourced from the downstream task (i.e., bug report identification). To identify whether푅is bug related, BERT outputs a binary prediction. 2.3 Active Learning Since the labeling process of reports itself is expensive and labor-intensive, the available number of labeled data samples used to update/train the model in a timely manner is often dramatically smaller compared with the number of unlabeled ones. In addition, even if there exists a good amount of labeled reports, training/fine-tuning all of them requires a complex enough model and lengthy training, leading to a drastic increase in the requirements of resources and time. For example, Zhao et al.[104] spent four hours for updating/training a deep neural network on 4,000 bug reports. If this model were to be trained on the million-level datasets (similar to those used in this paper), it would take approximately 53 days. Simply updating the neural language model on some randomly selected data samples works only under the assumption that all available data is equally informative. However, in real-world scenarios, especially in the context of software report analytics, this is seldom the case. In fact, a nontrivial amount of reports are much less informative than others, and hence blindly training on any data can lead to poorly-performed models [99]. Therefore, it is important to determine which reports are the most representative for labeling, hence making better use of the developers’ efforts. To that end, active learning, which queries the most representative samples for labeling and updating the underlying model, presents a promising solution [29, 83]. Given an initial set of labeled data풟 푙 which trains a model푓 0 together with a pool of unlabeled data풟 푢 , active learning improves the performance of a model by querying a set of푘representative, unlabeled data samples풟 푞 for the human to label (|풟 푞 | ≤ |풟 푢 |), which are then used to update a new model푓 푡 , at timestep푡 ∈ 0, ...,푇.풟 푞 will be removed from풟 푢 and added to풟 푙 . The goal is to improve the accuracy of those updated models푓 푡 throughout the timesteps using a separate test set. Active learning can be broadly divided into two categories: stream-based and pool-based [5]. In the former, each data sample is drawn from some distribution in a streaming manner and the active learning needs to decide immediately whether to query the human to label this sample or not (푘=1 and|풟 푢 |=1). The latter, which is a more realistic setting and is the focus of this work, allows flexible access to a pool of unlabeled data (푘>1 and|풟 푢 |>1). Pool-based active learning fits our problem better since it is often the case that new reports accumulate rapidly [46, 73, 90]. Among various pool-based strategies, three notable ones include uncertainty sampling, margin sampling, and confidence-based sampling. Uncertainty sampling focuses on selecting samples for which the model is least confident in its predictions. For instance, it might prioritize samples where the predicted probability is close to 0.5, indicating high uncertainty. Margin sampling chooses samples based on the margin between the model’s top two predictions, targeting those with small margins as they are less confidently predicted. It is worth noting that in binary classification tasks, margin sampling effectively becomes equivalent to uncertainty sampling. This equivalence arises because, in binary tasks, the margin between the two top predictions is directly related to the model’s uncertainty. Confidence-based sampling, on the other hand, selects samples where the model’s confidence in its predictions is high, often in a reverse manner where the model ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:7 queries examples it is confident about but uncertain if they are correctly labeled. By applying these strategies, active learning ensures that the most informative and useful samples are prioritized for labeling, thus enhancing the efficiency and effectiveness of the training process. 2.4 Challenges However, several challenges still exist when adopting active learning to train a reliable and gener- alizable bug report identification model: •Challenge 1: Working on Cross-Project Reports. There could be a limited number of reports for a single project to learn and generalize, therefore, utilizing reports from different projects to update the models and make predictions therein, is of high demand since building or updating a model for each individual project is inefficient. However, it is challenging to handle cross-project reports while effectively exploiting information from the newly queried and labeled reports made by the developers. • Challenge 2: Considering Developers’ Labeling Effort Classic active learning primarily aims to improve accuracy, but little attention has been paid to easing the efforts required to label the data samples. This, if ignored, can cause severe consequences of cognitive fatigue in bug report identification with active learning [81]: there can be a significant delay in updating the model or the reliability of labeling might be compromised when the reports to be labeled by the software developers are always highly complex. •Challenge 3: Extracting Meaningful Information from Unlabeled Reports. Indeed, the labeled reports can be useful for improving the model, but they might still be insufficient considering the largely imbalanced ratio between the number of unlabeled reports and the amount of reports that can be labeled. It would be beneficial, but challenging to further extract meaningful information from the unlabeled ones to enrich the data for updating/training. In the following, we delineateMNALthat is specifically designed to overcome the challenges above. 3 SYNERGIZING MUTUALISTIC RELATION WITH NEURAL ACTIVE LEARNING FOR IDENTIFYING BUG REPORTS In this section, we elaborate on the key properties and detailed designs of the MNAL framework. 3.1 Key Properties 3.1.1Neural Active Learning. From the literature, neural language models [14,21,36,95,106] and active learning [29,82,84,86,93] have been independently studied for identifying bug reports, but they are rarely considered together.MNALcombines a neural language model (e.g., BERT) with active learning, achieving the strength of both semantic understanding and human-machine teaming. In particular, this unique combination poses both challenges and opportunities: the limited set of newly human-labeled reports might not be sufficient for the neural language model to improve. Yet, the neural language model provides the embedding of the report in a latent space, serving as the foundation that allows us to design more sophisticated methods to enrich the data throughout active learning (in the Pseudo-labeling component). Since the neural language model operates on the texts of the reports,MNALseamlessly works cross-projects, including both the continuous update with humans and prediction (addressing Challenge 1), e.g., it might query the reports to developers specialized in different projects and identifies the newly submitted reports for those projects. It is also worth noting thatMNALis model-agnostic, i.e., by design, it can be paired with any neural language model as long as the model supports latent embedding of the reports. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:8Guoming Long, Shihai Wang, Hui Fang, and Tao Chen Timestep 1 Learning with Neural Language Model Prediction Updating Reports from Project N Reports from Project 1 ... New report from Project N New report from Project 1 ... (a) Learning with a neural language model Classic Active Learning Reports from Project 1New report from Project 1 Prediction Updating ...... Timestep 1 Timestep n Classic Active Learning Reports from Project 1New report from Project 1 Prediction Updating (b) Classic active learning Neural Active Learning Prediction Updating ...... Timestep 1 Timestep n Neural Active Learning Prediction Updating Reports from Project N Reports from Project 1 ... Reports from Project N Reports from Project 1 ... New report from Project N New report from Project 1 ... New report from Project N New report from Project 1 ... (c) Neural active learning Fig. 2. Illustrating the difference among learning with a neural language model, classic active learning, and neural active learning for bug report identification. Figure 2 illustrates the difference between neural active learning, learning with a neural language model, and classic active learning alone. As can be seen, the neural language model supports cross- project training/prediction but it neither handles the model change nor actively makes queries to the developers. Active learning deals with the continuous update and the queries to humans well but it does not usually work on cross-project reports, as it is often underpinned by a statistical classifier. As such, the process needs to be repeated from scratch for each project. Neural active learning, in contrast, combines the best of both worlds. The ability of neural active learning ability to make active queries while leveraging advanced neural network models instead of traditional statistical classifiers, which are most commonly used with active learning. 3.1.2 Mutualistic Relation. While active learning can be considered a common iterative learning approach for humans and machines to work as a team during training,MNALgoes beyond the classic active learning by introducing the mutualistic relation between humans (software developers) and machines (neural language model) when enriching the knowledge learned, enabling a better human-machine teaming (addressing Challenge 2 and Challenge 3). As can be seen from Figure 3, this enables a bidirectional benefit: • On one hand, the effort-aware uncertainty sampling would select unlabeled reports that re- quire an acceptable level of effort to label (the Effort-aware Uncertainty Sampling component). This is for the benefit of the software developers, making their jobs easier. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:9 Initialization Effort-aware Uncertainty Sampling Pseudo-labeling Neural Language Model Software Developers Nonbug report Bug report Labeled reports (cross-project) Unlabeled reports (cross-project) Mutualistic Relation 12 3 4 5 6 7 8 Control flowData/model involved Updating/TrainingPrediction Fig. 3. The workflow ofMNALthat supports enhanced human-machine teaming. The serial numbers refer to the order of workflow. • On the other hand, the Pseudo-labeling component enriches the data to train the neural language model, thanks to its ability to represent the report in a latent space. These, together with the newly human-labeled reports, are expected to improve the learning accuracy and efficiency, thereby being beneficial for the model. Such a correlation between software developers and the neural language model is a typical mutualistic relation—they “work together in a team” to enrich the knowledge learned about identifying bug reports, each benefiting from the relationship while enhancing the efficiency of human-machine co-boosted bug report identification. 3.2 Overview The iterative workflow of our proposed framework,MNAL, is illustrated in Figure 3 and Algorithm 1. This process is designed to create an effective human-machine team by integrating active learning with a pseudo-labeling strategy. The workflow, as detailed by the serial numbers in the figure, can be understood in three main phases: Phase 1: Active Sampling of Informative Reports (Steps 1 ○- 2 ○). The cycle begins with an initialized model trained on previously labeled data. 1 ○:The workflow is triggered. A neural language model is initialized using the pool of all available labeled reports which corresponds to the Algorithm lines 2-3. The system directs the pool of unlabeled reports to the Effort-aware Uncertainty Sampling module. 2 ○:This step is to useEffort-aware Uncertainty Samplingto intelligently sample a query size of푘representative unlabeled reports (from within or cross-project). for (one or more) software developers of the relevant projects to decide whether they are bug related (Algorithm line 4). Phase 2: Human-Machine Collaborative Labeling (Steps 3 ○- 6 ○). This phase represents the core “Mutualistic Relation” where human expertise and machine intelligence collaborate to expand the labeled dataset efficiently. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:10Guoming Long, Shihai Wang, Hui Fang, and Tao Chen Algorithm 1: Pseudo code for the learning in MNAL Input: A pool of unlabeled reports 풟 푢 ; a pool of all labelled reports 풟 푙 and a pre-trained neural language model ℳ 1 for 푡 ∈ 0, ...,푇 do 2 if ℳ=∅ or 풟 푙 has been updated since last timestep then 3ℳ← update a neural language model using 풟 푙 4 풟 푞 ←EUSampling(ℳ,풟 푢 ) /* effort-aware uncertainty sampling the top 푘 unlabeled reports. */ 5 풟 푞 ← the developers label all the queried reports 6 풟 푢 = remove all reports in 풟 푞 from 풟 푢 7 풟 푝 ←pseudoLabeling(풟 푙 ,풟 푞 ,풟 푢 )/* pseudo-labeling additional 푘 unlabeled reports. */ 8 풟 푢 = remove all reports in 풟 푝 from 풟 푢 9 풟 푙 = 풟 푙 ∪ 풟 푞 ∪ 풟 푝 10update ℳ with 풟 푙 3 ○:Software developers invest their effort to manually inspect and label the challenging reports sampled in Step 2 ○. These new, high-quality human-provided labels are added to theLabeled reports dataset (Algorithm line 5). 4 ○:The updatedLabeled reportsdataset are immediately used to fine-tune theNeural Language Model. This step ensures the model learns directly from expert knowledge (Algo- rithm line 6) 5 ○:The newly human-labeled reports are then employed in thePseudo-labelingmodule to assign labels to the most similar unlabeled reports (Algorithm line 7). 6 ○:The pseudo-labeled reports are incorporated with the labeled reports in labeled pool to train the neural language model together (Algorithm lines 8–10). Phase 3: Model Consolidation and Iteration (Steps 7 ○ - 8 ○ ). In the final phase, the model is consoli- dated and prepared for both prediction and the next cycle of learning. 7 ○: The resulting robust and more capable model is then deployed for thePredictiontask, automatically classifying new, incoming reports as “Bug report” or “Nonbug report”. 8 ○: The process is designed to be iterative. The expanded set of labeled reports (both human- and pseudo-labeled) forms the basis for theInitializationof the next learning cycle, allowing the model to continuously improve over time as more reports are processed. Particularly, in the pool of labeled reports, any incorrect labels due to human mistakes in the labeling or inaccurate pseudo-labeling can be corrected when discovered. 3.3 Initialization InitializingMNALcan use two distinct strategies: cold start and warm start. These strategies determine how the initial language model is created and subsequently updated to fulfill the task of bug report identification. The choice between cold and warm starts hinges on the availability and adequacy of a pre-trained neural language model, which can be case-dependent. 3.3.1Cold Start. WhenMNALis used completely from scratch, an initial language model needs to be trained by using the available labeled reports from any project, thanks to the semantic attention to the naturalness of language/code in the reports. Noteworthily, the number of reports used in the ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:11 initialization needs not be consistent with the query size and they can come from cross-project since it is important to provide a good general foundation. Most commonly, the initial number of labeled reports is set according to the balance between the training overhead and anticipated model performance. The cold start strategy benefits from creating a robust initial model capable of handling diverse bug reports from the outset. However, this approach involves a significant initial computational cost and time investment. The model’s performance heavily relies on the quality and variety of the labeled reports used for training. A well-balanced and diverse training set can lead to a more generalized and effective model, but acquiring such a dataset can be resource-intensive. 3.3.2 Warm Start. If the neural language model can be pre-trained, we do not need any initial training since the interactive model updating process within MNAL can naturally serve as the fine- tuning process. In this way, we gradually specializeMNALto identify bug reports for the specific problem of bug report identification. For the subsequent timesteps following the first one, however, any newly labeled reports or reports with corrected labels in the pool of labeled reports after the last update would be used to update the model. Noteworthily, such a fine-tuning process is much more lightweight compared with training the neural language model from scratch. The Warm Start strategy offers significant advantages in terms of efficiency, as it builds on an already pre-trained model, reducing initial computational requirements. This approach allows for quicker adaptation to specific datasets and problems, enabling faster convergence and deployment. However, the pre-trained model’s initial quality and relevance to the specific task are crucial. If the pre-trained model is not well-aligned with the target domain, the performance gains might be limited, necessitating more extensive fine-tuning. 3.4 Effort-aware Uncertainty Sampling 3.4.1 Uncertainty. In classic (pool-based) active learning, at every timestep, the key is to decide what data samples (unlabeled reports), from the pool of unlabeled data, should be sent to the human for labeling. This is important, as to what extent those samples are representative determines the accuracy improvement in active learning. From the literature on exploiting active learning in bug report identification and other software engineering tasks, the most common criterion for choosing the data sample labeling is uncertainty [29, 82, 93]: 푈(푥)=− ∑︁ 푦 푝(푦| x) log 2 (푝(푦|x))(1) The above is basically a measurement of information entropy [41] wherebyxrepresents the embedded and vectorized input text of an unlabeled report푥;푦stands for the label of whether the report is bug related or not being predicted by the neural language model;푝(푦|x)represents the conditional probability of a possible label푦provided the inputxextracted from the neural language model (e.g., at thesoftmaxlayer of BERT). A larger entropy indicates higher uncertainty that the current language model knows aboutx. The motivation behind the use of uncertainty in active learning is intuitive: since the underlying language model often performs poorly on the types of data samples that it has never seen before, it would be best if we could train it with the data that is currently known the least. Since the uncertainty measurement of a report simply reflects the information the current language model knows about a report, we should choose the top푘ones with the highest uncertainty and send them to the software developers for labeling in the active learning. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:12Guoming Long, Shihai Wang, Hui Fang, and Tao Chen iterative solver the current recursive one causes tall stack traces when something goes wrong Title Description (a) High readability but low identifiability support “OpenBSD httpd” Nginx does not have /etc/nginx/sites- enabled,available, would need to include them by modifying /etc/nginx/nginx.conf Apache2 is named apache-httpd on openbsd, does not have /etc/apache2/sites- enabled,available, would need to include them by modifying /etc/nginx/httpd2.conf, which is named /etc/nginx/apache2.conf on debian. ...... Title Description (b) High identifiability but low readability Fig. 4. Excerpt of the exampled reports with different scores of readability and identifiability. 3.4.2 What is the limitation? While selecting only the most uncertain data samples has been proven to be helpful for model accuracy [82], it still relies on humans to provide reliable labels. This, however, can be problematic in the other aspects of bug report identification because: •We found that the most uncertain reports are likely to be rather lengthy, especially when the report consists of both natural language and code snippets [61]. •When the reports are complex, the reliability of the labeled report becomes an issue [76,94, 109]. The above would inevitably require large efforts from the software developers to label the reports for the benefits of active learning due to cognitive fatigue. To tackle this unwanted issue, when recommending the data samples (reports) for human-labeling, it is essential to consider the effort required, which is a challenge that theMNALaims to address via introducing a mutualistic relation. That is, in addition to providing labeled data for the benefits of the neural language model, the process should also be effort-aware, thereby taking into account the benefits of software developers who actually label the reports. In particular, we exploit and propose two effort-aware metrics in the sampling process, namely readability [25] and identifiability, of which the details are explained in the following sections. 3.4.3Readability. In addition to the uncertainty, inMNAL, we measure the readability of a report as part of the human effort for labeling when choosing which should be labeled by software developers. In essence, our goal is to assess the ease with which a piece of text can be comprehended by its readers. To that end, we use the Flesch reading-ease score [25]—a well-known metric from the psychology domain—to calculate the readability score of the reports, which can be formally defined as: 푅(푥)= 206.83− 1.015 푤 푠 푒 − 84.6 푠 푦 푤 (2) whereby there are two important terms: •Sentence Structure: The first term, 푤 푠 푒 , evaluates the sentence structure through a ratio between the number of words (푤) and the number of sentences (푠 푒 ) in the unlabeled report푥. A lower ratio implies a report with shorter content and/or more concise sentences, rendering ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:13 it more readable for the software developers. In contrast, a higher ratio suggests a longer report or one with much longer sentences, which are often more difficult to read. •Word Complexity: 푠 푦 푤 is the second term that assesses the ratio between the number of syllables (푠 푦 ) and the number of words. Intuitively, a higher proportion of syllables against the total number of words indicates that the words are generally long and complex, hence the readability of the report tends to be low. On the other hand, a higher amount of shorter words often renders the report easier to read. Clearly, a higher value of 푅(푥) is preferred. Figure 4 shows two contrasted examples of reports with high and low readability, respectively. 3.4.4 Identifiability. While the readability is a good indicator of the required human effort for labeling inMNAL, a more readable report does not necessarily mean that it is also easier to be labeled by the software developers. This is because readability does not reflect the amount of information a report contains for performing the labeling. Taking the example from Figure 4 again, clearly, the readability of Figure 4a is excellent since it is short, but it contains very little information that can barely help one to identify whether it is bug related. To mitigate this issue, we further introduce a metric namely identifiability, aiming to ensure that the chosen reports for labeling are not only understandable but also more distinguishable in terms of their bug related nature. This can be formally defined as: 퐼(푥)= 푇 푟 +푇 푖 푤 (3) where푤is again the number of words in an unlabeled report푥;푇 푟 and푇 푖 denote the common “Bug Report Relevant Terms” and “Bug Report Irrelevant Terms”, respectively 1 , as identified in the literature [39, 60]: •Bug Report Relevant Terms: Bug reports are expected to be concise and informative, containing essential terms that describe the bug and issue. Here, the most common terms are compiled from the literature including“error”,“bug”,“reproduce”,“issue”,“behavior”, “debug”, “failed”, “expected”, and “crash”. • Bug Report Irrelevant Terms: In contrast, reports may also include terms that are less relevant to the bugs. These terms normally appear in non-bug reports such as those describing feature enhancements, questions, or documentation, and hence serve as strong indications that the report is not bug related. In this work, we also extracted the terms from exist- ing studies, including“add”,“would”,“like”,“use”,“feature”,“request”,“support”, “improvement”, “want”, and “documentation”. Among all the terms in a report, if there is a high proportion for either (or both) of the above categories of terms, then we say the report would contain more helpful information for the software developers to label whether it is bug related. Figure 4b is a typical example of high identifiability (although with low readability) since it contains a larger proportion of the terms that are irrelevant to bugs. Otherwise, the identifiability is said to be low, e.g., Figure 4a. 3.4.5Quality-Effort Score. The final score of an unlabeled report푥(and its embeddingx) being used in our effort-aware uncertainty sampling is measured by an acquisition function that aggregates the (normalized) uncertainty, readability, and identifiability, defined as: 푠푐표푟푒(푥)=푈(푥)+ 푅(푥)+ 퐼(푥)(4) in which the uncertainty indicates the potential quality improvement of the underlying language model using the newly-labeled reports while readability and identifiability reflect the efforts required 1 Note that not all the terms in a report can fit into these two categories. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:14Guoming Long, Shihai Wang, Hui Fang, and Tao Chen Algorithm 2: EUSampling function Input: All remaining unlabeled reports 풟 푢 ; the current neural language model ℳ Output: The 푘 unlabeled reports to be queried 풟 푞 1 for∀푟 푖 ∈ 풟 푢 do 2 푠 푖 = get the effort-accuracy score of 푟 푖 using ℳ and normalization via Equation 1-4 3 if |풟 푞 |< 푘 or 푠 푖 > the score of the at least one entry of 풟 푞 then 4if |풟 푞 |= 푘 then 5remove the last entry of 풟 푞 6풟 푞 ← 푟 푖 7sort 풟 푞 descendingly 8 return 풟 푞 by the software developers to label those reports. The higher the value of the score, the better. Since all three metrics can come with drastically different scales, we use max-min standardization to make them commensurable, in which the bounds are their extreme values found so far throughout the neural active learning inMNAL 2 .Given the time constraints, we opted to use an initial setting where all weights of uncertainty, readability, and identifiability are set to 1, which yielded good experimental results. This choice was pragmatic and allowed us to demonstrate the feasibility and benefits of our approach without extensive parameter tuning. There is no specific preference towards any of the three objectives (uncertainty, readability, and identifiability), i.e., all of them are equally important. This is the reason that in the quality score calculation withinMNAL, they are given the same weight, and hence none of them is prioritized against the others. Ultimately, the aim of the effort-aware uncertainty sampling is to recommend top푘(where푘 in this experiment is 300, 500, or 700) unlabeled reports measured by the quality-effort score for human labeling. Since such a recommendation considers the labeling effort, it contributes to the benefit of the human side within the mutualistic relation inMNAL. An algorithmic illustration can be found in Algorithm 2. 3.5 Pseudo-labeling Indeed, given the nature of active learning, the reports labeled by the software developers can serve as the beneficial effect to a machine learner contributed by humans—the other end of the mutualistic relation. However, using the human-labeled reports alone to consolidate the neural language model can easily suffer a difficult trade-off: if we use too few newly human-labeled reports, then it is hard for the updated model to be effective as they might only add a little extra information; this is especially true for updating/fine-tuning a neural language model within neural active learning. In contrast, if we assume that there are many reports that can be queried, the software developers might be overwhelmed by the labeling task, even though the efforts might have already been reduced via our effort-aware uncertainty sampling. To mitigate the above, we hypothesize that those human-labeled reports, which bear strong domain understanding as being processed by the software developers, contain rich information that can guide us to extract useful information from the pool of unlabeled reports. In particular, those unlabeled reports that are more similar to the human-labeled reports are more likely to belong to the same label. For example, Figure 5 respectively shows two pairs of bug related and non-bug related reports with the closest distance searched during the pseudo-labeling process, 2 The bounds might be updated across different timesteps. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:15 Search failed with error failed to connect 10.97.14.83:21123, reason: context canceled after querynode pod kill for kakfa version ...... 00:39:55 Traceback (most recent call last): ...... 00:39:55 File "/usr/local/lib/python3.7/dist- packages/pymilvus/orm/collection.py", line 713, in search 00:39:55 partition_names, output_fields, round_decimal, timeout=timeout, schema=schema_dict, **kwargs) ...... 00:39:55 File "/usr/local/lib/python3.7/dist- packages/pymilvus/client/grpc_handler.py", line 401, in_execute_search_requests 00:39:55 raise MilvusException(response.status.error_code, response.status.reason) 00:39:55 pymilvus.exceptions.MilvusException: <MilvusException: (code=1, message=fail to search on all shard leaders, err=fail to Search, QueryNode ID=39, reason=Search 37 failed, reason err err: failed to connect 10.97.14.83:21123, reason: context canceled ...... Title Description Search raise error: syncTimestamp Failed:context deadline exceeded ...... [2022-02-09 18:14:42,320] [ ERROR] - Traceback (most recent call last): ...... File "/src/milvus_benchmark/client.py", line 322, in query result = self._milvus.search(tmp_collection_name, **params) File "/usr/local/lib/python3.6/site-packages/ pymilvus/client/stub.py", line 829, in search partition_names, output_fields, timeout, round_decimal, **kwargs) ...... File "/usr/local/lib/python3.6/site-packages/ pymilvus/client/grpc_handler.py", line 382, in _execute_search_requests raise BaseException(response.status.error_code, response.status.reason) pymilvus.client.exceptions.BaseException: <BaseException: (code=1, message=syncTimestamp Failed:context deadline exceeded)> (milvus_benchmark.main:98) ...... Title Description (a) Similar bug related reports Add support for developers to set admin version So extension users can make use of adobe/ helix-sidekick#240 Title Description Add support for developers to set admin version add a way to configure an admin version the sidekick should use when requesting the admin. this can be useful to test new admin features with the sidekick. the admin version can be selected with the hlx- admin-version query parameter. Title Description (b) Similar non-bug related reports Fig. 5. Excerpt of the exampled reports that are the most similar and with the same label. The red texts highlight the parts that cause both reports to have high semantic and syntactical similarity. Algorithm 3: pseudoLabeling function Input: Historically labeled reports 풟 푙 ; the set of newly human-labeled reports 풟 푞 ; all remaining unlabeled reports 풟 푢 Output:The set of pseudo-labeled reports풟 푝 for further consolidating the neural language model 1 ℳ← update an intermediate version of the neural language model ℳ using all labeled reports so far 풟 푙 ∪ 풟 푞 2 for∀푟 푖 ∈ 풟 푞 do 3 풟 ′ 푞 ← ⟨푟 푖 ,v i ⟩= get embedding vectorv i when predicting 푟 푖 using ℳ 4 for∀푟 푗 ∈ 풟 푢 do 5 풟 ′ 푢 ← ⟨푟 푗 ,v j ⟩= get embedding vectorv j when predicting 푟 푗 using ℳ 6 for∀⟨푟 푖 ,v i ⟩ ∈ 풟 ′ 푞 do 7 푑 푠 =−1 8 푟 푠 =∅ 9 for∀⟨푟 푗 , v j ⟩ ∈ 풟 ′ 푢 do 10푑 푖푗 = calculate the 2-norm distance betweenv i andv j 11if 푑 푠 =−1 or 푑 푖푗 < 푑 푠 then 12푟 푠 = 푟 푗 13푑 푠 =푑 푖푗 14 풟 푝 ← 푟 푠 with the label of 푟 푖 15 return 풟 푝 which tends to be semantically or syntactically similar. Therefore, our goal is to pseudo-label those most similar unlabeled reports and incorporate them to train the neural language model together ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:16Guoming Long, Shihai Wang, Hui Fang, and Tao Chen Bug related reportNon-bug related reportUnlabeled report (a) Embed both human-labeled and unlabeled reports (b) Examine the distance of all human-labeled/unlabeled pairs (c) Label the nearest unlabeled re- port to each human-labeled one Fig. 6. Illustrating the key steps of the pseudo labeling process inMNALunder a projected 2D space. The points represent the vectorized embedding of the reports produced by the neural language model, e.g., a fine-tuned BERT. with the human-labeled ones. In this way, we can increase the number of labeled reports in learning without asking software developers to label more during the active learning process, hence boosting the mutualistic relation therein. Since the accurate “measurement of similar reports” relies on a comprehensive representation that quantifies the report texts, this can only be made possible within the unique context of neural active learning. Specifically, as shown in Algorithm 3, our pseudo-labeling process inMNALfollows the steps below: (1)To fully exploit the valuable data samples obtained so far for the next step, we firstly update the neural language modelℳby training/fine-tuning it with all the available sets of labeled reports (including the newly human-labeled ones) at line 1. This will create an intermediate version of the model solely used for pseudo-labeling. (2)We feed the newly human-labeled reports,풟 푞 , and all the remaining pool of unlabeled reports (풟 푢 ) intoℳ. This is to obtain the vectorized embedding of those reports for measuring their semantic similarity, for which we use the last encoding layer of the neural language model (e.g., a fine-tuned BERT), as in Figure 6a and lines 3-5. This is only plausible with the power of a neural language model that can meaningfully represent the report texts in a latent space. (3)As shown in Figures 6b and 6c, for each newly human-labeled report푟 푖 , we seek to find its most similar unlabeled counterpart 푟 푠 from all possible unlabeled report 푟 푗 (lines 6-14): 푟 푠 = arg min 푟 푗 ∈풟 푢 ∥ v i −v j ∥ 2 (5) whereby∥·∥ 2 denotes the Euclidean distance between the 768-dimensional vectorsv i andv j , which are the BERT-based embeddings for푟 푖 and푟 푗 , respectively. Finally, we assign the label of 푟 푖 to the identified 푟 푠 as its pseudo label. The above process will be repeated in the next timestep during the mutualistic neural active learning. 4 EXPERIMENTAL SETUP In this section, we elaborate on the details of our experiment settings. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:17 Table 1. Statistics of dataset. Training Set Test Set Bug report samples670,951 (52.6%)74,781 (52.5%) Non-bug report samples 604,930 (47.4%)67,539 (47.5%) Total samples1,275,881 (100%) 142,320 (100%) 4.1 Research Questions The research questions we aim to answer in this study are: • RQ1: How effective is the effort-aware uncertainty sampling? • RQ2: How useful is the pseudo-labeling? • RQ3: To what extent can MNAL improve an arbitrary neural language model? •RQ4: How effective isMNALagainst the state-of-the-art approaches for identifying bug reports? • RQ5: What do the human developers think aboutMNALin terms of the effort reduction for labeling reports? We explore RQ1 to assess the effectiveness and implication of querying unlabeled reports that are both good at improving accuracy and require a reasonable amount of effort to label. In RQ2, we ask how pseudo-labeled reports, which leverage the unique embedding of a neural language model, can complement the prediction performance in an active learning setting. Both RQ1 and RQ2 help to confirm the usefulness of the mutualistic relation inMNAL. Since the design ofMNALis agnostic to the underlying neural language model, we examine RQ3 to understand how well it can improve a neural language model compared with the case where the model is trained with data without any sophisticated sampling and pseudo-labeling. To verify the effectiveness ofMNAL, in RQ4, we compareMNALagainst four state-of-the-art active learning approaches for bug report identification, hence confirming the benefit of neural active learning. Finally, in RQ5, we conduct a qualitative study with human involvement to understand the true usefulness of MNAL. 4.2 Datasets We use the dataset from NLBSE’23 3 [44], which is one of the most comprehensive collections of datasets about repository reports to date. We entails the statistics of this dataset in Table 1 In particular, NLBSE’23 contains many smaller datasets and it documents the data in a more structured manner. It is chosen for our evaluation because of its following unique benefits: •Large Volume: The collection of datasets is of a large volume, containing a total of 1,275,881 labeled reports. A detailed breakdown reveals that 670,951 reports, accounting for 52.6% of the datasets, are bug reports. The remaining 604,930 reports are non-bug reports. Such a near-balanced distribution provides a robust foundation for training and evaluatingMNAL, ensuring that neither class is underrepresented. •Cross-Project: The datasets contain reports collected from different software projects (over 127,000) 4 , which might be of different domains from GitHub. This renders rigorous evaluation of the ability for usingMNALon different projects plausible, which is an important property for neural language model-based approaches. •Diverse Report Formats: The collected reports are of diverse formats and types, ranging from simple interface glitches to complex backend anomalies. The textual descriptions in these reports vary in length and complexity, with some providing detailed steps to reproduce 3 https://nlbse2023.github.io/tools/ 4 https://tinyurl.com/mr3h4my7 ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:18Guoming Long, Shihai Wang, Hui Fang, and Tao Chen the issue, while others offer a brief overview. This diversity in content and context makes the datasets particularly challenging and, at the same time, an ideal ground for testing the robustness of MNAL. For all experiments, we evaluate the approaches over all data from the datasets, i.e., all of the reports are part of the set of unlabeled reports, thereby we intentionally hide their true labels from the learning approaches. At every timestep, the query size is푘, meaning that there will be푘new reports to be labeled by the human for training. As will be discussed in Section 6.3, we examine different values of푘. For all experiments, we follow the warm start strategy by using pre-trained neural language models. The process begins by fine-tuning the model on an initial set of푘randomly selected labeled reports, which serves as the baseline model for the first timestep of active learning. 4.3 Experiment Preparation Here, we provide details on the experiments and implementation. We outline the necessary processes and setups required to prepare the experiments for MNAL and all other methods being compared: • Data Preprocessing: All reports are preprocessed following the steps below: (1) Symbols Removal: HTML tags and punctuation marks that appear in the reports are also removed. (2) Stop Words Removal: Stop words, such as“the”and“is”, are removed. They appear frequently in natural language but with little contribution to semantic meaning. (3) Case Conversion: All upper case texts are converted into lower case ones. • Data Splitting: For training/updating the neural language model or machine learning model (as in the state-of-the-art approaches), we adopt an 80/20 training/validation split when learning each set of푘reports. This ensures that the model is trained on a substantial amount of data while also having a separate set for hyperparameter tuning and validation. For testing, we uniformly sample 5,000 reports (with the true labels) from the test datasets based on their project, type, length, etc; these are excluded from the unlabeled report from the beginning. • Query Size: To examine different cases of labeling the reports in the process of active learning, we run experiments on three different query sizes, i.e.,푘=300,푘=500, and푘=700. It is the number of samples to query at each round. indeed, it is still an open question as to how many samples are sufficient (how large the푘should be) for learning in a given problem. In this work, we pragmatically examined these three푘values, which serves as a good trade-off between the effectiveness they provide and the cost they incur. •Human Involvement: From RQ1 to RQ4, we emulate human involvement by assigning the selected reports the true labels from the datasets. For RQ5, we use the judgments given by the human practitioners as the true labels. • Number of Timesteps and Repetitions: To emulate a realistic setting, we conduct 10 timesteps for active learning considering the balance between data sufficiency and the time it takes to run the experiment. The initial set of labeled reports for training is selected at random. To account for variability and ensure the robustness of our results, we repeat the entire experiment 10 runs with different seeds wherein each repeat would have different training/updating and testing splits. •Model and the Parameter Settings: For the neural language model inMNAL, we adopt BERT [17] by default, which is pre-trained on Wikipedia data, since it is the most standard and simple one for natural language processing. Such a model will be fine-tuned incrementally with푘samples from our downstream task of bug report identification. However, it can be easily replaced with more sophisticated natural language models such as RoBERTa [54]. We have also evaluatedMNALwith different neural language models to confirm its model-agnostic ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:19 nature (see Section 5.3). The hyperparameter settings of those models are either default or tuned to fit the scale of our experiment infrastructure, e.g., for BERT, we use 18 epochs, 32 batch size, learning rate of 3×10 −5 and Epsilon factor of 10 −8 . They were determined through preliminary fine-tuning using 10% of the training data as a validation set. For the hyperparameters of other approaches we compared, we set the same values as what was used in their corresponding work. All experiments are implemented using Python 3.10. We leverage the APIs from PyTorch and Transformers to build and train the neural language model inMNAL. All experiments are run on a high-performance machine equipped with AMD EPYC 7742 (Rome) 2.25GHz multi-core CPU. For GPU acceleration, we utilize an NVIDIA A100 with 40GB RAM. 4.4 Metrics Since our experiments are both quantitative and qualitative, we design/select different sets of metrics for them. 4.4.1Performance (Quantitative). We use the following widely used metrics in software repository mining to quantitatively measure the model performance on the testing data at the end of Timestep 푛: •Precision: Precision measures the proportion of correctly identified reports among all the reports retrieved, which is defined by: 푃푟푒푐푖푠푖표푛= 푡푝 푡푝+ 푓푝 (6) whereby푡푝and푓푝are the number of true positive and false positive predictions, respectively. A higher precision signifies that the classification minimizes false positives and predicts information that is highly relevant. • Recall: Recall measures the ability of a model to identify and retrieve all relevant reports: 푅푒푐푎푙= 푡푝 푡푝+ 푓푛 (7) where푓푛is the number of false negative predictions. A higher recall score indicates that the model can better reduce false negatives and ensure that a significant portion of relevant data is not overlooked. •Accuracy: Accuracy is a straightforward metric that calculates the ratio of correctly identified reports (both bug and non-bug) to the total number of reports. Formally, it is defined as: 퐴푐푢푟푎푐푦= 푡푝+ 푓푝 푡푝+ 푓푝+푡푛+ 푓푛 (8) whereby 푓푝 denotes the number of false positive predictions. •F1-score: The F1-score is a harmonic mean of precision and recall, offering a balanced measure between the model’s ability to correctly identify bug reports (precision) and its capability to capture all genuine bug reports within the datasets (recall), which is computed as: 퐹1 푠푐표푟푒= 2× 푃푟푒푐푖푠푖표푛× 푅푒푐푎푙 푃푟푒푐푖푠푖표푛+ 푅푒푐푎푙 (9) In scenarios where the cost of false positives and false negatives is significant, the F1-score becomes an important metric. Given the critical nature of bug report identification, where overlooking genuine bugs or misclassifying non-bug reports can have substantial implications, the F1-score serves as a cornerstone of our evaluation. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:20Guoming Long, Shihai Wang, Hui Fang, and Tao Chen 4.4.2Effort (Quantitative). It is also important to measure the efforts required to label the reports as part ofMNAL. Quantitatively, we do so by directly using the metrics for readability and identifiability of the queried report during Timestep 푛, as discussed in Section 3.4. 4.4.3Effort (Qualitative). Although the metrics for readability and identifiability used within the sampling ofMNALcan indicate the difficulty of the reports that required labeling, they do not reflect how humans think about the effort in the real scenario. To reflect that, we design Likert scale questions that ask one to individually rate the readability and identifiability that are queried to the developers. For each attribute, the question covers five qualitative scores where a higher score indicates a more perceived readability or identifiability. We will further discuss this in Section 5.5. In addition to the above, we also measure the time taken and monetary cost for the developers to label the queried reports recommended by MNAL. 4.5 Statistical Validation To ensure the comparative results are meaningful, we leverage the following statistical methods. 4.5.1Scott-Knott Test. For comparing multiple approaches, we apply the widely used Scott-Knott test [11,12,64,77]—a hierarchical clustering algorithm—to determine if there are statistically meaningful differences among them. In the context of our study, we employ the Scott-Knott test to ascertain if there are significant differences in the values of metrics and hence generate a ranking across different approaches. Formally, Scott-Knott test aims to find the best split by maximizing the differenceΔ in the expected mean before and after each split: Δ= |푙 1 | |푙| ( 푙 1 −푙) 2 + |푙 2 | |푙| ( 푙 2 −푙) 2 (10) whereby|푙 1 |and|푙 2 |are the sizes of two sub-lists (푙 1 and푙 2 ) from list푙with a size|푙|.푙 1 ,푙 2 , and푙 denote their mean. Recursively, in each iteration, two approaches are placed into different ranks if the results of their 10 repeated runs do not differ much according to the ˆ 퐴 12 effect size [85]; otherwise, they are said statistically similar. By leveraging the Scott-Knott test, we ensure that any differences observed in our results are statistically meaningful with indications of the better and worse while not requiring post-hoc correction. 4.5.2 Wilcoxon Sign-rank Test. We apply the Wilcoxon Sign-rank test [92] with푎=0.05 [2] to investigate the statistical significance of the pairwise comparisons over all 10 runs where applicable, as it is a non-parametric statistical test that makes little assumption about the data distribution and has been recommended in software engineering research [2]. It also assumes paired comparison between data points which fit our needs, since individual run is completed with independent seed. 5 RESULTS Here, we discuss and analyze the results of the experiments in great detail. 5.1 Effectiveness of Effort-aware Uncertainty Sampling in MNAL 5.1.1Operationalization. To answer RQ1, we conduct experiments to compareMNALwith three of its variants that use a different sampling scheme below (BERT is used as the default): • MNAL 푟푎푛 : This is the baseline such that at each timestep, the푘unlabeled reports are randomly sampled from the unlabeled pool. It emulates the case where the neural language model is directly used and updated without active learning. We realizeMNAL 푟푎푛 by replacing the effort-aware uncertainty sampling in MNAL as a random sampling. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:21 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score MNALMNAL ran MNAL un MNAL conf 246810 Timestep 0.78 0.80 0.82 0.84 0.86 F1-score (a) Query size 300 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score (b) Query size 500 246810 Timestep 0.84 0.85 0.86 0.87 0.88 F1-score (c) Query size 700 246810 Timestep 150 100 50 0 50 100 Readability (d) Query size 300 246810 Timestep 150 100 50 0 50 Readability (e) Query size 500 246810 Timestep 80 60 40 20 0 20 Readability (f ) Query size 700 246810 Timestep 0.0 0.1 0.2 0.3 0.4 0.5 Identifiability (g) Query size 300 246810 Timestep 0.1 0.2 0.3 0.4 Identifiability (h) Query size 500 246810 Timestep 0.1 0.2 0.3 0.4 Identifiability (i) Query size 700 Fig. 7. Comparing the effort-aware uncertainty sampling inMNALwith the state-of-the-art and baseline sam- plings over all 10 timesteps (10 runs each). The plots show the mean and standard deviation.MNAL 푟푎푛 ,MNAL 푢푛 andMNAL 푐표푛푓 represent the variants that use random, uncertainty and confidence-based sampling, respec- tively. • MNAL 푢푛 : This conducts the state-of-the-art sampling scheme in active learning based on uncertainty only [29, 82, 93]. All other components are identical to the MNAL. • MNAL 푐표푛푓 : This implements the confidence-based sampling scheme from active learning, as described in [74, 93]. All other components are identical to the MNAL. TheMNAL 푢푛 variant employs a state-of-the-art active learning sampling scheme based solely on uncertainty;MNAL 푟푎푛 is one that randomly chooses reports for human labeling, which is a baseline. All other components are identical to those inMNAL. We cannot simply rule out this component as in active learning, there has to be some mechanism to choose which reports are sent to the human for labeling, and MNAL 푟푎푛 is the simplest and naive baseline. Since there are multiple comparisons, we use the Scott-Knott test introduced in Section 4.5 to verify the statistical significance of the results. 5.1.2 Findings. We plot the traces of the key metrics for all 10 timesteps in Figure 7. Table 2 also reports on the overall results (on all timesteps) of more performance metrics and the effort metrics, together with the ranks from the Scott-Knott test. As can be seen,MNALobtains significantly ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:22Guoming Long, Shihai Wang, Hui Fang, and Tao Chen Table 2. Comparing the mean and standard deviation (SD) of the metrics values for the effort-aware uncer- tainty sampling inMNALand its variant counterparts that use existing samplings over all timesteps on 10 runs. 푟 denotes the rank produced by Scott-Knott test and the statistically best results are highlighted ingreen . Query size Approach F1-scoreAccuracyRecallPrecisionReadabilityIdentifiability 푟 Mean (SD) 푟 Mean (SD) 푟 Mean (SD) 푟 Mean (SD)푟 Mean (SD)푟 Mean (SD) MNAL 푟푎푛 2 0.849 (±0.013) 2 0.842 (±0.011) 2 0.837 (±0.026) 2 0.862 (±0.011)3 -53.530 (±14.955) 3 0.052 (±0.003) MNAL 푢푛 10.857 (±0.016)10.849 (±0.015)10.849 (±0.030)10.865 (±0.014)2 -36.049 (±23.043) 4 0.041 (±0.006) MNAL 푐표푛푓 3 0.827(±0.016) 3 0.819(±0.013) 3 0.812(±0.033) 4 0.842(±0.015)2 -34.575(±42.019) 2 0.071(±0.017) 300 MNAL2 0.850 (±0.019) 2 0.841 (±0.017)10.848 (±0.038) 3 0.852 (±0.016)15.750 (±32.774)10.188 (±0.109) MNAL 푟푎푛 2 0.86 (±0.009)2 0.851 (±0.008) 2 0.855 (±0.016) 2 0.864 (±0.009)4 -53.724 (±11.302) 3 0.053 (±0.002) MNAL 푢푛 10.864 (±0.014)10.856 (±0.013)10.859 (±0.024)10.869 (±0.011)3 -35.195 (±24.862) 4 0.040 (±0.006) MNAL 푐표푛푓 3 0.837(±0.012) 3 0.829(±0.01)3 0.829(±0.031) 4 0.847(±0.015)2 -28.908(±41.391) 2 0.073(±0.018) 500 MNAL2 0.861 (±0.013) 2 0.853 (±0.012)10.861 (±0.022) 3 0.862 (±0.012)1-0.119 (±20.631)10.143 (±0.087) MNAL 푟푎푛 3 0.863 (±0.008) 3 0.855 (±0.007) 2 0.858 (±0.016) 2 0.868 (±0.008)4 -52.903 (±11.866) 3 0.053 (±0.002) MNAL 푢푛 10.870 (±0.009)10.861 (±0.009)10.869 (±0.016)10.871 (±0.009)3 -35.872 (±18.985) 4 0.040 (±0.006) MNAL 푐표푛푓 4 0.848(±0.009) 4 0.839(±0.008) 3 0.839(±0.02)4 0.856(±0.012)2 -22.015(±26.695) 2 0.07(±0.017) 700 MNAL2 0.867 (±0.012) 2 0.859 (±0.012)10.869 (±0.023) 3 0.867 (±0.011)1-8.617(±18.488)10.134(±0.096) better results thanMNAL 푟푎푛 andMNAL 푐표푛푓 on all performance metrics across the cases since the random sampling and confidence-based always fails to select representative reports for human labeling, especially when the query size is larger. The effectiveness of our effort-aware uncertainty sampling is quantitatively remarkable. When compared with a random sampling baseline (MNAL 푟푎푛 ), MNALachieves a 98.1% improvement in readability and a 194.7% improvement in identifiability. Against the state-of-the-art uncertainty-only sampling (MNAL 푢푛 ),MNALstill shows a significant advantage with a 97.2% improvement in readability and a 283.6% improvement in identifiability. These figures, supported by the Scott-Knott test results in Table 2, demonstrate that the reports that MNALselects for human labeling are statistically and substantially easier for developers to process. This quantitative improvement translates to a significant reduction in human cognitive load. The reports selected byMNALare not only more likely to improve model performance but are also structured in a way that is more “human-friendly,” making the labeling task faster and less prone to fatigue. While there is a slight trade-off, resulting in marginally lower F1-scores compared to a pure uncertainty approach (MNAL 푢푛 ), the immense gain in labeling efficiency validates the core principle of our mutualistic framework: benefiting both the machine model and the human developer. The fluctuating results in readability and identifiability across timesteps are due to the evolving nature of the model; as the model learns, the uncertainty landscape shifts, causing different types of reports to become the most informative. Compared withMNAL 푢푛 ,MNALis still competitive but tends to be less effective for improving performance. This is understandable since the most useful reports for improving the performance of the neural language model might not be readable or identifiable. However, we see thatMNALachieves considerably better results thanMNAL 푢푛 on readability and identifiability (97.2% and 283.6% im- provement), thanks to the effort-aware uncertainty sampling. This indicates thatMNALis much more human-friendly which makes the labeling process effortless and time-saving for software developers, significantly benefiting one end of the mutualistic relation. Notably, in contrast to MNAL 푢푛 ,MNALleads to a dramatic gain on the effort metrics while suffering a relatively marginal loss on the effectiveness of prediction. What we found is that there is a clear conflicting relationship between uncertainty and readability/identifiability. The key reason is that the most uncertain report which can be most helpful to improve the accuracy might be difficult to read/identify the bug relevance. On the other hand, those that can enable a human to easily decide its bug relevance could be samples that are already well-learned by a model. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:23 An interesting observation is that, while the performance ofMNALis improved steadily with more timesteps, it tends to experience fluctuated results for readability and identifiability. This is because, as the uncertainty of the unlabeled reports evolves due to the continuously updated neural language model, the sampling landscape with respect to uncertainty, readability, and identifiability of the unlabeled reports also changes, leading to a different set of selected reports for labeling according to the effort-quality score. In particular, we see that there is a significant drop in the identifiability (and improvement in readability) from Timestep 1 to Timestep 2, this is still caused by the change in the uncertainty of the unlabeled reports. Since the neural language model has not been fine-tuned when calculating the uncertainty in Timestep 1, the model has not converged to a stable state compared with the case in Timestep 2. This has caused considerably different uncertainty results that led to a drastic shift in the landscape, hence the significant change in the identifiability metric. In contrast,MNAL 푟푎푛 andMNAL 푢푛 are not affected as the former relies on random exploration while the latter is only guided by the uncertainty value. We will further elaborate this observation with more evidence in Section 6.1. We can conclude the response to RQ1 as: To RQ1:Compared with the others, the effort-aware uncertainty sampling inMNALexhibits a competitive evolving performance while significantly reducing the effort of manually labeling the queried reports for the developers. 5.2 Usefulness of the Pseudo-labeling 5.2.1 Operationalization. To assess the benefit of the pseudo-labeling for RQ2, we compare the following approaches: • No modification: This denotes the MNAL variant without any pseudo labeling. • With augmentation: This is EDA [91], a state-of-the-art data augmentation technique, that seeks to address similar problem as the proposed pseudo labeling. • With pseudo labeling: This is the full version of MNAL with the proposed pseudo labeling. In essence, EDA and pseudo-labeling are two distinct techniques used to enhance machine learning models but address different aspects of model training. EDA focuses on improving text classification performance by artificially increasing the size and variability of training data through operations such as synonym replacement, random insertion, random swap, and random deletion. This helps models generalize better, especially with smaller datasets. On the other hand, pseudo- labeling is a semi-supervised learning technique where the model, after initial training on labeled data, generates predictions for unlabeled data. These predictions are then used as additional training data, effectively leveraging both labeled and unlabeled examples to refine the model’s performance. In particular, while EDA enriches the existing labeled data, pseudo-labeling expands the training dataset by incorporating predictions on unlabeled data. Since we have more than two approaches in the comparisons, we use the Scott-Knott test to verify the statistical significance between the results. Again, BERT is used as the default neural language model. 5.2.2Findings. Figure 8 and Table 3 show the trace along the timesteps and the overall statistical significance results over the cases, respectively. We see that the pseudo-labeling is indeed beneficial to the performance of the neural language model, even when compared with the EDA, as it achieves statistically better results on the majority of the performance metrics than the others. The inclusion of pseudo-labeling provides a clear, statistically significant boost to model performance. For instance, with a query size of 500,MNALwith pseudo-labeling achieves an F1-score of 0.864, ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:24Guoming Long, Shihai Wang, Hui Fang, and Tao Chen 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score No modification,With pseudo labelingWith augmentation 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.78 0.80 0.82 0.84 0.86 F1-score (a) Query size 300 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score (b) Query size 500 246810 Timestep 0.83 0.84 0.85 0.86 0.87 0.88 F1-score (c) Query size 700 246810 Timestep 125 100 75 50 25 0 25 50 Readability (d) Query size 300 246810 Timestep 80 60 40 20 0 20 Readability (e) Query size 500 246810 Timestep 60 50 40 30 20 10 0 10 20 Readability (f ) Query size 700 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (g) Query size 300 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (h) Query size 500 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (i) Query size 700 Fig. 8. Comparing EDA (data augmentation) againstMNALwith and without pseudo-labeling over all 10 timesteps (10 runs each). The plots show the mean and standard deviation. Table 3. Comparing the mean and standard deviation (SD) of the metrics values forMNAL, the version without pseudo-labeling, and EDA for bug report identification. The format is the same as Table 2. Query Size Approach F1-scoreAccuracyRecallPrecisionReadability Identifiability 푟 Mean (SD) 푟 Mean (SD) 푟 Mean (SD) 푟 Mean (SD)푟 Mean (SD) 푟 Mean (SD) No modification2 0.850 (±0.019) 2 0.841 (±0.017) 2 0.848 (±0.038) 2 0.852 (±0.016)2 5.750 (±32.774)10.188 (±0.109) With augmentation2 0.849 (±0.014) 2 0.841 (±0.013)10.853 (±0.028) 3 0.849 (±0.014)16.632 (±34.131) 2 0.179 (±0.11)300 With pseudo-labeling10.855 (±0.018)10.847 (±0.017)10.852 (±0.031)10.859 (±0.016)3 1.071 (±36.207) 3 0.155 (±0.104) No modification2 0.861 (±0.013) 2 0.853 (±0.012 3 0.861 (±0.022) 2 0.862 (±0.012)1-0.119 (±20.631) 2 0.143 (±0.087) With augmentation 2 0.860 (±0.012) 3 0.851 (±0.011)10.865 (±0.024) 3 0.856 (±0.01)3 -4.309 (±26.025)10.146 (±0.107)500 With pseudo-labeling10.864 (±0.015)10.856 (±0.014) 2 0.863 (±0.026)10.865 (±0.010)2 -1.671 (±18.956) 3 0.136 (±0.106) No modification2 0.867 (±0.012) 2 0.859 (±0.012) 1 0.869 (±0.023)10.867 (±0.011)2 -8.617 (±18.488)10.134 (±0.096) With augmentation3 0.865 (±0.09) 3 0.855 (±0.009) 1 0.87 (±0.017) 2 0.86 (±0.01)1-7.948 (±15.341)10.132 (±0.1)700 With pseudo-labeling10.869 (±0.011)10.861 (±0.011) 1 0.871 (±0.019)10.868 (±0.009)3 -9.530 (±18.465) 2 0.122 (±0.100) surpassing both the no-modification version (0.861) and the EDA-augmented version (0.860). This shows the effectiveness of exploiting unlabeled data as opposed to perturbing labeled data (as in EDA). Pseudo-labeling is highly effective because it allows the model to learn from the vast, untapped pool of unlabeled reports without requiring extra human effort. By leveraging the model’s own high-confidence predictions on unlabeled data that is semantically similar to human-labeled ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:25 reports, we enrich the training set with diverse, real-world examples. This method is more powerful than data augmentation (EDA), which merely creates synthetic variations of existing labeled data. There appears to be a slight degradation in readability and identifiability with pseudo-labeling, as the additional pseudo-labeled reports might lead to a model that produces better uncertainty for those reports of bad readability and identifiability in the subsequent timesteps. However, we see that the effect on readability and identifiability by pseudo-labeling is negligible with larger query sizes as the degradation is often statistically insignificant. This is because the more data samples used in the updating, the more reliable the neural language model becomes, hence helping the sampling to more accurately locate those reports with larger uncertainty while having better results on readability and identifiability in the landscape. Therefore, we can conclude the following for RQ2: To RQ2:The pseudo-labeling inMNALleads to a significant improvement in the performance of the neural language model with minimal effect on the readability and identifiability. 5.3 Model-Agnostic Nature of MNAL 5.3.1 Operationalization. To verify the model-agnostic nature ofMNAL, in RQ3, we equip it with three alternative neural language models that replace BERT, these are: •CodeBERT [26] A specifically designed model for code naturalness, in which the pre-training and the attention have been adopted to cater to the strong structure of code semantics. This fits our problem well since code is also very common in a report from GitHub. •RoBERTa [54] This is a robust and widely-used extension of BERT, known for its efficiency and accuracy in various natural language processing tasks. Its architecture and training methodology make it a strong contender in our study. • RTA [24] A pre-trained model tailored for analyzing bug reports via learning a universal representation. The pre-training is completed with two distinct learning objectives, including a masked language model and a contrastive learning objective. We selected CodeBERT, RoBERTa, and RTA for our experiments because these models represent the state-of-the-art in bug report identification tasks[24,62,65]. Our choice was based on their top performance and relevance to our task, ensuring a robust evaluation ofMNAL’s model independence. Specifically, our reasons are demonstrated below: • CodeBERT and RoBERTa are optimized and widely recognized models based on the BERT architecture: –CodeBERT is specifically designed for tasks in software engineering, such as code under- standing and generation. It extends the BERT architecture to handle both natural language and programming language tasks, making it highly relevant for evaluatingMNALin code- related domains. CodeBERT has been demonstrated to outperform other models in various software engineering tasks, ensuring that our method is tested against a strong baseline in this field [26]. –RoBERTa (Robustly optimized BERT approach) is known for its effectiveness in a wide range of NLP tasks. It improves upon BERT by utilizing a more extensive training dataset and better hyperparameter tuning, which has led to state-of-the-art results in many natural language understanding benchmarks. By including RoBERTa, we ensure thatMNALis evaluated using a highly optimized general-purpose text model, allowing us to assess its generalizability [54]. •RTA is a recent and specialized model trained to generate a universal representation of bug reports. Unlike CodeBERT and RoBERTa, which are more general-purpose, RTA is tailored ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:26Guoming Long, Shihai Wang, Hui Fang, and Tao Chen 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.83 0.84 0.85 0.86 0.87 0.88 F1-score MNALMNAL ran 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.81 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score (a) CodeBERT, Query size 300 246810 Timestep 0.83 0.84 0.85 0.86 0.87 0.88 F1-score (b) CodeBERT, Query size 500 246810 Timestep 0.84 0.85 0.86 0.87 0.88 F1-score (c) CodeBERT, Query size 700 246810 Timestep 0.84 0.85 0.86 0.87 0.88 F1-score (d) RoBERTa, Query size 300 246810 Timestep 0.84 0.85 0.86 0.87 0.88 F1-score (e) RoBERTa, Query size 500 246810 Timestep 0.845 0.850 0.855 0.860 0.865 0.870 0.875 0.880 0.885 F1-score (f ) RoBERTa, Query size 700 246810 Timestep 0.85 0.86 0.87 0.88 0.89 F1-score (g) RTA, Query size 300 246810 Timestep 0.860 0.865 0.870 0.875 0.880 0.885 0.890 F1-score (h) RTA, Query size 500 246810 Timestep 0.870 0.875 0.880 0.885 0.890 F1-score (i) RTA, Query size 700 Fig. 9. Comparing the F1-score of different neural language models with and without pairingMNALover all 10 timesteps (10 runs each). The plots show the mean and standard deviation. specifically for addressing multiple downstream software engineering problems, particularly bug report analysis. This model is designed to handle the nuances and specific requirements of software engineering tasks related to bug reports, such as prioritization and triage. By including RTA, we ensure thatMNALis tested against a model that is highly specialized, thereby verifying its effectiveness across both general and task-specific contexts [24]. Our selection process was guided by the need to evaluateMNALunder a diverse set of conditions: a general-purpose code model (CodeBERT), a general-purpose text model (RoBERTa), and a task- specific model (RTA). This diversity helps to demonstrate the model independence ofMNALby showing its effectiveness across different types of pre-trained language models. Additionally, these models were selected because they represent state-of-the-art techniques in their respective fields. By choosing widely recognized and effective models, we ensure that our findings are robust and generalizable. All the above neural language models can be pre-trained and fine-tuned, hence they can be triggered in a warm start manner underMNAL. As such, we pertain those models using the same datasets as published by their original authors. To assess the benefits ofMNAL, we pair each of the above models withMNALandMNAL 푟푎푛 , which uses random sampling to select reports to be labeled ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:27 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.83 0.84 0.85 0.86 0.87 0.88 F1-score MNALMNAL ran 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 125 100 75 50 25 0 25 50 Readability (a) CodeBERT, Query size 300 246810 Timestep 100 80 60 40 20 0 Readability (b) CodeBERT, Query size 500 246810 Timestep 100 80 60 40 20 0 20 Readability (c) CodeBERT, Query size 700 246810 Timestep 100 80 60 40 20 0 20 40 Readability (d) RoBERTa, Query size 300 246810 Timestep 60 40 20 0 20 Readability (e) RoBERTa, Query size 500 246810 Timestep 80 60 40 20 0 20 Readability (f ) RoBERTa, Query size 700 246810 Timestep 100 80 60 40 20 0 20 Readability (g) RTA, Query size 300 246810 Timestep 100 80 60 40 20 0 20 Readability (h) RTA, Query size 500 246810 Timestep 70 60 50 40 30 20 10 0 10 Readability (i) RTA, Query size 700 Fig. 10. Comparing the readability of different neural language models with and without pairing MNAL over all 10 timesteps (10 runs each). The plots show the mean and standard deviation. for updating the model without the notion of active learning. Again, since there are mainly pairwise comparisons, we use the Wilcoxon Sign-rank test for verifying statistical significance. 5.3.2Findings. Figures 9, 10, and 11, together with Table 4, demonstrate the traces for 10 timesteps and the overall results with the statistical test. Our results confirm thatMNALis model-agnostic. When paired with various neural language models (CodeBERT, RoBERTa, RTA),MNALconsistently and significantly outperforms the baseline where the same models are updated with randomly sampled data (i.e.,푝<0.001). For example, when applied to RoBERTa (query size 700),MNALimproves readability from -52.139 to -16.749 and identifiability from 0.053 to 0.137. The superiority is even more obvious in readability and identifiability, where the average improvements are 78.6% and 171.5% respectively across all models. This proves that the benefits ofMNALare not tied to a specific model architecture but stem from the framework itself. By intelligently selecting which data to label (effort-aware sampling) and augmenting it with pseudo-labeled reports,MNALprovides a superior training signal to any underlying neural language model. This allows organizations to applyMNALto their preferred or existing models, enhancing both performance and human efficiency without being locked into a single technology. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:28Guoming Long, Shihai Wang, Hui Fang, and Tao Chen 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.83 0.84 0.85 0.86 0.87 0.88 F1-score MNALMNAL ran 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (a) CodeBERT, Query size 300 246810 Timestep 0.1 0.2 0.3 0.4 Identifiability (b) CodeBERT, Query size 500 246810 Timestep 0.05 0.10 0.15 0.20 0.25 0.30 0.35 0.40 Identifiability (c) CodeBERT, Query size 700 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (d) RoBERTa, Query size 300 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (e) RoBERTa, Query size 500 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (f ) RoBERTa, Query size 700 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (g) RTA, Query size 300 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (h) RTA, Query size 500 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (i) RTA, Query size 700 Fig. 11. Comparing the Identifiability of different neural language models with and without pairingMNALover all 10 timesteps (10 runs each). The plots show the mean and standard deviation. We still observe that there are fluctuations in the readability and identifiability due to the same reason mentioned for RQ1. The performance in terms of the F1-score, on the other hand, is improved in a steady state. In conclusion, we say that: To RQ3:MNALis indeed model-agnostic: no matter which underlying neural language model is employed, theMNALframework can still considerably improve their performance with signifi- cantly reduced efforts required for labeling. 5.4 Comparing MNAL with State-of-the-art Approaches 5.4.1 Operationalization. To confirm the overall benefit ofMNAL, we compare it against state-of- the-art approaches that rely on active learning for bug report identification. Table 5 illustrates the compared approaches published since 2015. As can be seen, those approaches leverage mainly TF- IDF to quantify the reports, which are paired with different machine learning models. In particular, all of them mainly rely on uncertainty for the sampling with active learning.EMBLEMis a unique ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:29 Table 4. Comparing the mean and standard deviation (SD) of the metrics values forMNALunder different neural language models against updating them directly. Where ‡ indicates the value is higher than the counterpart with high statistical significance (i.e., 푝< 0.001). Query Size ModelApproachF1-scoreAccuracyRecallPrecisionReadabilityIdentifiability MNAL 푟푎푛 0.853 (±0.014) 0.846 (±0.013) 0.841 (±0.025) 0.866 (±0.011)-51.532 (±14.501) 0.053 (±0.003) CodeBERT MNAL0.864 (±0.016) ‡ 0.855 (±0.015) ‡ 0.863 (±0.027) ‡ 0.865 (±0.01)-5.500 (±40.631) ‡ 0.165 (±0.118) ‡ MNAL 푟푎푛 0.861 (±0.010) 0.852 (±0.009) 0.86 (±0.021)0.861 (±0.010)-50.017 (±14.424) 0.053 (±0.003) RoBERTa MNAL 0.868 (±0.011) ‡ 0.858 (±0.011) ‡ 0.874 (±0.017) ‡ 0.862 (±0.012)-0.473 (±24.745) ‡ 0.157 (±0.117) ‡ MNAL 푟푎푛 0.875 (±0.007) 0.866 (±0.007) 0.880 (±0.015) 0.870 (±0.010)-51.317 (±13.598) 0.053 (±0.003) 300 RTA MNAL 0.880 (±0.010) ‡ 0.871 (±0.010) ‡ 0.891 (±0.017) ‡ 0.870 (±0.011)-5.331 (±31.372) ‡ 0.156 (±0.121) ‡ MNAL 푟푎푛 0.862 (±0.010) 0.854 (±0.009) 0.855 (±0.019) 0.869 (±0.008)-52.605 (±11.605) 0.053 (±0.002) CodeBERT MNAL 0.870 (±0.013) ‡ 0.861 (±0.012) ‡ 0.872 (±0.024) ‡ 0.868 (±0.009)-14.026 (±24.921) ‡ 0.135 (±0.102) ‡ MNAL 푟푎푛 0.865 (±0.008) 0.856 (±0.007) 0.865 (±0.015) 0.865 (±0.007)-52.010 (±10.750) 0.053 (±0.002) RoBERTa MNAL 0.873 (±0.010) ‡ 0.864 (±0.010) ‡ 0.878 (±0.019) ‡ 0.867 (±0.008)-12.000 (±23.367) ‡ 0.137 (±0.109) ‡ MNAL 푟푎푛 0.880 (±0.005) 0.871 (±0.005) 0.887 (±0.011) 0.873 (±0.008)-51.033 (±10.880) 0.053 (±0.002) 500 RTA MNAL0.884 (±0.007) ‡ 0.875 (±0.007) ‡ 0.893 (±0.012) ‡ 0.875 (±0.008)-16.241 (±27.093) ‡ 0.141 (±0.122) ‡ MNAL 푟푎푛 0.866 (±0.008) 0.858 (±0.007) 0.862 (±0.015) 0.871 (±0.008)-54.596 (±14.400) 0.052 (±0.002) CodeBERT MNAL0.875 (±0.010) ‡ 0.866 (±0.010) ‡ 0.878 (±0.018) ‡ 0.871 (±0.007)-13.356 (±15.890) ‡ 0.126 (±0.087) ‡ MNAL 푟푎푛 0.867 (±0.006) 0.859 (±0.006) 0.867 (±0.013) 0.868 (±0.006)-52.139 (±11.494) 0.053 (±0.002) RoBERTa MNAL0.875 (±0.010) ‡ 0.866 (±0.010) ‡ 0.882 (±0.015) ‡ 0.869 (±0.008)-16.749 (±24.792) ‡ 0.137 (±0.102) ‡ MNAL 푟푎푛 0.881 (±0.004) 0.873 (±0.004) 0.887 (±0.010) 0.875 (±0.007)-51.531 (±9.335)0.053 (±0.002) 700 RTA MNAL 0.885 (±0.006) ‡ 0.877 (±0.006) ‡ 0.894 (±0.011) ‡ 0.877 (±0.006)-16.551 (±21.921) ‡ 0.139 (±0.105) ‡ Table 5. The state-of-the-art active learning approaches for bug report identification or a similar problem. (TF-IDF, NBM, RF, and SVM stand for Term Frequency–Inverse Document Frequency, Multinomial Naive Bayes, Random Forest, and Support Vector Machine, respectively.) NameModelSamplingPseudo Labeling Year Thung et al. [82]TF-IDF and SVMUncertaintyNo2015 hbrPredictor [93] TF-IDF and NBM UncertaintyNo2021 Ge et al. [29]TF-IDF and RFUncertaintyNo2021 EMBLEM [84]TF-IDF and SVMUncertainty and Certainty No2022 Table 6. The state-of-the-art cross project approaches for bug report identification or a similar problem. (MLP and BERT stands for Multilayer Perceptron and Bidirectional encoder representations from transformers, respectively.) NameModelYear Herbold et al. [37] auto-fastText2020 Perez et al. [71]TF-IDF and MLP2021 MHNurf [56]Attention Network 2022 Meher et al. [62]BERT2024 example in which the certainty metric is used to further extract the reports from the set of highly uncertain samples. However, none of them have exploited pseudo-labeling to enrich the amount of labeled reports. In addition, we also conducted a comparison betweenMNALand the state-of-the-art cross-project methods. Table 6 presents the state-of-the-art methods we used for comparison. Since those methods follow a typical supervised learning paradigm, we used 3,000, 5,000 and 7,000 training data sizes to train the state-of-the-art cross-project method, whileMNALwas trained using the 10 time step with 300, 500 and 700 query sizes. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:30Guoming Long, Shihai Wang, Hui Fang, and Tao Chen Furthermore, we also compareMNALagainst a state-of-the-art Large Language Model (LLM) - GPT-4o-mini 5 . Our implementation leverages the GPT-4o-mini model to automatically classify bug reports using few-shot learning, varying the number of provided examples (n-shot) from 0 to 5 to analyze the performance improvement with additional labeled examples, where 1 shot here equivalent to a pair of bug report and non-bug report. The implementation includes several steps. We structured the prompts to be concise yet informative, asking the model directly if a given text was a bug report, to which it should respond with a binary answer (0 for no, 1 for yes). Specifically, we designed the basic prompt for zero-shot learning, where the model was asked directly if a given text was a bug report. The prompt was formulated as: “Is this a bug report? Respond with only 1 for yes or 0 for no:푇” followed by the descriptions in the bug report푇to be predicted. For few-shot in context learning, we include labeled examples in the prompt to provide context and improve the model’s understanding. A few-shot prompt with a pair of positive and negative examples was structured as: “Here are some examples of bug reports and non-bug reports: “Here are some examples of bug reports and non-bug reports:퐸 1 ,퐸 1 , ...,퐸 푛 . Now, is this a bug report? Respond with only 1 for yes or 0 for no: T", where퐸 1 ,퐸 1 , ...,퐸 푛 are description of different non-bug relevant reports and푇is the bug report to be predicted. To ensure the validity of our few-shot prompting testing, for each test example, we randomly select an example text from the labeled training data to use as a prompt. Finally, we prepared the same test data used forMNALto evaluate the performance of GPT-4o-mini. The observation that 1-shot learning outperforms 5-shot learning is indeed counterintuitive, but a reasonable explanation can be: the examples used for the few-shot prompts were selected randomly from the labeled training data for each report being tested. The superior performance of the 1-shot setting suggests that a single, randomly selected example was often sufficient and, on average, provided a clearer, more direct signal to the model for this binary classification task. Conversely, when providing five randomly selected examples, the probability of introducing noise, ambiguity, or even contradictory information into the prompt increases significantly. The bug reports within the dataset are noted to be diverse in format and complexity. A random set of five reports could easily include examples that are atypical, poorly written, or present conflicting patterns. This noisy and more complex context may have confused the model, hindering its ability to identify the core task, rather than refining its understanding. This phenomenon, where performance does not monotonically increase with the number of examples, can occur in few-shot learning when the quality and direct relevance of the examples are more critical than the sheer quantity. Essentially, the focused context of a single good example (selected by chance) proved more effective than the potentially confusing context of five random examples. To reduce unnecessary noise when running the state-of-the-art approaches, we use the publicly available code published by their authors. Since there are more than two comparisons, we use the Scott-Knott test and Wilcoxon sign-rank test to verify the statistical significance. 5.4.2Findings. As illustrated in Figure 12, remarkably,MNALsignificantly outperforms the state-of- the-art active learning approaches in all cases starting from the first timestep. At a query size of 700, MNAL’s F1-score is 0.869, while the next best approach (Ge et al.) scores only 0.829, as shown in Table 7. This advantage stems from its holistic design; unlike prior active learning methods that rely on simpler models (e.g., TF-IDF) and focus only on uncertainty,MNALcombines the semantic power of neural networks with a dual focus on model accuracy and human efficiency. The combined effect of pseudo-labeling and the neural language model within active learning leads to superior F1-scores. Apart from the performance in identifying bug reports, the effort-aware uncertain sampling results in drastically improved readability and identifiability for the queried reports, with up to 95.8% and 196.0% improvements, respectively. The statistical test from Table 7 further confirms that 5 https://openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:31 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.80 0.82 0.84 0.86 0.88 F1-score MNALhbrPredictorGe et al.Thung et al.EMBLEM 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.78 0.80 0.82 0.84 0.86 F1-score (a) Query size 300 246810 Timestep 0.80 0.82 0.84 0.86 0.88 F1-score (b) Query size 500 246810 Timestep 0.80 0.81 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score (c) Query size 700 246810 Timestep 120 100 80 60 40 20 0 20 40 Readability (d) Query size 300 246810 Timestep 120 100 80 60 40 20 0 20 Readability (e) Query size 500 246810 Timestep 100 80 60 40 20 0 20 Readability (f ) Query size 700 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (g) Query size 300 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (h) Query size 500 246810 Timestep 0.1 0.2 0.3 0.4 Identifiability (i) Query size 700 Fig. 12. ComparingMNALwith state-of-the-art active learning approaches for bug report identification over all 10 timesteps (10 runs each). The plots show the mean and standard deviation. MNALachieves superior results on nearly all performance metrics. Although compared state-of-the- art approaches also rely on active learning, they do not consider identifiability, reliability, or the pseudo-labeling process. Consequently, they incur much larger human efforts, as seen from the identifiability and reliability values in Table 7, without achieving better accuracy than MNAL. Similarly, as shown in Table 8, we also compareMNALagainst the state-of-the-art cross-project approaches for bug report identification. Remarkably,MNALdemonstrates significant improvements over other cross-project methods, not only in terms of accuracy and F1-score but also readability and identifiability. Unlike traditional approaches, which mainly rely on a passive learning approach, the integration of neural active learning and effort-aware uncertainty sampling inMNALhas led to a considerable reduction in human labeling efforts. This is further supported by the superior performance metrics across all evaluation criteria compared to the SOTA cross-project approach [62] using the advanced neural language model BERT, while maintaining a competitive recall. Overall, MNALeffectively improves labeling efficiency and reduces labeling effort, providing a more reliable solution for cross-project bug report identification. In comparison with LLMs,MNAL’s performance is also impressive. As shown in Table 9,MNAL’s F1-score is 16.2% higher than GPT-4o-mini’s zero-shot baseline and 8.7% higher than the LLM’s best one-shot performance. This demonstrates that a smaller, specialized model fine-tuned with ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:32Guoming Long, Shihai Wang, Hui Fang, and Tao Chen Table 7. Comparing the mean and standard deviation (SD) of the metrics values forMNALand the state-of- the-art approaches for bug report identification. The format is the same as Table 2. Query Size Approach F1-scoreAccuracyRecallPrecisionReadability Identifiability 푟 Mean (SD) 푟 Mean (SD) 푟 Mean (SD) 푟 Mean (SD)푟 Mean (SD)푟 Mean (SD) EMBLEM4 0.797 (±0.011) 4 0.775 (±0.010) 4 0.831 (±0.025) 3 0.767 (±0.014)2 -87.015 (±13.525) 2 0.050 (±0.002) hbrPredictor4 0.796 (±0.006) 5 0.756 (±0.005)10.894 (±0.021) 4 0.718 (±0.009)2 -86.748 (±13.417) 2 0.050 (±0.003) Ge et al. 2 0.815 (±0.013) 2 0.803 (±0.014) 5 0.819 (±0.031) 2 0.813 (±0.024)2 -86.625 (±13.521) 2 0.050 (±0.003) Thung et al.3 0.802 (±0.009) 3 0.778 (±0.010) 3 0.842 (±0.021) 3 0.766 (±0.014)2 -86.505 (±12.561) 2 0.050 (±0.002) 300 MNAL10.855 (±0.018)10.847 (±0.017) 2 0.852 (±0.031)10.859 (±0.016)11.770 (±36.207)10.167 (±0.104) EMBLEM4 0.810 (±0.009) 4 0.789 (±0.009) 5 0.842 (±0.015) 4 0.779 (±0.011)2 -88.767 (±12.004) 2 0.050 (±0.001) hbrPredictor5 0.803 (±0.003) 5 0.764 (±0.005)10.902 (±0.011) 5 0.724 (±0.008)2 -87.837 (±10.198) 2 0.051 (±0.002) Ge et al.2 0.823 (±0.008) 2 0.805 (±0.010) 3 0.853 (±0.023) 2 0.797 (±0.021)2 -87.202 (±10.415) 2 0.050 (±0.002) Thung et al.3 0.814 (±0.009) 3 0.794 (±0.010) 4 0.846 (±0.018) 3 0.784 (±0.013)2 -87.023 (±10.714) 2 0.051 (±0.002) 500 MNAL 10.864 (±0.015)10.856 (±0.014) 2 0.863 (±0.026)10.865 (±0.010)1-2.511 (±18.956)10.145 (±0.106) EMBLEM4 0.819 (±0.008) 4 0.801 (±0.009) 4 0.844 (±0.015) 4 0.795 (±0.009)2 -88.860 (±9.759) 2 0.051 (±0.001) hbrPredictor 5 0.806 (±0.003) 5 0.770 (±0.005)10.899 (±0.010) 5 0.731 (±0.007)2 -88.194 (±7.588) 2 0.050 (±0.001) Ge et al.2 0.829 (±0.007) 2 0.817 (±0.006) 5 0.835 (±0.017) 2 0.825 (±0.010)2 -88.113 (±7.541) 2 0.050 (±0.001) Thung et al.3 0.821 (±0.008) 3 0.804 (±0.008) 3 0.848 (±0.015) 3 0.797 (±0.009)2 -88.102 (±7.290) 2 0.050 (±0.001) 700 MNAL 10.869 (±0.011)10.861 (±0.011) 2 0.871 (±0.019)10.868 (±0.009)1-10.555 (±18.465)10.132 (±0.100) Table 8. Comparing the mean and standard deviation of the metrics values forMNALand the state-of-the-art cross project approaches for bug report identification. The format is the same as Table 2. Training Size Approach F1-scoreAccuracyRecallPrecisionReadabilityIdentifiability 푟 Mean (SD) 푟 Mean (SD) 푟 Mean (SD) 푟 Mean (SD)푟 Mean (SD)푟 Mean (SD) Herbold et al.5 0.700 (±0.016) 5 0.696 (±0.012) 5 0.708 (±0.005) 5 0.706 (±0.021)2 -86.896 (±12.999) 2 0.051 (±0.001) Perez et al.4 0.712 (±0.015) 4 0.715 (±0.012) 4 0.721 (±0.022) 4 0.722 (±0.024)2 -87.5678 (±12.152) 2 0.050 (±0.003) MHNurf3 0.804 (±0.043) 3 0.795 (±0.015) 3 0.814 (±0.011) 3 0.806 (±0.035)2 -86.650 (±13.658) 2 0.050 (±0.002) Meher et al.2 0.847 (±0.004) 2 0.838 (±0.008) 2 0.836 (±0.021) 2 0.853 (±0.014)2 -85.432 (±13.431) 2 0.051 (±0.001) 3000 MNAL10.855 (±0.018)10.847 (±0.017)10.852 (±0.031)10.859 (±0.016)11.770 (±36.207)10.167 (±0.104) Herbold et al.5 0.711 (±0.006) 5 0.704 (±0.004) 5 0.700 (±0.002) 4 0.713 (±0.06)2 -86.958 (±11.576) 2 0.050 (±0.003) Perez et al.4 0.724 (±0.007) 4 0.734 (±0.008) 4 0.744 (±0.011) 4 0.736 (±0.012)2 -86.388 (±12.547) 2 0.051 (±0.001) MHNurf3 0.816 (±0.006) 3 0.807 (±0.018) 3 0.821 (±0.015) 3 0.815 (±0.018)2 -88.494 (±11.979) 2 0.051 (±0.003) Meher et al.2 0.857 (±0.009) 2 0.848 (±0.007) 2 0.858 (±0.017) 2 0.861 (±0.013)2 -86.754 (±12.567) 2 0.051 (±0.002) 5000 MNAL10.864 (±0.015)10.856 (±0.014)10.863 (±0.026)10.865 (±0.010)1-2.511 (±18.956)10.145 (±0.106) Herbold et al.5 0.721 (±0.015) 5 0.715 (±0.021) 5 0.709 (±0.021) 5 0.719 (±0.007)2 -88.861 (±8.875)2 0.051 (±0.002) Perez et al.4 0.732 (±0.004) 4 0.741 (±0.007) 4 0.752 (±0.008) 4 0.741 (±0.013)2 -88.056 (±6.994)2 0.050 (±0.003) MHNurf 3 0.824 (±0.013) 3 0.812 (±0.012) 3 0.828 (±0.008) 3 0.821 (±0.004)2 -88.339 (±6.856)2 0.050 (±0.001) Meher et al.2 0.864 (±0.015) 2 0.852 (±0.011) 2 0.859 (±0.023) 2 0.866 (±0.013)2 -87.875 (±7.067)2 0.051 (±0.006) 7000 MNAL10.869 (±0.011)10.861 (±0.011)10.871 (±0.019)10.868 (±0.015)1-10.555 (±18.413)10.132 (±0.100) Table 9. Comparing the performance ofMNALand the state-of-the-art pseudo-labeling model and LLMs in bug report identification task. The format is the same as Table 4. Approach PromptingF1-score Accuracy Recall Precision AUC GPT-4o-mini 0-shot0.7070.563 ‡ 0.9890.5500.533 1-shot0.7820.7320.9010.6900.720 2-shot0.6530.5630.7730.5660.548 3-shot 0.6480.5160.8350.5290.494 4-shot0.6890.5340.9680.5340.504 5-shot0.6680.5260.8940.5330.500 HINT -0.8480.8450.8210.877 ‡ 0.889 ‡ MNAL -0.869 ‡ 0.861 ‡ 0.8710.8680.878 the intelligent, human-in-the-loopMNALframework can achieve better performance on a specific task than a massive, general-purpose model. Furthermore, we comparedMNALwith HINT, another ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:33 state-of-the-art approach that utilizes pseudo-labeling. As shown in Table 9, HINT outperforms MNALon the Precision and AUC metrics. However,MNALdemonstrates superior performance in F1- score, Accuracy, and Recall. This is significant because, fundamentally, HINT does not incorporate the human-machine collaborative aspect that is central to our work.MNAL’s ability to actively reduce annotator fatigue by selecting for readable and identifiable reports represents a key practical advantage not captured by these metrics alone. Therefore, for RQ4, we conclude that: To RQ4:MNALsignificantly outperforms other state-of-the-art approaches on different perfor- mance metrics, readability and identifiability. 5.4.3 Analysis ofMNAL’s Superiority over Active Learning Counterparts. The experimental results above demonstrate thatMNALconsistently and significantly outperforms state-of-the-art active learning approaches for bug report identification across performance and effort metrics. The underlying reasons for this superiority can be attributed to several design choices that fundamentally advance beyond traditional active learning frameworks. (1) MNALis built on Neural Active Learning, which pairs a powerful neural language model with the active learning. In contrast, the compared state-of-the-art methods rely on statistical machine learning models like SVM, NBM, or Random Forest, combined with TF-IDF for text representation. TF-IDF can capture keyword frequency but fails to understand the deeper semantic and contextual nuances of language found in bug reports.MNAL, by using models like BERT, leverages contextualized embeddings that provide a much richer, more meaningful representation of the reports. (2) MNALintroduces a more comprehensive, human-centric sampling strategy. The compared active learning methods primarily use uncertainty sampling to select reports for labeling. While this can improve model accuracy, it often queries complex and hard-to-understand reports, leading to cognitive fatigue for developers.MNAL’s effort-aware uncertainty sampling explicitly addresses this by creating a mutualistic relationship between the human and the machine. It balances the model’s need for informative samples (uncertainty) with the developer’s need for reports that are easier to process (readability and identifiability). This multi-objective approach not only reduces human labeling effort significantly but also ensures the quality and timeliness of the labels provided, indirectly benefiting the model. (3) MNALenriches the training data at each step through pseudo-labeling, a feature absent in the other active learning approaches. After developers label a small set of queried reports, MNALleverages the neural model’s rich embedding space to find the most similar unlabeled reports and assigns them a “pseudo-label”. This process effectively multiplies the value of each human-provided label, expanding the training set with high-confidence samples without incurring additional human labeling cost. 5.5 Human-centric Case Study 5.5.1 Operationalization. To fully verify the effort reduction of active learning achieved by the mutualistic relation introduced inMNALand demonstrate the practical values ofMNAL, we conduct a case study involving human developers who actually label the reports during the training/updating. We recruit 10 software engineering research students who have the necessary understanding of software development and have identified Github reports before, but their years of experience vary between one and four years. Given that bug identification is a relatively straightforward task based on objective evidence (e.g., crash logs, reproduction steps), existing literature suggests that such students are qualified proxies for industrial developers in this context. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:34Guoming Long, Shihai Wang, Hui Fang, and Tao Chen Table 10. ComparingMNALandMNAL 푟푎푛 in the qualitative study involving humans. The format is the same as Table 4. Type Metric MNAL 푟푎푛 MNAL Effort Mean/deviation of readability score0.79 (±0.83)0.20 (±0.47) ‡ Mean/deviation of identifiability score0.71 (±0.88)0.25 (±0.48) ‡ Time Total labeling time for a timestep (300 reports)128 mins40 mins ‡ Expected number of reports labeled per hour140460 ‡ Money Total monetary cost for labeling in a timestep (300 reports) £21.91£6.67 ‡ In particular, the participants are asked to label two sets of 300 reports queried byMNALand MNAL 푟푎푛 (from RQ1) in one timestep, respectively. To avoid perception bias, each participant labels 60 randomly selected reports of which half are fromMNAL, but the participants have no knowledge about the source of their assigned reports. Upon completing the labeling of a report, the participants are asked to rate how they feel about its readability and identifiability using a Likert scale ranging from 0 to 4, where 0 denotes the most readable/identifiable level. This enables us to qualitatively assessMNALin terms of human cognition. In addition, the participants are paid £10.42/hour in compliance with the national minimum wage legislation in the UK, thus serving as a basis to evaluate the real cost savings when applying MNAL. To ensure the quality of the study, we implemented two key procedures: •Shuffled order: The 60 reports assigned to each participant were presented in a randomly shuffled order to prevent ordering effects and perception bias. •Mandatory breaks: To avoid fatigue-induced errors, participants were required to take a mandatory break every 20 reports or were allowed to complete the task across multiple sessions. Furthermore, to enhance the reliability of our findings, we invited an expert an expert to conduct a post-hoc spot check by re-evaluating 10% of the human-labeled results with equal split from both two sets (i.e., 60 reports). The consistency/inter-rater reliability of labels between the expert and students on the same set of reports reaches 93.33% with a Cohen’s휅=0.867, confirming the reliability of the results given by the participants. 5.5.2Findings. The human case study provides compelling real-world evidence ofMNAL’s benefits. From Table 10, we see that reports selected byMNALwere significantly easier to handle. Compared with the random baseline (MNAL 푟푎푛 ), participants’ qualitative ratings showed a 74.7% improvement in perceived readability and a 64.8% improvement in perceived identifiability. This subjective feedback was mirrored by objective measurements. When comparing the saving of labeling efforts, MNAL 푟푎푛 needs 128 minutes to label all 300 reports whileMNALonly needs 40 minutes—a remarkable reduction of 68.8%. This implies thatMNALcan enable humans to label 460 reports per hour, in contrast to the 140 reports byMNAL 푟푎푛 . The benefits provided byMNALare even more clear when considering the monetary cost; to label 300 reports,MNALcosts only £6.67, which is more than 3x cheaper than the £21.91 incurred byMNAL 푟푎푛 . These results directly validate the practical utility of MNAL, confirming it provides a tangible, efficient, and cost-effective solution for the human-intensive task of bug report labeling in practice. To take a closer look at each individual participant, we observe similar results, as shown in Figure 13. The significant reduction in time and cost is a direct result of the effort-aware sampling, which successfully filters for reports that are less ambiguous and require less cognitive effort to classify. As can be seen, this holds true across all 10 participants, regardless of their years of experience; almost all participants ratedMNALbetter than its counterpart in terms of readability and ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:35 00.511.52 1 2 3 4 5 6 7 8 9 10 Mean/deviation of readability score Participant number MNAL MNAL 푟푎푛 00.511.52 1 2 3 4 5 6 7 8 9 10 Mean/deviation of identifiability score Participant number MNAL MNAL 푟푎푛 01020 1 2 3 4 5 6 7 8 9 10 Total labeling time (minute) Participant number MNAL MNAL 푟푎푛 Fig. 13. ComparingMNALandMNAL 푟푎푛 with respect to every individual of the 10 human participants (each labels 30 reports). For each participant, the pairwise comparison shows 푝< 0.001. identifiability (i.e., a lower score means better). In particular, a few participants even considered all 30 reports queried byMNALas “highly readable” or “highly identifiable” (an average rating of 0). Similarly, when comparing the total labeling time of 30 reports for each participant,MNALneeds considerably shorter time on each case. For example, Participant 6 used 5.73 minutes to label the reports queried by MNAL, but this became 26.62 minutes for those from MNAL 푟푎푛 . Therefore, in response to RQ5, we conclude that: To RQ5:Comparing toMNAL 푟푎푛 ,MNALsignificantly reduces the effort for labeling the reports, within the same time frame, 3×monetary saving while having 74.7% and 64.8% improvement in the qualitative readability and identifiability score. 6 DISCUSSION In this section, we discuss a few notable observations drawn from the experiments and the factors to consider when applying MNAL in practical scenarios. 6.1 The Changing Trade-off on Uncertainty, Readability, and Identifiability It is not hard to understand that the uncertainty, readability, and identifiability involved in effort- aware uncertainty sampling are potentially conflicting. By using an aggregated quality-effort score with equal weights among the three, we seek to arrive at the trade-off points that did not show discriminated preferences over any of those metrics. However, although the readability and identifiability of a report never change, the sampling landscape can still evolve between different timesteps due to the varying uncertainty of the reports produced by the updated neural language model. Therefore, within the landscape, the exact positions of the reports with the best quality-effort scores can shift. Figure 14 illustrates the changing sampling landscapes across the timesteps for one run under MNALwith a query size of 300. Since it is unrealistic to visualize more than 1 million points, for each timestep, we plot the top 1,000 best reports based on the quality-effort score. We also highlight the 700 reports, which have not been selected from the 1,000 best ones for querying, with their new positions in the subsequent timestep after the changed landscape. This helps us to better understand the landscape shift. The 700 points depicted from the previous timestep are selected because the top 300 points from previous timestep have already been chosen and placed into the ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:36Guoming Long, Shihai Wang, Hui Fang, and Tao Chen 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 0 0.2 0.4 −1,000 −500 0 0 0.5 1 Uncertainty Readability Identifiability Top 1000 Top 300 Top 700 reports from the last timestep 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 0 0.2 0.4 −1,000 −500 0 0 0.5 1 Uncertainty Readability Identifiability (a) Timestep 1 0 0.2 0.4 −1,000 −500 0 0 0.5 1 Uncertainty Readability Identifiability (b) Timestep 2 0 0.2 0.4 −1,000 −500 0 0 0.5 1 Uncertainty Readability Identifiability (c) Timestep 3 0 0.2 0.4 −1,000 −500 0 0 0.5 1 Uncertainty Readability Identifiability (d) Timestep 4 0 0.2 0.4 −1,000 −500 0 0 0.5 1 Uncertainty Readability Identifiability (e) Timestep 5 0 0.2 0.4 −1,000 −500 0 0 0.5 1 Uncertainty Readability Identifiability (f ) Timestep 6 0 0.2 0.4 −1,000 −500 0 0 0.5 1 Uncertainty Readability Identifiability (g) Timestep 7 0 0.2 0.4 −1,000 −500 0 0 0.5 1 Uncertainty Readability Identifiability (h) Timestep 8 0 0.2 0.4 −1,000 −500 0 0 0.5 1 Uncertainty Readability Identifiability (i) Timestep 9 Fig. 14. The changing sampling landscapes considered byMNALacross the timesteps when푘=300, i.e., only the top 300 reports according to the quality-effort score are queried in a timestep. labeled pool in that previous timestep. Thus, including them again in the current timestep would not provide meaningful insights as they are no longer part of the unlabeled set. Therefore, the top 700 points shown represent the remaining subset of the top 1000 points from the last timestep that have not yet been labeled. Notably, a key information that we are trying to illustrate is how the uncertainty of the same points selected from timestep푡and using the model at timestep푡has been changed using the updated model at timestep푡 +1. Therefore, although we seek to illustrate the newly selected top 1000 points according to the quality-effort score at each timestep, the best 300 points from the top 1000 ones selected at timestep푡would have already been used to update the model for timestep 푡 +1, therefore, it makes less sense to include those 300 points since they are part of the training data for the model at timestep푡 +1 (their uncertainty would be close to zero). As such, in addition to the best 1000 newly selected points, we illustrate the uncertainty of the same remaining 700 points (from 1000 selected at timestep푡+1) using the updated model at timestep푡+1, and compare how the uncertainty has been shifted with the same 700 points at timestep 푡 . ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:37 Clearly, we see that there are obvious discrepancies in the landscape between different timesteps due to the changes of uncertainty on the unlabeled reports, hence the positions of the best 1,000 reports in terms of the quality-effort score have also been shifted. A key information we can observe is that there is always a considerable change in terms of the uncertainty between timesteps for the remaining unselected reports (e.g., comparing the◦at Timestep 2 and the×at Timestep 3). This makes sense, since at Timestep푛+1 the model has been updated with the top 300 reports selected from Timestep푛, then the remaining 700 best reports from Timestep푛would also be very likely to become known by the model, hence a significant drop of the uncertainty for those reports from the updated model at Timestep 푛+ 1. Interestingly, if we look closer at the relative positions of the unselected reports in the landscape of a Timestep푛(the◦) and their corresponding positions in the landscape of the subsequent Timestep 푛+1 (the×), the overall change of uncertainty for those between Timestep 1 and Timestep 2 are larger than that of those between any following timesteps. This has caused a significantly bigger shift of the 1,000 best reports in Timestep 2 compared with those in Timestep 1. That is, the 1,000 best reports have shifted from Timestep 1 to Timestep 2 on all three dimensions whereas those best reports between other timesteps mainly move along the dimension of uncertainty, e.g., from Timestep 2 to Timestep 3. We found that these observations are due to the fact that the neural language model has not yet been fine-tuned with the GitHub reports when conducting the sampling at Timestep 1, therefore it has not reached a stable state, leading to larger change on the uncertainty in Timestep 1 and hence the bigger landscape shift. The following timesteps (from Timestep 2 onwards), in contrast, would rely on a model that has already been updated with some reports, therefore the change in the landscape is smaller. The above is also the key reason that, in Section 5, there are always considerable changes in readability and identifiability underMNALfrom Timestep 1 to Timestep 2. Another worth noting finding is that, for the same unselected reports among the best 1,000 ones in a Timestep푛, the change between their uncertainty in the landscape of푛and that of the subsequent Timestep푛+1 tends to shrink with more updates, e.g., the difference of uncertainty between the◦in Timestep 9 and the×in Timestep 10 is much smaller than that between the◦ in Timestep 2 and the×in Timestep 3. This is because the neural language model inMNALhas learned from increasingly more labeled reports for identifying their bug relevance, which gradually consolidates itself, hence the changes in the uncertainty of reports are reduced. 6.2 On the Trade-off Between Human-Friendly and Model-Beneficial Data During model training, we realize that samples friendly to human interpretation do not necessarily contribute positively to the model’s identification performance. This section elaborates on this trade-off, showing the characteristics of data that, while potentially difficult for humans to parse, are information-rich and beneficial for training a neural language model. It is methodologically challenging to systematically isolate the most impactful training instances for two main reasons. First, it is difficult to precisely attribute improvements in model accuracy to a specific data point, as the contribution of any single report depends on the model’s state and the stage of training at which it is introduced. Second, the use of batch training complicates this analysis: when a performance increase follows a particular batch, it becomes hard to determine exactly which individual report within that batch was responsible for the improvement. Despite these challenges, we can illustrate the trade-off between readability and informational value through contrasting examples from our dataset. Consider the well-structured bug report shown in Figure 15a. This report is clear, concise, and follows a standard template, making it very easy for a human to understand the issue. It describes the bug, provides steps to reproduce it, and lists the expected behavior and system information. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:38Guoming Long, Shihai Wang, Hui Fang, and Tao Chen [BUG] Must toggle "Force TTY" to disable TTY mode Title **Describe the bug** I use btop on RISC-V with qemu. It is exciting to see that btop really works on such a niche platform, however, I have to **toggle** "Force TTY" to disable TTY mode, no matter it was true or false before. **To Reproduce** 1. Download ubuntu-20.04.2-preinstalled-server-riscv64 and run it on qemu 2. Build and run btop 3. Now TTY mode is enabled on **each** start 4. Toggle "Force TTY" to disable it **Expected behavior** The value of "Force TTY" should be respected. **Info (please complete the following information):** - btop++ version: 1.2.7 (commit 7d86602e) - Compiler and version: g++-10 (Ubuntu 10.3.0-1ubuntu1~20.04) 10.3.0 - Architecture: riscv - Platform: Linux - Kernel: 5.13.0-1026-generic - Terminal used: probably qemu's default TTY - Font used: Monospace Description (a) Human-friendly exampled report Cannot stream mp4 videos on 2.0.2-1 Title Code is pretty simple ``` source = playlist(reload_mode="watch", mode="normal", "list.m3u") enc=%ffmpeg( format="flv", %audio.copy, %video.copy) output.url( fallible=true, url="rtmp://localhost:1935/live", enc, source) ``` please check logs below ``` 2022/01/15 05:27:10 >>> LOG START 2022/01/15 05:27:09 [main:3] Liquidsoap 2.0.2 2022/01/15 05:27:09 [main:3] Using: bytes=[distributed with OCaml 4.02 or above] pcre=7.5.0 sedlex=2.5 menhirLib=20 211128 curl=0.9.2 dtools=0.4.4 duppy=0.9.2 cry=0.6.5 m=0.7.2 ogg=0.7.0 ogg.decoder=0.7.0 vorbis=0.8.0 vorbis.decod er=0.8.0 mad=0.5.1 dynlink=[distributed with Ocaml] lame=0.3.5 ffmpeg=1.1.1 samplerate=0.1.6 taglib=0.3.8 camomile= 1.0.2 2022/01/15 05:27:09 [dynamic.loader:3] Could not find dynamic module for fdkaac encoder. 2022/01/15 05:27:09 [clock:3] Using builtin (low-precision) implementation for latency control 2022/01/15 05:27:10 [list.m3u:4] Reloading playlist. 2022/01/15 05:27:10 [request:5] Resolving request [[list.m3u]]. 2022/01/15 05:27:10 [request:5] Resolved to [[list.m3u]]. 2022/01/15 05:27:10 [playlist parser:4] Trying youtube-dl parser 2022/01/15 05:27:10 [playlist parser:4] Trying application/x-cue parser (130 lines of log omitted) I can stream with `ffmpeg -v debug -re -f concat -safe 0 -i ./list.txt -c copy -f flv rtmp://localhost:1935/live` without any issue. by the way, single file works fine. Description (b) Model-beneficial exampled report Fig. 15. Excerpt of the human-friendly vs. model-beneficial exampled reports From the model’s perspective, however, the information in this report may be limited. Its vocabulary and structure are conventional, offering few new patterns for a model already trained on vast quantities of similar text. While it effectively communicates the bug to a human, its informational density is relatively low and contributes minimally to enhancing the model’s ability to diagnose more complex or obscure issues. In contrast, the second report, shown in Figure 15b, is less accessible to a human reader, consisting primarily of a short code snippet and a long, dense log file. Despite its lack of human-friendly structure, this report is valuable for the model. The log file contains a high density of specific technical information—library function calls, precise error messages (e.g., Server error: Already publishing), and stack traces (e.g., Avutil.Error(Operation not permitted)). Such data provide rich contextualized patterns, enabling the model to learn complex correlations among specific library function calls, event sequences in logs, and relevant bug types. 6.3 On the Implications of Pseudo-labeled Reports While having some pseudo-labeled reports as part of the updating/training process inMNALis certainly beneficial for the performance, it remains unclear how many such reports are sufficient. To investigate that, we examine and compareMNALunder varying numbers of pseudo-labeled reports used at each timestep, i.e., the top푠most similar reports would be pseudo-labeled according ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:39 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score s = 1s = 2s = 3 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.80 0.81 0.82 0.83 0.84 0.85 0.86 0.87 F1-score (a) Query size 300 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score (b) Query size 500 246810 Timestep 0.84 0.85 0.86 0.87 0.88 F1-score (c) Query size 700 246810 Timestep 150 100 50 0 50 Readability (d) Query size 300 246810 Timestep 60 40 20 0 20 Readability (e) Query size 500 246810 Timestep 60 40 20 0 20 Readability (f ) Query size 700 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (g) Query size 300 246810 Timestep 0.1 0.2 0.3 0.4 0.5 Identifiability (h) Query size 500 246810 Timestep 0.1 0.2 0.3 0.4 Identifiability (i) Query size 700 Fig. 16. ComparingMNALunder different numbers of pseudo-labeled reports used at each timestep over all 10 timesteps (10 runs each). The plots show the mean and standard deviation. to each human-labeled report (by default the pseudo-labeling approach assumes푠=1). As such, the total number of pseudo-labeled reports per timestep would be 푠×푘 where 푘 is the query size. Figure 16 shows the results. As can be seen, we observe no statistical significance comparing the results obtained by different푠values on all metrics. This means that using more pseudo- labeled reports as part of the updating/training does not help the performance but only introduces unnecessary overhead. Therefore, we suggest considering only the most similar reports to their corresponding human-labeled ones in pseudo-labeling, i.e., 푠= 1. 6.4 Additional Design Justifications 6.4.1Cold Start Strategy. WhileMNALis designed to support both cold and warm start initializations as described in Section 3.3, our experimental evaluations have focused on the warm start strategy. This decision was based on several practical and methodological considerations. The warm start approach, which involves fine-tuning a pre-trained neural language model, represents the current state-of-the-art and most common practice for NLP tasks. Models like BERT and RoBERTa are pre-trained on vast general-language corpora, which provides a powerful ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:40Guoming Long, Shihai Wang, Hui Fang, and Tao Chen Table 11. Comparing the mean and standard deviation (SD) of the metrics values for different ratios of bug reports to non-bug reports. Ratio F1-scoreAccuracyAUCPrecisionReadability Identifiability 푟 Mean (SD) 푟 Mean (SD) 푟 Mean (SD) 푟 Mean (SD)푟 Mean (SD) 푟 Mean (SD) 1:110.874 (±0.0050)10.869 (±0.0050)10.926 (±0.0040) 3 0.881 (±0.0050)5 6.073 (±5.8000)10.189 (±0.0040) 1:42 0.870 (±0.0023) 2 0.851 (±0.0032) 3 0.918 (±0.0043) 4 0.804 (±0.0056)4 9.746 (±2.9677) 3 0.156 (±0.0052) 1:93 0.843 (±0.0053) 3 0.809 (±0.0085) 4 0.906 (±0.0114) 5 0.744 (±0.0107)3 9.705 (±3.5003) 5 0.147 (±0.0062) 4:14 0.782 (±0.0115) 4 0.802 (±0.0083) 2 0.912 (±0.0051) 2 0.928 (±0.0042)2 13.649 (±3.8468) 2 0.179 (±0.0035) 9:1 5 0.659 (±0.0240) 5 0.725 (±0.0139) 5 0.897 (±0.0121)10.947 (±0.0036)115.409 (±3.2403) 4 0.164 (±0.0050) foundation that is more resource-efficient and often more effective than training a model from the ground up (a cold start). Furthermore, a cold start would require training a neural language model from scratch, a process that is computationally intensive and time-consuming, especially with a dataset of over a million reports. As we will discuss in Section 6.6, even the fine-tuning process for one timestep can take nearly an hour on high-performance hardware; a full training cycle from scratch would be substantially more demanding and was considered beyond the scope of this study. Therefore, we focused on demonstrating the significant valueMNALadds to the more pragmatic and widely adopted warm start workflow. The core contributions—effort-aware uncertainty sampling and pseudo-labeling—are effectively validated by showing how they improve the standard fine-tuning process in a continuous, active learning setting. 6.4.2Imbalance Experiment. Our primary experiments were conducted using the NLBSE’23 dataset, which is relatively balanced with a 52.6% to 47.4% ratio of bug reports to non-bug reports in the training set. While this provides a controlled environment for evaluation, real-world software repositories often exhibit highly skewed distributions where one class significantly outnumbers the other. To assess the robustness and practical viability ofMNALunder such conditions, we conducted an additional experiment to evaluate its performance on imbalanced datasets. We simulated various imbalance scenarios by creating five different training sets from our original data, each with a distinct ratio of bug reports to non-bug reports: 1:1 (balanced), 1:4, 1:9, 4:1, and 9:1. All other experimental parameters, including the underlying neural language model and the active learning process, were kept consistent with our main evaluation. The results of this experiment are presented in Table 11. As anticipated, classification performance is influenced by the degree of class imbalance. The F1-score, a metric sensitive to the balance between precision and recall, shows a clear degradation as the data becomes more skewed. For instance, the F1-score dropped from 0.874 in the balanced 1:1 scenario to 0.843 for the 1:9 ratio and more significantly to 0.659 for the 9:1 ratio. This behavior is expected, as the model’s learning process is naturally biased by the majority class. When non-bug reports are dominant (1:9), the model’s precision decreases to 0.744, indicating a higher rate of false positives. Conversely, when bug reports form the vast majority (9:1), precision reaches a high of 0.947, but the low F1-score suggests a very poor recall, meaning the model struggles to correctly identify the few non-bug reports. Interestingly, the effort metrics also reveal a notable trend. The readability score is highest (least readable) in the 9:1 scenario (15.409), suggesting that when the model is faced with a highly skewed distribution, the reports it identifies as most uncertain may also be the most complex or poorly structured. Despite these challenges, the results confirm thatMNALremains effective even in imbalanced settings. The framework’s core mechanisms continue to guide the model’s ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:41 246810 Timestep 0.82 0.83 0.84 0.85 0.86 0.87 0.88 F1-score Ratio = 1Ratio = 2Ratio = 3 246810 Timestep 0.865 0.870 0.875 0.880 0.885 0.890 F1-score (a) F1 Score 246810 Timestep 0.09 0.10 0.11 0.12 0.13 0.14 0.15 0.16 0.17 Identifiability (b) Readability 246810 Timestep 25 20 15 10 5 0 5 10 Readability (c) Identifiability Fig. 17. Performance upper bound experiment result ofMNALwith metrics of F1 score, readablity and identifi- ability. learning process effectively, demonstrating its resilience and suitability for deployment in practical, real-world environments where balanced data cannot be guaranteed. 6.4.3 Performance Upper Bound. AlthoughMNALconsistently outperforms the baselines under practical query sizes, an exact performance upper bound remains unknown due to computational constraints. The full dataset contains over 1.27 million reports, making exhaustive fine-tuning prohibitively expensive. To approximate this bound, we conducted our largest feasible experiment. We continued this process until the performance gains became marginal with a step size of 3,000 reports, using a termination criterion of less than 0.1% F1-score change over consecutive timesteps. This experiment yielded a final F1-score of 0.891, at which point the performance curve began to plateau over 8 timesteps (for a total of 24,000 reports), as shown in Figure 17. While we cannot claim this as the true upper bound, the results imply thatMNALhas the potential to reach even higher performance with more extensive resources. We acknowledge this as a limitation and leave the precise measurement of the upper bound as future work. 6.4.4 Diversity Sampling Method. Our current effort-aware sampling method primarily balances model uncertainty with the readability and identifiability of reports for human labelers. A potential alternative is to incorporate sample diversity, a common strategy in active learning to ensure the queried samples are not redundant and cover a wider feature space. To investigate this, we conducted an experiment replacing our sampling method with a diversity-based approach. The results at the final timestep showed a minor 1.14% increase in F1-score. However, this was accompanied by a significant decrease in readability (365.48% reduction) and identifiability (52.14% reduction). This suggests that the most diverse samples, while potentially informative for the model, are much more difficult for human developers to label. Since a core contribution ofMNALis reducing the human effort in the labeling process, we conclude that our effort-aware approach provides a better balance for effective human-machine collaboration in this context. 6.4.5Model Selection for Model-Agnostic Experiment and Justification. Our model selection process for Section 5.3 was guided by the need to evaluateMNALunder varied conditions, and we chose models that represent different specializations: •A General-Purpose Text Model (RoBERTa): We included RoBERTa as it is a "robust and widely- used extension of BERT" known for its SOTA performance on a wide range of natural language understanding benchmarks. This testsMNAL’s generalizability with a highly optimized, all- purpose model. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:42Guoming Long, Shihai Wang, Hui Fang, and Tao Chen •A Code-Specific Model (CodeBERT): Given that bug reports on GitHub frequently contain code snippets, we chose CodeBERT, which is "a specifically designed model for code natural- ness". Its ability to handle both programming and natural languages makes it highly relevant for our domain. •A Task-Specific Model (RTA): To testMNALin a highly specialized context, we used RTA, a model “tailored for analyzing bug reports via learning a universal representation”. While we initially performed preliminary experiments using T5, we finally excluded those results from the manuscript because the model consistently failed to generate meaningful outcomes for our specific task. Our investigation revealed several reasons for T5’s poor performance: Architectural Mismatch for Classification: T5 is an encoder-decoder model designed pri- marily for generative, text-to-text tasks (e.g., summarization, translation). Our task, bug report identification, is a binary classification problem. While T5 can be adapted for classification by training it to generate specific labels (e.g., "bug" or "not bug"), this approach is often less efficient and stable than using encoder-only models like BERT. BERT-style architectures are purpose-built for discriminative tasks, using the token representation and a simple classification head, which is a more direct and suitable architecture for this problem. Training Instability and Failure to Converge: During fine-tuning, the T5 model exhibited significant instability. Despite extensive hyperparameter tuning (adjusting learning rates, batch sizes, optimizers, and schedulers), the training loss failed to converge, and the model’s accuracy on the validation set remained close to random chance. This suggests that the model architecture and its pre-training objectives are not easily adaptable to learning the specific discriminative features required to distinguish bug from non-bug reports in our dataset. We believe this architectural mismatch is the primary reason for T5’s poor performance. Encoder- only models are inherently better suited for this classification task, and our results with BERT, RoBERTa, CodeBERT, and RTA confirm this. 6.5 Validity of Readability Metrics with Non-Natural Language Tokens A potential concern raised regarding our effort-aware sampling is whether the FRE score remains a valid proxy for labeling effort given that GitHub Issues frequently contain non-natural language elements, such as code snippets, stack traces, log entries, and file paths. These elements can significantly skew traditional linguistic statistics (e.g., word length and sentence count), sometimes resulting in extremely low or even negative readability scores. Processing of Non-Natural Language. In our implementation, we do not aggressively filter out these symbols. Instead, we treat them as part of the overall textual information that a software developer must process. While tokens like version numbers or complex file paths may not follow standard grammatical structures, they contribute to the visual and cognitive complexity of the report. Validity in the Software Engineering Context. We argue that the FRE score remains an effective indicator of labeling effort in this domain for the following reasons: •Correlation with cognitive load: Non-natural language elements, particularly dense stack traces or raw logs, inherently increase the time and mental effort required for a developer to comprehend the core issue. A lower (or negative) FRE score accurately reflects this increased "reading cost." Thus, the metric’s sensitivity to these tokens is a feature, not a bug, in the context of measuring labeling effort. •Empirical consistency: In our dataset, bug reports with extensive technical jargon and code fragments consistently received lower FRE scores compared to descriptive, text-heavy reports. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:43 This aligns with the practical experience of developers who find "noisy" issues more taxing to triage. Potential Restrictions. However, we acknowledge certain potential restrictions. The FRE metric might be less reliable in extreme cases where a report consists exclusively of a massive log dump without any descriptive context. In such scenarios, the metric might over-penalize the report’s complexity. While our results demonstrate a significant reduction in overall labeling effort, future work could explore hybrid metrics that specifically weight code-to-text ratios to further refine the estimation of human labeling costs in highly technical repositories. 6.6 Scalability, Overheads, and Resources 6.6.1Scalability. Apart from the model training, scalability can be related to the retrieval, especially the pseudo labeling process. Our current implementation utilizes an exact nearest neighbor search via optimizedpairwise_distancescomputations. While the theoretical complexity is푂(|풟 푞 |· |풟 푢 |), the use of matrix-based operations on GPUs significantly accelerates this process. For scenarios with millions of reports, our framework can be seamlessly integrated with Approximate Nearest Neighbor (ANN) libraries such as FAISS or HNSW to achieve sub-linear retrieval time. Another factor that affects scalability is the updating frequency, for whichMNALrelies on the number of reports needed for the update (i.e., up to 700). This lies in the trade-off between the timeliness of the model and efficiency, where we assume that the frequency can be case dependent. 6.6.2 Overhead of Updating/Training Time. revisionMNALsupports warm starts by using a pre- trained neural language model, thus the key training overhead thereof would be the updating process that fine-tunes the model. On a machine of 2.25GHz CPU accelerated by a typical CUDA- enabled A100 GPU and under over one million unlabeled reports from different projects,MNALneeds ≈53 minutes to complete one cycle of the update within each timestep (with 700 query size). Further analysis of the breakdown reveals that the actual training/fine-tuning of the neural language model only takes≈2 minutes because only 700 new reports together with any previously labeled reports are updated/trained. The main proportion of the overhead comes from the retrieval procedure: there are≈25 minutes for the effort-aware uncertainty sampling and≈26 minutes for pseudo labeling. We would like to stress that the above is mainly caused by the fact that there are over one million unlabeled reports to be processed therein; most of the time required is for iterating over that huge pool of reports. Certainly, the overhead is expected to be reduced with fewer accumulated unlabeled reports. The inference time for automatically identifying a new report, on the other hand, is negligible at the magnitude of milliseconds. Since the interval between different timesteps is flexible, i.e., there could be only one update every a few days or even weeks, an update of less than 1 hour is often considered acceptable as MNALcan be left to run the update overnight, once the queried reports have been labeled, which is not uncommon for updating some large software systems in practice. In particular, considering the dramatic improvement in accuracy and the reduction of human labeling efforts against the state-of-the-art approaches, the benefits of MNAL outweigh its updating/training overhead. 6.6.3Resources. With the data scale over one million (labeled and unlabeled) samples, the memory footprint of the process is stable, requiring approximately 3–5 GB of VRAM for the BERT model and 2–8 GB of system RAM for storing the distance matrix, depending on the specific sample size. Since the neural embedding space evolves as the neural language model inMNALis fine-tuned, we re-calculate the embeddings in each iteration to maintain the accuracy of the pseudo-labels. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:44Guoming Long, Shihai Wang, Hui Fang, and Tao Chen 6.7 Practical Application of MNAL 6.7.1 UsingMNAL.MNALworks cross-project, meaning that the training, querying unlabeled re- ports to the software developers, pseudo-labeling reports, and prediction therein can all occur for different software projects. This providesMNALwith the ability to serve as a centralized bug report identification system for any organization. For example, companies can adoptMNALto handle the repositories of all their projects. In particular, depending on the quality-effort score, the top푘 unlabeled reports that would be queried might not be constrained to one project since the pool of unlabeled reports contains reports from all the projects of a company. As such, it is possible forMNALto ask developers who are specialized in different projects for the labeling, depending on which projects the queried reports belong. Of course, when there is only one project,MNALnaturally reduces to a typical within-project setting. The warm start property inMNALcan also largely improve its applicability, thanks to the pre- training nature of many neural language models. That is, even when applyingMNALfor the very first time, it is possible to build it from a pre-trained version elsewhere, hence saving a dramatic amount of resources and time. 6.7.2Human Labeling Effort. Indeed, in a case where there is plenty of labeled data, human labeling efforts can indeed be costly. However, in our context, we target the scenarios in which all the labeled reports have been used for training, and we are hitting the point where no more labels can be used (or the labeled data is very limited). For example, in a recent large-scale study over GitHub [87], it has been found that 54.4% of the issues studied have no label at all. In such a condition, there is no other way to further improve the model but to use some more labeled data, which should ideally provided by humans. To that end, active learning is a paradigm that is well-suited to a scenario that requires human-machine interaction. We agree that the human labeling process is expensive, which is why in MNAL, we specifically design two strategies to relieve that: • In the Effort-aware Uncertainty Sampling, we consider readability and identifiability, alongside uncertainty. Those two objectives aim primarily to reduce the labeling effort required by humans, as the selected reports to be labeled are often more information-intuitive. • In the Pseudo-labeling, we create pseudo labels, guided by a limited set of human-labeled data, which greatly reduce the number of labels required from humans. The above designs have significantly reduced the human efforts needed in the active learning process while keeping promising effectiveness, as shown by the experiment results. 6.7.3Applicable Scenarios ofMNAL. Since the neural language model is the base model inMNAL, it naturally works for both single and cross-project scenarios. However, it is often more beneficial for an approach to work on cross-project cases because this allows for more diverse training data, and advanced language models can handle such complex data. Therefore, we highlight this as a key feature ofMNAL; however, this does not mean that it can only work on cross-project data. Our model can also be directly applied to single-project cases, especially those with a particularly large number of bug reports, which allows our approach to have sufficient data for training.MNALcan also serve as a complementary approach with tools from other software engineering tasks, e.g., the report can be used to guide testing [19,59]; to identify key/problematic options in configuration modeling [31–33,96]; to pinpoint useful information for configuration tuning [6,7,10,97] and runtime self-adaptation [9, 100]; and to assist explainability [8, 101]. 6.8 Limitations While theMNALframework presents numerous advantages, it also has certain limitations and potential drawbacks that may arise: ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:45 •Model Training Dependency on GitHub Issues: The model developed through our experiments is specifically trained on GitHub issue data. IfMNALis to be applied to other bug report repositories, it requires retraining with data from those specific repositories. This dependency on retraining can limit the immediate applicability of the model across different platforms or environments. •Training Data Size and Time Cost: When retrainingMNALwith large datasets, there may be significant time and computational costs associated with the process. Handling large volumes of training data can lead to extended training times, potentially impacting the efficiency and scalability of the framework. In contrast, baseline methods such as those based on TF-IDF are less computationally expensive, as they do not require the same level of deep learning infrastructure, making them faster to deploy but at the cost of reduced accuracy in identifying complex patterns. However, they may not provide as good performance asMNALdue to their inability to capture nuanced semantic relationships in the text. •Lack of Transparency in Neural Language Models: Neural language models are often described as “black boxes,” and the predictions made byMNALmight lack transparency. This characteristic can make it challenging to understand the rationale behind the model’s decisions and may affect the trustworthiness of the results. Other more interpretable models, such as decision trees, are free of this issue. Again, baseline methods such as those based on TF-IDF can provide more transparency, as they rely on straightforward statistical techniques that are easier to interpret. These methods make it clear how certain terms influence the output, which adds a level of explainability that deep learning models may lack. However, this transparency often comes at the cost of performance. This is essentially a trade-off between transparency and performance, and we chose better performance over TF-IDF-based methods, which are more transparent but offer weaker performance. • Integration Challenges: Organizations with established systems may encounter difficulties when integratingMNALinto their existing processes. The complexity of adapting and incorpo- rating our framework into pre-existing workflows could pose practical challenges and hinder its adoption in certain settings. 7 THREATS TO VALIDITY We now discuss the potential threats in this work with respect to different aspects of validity. 7.1 Internal Validity A potential threat to validity is the choice of푘, the number of samples queried per round. We selected푘=300,푘=500, and푘=700 as a pragmatic balance between effectiveness and cost, but these values may not be optimal. This choice was not rigorously optimized, and different푘 values could yield varying results.MNALand its underlying neural language model (e.g., BERT) contains several parameters that require pre-defining their values, which may raise threats to internal validity. To that end, we tune that to fit with our experiment infrastructure, e.g., 18 epochs, 32 batch size, learning rate of 3×10 −5 and Epsilon factor of 10 −8 . Indeed, the use of BERT is only a pragmatic choice and can lead to bias. To mitigate that, in RQ3, we examine additional three neural language models withMNALfor confirming its model-agnostic property. Yet, hidden interactions between those settings that we have not discovered are always possible. The definition of identifiability is straight-forward. It does not mean to capture the semantics but rather, it serves as domain-specific keyword lists that help to filter those reports that are highly readable but still contain less information about bug relevance. Besides, it has the benefit of lightweight computation while fully capturing the semantics would certainly be more expensive. However, such a simple ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:46Guoming Long, Shihai Wang, Hui Fang, and Tao Chen metric works really well in our large-scale dataset and we leave the capture of richer semantics as part of the future work, which might further improve the effectiveness of MNAL. Not knowing the upper bound of this method could be another internal threat. Feeding the models with data from the dataset is the most ideal scenario. However, practically, we do not have the necessary resources to do that. This is because conducting a 10 repeated time on a standard BERT after 10 timesteps with a total of 7,000 samples has already consumed 9 hours of training. Considering the fact that the dataset has 1,275,881 reports, the time/resource required in fine-tuning increases dramatically. Not to mention the complexity of running the entire experiment, which involves varying query sizes and different modification settings. 7.2 Construct Validity The metrics in the experiment can introduce bias on construct validity. Their interpretation can vary based on the context and the specific nuances of the dataset. It is also worth noting that there’s the potential risk of overemphasizing certain metrics at the expense of others, leading to a biased understanding of the true performance. To mitigate this, we use several most common performance metrics together with those that measure the efforts of labeling. We repeat the experiment 10 runs and use statistical tests to ensure the reliability of the conclusion. For our qualitative study, the participants involved might not truly represent real developers, and their judgments may be subjective, but this constraint is not uncommon for studies with humans and fits with real-world scenarios well. 7.3 External Validity Threats to external validity can be sourced from different aspects. For example, the choice of the NLBSE dataset and the model could influence the generalizability of our findings. Yet, this dataset, to the best of our knowledge, is the most comprehensive one to date while of large volume (>1 million reports), covers cross-projects, and contains diverse report formats. To fully evaluateMNAL, we have assessed it over three neural language models and against four other state-of-the-art approaches. Yet, admittedly, adding more subjects may prove fruitful. The qualitative human study involved 10 participants, which may limit the external validity. Though this is not a significantly large number, it is a reasonable sample size to start with, provided that appropriate statistical tests are used, as suggested in the guideline of human-involved experiments in software engineering [47]. 8 RELATED WORK We now discuss the related work in light of the contributions from MNAL. 8.1 Supervised Learning to Identify Bug Reports Supervising learning that assumes a model can be built with a sufficient amount of data beforehand has been a major research direction for bug report identification in the past decade. Among others [14,21,22,36,48,49,53,72,75,95,106,108], majority of the work did not distinguish the type of bug that a report is associated with, i.e., as long as they are bug related then the model should put classify them into the same category.MNALalso falls into the same type of approaches that do not distinguish the bug type. In contrast, others focus on identifying reports relevant to a particular bug type, e.g., high-impact bugs [98,99,108], security bugs [16,34,42,48], or performance bugs [4], etc. Yet, regardless of whether a specific bug type is considered, the proposed approach for bug report identification can be equally applicable with no or minor amendments. In what follows, we discuss them in terms of the model proposed. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:47 8.1.1Statistical Machine Learning Model. Traditionally, report identification with machine learning has been leveraging simple but interpretable models that are paired with certain embedding methods, which most commonly include term-frequency and term frequency-inverse document frequency. Among others, Yang et al.[99] and Kukkar and Mohana [48] use푘N to aid developers with extracting information from the reports for debugging. Ding et al.[18] use Naive Bayes (NB), and Yang et al.[98] and Zhou et al.[108] use Naive Bayes Multinomial (NBM), respectively, to classify bug related reports. The probabilistic nature of those models offers insights into the likelihood of a report is bug related or not. More complex models exist for bug report identification. For example, Polpinij [72] and Fan et al.[22] use Support Vector Machines (SVM) while Guo et al.[34], Brown et al.[4] and Xia et al.[95] adopt Multilayer Perceptron (MLP). SVM and MLP demonstrate versatility in capturing intricate patterns and decision boundaries present in bug reports. Furthermore, by pairing with encoding techniques such as Doc2Vec [50], the generalization abilities of SVM and MLP make them valuable for bug identification for bug reports with various formats and characteristics. An alternative learning paradigm is ensemble learning, such as random forest. In the realm of bug report identification, those ensemble models emerge as a versatile and powerful approach, synergizing individual models to enhance overall performance and reliability as demonstrated by various research studies in this domain [53, 79, 108]. However, due to the restricted ability of statistical machine learning models to understand and represent the nature of texts, the above work is often ineffective in extracting and parsing the information of a submitted report. 8.1.2 Neural Language Model. To overcome the limitation of the statistical machine learning model, existing work has started to explore the power of neural language models for bug report identification. For example, Zheng et al.[106] incorporate attention-based Bi-directional Long Short-Term Memory (BiLSTM)—a typical language model that handles long term dependencies with improved computational efficiency—to automatically extract text features from vulnerability descriptions the submitted reports. By pairing withFastText[43] for word embedding, BiLSTM is designed with a tailored loss function that fits the characteristics of bug reports. Attention weights are also extracted to highlight significant text features aiding in the identification of tactical vulnerabilities. Similarly, Convolutional Neural Network (CNN), which is commonly used for image processing, has also been adopted for identifying bug reports [14,21,36,106], in which the word embedding is represented as a matrix of image pixels. The structure of convolutional layers helps capture local patterns and features while pooling layers behind allows reduced dimension. This approach is effective for tasks like text classification or sentiment analysis, where local relationships between words are crucial [13]. Additionally, pre-trained word embedding or convolutional layers can be used to leverage knowledge from large text corpora. In particular, Cho et al.[14] exploit not only the information from the reports but also the key messages from the user manual for identifying whether the report is bug related. A unique characteristic of applying those CNN models, when used for bug report identification, is that they often work better with word embedding techniques such as FastText[43] orGloVe[70]. Interestingly, Zheng et al.[106] propose a multi-modal model-based that combines image and text contained in a submitted report, which tends to improve the overall prediction performance. Comparing to CNN and LSTM, BERT excels in capturing contextualized word representations through bidirectional context analysis. Its self-attention mechanism allows for better handling of long-range dependencies, and pre-training on large corpora enables efficient transfer of knowledge from general text classification task to bug report identification domain. The utilization of BERT as a model has been extensively explored in existing literature. For example, Meher et al.[62] ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:48Guoming Long, Shihai Wang, Hui Fang, and Tao Chen employ a heuristic annotation approach to annotate a substantial dataset of software bug reports. Subsequently, they leverage BERT for the classification of these carefully curated bug reports. The utilization of BERT proves advantageous, capitalizing on the contextual features embedded in the bug reports, facilitated by the attention mechanism. Similarly, Nadeem et al.[65] introduce a methodology for classifying bug reports in a multi-label setting. The authors employ an off- the-shelf neural network named RoBERTa, fine-tuning it for the bug report classification task. As an optimized variant of BERT, RoBERTa utilizes an enhanced training methodology and larger datasets, resulting in superior performance compared to BERT. Their preference for RoBERTa over BERT is driven by its state-of-the-art results on benchmark datasets. In a distinctive approach, Gomes et al.[30] present a methodology that utilizes BERT as an embedding method. They extract features using BERT and subsequently input the data into statisti- cal machine learning models. Their findings reveal that the long-lived bug report identification achieved through BERT-based feature extraction consistently outperformed TF-IDF. Specifically, SVM and Random Forest classifiers demonstrated superior performance across nearly all datasets when utilizing BERT as the feature extraction method. Furthermore, Shi et al.[78] identify the bug report from separated dialogs by utilizing a graph- based context embedding approach to create a dialog graph. Furthermore, they employ a two-layer graph neural network to assimilate contextual information. This approach proves effective in capturing the graphical context of utterances within a single dialog, comprehending structure-level context for each vertex in a specified graph relationship, and encapsulating high-level contextual information. Despite the adoption of various neural language models, existing work has not adopted powerful models that are compatible with pre-training capacity (e.g., BERT), which renders cross-project identification plausible. This is one of the key components inMNAL. Further, the above often assumes a sufficient amount of labeled reports is readily available and one is willing to accept a significant amount of training overhead (even for fine-tuning), which might not be applicable to all real-world scenarios.MNAL, on the other hand, leverage neural active learning for bug report identification via boosted human-machine teaming. 8.2 Active Learning for Bug Report Identification To mitigate the prerequisite on the amount of labeled data and to relax the requirements of training on all data samples, the paradigm of active learning has been a promising direction for bug report identification [29,82,84,86,93]. Wang et al.[86] and Ge et al.[29] adopt active learning for identifying bug related reports where the queried samples are determined by using local neighborhoods of previously useful reports for improving model performance. Similarly, Wu et al.[93] proposehbrPredictor, which also relies on active learning to identify high-impact bug related reports but they additionally exploits stronger human interaction by combining interactive machine learning with active learning. There are also approaches that work with extended active learning.EMBLEM[84] is an active learning approach for bug report identification, which relies on the active learning strategy that is a mix of certainty and uncertainty in the selection of reports to query. There also exists a combination of semi-supervised learning and active learning with an aim to reduce human efforts [82]. Other similar work includes HINT [28], which focuses on hybrid pseudo-labeled data selection and noise-tolerant training to improve pre-trained code models by mitigating noise in pseudo- labeled data. While both MNAL and HINT leverage pseudo-labeling to enhance model performance , our direct experimental comparison reveals that MNAL achieves a better F1-score, accuracy, and recall. More fundamentally, the approaches differ in their core objectives. HINT’s primary contribution is handling noisy pseudo-labels. In contrast, MNAL establishes a human-machine ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:49 collaborative loop, centering on active learning and using readability and identifiability metrics to reduce the effort for human labelers, a crucial aspect not addressed by HINT. The above work mainly targets within-project while they did not adopt a neural language model, In contrast,MNALworks for cross-project, and the selection of queried reports directly optimizes for better readability and identifiability. In addition, they did not directly select reports that are naturally more readable and identifiable which is one of the key contributions inMNAL. We also enrich the samples to update the neural language model by considering pseudo-labeled reports. Such a mutualistic relation, underpinned by neural active learning, has enabledMNALto achieve more promising results. 8.3 Empirical Studies for Bug Report Identification To better design models for bug report identification, empirical studies have been conducted to understand the characteristics of the identification problem. Among others, Pandey et al.[68] examine six statistical machine learning models in identifying bug related reports for three open- source projects. Their experiments reveal that random forests and SVM with specific kernels tend to be superior to the others. Some other studies focus on techniques that deal with data unbalancing. For example, Zheng et al.[105] study six imbalance mitigation methods and found that the performance of the bug identification can be notably enhanced in over 78% of cases by employing class rebalancing methods such as Rose [58] and Mahakil [3] that is paired with CNN. Other work that concentrates on understanding how mislabeling can affect the model performance in identifying bug reports also exist [1,94]. They discovery that the prediction model exhibits significantly better performance on clean datasets compared to noisy datasets and employing advanced neural language models like RoBERTa is recommended for bug report identification. We considerMNALa general and flexible approach, which can be complementary to some of the above findings. For example, while we do not consider mislabeled reports due to human errors, some of the findings from Afric et al.[1] and Wu et al.[94] can serve as the foundation to take that into account, hence achieving labeling with better quality as part of the neural active learning. 9 CONCLUSION In this paper, we introduced a novel cross-project frameworkMNALfor automated and more effective identification of bug reports from GitHub repositories co-boosted by human-machine collaboration. Going beyond traditional active learning,MNALintroduces a mutualistic relation between humans (software developers) and machines (neural language model) for enhanced human-machine teaming when updating/training the model. This is achieved through effort-aware uncertainty sampling, which queries unlabeled reports requiring an acceptable effort level for labeling. Additionally, we proposed a pseudo-labeling approach to enrich the data for updating/training the neural language model that represents the reports in a latent space. To investigate the effectiveness of our approach, we experiment on large-scale bug report datasets containing a total of 1,275,881 reports collected from different software projects (over 127,000). The experimental results reveal that (1)The effort-aware uncertainty sampling inMNALdemonstrates competitive evolving perfor- mance while significantly reducing the effort required for manual labeling of queried reports by developers. This results in a substantial 98.1% and 194.7% reliability and identifiability improvement, respectively, both of which demonstrate statistical significance overall. (2)Pseudo-labeling inMNALcontributes to a substantial improvement in the performance of the neural language model, with minimal impact on readability and identifiability. (3) MNALis prone to be model-agnostic, showing considerable performance improvement re- gardless of the underlying neural language model employed. This improvement comes with ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:50Guoming Long, Shihai Wang, Hui Fang, and Tao Chen significantly reduced efforts required for labeling. Specifically, the superiority lies in read- ability and identifiability, with improvements of 78.6% and 171.5%, respectively, supported by significant statistical evidence. (4) MNALsignificantly outperforms other state-of-the-art approaches on various performance metrics and with up to 95.8% and 196.0% improvement for readability and identifiability, respectively. (5) Through a case study with human involvement, we show that, in comparison with randomly selected reports for human labeling,MNALachieves an effort reduction within the same time frame, leading to 3×monetary savings, while also showing 74.7% and 64.8% improvement in qualitative readability and identifiability, respectively. This work is merely one step towards an enhanced human-machine collaborative approach for software data analytics, in which the convenience and benefit of human involvement are explicitly taken into account while the model is also greatly improved. Along this thread of research, the future opportunities of intervention are vast, including better handling of mislabeled reports due to human labeling error and improved efficiency of utilizing data, e.g., by means of combining few-short learning and neural active learning. DATA AVAILABILITY STATEMENT The dataset and replication package supporting this study are publicly available at Zenodo: https: //zenodo.org/records/19193244. ACKNOWLEDGMENT This work was supported by a NSFC Grant (62372084) and a UKRI Grant (10054084). REFERENCES [1] Petar Afric, Davor Vukadin, Marin Silic, and Goran Delac. 2023. Empirical Study: How Issue Classification Influences Software Defect Prediction. IEEE Access 11 (2023), 11732–11748. https://doi.org/10.1109/ACCESS.2023.3242045 [2] Andrea Arcuri and Lionel C. Briand. 2011. A practical guide for using statistical tests to assess randomized algorithms in software engineering. In Proceedings of the 33rd International Conference on Software Engineering, ICSE 2011, Waikiki, Honolulu , HI, USA, May 21-28, 2011, Richard N. Taylor, Harald C. Gall, and Nenad Medvidovic (Eds.). ACM, 1–10. https://doi.org/10.1145/1985793.1985795 [3]Kwabena Ebo Bennin, Jacky Keung, Passakorn Phannachitta, Akito Monden, and Solomon Mensah. 2017. Mahakil: Diversity based oversampling approach to alleviate the class imbalance issue in software defect prediction. IEEE Transactions on Software Engineering 44, 6 (2017), 534–550. [4] Selasie Aformaley Brown, Benjamin Asubam Weyori, Adebayo Felix Adekoya, Patrick Kwaku Kudjo, Solomon Mensah, and Samuel Abedu. 2021. DeepLaBB: a deep learning framework for blocking bugs. In 2021 International Conference on Cyber Security and Internet of Things (ICSIoT). IEEE, 22–25. [5]Davide Cacciarelli and Murat Kulahci. 2023. Active learning for data streams: a survey. Machine Learning (2023), 1–55. [6]Pengzhou Chen and Tao Chen. 2026. PromiseTune: Unveiling Causally Promising and Explainable Configuration Tuning. In 48th IEEE/ACM International Conference on Software Engineering (ICSE). ACM. [7] Pengzhou Chen, Tao Chen, and Miqing Li. 2024. MMO: Meta Multi-Objectivization for Software Configuration Tuning. IEEE Trans. Software Eng. 50, 6 (2024), 1478–1504. https://doi.org/10.1109/TSE.2024.3388910 [8]Pengzhou Chen, Jingzhi Gong, and Tao Chen. 2025. Accuracy Can Lie: On the Impact of Surrogate Model in Configuration Tuning. IEEE Trans. Software Eng. 51, 2 (2025), 548–580. https://doi.org/10.1109/TSE.2025.3525955 [9]Tao Chen. 2022. Lifelong Dynamic Optimization for Self-Adaptive Systems: Fact or Fiction?. In IEEE International Conference on Software Analysis, Evolution and Reengineering, SANER 2022, Honolulu, HI, USA, March 15-18, 2022. IEEE, 78–89. https://doi.org/10.1109/SANER53432.2022.00022 [10]Tao Chen and Miqing Li. 2021. Multi-objectivizing software configuration tuning. In ESEC/FSE ’21: 29th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, Athens, Greece, August 23-28, 2021, Diomidis Spinellis, Georgios Gousios, Marsha Chechik, and Massimiliano Di Penta (Eds.). ACM, 453–465. https://doi.org/10.1145/3468264.3468555 ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:51 [11]Tao Chen and Miqing Li. 2023. Do Performance Aspirations Matter for Guiding Software Configuration Tuning? An Empirical Investigation under Dual Performance Objectives. ACM Trans. Softw. Eng. Methodol. 32, 3 (2023), 68:1–68:41. https://doi.org/10.1145/3571853 [12]Tao Chen and Miqing Li. 2023. The Weights Can Be Harmful: Pareto Search versus Weighted Search in Multi- objective Search-based Software Engineering. ACM Trans. Softw. Eng. Methodol. 32, 1 (2023), 5:1–5:40.https: //doi.org/10.1145/3514233 [13] Jian Cheng, Peisong Wang, Gang Li, Qinghao Hu, and Hanqing Lu. 2018. Recent advances in efficient computation of deep convolutional neural networks. Frontiers Inf. Technol. Electron. Eng. 19, 1 (2018), 64–77. https://doi.org/10.1631/ FITEE.1700789 [14]Heetae Cho, Seonah Lee, and Sungwon Kang. 2022. Classifying issue reports according to feature descriptions in a user manual based on a deep learning model. Inf. Softw. Technol. 142 (2022), 106743. https://doi.org/10.1016/J.INFSOF. 2021.106743 [15] Gabriele Ciravegna, Frédéric Precioso, Alessandro Betti, Kevin Mottin, and Marco Gori. 2023. Knowledge-Driven Active Learning. In Machine Learning and Knowledge Discovery in Databases: Research Track - European Conference, ECML PKDD 2023, Turin, Italy, September 18-22, 2023, Proceedings, Part I (Lecture Notes in Computer Science, Vol. 14169), Danai Koutra, Claudia Plant, Manuel Gomez-Rodriguez, Elena Baralis, and Francesco Bonchi (Eds.). Springer, 38–54. https://doi.org/10.1007/978-3-031-43412-9_3 [16] Dipok Chandra Das and Md Rayhanur Rahman. 2018. Security and performance bug reports identification with class-imbalance sampling and feature selection. In 2018 Joint 7th International Conference on Informatics, Electronics & Vision (ICIEV) and 2018 2nd International Conference on Imaging, Vision & Pattern Recognition (icIVPR). IEEE, 316–321. [17]Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), Jill Burstein, Christy Doran, and Thamar Solorio (Eds.). Association for Computational Linguistics, 4171–4186. https://doi.org/10.18653/V1/N19-1423 [18]Jianshu Ding, Guisheng Fan, Huiqun Yu, and Zijie Huang. 2021. Automatic Identification of High Impact Bug Report by Test Smells of Textual Similar Bug Reports. In 21st IEEE International Conference on Software Quality, Reliability and Security, QRS 2021, Hainan, China, December 6-10, 2021. IEEE, 446–457. https://doi.org/10.1109/QRS54544.2021.00056 [19]Chengwen Du and Tao Chen. 2025. Causally Perturbed Fairness Testing. ACM Trans. Softw. Eng. Methodol. (Oct. 2025). https://doi.org/10.1145/3773088 Just Accepted. [20]Junwei Du, Xinshuang Ren, Haojie Li, Feng Jiang, and Xu Yu. 2023. Prediction of bug-fixing time based on distinguish- able sequences fusion in open source software. J. Softw. Evol. Process. 35, 11 (2023). https://doi.org/10.1002/SMR.2443 [21]Xiaoting Du, Zheng Zheng, Guanping Xiao, Zenghui Zhou, and Kishor S. Trivedi. 2022. DeepSIM: Deep Semantic Information-Based Automatic Mandelbug Classification. IEEE Trans. Reliab. 71, 4 (2022), 1540–1554. https://doi.org/ 10.1109/TR.2021.3110096 [22]Yuanrui Fan, Xin Xia, David Lo, and Ahmed E. Hassan. 2020. Chaff from the Wheat: Characterizing and Determining Valid Bug Reports. IEEE Trans. Software Eng. 46, 5 (2020), 495–525. https://doi.org/10.1109/TSE.2018.2864217 [23] Fan Fang, John Wu, Yanyan Li, Xin Ye, Wajdi Aljedaani, and Mohamed Wiem Mkaouer. 2021. On the classification of bug reports to improve bug localization. Soft Comput. 25, 11 (2021), 7307–7323. https://doi.org/10.1007/s00500-021- 05689-2 [24] Sen Fang, Tao Zhang, Youshuai Tan, He Jiang, Xin Xia, and Xiaobing Sun. 2023. RepresentThemAll: A Universal Learning Representation of Bug Reports. In 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023. IEEE, 602–614. https://doi.org/10.1109/ICSE48619.2023.00060 [25]James N Farr, James J Jenkins, and Donald G Paterson. 1951. Simplification of Flesch reading ease formula. Journal of applied psychology 35, 5 (1951), 333. [26]Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020 (Findings of ACL, Vol. EMNLP 2020), Trevor Cohn, Yulan He, and Yang Liu (Eds.). Association for Computational Linguistics, 1536–1547. https://doi.org/10.18653/V1/2020.FINDINGS-EMNLP.139 [27] Yifan Fu, Xingquan Zhu, and Bin Li. 2013. A survey on instance selection for active learning. Knowl. Inf. Syst. 35, 2 (2013), 249–283. https://doi.org/10.1007/s10115-012-0507-8 [28]Shuzheng Gao, Wenxin Mao, Cuiyun Gao, Li Li, Xing Hu, Xin Xia, and Michael R. Lyu. 2024. Learning in the Wild: Towards Leveraging Unlabeled Data for Effectively Tuning Pre-trained Code Models. In Proceedings of the 46th IEEE/ACM International Conference on Software Engineering, ICSE 2024, Lisbon, Portugal, April 14-20, 2024. ACM, 80:1–80:13. https://doi.org/10.1145/3597503.3639216 ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:52Guoming Long, Shihai Wang, Hui Fang, and Tao Chen [29]Xiuting Ge, Chunrong Fang, Meiyuan Qian, Yu Ge, and Mingshuang Qing. 2022. Locality-based security bug report identification via active learning. Inf. Softw. Technol. 147 (2022), 106899. https://doi.org/10.1016/j.infsof.2022.106899 [30]Luiz Alberto Ferreira Gomes, Ricardo da Silva Torres, and Mario Lúcio Côrtes. 2023. BERT- and TF-IDF-based feature extraction for long-lived bug prediction in FLOSS: A comparative study. Inf. Softw. Technol. 160 (2023), 107217. https://doi.org/10.1016/J.INFSOF.2023.107217 [31]Jingzhi Gong and Tao Chen. 2023. Predicting Software Performance with Divide-and-Learn. In Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ESEC/FSE 2023, San Francisco, CA, USA, December 3-9, 2023, Satish Chandra, Kelly Blincoe, and Paolo Tonella (Eds.). ACM, 858–870. https://doi.org/10.1145/3611643.3616334 [32]Jingzhi Gong and Tao Chen. 2024. Predicting Configuration Performance in Multiple Environments with Sequential Meta-Learning. Proc. ACM Softw. Eng. 1, FSE (2024), 359–382. https://doi.org/10.1145/3643743 [33] Jingzhi Gong, Tao Chen, and Rami Bahsoon. 2025. Dividable Configuration Performance Learning. IEEE Trans. Software Eng. 51, 1 (2025), 106–134. https://doi.org/10.1109/TSE.2024.3491945 [34]Shikai Guo, Rong Chen, Hui Li, Tianlun Zhang, and Yaqing Liu. 2019. Identify Severity Bug Report with Distribution Imbalance by CR-SMOTE and ELM. Int. J. Softw. Eng. Knowl. Eng. 29, 2 (2019), 139–175. https://doi.org/10.1142/ S0218194019500074 [35] Som Gupta and Sanjai Gupta. 2021. Bug Reports and Deep Learning Models. International Journal of Computer Science and Mobile Computing 10 (12 2021), 21–26. https://doi.org/10.47760/ijcsmc.2021.v10i12.003 [36]Jianjun He, Ling Xu, Yuanrui Fan, Zhou Xu, Meng Yan, and Yan Lei. 2020. Deep Learning Based Valid Bug Reports Determination and Explanation. In 31st IEEE International Symposium on Software Reliability Engineering, ISSRE 2020, Coimbra, Portugal, October 12-15, 2020, Marco Vieira, Henrique Madeira, Nuno Antunes, and Zheng Zheng (Eds.). IEEE, 184–194. https://doi.org/10.1109/ISSRE5003.2020.00026 [37]Steffen Herbold, Alexander Trautsch, and Fabian Trautsch. 2020. On the feasibility of automated prediction of bug and non-bug issues. Empir. Softw. Eng. 25, 6 (2020), 5333–5369. https://doi.org/10.1007/S10664-020-09885-W [38]Kim Herzig, Sascha Just, and Andreas Zeller. 2013. It’s not a bug, it’s a feature: how misclassification impacts bug prediction. In 35th International Conference on Software Engineering, ICSE ’13, San Francisco, CA, USA, May 18-26, 2013, David Notkin, Betty H. C. Cheng, and Klaus Pohl (Eds.). IEEE Computer Society, 392–401.https: //doi.org/10.1109/ICSE.2013.6606585 [39] Abram Hindle, Neil A. Ernst, Michael W. Godfrey, and John Mylopoulos. 2011. Automated topic naming to support cross-project analysis of software maintenance activities. In Proceedings of the 8th International Working Conference on Mining Software Repositories, MSR 2011 (Co-located with ICSE), Waikiki, Honolulu, HI, USA, May 21-28, 2011, Proceedings, Arie van Deursen, Tao Xie, and Thomas Zimmermann (Eds.). ACM, 163–172. https://doi.org/10.1145/1985441.1985466 [40]Pieter Hooimeijer and Westley Weimer. 2007. Modeling bug report quality. In 22nd IEEE/ACM International Conference on Automated Software Engineering (ASE 2007), November 5-9, 2007, Atlanta, Georgia, USA, R. E. Kurt Stirewalt, Alexander Egyed, and Bernd Fischer (Eds.). ACM, 34–43. https://doi.org/10.1145/1321631.1321639 [41] Edwin T Jaynes. 1957. Information theory and statistical mechanics. Physical review 106, 4 (1957), 620. [42]Yuan Jiang, Pengcheng Lu, Xiaohong Su, and Tiantian Wang. 2020. LTRWES: A new framework for security bug report detection. Inf. Softw. Technol. 124 (2020), 106314. https://doi.org/10.1016/J.INFSOF.2020.106314 [43] Armand Joulin, Edouard Grave, Piotr Bojanowski, Matthijs Douze, Hervé Jégou, and Tomás Mikolov. 2016. FastText.zip: Compressing text classification models. CoRR abs/1612.03651 (2016). arXiv:1612.03651 http://arxiv.org/abs/1612.03651 [44]Rafael Kallis, Maliheh Izadi, Luca Pascarella, Oscar Chaparro, and Pooja Rani. 2023. The NLBSE’23 Tool Competition. In Proceedings of The 2nd International Workshop on Natural Language-based Software Engineering (NLBSE’23). [45]Rafael Kallis, Andrea Di Sorbo, Gerardo Canfora, and Sebastiano Panichella. 2019. Ticket Tagger: Machine Learning Driven Issue Classification. In 2019 IEEE International Conference on Software Maintenance and Evolution, ICSME 2019, Cleveland, OH, USA, September 29 - October 4, 2019. IEEE, 406–409. https://doi.org/10.1109/ICSME.2019.00070 [46]Jaweria Kanwal and Onaiza Maqbool. 2012. Bug Prioritization to Facilitate Bug Report Triage. J. Comput. Sci. Technol. 27, 2 (2012), 397–412. https://doi.org/10.1007/S11390-012-1230-3 [47] Amy J. Ko, Thomas D. LaToza, and Margaret M. Burnett. 2015. A practical guide to controlled experiments of software engineering tools with human participants. Empir. Softw. Eng. 20, 1 (2015), 110–141. https://doi.org/10.1007/S10664- 013-9279-3 [48] Ashima Kukkar and Rajni Mohana. 2018. A supervised bug report classification with incorporate and textual field knowledge. Procedia computer science 132 (2018), 352–361. [49] C Kwak and S Lee. 2022. Issue report classification using a multimodal deep learning technique. (2022). [50] Quoc V. Le and Tomás Mikolov. 2014. Distributed Representations of Sentences and Documents. In Proceedings of the 31th International Conference on Machine Learning, ICML 2014, Beijing, China, 21-26 June 2014 (JMLR Workshop and Conference Proceedings, Vol. 32). JMLR.org, 1188–1196. http://proceedings.mlr.press/v32/le14.html ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:53 [51]Dong-Gun Lee and Yeong-Seok Seo. 2020. Improving bug report triage performance using artificial intelligence based document generation model. Hum. centric Comput. Inf. Sci. 10 (2020), 26. https://doi.org/10.1186/s13673-020-00229-7 [52]Bin Li, Ying Wei, Xiaobing Sun, Lili Bo, Dingshan Chen, and Chuanqi Tao. 2022. Towards the identification of bug entities and relations in bug reports. Autom. Softw. Eng. 29, 1 (2022), 24. https://doi.org/10.1007/s10515-022-00325-1 [53]Hui Li, Yang Qu, Shikai Guo, Guofeng Gao, Rong Chen, and Chen Guo. 2020. Surprise Bug Report Prediction Utilizing Optimized Integration with Imbalanced Learning Strategy. Complex. 2020 (2020), 8509821:1–8509821:14. https://doi.org/10.1155/2020/8509821 [54]Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A Robustly Optimized BERT Pretraining Approach. CoRR abs/1907.11692 (2019). arXiv:1907.11692 http://arxiv.org/abs/1907.11692 [55]Guoming Long and Tao Chen. 2022. On Reporting Performance and Accuracy Bugs for Deep Learning Frameworks: An Exploratory Study from GitHub. In EASE 2022: The International Conference on Evaluation and Assessment in Software Engineering 2022, Gothenburg, Sweden, June 13 - 15, 2022, Miroslaw Staron, Christian Berger, Jocelyn Simmonds, and Rafael Prikladnicki (Eds.). ACM, 90–99. https://doi.org/10.1145/3530019.3530029 [56]Guoming Long, Tao Chen, and Georgina Cosma. 2022. Multifaceted Hierarchical Report Identification for Non- Functional Bugs in Deep Learning Frameworks. In 29th Asia-Pacific Software Engineering Conference, APSEC 2022, Virtual Event, Japan, December 6-9, 2022. IEEE, 289–298. https://doi.org/10.1109/APSEC57359.2022.00041 [57] Guoming Long, Jingzhi Gong, Hui Fang, and Tao Chen. 2025. Learning Software Bug Reports: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. (July 2025). https://doi.org/10.1145/3750040 Just Accepted. [58]Nicola Lunardon, Giovanna Menardi, and Nicola Torelli. 2014. ROSE: a package for binary imbalanced learning. R journal 6, 1 (2014). [59] Youpeng Ma, Tao Chen, and Ke Li. 2025. Faster Configuration Performance Bug Testing with Neural Dual-Level Prioritization. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 988–1000. https://doi.org/10.1109/ICSE55347.2025.00201 [60]Walid Maalej and Hadeer Nabil. 2015. Bug report, feature request, or simply praise? On automatically classifying app reviews. In 23rd IEEE International Requirements Engineering Conference, RE 2015, Ottawa, ON, Canada, August 24-28, 2015, Didar Zowghi, Vincenzo Gervasi, and Daniel Amyot (Eds.). IEEE Computer Society, 116–125. https: //doi.org/10.1109/RE.2015.7320414 [61] Senthil Mani, Rose Catherine, Vibha Singhal Sinha, and Avinava Dubey. 2012. AUSUM: approach for unsupervised bug report summarization. In 20th ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE-20), SIGSOFT/FSE’12, Cary, NC, USA - November 11 - 16, 2012, Will Tracz, Martin P. Robillard, and Tevfik Bultan (Eds.). ACM, 11. https://doi.org/10.1145/2393596.2393607 [62] Jyoti Prakash Meher, Sourav Biswas, and Rajib Mall. 2024. Deep learning-based software bug classification. Inf. Softw. Technol. 166 (2024), 107350. https://doi.org/10.1016/J.INFSOF.2023.107350 [63]Fanqi Meng, Xuesong Wang, Jingdong Wang, and Peifang Wang. 2022. Automatic Classification of Bug Reports Based on Multiple Text Information and Reports’ Intention. In Theoretical Aspects of Software Engineering - 16th International Symposium, TASE 2022, Cluj-Napoca, Romania, July 8-10, 2022, Proceedings (Lecture Notes in Computer Science, Vol. 13299), Yamine Aït Ameur and Florin Craciun (Eds.). Springer, 131–147. https://doi.org/10.1007/978-3- 031-10363-6_9 [64]Nikolaos Mittas and Lefteris Angelis. 2013. Ranking and Clustering Software Cost Estimation Models through a Multiple Comparisons Algorithm. IEEE Trans. Software Eng. (2013). [65]Anas Nadeem, Muhammad Usman Sarwar, and Muhammad Zubair Malik. 2022. Automatic Issue Classifier: A Transfer Learning Framework for Classifying Issue Reports. CoRR abs/2202.06149 (2022). arXiv:2202.06149 https: //arxiv.org/abs/2202.06149 [66] Behzad Soleimani Neysiani and Seyed Morteza Babamir. 2019. New methodology for contextual features usage in duplicate bug reports detection: dimension expansion based on manhattan distance similarity of topics. In 2019 5th international conference on web research (ICWR). IEEE, 178–183. [67] Behzad Soleimani Neysiani, Seyed Morteza Babamir, and Masayoshi Aritsugi. 2020. Efficient feature extraction model for validation performance improvement of duplicate bug report detection in software bug triage systems. Inf. Softw. Technol. 126 (2020), 106344. https://doi.org/10.1016/J.INFSOF.2020.106344 [68] Nitish Pandey, Debarshi Kumar Sanyal, Abir Hudait, and Amitava Sen. 2017. Automated classification of software issue reports using machine learning techniques: an empirical study. Innov. Syst. Softw. Eng. 13, 4 (2017), 279–297. https://doi.org/10.1007/S11334-017-0294-1 [69] Sebastiano Panichella, Gerardo Canfora, and Andrea Di Sorbo. 2021. "Won’t We Fix this Issue? " Qualitative char- acterization and automated identification of wontfix issues on GitHub. Inf. Softw. Technol. 139 (2021), 106665. https://doi.org/10.1016/J.INFSOF.2021.106665 ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:54Guoming Long, Shihai Wang, Hui Fang, and Tao Chen [70]Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global Vectors for Word Representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, Alessandro Moschitti, Bo Pang, and Walter Daelemans (Eds.). ACL, 1532–1543. https://doi.org/10.3115/V1/D14-1162 [71]Quentin Perez, Pierre-Antoine Jean, Christelle Urtado, and Sylvain Vauttier. 2021. Bug or not bug? That is the question. In 29th IEEE/ACM International Conference on Program Comprehension, ICPC 2021, Madrid, Spain, May 20-21, 2021. IEEE, 47–58. https://doi.org/10.1109/ICPC52881.2021.00014 [72]Jantima Polpinij. 2021. A method of non-bug report identification from bug report repository. Artif. Life Robotics 26, 3 (2021), 318–328. https://doi.org/10.1007/S10015-021-00681-3 [73]Ketan Rathor, Jaspreet Kaur, Ullal Akshatha Nayak, S Kaliappan, Ramya Maranan, and V Kalpana. 2023. Technological Evaluation and Software Bug Training using Genetic Algorithm and Time Convolution Neural Network (GA-TCN). In 2023 Second International Conference on Augmented Intelligence and Sustainable Systems (ICAISS). IEEE, 7–12. [74] Pengzhen Ren, Yun Xiao, Xiaojun Chang, Po-Yao Huang, Zhihui Li, Brij B. Gupta, Xiaojiang Chen, and Xin Wang. 2022. A Survey of Deep Active Learning. ACM Comput. Surv. 54, 9 (2022), 180:1–180:40. https://doi.org/10.1145/3472291 [75]Gema Rodríguez-Pérez, Jesús M. González-Barahona, Gregorio Robles, Dorealda Dalipaj, and Nelson Sekitoleko. 2016. BugTracking: A Tool to Assist in the Identification of Bug Reports. In Open Source Systems: Integrating Communities - 12th IFIP WG 2.13 International Conference, OSS 2016, Gothenburg, Sweden, May 30 - June 2, 2016, Proceedings (IFIP Advances in Information and Communication Technology, Vol. 472), Kevin Crowston, Imed Hammouda, Björn Lundell, Gregorio Robles, Jonas Gamalielsson, and Juho Lindman (Eds.). Springer, 192–198. https://doi.org/10.1007/978-3-319- 39225-7_16 [76]Tommaso Dal Sasso, Andrea Mocci, and Michele Lanza. 2016. What Makes a Satisficing Bug Report?. In 2016 IEEE International Conference on Software Quality, Reliability and Security, QRS 2016, Vienna, Austria, August 1-3, 2016. IEEE, 164–174. https://doi.org/10.1109/QRS.2016.28 [77]Andrew Jhon Scott and M Knott. 1974. A cluster analysis method for grouping means in the analysis of variance. Biometrics (1974), 507–512. [78]Lin Shi, Fangwen Mu, Yumin Zhang, Ye Yang, Junjie Chen, Xiao Chen, Hanzhi Jiang, Ziyou Jiang, and Qing Wang. 2022. BugListener: Identifying and Synthesizing Bug Reports from Collaborative Live Chats. In 44th IEEE/ACM 44th International Conference on Software Engineering, ICSE 2022, Pittsburgh, PA, USA, May 25-27, 2022. ACM, 299–311. https://doi.org/10.1145/3510003.3510108 [79]Rui Shu, Tianpei Xia, Laurie A. Williams, and Tim Menzies. 2019. Better Security Bug Report Classification via Hyperparameter Optimization. CoRR abs/1905.06872 (2019). arXiv:1905.06872 http://arxiv.org/abs/1905.06872 [80]Chengnian Sun, David Lo, Siau-Cheng Khoo, and Jing Jiang. 2011. Towards more accurate retrieval of duplicate bug reports. In 26th IEEE/ACM International Conference on Automated Software Engineering (ASE 2011), Lawrence, KS, USA, November 6-10, 2011, Perry Alexander, Corina S. Pasareanu, and John G. Hosking (Eds.). IEEE Computer Society, 253–262. https://doi.org/10.1109/ASE.2011.6100061 [81]Alaa Tharwat and Wolfram Schenck. 2023. A Survey on Active Learning: State-of-the-Art, Practical Challenges and Research Directions. Mathematics 11, 4 (2023), 820. [82] Ferdian Thung, Xuan-Bach Dinh Le, and David Lo. 2015. Active semi-supervised defect categorization. In Proceedings of the 2015 IEEE 23rd International Conference on Program Comprehension, ICPC 2015, Florence/Firenze, Italy, May 16-24, 2015, Andrea De Lucia, Christian Bird, and Rocco Oliveto (Eds.). IEEE Computer Society, 60–70.https: //doi.org/10.1109/ICPC.2015.15 [83]Ha Manh Tran, Son Thanh Le, Van Sinh Nguyen, and Phong Thanh Ho. 2020. An Analysis of Software Bug Reports Using Machine Learning Techniques. SN Comput. Sci. 1, 1 (2020), 4:1–4:11. https://doi.org/10.1007/s42979-019-0004-1 [84]Huy Tu, Zhe Yu, and Tim Menzies. 2022. Better Data Labelling With EMBLEM (and how that Impacts Defect Prediction). IEEE Trans. Software Eng. 48, 2 (2022), 278–294. https://doi.org/10.1109/TSE.2020.2986415 [85]András Vargha and Harold D. Delaney. 2000. A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong. [86] Junjie Wang, Song Wang, Qiang Cui, and Qing Wang. 2016. Local-based active classification of test report to assist crowdsourced testing. In Proceedings of the 31st IEEE/ACM International Conference on Automated Software Engineering, ASE 2016, Singapore, September 3-7, 2016, David Lo, Sven Apel, and Sarfraz Khurshid (Eds.). ACM, 190–201. https://doi.org/10.1145/2970276.2970300 [87]Jun Wang, Xiaofang Zhang, Lin Chen, and Xiaoyuan Xie. 2022. Personalizing label prediction for GitHub issues. Inf. Softw. Technol. 145 (2022), 106845. https://doi.org/10.1016/J.INFSOF.2022.106845 [88] Shihai Wang and Tao Chen. 2026. Conjecture and Inquiry: Quantifying Software Performance Requirements via Interactive Retrieval-Augmented Preference Elicitation. In Findings of the Association for Computational Linguistics: ACL 2026. ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. Human-Machine Co-boosted Bug Report Identification with Mutualistic Neural Active Learning111:55 [89]Shihai Wang and Tao Chen. 2026. Light over Heavy: Automated Performance Requirements Quantification with Linguistic Inducement. In 48th IEEE/ACM International Conference on Software Engineering (ICSE). ACM. [90] Wenyao Wang, Chenhao Wu, and Jie He. 2023. CLeBPI: Contrastive Learning for Bug Priority Inference. Inf. Softw. Technol. 164 (2023), 107302. https://doi.org/10.1016/J.INFSOF.2023.107302 [91]Jason W. Wei and Kai Zou. 2019. EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (Eds.). Association for Computational Linguistics, 6381–6387. https://doi.org/10.18653/V1/D19-1670 [92] Frank Wilcoxon. 1945. Individual Comparisons by Ranking Methods. [93]Xiaoxue Wu, Wei Zheng, Xiang Chen, Yu Zhao, Tingting Yu, and Dejun Mu. 2021. Improving high-impact bug report prediction with combination of interactive machine learning and active learning. Inf. Softw. Technol. 133 (2021), 106530. https://doi.org/10.1016/j.infsof.2021.106530 [94]Xiaoxue Wu, Wei Zheng, Xin Xia, and David Lo. 2022. Data Quality Matters: A Case Study on Data Label Correctness for Security Bug Report Prediction. IEEE Trans. Software Eng. 48, 7 (2022), 2541–2556. https://doi.org/10.1109/TSE. 2021.3063727 [95] Wensheng Xia, Ying Li, Tong Jia, and Zhonghai Wu. 2019. BugIdentifier: An Approach to Identifying Bugs via Log Mining for Accelerating Bug Reporting Stage. In 19th IEEE International Conference on Software Quality, Reliability and Security, QRS 2019, Sofia, Bulgaria, July 22-26, 2019. IEEE, 167–175. https://doi.org/10.1109/QRS.2019.00033 [96]Zezhen Xiang, Jingzhi Gong, and Tao Chen. 2026. Dually Hierarchical Drift Adaptation for Online Configuration Performance Learning. In 48th IEEE/ACM International Conference on Software Engineering (ICSE). ACM. [97] Gangda Xiong and Tao Chen. 2025. CoTune: Co-evolutionary Configuration Tuning. In 40th IEEE/ACM International Conference on Automated Software Engineering, ASE 2025, Seoul, Korea, Republic of, November 16-20, 2025. IEEE, 1490–1502. https://doi.org/10.1109/ASE63991.2025.00126 [98]Xinli Yang, David Lo, Qiao Huang, Xin Xia, and Jianling Sun. 2016. Automated Identification of High Impact Bug Reports Leveraging Imbalanced Learning Strategies. In 40th IEEE Annual Computer Software and Applications Conference, COMPSAC 2016, Atlanta, GA, USA, June 10-14, 2016. IEEE Computer Society, 227–232. https://doi.org/10. 1109/COMPSAC.2016.67 [99] Xinli Yang, David Lo, Xin Xia, Qiao Huang, and Jian-Ling Sun. 2017. High-Impact Bug Report Identification with Imbalanced Learning Strategies. J. Comput. Sci. Technol. 32, 1 (2017), 181–198. https://doi.org/10.1007/s11390-017- 1713-3 [100]Yulong Ye, Tao Chen, and Miqing Li. 2025. Distilled Lifelong Self-Adaptation for Configurable Systems. In 47th IEEE/ACM International Conference on Software Engineering, ICSE 2025, Ottawa, ON, Canada, April 26 - May 6, 2025. IEEE, 1333–1345. https://doi.org/10.1109/ICSE55347.2025.00094 [101]Yulong Ye, Hongyuan Liang, Chao Jiang, Miqing Li, and Tao Chen. 2026. Revealing Domain-Spatiality Patterns for Configuration Tuning: Domain Knowledge Meets Fitness Landscapes. ACM Trans. Softw. Eng. Methodol. (March 2026). https://doi.org/10.1145/3803859 Just Accepted. [102] Syed Farhan Alam Zaidi, Faraz Malik Awan, Minsoo Lee, Honguk Woo, and Chan-Gun Lee. 2020. Applying Convolu- tional Neural Networks With Different Word Representation Techniques to Recommend Bug Fixers. IEEE Access 8 (2020), 213729–213747. https://doi.org/10.1109/ACCESS.2020.3040065 [103] Jie Zhang, Xiaoyin Wang, Dan Hao, Bing Xie, Lu Zhang, and Hong Mei. 2015. A survey on bug-report analysis. Sci. China Inf. Sci. 58, 2 (2015), 1–24. https://doi.org/10.1007/S11432-014-5241-2 [104]Yu Zhao, Ting Su, Yang Liu, Wei Zheng, Xiaoxue Wu, Ramakanth Kavuluru, William G. J. Halfond, and Tingting Yu. 2022. ReCDroid+: Automated End-to-End Crash Reproduction from Bug Reports for Android Apps. ACM Trans. Softw. Eng. Methodol. 31, 3 (2022), 36:1–36:33. https://doi.org/10.1145/3488244 [105]Wei Zheng, Yuxing Xun, Xiaoxue Wu, Zhi Deng, Xiang Chen, and Yulei Sui. 2021. A Comparative Study of Class Rebalancing Methods for Security Bug Report Classification. IEEE Trans. Reliab. 70, 4 (2021), 1658–1670. https: //doi.org/10.1109/TR.2021.3118026 [106]Wei Zheng, Manqing Zhang, Hui Tang, Yuanfang Cai, Xiang Chen, Xiaoxue Wu, and Abubakar Omari Abdallah Semasaba. 2021. Automatically Identifying Bug Reports with Tactical Vulnerabilities by Deep Feature Learning. In 32nd IEEE International Symposium on Software Reliability Engineering, ISSRE 2021, Wuhan, China, October 25-28, 2021, Zhi Jin, Xuandong Li, Jianwen Xiang, Leonardo Mariani, Ting Liu, Xiao Yu, and Nahgmeh Ivaki (Eds.). IEEE, 333–344. https://doi.org/10.1109/ISSRE52982.2021.00043 [107] Yaqin Zhou and Asankhaya Sharma. 2017. Automated identification of security issues from commit messages and bug reports. In Proceedings of the 2017 11th Joint Meeting on Foundations of Software Engineering, ESEC/FSE 2017, Paderborn, Germany, September 4-8, 2017, Eric Bodden, Wilhelm Schäfer, Arie van Deursen, and Andrea Zisman (Eds.). ACM, 914–919. https://doi.org/10.1145/3106237.3117771 ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026. 111:56Guoming Long, Shihai Wang, Hui Fang, and Tao Chen [108]Yu Zhou, Yanxiang Tong, Ruihang Gu, and Harald C. Gall. 2016. Combining text mining and data mining for bug report classification. J. Softw. Evol. Process. 28, 3 (2016), 150–176. https://doi.org/10.1002/SMR.1770 [109] Thomas Zimmermann, Rahul Premraj, Nicolas Bettenburg, Sascha Just, Adrian Schröter, and Cathrin Weiss. 2010. What Makes a Good Bug Report? IEEE Trans. Software Eng. 36, 5 (2010), 618–643. https://doi.org/10.1109/TSE.2010.63 Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009 ACM Trans. Softw. Eng. Methodol., Vol. 37, No. 4, Article 111. Publication date: August 2026.