Paper deep dive
A Unified Algebraic Framework for Classification Performance Evaluation
Ronaldo C. Prati
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 7/7/2026, 10:58:32 AM
Summary
The paper introduces a unified algebraic framework for evaluating classification performance across binary, multiclass, multilabel, ordinal, hierarchical, cost-sensitive, and soft-label settings. It represents actual and predicted labels as binary indicator matrices and uses three aggregation operators (global, column-wise, row-wise) to automatically extend any binary performance measure to all settings. The framework establishes theoretical equivalences between averaging schemes, characterizes skew-invariant measures, and demonstrates that micro-precision, micro-recall, and micro-F1 equal accuracy in multiclass settings.
Entities (8)
Relation Signals (8)
Unified Algebraic Framework → encompasses → Classification Settings
confidence 95% · encompasses binary, multiclass, multilabel, ordinal, hierarchical, cost-sensitive, and soft-label settings within a single formalism.
Global Aggregation Operator → correspondsto → Micro-averaging
confidence 92% · global ($\Sigma_1$)... corresponding to micro... averaging.
Column-wise Aggregation Operator → correspondsto → Macro-averaging
confidence 92% · column-wise ($\Sigma_m$)... corresponding to macro/weighted... averaging.
Row-wise Aggregation Operator → correspondsto → Exemplar averaging
confidence 92% · row-wise ($\Sigma_n$)... corresponding to... exemplar averaging.
Binary Indicator Matrices → combinedwith → Aggregation Operators
confidence 90% · The foundation is a representation of actual and predicted labels as binary indicator matrices, combined with three aggregation operators
Micro-precision, Micro-recall, Micro-F1 → equalto → Accuracy
confidence 90% · micro-precision, micro-recall, and micro-F1 are all equal to accuracy in multiclass settings.
Product t-norm → preserves → Confusion-matrix partition
confidence 85% · the product t-norm is the unique one preserving the confusion-matrix partition
Cost Matrix → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:We propose a unified algebraic framework for classification performance evaluation that encompasses binary, multiclass, multilabel, ordinal, hierarchical, cost-sensitive, and soft-label settings within a single formalism. The foundation is a representation of actual and predicted labels as binary indicator matrices, combined with three aggregation operators -- global, column-wise, and row-wise -- that correspond exactly to micro, macro/weighted, and exemplar averaging. Any binary performance measure expressed in terms of true/positive/negative counts extends automatically to all settings by substituting these operators, generating multiclass and multilabel versions without measure-specific derivations. The framework further accommodates soft classifier outputs via argmax or thresholding, soft ground truth via triangular norms, ordinal classification via membership functions or cumulative encodings, and cost-sensitive evaluation via a cost matrix that subsumes MAE and MSE as special cases. We establish several theoretical results: micro-averaging equals denominator-weighted macro-averaging; the product $t$-norm is the unique one preserving the confusion-matrix partition; skew-invariant measures are characterised as functions of recall and specificity; and micro-precision, micro-recall, and micro-$F_1$ are all equal to accuracy in multiclass settings. Empirical illustrations on synthetic and real data confirm the theoretical findings.
Tags
Links
- Source: https://arxiv.org/abs/2607.04028v1
- Canonical: https://arxiv.org/abs/2607.04028v1
Trouble viewing inline? Open PDF directly →
Full Text
120,260 characters extracted from source content.
Expand or collapse full text
A Unified Algebraic Framework for Classification Performance Evaluation Ronaldo C. Prati ronaldo.prati@ufabc.edu.br Universidade Federal do ABC (UFABC), Av. dos Estados, 5001, Santo André, SP, 09210-580, Brazil Abstract We propose a unified algebraic framework for classification performance evaluation that encompasses binary, multiclass, multilabel, ordinal, hierarchical, cost-sensitive, and soft-label settings within a single formalism. The foundation is a representation of actual and predicted labels as binary indicator matrices, combined with three aggregation operators — global, column-wise, and row-wise — that correspond exactly to micro, macro/weighted, and exemplar averaging. Any binary performance measure expressed in terms of true/positive/negative counts extends automatically to all settings by substituting these operators, generating multiclass and multilabel versions without measure-specific derivations. The framework further accommodates soft classifier outputs via argmax or thresholding, soft ground truth via triangular norms, ordinal classification via membership functions or cumulative encodings, and cost-sensitive evaluation via a cost matrix that subsumes MAE and MSE as special cases. We establish several theoretical results: micro-averaging equals denominator-weighted macro-averaging; the product t-norm is the unique one preserving the confusion-matrix partition; skew-invariant measures are characterised as functions of recall and specificity; and micro-precision, micro-recall, and micro-F1F_1 are all equal to accuracy in multiclass settings. Empirical illustrations on synthetic and real data confirm the theoretical findings. keywords: classification performance measures , multilabel classification , evaluation framework , micro-averaging , macro-averaging , cost-sensitive evaluation , confusion matrix , †journal: Information Fusion 1 Introduction Evaluating the performance of classification algorithms is an essential step for machine learning researchers and practitioners [21, 28]. The process of comparing, selecting and tuning the parameters of learning algorithms is generally based on the performance evaluation by means of quality measures of the learned models, evaluated on test data. Despite its universality, evaluating classification models is a complex and still open-ended challenge [20, 2]. In the last decades, numerous performance measures have been proposed to evaluate classification models [31, 12, 4, 22]. The choice of a particular measure generally depends on the problem at hand, as well as the preferences of a particular domain. In medicine, sensitivity and specificity are often used; in information retrieval, precision and recall are preferred; in cost-sensitive applications, misclassification costs must be incorporated; and in settings with uncertain or crowdsourced labels, soft evaluation criteria are required. The complexity of evaluation grows substantially as the classification setting moves beyond the standard binary case. In multiclass problems, multiple classes compete and errors between different pairs of classes need not be treated equally. In multilabel settings, each example may simultaneously belong to several classes, creating interactions between labels that scalar measures must somehow summarize. In ordinal and hierarchical classification, the structure of the class space carries information about error severity. In single-class, open-set, and abstaining classifiers, the boundary between known and unknown is itself part of the prediction. Across all these settings, the same underlying measure — say, F1F_1 — can be computed in multiple ways (micro, macro, weighted, exemplar), each emphasising different aspects of performance and potentially producing contradictory rankings of competing classifiers, as we demonstrate empirically. A further complication is that evaluation methodology has not kept pace with the diversity of modern classification architectures. Binary measures are well-studied, but their extensions to non-standard settings are typically derived on a case-by-case basis, leading to a fragmented landscape in which the relationships between measures, and the conditions under which they agree or disagree, are poorly understood. In this paper, we address this fragmentation by proposing a unified algebraic framework for classification performance evaluation. The foundation is a representation of actual and predicted labels as binary indicator matrices, combined with three aggregation operators — global (Σ1 _1), column-wise (Σm _m), and row-wise (Σn _n) — corresponding to micro, macro/weighted, and exemplar averaging. Any binary measure expressed in terms of tptp, fpfp, fnfn, tntn extends automatically to multiclass, multilabel, ordinal, hierarchical, cost-sensitive, and soft-label settings by substituting these operators. The framework accommodates soft classifier outputs via argmax or thresholding, soft ground truth via triangular norms, ordinal structure via membership functions or cumulative encodings, and cost sensitivity via a cost matrix. We establish theoretical results on the relationships between aggregation schemes, characterise which measures are sensitive to class imbalance, identify redundancies among measures, and show that micro-precision, micro-recall, and micro-F1F_1 are all equal to accuracy in multiclass settings. Experiments on synthetic and real multilabel data confirm the practical significance of these results. 2 Related Work Research on classification evaluation measures can be grouped into four broad streams: cataloguing binary measures, extending individual measures to multiclass or multilabel settings, proposing unified views of performance evaluation, and handling non-standard settings such as cost-sensitivity or label uncertainty. We position our contribution with respect to each. The most common type of contribution in this area is a systematic survey of existing measures and their properties. Sokolova and Lapalme [31] analyse twelve binary measures across five properties (consistency, correctness, completeness, reliability, and expected behaviour) and extend a subset to multiclass and multilabel settings. Ferri et al. [12] conduct an experimental comparison of measures on synthetic and real data, showing that rankings of classifiers are highly sensitive to measure choice. Prati et al. [28] survey graphical methods for performance evaluation and their relationship to scalar measures. While these works provide valuable taxonomies, they analyse measures individually and do not provide a generative mechanism for producing multiclass or multilabel versions automatically. In particular, Sokolova and Lapalme’s property framework must be applied separately to each measure and each extension; Propositions 2 and 3 of this paper show that two of their key properties (boundary correctness and monotonicity) are preserved by all three aggregation operators simultaneously, covering every derived measure in Table 3 at once. The idea of decomposing multiclass evaluation into per-class binary problems dates at least to Sebastiani [30] in the context of text categorisation, where per-class tpjtp_j, fpjfp_j, fnjfn_j, tnjtn_j counts and their micro and macro averages are defined. Tsoumakas and Katakis [33] and Zhang and Zhou [36] survey multilabel classification and catalogue example-based and label-based measures, including Hamming loss and exemplar precision and recall. Godbole and Sarawagi [17] extend precision, recall and F1F_1 to multilabel problems using set-intersection semantics. Our framework subsumes all of these extensions: as shown in Table 3, they correspond to specific choices of aggregation operator and measure formula. Moreover, our framework does so for any binary measure simultaneously, rather than for specific measures individually. The closest work in spirit to ours is Hernández-Orallo et al. [20], who propose a unified view of performance metrics as expected classification losses under threshold variation, covering binary and cost-sensitive settings via cost curves and the ROC framework. Their approach focuses on continuous score-based classifiers and the relationship between thresholds and loss functions, whereas ours operates at the level of the confusion matrix and provides an explicit algebraic mechanism for generating multiclass and multilabel measures. The two frameworks are therefore complementary: theirs gives a decision-theoretic unification for binary and cost-sensitive cases, ours gives an algebraic unification across all classification settings. Kautz et al. [22] propose a generic multiclass performance measure derived from geometrical considerations on the confusion matrix, while Carbonero-Ruz et al. [2] propose a two-dimensional accuracy-based measure. Both works target specific gaps in multiclass evaluation rather than a generative framework. Cost-sensitive evaluation has a long history [9, 7, 20], typically addressed by incorporating misclassification costs directly into the learning objective or by post-hoc reweighting of confusion matrix entries. Our framework formalises cost sensitivity via a cost matrix that acts as a bilinear form on the indicator matrices (Eq. 14), unifying misclassification cost, MAE and MSE as special cases (Theorem 3). Evaluation under label uncertainty—arising from annotator disagreement or probabilistic ground truth—has received increasing attention in crowdsourcing [28] and learning from crowds, but existing work typically aggregates labels into a single hard assignment before evaluation. Our soft-label extension instead propagates uncertainty directly through the confusion matrices via t-norms, making the treatment of label ambiguity an explicit, interpretable part of the evaluation protocol. To the best of our knowledge, no prior work connects these settings—multiclass, multilabel, ordinal, cost-sensitive, and uncertain-label—within a single algebraic framework. Notation Throughout the paper we use the following conventions. Scalars are denoted by plain lowercase italic letters: tptp, fpfp, fnfn, tntn are aggregate confusion-matrix counts, and n, m are dimensions. Vectors are denoted by bold lowercase letters ( y, tp, etc.) and matrices by bold uppercase letters ( Y, TP, etc.). Bold uppercase matrices TP, FP, FN, TN appear in text when referring to the matrices as objects; subscripted entries i,j TP_i,j are used in proofs where a single element is meant. More specifically: in the binary case (Section 3), ∈0,1n tp∈\0,1\^n is the element-wise true-positive indicator vector and tp=Σ1()∈ℤ≥0tp= _1( tp) _≥ 0 its scalar sum. In the multiclass and multilabel case, ∈0,1n×m TP∈\0,1\^n× m is the true-positive indicator matrix. Applying Σm _m yields a per-class count vector ∈ℤ≥0m tp _≥ 0^m; applying Σn _n yields a per-example count vector ∈ℤ≥0n tp _≥ 0^n; and applying Σ1 _1 yields the global scalar tp∈ℤ≥0tp _≥ 0. The type of tp (indicator vector, count vector, or scalar) is determined by which aggregation operator has been applied. The same conventions hold for fp, fn, tn and their uppercase counterparts. Actual and predicted label matrices are Y and Y; their binary-case vector counterparts are y and y. 3 The binary case Consider a binary classification problem with a test set111We assume a single fixed test set throughout this section. The extension to resampling procedures such as cross-validation, where predictions are aggregated across multiple test folds, is formalised in Section 6.2. of n examples, where each example is associated with one of two predefined, non-overlapping classes c1c_1 and c2c_2. Without loss of generality, these two classes are also called positive and negative classes, respectively, and denoted as c+c_+ and c−c_-. A binary classifier will assign a predicted class to each test example. Most of the performance evaluation measures are based on the cross-tabulation of actual and predicted class values (the Confusion Matrix), as shown in Table 1, where true positives/negatives (tp/tntp/tn) corresponds to the count of examples which belong to the positive/negative class and are correctly classified as positive/negative; false positives/negatives (fp/fnfp/fn) corresponds to the count of examples which belong to the positive/negative class and are incorrectly classified as positive/negative; actual positives/negatives (tpos/tnegtpos/tneg) corresponds to the total of examples of the positive/negative class; predicted positives/negatives (ppos/pnegppos/pneg) corresponds to the total of examples predicted as positive/negative; and the total number of examples (n) is the sample size. Predicted positive negative Actual positive tptp fnfn tpostpos negative fpfp tntn tnegtneg pposppos pnegpneg n Table 1: Binary confusion matrix. Different performance measures are derived from the cells of the confusion matrix, each encoding a different evaluation priority. Table 2 lists the eleven measures used throughout the paper: some treat both classes symmetrically (accuracy, error rate), some focus on the positive class (precision, recall, FβF_β, Jaccard), some measure balance across both classes (balanced accuracy, G-mean), and some capture global agreement (MCC [26], Cohenś κ [3]). Measure Formula Interpretation Accuracy tp+tn tp+tnn Proportion of correctly classified examples Error Rate fp+fn fp+fnn Proportion of incorrectly classified examples Precision tptp+fp tptp+fp Proportion of predicted positives that are truly positive Recall tptp+fn tptp+fn Proportion of actual positives correctly classified Specificity tntn+fp tntn+fp Proportion of actual negatives correctly classified FβF_β-measure (β=1β\!=\!1: F1F_1) (1+β2)tp(1+β2)tp+β2fn+fp (1+β^2)\,tp(1+β^2)\,tp+β^2fn+fp Weighted harmonic mean of precision and recall Jaccard index tptp+fp+fn tptp+fp+fn Intersection over union of predicted and actual positives Balanced accuracy 12(tptp+fn+tntn+fp) 12\! ( tptp+fn+ tntn+fp ) Average of recall and specificity G-mean tptp+fn⋅tntn+fp tptp+fn· tntn+fp Geometric mean of recall and specificity MCC [26] tp⋅tn−fp⋅fn(tp+fp)(tp+fn)(tn+fp)(tn+fn) tp· tn-fp· fn (tp+fp)(tp+fn)(tn+fp)(tn+fn) Correlation between observed and predicted classifications Cohen’s κ [3] po−pe1−pe p_o-p_e1-p_e, po=tp+tnnp_o\!=\! tp+tnn, pe=tpos⋅ppos+tneg⋅pnegn2p_e\!=\! tpos· ppos+tneg· pnegn^2 Agreement corrected for chance Table 2: Binary performance measures used as the basis for the framework extensions in Table 3. Although some of these measures, such as accuracy or error rate, extend naturally to multiclass, multilabel, ordinal, and other settings, others focus on a single class and require an averaging scheme to be applied. The plethora of possible extensions and averaging choices motivates the unified framework developed in the remainder of this paper. To provide a unified view across all classification settings, let us reframe these measures in terms of binary indicator vectors. Let y be a binary vector where, for each example eie_i, yi=(ActualClass(ei)=c+)y_i=I(ActualClass(e_i)=c_+) for i∈[1,n]i∈[1,n], and (⋅)I(·) is an indicator function whose value is 1 if its argument is true and zero otherwise. Similarly, let y be a binary indicator vector of size n defined as y^i=(PredictedClass(ei)=c+) y_i=I(PredictedClass(e_i)=c_+). Definition 1. Based on those two vectors, we can define the indicator vectors tp, fp, fn and tn as follows: tp =∧ = y y fp =¬∧ = y y fn =∧¬ = y y tn =¬∧¬ = y y where ∧ and ¬ are element-wise Boolean operators and and not, respectively. Each vector has value 1 precisely where the corresponding true/false positive/negative outcome occurs for example eie_i. The scalar confusion-matrix counts follow by summing: tp=Σ1()tp= _1( tp), and analogously for fpfp, fnfn, tntn, where Σ1 _1 denotes the global summation operator introduced next. The counts of the confusion matrix can be obtained by applying a summation function to the indicator vectors tp, fp, fn and tn. For binary classification problems, a common choice is the total of 1’s in the resulting binary indicator vectors. We will use the symbol Σ1 _1 for this summation function. The 11 symbol indicates that the result of the aggregation function is a single value. For instance, the true positive count can be calculated by tp=Σ1()tp= _1( tp). For binary classification problems, this notation may look clumsy, but it will be useful for relating to multiclass and multilabel cases, as will be discussed later in this paper. Furthermore, it allows some algebraic manipulations that may provide different perspectives for some measures. For instance, the error rate can be rewritten as: error rate =fp+fn=Σ1(fp)+Σ1(fn)n = fp+fnn= _1(fp)+ _1(fn)n (1) =Σ1(¬∧^)+Σ1(∧¬^)n = _1( y y)+ _1( y y)n (2) =Σ1((¬∧^)∨(∧¬^))n = _1(( y y) ( y y))n (3) =Σ1(⊕^)n = _1( y y)n (4) where ∨ and ⊕ are the or and xor bit-wise Boolean operators. Eq. (2) can be directly obtained from (1) by substituting the vectorial definition of fpfp and fnfn. As these vectors are mutually exclusive, we can rewrite (2) as (3), and finally (4) can be obtained from (3) by some Boolean algebraic manipulation. In other words, the error rate equals Σ1(⊕^)/n _1( y y)/n, the normalised count of bit disagreements between the actual and predicted indicator vectors. This formulation generalises directly to the multilabel matrix setting, where it recovers Hamming loss [33]: HammingLoss=Σ1(⊕^)/(nm)HammingLoss= _1( Y Y)/(nm) (Section 5.1, Proposition 4). 4 Extension to multiclass and multilabel cases In multiclass problems, each example is associated with only one of the m>2m>2 classes ∈c1,…,cm∈\c_1,…,c_m\. Furthermore, in multilabel problems, each example can be associated with one or more of the m≥2m≥ 2 labels ∈l1,…,lm∈\l_1,…,l_m\. In multiclass problems, classes are mutually exclusive, as an example cannot belong to more than one class. This does not apply to multilabel problems, where more than one label can be associated to each example. While some binary measures extend naturally to these settings, others focus on a single class or label and require an aggregation scheme to summarise results across classes, labels, or examples. We formalise the three most common schemes below, after introducing the matrix representation they operate on. The binary indicator vectors of Section 3 extend directly to a matrix representation. The actual classes or labels of example eie_i are encoded in the binary indicator matrix ∈0,1n×m Y∈\0,1\^n× m, with n rows (examples) and m columns (classes or labels), where Yi,j=(ActualClass(ei)=cj)Y_i,j=I(ActualClass(e_i)=c_j) in the multiclass case and Yi,j=(lj is relevant for ei)Y_i,j=I(l_j is relevant for e_i), for i∈[1,…,n]i∈[1,…,n] and j∈[1,…,m]j∈[1,…,m]. Similarly, the predicted classes or the predicted labels are represented by the binary indicator matrix Y, with n rows and m columns where Y^i,j=(PredictedClass(ei)=cj) Y_i,j=I(PredictedClass(e_i)=c_j) in the multiclass case and Y^i,j=(lj is predicted relevant for ei) Y_i,j=I(l_j is predicted relevant for e_i). For multiclass problems, this representation is equivalent to the one-hot encoding scheme, where a binary indicator column vector is used to represent each class. The column vectors are stacked together to form the binary indicator matrix, and for each row only one entry is equal to 1. For multilabel problems, it is common to associate with each example a binary vector where the value 1 is used to represent relevant labels for that example, and a 0 for non-relevant labels. Unlike the multiclass case, more than one entry per row may equal 1. Definition 2. Similarly to the binary class case, we can define TP, FP, FN and TN matrices in terms of the matrices Y and Y as follows: TP =∧ = Y Y FP =¬∧ = Y Y TN =¬∧¬ = Y Y FN =∧¬ = Y Y where ∧ and ¬ are element-wise Boolean operators and and not, respectively. These matrices are also binary indicator matrices, where 1 appears only where a true/false positive/negative case occurs for example eie_i and class cjc_j or label ljl_j. The true/false positive/negative counts can also be obtained by summing the number of 1s in the indicator matrices. However, unlike the binary case where the resulting operations are binary indicator vectors and only a single aggregation function is generally applied, for multiclass and multilabel cases we can define three different aggregation functions to be applied: 1. Global aggregation Σ1 _1 which returns the sum of 1s in the entire resulting matrix. 2. Column-wise aggregation Σm _m, which returns a vector of length m corresponding to the column-wise sum of 1s in the resulting matrix, giving counts per class or per label; and 3. Row-wise aggregation Σn _n, which returns a vector of length n corresponding to the row-wise sum of 1s in the resulting matrix, giving counts per example. These aggregation functions can be used to define the different averaging versions of performance measures. Definition 3. Micro-averaging can be defined in terms of Σ1 _1. Computing tp=Σ1()tp= _1( TP), fp=Σ1()fp= _1( FP), tn=Σ1()tn= _1( TN) and fn=Σ1()fn= _1( FN) is equivalent to the computing of the global counts, as these matrices will have as many 1s as the number of true/false positive/negative counts. Definition 4. Macro-averaging and weighted averaging can be defined in terms of Σm _m. By computing =Σm() tp= _m( TP), =Σm() fp= _m( FP), =Σm() tn= _m( TN) and =Σm() fn= _m( FN), we end up with vectors of length m where each position corresponds to a per class or per label counts of true/false positive/negative rates. These vectors can be used to compute a vectorized version of the performance measures, obtaining a vector of length m where each position corresponds to a per class or per label measure. Macro averaging corresponds to the unweighted average of this vector, while weighted averaging uses the class or label prevalence to compute a weighted average of this vector. Definition 5. Exemplar-averaging can be defined in terms of Σn _n. By computing =Σn() tp= _n( TP), =Σn() fp= _n( FP), =Σn() tn= _n( TN) and =Σn() fn= _n( FN), we end up with vectors of length n where each position corresponds to a per instance counts of true/false positive/negative rates. These vectors can be used to compute a vectorized version of the performance measures, obtaining a vector of length n where each position corresponds to a per instance measure. Exemplar averaging corresponds to the unweighted average of this vector. 5 Extending Binary Measures to Multiclass and Multilabel The central contribution of the proposed framework is that any binary performance measure M expressible as a function of tptp, fpfp, fnfn and tntn can be extended to multiclass and multilabel settings automatically, by replacing those scalar counts with the output of the three aggregation operators. Concretely, let M(tp,fp,fn,tn)M(tp,fp,fn,tn) denote a generic binary measure. Its three extensions are: Mmicro M^micro =M(Σ1(),Σ1(),Σ1(),Σ1()) =M\! ( _1( TP),\; _1( FP),\; _1( FN),\; _1( TN) ) (5) Mjclass M_j^class =M([Σm()]j,[Σm()]j,[Σm()]j,[Σm()]j) =M\! ([ _m( TP)]_j,\;[ _m( FP)]_j,\;[ _m( FN)]_j,\;[ _m( TN)]_j ) (6) Miexample M_i^example =M([Σn()]i,[Σn()]i,[Σn()]i,[Σn()]i) =M\! ([ _n( TP)]_i,\;[ _n( FP)]_i,\;[ _n( FN)]_i,\;[ _n( TN)]_i ) (7) Eq. (5) gives the micro-average version of M; Eq. (6) gives a per-class (or per-label) scalar whose unweighted or prevalence-weighted average yields the macro or weighted-average version; and Eq. (7) gives a per-example scalar whose average yields the exemplar version. The framework thus acts as a measure generator: one binary formula, three aggregation operators, and an entire family of multiclass and multilabel measures follows automatically. A comprehensive catalogue of more than 50 binary measures with their micro, macro, and exemplar extensions — over 400 novel named measures in total — is provided as supporting information. As a concrete illustration, consider informedness [27] (also known as Youdenś J), defined as recall+specificity−1recall+specificity-1. This measure does not appear in Table 3, yet its three extensions follow immediately from Eqs. (5)–(7): micro-informedness pools global counts before computing recall+specificity−1recall+specificity-1; macro-informedness averages per-class values recallj+specificityj−1recall_j+specificity_j-1 over all m classes; and exemplar-informedness averages the per-example analogue—none of which has a standard named version in the multiclass or multilabel literature. Table 3 applies this recipe to the measures introduced in Table 2, showing the result of each aggregation operator and its established name where one exists; novel marks cases for which no standard named version appears in the literature. Measure Σ1 _1 (micro) Σm _m (per-class → macro) Σn _n (per-example → exemplar) Precision Micro-precision Macro-precision Exemplar-precision Recall Micro-recall Macro-recall Exemplar-recall Specificity Micro-specificity Macro-specificity Exemplar-specificity FβF_β Micro-FβF_β Macro-FβF_β Exemplar-FβF_β Jaccard Micro-Jaccard Macro-Jaccard (mean IoU) Exemplar-Jaccard Error rate Hamming loss† Per-class error rate Per-example error rate Accuracy Label accuracy∗ Per-class accuracy Per-example accuracy∗ Balanced accuracy Micro-BA Macro-BA Exemplar-BA (novel) G-mean Micro-GM Macro-GM Exemplar-GM (novel) MCC Micro-MCC‡ Macro-MCC (novel) Exemplar-MCC (novel) Cohen’s κ Micro-κ§ Macro-κ§ Exemplar-κ (novel)§ † Hamming loss [33] =Σ1(⊕^)/(nm)= _1( Y Y)/(nm); see Section 3. ∗ See the discussion in Section 5.1. ‡ Related to but distinct from Gorodkin’s multiclass MCC; see Section 5.1. § pep_e depends on marginal class distributions and does not factor purely into tptp/fpfp/fnfn/tntn; an extension is possible but requires auxiliary per-class or per-example marginals (Section 5.1). Table 3: Extensions of binary performance measures (see Table 2 for binary formulas) to multiclass and multilabel settings via the three aggregation operators. Novel indicates cases for which no established named measure exists in the literature. 5.1 Notable cases As shown in Section 3, the micro extension of error rate—i.e., Σ1 _1 applied to (fp+fn)/(tp+fp+fn+tn)(fp+fn)/(tp+fp+fn+tn)—coincides with Hamming loss when normalised by nmnm [33]. This is an instance of a general pattern: measures introduced as specific to multilabel classification often turn out to be micro-averages of simpler binary measures under this framework. Applying Σ1 _1 to the accuracy formula yields label accuracy, (Σ1()+Σ1())/(nm)( _1( TP)+ _1( TN))/(nm), which counts correct label assignments across all examples and all labels. This measure appears in the multilabel literature under various names but is rarely connected to binary accuracy explicitly. For multiclass problems (one-hot encoding), the simpler quantity Σ1()/n _1( TP)/n recovers standard accuracy directly, since each correctly classified example contributes exactly one entry to TP. A subtlety arises with the exemplar extension: per-example accuracy equals (tpi+tni)/m(tp_i+tn_i)/m. In the multiclass one-hot case this equals (m−1)/m(m-1)/m for a correctly classified example and (m−2)/m(m-2)/m for an incorrect one, so its average is a linear transformation of standard accuracy rather than accuracy itself. In the multilabel case the exemplar version is a standard example-based accuracy used in the literature. This asymmetry reflects a broader point: measures involving tntn behave differently under multiclass one-hot encoding than under multilabel encoding, because the number of true negatives per example grows with m in the former. Practitioners should be aware of this when applying measures that involve tntn in multiclass settings. MCC is particularly interesting because its extension to multiclass settings has been debated in the literature, with competing proposals [18, 20]. The present framework provides a natural and interpretable resolution. Applying Σm _m yields a per-class MCC for each one-vs-rest decomposition; its unweighted average is a macro-MCC whose value is directly interpretable as the average binary correlation across classes. Applying Σ1 _1 pools all counts globally before computing the correlation, giving a micro-MCC that is related to Gorodkin’s proposal but operates on the binary indicator matrices rather than the full m×m× m confusion matrix. Applying Σn _n gives a per-example MCC for multilabel problems, for which no standard named version currently exists. Cohen’s κ [3] requires additional care because the expected agreement term pe=(tpos⋅ppos+tneg⋅pneg)/n2p_e=(tpos· ppos+tneg· pneg)/n^2 depends on marginal class distributions, so it does not factor purely into tptp, fpfp, fnfn, tntn. Within the matrix framework, however, these marginals are row and column sums of Y and Y, and pep_e can be computed separately for each class column or each example row alongside the standard counts. The three aggregation schemes can then be applied, at the cost of tracking these auxiliary marginals alongside the standard tptp, fpfp, fnfn, tntn counts. 5.2 Connection to Standard Implementations The framework’s aggregation operators correspond directly to the average parameter used in widely adopted machine learning libraries such as scikit-learn [20]. Specifically: 1. average=’micro’ pools all per-label counts globally before computing the measure, corresponding to Σ1 _1. 2. average=’macro’ computes the measure per class and takes the unweighted mean, corresponding to Σm _m with uniform weights. 3. average=’weighted’ computes the measure per class and takes a prevalence-weighted mean, corresponding to Σm _m with weights wj=tposj/nw_j=tpos_j/n. 4. average=’samples’ computes the measure per example and takes the unweighted mean, corresponding to Σn _n. The framework provides the theoretical foundation for these four options and their relationships. In particular, Corollary 1 explains precisely when micro and weighted agree (when per-class denominators are equal), Corollary 4 gives the exact magnitude of their divergence as a covariance, and Theorem 5 explains why micro precision, recall, and F1F_1 are always identical to accuracy in multiclass settings — a fact that practitioners encounter empirically but that lacks an explicit justification in most library documentation. The framework also identifies what is absent: none of the four average options corresponds to cost-sensitive, soft-label, or ordinal evaluation, all of which are covered by the extensions in Section 6. 6 Extended Settings and Special Cases So far we have assumed that the classifier produces a hard assignment — a single class (multiclass) or a subset of labels (multilabel) — for each example. In practice, many classifiers produce soft outputs: a matrix ^∈[0,1]n×m P∈[0,1]^n× m, where P^i,j P_i,j represents the estimated probability, score, or membership degree of example eie_i with respect to class cjc_j or label ljl_j. The framework accommodates these outputs by converting P into an indicator matrix Y before computing any performance measure. In multiclass problems, a probabilistic classifier typically produces a matrix P whose rows sum to one (a proper probability distribution over classes). Because the predicted class is the one with highest probability, the indicator matrix is obtained by applying an argmax per row: Y^i,j=(j=argmaxk∈[1,m]P^i,k). Y_i,j=I\! (j= k∈[1,m] \; P_i,k ). (8) This guarantees that each row of Y has exactly one entry equal to 1, preserving the mutual-exclusivity constraint of multiclass problems. In multilabel problems, or when the classifier outputs a degree of membership rather than a proper probability, each entry P^i,j P_i,j is converted independently by thresholding at a value θ∈(0,1)θ∈(0,1): Y^i,j=(P^i,j≥θ). Y_i,j=I\! ( P_i,j≥θ ). (9) The default choice is θ=0.5θ=0.5, though θ can be tuned per label or set to produce a fixed number of predicted labels per example. In the binary case, Eq. (9) reduces to the standard score-to-decision rule, so the binary and multilabel formulations are consistent. Note that the choice of conversion method affects the resulting Y and consequently all derived performance measures. In particular, the argmax in Eq. (8) always produces exactly one predicted class per example, while thresholding in Eq. (9) may produce zero or more than m predicted labels. The threshold θ is therefore an additional degree of freedom that is part of the classifier’s decision boundary, not of the evaluation framework itself. The discussion so far has assumed that the actual label matrix Y is binary, reflecting crisp ground-truth assignments. This assumption does not always hold. In crowdsourced annotation, for instance, Yi,jY_i,j may represent the proportion of annotators who assigned label ljl_j to example eie_i [28]. In ordinal or hierarchical classification, partial membership in a class may be inherently meaningful. In knowledge distillation, soft teacher labels are used directly as targets. In all these cases, ∈[0,1]n×m Y∈[0,1]^n× m is a soft label matrix, and the same applies to Y when the classifier’s output is not thresholded. The framework extends naturally to this setting by replacing the Boolean element-wise operators with their fuzzy counterparts. The key ingredient is a triangular norm (t-norm), which generalises the Boolean conjunction to the unit interval [23, 35]. Definition 6. A t-norm is a function T:[0,1]2→[0,1]T:[0,1]^2→[0,1] satisfying, for all a,b,c∈[0,1]a,b,c∈[0,1]: 1. Commutativity: T(a,b)=T(b,a)T(a,b)=T(b,a); 2. Associativity: T(a,T(b,c))=T(T(a,b),c)T(a,T(b,c))=T(T(a,b),c); 3. Monotonicity: b≤c⇒T(a,b)≤T(a,c)b≤ c T(a,b)≤ T(a,c); 4. Boundary condition: T(a,1)=aT(a,1)=a. Intuitively, T(a,b)T(a,b) measures the degree to which both a and b hold simultaneously. The three most common t-norms are shown in Table 4, together with their standard complements and the induced notion of fuzzy disjunction (t-conorm S, defined as S(a,b)=1−T(1−a,1−b)S(a,b)=1-T(1-a,1-b)). T(a,b)T(a,b) (conjunction) S(a,b)S(a,b) (disjunction) Interpretation Gödel (min) min(a,b) (a,b) max(a,b) (a,b) Weakest overlap Product a⋅ba· b a+b−aba+b-ab Probabilistic independence Łukasiewicz max(0,a+b−1) (0,a+b-1) min(1,a+b) (1,a+b) Strongest overlap Table 4: The three principal t-norms and their induced t-conorms, all using the standard complement N(a)=1−aN(a)=1-a. Using a generic t-norm T and the standard complement N(a)=1−aN(a)=1-a, Definition 2 generalises to: i,j TP_i,j =T(Yi,j,Y^i,j) =T(Y_i,j,\; Y_i,j) i,j FP_i,j =T(N(Yi,j),Y^i,j) =T(N(Y_i,j),\; Y_i,j) i,j FN_i,j =T(Yi,j,N(Y^i,j)) =T(Y_i,j,\;N( Y_i,j)) i,j TN_i,j =T(N(Yi,j),N(Y^i,j)) =T(N(Y_i,j),\;N( Y_i,j)) Each choice of t-norm yields a different interpretation of the fuzzy confusion matrices. The Gödel (minimum) t-norm sets i,j=min(Yi,j,Y^i,j) TP_i,j= (Y_i,j, Y_i,j), which only credits the smaller of the two degrees and is conservative in the sense that even a highly confident prediction of a weakly relevant label contributes little. The product t-norm sets i,j=Yi,j⋅Y^i,j TP_i,j=Y_i,j· Y_i,j, which has a natural probabilistic interpretation: if Yi,jY_i,j is the probability that eie_i truly belongs to class cjc_j and Y^i,j Y_i,j is the predicted probability, then i,j TP_i,j is the joint probability of both events under independence. The Łukasiewicz t-norm is the most demanding: i,j=max(0,Yi,j+Y^i,j−1) TP_i,j= (0,Y_i,j+ Y_i,j-1), contributing a non-zero value only when both degrees are high enough to “overlap” above the unit threshold, which makes it suitable when only high-confidence agreement should be counted. The three choices are ordered pointwise as TL≤TP≤TMT_L≤ T_P≤ T_M for all (a,b)∈[0,1]2(a,b)∈[0,1]^2, meaning Łukasiewicz produces the smallest TP (and TN) entries and the largest FP and FN entries. Correspondingly, any performance measure in Table 3 will be most lenient under TMT_M and most strict under TLT_L. For evaluation with probabilistic ground truth (e.g., annotator proportions), the product t-norm is the most principled choice because it is consistent with the probabilistic calculus. In all three cases, when Y and Y are binary, T(a,b)T(a,b) reduces to the Boolean AND, recovering Definition 2 exactly. The three aggregation operators Σ1 _1, Σm _m and Σn _n apply unchanged to the resulting real-valued matrices, and all measures in Table 3 inherit a fuzzy interpretation without any further modification. This opens the door to evaluation protocols that are sensitive to annotator uncertainty or calibrated classifier confidence, with the t-norm as an explicit, interpretable hyperparameter of the evaluation procedure. Proper scoring rules — most notably the Brier score [1] 1nmΣ1((−^)∘2) 1nm _1(( Y- P) 2) and the log-loss −1nmΣ1(∘log^)- 1nm _1( Y P), where ∘ denotes the Hadamard product and ∘2 2 element-wise squaring — assess calibration: whether the predicted probabilities P faithfully represent the true label frequencies. These rules operate on the raw probability matrix before any thresholding or argmax, and do not factor into TP, FP, FN, TN counts because they involve the continuous difference between Y and P rather than their Boolean or fuzzy combination. Proper scoring rules [16] thus represent a complementary evaluation paradigm that is adjacent to, but outside, the current framework: where the framework measures discrimination (whether predictions correctly classify or rank examples), proper scoring rules measure calibration (whether predicted confidence levels are accurate). The two paradigms are related: under the product t-norm with soft Y and P, the quantity Σ1()=Σ1(∘^) _1( TP)= _1( Y P) measures joint degree-of-membership, which is related to but distinct from the negative Brier score (which additionally penalises overconfidence via the squared-deviation term). Practitioners requiring calibration assessment should apply a proper scoring rule to the raw P matrix as a complement to the discrimination measures provided by this framework. In multi-output (or multi-task) classification, each example is associated with T target variables, where target t takes a value in its own class set of size mtm_t. The indicator-matrix framework accommodates this setting by representing all targets jointly in a single block-structured indicator matrix. If all targets share the same number of classes m (a common case), the actual and predicted matrices are: =[(1)|(2)|⋯|(T)]∈0,1n×Tm, Y= [ Y^(1)\,|\, Y^(2)\,|\,·s\,|\, Y^(T) ]∈\0,1\^n× Tm, where each block (t)∈0,1n×m Y^(t)∈\0,1\^n× m is a one-hot matrix for target t, and Y is constructed analogously. The block structure enforces that within each block of m consecutive columns, exactly one entry per row equals 1. Predicted indicator matrices are obtained by applying argmax independently to each target’s probability block: Y^i,j(t)=(j=argmaxk∈[1,m]P^i,k(t)),t∈[1,T], Y^(t)_i,j=I\! (j= k∈[1,m] \; P^(t)_i,k ), t∈[1,T], so that mutual exclusivity is enforced per target rather than globally. When targets have different class counts mtm_t, the blocks have different widths and can be padded or handled separately. Within this structure, the three aggregation operators produce natural evaluation summaries. Column-wise aggregation Σm _m applied to the full n×Tmn× Tm matrix yields per-class-per-target counts, from which per-class or per-target measures can be extracted. A fourth aggregation level, target-wise ΣT _T, averages over the T blocks: Mtarget=1T∑t=1TM(Σ1((t)),…),M^target= 1T _t=1^TM\! ( _1( TP^(t)),\;… ), yielding the average of per-target measures. This is analogous to macro-averaging over targets, and by Theorem 1 it relates to micro-averaging over all (n×Tm)(n× Tm) entries via a target-denominator weighting. The practical implication is that multi-output evaluation inherits the same micro-macro trade-off as single-target evaluation: if some targets are easier than others, target-wise (macro) averaging treats them equally while global (micro) averaging is dominated by the targets with more class entries. Two common strategies for extending binary classifiers to multiclass problems are One-vs-All (OVA, also called One-vs-Rest) [29] and One-vs-One (OVO). Both fit naturally into the indicator-matrix framework, but in structurally different ways. In OVA, m binary classifiers are trained, one per class, each distinguishing cjc_j from all remaining classes. Each classifier produces a score P^i,j P_i,j, and the n×mn× m probability matrix P is constructed by stacking the m score vectors column-wise. The column-wise aggregation Σm _m then evaluates each binary classifier independently: applying the framework to column j yields the precision, recall, F1F_1, and other measures for the j-vs-rest problem in isolation. When argmax is applied across columns to resolve the final multiclass prediction (Eq. 8), Y becomes one-hot and Σ1 _1 gives the standard multiclass accuracy. The OVA architecture thus exposes two natural evaluation levels simultaneously: the per-classifier binary performance (via Σm _m before argmax) and the final multiclass performance (via Σ1 _1 after argmax). In OVO, (m2) m2 binary classifiers are trained, one for each pair of classes (cj,ck)(c_j,c_k). The natural indicator structure is an n×(m2)n× m2 matrix of binary predictions restricted to the relevant subsets of examples, extending the column dimension from m to (m2) m2. The aggregation operators generalise accordingly: Σm _m applied to the (m2) m2 columns evaluates each pairwise classifier independently, while a voting step aggregates the pair-level decisions into a per-class score and then resolves the final prediction. OVO evaluation can therefore be performed either at the pair level (one binary problem per pair) or at the final multiclass level (after voting), again corresponding to two choices of aggregation operator on two different matrices. A practically important consequence of this distinction is that the individual binary classifiers and the final multiclass predictor can disagree in their performance ordering across competing models: a classifier that is strong on each binary subproblem may accumulate errors on the same examples after resolution, yielding a weaker final predictor, and vice versa. The framework makes this discrepancy explicit by treating the pre-resolution and post-resolution matrices as two distinct objects, each with its own evaluation. Some classifiers may decline to make a prediction for examples they consider too uncertain, a behaviour known as rejection or abstention [4]. In the indicator-matrix framework, abstention is represented simply as an all-zero row in Y: the classifier predicts no class or label for that example. This is naturally accommodated by the multilabel encoding, which allows ∅ as a valid predicted set. An abstained example contributes fpi=tni=0fp_i=tn_i=0 and fni=Σn()i=Σn()ifn_i= _n( FN)_i= _n( Y)_i to the per-example counts, i.e., all true labels are counted as false negatives. Under row-wise aggregation Σn _n, the per-example measure for abstained examples reflects the full cost of not predicting any label, which may then be aggregated with or without abstained examples depending on the evaluation protocol. Practitioners who wish to evaluate only on non-abstained examples can achieve this by masking abstained rows, as described next. In single-class (or one-class) classification [32], a classifier is trained on examples from a single target class and must decide, at test time, whether each example belongs to that class or is an outlier. This setting maps to the framework in three complementary ways, depending on the architecture and the availability of test-set labels. In the simplest case — a single one-class classifier — the target class plays the role of c+c_+ and all outliers play the role of c−c_-, so the binary indicator vectors y and y are defined exactly as in Section 3. All binary measures apply directly: recall corresponds to the detection rate (proportion of true inliers accepted), precision to the purity of accepted examples, and FPR to the false alarm rate. Soft classifier scores are converted to decisions via Eq. (9). When m independent one-class classifiers are trained, one per class, each produces a binary accept/reject decision per example. Stacking these decisions column-wise gives a predicted indicator matrix ^∈0,1n×m Y∈\0,1\^n× m in which each classifier’s output occupies one column. Because classifiers are independent, a row may contain zero entries (all classifiers reject the example — the abstention case of Section 6), exactly one entry, or multiple entries (several classifiers accept it simultaneously). This is precisely the multilabel encoding, and the three aggregation operators yield per-class acceptance rates (Σm _m), per-example acceptance counts (Σn _n), and global totals (Σ1 _1) without further modification. The third case arises when the test set contains labeled inliers but unlabeled or unknown outliers — a common setting in novelty detection. Since the true negative label is unknown for outlier examples, their contribution to the confusion matrices is uncertain. This maps directly to the partial-label masking described next: setting Mi,j=0M_i,j=0 for examples whose true membership in class j is unknown restricts aggregation to observed entries only. Precision on accepted examples remains computable from inlier labels alone; recall requires knowledge of which examples truly belong to the target class and is therefore undefined when outlier labels are absent. Open-set recognition extends multiclass classification by allowing test examples to belong to classes unseen during training, which must be detected rather than classified into a known class. The framework accommodates this setting by augmenting the indicator matrices with an explicit unknown class column: Yi,unk=1Y_i,unk=1 for examples whose true class is not among the m known classes, and Y^i,unk=1 Y_i,unk=1 when the classifier triggers its rejection criterion. With this (m+1)(m+1)-column augmented matrix, open-set recognition reduces to a standard (m+1)(m+1)-class problem and all measures in Table 3 apply directly — including the multiclass micro-collapse result of Theorem 5, now for m+1m+1 classes. If the unknown class is not explicitly modelled in Y, open-set examples that receive no predicted class map to the abstention case (all-zero rows) described above, and can be masked out of evaluation using the partial-label mechanism of Eq. (10). In multi-instance learning (MIL) [6], each training example consists of a bag of instances, and the label is assigned at the bag level rather than the instance level. The canonical assumption is that a bag is positive if at least one of its instances is positive. The indicator-matrix framework applies directly at the bag level by treating each bag as a single example with its bag-level label vector, so all binary and multiclass measures extend without modification. Instance-level evaluation — determining which instances within a positive bag are responsible for the positive label — requires a different representation in which instances rather than bags form the rows of the indicator matrices. This introduces a two-level hierarchy (instances nested within bags) that can be handled by a two-stage application of the aggregation operators: Σn _n first within each bag (instance-level aggregation), then Σ1 _1 or Σm _m across bags. Full MIL instance-level evaluation with this two-stage scheme remains an open direction. In label ranking (also called preference learning [15]), the classifier assigns a ranking over the m labels to each example rather than a binary relevance vector. The output for example eie_i is a permutation σi:[1,m]→[1,m] _i [1,m]→[1,m], where σi(j) _i(j) gives the rank assigned to label ljl_j. This cannot be directly encoded as a binary indicator matrix: the information is relational (pairwise comparisons between labels) rather than absolute. Rank-based evaluation measures — such as Kendall’s τ, average precision, or normalised discounted cumulative gain (nDCG) — operate on these permutations and do not factor into tptp, fpfp, fnfn, tntn counts. The framework can accommodate thresholded label ranking, where labels ranked above position k are treated as relevant: Y^i,j=(σi(j)≤k) Y_i,j=I( _i(j)≤ k), which maps back to Eq. (9) with a rank-derived threshold. Full ranking evaluation represents a genuine boundary of the current framework and a natural direction for future extension, for instance by replacing the indicator matrix with a preference matrix P∈−1,0,1n×m×mP∈\-1,0,1\^n× m× m encoding pairwise label comparisons. In weakly supervised settings, not all entries of the label matrix Y are observed: an annotator may have labelled example eie_i for label ljl_j but not for lkl_k. Let ∈0,1n×m M∈\0,1\^n× m be a binary observation mask, where Mi,j=1M_i,j=1 if the true label of example eie_i for class or label j is known and 0 if it is missing. The three aggregation operators extend to partial labels by restricting sums to observed entries: Σ1() M_1( TP) =Σ1(∘), = _1( TP M), Σm() M_m( TP) =Σm(∘), = _m( TP M), Σn() M_n( TP) =Σn(∘). = _n( TP M). (10) All other operators ( FP, FN, TN) are masked analogously. The denominators of all measures in Table 3 are then adjusted to the number of observed entries rather than nmnm, ensuring that missing labels do not bias the evaluation. This formulation naturally handles the case of abstained predictions (mask out abstained rows) and semi-supervised evaluation (mask out unlabelled examples), unifying several evaluation scenarios that are typically treated separately. In ordinal classification, the m classes carry a natural ordering c1≺c2≺⋯≺cmc_1 c_2 ·s c_m, and misclassification errors should be penalised proportionally to their distance in the ordering rather than treated uniformly as in the standard 0/1 case. The framework accommodates this through two complementary extensions. The first, and most direct, extension treats ordinal classification as a special case of soft labelling. If example eie_i truly belongs to class ckic_k_i, its label matrix row is defined by an ordinal membership function f:0,…,m−1→[0,1]f:\0,…,m-1\→[0,1], where the argument is the ordinal distance to the class in question: Yi,j=f(|ki−j|).Y_i,j=f\! (|k_i-j| ). (11) The function f must satisfy f(0)=1f(0)=1 (a class is fully a member of itself) and be non-increasing (membership decreases with distance). The predicted matrix Y is constructed analogously from k^i k_i using the same function, or kept as a standard one-hot matrix if the classifier produces hard predictions. Table 5 lists common choices and their properties. f(d)f(d) Properties One-hot (d=0)I(d=0) No ordinal sensitivity; standard multiclass Linear 1−d/(m−1)1-d/(m-1) Penalises errors proportionally to distance Exponential exp(−d/σ) (-d/σ) Controlled decay; σ is a scale parameter Step (d≤δ)I(d≤δ) Tolerance window of width δ Table 5: Common ordinal membership functions for use in Eq. (11). The one-hot case recovers standard binary indicator matrices. Once Y is defined via Eq. (11), the TP, FP, FN, TN matrices are computed as before using a t-norm, and the aggregation operators and measures of Table 3 follow without modification. The combination of membership function and t-norm gives a rich two-parameter family of ordinal-sensitive evaluation protocols. The second extension uses a cumulative binary encoding, standard in ordinal regression [14, 28]. Instead of a one-hot row, example eie_i is represented by a binary row where: Yi,j=(ki≥j),j∈[1,m].Y_i,j=I(k_i≥ j), j∈[1,m]. (12) Each column j then encodes a binary subproblem: “is the true class at least cjc_j?”. The label matrix Y remains binary (no t-norm is needed), but its structure is richer than one-hot: each row is a vector of the form (1,…,1,0,…,0)(1,…,1,0,…,0), with the transition from 1 to 0 at position kik_i. Applying the aggregation operators column-wise via Σm _m yields per-threshold performance measures, while Σ1 _1 yields a global summary over all thresholds simultaneously. This is directly analogous to how AUC integrates performance across decision thresholds in the binary case, and suggests a natural ordinal generalisation of AUC within the framework. In hierarchical classification, classes are organised in a taxonomy and misclassification severity depends on the distance between the true and predicted class in that taxonomy. This setting is formalised as a special case of cost-sensitive evaluation in Section 6.1. In semantic segmentation each pixel is treated as an example and each semantic class as a label, yielding an indicator matrix with n=n= (number of pixels) and m=m= (number of classes). Under this mapping, several standard evaluation metrics are instances of the framework: per-class IoU is Jaccard applied column-wise via Σm _m; mean IoU (mIoU) [25] — the dominant benchmark metric for semantic segmentation — is macro-Jaccard; pixel accuracy is micro-accuracy (Σ1 _1); and mean pixel accuracy is macro-recall. The Dice coefficient [5] used in medical image segmentation equals F1F_1 with β=1β=1, and its per-class and mean versions are macro-F1F_1 and exemplar-F1F_1 respectively. For object detection, a predicted bounding box is accepted as a true positive when its IoU with the ground-truth box exceeds a threshold τ; this maps to Eq. (9) with IoU playing the role of P^i,j P_i,j and τ the role of θ. The resulting per-class average precision (AP) curves and their mean (mAP) [10] are then instances of the AUC formulation in Proposition 7, with τ as the threshold parameter. The three aggregation operators yield micro-AP (pooled over all classes), macro-AP (mAP, the standard), and exemplar-AP (per-image average) without further modification. 6.1 Cost-Sensitive Evaluation The indicator matrix framework extends naturally to cost-sensitive evaluation, where different types of misclassification carry different penalties. In the binary case, this is typically encoded as two scalar costs cfpc_fp and cfnc_fn. For multiclass and multilabel problems, the general object is a cost matrix ∈ℝ≥0m×m C _≥ 0^m× m, where Cj,kC_j,k denotes the cost incurred when an example whose true class is cjc_j is predicted as ckc_k. By convention, Cj,j=0C_j,j=0 for all j (no cost for correct predictions), though benefits for correct classification can also be incorporated by allowing negative entries. Given cost matrix C, the cost associated with predicting class ckc_k for example eie_i depends on its true class distribution, encoded in row i of Y. The expected cost per (example, predicted label) pair is: ()i,k=∑j=1mYi,jCj,k,( Y C)_i,k= _j=1^mY_i,j\,C_j,k, (13) where Y C is an ordinary matrix product. This produces an n×mn× m cost-weight matrix whose (i,k)(i,k) entry is the cost incurred if example eie_i is predicted as class ckc_k. The total cost over the test set is then: Cost(,^,)=Σ1(()∘^),Cost( Y, Y, C)= _1\! (( Y C) Y ), (14) which sums the cost weights only where Y^i,k=1 Y_i,k=1, i.e., where a prediction is actually made. The three aggregation operators then apply to ()∘^( Y C) Y exactly as before, yielding micro, macro, and exemplar cost-sensitive summaries. Several important special cases follow directly from Eq. (14). With m=2m=2, =(0cfncfp0) C= ( smallmatrix0&c_fn\\ c_fp&0 smallmatrix ), and one-hot Y and Y, Eq. (14) reduces to cfn⋅fn+cfp⋅fpc_fn· fn+c_fp· fp, which is the standard cost-sensitive misclassification cost. Normalised by the minimum achievable cost, this recovers the cost curve framework of Hernández-Orallo et al. [20]. If Cj,k=(j≠k)C_j,k=I(j≠ k) (unit cost for any error, zero for correct), Eq. (14) yields the standard 0/1 misclassification count, i.e., Σ1()+Σ1() _1( FP)+ _1( FN) up to a factor of 1/(m−1)1/(m-1). The standard indicator framework is therefore a degenerate case of the cost-sensitive one. Setting Cj,k=|j−k|C_j,k=|j-k| or Cj,k=(j−k)2C_j,k=(j-k)^2 yields total costs equivalent to mean absolute error (MAE) and mean squared error (MSE) of the predicted class indices, respectively. This establishes a direct link between ordinal classification losses and the cost-sensitive framework: MAE and MSE are simply instances of Eq. (14) with specific cost matrices. Combined with the ordinal membership function of Eq. (11), which softens Y, and a t-norm, which softens the conjunction, the framework provides a three-parameter family of cost-sensitive, ordinal-aware evaluation protocols. Hierarchical classification (Theorem 3) generalises the ordinal case to non-linear taxonomies. Setting Cj,k=d(cj,ck)C_j,k=d(c_j,c_k), where d is a distance in the class taxonomy (e.g., the number of edges in the class tree or the depth of the lowest common ancestor), yields a cost matrix that penalises coarse-grained errors more than fine-grained ones. For a general tree, Cost(,^,)Cost( Y, Y, C) computes the total hierarchical loss and the three aggregation operators yield micro, macro, and exemplar hierarchical summaries. Standard hierarchical precision, recall, and F1F_1 [33] are recovered without additional machinery. Any measure in Table 3 can be made cost-sensitive by replacing the raw TP, FP, FN, TN matrices with cost-weighted counterparts. Specifically, one may define: i,jC TP^C_i,j =Cj,j+⋅T(Yi,j,Y^i,j), =C^+_j,j· T(Y_i,j, Y_i,j), i,jC FP^C_i,j =(∑k≠jCk,jYi,k)⋅Y^i,j, = ( _k≠ jC_k,j\,Y_i,k )· Y_i,j, where Cj,j+C^+_j,j is a benefit for correct prediction of class j (often zero) and the FP weight sums the costs of all true classes that were incorrectly predicted as cjc_j. The resulting cost-weighted precision, recall, F-measure, and so on, are then computed by substituting these matrices into the formulas of Table 3. This provides a principled and unified route to cost-sensitive versions of a large family of evaluation measures, without requiring separate derivations for each. 6.2 Aggregation over Resampling Folds In practice, classifiers are evaluated using resampling procedures such as k-fold cross-validation, which partition the data into k train-test splits and average results over the k test folds. This introduces a fourth level of aggregation — fold-wise — alongside the three already defined. Let (f) Y^(f) and ^(f) Y^(f) denote the actual and predicted indicator matrices for fold f∈[1,k]f∈[1,k]. The fold-wise aggregation operator Σk _k combines the k per-fold values into a single summary. Two natural conventions exist: 1. Pooled evaluation concatenates all per-fold predictions before computing the measure: Mpooled=M(Σ1(⨁f(f)),…)M^pooled=M\! ( _1\! ( _f TP^(f) ),… ), where ⨁ denotes row-wise stacking of matrices. This is equivalent to treating the entire dataset as a single test set and is the correct choice when a single global estimate of performance is desired. 2. Averaged evaluation computes the measure separately on each fold and then averages: Mavg=1k∑fM(Σ1((f)),…)M^avg= 1k _fM\! ( _1( TP^(f)),… ). This estimates the expected performance of the classifier trained on a dataset of size n(k−1)/kn(k-1)/k, and is more appropriate when generalisation to new training sets is the object of interest. By Theorem 1, pooled evaluation is a weighted version of averaged evaluation with fold-wise weights equal to the fold denominators. When all folds are the same size and have the same class distribution (stratified k-fold), the two coincide. When folds differ — as is common with temporal splits or group-based cross-validation — the two can diverge substantially, for exactly the same reason that micro and macro averages diverge under class imbalance. Practitioners should therefore make explicit which convention they adopt, as the choice affects the interpretation of the reported measure. 7 Theoretical Properties The aggregation operators defined in Section 3 and their interplay with binary measures give rise to several structural results. We collect the most useful ones here. The redefinition of the different averaging methods in terms of the global, column and row wise aggregation functions allows the derivation of relations among the different measures averaging methods. Theorem 1 (Micro-averaging as weighted macro- or exemplar-averaging). Micro-averaging is equivalent to a weighted version of macro-averaging or exemplar-averaging, where the weight of each class (or example) equals the denominator of the corresponding local measure. Proof. Performance measures are a ratio between two terms. Let’s write a performance measure M generically as: M=numdenM= numden where numnum is the numerator and denden the denominator terms of the measure M. In macro or exemplar averaging, we first compute a vector M where each element is a per class (or label) or per example value of the measure, and then compute its (unweighted) average. We can also compute an weighted average ¯ M as: ¯=∑iwiMi∑iwi=∑iwinumideni∑iwi M= _iw_iM_i _iw_i= _iw_i num_iden_i _iw_i where wiw_i is the weight for MiM_i. By taking wi=deniw_i=den_i, we have ¯=∑iwinumideni∑iwi=∑iwinumiwi∑iwi=∑inumi∑ideni M= _iw_i num_iden_i _iw_i= _i w_i num_i w_i _iw_i= _inum_i _iden_i which is equivalent to the micro-average of M, Q.E.D. ∎ In other words, computing global counts and then the measure (micro-average) is equivalent to computing a weighted average of per-class or per-example measure values, where each weight equals the denominator of the corresponding local measure. This is a consequence of the fact that the micro-average pools numerators and denominators separately before dividing, while the weighted average divides locally and then re-weights by the same denominators. Corollary 1 (Micro equals unweighted macro iff denominators are balanced). Let M=num/denM=num/den be a performance measure and let denjden_j denote its denominator computed per class j via Σm _m. Then Mmicro=MmacroM^micro=M^macro if and only if all per-class denominators are equal, i.e. denj=denkden_j=den_k for all j,k∈[1,m]j,k∈[1,m]. Proof. From Theorem 1, Mmicro=∑jdenjMj/∑jdenjM^micro= _jden_jM_j/ _jden_j, which is the weighted average of the MjM_j with weights denjden_j. A weighted average equals the unweighted average Mmacro=1m∑jMjM^macro= 1m _jM_j if and only if all weights are equal. ∎ Corollary 1 makes precise the widely observed phenomenon that micro and macro averages diverge on imbalanced data. For recall, denj=tposjden_j=tpos_j (actual positives per class), so micro and macro recall coincide only when all classes have the same number of examples. For precision, denj=pposjden_j=ppos_j (predicted positives per class), so the condition depends on the classifier’s output distribution rather than the true class distribution. For FβF_β, the denominator mixes both, and the balancing condition is more complex. Theorem 2 (Partition property of the product t-norm). The product t-norm TP(a,b)=abT_P(a,b)=ab is the unique t-norm such that, for all Yi,j,Y^i,j∈[0,1]Y_i,j, Y_i,j∈[0,1], i,jTP+i,jTP+i,jTP+i,jTP=1. TP^T_P_i,j+ FP^T_P_i,j+ FN^T_P_i,j+ TN^T_P_i,j=1. Proof. Let a=Yi,ja=Y_i,j and b=Y^i,jb= Y_i,j. Under TPT_P and the standard complement N(x)=1−xN(x)=1-x: i,j+i,j+i,j+i,j TP_i,j+ FP_i,j+ FN_i,j+ TN_i,j =ab+(1−a)b+a(1−b)+(1−a)(1−b) =ab+(1-a)b+a(1-b)+(1-a)(1-b) =b[a+(1−a)]+(1−b)[a+(1−a)] =b [a+(1-a) ]+(1-b) [a+(1-a) ] =b+(1−b)=1. =b+(1-b)=1. To see uniqueness, observe that the partition property requires T(a,b)+T(1−a,b)+T(a,1−b)+T(1−a,1−b)=1T(a,b)+T(1-a,b)+T(a,1-b)+T(1-a,1-b)=1 for all a,ba,b. Setting a=b=1/2a=b=1/2 gives 4T(1/2,1/2)=14T(1/2,1/2)=1, so T(1/2,1/2)=1/4T(1/2,1/2)=1/4. The Gödel t-norm gives TM(1/2,1/2)=1/2≠1/4T_M(1/2,1/2)=1/2≠ 1/4 and the Łukasiewicz t-norm gives TL(1/2,1/2)=0≠1/4T_L(1/2,1/2)=0≠ 1/4, so neither satisfies the partition property. ∎ Theorem 2 provides a principled justification for preferring TPT_P in probabilistic settings: it is the only t-norm under which the fuzzy confusion matrices partition the unit measure, in exact analogy with how the four cells of the crisp confusion matrix partition the total count n. The Gödel and Łukasiewicz t-norms do not share this property, so measures computed from them may have denominators that differ from the expected nmnm, complicating interpretation. Theorem 3 (Ordinal losses as cost-sensitive classification). Let Y and Y be one-hot indicator matrices for a multiclass problem with ordered classes c1≺⋯≺cmc_1 ·s c_m. Then: 1. With cost matrix Cj,k=|j−k|C_j,k=|j-k|, 1nCost(,^,)=MAE 1n\,Cost( Y, Y, C)=MAE; 2. With cost matrix Cj,k=(j−k)2C_j,k=(j-k)^2, 1nCost(,^,)=MSE 1n\,Cost( Y, Y, C)=MSE. Proof. Since Y is one-hot with Yi,ki=1Y_i,k_i=1 and Y is one-hot with Y^i,k^i=1 Y_i, k_i=1, the product ()i,k=∑jYi,jCj,k=Cki,k( Y C)_i,k= _jY_i,jC_j,k=C_k_i,k. Applying Σ1 _1 to ()∘^( Y C) Y then gives ∑i∑kCki,kY^i,k=∑iCki,k^i _i _kC_k_i,k Y_i,k= _iC_k_i, k_i. Substituting Cj,k=|j−k|C_j,k=|j-k| yields ∑i|ki−k^i|=n⋅MAE _i|k_i- k_i|=n·MAE, and substituting Cj,k=(j−k)2C_j,k=(j-k)^2 yields ∑i(ki−k^i)2=n⋅MSE _i(k_i- k_i)^2=n·MSE. ∎ Theorem 3 unifies ordinal classification losses with cost-sensitive classification: MAE and MSE are not separate evaluation paradigms but instances of the general cost-sensitive framework with specific cost matrices. Combined with Eq. (11), which encodes ordinal structure in Y itself, there are therefore two orthogonal ways to introduce ordinal sensitivity—through the label matrix or through the cost matrix—that can be combined independently. Corollary 2 (Cost-sensitive micro equals weighted macro). Let MCM^C be any performance measure computed using cost-weighted confusion matrices as in Section 6.1. Then MC,microM^C,micro equals the weighted average of MC,class−jj\M^C,class-j\_j, with weights equal to the per-class denominator of MCM^C. Proof. The cost weighting modifies the numerators of the confusion matrix entries but leaves the structure of the measure as a ratio num/dennum/den unchanged. Theorem 1 applies to any such ratio regardless of how numnum is computed, so the result follows immediately. ∎ 7.1 Sensitivity to Class Skew Class imbalance is one of the most practically consequential sources of variation in evaluation outcomes. We now formalise what it means for a measure to be sensitive or insensitive to the class distribution, and use the indicator matrix framework to characterise which measures in Table 3 belong to each category and how the three aggregation operators interact with skew. Fix the conditional behaviour of a classifier, i.e., the rates rjk=P(c^k∣cj)r_jk=P( c_k c_j) at which it assigns class ckc_k to examples whose true class is cjc_j. Varying the class priors πj=P(cj) _j=P(c_j) while keeping rjkr_jk fixed changes tptp, fpfp, fnfn, tntn in a structured way: counts involving the true-positive class c+c_+ scale with π+ _+, while counts involving the true-negative class c−c_- scale with 1−π+1- _+. In matrix terms, changing πj _j rescales row j of the indicator matrix Y by a factor λj>0 _j>0, while leaving the conditional structure of Y unchanged. Definition 7 (Skew-invariant measure). A binary measure M(tp,fp,fn,tn)M(tp,fp,fn,tn) is skew-invariant if M(λtp,μfp,λfn,μtn)=M(tp,fp,fn,tn)M(λ\,tp,\;μ\,fp,\;λ\,fn,\;μ\,tn)=M(tp,fp,fn,tn) for all λ,μ>0λ,μ>0. A measure that is not skew-invariant is skew-sensitive. The λ factor scales all counts tied to the positive class (true positives and false negatives), while μ scales all counts tied to the negative class (false positives and true negatives). Skew-invariance therefore means the measure is unaffected by arbitrary rescaling of the two classes independently. Theorem 4 (Characterisation of skew-invariant measures). A measure M(tp,fp,fn,tn)M(tp,fp,fn,tn) is skew-invariant if and only if it can be written as a function of recall rec=tp/(tp+fn)rec=tp/(tp+fn) and specificity spec=tn/(tn+fp)spec=tn/(tn+fp) alone. Proof. (⇐ ) If M=f(rec,spec)M=f(rec,spec), then under the scaling (λ,μ)(λ,μ): rec′ =λtpλtp+λfn=tptp+fn=rec, = λ\,tpλ\,tp+λ\,fn= tptp+fn=rec, spec′ =μtnμtn+μfp=tntn+fp=spec. = μ\,tnμ\,tn+μ\,fp= tntn+fp=spec. Both arguments of f are invariant, so M′=f(rec′,spec′)=f(rec,spec)=M =f(rec ,spec )=f(rec,spec)=M. (⇒ ) Suppose M is skew-invariant. Setting λ=1/(tp+fn)λ=1/(tp+fn) and μ=1/(tn+fp)μ=1/(tn+fp) gives M=M(tp/(tp+fn),fp/(tn+fp),fn/(tp+fn),tn/(tn+fp))M=M (tp/(tp+fn),\;fp/(tn+fp),\;fn/(tp+fn),\;tn/(tn+fp) ). The four normalised arguments are not independent: they satisfy rec+fn/(tp+fn)=1rec+fn/(tp+fn)=1 and spec+fp/(tn+fp)=1spec+fp/(tn+fp)=1. So M depends on at most two free quantities, which can be taken to be recrec and specspec. ∎ Table 6 classifies each measure in Table 3 accordingly. Skew-invariant? Reason Recall Yes =tp/(tp+fn)=tp/(tp+fn); scales with λ only Specificity Yes =tn/(tn+fp)=tn/(tn+fp); scales with μ only Balanced accuracy Yes Average of recall and specificity G-mean Yes Geometric mean of recall and specificity Precision No tp/(tp+fp)tp/(tp+fp) mixes λ and μ FβF_β No Involves precision Jaccard No tp/(tp+fp+fn)tp/(tp+fp+fn) mixes λ and μ Accuracy No (tp+tn)/(tp+tn+fp+fn)(tp+tn)/(tp+tn+fp+fn) mixes both MCC No Involves products of mixed terms Error rate No (fp+fn)/n(fp+fn)/n mixes both Cohen’s κ No Depends on marginal distributions Table 6: Skew-invariance of each measure in Table 3. A measure is skew-invariant iff it is a function of recall and specificity only (Theorem 4). Remark 1 (Multiclass specificity is not skew-invariant). Theorem 4 is stated for the binary case, where the positive and negative classes are fixed and their conditional rates are independent of class priors. In multiclass settings with one-vs-rest decomposition, the situation is more subtle: the false-positive count for class j is fpj=∑k≠jRkjπkfp_j= _k≠ jR_kj\, _k, which depends on the prevalences of all other classes. Consequently, per-class specificity tnj/(tnj+fpj)tn_j/(tn_j+fp_j) is not invariant to class priors in multiclass, even when conditional rates RkjR_kj are fixed. This means that balanced accuracy and G-mean, while skew-invariant in binary classification, exhibit residual skew sensitivity in multiclass settings via the one-vs-rest specificity term. Figure 1 confirms this: macro-recall is exactly flat, while macro-balanced accuracy drifts slightly as π1 _1 increases. Corollary 3 (Micro-averaging reintroduces skew sensitivity). Let M be a skew-invariant measure. Then MmacroM^macro and MexemplarM^exemplar are skew-invariant, but MmicroM^micro is skew-sensitive whenever the class distribution is non-uniform. Proof. Since M is skew-invariant, each per-class value MjM_j depends only on the row rates rjjr_j (recall for class j) and the complementary rate (specificity for class j), neither of which involves class priors. Macro-averaging Mmacro=1m∑jMjM^macro= 1m _jM_j and exemplar-averaging are therefore independent of class priors. For micro-averaging, Theorem 1 gives Mmicro=∑jwjMj/∑jwjM^micro= _jw_jM_j/ _jw_j where wj=denjw_j=den_j. For recall, denj=tposj=πj⋅nden_j=tpos_j= _j· n, so wj∝πjw_j _j. When the πj _j are non-uniform, the weights differ, and the weighted average differs from the unweighted one (by Corollary 1), making MmicroM^micro depend on the πj _j. ∎ Corollary 3 has an important practical implication: practitioners who require skew-invariant evaluation should use macro-averaging (or exemplar-averaging in multilabel settings) rather than micro-averaging, even when the per-class measure itself is skew-invariant. Micro-averaging is equivalent to prevalence-weighted macro-averaging, and thus systematically rewards performance on majority classes. Corollary 4 (Gap between micro and macro as a function of skew). For any measure M and class distribution =(π1,…,πm) π=( _1,…, _m), Mmicro−Mmacro=Cov(,),M^micro-M^macro=Cov_ π( w, M), where CovCov_ π denotes the covariance under the distribution π and =(M1,…,Mm) M=(M_1,…,M_m) are the per-class measure values. Proof. Let w¯=1m∑jwj w= 1m _jw_j and M¯=Mmacro M=M^macro. Then: Mmicro=∑jwjMj∑jwj=m∑jwj⋅1m∑jwjMj=M¯+∑j(wj−w¯)(Mj−M¯)∑jwj,M^micro= _jw_jM_j _jw_j= m _jw_j· 1m _jw_jM_j= M+ _j(w_j- w)(M_j- M) _jw_j, where the last term is Cov(,)Cov_ π( w, M) under the normalised weight distribution. ∎ Corollary 4 gives an interpretable decomposition: micro and macro averages agree only when class weights and per-class performance are uncorrelated. If a classifier performs better on larger classes (a common outcome when training data is imbalanced), the covariance is positive and micro exceeds macro. If it performs better on smaller classes (e.g., after oversampling), the gap reverses. The magnitude of the gap is thus a diagnostic for the interaction between class imbalance and classifier behaviour. 7.2 Consistency and the Aggregation Operators A measure M is consistent with a loss L if the classifier that minimises the expected loss [L]E[L] also maximises M [20]. Consistency is a desirable property because it guarantees that optimising M during model selection or threshold tuning will not lead to a classifier that is suboptimal under L. We say that MmicroM^micro is ERM-consistent222Empirical Risk Minimisation (ERM) is the standard training paradigm: find f^=argminf1n∑iL(yi,f^(xi)) f= _f 1n _iL(y_i, f(x_i)) for some loss L. A measure is ERM-consistent if the classifier that minimises L in expectation also maximises the measure, so that standard gradient-based training automatically produces a classifier that is optimal under the measure without any special objective engineering. when it is consistent with the per-label 0/1 loss minimised by standard ERM training. The choice of aggregation operator has direct consequences for this property. Proposition 1 (Fixed-denominator micro-measures are ERM-consistent). Let M(tp,fp,fn,tn)=num/denM(tp,fp,fn,tn)=num/den be a binary measure whose denominator denden depends only on the data distribution (not on the classifier’s predictions). Then MmicroM^micro is maximised by the same classifier that minimises the per-label expected 0/1 loss [(Yij≠Y^ij)]E[I(Y_ij≠ Y_ij)]. Proof. Since den=Σ1(+)den= _1( TP+ FN) for recall, or Σ1(+) _1( TN+ FP) for specificity, denden is a fixed constant determined by the class distribution and not by Y Y. Maximising Mmicro=Σ1()/denM^micro= _1( TP)/den is therefore equivalent to maximising Σ1() _1( TP), i.e., minimising Σ1()+Σ1() _1( FP)+ _1( FN), which is proportional to the per-label 0/1 loss summed over the test set. ∎ Proposition 1 covers recall and specificity. For variable-denominator measures such as precision and FβF_β (whose denominator ppos=Σ1(^)ppos= _1( Y) depends on the classifier), the argument breaks down: a classifier can inflate its micro-precision by predicting fewer positives, creating a trade-off that is not captured by 0/1 loss. The standard resolution is to jointly optimise precision and recall via the FβF_β measure, but even this is not consistent with 0/1 loss in general. The situation is more delicate for macro-averaged measures. Because Σm _m weights classes equally regardless of their prevalence, the Bayes-optimal classifier under macro-F1F_1 is generally not the MAP classifier, and requires class-dependent threshold adjustments [24, 20]. Corollary 3 gives the formal reason: macro-averaging is equivalent to a uniform class-weighted micro-average, which corresponds to a reweighted 0/1 loss where minority classes receive higher weight. Optimising this reweighted loss requires either reweighted training or post-hoc threshold tuning per class. Exemplar-averaging (Σn _n) occupies an intermediate position: it weights examples equally but computes the measure per example, which for multilabel problems corresponds to averaging instance-level losses. For measures such as recall and F1F_1, exemplar-averaging is consistent with the corresponding per-instance loss, making it the natural evaluation choice in multilabel settings where each example’s label subset should be treated as a unit. In summary, the aggregation operator determines the implicit loss that the measure is consistent with: 1. Σ1 _1 (micro) ↔ uniform per-label 0/1 loss, consistent with standard ERM; 2. Σm _m (macro) ↔ class-reweighted loss, requiring specialised optimisation; 3. Σn _n (exemplar) ↔ per-instance loss, natural for multilabel evaluation. This correspondence provides a principled guide for choosing an aggregation operator based on the learning objective rather than on convention. 7.3 Preservation of Desirable Properties A natural concern when extending binary measures “for free” via the framework is whether the extensions inherit the desirable properties of their binary originals. We formalise two such properties and show they are preserved under all three aggregation operators. Definition 8 (Boundary correctness). A binary measure M(tp,fp,fn,tn)M(tp,fp,fn,tn) is boundary correct if M attains its theoretical maximum when all predictions are correct (fp=fn=0fp=fn=0) and its theoretical minimum when no prediction is correct (tp=tn=0tp=tn=0). Definition 9 (Monotonicity). A binary measure M(tp,fp,fn,tn)M(tp,fp,fn,tn) is monotone if it is non-decreasing in tptp and tntn, and non-increasing in fpfp and fnfn, when all other arguments are held fixed. Proposition 2 (Boundary correctness is preserved). Let M be boundary correct. Then MmicroM^micro, MmacroM^macro, and MexemplarM^exemplar are all boundary correct: they attain their maximum when = Y= Y and their minimum when == TP= TN=0. Proof. When = Y= Y, every element satisfies Yi,j=Y^i,jY_i,j= Y_i,j, so == FP= FN=0 and TP, TN are maximised element-wise. Under Σ1 _1 this gives fp=fn=0fp=fn=0, which by boundary correctness of M maximises MmicroM^micro. The same argument holds column-wise and row-wise, so MjM_j and MiM_i are maximised for every class j and every example i, hence their averages MmacroM^macro and MexemplarM^exemplar are maximised. The minimum case follows analogously. ∎ Proposition 3 (Monotonicity is preserved). Let M be monotone. Then MmicroM^micro, MmacroM^macro, and MexemplarM^exemplar are monotone: correcting any single prediction (changing a false positive or false negative to a true positive or true negative in Y) does not decrease any of the three extended measures. Proof. Correcting a single prediction at position (i∗,j∗)(i^*,j^*) decreases i∗,j∗ FP_i^*,j^* or i∗,j∗ FN_i^*,j^* by 1 and increases i∗,j∗ TP_i^*,j^* or i∗,j∗ TN_i^*,j^* by 1, leaving all other entries unchanged. Under Σ1 _1, the global counts tptp and tntn increase and fpfp or fnfn decreases, so MmicroM^micro does not decrease by monotonicity of M. Under Σm _m, only the counts for class j∗j^* change in the same direction, so Mj∗M_j^* does not decrease, and all other MjM_j are unchanged; therefore MmacroM^macro does not decrease. Under Σn _n, only Mi∗M_i^* changes, and by the same argument it does not decrease; hence MexemplarM^exemplar does not decrease. ∎ Both properties hold for all binary measures in Table 2 (precision, recall, F1F_1, balanced accuracy, G-mean, Jaccard, MCC), and Propositions 2 and 3 guarantee that every measure generated by the framework from a well-behaved binary measure inherits both properties simultaneously. This contrasts with ad-hoc measure extensions, which must verify these properties case by case [31]. 7.4 Redundancy of Measures The indicator-matrix framework makes it straightforward to identify when two measures carry identical information — either because they are algebraic functions of each other, or because they always produce the same ranking of classifiers. Two measures are algebraically redundant if one is a deterministic function of the other for all possible confusion matrices. Theorem 5 (Micro-averaging collapse in multiclass). In a multiclass problem with one-hot indicator matrices Y and Y, Mprecisionmicro=Mrecallmicro=MF1micro=accuracy.M^micro_precision\;=\;M^micro_recall\;=\;M^micro_F_1\;=\;accuracy. Proof. Let C=Σ1()C= _1( TP) denote the number of correctly classified examples. Since each row of Y has exactly one entry equal to 1 (one-hot), Σ1()=n _1( Y)=n. Similarly Σ1(^)=n _1( Y)=n. Each incorrectly classified example contributes exactly one false positive (the predicted class) and one false negative (the true class), so Σ1()=Σ1()=n−C _1( FP)= _1( FN)=n-C. Therefore: Mprecisionmicro M^micro_precision =C+(n−C)=Cn, = CC+(n-C)= Cn, Mrecallmicro M^micro_recall =C+(n−C)=Cn, = CC+(n-C)= Cn, MF1micro M^micro_F_1 =2C2C+(n−C)+(n−C)=Cn, = 2C2C+(n-C)+(n-C)= Cn, all of which equal the standard multiclass accuracy C/nC/n. ∎ Theorem 5 has a direct practical implication: in multiclass settings, reporting micro-precision, micro-recall, and micro-F1F_1 alongside accuracy is entirely redundant — all four numbers are identical. Their apparent differences in the literature arise solely from multilabel settings, where the one-hot constraint is lifted. Corollary 5 (Micro-Jaccard in multiclass). In a multiclass problem with one-hot indicator matrices, MJaccardmicro=C2n−C.M^micro_Jaccard= C2n-C. Consequently, MJaccardmicroM^micro_Jaccard is a monotone increasing function of accuracy and carries no additional information. Proof. Σ1()=Σ1()=n−C _1( FP)= _1( FN)=n-C, so micro-Jaccard =C/(C+(n−C)+(n−C))=C/(2n−C)=C/(C+(n-C)+(n-C))=C/(2n-C), which is strictly increasing in C/nC/n. ∎ The framework also makes the following exact algebraic redundancy immediate. Proposition 4 (Hamming loss and label accuracy). For any indicator matrices ,^∈0,1n×m Y, Y∈\0,1\^n× m, HammingLoss+LabelAccuracy=1.HammingLoss+LabelAccuracy=1. Proof. HammingLoss =Σ1(⊕^)/(nm)= _1( Y Y)/(nm) and LabelAccuracy =(Σ1()+Σ1())/(nm)=( _1( TP)+ _1( TN))/(nm). Since every (i,j)(i,j) entry satisfies either Yij=Y^ijY_ij= Y_ij (contributing to accuracy) or Yij≠Y^ijY_ij≠ Y_ij (contributing to Hamming loss), the two quantities partition the nmnm entries, giving their sum equal to 1. ∎ A more general redundancy result follows from the structure of the confusion matrix. Proposition 5 (Degrees-of-freedom bound). For an m-class problem with fixed class counts n1,…,nmn_1,…,n_m (row sums), the confusion matrix has exactly m(m−1)m(m-1) degrees of freedom. Consequently, any collection of more than m(m−1)m(m-1) scalar measures computed from the confusion matrix is necessarily algebraically redundant. Proof. The confusion matrix ∈ℤ≥0m×m W _≥ 0^m× m has m2m^2 entries. The constraint ∑kWjk=nj _kW_jk=n_j fixes each row sum, removing m degrees of freedom. The remaining m2−m=m(m−1)m^2-m=m(m-1) entries are free (subject to non-negativity). Any scalar measure is a function of W, so a set of more than m(m−1)m(m-1) measures must have at least one that is determined by the others. ∎ For binary classification (m=2m=2), this gives 2 degrees of freedom: any two independent measures — e.g., recall and specificity — fully determine all other binary measures. For m=3m=3, six independent measures suffice; for m=4m=4, twelve. This provides a principled upper bound on the number of non-redundant measures a practitioner should report. Two measures are ranking-equivalent if they always produce the same ordering of classifiers, i.e., one is a strictly monotone transformation of the other on the relevant domain. Proposition 6 (Accuracy and error rate are ranking-equivalent). For any two classifiers evaluated on the same test set, accuracy≥1_1≥ accuracy2 if and only if error rate≤1_1≤ error rate2. Proof. Error rate =1−=1- accuracy, so the ranking is reversed. The two measures are thus ranking-equivalent up to direction. ∎ A less obvious case is Cohen’s κ and accuracy. Since κ=(acc−pe)/(1−pe)κ=(acc-p_e)/(1-p_e) and pep_e depends on the predicted class distribution (which varies across classifiers), κ is not in general a monotone function of accuracy: two classifiers with the same accuracy may differ in κ if their predicted class distributions differ. Ranking equivalence between κ and accuracy therefore holds only conditional on the predicted class distribution being fixed, a condition that is rarely satisfied in practice. Within the indicator-matrix framework, this is seen directly: pe=1n2∑j[Σm()]j⋅[Σm(^)]jp_e= 1n^2 _j[ _m( Y)]_j·[ _m( Y)]_j, which depends on the column sums of Y and therefore on the classifier. Proposition 7 (ROC curve and AUC). For a binary classifier with score output and threshold θ∈(0,1)θ∈(0,1), let ^(θ) Y(θ) denote the indicator vector obtained from Eq. (9). The ROC curve is the parametric curve: FPR(θ) (θ) =Σ1((θ))Σ1((θ))+Σ1((θ)), = _1( FP(θ)) _1( FP(θ))+ _1( TN(θ)), TPR(θ) (θ) =Σ1((θ))Σ1((θ))+Σ1((θ)), = _1( TP(θ)) _1( TP(θ))+ _1( FN(θ)), and ROC(θ)=(FPR(θ),TPR(θ))ROC(θ)= (FPR(θ),\;TPR(θ) ), and the area under the ROC curve (AUC) is ∫01TPR(θ)dFPR(θ) _0^1TPR(θ)\,d\,FPR(θ). In multiclass settings under OVA, column-wise aggregation Σm _m yields one ROC curve per class, and the unweighted average of per-class AUC values gives macro-AUC [19]. Proposition 7 shows that the entire ROC apparatus is a consequence of varying the threshold θ in Eq. (9) and applying Σ1 _1 at each value. All points on the ROC curve correspond to a particular choice of threshold, and choosing a fixed θ (e.g., θ=0.5θ=0.5) picks a single point on the curve — the operating point used for computing precision, recall, and F1F_1 in Table 3. AUC integrates the performance over all operating points [11, 13] and is therefore threshold-independent, unlike the measures in Table 3, which are all threshold-specific. Macro-AUC under OVA is also an instance of Σm _m — the same operator used for macro-precision and macro-recall — applied to the AUC formula rather than a fixed-threshold measure. Proposition 8 (Ordinal AUC as column-wise aggregation). Let ^∈[0,1]n×m P∈[0,1]^n× m be a probabilistic classifier’s output and define, for each threshold j∈[1,m−1]j∈[1,m-1], a binary classification problem with labels i(j)=(ki≥j)y^(j)_i=I(k_i≥ j) and scores p^i(j)=P^i,j p^(j)_i= P_i,j. Let AUC(j)AUC^(j) denote the area under the ROC curve for threshold j. Then the ordinal AUC, AUCord=1m−1∑j=1m−1AUC(j),AUC^ord= 1m-1 _j=1^m-1AUC^(j), is the unweighted average of per-column AUCs induced by the cumulative binary encoding of Eq. (12), and is therefore an instance of Σm _m applied to the binary AUC formula. Proposition 8 shows that ordinal AUC arises naturally within the framework via the column-wise aggregation operator, paralleling the way standard AUC integrates performance over decision thresholds in binary classification. The (m−1)(m-1) normalisation excludes the degenerate threshold j=mj=m (which would yield all-positive labels), consistent with standard practice. 7.5 Dominance Relations Classifier A dominates classifier B under measure M if M(A)≥M(B)M(A)≥ M(B) for all possible test sets drawn from the same distribution. A stronger notion is measure-wise dominance: A dominates B if M(A)≥M(B)M(A)≥ M(B) for every measure M in Table 3 simultaneously. By Proposition 5, all scalar measures in Table 3 are functions of the same m(m−1)m(m-1) confusion matrix entries, so measure-wise dominance reduces to element-wise dominance of the (normalised) confusion matrix: A dominates B if and only if A’s confusion matrix assigns at least as much probability to correct predictions and at most as much to each type of error as B’s, for every cell simultaneously. In binary classification (m=2m=2, two degrees of freedom), this reduces to dominance in the (recall, specificity) space, which is exactly ROC dominance: A dominates B if and only if A’s ROC curve lies pointwise above B’s (Proposition 7). The result is clean because two degrees of freedom span a two-dimensional space in which a total order on the Pareto frontier is easy to characterise. For m>2m>2, the dominance relation becomes m(m−1)m(m-1)-dimensional and Pareto-optimal classifiers form a higher-dimensional surface. In this case there typically exists no single classifier that dominates all others across all measures simultaneously, which is the formal justification for reporting multiple complementary measures rather than a single summary statistic: different measures emphasise different cells of the confusion matrix, and the choice among them should reflect the cost structure of the application — which the framework encodes explicitly via the cost matrix C of Section 6.1. 8 Empirical Illustrations We present three synthetic illustrations designed to verify the theoretical results of Section 7 and demonstrate the practical implications of the framework. All experiments are fully reproducible from the indicator-matrix computations described in Section 3; no external datasets are required. Code is available in the supplementary material. 8.1 Micro vs. Macro under Class Skew We construct a 4-class classification problem with fixed conditional rates (confusion-matrix rows RjR_j), i.e., the probability that a classifier assigns class ckc_k to an example of true class cjc_j is held constant across all trials. We then vary the prevalence π1 _1 of the majority class from 0.250.25 (uniform) to 0.700.70 (strongly skewed), distributing the remaining probability equally among the three minority classes. All quantities are computed analytically from the rates, eliminating sampling noise. Figure 1 shows micro and macro averages for four measures. Recall and balanced accuracy confirm Theorem 4 and Corollary 3: macro-recall and macro-balanced accuracy are perfectly flat (skew-invariant, since both are functions of recall and specificity alone), while their micro counterparts drift steadily as π1 _1 increases. F1F_1 and MCC are skew-sensitive under both aggregation schemes, though the magnitude of drift differs substantially between micro and macro. Figure 1: Micro vs. macro averages as a function of majority-class prevalence π1 _1. Conditional rates are held fixed throughout. Recall and balanced accuracy are flat under macro (skew-invariant; Theorem 4) but drift under micro (Corollary 3). F1F_1 and MCC are skew-sensitive under both schemes. 8.2 Micro-Macro Gap Equals the Covariance To verify Corollary 4 empirically, we generate 1,200 random 5-class classifiers by drawing class priors π from a symmetric Dirichlet distribution and confusion-matrix rates from a Dirichlet with concentration 3 (encouraging moderately good classifiers). For each trial we compute, analytically, both Mmicro−MmacroM^micro-M^macro and Cov(,)Cov(w,M) using the measure-specific denominator weights: wj∝tposjw_j tpos_j for recall and wj∝2tpj+fpj+fnjw_j 2tp_j+fp_j+fn_j for F1F_1. Figure 2 shows that all points lie on the y=xy=x diagonal to numerical precision, confirming the identity for both measures. The range of the gap is wider for recall (±0.15± 0.15) than for F1F_1 (±0.05± 0.05 for typical classifiers), reflecting the fact that F1F_1’s denominator weights partially absorb skew variation. Crucially, points appear on both sides of zero: when the classifier performs best on the majority class (positive covariance), micro exceeds macro; when it performs best on minority classes (negative covariance), micro falls below macro. Figure 2: Micro−-macro gap against Cov(,)Cov(w,M) for 1,200 random 5-class classifiers. All points lie on y=xy=x (dashed), verifying Corollary 4. Weights are measure-specific: wj∝tposjw_j tpos_j for recall and wj∝2tpj+fpj+fnjw_j 2tp_j+fp_j+fn_j for F1F_1. 8.3 T-norm Ordering under Soft Ground Truth We construct a 3-class problem where both the ground-truth matrix Y and the predicted matrix Y are real-valued. The true labels are soft, generated from a Dirichlet distribution centred at the true class with concentration parameter α controlling label certainty: small α produces near-uniform rows (high annotator uncertainty) and large α produces near-one-hot rows (confident annotation). Predicted probabilities are generated by a fixed noisy classifier whose outputs are also smoothed via a Dirichlet with a peak at the predicted class. Figure 3 reports recall, precision, and mean TP entry across the three t-norms. The rightmost panel directly confirms the pointwise ordering TL≤TP≤TMT_L≤ T_P≤ T_M for the raw TP values (Theorem 2): Łukasiewicz is the most conservative, Gödel the most lenient, and product lies between. The derived measures (recall, precision) do not always preserve this ordering, because both the numerator and the denominator of a ratio change with the t-norm. This illustrates the nuance discussed in Section 6: the t-norm ordering is a property of the confusion matrices themselves, not an automatic guarantee for every derived measure. Figure 3: Effect of t-norm choice on recall, precision, and mean TP under soft ground truth (α controls annotation certainty). The TP ordering TL≤TP≤TMT_L≤ T_P≤ T_M is exact (right panel), while derived measures need not preserve it (left and centre panels). 8.4 Real Multilabel Data: Aggregation Scheme Determines Classifier Selection The synthetic experiments above verify the theoretical results analytically. We complement them with a real multilabel experiment whose purpose is different: to show that the choice of aggregation scheme has direct practical consequences — it can change which classifier a practitioner selects. We use the yeast dataset from the MULAN repository [34, 8], a standard multilabel benchmark with 2,417 examples, 103 features, and 14 labels. Label prevalences range from 0.014 to 0.751, making it substantially imbalanced. We evaluate four multilabel classifiers based on the Binary Relevance (BR) decomposition with different base learners: logistic regression (BR-LR), support vector machines (BR-SVM), random forests (BR-RF), and k-nearest neighbours (BR-kNN). All measures are computed via 10-fold cross-validation using the indicator-matrix framework; the three aggregation operators are applied to the same predicted matrices, so differences in rankings arise solely from the aggregation choice, not from different training procedures. Figure 4(a) shows the classifier rankings under ten measure–aggregation combinations. The most striking result is the complete inversion between macro-F1F_1 and exemplar-F1F_1: BR-kNN ranks first under macro-F1F_1 (0.3910.391) but last under exemplar-F1F_1 (0.5980.598), while BR-SVM ranks last under macro-F1F_1 (0.3560.356) but first under exemplar-F1F_1 (0.6240.624). Kendall’s τ between these two rankings is −1.000-1.000 — the maximum possible disagreement. Micro-F1F_1 agrees with the exemplar measures (BR-SVM first, BR-kNN last), not with macro-F1F_1. This result follows directly from the theory. Yeast has one label with prevalence 0.0140.014: macro-F1F_1, which weights all labels equally regardless of prevalence, gives this rare label the same influence as the most common one (prevalence 0.7510.751). A classifier that performs well on rare labels — even at the cost of worse average-example coverage — will rank high under macro-F1F_1. BR-kNN exhibits this behaviour. Exemplar-F1F_1, by contrast, weights each example equally and evaluates how completely each example’s label set is predicted; it is the aggregation scheme consistent with per-instance loss (Proposition 1), the theoretically appropriate choice when the downstream task requires complete label assignment per example. BR-SVM performs best on this criterion. Figure 4(b) shows the full matrix of Kendall’s τ values between all pairs of measures. Several patterns are immediately visible. First, the τ=−1.00τ=-1.00 cell between macro-F1F_1 (M-F1) and exemplar-F1F_1 (ex-F1) stands out as the only case of maximum disagreement. Second, micro-F1F_1 (μ-F1) achieves τ=1.00τ=1.00 with both exemplar-MCC (ex-MCC) and exemplar-BA (ex-BA), despite these belonging to different measure families — an empirical illustration of the redundancy results of Section 7.4. Third, macro-BA (M-BA) achieves τ=1.00τ=1.00 with subset accuracy, a non-obvious agreement explained by the fact that both measures give high weight to complete-label correctness on the dominant labels. Fourth, macro-F1F_1 has τ≤−0.33τ≤-0.33 with all micro and exemplar measures, confirming that the choice of macro averaging on this imbalanced dataset systematically inverts the conclusions drawn from any other aggregation scheme. (a) Classifier rankings on the yeast multilabel dataset (14 labels, prevalences 0.014–0.751) under ten measure–aggregation combinations. Green = rank 1 (best), red = rank 4 (worst). ★ marks measures that are theoretically consistent with per-instance loss (Proposition 1). Note the complete rank reversal between macro-F1F_1 and exemplar-F1F_1: the classifiers ranked first and last are swapped. (b) Kendall’s τ between all pairs of measures on yeast. Red = perfect agreement (τ=1τ=1), blue = perfect disagreement (τ=−1τ=-1). The τ=−1.00τ=-1.00 between macro-F1F_1 (M-F1) and exemplar-F1F_1 (ex-F1) is the result of central interest. Micro measures (μ-F1, μ-MCC) agree with exemplar measures but disagree with macro-F1F_1. 9 Conclusion We have presented a unified algebraic framework for classification performance evaluation, grounded in binary indicator matrices and three aggregation operators. The framework’s key practical contribution is that it acts as a measure generator: any binary measure expressible in terms of tptp, fpfp, fnfn, tntn yields micro, macro/weighted, and exemplar multiclass and multilabel versions automatically, with no measure-specific derivation required. Table 3 instantiates this for eleven common measures, identifying novel versions where no standard named measure previously existed. Beyond this generative property, the framework unifies several settings that are typically treated independently. Soft classifier outputs are accommodated via argmax (multiclass) or thresholding (multilabel). Soft and probabilistic ground truth is handled via triangular norms, with the product t-norm singled out by its unique partition property (Theorem 2). Ordinal classification connects to the framework through ordinal membership functions and cumulative binary encodings, with ordinal AUC emerging as column-wise aggregation of binary AUC (Proposition 8). Cost-sensitive evaluation enters via a cost matrix acting as a bilinear form on the indicator matrices, with MAE and MSE recovered as algebraic special cases (Theorem 3). The theoretical results establish structural properties that hold across all measures and settings simultaneously. The micro-equals-weighted-macro theorem (Theorem 1) makes precise why micro and macro averages diverge, and Corollary 4 identifies the divergence exactly as the covariance between class weights and per-class performance. The skew-invariance characterisation (Theorem 4) shows that only measures expressible as functions of recall and specificity are immune to class imbalance, and Corollary 3 shows that micro-averaging reintroduces skew sensitivity even for such measures. The redundancy results (Section 7.4) reveal that micro-precision, micro-recall, and micro-F1F_1 are all equal to accuracy in multiclass settings — a collapse that is invisible without the matrix perspective — and that the confusion matrix has at most m(m−1)m(m-1) degrees of freedom, bounding the number of non-redundant measures that can usefully be reported. Several directions remain open. First, the framework currently operates at the level of a fixed test set; extending it to cover resampling procedures such as cross-validation, where results are averaged over test partitions, would be straightforward but has not been formalised here. Second, the redundancy analysis in Section 7.4 identifies algebraic and ranking redundancy but does not address statistical redundancy — whether two measures carry the same information in expectation across classifiers and datasets, a question that requires an empirical or information-theoretic treatment. Third, the soft-label extension raises the question of which t-norm is most appropriate for a given application; while the product t-norm is justified by the partition property, the choice in practice may depend on the downstream task and the nature of the label uncertainty. Fourth, the framework could be extended to structured output prediction (sequences, trees, graphs), where the indicator matrix would need to be replaced by a more complex structure. Finally, the relationship between the measure framework proposed here and the decision-theoretic framework of Hernández-Orallo et al. [20], which unifies measures via threshold choice and expected loss, deserves a more detailed comparative analysis. Funding This research was partially supported by the Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq) and the Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP), Brazil. CRediT Author Statement Ronaldo C. Prati: Conceptualization, Methodology, Formal Analysis, Software, Validation, Writing – Original Draft, Writing – Review & Editing. Declaration of Generative AI and AI-assisted Technologies During the preparation of this work the author used Claude (Anthropic) in order to assist with literature search, drafting and revising manuscript text, implementing experiment code, and checking LaTeX syntax. After using this tool, the author reviewed and edited all content as needed and takes full responsibility for the content of the published article. References [1] G. W. Brier (1950) Verification of forecasts expressed in terms of probability. Mon. Weather Rev. 78 (1), p. 1–3. Cited by: §6. [2] M. Carbonero-Ruz, F. J. Martínez-Estudillo, F. Fernández-Navarro, D. Becerra-Alonso, and A. C. Martínez-Estudillo (2017) A two dimensional accuracy-based measure for classification performance. Inf. Sci. 382-383, p. 60–80. External Links: Document Cited by: §1, §2. [3] J. Cohen (1960) A coefficient of agreement for nominal scales. Educ. Psychol. Meas. 20 (1), p. 37–46. Cited by: Table 2, §3, §5.1. [4] F. Condessa, J. M. Bioucas-Dias, and J. Kovacevic (2017) Performance measures for classification systems with rejection. Pattern Recognit. 63, p. 437–450. External Links: Document Cited by: §1, §6. [5] L. R. Dice (1945) Measures of the amount of ecologic association between species. Ecology 26 (3), p. 297–302. Cited by: §6. [6] T. G. Dietterich, R. H. Lathrop, and T. Lozano-Pérez (1997) Solving the multiple instance problem with axis-parallel rectangles. Artif. Intell. 89 (1–2), p. 31–71. Cited by: §6. [7] C. Drummond and R. C. Holte (2006) Cost curves: an improved method for visualizing classifier performance. Mach. Learn. 65 (1), p. 95–130. Cited by: §2. [8] A. Elisseeff and J. Weston (2001) A kernel method for multi-labelled classification. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 14, Cambridge, MA. Cited by: §8.4. [9] C. Elkan (2001) The foundations of cost-sensitive learning. In Proceedings of the 17th International Joint Conference on Artificial Intelligence (IJCAI), Seattle, WA, p. 973–978. Cited by: §2. [10] M. Everingham, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman (2010) The PASCAL visual object classes (VOC) challenge. Int. J. Comput. Vis. 88 (2), p. 303–338. External Links: Document Cited by: §6. [11] T. Fawcett (2006) An introduction to ROC analysis. Pattern Recognit. Lett. 27 (8), p. 861–874. External Links: Document Cited by: §7.4. [12] C. Ferri, J. Hernández-Orallo, and R. Modroiu (2009) An experimental comparison of performance measures for classification. Pattern Recognit. Lett. 30 (1), p. 27–38. External Links: Document Cited by: §1, §2. [13] P. A. Flach (2003) The geometry of ROC space: understanding machine learning metrics through ROC isometrics. In Proceedings of the 20th International Conference on Machine Learning (ICML), Washington, DC, p. 194–201. Cited by: §7.4. [14] E. Frank and M. Hall (2001) A simple approach to ordinal classification. In Proceedings of the 12th European Conference on Machine Learning (ECML), Lecture Notes in Computer Science, Vol. 2167, Freiburg, Germany, p. 145–156. Cited by: §6. [15] J. Fürnkranz and E. Hüllermeier (Eds.) (2010) Preference learning. Springer, Berlin, Heidelberg. External Links: Document Cited by: §6. [16] T. Gneiting and A. E. Raftery (2007) Strictly proper scoring rules, prediction, and estimation. J. Amer. Statist. Assoc. 102 (477), p. 359–378. Cited by: §6. [17] S. Godbole and S. Sarawagi (2004) Discriminative methods for multi-labeled classification. In Advances in Knowledge Discovery and Data Mining (PAKDD), LNCS, Vol. 3056, Sydney, Australia, p. 22–30. Cited by: §2. [18] J. Gorodkin (2004) Comparing two K-category assignments by a K-category correlation coefficient. Comput. Biol. Chem. 28 (5–6), p. 367–374. External Links: Document Cited by: §5.1. [19] D. J. Hand and R. J. Till (2001) A simple generalisation of the area under the ROC curve for multiple class problems. Mach. Learn. 45 (2), p. 171–186. Cited by: Proposition 7. [20] J. Hernández-Orallo, P. A. Flach, and C. Ferri (2012) A unified view of performance metrics: translating threshold choice into expected classification loss. J. Mach. Learn. Res. 13, p. 2813–2869. Cited by: §1, §2, §2, §5.1, §5.2, §6.1, §7.2, §7.2, §9. [21] N. Japkowicz and M. Shah (2011) Evaluating learning algorithms: A classification perspective. Cambridge University Press, Cambridge. External Links: ISBN 9780521196000 Cited by: §1. [22] T. Kautz, B. M. Eskofier, and C. F. Pasluosta (2017) Generic performance measure for multiclass-classifiers. Pattern Recognit. 68, p. 111–125. Cited by: §1, §2. [23] E. P. Klement, R. Mesiar, and E. Pap (2000) Triangular norms. Kluwer Academic Publishers, Dordrecht. Cited by: §6. [24] O. Koyejo, N. Natarajan, P. Ravikumar, and I. S. Dhillon (2014) Consistent multilabel classification. In Advances in Neural Information Processing Systems (NeurIPS), Vol. 27, Red Hook, NY, p. 3301–3309. Cited by: §7.2. [25] J. Long, E. Shelhamer, and T. Darrell (2015) Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Boston, MA, p. 3431–3440. Cited by: §6. [26] B. W. Matthews (1975) Comparison of the predicted and observed secondary structure of T4 phage lysozyme. Biochim. Biophys. Acta 405 (2), p. 442–451. External Links: Document Cited by: Table 2, §3. [27] D. M. W. Powers (2011) Evaluation: from precision, recall and F-measure to ROC, informedness, markedness and correlation. J. Mach. Learn. Technol. 2 (1), p. 37–63. Cited by: §5. [28] R. C. Prati, G. E. A. P. A. Batista, and M. C. Monard (2011) A survey on graphical methods for classification predictive performance evaluation. IEEE Trans. Knowl. Data Eng. 23 (11), p. 1601–1618. External Links: Document Cited by: §1, §2, §2, §6, §6. [29] R. Rifkin and A. Klautau (2004) In defense of one-vs-all classification. J. Mach. Learn. Res. 5, p. 101–141. Cited by: §6. [30] F. Sebastiani (2002) Machine learning in automated text categorization. ACM Comput. Surv. 34 (1), p. 1–47. External Links: Document Cited by: §2. [31] M. Sokolova and G. Lapalme (2009) A systematic analysis of performance measures for classification tasks. Inf. Process. Manage. 45 (4), p. 427–437. External Links: Document Cited by: §1, §2, §7.3. [32] D. M. J. Tax and R. P. W. Duin (2004) Support vector data description. Mach. Learn. 54 (1), p. 45–66. Cited by: §6. [33] G. Tsoumakas and I. Katakis (2007) Multi-label classification: an overview. IJDWM 3 (3), p. 1–13. External Links: Document Cited by: §2, §3, §5.1, Table 3, §6.1. [34] G. Tsoumakas, E. Spyromitros-Xioufis, J. Vilcek, and I. Vlahavas (2011) MULAN: a Java library for multi-label learning. In Journal of Machine Learning Research, Vol. 12, p. 2411–2414. Cited by: §8.4. [35] L. A. Zadeh (1965) Fuzzy sets. Inf. Control 8 (3), p. 338–353. External Links: Document Cited by: §6. [36] M. Zhang and Z. Zhou (2014) A review on multi-label learning algorithms. IEEE Trans. Knowl. Data Eng. 26 (8), p. 1819–1837. Cited by: §2.