Paper deep dive
Goodness-of-pronunciation without phoneme time alignment
Jeremy H. M. Wong, Nancy F. Chen
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 95%
Last extracted: 3/27/2026, 1:33:08 AM
Summary
This paper proposes a method for speech evaluation that eliminates the need for phoneme-level time alignment, which is typically required by frame-synchronous ASR models. By leveraging weakly-supervised models like Whisper, the authors compute phoneme posteriors from a phoneme-level confusion network and use a cross-attention architecture to combine phoneme and frame-level features. This approach enables speech evaluation for low-resource languages, as demonstrated by experiments on English (speechocean762) and Tamil datasets.
Entities (5)
Relation Signals (3)
cross-attention architecture → replaces → phoneme time alignment
confidence 95% · Phoneme and frame-level features are combined using a cross-attention architecture, obviating phoneme time alignment.
wav2vec 2.0 XLSR-53 → providesfeaturesfor → speech evaluation
confidence 90% · The SSL model here is wav2vec 2.0 XLSR-53... The per-frame features are then converted to per-phoneme representations... and used as inputs to the speech evaluation model.
Whisper → usedfor → Feature Extraction
confidence 90% · This paper proposes to leverage existing weakly-supervised multi-lingual speech foundation models to extract the features needed for speech evaluation.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:In speech evaluation, an Automatic Speech Recognition (ASR) model often computes time boundaries and phoneme posteriors for input features. However, limited data for ASR training hinders expansion of speech evaluation to low-resource languages. Open-source weakly-supervised models are capable of ASR over many languages, but they are frame-asynchronous and not phonemic, hindering feature extraction for speech evaluation. This paper proposes to overcome incompatibilities for feature extraction with weakly-supervised models, easing expansion of speech evaluation to low-resource languages. Phoneme posteriors are computed by mapping ASR hypotheses to a phoneme confusion network. Word instead of phoneme-level speaking rate and duration are used. Phoneme and frame-level features are combined using a cross-attention architecture, obviating phoneme time alignment. This performs comparably with standard frame-synchronous features on English speechocean762 and low-resource Tamil datasets.
Tags
Links
- Source: https://arxiv.org/abs/2603.25150v1
- Canonical: https://arxiv.org/abs/2603.25150v1
Trouble viewing inline? Open PDF directly →
Full Text
48,132 characters extracted from source content.
Expand or collapse full text
Goodness-of-pronunciation without phoneme time alignment Jeremy H. M. Wong and Nancy F. Chen Abstract In speech evaluation, an Automatic Speech Recognition (ASR) model often computes time boundaries and phoneme posteriors for input features. However, limited data for ASR training hinders expansion of speech evaluation to low-resource languages. Open-source weakly-supervised models are capable of ASR over many languages, but they are frame-asynchronous and not phonemic, hindering feature extraction for speech evaluation. This paper proposes to overcome incompatibilities for feature extraction with weakly-supervised models, easing expansion of speech evaluation to low-resource languages. Phoneme posteriors are computed by mapping ASR hypotheses to a phoneme confusion network. Word instead of phoneme-level speaking rate and duration are used. Phoneme and frame-level features are combined using a cross-attention architecture, obviating phoneme time alignment. This performs comparably with standard frame-synchronous features on English speechocean762 and low-resource Tamil datasets. I Introduction Speech evaluation aims to automatically grade the oral proficiency of a speaker. This is useful in computer-aided language learning scenarios, such as to provide feedback to language learners practising to speak with automatic applications. This paper focuses on evaluation of read speech, where the input is speech and the text that should have been read, and the output is a proficiency score. Like many other tasks, the challenge of data scarcity is often encountered in speech evaluation. This is especially so when developing a system for a low-resource language. In systems such as [41, 13, 34, 7, 35, 37], which this paper’s setup is an extension of, a frame-synchronous Automatic Speech Recognition (ASR) model is first used to force align the speech with the transcript, to compute time boundaries and phoneme posteriors, which are then used to derive features to perform speech evaluation on. This ASR model needs to be trained on sufficiently diverse paired speech-text data to yield robust performance. The difficulty of collecting such data for low-resource languages hinders the expansion of speech evaluation to these languages. However, the function of the ASR model itself is only for feature extraction, while a separate model is often responsible for the primary speech evaluation task. It thus seems suboptimal that limited availability of ASR training data restricts the development of a speech evaluation system. The recent developments of speech foundation models offer hope to overcome this limitation. These foundation models are pre-trained on large quantities of diverse data, using either weakly-supervised learning [29] or Self-Supervised Learning (SSL) [2, 15, 8]. This relies on either diversely labelled or unlabelled data, which is easier to obtain than paired speech and annotated transcript data. Furthermore, many speech foundation models are open-source and easily available. These models have been shown to generalise over a broad range of tasks and languages, and may empower the expansion of downstream tasks to low-resource languages. This paper aims to ease the expansion of speech evaluation to low-resource languages, by overcoming the scarcity of paired speech-text data for ASR model development. The proposal is to leverage existing weakly-supervised multi-lingual speech foundation models to extract the features needed for speech evaluation. However, some multi-lingual weakly-supervised models, such as Whisper [29], are not frame-synchronous and do not explicitly model phonemes. These hinder the computation of time boundaries and phoneme posteriors. The issues described in more detail are that, first, the model may not express phonemes in its output and may format the text. For example, Whisper outputs byte-level byte-pair encoding [30] of the display-formatted text. Second, the frame-asynchronous nature of the model hinders the computation of time alignments. This paper proposes to address these incompatibilities by computing phoneme posteriors from a phoneme-level Confusion Network (CN), obtained from an N-best list generated by the weakly-supervised model. The N-best list is text normalised and mapped to phonemes through a lexicon. It is also proposed to replace the phoneme-level Speaking Rate (SR) and Normalised Duration (ND) features, which require phoneme time alignments, with the word level, for which word time alignments can be computed from the Whisper attention weights. SSL models offer another way to alleviate data scarcity. Features extracted from these are informative for a variety of downstream tasks. A task-specific downstream model can be trained upon these features on limited labelled data. Using SSL features has been found to often improve the downstream model performance [2]. For speech evaluation, SSL features have been used in mispronunciation detection [26, 38, 40] and predicting other proficiency aspects [4, 3]. The transcription of the speech can also be used as a complementary input to the model [17]. This paper likewise also uses SSL features as one input to a speech evaluation model. In setups such as [41, 13, 34, 7, 35, 37], input features to the speech evaluation model are expressed per phoneme. Using per-frame SSL features together with per-phoneme features requires a phoneme time alignment. In [7, 37], the SSL features are first averaged over the frames that are aligned with each canonical phoneme, and then concatenated with the other per-phoneme features. A lack of phoneme time alignments from frame-asynchronous weakly-supervised models hinders such per-phoneme averaging of the SSL features. Rather than averaging the SSL features per phoneme from known alignments, this paper instead proposes to use a cross-attention layer to combine the per-phoneme features with the per-frame SSL features. These proposals eliminate the need to train a language-specific ASR model for feature extraction, thereby easing the expansion of speech evaluation to new languages that are already supported by multi-lingual open-source models. Such language expansion is demonstrated by experimenting on the low-resource Tamil language. Although the experiments use Whisper in this paper, the proposals are also applicable to other open-source ASR models. Although these proposals obviate paired speech-text data for ASR model training, speech-text-score data is still needed to train the read-speech evaluation model. The text here can be the transcript that is meant to be read, which is more easily obtainable than an annotation of what the speaker actually uttered. On the other hand, for ASR model training, mismatches between the reference text and speech content may be detrimental to the model performance, and therefore additional effort is required to manually annotate what the speaker actually uttered. For speech evaluation, the reference score is manually annotated here. Works in [19, 18, 33] consider speech evaluation approaches that do not require annotated scores for training. I Speech evaluation model and features (a) Mean pool frame-level features per-phoneme, then concatenate with per-phoneme features (b) Transformer decoder cross-attention to combine per-phoneme and per-frame features Figure 1: Transformer models for sentence-level speech evaluation The task is to evaluate the oral proficiency of a speaker as they read off a transcript. This paper expands upon the speech evaluation system setup that follows the general architecture in [7, 37]. The baseline architecture is shown in Fig. 1(a). This computes per-phoneme features (i.e. one feature vector for each phoneme in the sentence) of the Goodness-Of-Pronunciation (GOP), Log-Phoneme Posterior (LPP), Log-Posterior Ratio (LPR), SR, ND, and Phoneme Embedding (PE). Per-frame features of pitch and embeddings extracted from a weighted sum across layers of an SSL model are also computed. The SSL model here is wav2vec 2.0 XLSR-53 [9], which is pre-trained on multiple languages. The per-frame features are then converted to per-phoneme representations by mean-pooling over the frames that are aligned with each phoneme. These are then concatenated with the other per-phoneme features, and used as inputs to the speech evaluation model. The speech evaluation model first linearly projects the input to a reduced 24 dimensions, followed by 3 layers of 8-headed transformer encoder layers. Single-headed sentence pooling then converts the per-phoneme sequence to a sentence-level representation. There is then a ReLU hidden layer, followed by a softmax output that classifies between possible sentence-level oral proficiency scores. Each of the feature types is now described in more detail. Let the time-normalised log-posterior probability of a phoneme be ϕi(ρ)=1te(ρiref)−ts(ρiref)+1∑t=ts(ρiref)te(ρiref)logP(ρt=ρ|), _i (ρ )= 1t_e ( _i^ref )-t_s ( _i^ref )+1 _t=t_s ( _i^ref )^t_e ( _i^ref ) P ( _t=ρ |O ), (1) where ρ is a phoneme, ρiref _i^ref is the iith correct canonical phoneme in the sentence that should have been read, tst_s and tet_e are the start and end frame indexes of the occurrence of that phoneme respectively, t is a frame index, and O is the acoustic observation. When using a frame-synchronous ASR model, the times can be computed from a numerator lattice forced-alignment between the speech and the text that should have been read. In [16], the per-frame phoneme posterior, P(ρt|)P ( _t |O ), is computed as the softmax output of the neural network acoustic model, avoiding the computation expense of denominator lattice forward-backward operations. The duration normalisation encourages GOP features from phonemes with different durations to reside within similar dynamic ranges. The GOP is the time-normalised log-posterior probability of a canonical phoneme [16], GOPi=ϕi(ρiref).GOP_i= _i ( _i^ref ). (2) A commonly-used variant is to measure the margin between the log-posteriors of the canonical and most likely phonemes [16], GOPmargin,i=ϕi(ρiref)−maxρϕi(ρ).GOP_margin,i= _i ( _i^ref )- _ρ _i (ρ ). (3) LPP and LPR features are often also used for speech evaluation [16]. LPP is defined as a vector of the log-posteriors of all phonemes, such that LPPi=[ϕi(ρ),∀ρ∈],LPP_i= [ _i (ρ ),∀ρ ], (4) where P is the set of all phonemes. The LPR is the log-ratio between the posteriors of each phoneme and the canonical, LPRi=[ϕi(ρiref)−ϕi(ρ),∀ρ∈].LPR_i= [ _i ( _i^ref )- _i (ρ ),∀ρ ]. (5) The GOP, LPP, and LPR provide information about confusability between phonemes that are pronounced, as perceived by the ASR model. The SR is the rate at which the canonical phoneme is spoken [41], speaking ratei=1te(ρiref)−ts(ρiref).speaking rate_i= 1t_e ( _i^ref )-t_s ( _i^ref ). (6) The ND is the time taken to utter the canonical phoneme, normalised over all phonemes and utterances [41], normalised phoneme durationi=te(ρiref)−ts(ρiref)−μσ,normalised phoneme duration_i= t_e ( _i^ref )-t_s ( _i^ref )-μσ, (7) where μ and σ are the mean and standard deviation of the un-normalised phoneme durations, te−tst_e-t_s, respectively. These features are indicative of the speaker’s fluency. PE may represent semantic information, analogously to word embeddings [22]. These were extracted from a 32-node recurrent neural network continuous skip-gram language model [22], which was trained on the non-silence canonical phoneme sequences of the speech evaluation datasets. Pitch was computed using the approach in [12], and may inform about the speaker’s intonation. I Computing features from Whisper Open-source multi-lingual models allow for easy development of speech processing on a variety of languages, many of which that are supported are considered as low-resourced. For speech evaluation, this may simplify the expansion of applications to new languages. One such model is Whisper, which is trained in a weakly-supervised manner toward a variety of tasks, one of which is ASR. Whisper is designed with an attention encoder-decoder architecture [6], which is frame-asynchronous and thus does not directly yield time alignments. Whisper also outputs formatted byte-pair encoding tokens, from which the computation of phoneme posteriors is not trivial. The formatted output may also not adequately express disfluencies, which may be useful information for speech evaluation. Work in [20] considers improving the disfluency expression of Whisper using prompt tuning, and is orthogonal to this paper. Consider the text formatting and lack of expression of phonemes. The weakly-supervised model can be used to compute ASR hypotheses, expressed as an N-best list of word sequences. This paper proposes to use rule-based text normalisation to standardise casing, remove punctuations, and express numbers in spoken form. A lexicon is used to convert words into phoneme sequences. Although Whisper can be prompted to transcribe in a specified target language, words that do not belong to that language may still be found within the decoded results. It may be reasonable to assume that these foreign words are phonetically related to the input speech, and may therefore still be useful when computing phoneme confusions. Transliteration or romanisation can be used for non-Latin and Latin script target languages respectively, to convert these to similarly sounding character sequences in the script of the target language. A Grapheme-to-Phoneme (G2P) model can supplement the lexicon for out-of-vocabulary words. Multiple alternative pronunciations can be considered to improve generalisability, together with prior pronunciation probabilities computed from the G2P model. This paper proposes to compute per-phoneme phoneme posteriors, P(ρi|)P ( _i |O ), from a phoneme-level CN, instead of as (1). A CN [21, 11, 39] is a representation of competing decoded hypotheses, which assumes that the probability of the phoneme at each position along the sequence is conditionally independent of phonemes at other positions along the sequence. In this way, the probability of the phoneme sequence can be expressed as P(|)≈∏iP(ρi|).P ( ρ |O )≈ _iP ( _i |O ). (8) A CN can be approximated from an N-best list by using approaches of [21, 11, 39], with [39] being used in this paper. The CN can be interpreted to express a confidence in the model’s hypothesis. A related approach in [1] instead trains a Whisper-based model to directly predict confidence scores from a given hypothesis. GOP-like features, referred to as CN-GOP, are computed from the log-posteriors of the phoneme-level CN, CN-GOPi=logP(ρi=ρiref|).CN-GOP_i= P ( _i= _i^ref |O ). (9) Analogously to (3), a margin between the log-posteriors of the canonical and most likely phonemes can also be computed as CN-GOPmargin,i=logP(ρi=ρiref|)−maxρlogP(ρi=ρ|).CN-GOP_margin,i= P\! ( _i\!=\! _i^ref |O )- _ρ P\! ( _i\!=\!ρ |O )\!. (10) Duration normalisation is omitted, to accommodate for the absence of phoneme time alignments from a frame-asynchronous model. Analogous LPP and LPR features can also be computed from the CN. In summary, the proposed steps to compute the phoneme-level CN from frame-asynchronous non-phonemic formatted hypotheses are as follows. 1. Beam-search ASR decoding with the weakly-supervised model is used to compute an N-best list of formatted hypotheses. 2. The N-best word sequences are then text normalised. 3. Transliteration or romanisation is applied to accommodate for hypothesised words that are written in a script that differs from the target language. 4. The N-best list is filtered to remove identical word sequences, retaining that with the largest posterior. 5. The word sequences are converted to phoneme sequences through a lexicon. 6. A phoneme-level CN is computed from the phoneme-level N-best list. This is done by first expressing the N-best list as a phoneme lattice, with branched paths for competing N-best hypotheses and also branched paths for the alternative pronunciations of each word. The branch scores can be computed from the N-best posteriors and the G2P pronunciation priors. The algorithm in [39] is then used to compute a CN from the lattice. For read speech, GOP, LPP, and LPR features are computed for each canonical phoneme. This is computed by first finding a minimum edit distance alignment between the CN and the canonical phoneme sequence of the transcript. Each confusion set is categorised as either an insertion, deletion, or matched with a canonical phoneme. If matched, then the phoneme posterior can be read off from the confusion set, and phonemes that are not expressed in the confusion set are assumed to have zero probability. For a deletion, it is assumed that all probability is allocated to silence, and thus the non-silence phonemes are assigned zero probability. Insertions can be ignored, as GOP, LPP, and LPR features only need to be computed for each canonical phoneme. A limitation is that pronunciation scores in the lexicon are priors, without considering the observed audio. Furthermore, the pronunciation variants present in the lexicon may not adequately capture idiosyncrasies present in non-native speech. An alternative to beam-search decoding in step 1 is to instead generate hypotheses by sampling from the posterior. Initial tests found that for speechocean762, a high temperature needs to be used to obtain a set of diverse hypotheses. On the other hand, beam-search consistently yielded diverse hypotheses without needing to apply a temperature. Applying a temperature may impact the distribution expressed in the set of hypotheses with their sentence-level posteriors, which may then affect the GOP, LPP, and LPR features that are computed from these. This paper therefore favours using beam-search decoding over sampling generation. The model’s frame-asynchronous nature hinders the direct computation of the SR and ND. However, it is possible to estimate token-level time alignments through a dynamic time warping on the cross-attention weights. If the tokens are sub-word units, then word-level SR and ND features can be computed from these, and used instead of phoneme-level features. IV Alignment-free cross-attention The speech evaluation models in [7, 37] have inputs with sequence lengths corresponding to the number of canonical phonemes in the utterance. Per-frame features extracted from pre-trained SSL models and pitch can supplement the per-phoneme features. In [7, 37], this is achieved by mean pooling the per-frame features over the frames that are aligned to each canonical phoneme. An example of a model with this feature combination is shown in Fig. 1(a). An absence of a phoneme time alignment hinders the joint usage of phoneme-level and frame-level features. This paper proposes that a cross-attention, such as a transformer decoder [31], can allow the joint usage of phoneme-level and frame-level features, without needing to supply a time alignment. An example of this is illustrated in Fig. 1(b). The phoneme-level input sequence is used as the query to the cross-attention, thereby determining the output sequence length, while the frame-level input sequence is used as both the key and value. This may allow the model to learn how to associate between phonemes and frames, thereby learning to align. However, this approach allows each phoneme position to attend over the entire frame sequence. Learning from the limited training data about an adequate association of which frames belong to which canonical phoneme may be difficult. Word time alignments can be used to restrict each phoneme position to only attend to frames that are aligned to the word that the phoneme belongs to. Although such a constraint is less restrictive than using a phoneme time alignment, it is easier to obtain a word rather than a phoneme time alignment from Whisper. V Experiments V-A Setup Experiments were performed on two language datasets, speechocean762 [42] in English and an internal Tamil dataset that is also used in [41, 36]. Speechocean762 comprises 2500 sentences of English read speech from 125 native Mandarin speakers in each of the train and test sets. The data is annotated with various types of proficiency scores at the phoneme, word, and sentence levels. The experiments in this paper only consider pronunciation accuracy, which is annotated by 5 human raters per sentence, along a scale from 0 to 10 for the sentence and word levels, and from 0 to 2 for the phoneme level. A consensus reference score was obtained by averaging the scores of the 5 raters, which differs from [42] that instead uses the median score. The Tamil data comprises 2118 and 2093 sentences in the train and test sets respectively. These were read speech from 100 speakers from Singapore, between 9 to 16 years old. The sentence-level pronunciation accuracy was annotated by 4 raters along a scale from 1 to 5. A consensus reference score was again computed as the average of the multiple rater scores. A baseline approach for speech evaluation is to compute GOP, LPP, LPR, SR, and ND from a frame-synchronous hybrid neural network-Hidden Markov model [5] ASR model. For speechocean762, this was trained following the setup in [42]. The acoustic model used a Time Delay Neural Network (TDNN) [32], and was trained on the Librispeech 960 hours data [24], following the standard Kaldi [27] recipe, up to the cross-entropy stage. For Tamil, a lattice-free [28] hybrid model was trained on 220 hours of internal read closed-talking speech from 710 adults, previously used in [41, 36]. The acoustic model interleaved TDNN and long short-term memory layers [25]. Forced alignment of the speech evaluation data using these ASR models was used to compute the GOP, LPP, LPR, SR, and ND. SSL embeddings, PE, and pitch features were also computed, as described in section I. The baseline speech evaluation architecture is illustrated in Fig. 1(a) and is referred to as the phoneme pooling model. The proposed alignment-free architecture shown in Fig. 1(b) and described in section IV is referred to as the cross-attention model. The speech evaluation model was trained toward the cross-entropy criterion, with early stopping measured on a 10% held-out validation set. During inference, the score was hypothesised as the mean of the discrete output posterior. Following [42], both the hypothesis and reference scores were first rounded to the closest integers before computing the Pearson’s Correlation Coefficient (PCC) and Mean Squared Error (MSE) evaluation metrics. V-B Features from Whisper The first experiment investigates the use of multiple alternative pronunciations and pronunciation priors when computing the GOP, LPP, and LPR from Whisper, each on its own without combining with other features. The speechocean762 test set was used. Whisper-large-v3 was decoded with a beam size of 200, using English prompted as the target language. Non-Latin script words were romanised using Unidecode. The CMU lexicon was used to map words to phoneme sequences, which contains up to 6 pronunciation variants for each word. A G2P model from [23] was used to supplement 6 pronunciation variants for each word that was not contained within the CMU lexicon, and also to compute pronunciation priors. This initial investigation of the stand-alone quality of the features followed the baseline setup presented in [42], where per-phoneme Support Vector Regressors (SVR) [10] were trained on each of the feature types to compute the phoneme-level pronunciation accuracy. TABLE I: Phoneme pronunciation accuracy of per-phoneme SVRs trained on GOP, LPP, and LPR from Whisper, using multiple pronunciations and priors Number of Pronunciation PCC↑ / MSE↓ for Whisper feature pronunciations prior CN-GOP CN-GOPmargin LPP & LPR 1 no 0.24 / 0.95 0.28 / 0.99 0.42 / 0.16 up to 6 no 0.29 / 0.88 0.41 / 0.99 0.48 / 0.15 up to 6 yes 0.27 / 0.92 0.34 / 1.00 0.48 / 0.14 The results in Table I show that using multiple pronunciations in the lexicon improves performance, while using pronunciation priors may not yield further gains. When pronunciation priors were not used, all pronunciation variants were treated as equally likely. The lack of benefit of pronunciation priors may be because the priors computed by the G2P model are independent of the acoustic observation. Subsequent experiments used multiple pronunciations without priors. TABLE I: Compare features from hybrid model and Whisper with different beam sizes, to compute phoneme pronunciation accuracy, using per-phoneme SVRs Features N-best PCC / MSE for feature from size GOP GOPmargin LPP & LPR hybrid - 0.28 / 0.60 0.25 / 0.69 0.47 / 0.16 Whisper 10 0.31 / 0.91 0.37 / 0.99 0.49 / 0.17 100 0.30 / 0.90 0.41 / 1.00 0.47 / 0.15 200 0.29 / 0.88 0.41 / 0.99 0.48 / 0.15 The next experiment compares the Whisper-generated features using different beam sizes, against features computed from a hybrid model. Per-phoneme SVRs were again used to model phoneme pronunciation accuracy. The results in Table I suggest that the feature quality may be fairly independent of the decoding beam size, with the only substantial improvement being in the PCC for CN-GOPmargin when increasing the beam from 10 to 100. It therefore seems reasonable to use a smaller decoding beam to reduce computational cost. However, the remaining experiments used a beam size of 200, as the computation was still feasible with the dataset sizes that were used. The combination of LPP and LPR features computed from Whisper performs comparably to those computed from a hybrid model. For GOPmargin, features extracted from Whisper have better PCC but worse MSE. The degraded MSE may be due to an interaction between the sparsity of the CN posteriors and the per-phoneme modelling of the SVRs. TABLE I: Sentence-level pronunciation accuracy of phoneme pooling model on GOP, LPP, and LPR features from Whisper Features from GOP type Use LPP & LPR PCC MSE - - no 0.709 1.138 hybrid GOPmargin yes 0.753 1.051 Whisper CN-GOPmargin yes 0.748 1.042 Whisper CN-GOP yes 0.745 1.026 A phoneme pooling model was then trained using the phoneme-level GOPmargin, LPP, LPR, SR, ND, and PE features, together with the frame-level SSL and pitch features. This computed the sentence-level pronunciation accuracy. Table I compares omitting the GOP, LPP, and LPR features (top row), as well as replacing these with features computed from Whisper. Omitting these three features degrades performance, showing their complementarity to the other features. Features computed from Whisper perform comparably to those computed from a hybrid model. TABLE IV: Inclusion of SR and ND features from Whisper Use SR and ND PCC MSE no 0.729 1.125 hybrid phoneme level 0.753 1.051 Whisper word level 0.743 1.031 Table IV then investigates using word-level SR and ND features computed from Whisper alignments, using the same phoneme pooling model. Each canonical phoneme position was assigned the SR and ND of the word that the phoneme belonged to. The GOPmargin, LPP, and LPR features here were computed from a hybrid model. The results again show that omitting SR and ND features degrades performance. Using word-level SR and ND computed from a Whisper alignment performs comparably against using phoneme-level SR and ND from a hybrid model. V-C Alignment-free cross-attention model The models used thus far have combined the per-phoneme GOPmargin, LPP, LPR, SR, ND, and PE features together with the per-frame SSL and pitch features, by averaging the per-frame features over the frames that are aligned with each canonical phoneme. A phoneme time alignment is not easily available from a frame-asynchronous model that does not explicitly express phonemes, like Whisper. Section IV proposes a cross-attention approach, that allows for combination between these features without phoneme time alignments. This experiment investigates using this model architecture, shown in Fig. 1(b). The per-phoneme and per-frame features were separately projected to 24 dimensions. An 8-headed 24-dimensional transformer decoder layer was used to combine these feature projections. The output of this was then parsed through 2 transformer encoder layers, followed by sentence-level attention pooling, ReLU feed-forward, and a softmax output. TABLE V: Combine per-phoneme and per-frame features either by averaging over the frames for each phoneme or by using cross-attention Features from Combine features by PCC MSE hybrid phoneme pooling 0.753 1.051 cross-attention 0.718 1.095 Whisper cross-attention 0.735 1.051 + restrict attention to word 0.747 1.027 The results shown in Table V start from a baseline configuration of using features from a hybrid model and used the phoneme pooling architecture. The features were then kept the same, but the model architecture was replaced with the cross-attention approach. This yields a performance degradation, suggesting the benefit that a phoneme time alignment can bring when associating between the per-frame and per-phoneme features. While still using the cross-attention architecture, the hybrid model features were replaced with the CN-GOPmargin, LPP, LPR, and word SR and ND features from Whisper. This configuration does not need a hybrid model. This performs similarly to the baseline for MSE, but still has a degraded PCC. The cross-attention allows each phoneme position to attend over all frames. Section IV proposes that a word time alignment can be used to restrict the cross-attention, such that each phoneme only attends to frames that are aligned to the same word as that phoneme. The results in the bottom row of Table V suggest that such an attention constraint may be useful, and brings the performance back to being more comparable with the baseline. Thus, speech evaluation can be done without ASR training data. V-D Word-level posteriors Section I describes how phoneme-level posteriors can be computed from a CN representation of ASR hypotheses. These can then be used to compute GOP, LPP, and LPR features for speech evaluation. Analogous posteriors can also be expressed at the word level, and be used to compute word-level GOP features. These may be complementary to the phoneme-level features. This experiment investigates the usefulness of these word-level GOP features. The following steps were used to compute the features. The 200-best list from Whisper beam search was text normalised, romanised, and filtered for repetitions. A word-level CN was then computed from this N-best list. The CN was aligned with the reference transcription. For a confusion set that was matched with a reference word, the word posteriors could be read off the confusion set, with zero probability assigned to words that were not in the confusion set. For deletions, all non-silence word probabilities were assumed to be zero. Insertions were again ignored. Analogous word-level GOP-like features, referred to as CN-WGOP and CN-WGOPmargin were computed from the word posteriors. As a sanity check, the PCC between CN-WGOP and the reference word-level pronunciation accuracy on the speechocean762 test set is 0.477, suggesting a reasonable correlation with the human-labelled word accuracy. TABLE VI: Inclusion of word GOP features in cross-attention model Use word-level GOP PCC MSE no 0.747 1.027 CN-WGOPmargin 0.745 1.040 CN-WGOP 0.737 1.077 A baseline setup used a cross-attention model with word time alignment constrained attention, using phoneme CN-GOPmargin, LPP, LPR, and word SR and ND features from Whisper, together with PE, SSL, and pitch. The word-level GOP was then concatenated to the per-phoneme features, with each phoneme position in the sequence using the GOP of the word that the phoneme belonged to. Either CN-WGOP or CN-WGOPmargin was used. The sentence-level pronunciation accuracy results in Table VI suggest that using CN-WGOPmargin outperforms using CN-WGOP, but both do not yield performance improvements over the baseline. This suggests that word-level GOP may not be complementary to the existing phoneme-level and frame-level features for speech evaluation. V-E Extend speech evaluation to low-resource Tamil language Using a multi-lingual open-source weakly-supervised model to compute features and time alignments for speech evaluation alleviates the need for paired speech-text data, simplifies extending speech evaluation to new languages supported by the weakly-supervised model, and may allow speech evaluation applications to benefit from the latest open-source model releases. The experiments thus far have considered speech evaluation in English, which is a reasonably high-resourced language. It is more difficult to obtain large quantities of diverse paired speech-text data for Tamil, to train an ASR model. Internal Tamil ASR training data needed to be collected here, because of the sparsity of open-source alternatives. It is often difficult to share such internal data, due to privacy constraints. Thus, Tamil is considered a low-resource language, and the proposal here can be applied when no Tamil ASR data is available at all. Whisper supports Tamil ASR, and thus can be used out-of-the-box to generate features and time alignments for speech evaluation. This section demonstrates the application of the proposed approach to Tamil speech evaluation. In the previous experiments on English, hypothesised words from Whisper that were not in Latin script were text-normalised and romanised. The Tamil N-best lists were text-normalised to remove punctuation and convert numbers into spoken form. Open-source transliteration tools for Tamil as the target script are not as readily available as romanisation tools for a target with Latin script. As such, a simpler approach was used for Tamil by omitting N-best hypotheses that contained non-Tamil characters. The Tamil words were converted into phoneme sequences using a lexicon obtained from Babel [14]. This expresses up to 4 pronunciation variants for each word. A G2P model trained on the Babel lexicon was used to generate 4 pronunciation variants for hypothesised Tamil words that were not contained in the Babel lexicon. TABLE VII: Apply Whisper features and cross-attention model to low-resource Tamil language Features from Combine features by PCC MSE hybrid phoneme pooling 0.547 0.637 cross-attention 0.611 0.699 Whisper cross-attention 0.611 0.573 + restrict attention to word 0.629 0.569 The experiment from section V-C is applied to Tamil in Table VII. This compares a baseline hybrid model features approach and the proposed approach of computing features and time alignments from Whisper, on Tamil. As with the previous English experiments, the baseline approach used GOPmargin, LPP, LPR, SR, ND, and phoneme forced alignments from the hybrid model. These were combined with PE, pitch, and SSL features. The pitch and SSL features were averaged over the frames that were aligned with each canonical phoneme. The proposed approach used CN-GOPmargin, LPP, LPR, word-level SR and ND, and word-level time alignments computed from Whisper. Together with PE, these were combined with pitch and SSL features using cross-attention. Word posteriors were not used. The results show that using a cross-attention architecture with features from a hybrid model improves PCC but degrades MSE, compared to a phoneme pooling architecture that uses a hybrid forced alignment. It is thus difficult to conclude how changing the architecture impacts performance. However, changing the features to those from Whisper improves both PCC and MSE, compared to hybrid features with a phoneme pooling model. Restricting the transformer decoder attention according to the Whisper word time alignment further improves the performance. This suggests that on a low-resource language, like Tamil, speech evaluation can be performed without any need for paired speech-text training data, by using an easily available multi-lingual open-source ASR model. This approach not only has comparable performance, but may surpass the performance of the hybrid baseline. Perhaps, in a low-resource scenario, it may be beneficial to rely on a multi-lingual open-source model that was trained on a large quantity of diverse data, rather than attempting to source for limited target language paired speech-text data to train an ASR model. V-F Confusion network decoding of Whisper TABLE VIII: Confusion network decoding of Whisper N-best list N-best size 1-best WER (%)↓ CN decode WER (%) 1 (greedy) 17.40 - 10 16.25 15.66 100 16.18 15.41 200 16.22 15.36 Throughout this paper, CNs computed from Whisper N-best lists are repeatedly used to generate features for speech evaluation. With the availability of these CNs, it seems convenient to assess whether CN decoding for ASR has any benefit for a large, weakly-supervised, attention encoder-decoder model like Whisper. Table VIII shows the Word Error Rate (WER) measured on the speechocean762 test set when either selecting the top hypothesis from the N-best list or when first computing a CN, and then selecting the top word from each confusion set. The results suggest that CN decoding consistently outperforms selecting the 1-best, or top, hypothesis. When selecting the top hypothesis, increasing the decoding beam size only yields improvements up to a beam size of 100. For CN decoding, increasing the beam size continues to yield improvements up to the largest experimented beam size of 200. This suggests that applying CN decoding to a large, weakly-supervised model like Whisper may still be beneficial. VI Conclusion This paper has considered how to alleviate the reliance of speech evaluation on expensive target language paired speech-text data to train a phonemic frame-synchronous ASR model, by relaxing the requirements to allow the use of more easily obtainable open-source multi-lingual frame-asynchronous models. Alternative GOP, SR, and ND features have been proposed without requiring phoneme time alignments, and instead using CNs. A cross-attention model architecture has also been proposed to combine between per-phoneme and per-frame features. The experiments show that these perform comparably to the hybrid baseline on English speechocean762, and even outperform the baseline on low-resource Tamil. Thus, speech evaluation can be done without expensive speech-text data. References [1] V. Aggarwal, S. S. Nair, Y. Verma, and Y. Jogi (2025-04) Adopting Whisper for confidence estimation. In ICASSP, Hyderabad, India. Cited by: §I. [2] A. Baevski, H. Zhou, A. Mohamed, and M. Auli (2020-12) Wav2vec 2.0: a frameword for self-supervised learning of speech representations. In NeurIPS, Vancouver, Canada, p. 12449–12460. Cited by: §I, §I. [3] S. Bannò, K. M. Knill, M. Matassoni, V. Raina, and M. J. F. Gales (2023-08) Assessment of L2 oral proficiency using self-supervised speech representation learning. In SLaTE, Dublin, Ireland, p. 126–130. Cited by: §I. [4] S. Bannò and M. Matassoni (2023-01) Proficiency assessment of L2 spoken English using wav2vec 2.0. In SLT, Doha, Qatar, p. 1088–1095. Cited by: §I. [5] H. A. Bourlard and N. Morgan (1994) Connectionist speech recognition: a hybrid approach. Kluwer Academic Publishers. Cited by: §V-A. [6] W. Chan, N. Jaitly, Q. Le, and O. Vinyals (2016-03) Listen, attend and spell: a neural network for large vocabulary conversational speech recognition. In ICASSP, Shanghai, China, p. 4960–4964. Cited by: §I. [7] F.-A. Chao, T.-H. Lo, T.-I. Wu, Y.-T. Sung, and B. Chen (2022-11) 3M: an effective multi-view, multi-granularity, and multi-aspect modeling approach to English pronunciation assessment. In APSIPA ASC, Chiang Mai, Thailand, p. 575–582. Cited by: §I, §I, §I, §IV. [8] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao, J. Wu, L. Zhou, S. Ren, Y. Qian, Y. Qian, J. Wu, M. Zeng, X. Yu, and F. Wei (2022-10) WavLM: large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing 16 (6), p. 1505–1518. Cited by: §I. [9] A. Conneau, A. Baevski, R. Collobert, A. Mohamed, and M. Auli (2021-08) Unsupervised cross-lingual representation learning for speech recognition. In Interspeech, Brno, Czechia, p. 2426–2430. Cited by: §I. [10] H. Drucker, C. J. C. Burges, L. Kaufman, A. Smola, and V. Vapnik (1996-12) Support vector regression machines. In NIPS, Denver, USA, p. 155–161. Cited by: §V-B. [11] G. Evermann and P. C. Woodland (2000-06) Large vocabulary decoding and confidence estimation using word posterior probabilities. In ICASSP, Istanbul, Turkey, p. 1655–1658. Cited by: §I, §I. [12] P. Ghahremani, B. BabaAli, D. Povey, K. Riedhammer, J. Trmal, and S. Khudanpur (2014-05) A pitch extraction algorithm tuned for automatic speech recognition. In ICASSP, Florence, Italy, p. 2494–2498. Cited by: §I. [13] Y. Gong, Z. Chen, I.-H. Chu, P. Chang, and J. Glass (2022-05) Transformer-based multi-aspect multi-granularity non-native English speaker pronunciation assessment. In ICASSP, Singapore, p. 7262–7266. Cited by: §I, §I. [14] M. P. Harper (2011) IARPA babel program. Note: http://w.iarpa.gov/index.php/research-programs/babel Cited by: §V-E. [15] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed (2021-10) HuBERT: self-supervised speech representation learning by masked prediction of hidden units. IEEE/ACM Transactions on Audio, Speech, and Language Processing 29 (), p. 3451–3460. Cited by: §I. [16] W. Hu, Y. Qian, F. K. Soong, and Y. Wang (2015-03) Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiers. Speech Communication 67, p. 154–166. Cited by: §I, §I, §I. [17] E. Kim, J.-J. Jeon, H. Seo, and H. Kim (2022-09) Automatic pronunciation assessment using self-supervised speech representation learning. In Interspeech, Incheon, South Korea, p. 1411–1415. Cited by: §I. [18] Y. Liang, K. Song, S. Mao, H. Jiang, L. Qiu, Y. Yang, D. Li, L. Xu, and L. Qiu (2023-08) End-to-end word-level pronunciation assessment with MASK pre-training. In Interspeech, Dublin, Ireland, p. 969–973. Cited by: §I. [19] H. Liu, M. Shi, and Y. Wang (2023-08) Zero-shot automatic pronunciation assessment. In Interspeech, Dublin, Ireland, p. 1009–1013. Cited by: §I. [20] R. Ma, M. Qian, M. J. F. Gales, and K. M. Knill (2023-08) Adapting an ASR foundation model for spoken language assessment. In SLaTE, Dublin, Ireland, p. 104–108. Cited by: §I. [21] L. Mangu, E. Brill, and A. Stolcke (2000-10) Finding consensus in speech recognition: word error minimization and other applications of confusion networks. Computer Speech and Language 14 (4), p. 373–400. Cited by: §I, §I. [22] T. Mikolov, K. Chen, G. Corrado, and J. Dean (2013-05) Efficient estimation of word representations in vector space. In ICLR, Scottsdale, USA. Cited by: §I. [23] J. R. Novak, N. Minematsu, and K. Hirose (2016-11) Phonetisaurus: exploring grapheme-to-phoneme conversion with joint n-gram models in the WFST framework. Natural Language Engineering 22 (6), p. 907–938. Cited by: §V-B. [24] V. Panayotov, G. Chen, D. Povey, and S. Khudanpur (2015-04) Librispeech: an ASR corpus based on public domain audio books. In ICASSP, Brisbane, Australia, p. 5206–5210. Cited by: §V-A. [25] V. Peddinti, Y. Wang, D. Povey, and S. Khudanpur (2018-03) Low latency acoustic modeling using temporal convolution and LSTMs. IEEE Signal Processing Letters 25 (3), p. 373–377. Cited by: §V-A. [26] L. Peng, K. Fu, B. Lin, D. Ke, and J. Zhang (2021-08) A study on fine-tuning wav2vec2.0 model for the task of mispronunciation detection and diagnosis. In Interspeech, Brno, Czechia, p. 4448–4452. Cited by: §I. [27] D. Povey, A. Ghoshal, G. Boulianne, L. Burget, O. Glembek, N. Goel, M. Hannemann, P. Motlíček, Y. Qian, P. Schwarz, J. Silovský, G. Stemmer, and K. Veselý (2011-12) The Kaldi speech recognition toolkit. In ASRU, Hawaii, USA. Cited by: §V-A. [28] D. Povey, V. Peddinti, D. Galvez, P. Ghahremani, V. Manohar, X. Na, Y. Wang, and S. Khudanpur (2016-09) Purely sequence-trained neural networks for ASR based on lattice-free MMI. In Interspeech, San Francisco, USA, p. 2751–2755. Cited by: §V-A. [29] A. Radford, J. W. Kim, T. Xu, G. Brockman, C. McLeavey, and I. Sutskever (2023-07) Robust speech recognition via large-scale weak supervision. In ICML, Honolulu, USA, p. 28492–28518. Cited by: §I, §I. [30] R. Sennrich, B. Haddow, and A. Birch (2016-08) Neural machine translation of rare words with subword units. In ACL, Berlin, Germany, p. 1715–1725. Cited by: §I. [31] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017-12) Attention is all you need. In NIPS, Long Beach, USA, p. 5998–6008. Cited by: §IV. [32] A. Waibel, T. Hanazawa, G. E. Hinton, K. Shikano, and K. J. Lang (1989-03) Phoneme recognition using time-delay neural networks. IEEE Transactions on Acoustics, Speech, and Signal Processing 37 (3), p. 328–339. Cited by: §V-A. [33] Y. Wang, Z. Wu, J. Nese, A. Kamata, V. Nilabh, and E. C. Larson (2024-04) Improving oral reading fluency assessment through sub-sequence matching of acoustic word embeddings. In ICASSP, Seoul, South Korea, p. 10766–10770. Cited by: §I. [34] J. H. M. Wong, H. Zhang, and N. F. Chen (2022-09) Variations of multi-task learning for spoken language assessment. In Interspeech, Incheon, South Korea, p. 4456–4460. Cited by: §I, §I. [35] J. H. M. Wong, H. Zhang, and N. F. Chen (2023-08) Distilling knowledge from Gaussian process teacher to neural network student. In Interspeech, Dublin, Ireland, p. 426–430. Cited by: §I, §I. [36] J. H. M. Wong, H. Zhang, and N. F. Chen (2023-07) Modelling inter-rater uncertainty in spoken language assessment. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31, p. 2886–2898. Cited by: §V-A, §V-A. [37] J. H. M. Wong, H. Zhang, and N. F. Chen (2023-12) Variational Gaussian process data uncertainty. In ASRU, Taipei, Taiwan. Cited by: §I, §I, §I, §IV. [38] M. Wu, K. Li, W.-K. Leung, and H. Meng (2021-08) Transformer based end-to-end mispronunciation detection and diagnosis. In Interspeech, Brno, Czechia, p. 3954–3958. Cited by: §I. [39] H. Xu, D. Povey, L. Mangu, and J. Zhu (2011-10) Minimum Bayes risk decoding and system combination based on a recursion for edit distance. Computer Speech and Language 25 (4), p. 802–828. Cited by: item 6, §I, §I. [40] X. Xu, Y. Kang, S. Cao, B. Lin, and L. Ma (2021-08) Explore wav2vec 2.0 for mispronunciation detection. In Interspeech, Brno, Czechia, p. 4428–4432. Cited by: §I. [41] H. Zhang, K. Shi, and N. F. Chen (2021-08) Multilingual speech evaluation: case studies on English, Malay and Tamil. In Interspeech, Brno, Czechia, p. 4443–4447. Cited by: §I, §I, §I, §I, §V-A, §V-A. [42] J. Zhang, Z. Zhang, Y. Wang, Z. Yan, Q. Song, Y. Huang, K. Li, D. Povey, and Y. Wang (2021-08) Speechocean762: an open-source non-native English speech corpus for pronunciation assessment. In Interspeech, Brno, Czechia, p. 3710–3714. Cited by: §V-A, §V-A, §V-A, §V-B.