Paper deep dive
Using Prosody to Predict Syntactic Structure
Junghyun Min, Alex Warstadt, Tamar I. Regev, Tiago Pimentel, Ethan Gotlieb Wilcox
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 90%
Last extracted: 9/15/2026, 8:17:03 AM
Summary
This paper investigates the relationship between prosody and syntactic structure using an information-theoretic framework. The authors propose a modular, structure-agnostic method to quantify the mutual information between prosodic features (specifically word duration and inter-word pauses) and syntactic representations. Using multimodal language models on English speech-text corpora (audiobooks and spontaneous conversations), they find that prosody reduces syntactic uncertainty by up to 10.2%, particularly in spontaneous speech. The study supports theories like Prosodic Bootstrapping and Indirect Reference Theory, suggesting prosody carries significant but redundant syntactic information relative to text.
Entities (11)
Relation Signals (7)
Prosody → carriesinformationabout → Syntactic Structure
confidence 95% · Our results demonstrate that prosody contains measurable syntactic information...
Inter-word Pauses → reducesuncertaintyin → Syntactic Structure
confidence 90% · ...with prosodic features reducing syntactic uncertainty in spontaneous conversations by up to 10.2%.
Word Duration → reducesuncertaintyin → Syntactic Structure
confidence 90% · ...with prosodic features reducing syntactic uncertainty in spontaneous conversations by up to 10.2%.
T5 → usedtoestimate → Mutual Information
confidence 90% · We use modified T5 encoder-decoder transformer models... to predict linearized syntactic parses... to estimate this quantity
Spontaneous Conversations → exhibitshigherprosodicsyntacticinformationthan → Read Audiobooks
confidence 88% · ...carry more information in spontaneous over planned speech, as a proportion of the total structural uncertainty in each genre.
Prosodic Bootstrapping → issupportedby → Results
confidence 85% · Our findings offer new empirical support for several theoretical accounts... consistent with the Prosodic Bootstrapping theory...
Indirect Reference Theory → issupportedby → Results
confidence 85% · Our findings offer new empirical support for... the Indirect Reference approach...
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:While it is well-established that prosody carries crucial cues for syntactic structure, the degree and nature of correspondence between these two domains remains contested. We investigate the syntax-prosody interface through an information-theoretic lens, quantifying the interaction between prosodic features and syntactic representations as their mutual information. We provide a general-purpose framework for estimating this quantity over large speech-text corpora using multimodal language models. Our framework is structure-agnostic and modular, insofar as it can be used to measure the contributions of individual prosodic features or components of structure. We evaluate the syntax-prosody relationship for two features (word duration and inter-word pauses) across two domains--read audiobooks and spontaneous conversations--both in English. Our results demonstrate that prosody contains measurable syntactic information, with prosodic features reducing syntactic uncertainty in spontaneous conversations by up to 10.2%. Our findings offer new empirical support for several theoretical accounts of the syntax-prosody interface.
Tags
Links
- Source: https://arxiv.org/abs/2608.30260v1
- Canonical: https://arxiv.org/abs/2608.30260v1
Trouble viewing inline? Open PDF directly →
Full Text
77,910 characters extracted from source content.
Expand or collapse full text
Using Prosody to Predict Syntactic Structure Junghyun Min Affiliation: Georgetown University Alex Warstadt Affiliation: UC San Diego Tamar I. Regev Affiliation: MIT Tiago Pimentel Affiliation: ETH Zürich Correspondence:jm3743@georgetown.edu Ethan Gotlieb Wilcox Affiliation: Georgetown University Abstract While it is well-established that prosody carries crucial cues for syntactic structure, the degree and nature of correspondence between these two domains remains contested. We investigate the syntax-prosody interface through an information-theoretic lens, quantifying the interaction between prosodic features and syntactic representations as their mutual information. We provide a general-purpose framework for estimating this quantity over large speech-text corpora using multimodal language models. Our framework is structure-agnostic and modular, insofar as it can be used to measure the contributions of individual prosodic features or components of structure. We evaluate the syntax-prosody relationship for two features (word duration and inter-word pauses) across two domains---read audiobooks and spontaneous conversations---both in English. Our results demonstrate that prosody contains measurable syntactic information, with prosodic features reducing syntactic uncertainty in spontaneous conversations by up to 10.2%. Our findings offer new empirical support for several theoretical accounts of the syntax-prosody interface.11 1 We release code for our experiments and analyses at https://github.com/aatlantise/prosody-syntax-interface. 1 Introduction Prosody, or the “melody” of speech, carries information that is vital for successful communication. Individual components of the prosody channel—including word duration, pauses between words, pitch, and intensity—have been shown to encode information about word identity (Wolf et al., 2023; Wilcox et al., 2025), discourse functions (Wilson and Wharton, 2006; Hamlaoui et al., 2019), and meta-linguistic information (Mozziconacci, 2002; Yadavalli et al., 2025). Prosody has also long been known to carry vital cues about syntactic structure (Elfner, 2018). Children use prosody to learn the latent structure of their language (Soderstrom et al., 2003; Christophe et al., 2008); speakers use prosodic cues like pre-boundary lengthening to signal syntactic constituents (Klatt, 1975; Ferreira, 1993; Snedeker and Trueswell, 2003); and listeners rely on prosody to disambiguate between parses during real-time communication (Price et al., 1991; Millotte et al., 2007). While linguists studying the syntax–prosody interface agree that prosody and syntax interact, the manner and quantity of syntactic information carried by prosody remain debated. Different theories make different predictions about how tightly and in what ways the two are linked. For example, the Direct Reference Theory (DRT) posits that prosodic constituency is a direct representation of syntactic constituency, while the Indirect Reference Theory (IRT) allows mismatches between the two to satisfy phonological well-formedness constraints (see Section 2 for more details; Chomsky and Halle, 1968; Nespor and Vogel, 2007; Richards, 2016). These theories can be difficult to test due to limitations with current empirical methods. Much of the empirical evidence relies on case studies that examine individual phenomena in controlled settings (Klatt, 1975; Seidl, 2013; Zubizarreta, 1998). And studies that investigate the syntax–prosody relationship in larger datasets often rely on previous-generation statistical methods (Brierley, 2011), or make overly strong simplifying assumptions (Anu and Karjigi, 2014; Degano et al., 2024, e.g. omitting the effects of text context when predicting syntax;). In light of these limitations, this work proposes a new framework for measuring the relationship between syntax and prosody in large-scale natural corpora. Building on a line of recent work (Wolf et al., 2023; Regev et al., 2025; Wilcox et al., 2025; Yadavalli et al., 2025), we frame our method using the toolkit of information theory. We define syntactic information content of a prosodic feature as the mutual information (I [rgb]0.5,0.5,0.5I; Shannon, 1948) between that prosodic feature and a structural representation. In other words, syntactic information content is the extent to which knowing prosodic features reduces uncertainty over structural interpretations (i.e., syntactic parses) of a sentence. Our framework is structure-agnostic, insofar as it can be measured with respect to any type of structural representation. It is also contextual, with predictions conditionable on both the surrounding prosody or text context. And it is modular, insofar as it can be used to measure the contributions of individual prosodic features or components of a structural representation, such as phrase-boundary locations or POS tags. We implement a computational pipeline to measure the syntactic information content in two English text–speech corpora. We use modified T5 encoder-decoder transformer models (Raffel et al., 2020) to predict linearized syntactic parses, conditioned on either just text, just prosody, or prosody and text (see Section 4.1). We run a series of experiments to investigate: (1) how much syntactic information content is carried by individual prosodic features (word duration and inter-word pauses); (2) how syntactic information content changes across speech genre (planned vs. spontaneous speech); (3) how much information prosody carries beyond what is carried by the segmental information (i.e., words); and (4) what parts of the syntactic structure prosody contains information about (phrasal categories vs. phrase boundaries). Our results indicate that prosody carries information about syntactic structure, with duration and pause reducing uncertainty over syntactic representations by up to 10.2%. These features primarily signal boundary location, as opposed to phrase category identity, and carry more information in spontaneous over planned speech, as a proportion of the total structural uncertainty in each genre. We do not find evidence that prosody contains additional information about structure beyond what is carried by words; the syntactic information in duration and pause is redundant with that contained in text. We interpret our results as being consistent with the Prosodic Bootstrapping theory of syntactic acquisition (Soderstrom et al., 2003; Christophe et al., 2008) and the Indirect Reference approach to the syntax–prosody interface (Selkirk, 2011; Seidl, 2013). More broadly, we argue that our methods can go beyond previous-generation statistical approaches to reveal and quantify new statistical dependencies at the prosody-syntax interface. 2 The Syntax–Prosody Interface 2.1 Theoretical Linguistics Accounts of the syntax–prosody interface in theoretical linguistics often focus on the interplay between prosodic and syntactic well-formedness constraints. Different theories postulate different degrees of causal linking, which we argue makes predictions about the informational relationship between prosody and syntactic structure. Direct Reference Theory (DRT). DRT proposes that prosodic rules don’t employ separate phonological structures, but are formulated with respect to the underlying syntactic parses (Chomsky and Halle, 1968; Cooper and Paccia-Cooper, 1980; Kaisse and Shaw, 1985). This is supported by phonetic evidence showing that speakers consistently produce pre-boundary lengthening at the edges of syntactic constituents (Klatt, 1975). Under this view, prosody is isomorphic to syntax; mismatches between syntax and prosody are attributed to variance in underlying syntactic operations, such as extraposition (Wagner, 2015). We interpret DRT to predict a very high degree of syntactic information content, with a strict interpretation suggesting that prosodic information should come close to reducing all uncertainty about a sentence’s syntactic parse. Indirect Reference Theory (IRT). IRT argues that syntax and phonology are autonomous domains that communicate via a mediating mechanism (Nespor and Vogel, 2007; Seidl, 2013). Match Theory (Selkirk, 2011) formalizes this as a set of constraints that strive for isomorphism between syntactic and prosodic constituents, though this mapping can be overridden when certain phonological well-formedness constraints are ranked to outweigh isomorphism by Optimality Theory22 2 Optimality Theory considers grammatical rules as ranked and hierarchical; it selects the linguistic form that violates the least high-ranked rules. (Prince and Smolensky, 2004). IRT predicts a lower amount of information overlap; this is due to the fact that prosody and syntax each operate within its own framework of well-formedness constraints, with one causally influencing the other only when the two are in conflict. Prosody-Driven Syntax (PDS). PDS argues that the interface is bidirectional: syntactic well-formedness can drive prosodic choices, but prosodic well-formedness can also influence the syntax of a sentence. Certain syntactic operations like wh-movement, heavy NP shift, or argument scrambling are triggered specifically to satisfy prosodic requirements such as nuclear stress (Zubizarreta, 1998), or contiguity (Richards, 2016).33 3 Nuclear stress (Zubizarreta, 1998) describes how constituents move to align the discourse focus with the sentence’s primary phonological peak ; contiguity (Richards, 2016) posits that movement occurs to satisfy phonological phrasing requirements between a head and its dependent. We take PDS to make no strong predictions about the total degree of informational overlap, but rather about differences between genres, specifically spontaneous vs. planned speech. Because in planned (read-out-loud) speech, the syntax of a sentence is fixed, it cannot be modulated by prosodic choices. However, in spontaneous speech, as prosodic choices can cause changes in syntax, prosody should provide more information about structure. We therefore interpret PDS to predict a higher degree of syntactic information content in spontaneous, as opposed to planned speech. Prosodic Bootstrapping. Prosodic Bootstrapping is a theory of language acquisition that posits that infants use prosodic information to make inferences about the syntactic structure of a sentence, even when they are unaware of its semantics, or when it includes words that they haven’t yet learned. Proponents of the theory find evidence that infants as young as 6 months old are sensitive to prosodic markers of phrasal units (Soderstrom et al., 2003) and that they use this prosodic constituency information to constrain lexical segmentation (Christophe et al., 2008). Theories of prosodic bootstrapping predict non-zero syntactic information content in prosody, which is the basis of the bootstrapping. 2.2 Computational & Info-Theoretic Models Prosodic information has long been used as featural inputs to structure predictions in NLP systems (Kahn et al., 2005; Anu and Karjigi, 2014; Brierley, 2011; Cho, 2022; Min et al., 2025), and vice versa (Koehn et al., 2000; Dhamne et al., 2025). This study builds on a line of work quantifying the informational redundancy between prosody and non-prosodic linguistic channels. This work framed informational redundancy as mutual information (Pimentel et al., 2020) and used large language models to measure this quantity between prosody and segmental information operationalized as text (Wolf et al., 2023), pitch, and word identity (Wilcox et al., 2025), as well as to assess the temporal dynamics of the overlap (Regev et al., 2025). One shortcoming of this previous work was that because the computational pipelines made predictions over continuous-valued prosodic features, estimates (of entropy) were not bounded from below, limiting the interpretation of results. One recent contribution (Yadavalli et al., 2025) proposed a solution to this problem by using prosody to measure auxiliary discrete-valued tasks, such as emotion classification or sarcasm detection. The present work builds on this contribution, using syntax as our auxiliary task. 3 Modeling Framework In this section, we describe our formal framework for estimating the syntactic information carried by prosodic features, which we frame as the mutual information between prosody and syntax. To do so, we treat syntax ( [rgb]0.5,0.5,0 S), prosody ( [rgb]0,1,1 P), and text (W [rgb]1,0,1W) as random variables (RVs). A number of operationalizations and design choices follow. 3.1 Our Random Variables First, we consider a word, w [rgb]1,0,1w, to be an element from a vocabulary Σ . A text is then a sequence of words, ∈Σ∗ [rgb]1,0,1 w∈ ^*, which we can write as: =[w1,w2,⋯,wN] [rgb]1,0,1 w=[ [rgb]1,0,1w_1, [rgb]1,0,1w_2,·s, [rgb]1,0,1w_N]. Second, we consider a syntactic unit, s [rgb]0.5,0.5,0s, as an element from the set: Υ=∪ℬ = [rgb]0.5,0.5,0.5P∪ [rgb]0.5,0.5,0.5B, where [rgb]0.5,0.5,0.5P is a finite set of symbols representing phrasal category tags (e.g., N, SBAR), and ℬ [rgb]0.5,0.5,0.5B is a set containing an opening and closing bracket.44 4 In some of the experimental settings, we ignore phrasal categories, and thus Υ=ℬ = [rgb]0.5,0.5,0.5B. We then represent the syntax, ∈Υ∗ [rgb]0.5,0.5,0 s∈ ^*, of a text as a sequence of symbols drawn from Υ , e.g.,: (ROOT (S (NP PRP) (VP VBZ (NP DT N N)))). Finally, we treat a prosodic unit, p∈ℝ [rgb]0,1,1p , as a real-valued feature. In our experiments, this corresponds to word-level duration and pause length (measured in milliseconds; msms), but this could correspond to any prosodic feature. We then represent the prosody, ∈ℝd [rgb]0,1,1 p ^d, of a text as a sequence of prosodic units. We assume that there exists an unknown ground-truth distribution over the three variables above, p(,,)p( [rgb]1,0,1 w, [rgb]0.5,0.5,0 s, [rgb]0,1,1 p), and that for any tuple ⟨,,⟩ [rgb]1,0,1 w, [rgb]0.5,0.5,0 s, [rgb]0,1,1 p with non-zero probability, length()=length()=dlength( [rgb]1,0,1 w)=length( [rgb]0,1,1 p)=d. We use [rgb]1,0,1 W, [rgb]0.5,0.5,0 S, and [rgb]0,1,1 P to denote, respectively, sentence-level text-, syntax- and prosody-valued random variables. 3.2 Measuring Information Content We operationalize the information that prosody carries about syntax as the mutual information: I(,) [rgb]0.5,0.5,0.5I( [rgb]0,1,1 P, [rgb]0.5,0.5,0 S), following prior work (Wolf et al., 2023; Wilcox et al., 2025; Regev et al., 2025; Yadavalli et al., 2025). To estimate mutual information, we follow Pimentel et al. (2019)’s two-step process of estimation: We first decompose I [rgb]0.5,0.5,0.5I into the difference between an unconditional and a conditional entropy. We then use probabilistic models to upperbound each of these entropies (as further described in Section 3.3). With the “good mixed-pair assumption,” (Nair et al., 2007; Beknazaryan et al., 2019), the mutual information decomposes as:55 5 This assumption requires that H(∣) [rgb]0.5,0.5,0.5H( [rgb]0,1,1 P [rgb]0.5,0.5,0 s) be well-defined and finite for all values of [rgb]0.5,0.5,0 s. I(,)=H()−H(∣). [rgb]0.5,0.5,0.5I( [rgb]0,1,1 P, [rgb]0.5,0.5,0 S)= [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S)- [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]0,1,1 P). (1) In written or transcribed text, however, words carry substantial information about syntactic structure. It is thus a natural question to ask whether prosody carries information above and beyond what is carried by word identities. Beyond estimating the mutual information above, we thus additionally estimate the mutual information between prosody and syntax conditioned on word identities, I(,∣) [rgb]0.5,0.5,0.5I( [rgb]0,1,1 P, [rgb]0.5,0.5,0 S [rgb]1,0,1 W). We decompose it using the same logic as above: I(,∣)=H(∣)−H(∣,). [rgb]0.5,0.5,0.5I( [rgb]0,1,1 P, [rgb]0.5,0.5,0 S [rgb]1,0,1 W)= [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]1,0,1 W)- [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]1,0,1 W, [rgb]0,1,1 P). (2) If this value is positive, it means that prosody carries additional information about syntactic structure not carried by the text. 3.3 Entropy Estimation We follow Pimentel et al. (2019) in estimating entropy via a cross-entropy upper-bound. Let Hθ() [rgb]0.5,0.5,0.5H_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 S) be the cross-entropy between any distribution pθp_ [rgb]0.5,0.5,0.5θ and the ground truth distribution p. We can use it to upperbound the entropy via the inequality: Hθ()≥H() [rgb]0.5,0.5,0.5H_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 S)≥ [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S). The cross-entropy is defined as the expected surprisal of pθp_ [rgb]0.5,0.5,0.5θ taken with respect to p, where surprisal under pθp_ [rgb]0.5,0.5,0.5θ is defined as ι(x)=−logpθ(x) [rgb]0.5,0.5,0.5 (x)=- p_ [rgb]0.5,0.5,0.5θ(x). As p is unknown, we take a Monte Carlo estimate of the cross-entropy. Given a model pθp_ [rgb]0.5,0.5,0.5θ and a dataset n,n,nn=1N\ [rgb]1,0,1 w^n, [rgb]0.5,0.5,0 s^n, [rgb]0,1,1 p^n\_n=1^N sampled according to p, we can estimate it as: Hθ() [rgb]0.5,0.5,0.5H_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 S) =−∑∈Υ∗p()logpθ() =- _ [rgb]0.5,0.5,0 s∈ ^*p( [rgb]0.5,0.5,0 s) p_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 s) (3) ≈−1N∑n=1Nlogpθ(n),where n∼p(). ≈- 1N _n=1^N p_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 s^n), where [rgb]0.5,0.5,0 s^n p( [rgb]0.5,0.5,0 s). Analogous equations exist for the conditional entropies, e.g., Hθ(∣) [rgb]0.5,0.5,0.5H_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 S [rgb]0,1,1 P), but using the conditional distribution instead, e.g., pθ(∣)p_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 s [rgb]0,1,1 p). Notably, the difference between the cross-entropy and entropy, e.g., Hθ()−H() [rgb]0.5,0.5,0.5H_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 S)- [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S), is given by the Kullback-Leibler divergence (Kullback and Leibler, 1951) between pθp_ [rgb]0.5,0.5,0.5θ and p. This implies that, the more similar pθp_ [rgb]0.5,0.5,0.5θ and p are, the better our estimators will be. Estimating pθ()p_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 s). We train a language model from scratch on a corpus of linearized constituency parses until convergence. No conditioning information (e.g., text or prosody) is provided, and this model is thus composed of a simple decoder-only autoregressive component. Estimating conditional distributions pθ(∣)p_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 s [rgb]1,0,1 w), pθ(∣)p_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 s [rgb]0,1,1 p), and pθ(∣,)p_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 s [rgb]1,0,1 w, [rgb]0,1,1 p). We fine-tune an encoder-decoder language model to predict syntactic sequences (in the decoder) given a text sequence, a prosody sequence, or both sequences as input (in the encoder). While we train the decoder component of this model from scratch, the text encoder and cross-attention layers are loaded with pre-trained weights;66 6 From t5-base; see details in Section 4.1. this model can thus make use of text information it has learned during pre-training. All the cross-entropies reported here are over sentence-level variables, requiring no token-level normalization. 4 Methods The basis of our entropy estimation is multimodel encoder-decoder LMs which we train to predict syntactic structure from text input, prosody input, or no input. In this section, we detail our implementation pipeline, focusing on how we train LMs to predict (linearized) syntactic structures. We discuss model architectures in Section 4.1 and data (including datasets, syntactic-structure operationalisation, and statistical testing) in Section 4.2. 4.1 Encoder-Decoder Model Figure 1: Architecture of our encoder-decoder model. (a), (b), and (c) are used for estimations of H() [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S), H(∣) [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]1,0,1 W), H(∣,) [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]1,0,1 W, [rgb]0,1,1 P), respectively. To estimate H(∣) [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]0,1,1 P), we use an architecture symmetric to (b), but with an active Prosody encoder instead. Our sequence-to-sequence model is based on the base-sized encoder-decoder T5 architecture (Raffel et al., 2020), with several additions, illustrated in Figure 1. In order to process prosodic inputs, we construct a new prosody encoder from scratch, which differs from T5’s text encoder in several respects: It is shallower and narrower, with 2 layers, 2 heads per layer, and a hidden dimensionality of 88. Rather than learning an embedding matrix during training, we used a fixed embedding schema, with two sinusoidal functions used to construct an encoding of the prosodic feature’s magnitude MEME and absolute position encoding PEPE, given below. The two encodings are then concatenated before being passed to the transformer stack: ME(,d)=sin(200002i/D)if d=2icos(200002i/D)if d=2i+1 _( [rgb]0.5,0.5,0.5mag,d)= cases ( [rgb]0.5,0.5,0.5mag20000^2i/D )&if d=2i\\ ( [rgb]0.5,0.5,0.5mag20000^2i/D )&if d=2i+1\\ cases (4) PE(,d)=sin(100002i/D)if d=2icos(100002i/D)if d=2i+1 _( [rgb]0.5,0.5,0.5pos,d)= cases ( [rgb]0.5,0.5,0.5pos10000^2i/D )&if d=2i\\ ( [rgb]0.5,0.5,0.5pos10000^2i/D )&if d=2i+1\\ cases (5) Where [rgb]0.5,0.5,0.5mag is the prosodic feature binned into increments of 10ms10ms. The final hidden state of the encoder is projected to match the hidden state dimensions of the text encoder to allow for stable fusion via cross-attention to the decoder stack. For estimates of pθ(∣,)p_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 s [rgb]0,1,1 p, [rgb]1,0,1 w) that condition on both prosody and text, we add a cross-attention layer to fuse inputs across modalities. The cross-attention is performed with the layer’s query as text representation and key and value as prosody representations. Dropout and layer normalization are applied to the resulting fused representations. We add part-of-speech labels as additional tokens to the tokenizer, with one (and only one) token per POS label; we assign each of these new tokens a randomly initialized embedding. These additional tokens are used strictly for decoding (i.e., for predicting syntax) and are not used to tokenize the input text [rgb]1,0,1 w. The text encoder, embedding layers, and cross-attention layers are loaded with available t5-base pre-trained weights (Raffel et al., 2020). The decoder, prosody encoder, fusion layers, and LM head are randomly initialized and trained from scratch to predict syntactic parse tokens. We report hyperparameters in Appendix B. 4.2 Datasets Previous work using methodologies similar to ours has focused on prepared, read-out-loud speech (Wolf et al., 2023; Regev et al., 2025; Wilcox et al., 2025). To investigate how prosody may vary across speech styles, we consider two datasets in our experiments: LibriTTS is a dataset of audiobooks and their gold transcriptions,77 7 This may underestimate prosody’s information content; see Limitations. derived from the LibriSpeech corpus (Panayotov et al., 2015), with audiobooks from LibriVox,88 8 https://librivox.org/ and the original material from Project Gutenberg.99 9 https://w.gutenberg.org/ We use the aligned version of its clean subset using the Montreal Forced Aligner (McAuliffe et al., 2017, MFA; ), which comprises 44k sentences. Our second dataset is CANDOR (Conversation: A Naturalistic Dataset of Online Recordings), which consists of 1656 naturalistic, unscripted conversations between strangers that were collected over Zoom, whose transcription pipeline includes automatic speech recognition (ASR). We preprocess the dataset via the pipeline from Clark et al. (2025) and obtain pause and duration information, again by aligning them with MFA. Out of 750k sentences that are 5 words or longer, we filter for those that are not non-initial fragments due to speaker backchannel interruptions, resulting in 298k sentences. Punctuation marks are removed from the input text (as well as associated syntactic parse), as they are not part of word identities and carry syntactic information (Chafe, 1988). We report the dataset statistics in Table 1. Metric CANDOR LibriTTS Dataset size (sentences) 298k 44k Avg. words per sentence 13.2 17.5 Avg. tokens per full parse 64 76 Avg. tokens per brackets 33 38 Table 1: Dataset statistics for CANDOR (spontaneous speech) and LibriTTS (planned speech). Syntactic parses. Although our method could be employed with any structural representation, for our experiments, we represent syntactic structure as linearized constituency parses, following the Penn Treebank format. As we are interested in what types of syntactic information prosody contains, we generate two types of constituency trees, corresponding to differing levels of parse granularity: For our full parse condition, we obtain silver parse labels on our two datasets using Stanza (Qi et al., 2020).1010 10 Stanza employs the Penn Treebank framework. The framework processes disfluencies in its spoken subsets like Switchboard (Godfrey et al., 1992) by isolating them and annotating the remaining sentence. Stanza’s training data was not sourced from the spoken genres. From the silver parses, we remove any punctuation marks. For our bracket-only condition, we strip out all phrase category information, leaving only brackets that indicate the hierarchical structure of the sentence. Because we use autoregressive language models to estimate pθ()p_ [rgb]0.5,0.5,0.5θ( [rgb]0.5,0.5,0 s), it’s possible that the model can assign probability to non-valid syntactic parses. Thus, a certain amount of our probability can leak. To address this concern, we estimate the amount of leaked probability in Appendix A. Prosodic features. We measure the syntactic information content of word duration and inter-word pauses. We choose these features because they have been argued to provide cues for syntactic structure (Klatt, 1975; Ratner, 1986). Duration is the difference between the word’s onset and offset time, as taken from MFA-aligned word boundaries. Inter-word pauses are taken as the difference between a word’s offset and the onset of the next word; again, taken from MFA-aligned boundaries. Duration is normalized by the number of syllables in a word, using syllable data from CELEX (Baayen et al., 1995). Both measures are taken in milliseconds, but chunked into 10ms10ms bins in our prosody encoder (e.g. values 14ms14ms and 18ms18ms are represented in the same 10−19ms10-19ms bin). Figure 2: Estimated entropy in predicted syntactic structure given context across conditions. Genre Parse Type H() [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S) H(∣) [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]1,0,1 W) H(∣) [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]0,1,1 P_ pause) H(∣,) [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]1,0,1 W, [rgb]0,1,1 P_ pause) H(∣) [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]0,1,1 P_ dur) H(∣,) [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]1,0,1 W, [rgb]0,1,1 P_ dur) Spontaneous Brackets 13.9 2.83 12.7 3.03 12.4 3.06 Full parse 37.2 5.25 35.1 5.27 33.7 5.30 Planned Brackets 16.9 5.76 16.1 6.97 16.0 7.13 Full parse 48.2 12.8 46.2 13.6 46.6 13.5 Table 2: Syntactic entropy (nats per sequence) across datasets and input conditions. LibriTTS is a dataset of planned read-out-loud speech, while CANDOR is a dataset of naturalistic, spontaneous conversations. (,)( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P_ pause) (,)( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P_ dur) (,)( [rgb]0.5,0.5,0 S, [rgb]1,0,1 W) (,∣)( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P_ pause [rgb]1,0,1 W) (,∣)( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P_ dur [rgb]1,0,1 W) Genre Parse Type I [rgb]0.5,0.5,0.5I U [rgb]0.5,0.5,0.5U I [rgb]0.5,0.5,0.5I U [rgb]0.5,0.5,0.5U I [rgb]0.5,0.5,0.5I U [rgb]0.5,0.5,0.5U I [rgb]0.5,0.5,0.5I U [rgb]0.5,0.5,0.5U I [rgb]0.5,0.5,0.5I U [rgb]0.5,0.5,0.5U Spontaneous Brackets 1.20 8.64% 1.42 10.2% 11.0 79.6% −-0.20 −-7.14% −-0.24 −-8.32% Full parse 2.05 5.52% 3.45 9.27% 31.9 85.9% −-0.02 −-0.42% −-0.06 −-1.07% Planned Brackets 0.75 4.44% 0.84 4.99% 11.1 65.9% −-1.21 −-21.0% −-1.37 −-23.8% Full parse 2.06 4.27% 1.65 3.43% 35.4 73.5% −-0.89 −-6.95% −-0.73 −-5.70% Table 3: Syntactic information content in input feature as mutual information I [rgb]0.5,0.5,0.5I and uncertainty coefficient U [rgb]0.5,0.5,0.5U. All positive I [rgb]0.5,0.5,0.5I values are statistically significant (p<0.001p<0.001). Statistical tests. We test the statistical difference in sentence-level surprisal across conditions using our cross-validation splits. To this end, we use a Wilcoxon signed-rank test (Wilcoxon, 1945), as sequence-level surprisal values are paired across conditions. Notably, when applied to our setting, this test evaluates whether our MI estimates are positive significantly more often than expected by chance, and not if its average value is larger than zero across splits—thus being more resistant to outlier negative estimation errors (note that I [rgb]0.5,0.5,0.5I cannot, by definition, be negative). 5 Results Table 2 and Figure 2 outline our entropy estimates H( ) [rgb]0.5,0.5,0.5H($ [rgb]0.5,0.5,0 S$ ), H( ∣ ) [rgb]0.5,0.5,0.5H($ [rgb]0.5,0.5,0 S$ $ [rgb]0,1,1 P$ ), H( ∣ ) [rgb]0.5,0.5,0.5H($ [rgb]0.5,0.5,0 S$ $ [rgb]1,0,1 W$ ), H( ∣, ) [rgb]0.5,0.5,0.5H($ [rgb]0.5,0.5,0 S$ $ [rgb]0,1,1 P$,$ [rgb]1,0,1 W$ ) across input conditions and domain. Table 3 shows mutual information I [rgb]0.5,0.5,0.5I and uncertainty coefficient U [rgb]0.5,0.5,0.5U, defined as the ratio between I [rgb]0.5,0.5,0.5I and H() [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S). Prosody is labeled as [rgb]0,1,1 P_ pause and [rgb]0,1,1 P_ dur, when representing, respectively, pause and duration information. Overall, pause and duration carry similar amounts of syntactic information content in each condition. As noted by previous language modeling work (Godfrey et al., 1992; Zen et al., 2019, e.g.,), naturalistic, conversational speech has lower syntactic uncertainty across all conditions: both full and bracket parses, and both with and without text and prosody input. This may be due to a number of factors, including CANDOR’s larger dataset size that would allow a greater degree of convergence (298k vs. 44k samples), or the longer average sentence length in LibriTTS compared to CANDOR (17.5 vs. 13.2 words per sentence) and thus longer average syntax parses (76 vs. 64 tokens). It is also possible that the syntactic structures found in naturalistic, conversational speech are more predictable than those in recorded audiobooks. Answering RQ1: How much information about syntax is carried by word duration and pause? Our experiments show that both pause and duration carry measurable syntactic information, in both spontaneous and planned speech. Our statistical tests show that the sequence-level syntactic information I(,) [rgb]0.5,0.5,0.5I( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P_ pause) and I(,) [rgb]0.5,0.5,0.5I( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P_ dur) is significantly greater than zero in all input conditions (p<10−4p<10^-4). We find that pause can contain up to 2.06 nats of syntactic information, while duration can contain up to 3.45 nats. Words alone carry about 30 nats of syntactic information, suggesting that pause and duration carry around 10% of the information load that words do about syntax. Answering RQ2: Do prosodic features carry syntactic information beyond what is carried by words? We find that after controlling for the syntactic information already carried by words, both pause and duration carry little or no significant measurable syntactic information in planned speech. We hypothesize this may be due to the sparsity of sentences and constructions where syntax could be disambiguated with prosody. Our framework may also underestimate the syntactic information beyond what is carried by words, a possibility which we discuss in greater detail in Limitations. For the rest of the results section, we focus on the individual contributions of prosody (that is, I(,) [rgb]0.5,0.5,0.5I( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P), rather than on I(,∣) [rgb]0.5,0.5,0.5I( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P [rgb]1,0,1 W)). Answering RQ3: How does information content vary across speech styles? Duration carries more syntactic information in spontaneous than in planned speech, when quantified both in terms of total mutual information and uncertainty reduction. Pauses have higher total mutual information in spontaneous speech for our full parse condition, but less in our bracket-parse condition. However, when quantified in terms of uncertainty reduction, pauses contribute to a greater reduction in spontaneous speech in both conditions. The overall trend in our results is clear: in the datasets we use, prosody carries more syntactic information in spontaneous, rather than planned, speech. Sentence length may be a contributing factor in the difference, although utterance length differences could be thought of as an inherent part of the differences between these two genres. An analysis in Section 6 controlling for the length shows that the cross-genre difference remains significant after controlling for the length distribution. Answering RQ4: What parts of the syntactic structure does prosody contain information about? In both spontaneous and planned speech, phrase boundaries (represented by brackets) account for the majority of syntactic information content in prosody. Phrase boundaries contain up to around 50% of the syntactic information content in duration; 1.42 out of 3.45 nats in spontaneous speech and 0.84 out of 1.65 nats in planned speech. They contain up to 60% of the syntactic information content in pause; 1.20 out of 2.05 nats in spontaneous speech and 0.75 out of 2.06 nats in planned speech. This finding is in line with well-established phonological phenomena that connect prosodic features to phrase-structural boundaries, for example, pre-boundary lengthening (Klatt, 1975; Ferreira, 1993). 6 Sentence Features and Disfluencies As described in Section 3.3, our reported cross-entropies are over sentence-level variables, without token-level normalization. However, at the sentence level, it’s possible that estimates of syntactic information content interact with sequence-level properties, such as length and syntactic complexity. Thus, we perform three follow-up analyses, all using pointwise estimates of syntactic information content—ι(s)−ι(s,p) [rgb]0.5,0.5,0.5 ( [rgb]0.5,0.5,0s)- [rgb]0.5,0.5,0.5 ( [rgb]0.5,0.5,0s, [rgb]0,1,1p), i.e., the syntactic information of a single sequence—which are averaged across a dataset to obtain I(,) [rgb]0.5,0.5,0.5I( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P) values in the previous sections. Effects of sentence length and parse on sentence-level syntactic information content. We inspect the Pearson correlation between the pointwise syntactic information content and two sequence-level features: length l, as measured in number of words, and maximum tree depth 0pt. We conduct a separate correlation analysis for each dataset and each parse type (i.e., full parse and bracket-only). Figure 3: CANDOR sentence features versus pointwise syntactic information content. Each bin represents a single parse depth or sentence length value. We find that l and 0ptare weakly inversely correlated with syntactic information content in pause and duration in planned speech (|r|<0.05|r|<0.05 for all conditions, p<0.001p<0.001 in all conditions but full parse information carried by pause). We detail correlations in planned speech in Appendix C.1. In spontaneous speech, we observe a significant, moderate correlation for l (pause: r=0.52r=0.52; duration: r=0.42r=0.42) and 0pt(pause: r=0.42r=0.42; duration: r=0.39r=0.39), when considering phrase boundary information. When considering full parses, the correlations are still significant, but slightly weaker for both l (pause: r=0.29r=0.29; duration: r=0.25;r=0.25;) and 0pt(pause: r=0.23r=0.23; duration: r=0.23r=0.23). For all reported r values, p<0.001p<0.001. Overall, this analysis suggests that, when it comes to spontaneous speech, pause and duration contain greater phrase boundary information content in longer and syntactically deeper sentences. Effects of sentence length on cross-genre trends. In Section 5, we find that prosody carries more syntactic information in spontaneous speech. However, since length can directly influence the amount of sentence-level syntactic information, we perform a follow-up analysis controlling for sentence length. We sample subsets of both speech datasets so that the sample includes the same number of sentences for each length in number of words (5≤l≤255≤ l≤ 25). Even under exact length matching, spontaneous speech yields significantly higher syntactic information content for both duration (3.593.59 vs. 1.551.55 nats; p<.001p<.001) and pause (2.132.13 vs. 1.941.94 nats; p<10−17p<10^-17). Effects of disfluencies in spontaneous speech. We conduct a post-hoc analysis to assess the prevalence of disfluencies in our data and their impact on our results. While our data processing pipelines filter for sentence-initial disfluencies like restarts, we do not filter for sentence-medial disfluencies like repetitions and filled pauses, whose effects in our findings are unclear. On one hand, prosody could be used to signal disfluency, making it easier for a listener to focus on the parts of the sentence that are strictly necessary for syntactic interpretation. On the other hand, disfluencies could add noise into the prosody stream, making relevant information harder to extract. For this analysis, we define repetitions as identical adjacent tokens and filled pauses as the tokens um, uh, er, ah, and hmm. We find that 19.8% of our processed CANDOR sentences contain disfluencies. While I [rgb]0.5,0.5,0.5I is higher in disfluent subsets for all conditions, disfluent sentences are much longer (l¯=17.9 l=17.9 words) than fluent sentences (l¯=11.99 l=11.99 words). A length-controlled analysis across fluency subsets shows that length is the primary driver of such difference, and disfluencies have a limited impact on our results. We detail the analysis in Appendix C.2. 7 Discussion Implications for linguistic theories. Our results support the mainstream linguistic theories of the syntax-prosody interface outlined in Section 2. Starting with our two main theories of well-formedness, recall that Direct Reference Theory predicts a very high degree of informational overlap between prosody and syntax, whereas Indirect Reference predicts a moderate level. Our results support indirect reference—pauses and duration do carry syntactic information, but only up to about 10% of the total. Prosody-Driven Syntax predicts a higher degree of information in spontaneous speech over planned speech. Our results are in line with this prediction: I(,) [rgb]0.5,0.5,0.5I( [rgb]0,1,1 P, [rgb]0.5,0.5,0 S) values are higher in spontaneous speech, and U [rgb]0.5,0.5,0.5U values are uniformly higher here, too, by up to ≈6%p≈ 6\%p. Finally, our results can be interpreted as strong support for prosodic bootstrapping. We find that, even if a learner knows no words that are uttered in a sentence, they could still extract useful information about its syntax from prosody alone. Prosody as an error-correcting code. What do our results suggest about the role of prosody in language comprehension, more broadly? First and unsurprisingly, our results indicate that prosody largely signals phrase boundary information, and not phrase identities. This corroborates recent research with similar findings (Degano et al., 2024). Of course, we did not study all prosodic features, and others, such as pitch, may carry more information about identity (Wilcox et al., 2025, see e.g.,). Second, prosody plays a largely redundant role for signaling syntax, when word identity is already known. This suggests that, when it comes to syntax, prosody can be thought of as an error-correcting code (MacKay, 2003), i.e., information that can be used to reconstruct a message if the original signal is corrupted. Given that different linguistic channels may be corrupted by different sources of noise, using a separate prosodic channel as a source of redundancy for important syntactic information may be optimal given resource constraints. To this point, in Section 6, we find that syntactic information content in disfluent spontaneous sentences is higher than that in fluent spontaneous sentences. One possibility is that speakers adjust their prosodic usage in real-time to amplify its error-correcting role, for example to compensate for making disfluencies. However, prosody as an error-correcting signal may not extend beyond syntax; recent work by Yadavalli et al. (2025) has shown that prosody contributes significant information above and beyond words for other tasks, such as emotion detection. Generalization beyond English. Our findings are from English data only; there is no guarantee that they generalize to other languages. Given that pre-boundary lengthening and pause-based boundary marking are consistently observed cross-lingually (Vaissière, 1983; Fletcher, 2010), we predict that our high-level findings will hold when our methods are applied to multilingual datasets. However, the redundancy between syntactic information carried by words and that by prosodic features may vary significantly cross-lingually, as phonological richness (e.g., tonality) or word order flexibility may affect how prosodic features are used to signal syntactic structure. 8 Conclusion In this work, we design and implement a pipeline to measure syntactic information content in two prosodic features: word duration and pause. We confirm existing linguistic theories that prosodic features do carry measurable, statistically significant syntactic information content. More broadly, this work presents novel, machine learning methods for measuring the interaction between prosody and syntax over large corpora. We anticipate that our work will be useful for exploring a range of phenomena at the prosody-syntax interface, across a larger suite of datasets and languages. Limitations Limitations with our approach. Our cross-entropy estimates of entropy, which we use to measure the syntactic information content via mutual information, are not unbiased estimates. The better our model in predicting [rgb]0.5,0.5,0 S, the lower the estimated entropy may be. Thus, our estimates are an upper bound of the actual uncertainty in predicting [rgb]0.5,0.5,0 S from [rgb]1,0,1 W and [rgb]0,1,1 P. We consider the pre-trained text encoder to be an effective representation of [rgb]1,0,1 W. On the other hand, the prosody encoder which is trained from scratch and the fused representation of prosody and text encoder are likely less effective representations of [rgb]0,1,1 P and [rgb]1,0,1 W. Thus, our estimates of H(∣) [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]1,0,1 W) may be a much better estimate than our estimates of H(∣,) [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]1,0,1 W, [rgb]0,1,1 P), leading to a deflated measurement of I(,∣) [rgb]0.5,0.5,0.5I( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P [rgb]1,0,1 W). The negative I [rgb]0.5,0.5,0.5I estimates in Table 3 reflect this, as mutual informations are by definition always positive. Additionally, while we separately measure the syntactic information content in two features: duration and pause, it is possible that they contain redundant syntactic information. Future work may benefit from combining the two features, or estimating the redundant information content between the two features. Finally, we use gold transcripts for planned speech, versus ASR silver transcripts for spontaneous speech. This may reduce the total amount of syntax predictable from text, and thus affect our mutual information estimates. Model limitations. Despite the rising popularity of multimodality in language processing Zhang et al. (2024a), methods to reliably fuse text and real-valued input modes lag behind the likes of vision language models (Zhang et al., 2024b; Beyer et al., 2024) and speech and audio language models (Chu et al., 2023; Borsos et al., 2023). We believe our architecture can be further improved. First, the internal architecture of our prosody encoder can be optimized further via hyperparameter search. Our current encoders are fairly small and shallow, and it is possible that a deeper and larger encoder may be more effective. In addition, earlier fusion of text and prosody input may also positively affect model performance, to allow interaction between the two modes at the encoder stage. Second, although the output space of our experiments is extremely limited to 66 tokens representing POS and constituency labels and brackets, we currently utilize the entire pre-trained (Raffel et al., 2020) vocabulary. This setup may result in probability leakage; the models may be assigning probabilities to illegal tokens during inference, adversely affecting our entropy estimates. We investigate probability leakage in Appendix A. Decoupling input and output tokenizers may help estimate entropy in syntax prediction more effectively. Furthermore, models are known to be biased by tokenization, and to systematically assign less probability to strings with more tokens (Lesci et al., 2025); while we control for this in our spontaneous vs. read speech in Section 6, this may affect the correlations in Figure 3. Finally, the token embedding layer can also be decoupled between the encoder and the decoder. Other experimental limitations. In our study, we investigate the effects of only two prosodic features—pause and duration. However, there are many others, including pitch, loudness, tempo, and prominence, which is a composite prosodic feature. In addition, our pipeline used silver parses, meaning that errors in the parse itself may be propagated and impact our results. Finally, we only investigate English. The use of prosody is known to differ between languages, and running experiments on a wider set of languages is a necessary step to establish the generalizability of our results. Acknowledgements We thank Nathan Schneider, Amir Zeldes, members of the Georgetown University PICoL Lab, and anonymous reviewers for their help in shaping this work. This work was made possible in part by support from Georgetown University HPC (managed by Woonki Chung), the CLI Leon Node (managed by Abhishek Purushothama and Dan DeGenaro), and the Digital Research Alliance of Canada / Alliance de recherche numérique du Canada. Ethical Considerations Our work mainly concerns speech data, its transcriptions, and automatically generated annotations from NLP systems. The datasets and NLP systems are publicly available; we do not report any potential issues regarding privacy or safety. However, we acknowledge that our work is relevant to and contributes to research around large language models (LLMs) and other systems that are capable of generating and processing language data, possibly for malicious purposes. Finally, we disclose the use of commercial LLMs as brainstorming, coding, and writing assistants during the design and implementation of the experiments and during the writing of this paper. References Anu and Karjigi (2014) J.P. Anu and V. Karjigi Sentence segmentation for speech processing. In 2014 IEEE National Conference on Communication, Signal Processing and Networking (NCCSN), Vol. , p. 1–4. External Links: Document Cited by: §1, §2.2. Baayen et al. (1995) R. H. Baayen, R. Piepenbrock, and L. Gulikers The CELEX2 lexical database (release 2); LDC96L14. Distributed by the Linguistic Data Consortium, University of Pennsylvania, web download. Cited by: §4.2. Beknazaryan et al. (2019) A. Beknazaryan, X. Dang, and H. Sang On mutual information estimation for mixed-pair random variables. Statistics & Probability Letters 148, p. 9–16. External Links: ISSN 0167-7152, Document, Link Cited by: §3.2. Beyer et al. (2024) L. Beyer, A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang, D. Salz, M. Neumann, I. Alabdulmohsin, M. Tschannen, E. Bugliarello, T. Unterthiner, D. Keysers, S. Koppula, F. Liu, A. Grycner, A. A. Gritsenko, N. Houlsby, M. Kumar, K. Rong, J. Eisenschlos, R. Kabra, M. Bauer, M. Bosnjak, X. Chen, M. Minderer, P. Voigtlaender, I. Bica, I. Balazevic, J. Puigcerver, P. Papalampidi, O. J. Hénaff, X. Xiong, R. Soricut, J. Harmsen, and X. Zhai PaliGemma: A versatile 3B VLM for transfer. CoRR abs/2407.07726. External Links: Link Cited by: Model limitations.. Borsos et al. (2023) Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghidour AudioLM: A Language Modeling Approach to Audio Generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 (), p. 2523–2533. External Links: Document Cited by: Model limitations.. Brierley (2011) C. Brierley Prosody resources and symbolic prosodic features for automated phrase break prediction. Ph.D. Thesis, University of Leeds, University of Leeds. Note: Unpublished External Links: Link, ISBN 978-0-85731-125-2 Cited by: §1, §2.2. Chafe (1988) W. Chafe Punctuation and the prosody of written language. Written Communication 5 (4), p. 395–426. External Links: Document, Link, https://doi.org/10.1177/0741088388005004001 Cited by: §4.2. Cho (2022) J. Y. Cho Leveraging prosody for punctuation prediction of spontaneous speech. Ph.D. Thesis, University of Washington, (English). Note: Copyright - Database copyright ProQuest LLC; ProQuest does not claim copyright in the individual underlying works; Last updated - 2023-03-08 External Links: ISBN 9798837529122, Link Cited by: §2.2. Chomsky and Halle (1968) N. Chomsky and M. Halle The sound pattern of english. Studies in English, Harper & Row. External Links: ISBN 9780060412760, LCCN 67023446, Link Cited by: §1, §2.1. Christophe et al. (2008) A. Christophe, S. Millotte, S. Bernal, and J. Lidz Bootstrapping lexical and syntactic acquisition. Language and Speech 51 (1-2), p. 61–75. Note: PMID: 18561544 External Links: Document, Link, https://doi.org/10.1177/00238309080510010501 Cited by: §1, §1, §2.1. Chu et al. (2023) Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models. External Links: 2311.07919, Link Cited by: Model limitations.. Clark et al. (2025) T. H. Clark, M. Poliak, T. Regev, A. J. Haskins, C. Robertson, and E. Gibson The relationship between surprisal and prosodic prominence in conversation reflects intelligibility-oriented pressures. Cognitive Science 49 (10), p. e70134. External Links: Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1111/cogs.70134 Cited by: §4.2. Cooper and Paccia-Cooper (1980) W.E. Cooper and J. Paccia-Cooper Syntax and speech. Cognitive science series, Harvard University Press. External Links: ISBN 9780674860759, LCCN lc80016614, Link Cited by: §2.1. Degano et al. (2024) G. Degano, P. W. Donhauser, L. Gwilliams, P. Merlo, and N. Golestani Speech prosody enhances the neural processing of syntax. Communications Biology 7 (1), p. 748. External Links: Link Cited by: §1, §7. Dhamne et al. (2025) M. Dhamne, S. Raman, and P. Rao Predicting prosodic boundaries for children’s texts. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, C. Christodoulopoulos, T. Chakraborty, C. Rose, and V. Peng (Eds.), Suzhou, China, p. 31863–31873. External Links: Link, Document, ISBN 979-8-89176-332-6 Cited by: §2.2. Elfner (2018) E. Elfner The syntax-prosody interface: Current theoretical approaches and outstanding questions. Linguistics Vanguard 4 (1). External Links: Link, Document Cited by: §1. Ferreira (1993) F. Ferreira Creation of prosody during sentence production.. Psychological Review 100 (2), p. 233. Cited by: §1, §5. Fletcher (2010) J. Fletcher The prosody of speech: timing and rhythm. In The Handbook of Phonetic Sciences, p. 521–602. External Links: ISBN 9781444317251, Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781444317251.ch15 Cited by: §7. Godfrey et al. (1992) J.J. Godfrey, E.C. Holliman, and J. McDaniel SWITCHBOARD: telephone speech corpus for research and development. In [Proceedings] ICASSP-92: 1992 IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. 1, p. 517–520 vol.1. External Links: Document Cited by: §5, footnote 10. Hamlaoui et al. (2019) F. Hamlaoui, M. Żygis, J. Engelmann, and M. Wagner Acoustic correlates of focus marking in Czech and Polish. Language and Speech 62 (2), p. 358–377. Cited by: §1. Kahn et al. (2005) J. G. Kahn, M. Lease, E. Charniak, M. Johnson, and M. Ostendorf Effective use of prosody in parsing conversational speech. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, R. Mooney, C. Brew, L. Chien, and K. Kirchhoff (Eds.), Vancouver, British Columbia, Canada, p. 233–240. External Links: Link Cited by: §2.2. Kaisse and Shaw (1985) E. M. Kaisse and P. A. Shaw On the theory of lexical phonology. Phonology Yearbook 2, p. 1–30. External Links: ISSN 02658062, Link Cited by: §2.1. Klatt (1975) D. H. Klatt Vowel lengthening is syntactically determined in a connected discourse. Journal of Phonetics 3 (3), p. 129–140. External Links: ISSN 0095-4470, Document, Link Cited by: §1, §1, §2.1, §4.2, §5. Koehn et al. (2000) P. Koehn, S. Abney, J. Hirschberg, and M. Collins Improving intonational phrasing with syntactic information. In 2000 IEEE International Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No. 00CH37100), Vol. 3, p. 1289–1290. Cited by: §2.2. Kullback and Leibler (1951) S. Kullback and R. A. Leibler On information and sufficiency. The Annals of Mathematical Statistics 22 (1), p. 79–86. External Links: ISSN 00034851, Link Cited by: §3.3. Lesci et al. (2025) P. Lesci, C. Meister, T. Hofmann, A. Vlachos, and T. Pimentel Causal estimation of tokenisation bias. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 28325–28340. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: Model limitations.. MacKay (2003) D. J. MacKay Information theory, inference and learning algorithms. Cambridge University Press. External Links: Link Cited by: §7. McAuliffe et al. (2017) M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger Montreal Forced Aligner: Trainable Text-Speech Alignment Using Kaldi. In Proceedings of Interspeech 2017, Stockholm, Sweden, p. 498–502. External Links: Document Cited by: §4.2. Millotte et al. (2007) S. Millotte, R. Wales, and A. Christophe Phrasal prosody disambiguates syntax. Language and Cognitive Processes 22 (6), p. 898–909. External Links: Document, Link, https://doi.org/10.1080/01690960701205286 Cited by: §1. Min et al. (2025) J. Min, M. Lee, W. Lee, and Y. Lee Punctuation restoration improves structure understanding without supervision. In Proceedings of the 10th Workshop on Representation Learning for NLP (RepL4NLP-2025), V. Adlakha, A. Chronopoulou, X. L. Li, B. P. Majumder, F. Shi, and G. Vernikos (Eds.), Albuquerque, NM, p. 120–130. External Links: Link, Document, ISBN 979-8-89176-245-9 Cited by: §2.2. Mozziconacci (2002) S. Mozziconacci Prosody and emotions. In Speech Prosody 2002, p. 1–9. External Links: Document, ISSN 2333-2042 Cited by: §1. Nair et al. (2007) C. Nair, B. Prabhakar, and D. Shah On entropy for mixtures of discrete and continuous variables. External Links: cs/0607075, Link Cited by: §3.2. Nespor and Vogel (2007) M. Nespor and I. Vogel Prosodic phonology: with a new foreword. Studies in generative grammar, Mouton de Gruyter. External Links: ISBN 9783110197907, LCCN 2007039632, Link Cited by: §1, §2.1. Panayotov et al. (2015) V. Panayotov, G. Chen, D. Povey, and S. Khudanpur LibriSpeech: An ASR corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Vol. , p. 5206–5210. External Links: Document Cited by: §4.2. Pimentel et al. (2019) T. Pimentel, A. D. McCarthy, D. Blasi, B. Roark, and R. Cotterell Meaning to form: measuring systematicity as information. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, A. Korhonen, D. Traum, and L. Màrquez (Eds.), Florence, Italy, p. 1751–1764. External Links: Link, Document Cited by: §3.2, §3.3. Pimentel et al. (2020) T. Pimentel, J. Valvoda, R. H. Maudslay, R. Zmigrod, A. Williams, and R. Cotterell Information-theoretic probing for linguistic structure. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault (Eds.), Online, p. 4609–4622. External Links: Link, Document Cited by: §2.2. Price et al. (1991) P. J. Price, M. Ostendorf, S. Shattuck‐Hufnagel, and C. Fong The use of prosody in syntactic disambiguation. The Journal of the Acoustical Society of America 90 (6), p. 2956–2970. External Links: ISSN 0001-4966, Document, Link, https://pubs.aip.org/asa/jasa/article-pdf/90/6/2956/12175823/2956_1_online.pdf Cited by: §1. Prince and Smolensky (2004) A. Prince and P. Smolensky Optimality theory: constraint interaction in generative grammar. In Optimality Theory in Phonology, p. 1–71. External Links: ISBN 9780470756171, Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1002/9780470756171.ch1 Cited by: §2.1. Qi et al. (2020) P. Qi, Y. Zhang, Y. Zhang, J. Bolton, and C. D. Manning Stanza: a python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, A. Celikyilmaz and T. Wen (Eds.), Online, p. 101–108. External Links: Link, Document Cited by: §4.2. Raffel et al. (2020) C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research 21 (140), p. 1–67. External Links: Link Cited by: §1, §4.1, §4.1, Model limitations.. Ratner (1986) N. B. Ratner Durational cues which mark clause boundaries in mother-–child speech. Journal of Phonetics 14 (2), p. 303–309. External Links: ISSN 0095-4470, Document, Link Cited by: §4.2. Regev et al. (2025) T. I. Regev, C. Ohams, S. Xie, L. Wolf, E. Fedorenko, A. Warstadt, E. Wilcox, and T. Pimentel The time scale of redundancy between prosody and linguistic context. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 30476–30488. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1, §2.2, §3.2, §4.2. Richards (2016) N. Richards Contiguity theory. Linguistic Inquiry Monographs, MIT Press. External Links: ISBN 9780262332347, Link Cited by: §1, §2.1, footnote 3. Seidl (2013) A. Seidl Minimal indirect reference: a theory of the syntax-phonology interface. Outstanding Dissertations in Linguistics, Taylor & Francis. External Links: ISBN 9781136710216, Link Cited by: §1, §1, §2.1. Selkirk (2011) E. Selkirk The syntax-phonology interface. In The Handbook of Phonological Theory, p. 435–484. External Links: ISBN 9781444343069, Document, Link, https://onlinelibrary.wiley.com/doi/pdf/10.1002/9781444343069.ch14 Cited by: §1, §2.1. Shannon (1948) C. E. Shannon A mathematical theory of communication. The Bell System Technical Journal 27 (3), p. 379–423. External Links: Document Cited by: §1. Snedeker and Trueswell (2003) J. Snedeker and J. Trueswell Using prosody to avoid ambiguity: effects of speaker awareness and referential context. Journal of Memory and Language 48 (1), p. 103–130. External Links: ISSN 0749-596X, Document, Link Cited by: §1. Soderstrom et al. (2003) M. Soderstrom, A. Seidl, D. G. K. Nelson, and P. W. Jusczyk The prosodic bootstrapping of phrases: evidence from prelinguistic infants. Journal of Memory and Language 49 (2), p. 249–267. Cited by: §1, §1, §2.1. Vaissière (1983) J. Vaissière Language-independent prosodic features. In Prosody: Models and Measurements, A. Cutler and D. R. Ladd (Eds.), p. 53–66. External Links: ISBN 978-3-642-69103-4, Document, Link Cited by: §7. Wagner (2015) M. Wagner Phonological evidence in syntax. Syntax–theory and analysis: An international handbook, p. 1154–1198. Cited by: §2.1. Wilcox et al. (2025) E. Wilcox, C. Ding, G. Acampa, T. Pimentel, A. Warstadt, and T. I. Regev Using information theory to characterize prosodic typology: the case of tone, pitch-accent and stress-accent. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), W. Che, J. Nabende, E. Shutova, and M. T. Pilehvar (Eds.), Vienna, Austria, p. 24439–24451. External Links: Link, Document, ISBN 979-8-89176-251-0 Cited by: §1, §1, §2.2, §3.2, §4.2, §7. Wilcoxon (1945) F. Wilcoxon Individual comparisons by ranking methods. Biometrics Bulletin 1 (6), p. 80–83. External Links: ISSN 00994987, Link Cited by: §4.2. Wilson and Wharton (2006) D. Wilson and T. Wharton Relevance and prosody. Journal of Pragmatics 38 (10), p. 1559–1579. Note: Special Issue: Prosody and Pragmatics External Links: ISSN 0378-2166, Document, Link Cited by: §1. Wolf et al. (2023) L. Wolf, T. Pimentel, E. Fedorenko, R. Cotterell, A. Warstadt, E. G. Wilcox, and T. I. Regev Quantifying the redundancy between prosody and text. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, H. Bouamor, J. Pino, and K. Bali (Eds.), Singapore, p. 9765–9784. External Links: Link, Document Cited by: §1, §1, §2.2, §3.2, §4.2. Yadavalli et al. (2025) A. Yadavalli, T. Pimentel, T. I. Regev, E. Wilcox, and A. Warstadt What Do Prosody and Text Convey? Characterizing How Meaningful Information is Distributed Across Multiple Channels. External Links: 2512.16832, Link Cited by: §1, §1, §2.2, §3.2, §7. Zen et al. (2019) H. Zen, V. Dang, R. Clark, Y. Zhang, R. J. Weiss, Y. Jia, Z. Chen, and Y. Wu LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech. External Links: 1904.02882, Link Cited by: §5. Zhang et al. (2024a) D. Zhang, Y. Yu, J. Dong, C. Li, D. Su, C. Chu, and D. Yu M-LLMs: recent advances in MultiModal large language models. In Findings of the Association for Computational Linguistics: ACL 2024, L. Ku, A. Martins, and V. Srikumar (Eds.), Bangkok, Thailand, p. 12401–12430. External Links: Link, Document Cited by: Model limitations.. Zhang et al. (2024b) J. Zhang, J. Huang, S. Jin, and S. Lu Vision-language models for vision tasks: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 46 (8), p. 5625–5644. External Links: Document Cited by: Model limitations.. Zubizarreta (1998) M.L. Zubizarreta Prosody, focus, and word order. Linguistic Inquiry Monographs, MIT Press. External Links: ISBN 9780262740210, LCCN 98015459, Link Cited by: §1, §2.1, footnote 3. Uncontrolled Length-controlled (,)( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P_ pause) (,)( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P_ dur) (,)( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P_ pause) (,)( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P_ dur) Parse Type Subset I [rgb]0.5,0.5,0.5I U [rgb]0.5,0.5,0.5U I [rgb]0.5,0.5,0.5I U [rgb]0.5,0.5,0.5U I [rgb]0.5,0.5,0.5I U [rgb]0.5,0.5,0.5U I [rgb]0.5,0.5,0.5I U [rgb]0.5,0.5,0.5U Brackets Fluent 1.13 9.05% 1.36 10.9% 1.45 7.74% 1.68 8.96% Disfluent 1.45 7.45% 1.62 8.33% 1.40 7.49% 1.58 8.42% Full 1.20 8.64% 1.42 10.2% 1.43 7.62% 1.63 8.69% Full parse Fluent 1.90 5.70% 3.31 9.92% 2.21 4.61% 3.69 7.70% Disfluent 2.61 4.93% 3.91 7.39% 2.58 5.02% 3.88 7.55% Full 2.05 5.52% 3.45 9.27% 2.40 4.82% 3.79 7.63% Table 4: I [rgb]0.5,0.5,0.5I in nats U [rgb]0.5,0.5,0.5U across fluent, disfluent, and full datasets in uncontrolled (N=298kN=298k) and length-controlled (N=58kN=58k matched pairs, l¯=17.28 l=17.28 words) CANDOR subsets. In each subset, H [rgb]0.5,0.5,0.5H is computed separately, and the full “subset” rows are equivalent to those in Table 3. Appendix A Probability Leakage As discussed in Limitations, our model architecture Section 4.1 does not explicitly disallow illegal tokens while predicting syntactic parses. When non-zero probabilities are assigned to illegal tokens (e.g. tokens that are not a POS tag, a phrase tag, or a bracket), our H [rgb]0.5,0.5,0.5H estimates are inflated. In this section, we analyze such probability leakage in our trained models. For each trained model, we randomly select 1,000 sentences and measure at each prediction step the amount of probability assigned to illegal tokens. The aggregate probability is then divided by the total number of prediction steps to obtain the average probability mass assigned to illegal tokens at a given prediction step. We find that a significant probability mass is assigned to illegal tokens to full parse autoregressive models that estimate H() [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S): 15.7% in the planned speech (LibriTTS) model, and 11.7% in the spontaneous speech (CANDOR) model. Assuming a uniform distribution across 32k32k illegal tokens, the leakage contributes to mleak×lnmleak/N=1.92m_leak× m_leak/N=1.92 and 1.461.46 nats of inflated H [rgb]0.5,0.5,0.5H estimation, respectively, compared to an ideal model that assigns zero probability to illegal tokens. The remaining models’ leakage ranges from between 0.07% (from H(∣) [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]0,1,1 P_ pause) estimation on CANDOR), accounting for around 0.010.01 nats of inflation to 1.3% (from H(∣) [rgb]0.5,0.5,0.5H( [rgb]0.5,0.5,0 S [rgb]0,1,1 P_ dur) estimation on LibriTTS), accounting for around 0.180.18 nats of inflation. Models conditioned on brackets only leak no more than 0.001% probability mass, producing negligible inflation. Appendix B Modeling Hyperparameters We employ k-fold cross-validation where k=10k=10 for LibriTTS and k=5k=5 for CANDOR, resulting in 90:10 and 80:20 train-test splits for each fold, respectively. During training on each fold, we use the following hyperparameters: learning rate 3e-4 with a cosine scheduler, max length 256, batch size between 24 and 96, depending on GPU VRAM and modality, and early stopping patience 3. Each LibriTTS experiment requires around 4 GPU-days on a single Tesla L4 GPU, while each CANDOR experiment requires around 2 GPU-days on a single NVIDIA H100 GPU. Our models are trained to predit [rgb]0.5,0.5,0 S by teacher-forcing. Appendix C Details on Post-Hoc Analyses In Section 6, we perform analyses into how sentence features (sentence length, parse depth) and disfluency affects our findings. We provide additional detail on the two analyses in this section. C.1 Sentence Features and Sentence-level Syntactic Information Content in Planned Speech We find that l and 0ptare weakly inversely correlated with sequence-level syntactic information content in pause and duration in planned speech. The correlation is significant in all conditions when considering boundary information (p<0.001p<0.001), with r=−0.032r=-0.032 for l and pause, r=−0.029r=-0.029 for 0ptand pause; r=−0.040r=-0.040 for l and duration, r=−0.036r=-0.036 for 0ptand duration. The correlation is significant only in full parse information carried by duration with r=−0.036,p<0.001r=-0.036,p<0.001 for l and duration and r=−0.041,p<0.001r=-0.041,p<0.001 for 0ptand duration, while full parse information carried by pause is inversely, weakly, and insignificantly correlated: r=−0.003r=-0.003, p=0.49p=0.49 for l and pause, r=−0.009,p=0.054r=-0.009,p=0.054 for 0ptand pause. We plot sentence features and sentence-level syntactic information content that duration and pause carry in planned speech in Figure 4. Figure 4: LibriTTS sentence features (sentence length and maximum tree depth) versus pointwise syntactic information content. Each bin represents a single depth or length value. C.2 Effects of Disfluency on I(,) [rgb]0.5,0.5,0.5I( [rgb]0.5,0.5,0 S, [rgb]0,1,1 P) At first glance, the fluent and disfluent subsets vary significantly in I [rgb]0.5,0.5,0.5I and U [rgb]0.5,0.5,0.5U, as seen in Table 4. For example, in the disfluent subset, duration has 3.913.91 nats of I [rgb]0.5,0.5,0.5I with syntax, compared to 3.313.31 in the fluent subset. And pauses 2.612.61 nats of I [rgb]0.5,0.5,0.5I, vs. only 1.901.90 in the fluent subset. However, given the large difference in mean sentence length l~ l across the fluent (11.99 words) and disfluent (17.90 words) subsets, the length could be the primary driver in the cross-fluency variation. A length-controlled analysis, similar to our cross-genre analysis in Section 6, shows that l accounts for the majority of this difference, reducing the cross-subset gap in both I [rgb]0.5,0.5,0.5I and U [rgb]0.5,0.5,0.5U. After controlling for l, the difference in I [rgb]0.5,0.5,0.5I between the subsets and the full dataset drops from up to 40%40\% of the total value to within 7.5%7.5\% of the total value. To statistically confirm that l is in fact the main driver for the uncontrolled cross-subset difference, we perform a regression analysis by fitting ordinary least squares models. We compare a baseline model predicting Δι=ι(s)−ι(s,p) [rgb]0.5,0.5,0.5 = [rgb]0.5,0.5,0.5 ( [rgb]0.5,0.5,0s)- [rgb]0.5,0.5,0.5 ( [rgb]0.5,0.5,0s, [rgb]0,1,1p) using only l, against an augmented model containing both l and binary fluency information (fluent or disfluent). Across all conditions, adding fluency information to a regression model provides little added explanatory power, with ΔR2<0.01 R^2<0.01. That is, length is the main driver for the variance of sentence-level syntactic information content between fluent and disfluent subsets.