Paper deep dive
Token-Oriented Semantic Communication with Pretrained Vision Transformers
Jiwoong Im, Minwoo Kim, Jaeho Lee, Yo-Seb Jeon, Yongjune Kim
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 8/27/2026, 5:15:18 AM
Summary
The paper proposes a token-oriented semantic communication framework for client-server collaborative inference in resource-constrained edge systems. It addresses the high communication cost and interoperability issues of direct token embedding transmission by using a modular approach that coordinates a lightweight client-side Vision Transformer (ViT), a Learned Image Compression (LIC) model, and a large server-side ViT without end-to-end training. The framework leverages the spatial alignment between ViT patch tokens and LIC latent vectors to selectively transmit only task-relevant latents. Key innovations include layer-selective attention rollout for accurate token relevance estimation and surrogate token substitution to adapt the frozen server model for partial input reconstruction. Experiments on ImageNet demonstrate a superior rate-accuracy trade-off compared to existing semantic communication schemes and task-agnostic LIC models.
Entities (9)
Relation Signals (8)
Token-Oriented Semantic Communication Framework → evaluatedon → ImageNet
confidence 98% · Experiments on ImageNet show that the proposed framework achieves a more favorable rate–accuracy trade-off
Authors → affiliatedwith → POSTECH
confidence 95% · Jiwoong Im... are with the Department of Electrical Engineering, Pohang University of Science and Technology (POSTECH)
Token-Oriented Semantic Communication Framework → uses → Learned Image Compression (LIC)
confidence 95% · The framework is modular, coordinating... a learned image compression (LIC) model
Token-Oriented Semantic Communication Framework → uses → Vision Transformer (ViT)
confidence 95% · The framework coordinates... a lightweight client-side vision transformer (ViT)... and a large server-side ViT
Token-Oriented Semantic Communication Framework → employs → Layer-Selective Attention Rollout
confidence 94% · layer-selective attention rollout estimates token relevance from a selected range of attention layers
Token-Oriented Semantic Communication Framework → employs → Surrogate Token Substitution
confidence 94% · surrogate token substitution adapts the frozen server model by optimizing a single learnable token
Surrogate Token Substitution → enables → Parameter-Efficient Adaptation
confidence 93% · providing parameter-efficient input-level adaptation that improves inference with partial visual information
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Token communications realize the semantic communication principle at the granularity of transformer tokens, providing a promising direction for client--server collaborative inference in resource-constrained edge systems. However, directly transmitting token embeddings presents two practical challenges: substantial communication cost and limited interoperability across model-specific token embedding spaces. To address these challenges, we propose a \emph{token-oriented} semantic communication framework. In this framework, token-level task relevance determines which compressed image latents are transmitted, enabling token-granular transmission without directly transmitting token embeddings. The framework is modular, coordinating three pretrained components---a lightweight client-side vision transformer (ViT), a learned image compression (LIC) model, and a large server-side ViT---without end-to-end training. The key enabler is the one-to-one spatial alignment between ViT patch tokens and the LIC latent vectors, which allows token-level task relevance to directly determine which latent vectors are transmitted. Building on this alignment, token-aligned LIC selectively transmits task-relevant latents, layer-selective attention rollout estimates token relevance from a selected range of attention layers in a single forward pass, and surrogate token substitution adapts the frozen server model by optimizing a single learnable token. Experiments on ImageNet show that the proposed framework achieves a more favorable rate--accuracy trade-off than recent semantic communication schemes, hand-crafted codecs, and task-agnostic LIC models.
Tags
Links
- Source: https://arxiv.org/abs/2608.25410v1
- Canonical: https://arxiv.org/abs/2608.25410v1
Trouble viewing inline? Open PDF directly →
Full Text
72,748 characters extracted from source content.
Expand or collapse full text
Token-Oriented Semantic Communication with Pretrained Vision Transformers Thanks: Jiwoong Im, Minwoo Kim, Jaeho Lee, Yo-Seb Jeon, and Yongjune Kim are with the Department of Electrical Engineering, Pohang University of Science and Technology (POSTECH), Pohang 37673, South Korea (e-mail: jw3562, minwoo.kim, jaeho.lee, yoseb.jeon, yongjune@postech.ac.kr). Jiwoong Im and Minwoo Kim contributed equally to this work. Jiwoong Im, Minwoo Kim, Jaeho Lee, , Yo-Seb Jeon, , and Yongjune Kim Affiliation: Abstract Token communications realize the semantic communication principle at the granularity of transformer tokens, providing a promising direction for client–server collaborative inference in resource-constrained edge systems. However, directly transmitting token embeddings presents two practical challenges: substantial communication cost and limited interoperability across model-specific token embedding spaces. To address these challenges, we propose a token-oriented semantic communication framework. In this framework, token-level task relevance determines which compressed image latents are transmitted, enabling token-granular transmission without directly transmitting token embeddings. The framework is modular, coordinating three pretrained components—a lightweight client-side vision transformer (ViT), a learned image compression (LIC) model, and a large server-side ViT—without end-to-end training. The key enabler is the one-to-one spatial alignment between ViT patch tokens and the LIC latent vectors, which allows token-level task relevance to directly determine which latent vectors are transmitted. Building on this alignment, token-aligned LIC selectively transmits task-relevant latents, layer-selective attention rollout estimates token relevance from a selected range of attention layers in a single forward pass, and surrogate token substitution adapts the frozen server model by optimizing a single learnable token. Experiments on ImageNet show that the proposed framework achieves a more favorable rate–accuracy trade-off than recent semantic communication schemes, hand-crafted codecs, and task-agnostic LIC models. Index Terms: Semantic communications, token communications, edge computing, learned image compression, collaborative inference, vision transformer. I Introduction Recent advances in transformer architectures have fundamentally reshaped the internal representation and processing of modern artificial intelligence (AI) models [1]. Transformers process inputs as a sequence of tokens, each of which serves as a unified computational unit across multiple layers. This tokenized formulation has proven effective across a wide range of modalities, including language, images, audio, and video [1, 2, 3, 4, 5, 6]. In particular, token representations offer a structured, modular interface that is naturally compatible with global context modeling via attention mechanism [1]. As a result, tokens are now widely regarded not only as an internal feature format for transformer models, but also as a candidate unit for computation and communication. Meanwhile, with the emergence of AI-native wireless networks, communication paradigms are progressively shifting beyond conventional bit-level communication toward semantic communications [7, 8, 9, 10, 11, 12, 13]. Rather than optimizing human-perceived reconstruction fidelity quantified in terms of bit-level distortion, semantic communications attempt to enhance communication efficiency by selectively transmitting information relevant to downstream task objectives. Tokens provide a natural unit for this shift, and recent studies have accordingly investigated token communications, which realize the semantic communication principle at the granularity of transformer tokens [14, 15, 16, 17, 18, 19]. Operating at the token level reveals which tokens are relevant to the downstream task, so that the communication payload can be concentrated on inference-relevant tokens rather than the entire source, as in conventional bit-level communication. Realizing this advantage by transmitting token representations themselves, however, entails two practical considerations. The first concern is communication cost: each token consists of a high-dimensional embedding vector whose dimensionality increases with model capacity, so transmitting even a reduced number of tokens may incur a substantial payload. The second concern is the limited interoperability of token embedding spaces across independently trained models, as these embedding spaces can differ substantially because of differences in architectures, pretraining procedures, and downstream objectives, even at an identical model dimensionality [20]. To address both considerations, several studies jointly train the transmitter and receiver in an end-to-end manner, typically with a shared codebook that converts token embeddings into codeword indices [21, 14, 22, 23, 24]. Such fully integrated designs are effective within a given deployment, yet they reduce modularity, incur retraining costs, and couple the communication strategy to a specific downstream task. To retain the benefits of token-level processing while addressing these considerations, we propose a token-oriented semantic communication framework, considering image classification as the downstream task. Rather than compressing or exchanging token embeddings, the proposed framework compresses the input image with learned image compression (LIC), which exploits the statistics of natural images to attain a content-adaptive rate–distortion performance competitive with conventional image compression [25, 26, 27, 28]. The proposed framework integrates token-level task relevance into this compression. The key enabler is the spatial alignment between visual tokens and LIC latent vectors: in the widely used architectures, the vision transformer (ViT) patch size equals the LIC encoder’s spatial downsampling factor, so each visual token corresponds to exactly one latent vector [2, 29, 30, 26, 27, 28]. Task relevance is thereby incorporated into compression without modifying the LIC architecture or introducing additional training, and the communication payload is reduced by compressing in the image domain rather than in the model-specific embedding space. We use the term token-oriented to emphasize that token-level task relevance determines which token-aligned LIC latent vectors are transmitted, enabling transmission at token granularity without directly transmitting token embeddings. The resulting framework is fully modular, coordinating three separately pretrained models for client–server collaborative inference: a lightweight client-side ViT, an LIC-based neural compressor, and a large server-side ViT. Specifically, the client-side ViT estimates the task relevance of visual tokens, and this token-level relevance is used to guide the selection of latent representations within the compressor. The neural compressor then reduces communication payload by transmitting only the selected latent representations, while the server-side ViT performs downstream inference from the reconstructed image. Since no token embedding is exchanged across models, their embedding spaces need not be compatible, and each component can be replaced without retraining the others from scratch. Prior semantic communication frameworks that use token-level relevance to guide transmission without directly transmitting token embeddings can likewise be regarded as token-oriented [31, 32, 33, 34]. However, they differ in the transmitted representation, sending image patches in the pixel domain, either raw or scalar-quantized. Operating in the LIC latent domain thus distinguishes the proposed framework within this class and enhances the rate–accuracy trade-off. Upon this framework, we introduce three complementary components, each acting on one of the pretrained models, to further optimize the rate–accuracy trade-off: • Token-aligned LIC: Token-level task relevance is integrated into an LIC model to reduce the transmission rate while preserving task-relevant visual information. Exploiting the structural alignment between the visual tokens of ViT and the latent representations of LIC on a shared spatial grid, the token-level relevance map directly indexes the latents to be transmitted. Our approach selectively transmits only the latents at task-relevant positions, reducing the transmission rate while retaining compatibility with the underlying LIC architecture. • Layer-selective attention rollout: The client-side ViT estimates token-level task relevance precisely by aggregating attention weights over a selected range of transformer layers. This selective aggregation provides more accurate relevance estimates than the last-layer attention commonly adopted in prior semantic communication frameworks [31, 32, 33, 34], the attention rollout accumulated over all layers [35], and even gradient-weighted attention [36]. Moreover, unlike gradient-weighted attention [36], it requires only a single forward pass, avoiding the memory and latency overhead of gradient computation that is demanding for resource-constrained clients. • Learnable surrogate token substitution: Server-side inference with partial visual information is improved by substituting the token positions of the unselected regions with a single learnable surrogate token. Inference without any adaptation, whether using the directly reconstructed image or only the selected patches, degrades classification accuracy on the server, whereas retraining the large server backbone is costly and alters its behavior on complete inputs. Instead, we optimize only the single surrogate token with the backbone kept frozen, providing parameter-efficient input-level adaptation that improves inference with partial visual information while preserving the original complete-input behavior. It further improves the robustness of server-side inference over unreliable channels, where the tokens affected by channel impairments are likewise replaced by the surrogate token. Built from three pretrained models without end-to-end training, the proposed framework enables modular and communication-efficient collaborative inference. Across the evaluated rate regime, it attains a more favorable rate–accuracy trade-off than both recent semantic communication schemes [33, 34], which exploit task relevance without efficient compression, and task-agnostic LIC models [26, 27, 28], which optimize reconstruction without task relevance. The remainder of this paper is organized as follows. Section I reviews related work on token-level semantic communications, learned image compression, attention-guided importance estimations, and learnable input tokens. Section I presents an overview of our proposed token-oriented semantic communication framework. Section IV introduces the main technical contributions of this work, optimizing the rate–accuracy trade-off of the proposed framework: token-aligned LIC, layer-selective attention rollout, and surrogate token substitution. Section V reports the experimental results, and Section VI concludes the paper. I Related Work I-A Token-Level Semantic Communications TABLE I: Comparison of Token-Level Semantic Communication Frameworks for Collaborative Inference Framework Transmitted Token Importance Server-Side Modularity Representation Estimation Adaptation VQ-Based Token Comm. [14, 22, 23, 24] VQ Codebook Indices Auxiliary Network or None Joint Training × Prior Token-Oriented Comm. [31, 32, 33, 34] Raw or Quantized Pixels Last-Layer Attention None ✓ Proposed Token-Aligned Layer-Selective Surrogate Token ✓ LIC Latents Attention Rollout (Frozen Backbone) Semantic communication frameworks operating at token granularity differ in what they transmit, how they identify task-relevant content, and how the receiver is adapted for inference. As summarized in Table I, VQ-enabled token communications transmit the token representations themselves as vector-quantized codebook indices, which requires end-to-end joint training to establish a codebook shared between the two ends [14, 22, 23, 24]. Token-oriented communications instead guide transmission by token-level relevance while operating on pretrained models, yet prior frameworks in this class transmit image patches in the pixel domain [31, 32, 33, 34]. The proposed framework is likewise token-oriented but transmits LIC latents, thereby improving rate efficiency while retaining modularity, i.e., operation with pretrained models without end-to-end joint training. I-B Learned Image Compression With the advancement of machine learning, the learned image compression (LIC) paradigm [25, 26, 27, 28] has achieved rate–distortion performance competitive with conventional hand-crafted image compression. Among the various LIC architectures, we build upon the nonlinear transform coding framework [25] and its extension to a hyperprior network [26], which underlies many widely adopted LIC models. The end-to-end compression pipeline maps an input image X to a reconstruction ^∈ℝH×W×C X ^H× W× C through the following sequential pipeline: →gac→^c→gs^.X g_aX^c Q X^c g_s X. (1) Here, an encoder gag_a extracts the latent representation c∈ℝN×DcX^c ^N× D_c, where DcD_c denotes the model dimension of the neural compressor. Q introduces discretization to obtain the quantized latent ^c X^c, and a decoder gsg_s reconstructs the image. To effectively remove the remaining spatial redundancy of latents cX^c, [26] introduced a hyperprior network that transmits side information. Specifically, a hyper encoder hah_a extracts a hyper latent =ha(c)Z=h_a(X^c), which is quantized to Z. A hyper decoder hsh_s then decodes Z to estimate the conditional distribution of ^c X^c, modeled as a normal distribution with zero mean. Subsequently, [27] extended this architecture to jointly predict both the mean μ and variance σ of the elements of the latents. These predicted parameters define the precise probability mass function required by the arithmetic coder, allowing the entropy coding process to closely approach the theoretical Shannon entropy limit. I-C Attention-Guided Importance Estimation Vision transformer (ViT) is a transformer-based architecture for computer vision tasks [2]. To perform inference, an input image ∈ℝH×W×CX ^H× W× C is reshaped into a sequence of flattened 2D patches p∈ℝN×(P2⋅C)X^p ^N×(P^2· C), where (H,W)(H,W), C, and (P,P)(P,P) denote the resolution of the original image, the number of channels, and the resolution of each image patch, respectively. Here, N=HWP2N= HWP^2 denotes the resulting number of image patches. Each patch ipx_i^p is then tokenized into a visual token it∈ℝ1×Dx_i^t ^1× D by linear projection, forming the visual token matrix t∈ℝN×DX^t ^N× D whose iith row is itx_i^t. Then, the class token clsx_cls is prepended to the sequence of visual tokens. The input embedding of the ViT encoder ~t∈ℝ(N+1)×D X^t ^(N+1)× D is given by ~t=[cls;t+], X^t= [x_cls;X^t+E ], (2) where ∈ℝN×DE ^N× D denotes the standard learnable position embedding. Since token interactions in ViT are primarily mediated through multi-head self-attention blocks, the attention weights can be interpreted as modeling global dependencies among tokens that contribute to the final inference. Accordingly, many recent studies have estimated token relevance ∈ℝ(N+1)×(N+1)R ^(N+1)×(N+1) and token-level task relevance ¯∈ℝN r ^N from the attention weights computed during downstream inference. Below, we briefly review representative attention-guided token relevance methods. For brevity, we denote the attention weights at layer l and head h by l,h∈ℝ(N+1)×(N+1)A_l,h ^(N+1)×(N+1). • Last-layer attention: The most direct approach estimates token relevance using the attention weights from the final transformer layer: =h[L,h],R=E_h [A_L,h ], (3) where L denotes the number of transformer layers and h[⋅]E_h[·] represents averaging over attention heads. It is widely used in prior token-oriented communication studies owing to its simplicity [33, 34, 31, 32]. • Attention rollout: Attention rollout aggregates attention weights across all transformer layers to approximate the overall token dependencies captured by the model [35]: ~l=h[l,h]+N+1,=~L~L−1⋯~1, A_l=E_h [A_l,h ]+I_N+1, = A_L A_L-1·s A_1, (4) where N+1∈ℝ(N+1)×(N+1)I_N+1 ^(N+1)×(N+1) denotes the identity matrix that accounts for the residual connection in ViT. • Gradient-weighted attention: Gradient-weighted attention additionally reweights the attention rollout based on the gradient of the top-1 predicted logit with respect to the attention weights [36]. Although it provides precise task-relevance estimation, its reliance on gradient computation constitutes a practical limitation, which precludes its deployment on resource-constrained clients. The above methods share a common final step: the token-level task relevance ¯ r is extracted from the first row of R, which quantifies the relevance of the class token, the input to the classification head, to the visual tokens. Excluding the class-token self-relevance R1,1R_1,1, we normalize the N entries corresponding to the visual tokens as r¯i=R1,i+1∑j=1NR1,j+1, r_i= R_1,i+1 _j=1^NR_1,j+1, (5) so that ¯ r forms a distribution over the visual tokens. I-D Learnable Input Tokens Learnable input tokens, i.e., trainable vectors inserted into the input sequence, have been used to adapt or train vision transformers. Visual prompt tuning prepends a few learnable prompt tokens to the input and optimizes only them while freezing the backbone, enabling parameter-efficient task adaptation [37]. On the other hand, masked image modeling inserts a single shared mask token at masked positions, jointly optimized with the backbone for self-supervised reconstruction during pretraining [38, 39]. The proposed surrogate token substitution combines both aspects: a single learnable token substitutes missing positions and is optimized with the backbone frozen. It differs in purpose, being trained for the downstream classification objective to compensate for missing visual content at inference. I Overview of the Proposed Framework Fig. 1: Overview of the proposed token-oriented semantic communication framework based on pretrained ViT models. The circled numbers in the figure follow the same order as the line numbers in Algorithm 1. This section presents the client–server collaborative inference procedure of the proposed token-oriented semantic communication framework, deferring its technical components to Section IV. We consider a setting in which a resource-constrained client cooperates with a computationally capable server. The client employs a lightweight ViT-based classifier (e.g., DeiT-Tiny [29]) together with a neural compressor, whereas the server employs a large-scale ViT classifier (e.g., DeiT-I-Large [30]) and the corresponding neural decompressor. Our overall framework is illustrated in Fig. 1, where the circled numbers follow the same order as the line numbers in Algorithm 1. First, the client extracts visual tokens tX^t from the input image X and computes the token-level task relevance ¯ r using the lightweight ViT. Based on these scores, the client selects the task-relevant visual tokens and obtains a token-selection map ∈0,1Ns∈\0,1\^N, where N denotes the number of visual tokens. The client then generates the compressed bitstream b, which contains the visual latent representations corresponding to the positions of the selected tokens and the hyperpriors, and transmits (,)(b,s) to the server. After transmission, the server receives (^,^)( b, s) and reconstructs the image X, which is then converted into server-side visual tokens ^t X^t. From ^t X^t, the input sequence ~t X^t of the server-side ViT is constructed via surrogate token substitution. Finally, the server-side predictor produces the classification result y and returns it to the client. The overall inference procedure is described in Algorithm 1. Algorithm 1 Inference Procedure of the Proposed Framework Input: Input image X, token-selection threshold δ, client tokenizer fα:→tf_α:X ^t, server tokenizer fβ:^→^tf_β: X→ X^t, client classifier fθ:t→¯f_θ:X^t→ r, server classifier fϕ:~t→yf_φ: X^t→ y, neural compressor fψ:(,)→f_ψ:(X,s) , and neural decompressor fξ:(^,^)→^f_ξ:( b, s)→ X. Output: Classification result y. 1: t←fα()X^t← f_α(X) ⊳ Client-side visual token extraction 2: ¯←fθ(t) r← f_θ(X^t) ⊳ Token-level task relevance 3: Obtain token-selection map ∈0,1Ns∈\0,1\^N from ¯ r using threshold δ 4: ←fψ(,)b← f_ψ(X,s) ⊳ Compressed bitstream 5: Transmit (,)(b,s) over the communication channel 6: Server receives (^,^)( b, s) 7: ^←fξ(^,^) X← f_ξ( b, s) ⊳ Reconstructed image 8: ^t←fβ(^) X^t← f_β( X) ⊳ Server-side visual token extraction 9: Construct ~t X^t from ^t X^t via surrogate token substitution 10: y←fϕ(~t)y← f_φ( X^t) ⊳ Final inference result 11: Return y to the client We integrate token selection into LIC by retaining only the visual latent representations corresponding to the task-relevant token positions. Consequently, the latent payload is reduced according to the number of selected tokens. The transmitted bitstream also includes the hyperpriors and the selection map s, but the bits allocated to them are sufficiently small compared to those used for the visual latent representations. As a result, the bitstream size decreases approximately linearly with the number of selected tokens, as analyzed in Section IV-A. IV Main Technical Components In this section, we present three technical components of the proposed framework: (1) token-aligned learned image compression, (2) layer-selective attention rollout, and (3) learnable surrogate token substitution. The client first estimates the token-level task relevance of visual tokens via layer-selective attention rollout on a lightweight ViT model. It then performs token-wise on/off selection based on the estimated relevance, and the LIC model generates a bitstream containing the latent representations at the selected token positions and the hyperpriors. At the server, the reconstructed patches at unselected positions are unreliable because the corresponding latents are not transmitted. To prevent these unreliable patches from degrading downstream inference, the server replaces the tokens at those positions with a learnable surrogate token. IV-A Token-Aligned Learned Image Compression Given the token-level task relevance ¯ r estimated from the attention weights of the client-side ViT (Section IV-B), the client selects task-relevant tokens to reduce communication overhead. A straightforward approach is to transmit the selected image patches directly in the pixel domain [33] or after importance-aware scalar quantization [34]. However, both approaches encode each patch independently and thus cannot exploit the statistical redundancy that learned image compression captures, remaining far less rate-efficient than LIC models, as demonstrated in Section V. This motivates integrating token selection into the compression process itself. We propose token-aligned LIC, which integrates attention-aware token selection with an LIC model by operating in the latent domain, as shown in Fig. 2. The key enabler is a structural alignment between the visual tokens of ViT and the latent representations of LIC: the encoder maps an input image X to a latent representation cX^c, which is quantized to ^c∈ℝN×Dc X^c ^N× D_c with spatial resolution (HP,WP) ( HP, WP ), yielding N=HWP2N= HWP^2 spatial positions identical to the number of visual tokens. Note that both architectures conventionally reduce the input resolution by the same factor P=16P=16: ViT through a patch size of (16,16)(16,16) [2, 29, 30], and LIC through four stride-2 convolutional stages [26, 27, 28]. This one-to-one correspondence between the iith visual token itx_i^t and the iith latent vector ^ic x_i^c, denoting the iith rows of tX^t and ^c X^c, respectively, allows the token-selection map ∈0,1Ns∈\0,1\^N to directly index the latent representations, controlling which latent vectors are included in the transmitted bitstream without modifying the LIC architecture. Fig. 2: Proposed token-aligned learned image compression framework. Q, AE, and AD refer to the quantizer, arithmetic encoder, and arithmetic decoder, respectively. After latent encoding, only the latent representations at task-relevant token positions are retained and compressed into a bitstream. At the server, missing latent positions are imputed with hyperprior-predicted means before image reconstruction by the neural decoder. Given the task relevance ¯∈ℝN r ^N, we construct the selection map s using the attention-sum threshold selection of [33], where si=1s_i=1 indicates that the iith token is selected. Tokens are selected in descending order of relevance until the cumulative sum of relevance first exceeds a preset threshold δ, so that the number of selected tokens varies across images according to semantic content, such as object size and inference difficulty. We denote the resulting sets of selected and unselected spatial indices as =i∣si=1S=\i s_i=1\ and =i∣si=0U=\i s_i=0\, respectively. IV-A1 Selective Latent Transmission To selectively include only task-relevant latent representations in the transmitted bitstream, we exploit the structure of the hyperprior-based entropy model [26]. Specifically, the distribution of the ddth element of the iith latent x^i,dc x^c_i,d is fully parameterized by the predicted mean μi,d _i,d and scale σi,d _i,d, obtained from the hyper decoder hsh_s using the quantized hyperprior Z [27]. Since the latent elements are modeled as conditionally independent given Z, each element can be entropy-coded and decoded using only its own predicted parameters (μi,d,σi,d)( _i,d, _i,d), without access to any other latent element. Consequently, the client guides the arithmetic encoder to encode only the latent vectors at positions i∈i , bypassing entropy coding for positions i∈i entirely. The transmitted bitstream comprises the tuple (^c,^,) ( X^c_S,\, Z,\,s ), where ^c∈ℝ||×Dc X^c_S ^|S|× D_c stacks the retained latent vectors ^ic x^c_i, i∈i , in ascending order of spatial index. This selective encoding reduces the communication cost; the resulting rate reduction is quantified in the theoretical analysis below. IV-A2 Theoretical Analysis of Communication Efficiency To analyze the communication efficiency, we establish a theoretical bound demonstrating that an approximately linear reduction in bit rate is achievable. The analysis assumes an ideal entropy coder in which the code length of each symbol approaches its self-information. The number of bits required to encode the retained latents is given by B^c=⌈−∑i∈∑dlog2p(x^i,dc∣^)⌉.B_ X^c_S= - _i _d _2p\! ( x^c_i,d Z ) . (6) Analogously, the number of bits required to encode the complete latent matrix ^c X^c, i.e., over both selected and unselected positions, is B^c=⌈−∑i=1N∑dlog2p(x^i,dc∣^)⌉.B_ X^c= - _i=1^N _d _2p\! ( x^c_i,d Z ) . (7) Transmitting the complete latent representation together with the side information would then require Btot=B^c+B^,B_tot=B_ X^c+B_ Z, (8) where B^B_ Z denotes the number of bits allocated to the quantized hyperprior. As the proposed framework transmits only the retained latents, however, the actual number of transmitted bits is B=B^c+B^+B,B=B_ X^c_S+B_ Z+B_s, (9) where B_s denotes the number of bits allocated to the binary selection map. Consequently, the ratio of the actual to the total bits admits the upper bound BBtot≤η+ρ+(1−ρ)k,where k=H¯H¯, BB_tot≤η+ ρ+(1-ρ)k, k= H_U H_S, (10) where ρ=||||+||ρ= |S||S|+|U| denotes the token selection ratio, η=B^+BBtotη= B_ Z+B_sB_tot denotes the side-information overhead ratio including the hyperprior and the selection map, and H¯ H_S and H¯ H_U denote the average self-information per latent over the selected and unselected positions, respectively. In the regime k≈1k≈ 1, where the selected and unselected regions exhibit comparable information density, the bound simplifies to BBtot≲η+ρ, BB_tot η+ρ, (11) showing that the proposed selective transmission yields a rate reduction approximately linear in the selection ratio ρ, offset only by the side-information overhead η. The bound further characterizes the nonlinear regimes: the reduction is sublinear when the unselected region is information-sparse (k<1k<1), and superlinear when it is information-dense relative to the selected region (k>1k>1). IV-A3 Predicted-Mean Imputation Before latent decoding, the decoder imputes the unselected latent positions with their predicted means, derived from the received hyperprior Z following the mean-scale hyperprior architecture [27]. From a statistical perspective, the predicted mean is the conditional expectation of the local latent feature given the context captured by the hyperprior, and is thus a principled substitute for the missing latent. This choice matters because the LIC decoder uses convolutional layers with overlapping receptive fields: the reconstruction of each selected patch is influenced by the latents at neighboring positions, so the imputed values directly affect the quality of the neighboring selected patches. Substituting the unselected latents with their conditional expectations preserves the local statistical consistency of the latent map, improving downstream classification accuracy over zero padding, as shown in Section V-E. Furthermore, since ^∈ℝN×Dc μ ^N× D_c is already produced by the hyper decoder for entropy decoding, the imputation introduces negligible additional computational overhead. The iith row of the imputed latent representation ~c X^c at the decoder is ~ic=si^ic+(1−si)^i,i=1,…,N, x^c_i=s_i\, x^c_i+(1-s_i)\, μ_i, i=1,…,N, (12) where ^i μ_i denotes the iith row of μ. Note that predicted-mean imputation is introduced solely to improve the reconstruction of the selected image patches: the reconstructed patches at unselected positions are not forwarded to the server-side predictor, as described in Section IV-C. IV-B Layer-Selective Attention Rollout In transformer architectures, token interactions are primarily mediated by multi-head self-attention, whose attention weights capture global dependencies among tokens. For image classification with ViTs, the final prediction is produced from the class token, so the attention from the class token to the visual tokens naturally indicates how much each visual token contributes to the classification task. Fig. 3: Visualization of token-level task relevance from DeiT-Tiny [29]. The first column shows example ImageNet images labeled as ‘great gray owl,’ ‘Pembroke,’ and ‘polecat,’ respectively. The remaining columns, from left to right, show task-relevance maps obtained using last-layer attention, attention rollout, and layer-selective attention rollout from the 7th to the 12th layers, respectively. Along this line, last-layer class-token attention has been widely used for task relevance estimation owing to its simplicity [31, 32, 33, 34]. However, as shown in Fig. 3, last-layer attention often assigns high relevance to non-object regions, overestimating the task relevance of background patches. This limitation may arise because visual tokens are progressively contextualized across transformer layers [35, 40, 36]: by the final layer, each token aggregates information from other tokens and no longer represents only its own patch, so the class-token attention at a single layer does not faithfully reflect which patches originally contributed to the prediction. Attention rollout was originally proposed to trace attention propagation among language tokens in natural language processing [35]. To account for the progressive contextualization described above, it multiplies the attention matrices across all layers, thereby explicitly tracking the information flow from the input to the final representation. As shown in Fig. 3, the resulting map is largely concentrated on the main object. However, when the selected patches are transmitted for server-side inference, attention rollout yields lower classification accuracy than last-layer attention unless only a small fraction of patches is retained [33], indicating that accurate object localization does not necessarily translate into accurate task-relevance estimation. Fig. 4: Visualization of class-token attention weights across all layers of DeiT-Tiny. The original image is ‘polecat,’ same as Fig. 3. Note that the visualized attentions are averaged along the head dimension. To understand why both methods fall short, we visualize the class-token attention weights across all layers of DeiT-Tiny [29] in Fig. 4. The attention patterns are noisy and weakly interpretable in the early layers, concentrate on the main object in the middle layers, and drift toward subsets of the background in the final layers. This layer-dependent behavior explains the preceding observations: last-layer attention inherits the background drift of the final layers, whereas attention rollout accumulates the noise of the early layers. These observations motivate aggregating attention over a selected range of layers that excludes the noisy early layers, rather than relying on a single layer or on all layers. Therefore, we propose layer-selective attention rollout, which aggregates attention weights over a selected range of ViT layers. The layer-selective attention rollout from the LsL_sth layer to the LeL_eth layer, where Ls≤LeL_s≤ L_e, is computed as ~l=h[l,h]+N+1,=~Le~Le−1⋯~Ls, A_l=E_h [A_l,h ]+I_N+1, = A_L_e A_L_e-1·s A_L_s, (13) and the token-level relevance ¯ r is then obtained via (5). For DeiT-Tiny with L=12L=12 layers, we set (Ls,Le)=(7,12)(L_s,L_e)=(7,12), determined by the layer-range study in Section V-F. The last layer is included because it directly shapes the class token consumed by the classification head [2], while the noisy early layers are excluded. Examples of the resulting relevance maps are shown in Fig. 3, assigning relevance to the main object together with parts of its surrounding regions. As shown in Section V-F, layer-selective attention rollout estimates task relevance more accurately than last-layer attention and attention rollout, and matches or exceeds gradient-weighted attention [36] across our main operating region. Achieving this accuracy without gradients is what matters in practice: gradient-weighted attention requires a backward pass through the client-side ViT at every inference, and the additional computation and peak-memory footprint for storing intermediate activations are costly on resource-constrained clients. Layer-selective attention rollout attains this accuracy, exceeding gradient-weighted attention on our main client configuration, from a single forward pass, making precise task-relevance estimation practical on such clients. IV-C Surrogate Token Substitution Through token-aligned LIC, the server obtains a reconstructed image ^∈ℝH×W×C X ^H× W× C that retains the original spatial resolution, in which the patches at unselected positions are synthesized from the hyperprior-predicted means and thus reflect only a coarse statistical estimate of the original content at those positions. To prevent these patches from serving as misleading visual evidence for the server-side predictor, we introduce a single learnable token, termed the surrogate token, that replaces the tokens at unselected positions and is optimized while the pretrained backbone is kept frozen. The modified input token sequence ~t∈ℝ(N+1)×D X^t ^(N+1)× D is constructed by prepending the class token clsx_cls and assigning the surrogate token sur∈ℝ1×Dx_sur ^1× D to the unselected positions: ~1t x^t_1 =cls, =x_cls, (14) ~i+1t x^t_i+1 =si^it+(1−si)sur+i,i=1,…,N, =s_i\, x^t_i+(1-s_i)\,x_sur+e_i, i=1,…,N, (15) where ~it x^t_i and ie_i denote the iith row of the resulting token sequence ~t X^t and the position embedding E in (2), respectively. We optimize surx_sur for the downstream classification objective with the backbone frozen, so that the surrogate token learns to indicate the absence of information rather than supply misleading visual evidence. Without any adaptation, the server can perform inference directly on the reconstructed image by replacing tX^t in (2) with ^t X^t (direct-reconstruction inference), or exploit the variable-length input property of ViTs to use only the reconstructed patches at the selected positions (selected-patch inference). While some accuracy reduction is unavoidable when inferring from partial visual information, both adaptation-free baselines leave the frozen predictor unadapted to the incomplete input and thus miss the accuracy that adaptation could recover. At the opposite extreme, retraining the server-side ViT on incomplete images recovers this margin but is computationally expensive [41, 37, 42], and updating the backbone weights can degrade performance on complete or near-complete inputs, where the pretrained DeiT-I-Large [30] already attains its peak accuracy of 86.81 %86.81\,\% on ImageNet. Surrogate token substitution occupies the middle ground between these extremes: the cost of adaptation is a single learnable vector of D parameters (D=1024D=1024 for DeiT-I-Large), and since the backbone remains intact, the predictor’s behavior on complete inputs is identical to that of the original pretrained model. Section V-G demonstrates that it maintains downstream accuracy more effectively than both adaptation-free baselines. Surrogate token substitution further improves the robustness of server-side inference over unreliable channels. When the received bitstream differs from the transmitted one because of uncorrected channel impairments, a subset of the transmitted LIC latents may be erased. In this case, the server can replace the tokens at the affected positions with the surrogate token, reducing the impact of erasures on classification accuracy; this robustness gain is evaluated over a packet-erasure channel in Section V-C. V Experimental Results We evaluate the proposed token communication framework on ImageNet classification. After describing the experimental settings (Section V-A), we first present the main results: the rate–accuracy trade-off against prior token-oriented semantic communication baselines, hand-crafted codecs, and LIC frameworks (Section V-B), the robustness of surrogate token substitution under a packet-erasure channel (Section V-C), and the additional gain of entropy-aware image transmission (Section V-D). We then conduct ablation studies that isolate the contribution of each technical component: token-aligned LIC (Section V-E), layer-selective attention rollout (Section V-F), and surrogate token substitution (Section V-G). V-A Experimental Settings TABLE I: Model Complexity and Classification Accuracy on ImageNet-1k [43, 26, 29, 30] Model Parameters Memory MACs Classification (million) (MB) (G) Accuracy (%) DeiT-Tiny 5.7 22.9 1.26 72.2 DeiT-Small 22.1 88.2 4.61 79.8 DeiT-Base 86.6 346.3 17.58 81.8 DeiT-I-Large 304.4 1217.5 61.6 86.8 Neural Comp. 5.5 22.1 3.21 - Neural Decomp. 4.5 18.0 2.76 - In the proposed system, we assume that the client deploys DeiT-Tiny [29], while the server employs DeiT-I-Large [30], reflecting the disparity in memory and computational capacity between the client and the server. As shown in Table I, the accuracy gap between the two models is substantial: DeiT-I-Large attains 86.8 %86.8\,\%, whereas DeiT-Tiny reaches only 72.2 %72.2\,\%. However, this accuracy improvement comes with substantially higher resource requirements, including approximately 50×50× larger memory consumption and computational complexity than those of DeiT-Tiny. In contrast, the hyperprior-based neural compressor [26, 27] deployed on the client side requires only 22.1 MB22.1\,MB of memory, making it suitable for resource-constrained clients. Overall, the client-side models, consisting of DeiT-Tiny and the neural compressor, require only 45 MB45\,MB of memory and 4.47 GMACs4.47\,GMACs, whereas the server-side models, consisting of DeiT-I-Large and the neural decompressor, require more than 1.2 GB1.2\,GB of memory and approximately 64.36 GMACs64.36\,GMACs. We evaluate the framework on ImageNet [43] for image classification. Each image is center-cropped and resized to a resolution of (224,224)(224,224) pixels. We adopt publicly available pretrained ViT parameters, where DeiT-I-Large is additionally pretrained on ImageNet-21k [30]. The LIC model and the surrogate token are trained separately, with the settings described below. • LIC training: We train a hyperprior model [26] with a mean hyperprior [27] for image reconstruction on ImageNet, setting the channel dimensions of the latent representation and the hyperprior to 192192 and 128128, respectively. A separate LIC model is trained for each considered value of λ, while the remaining optimization settings follow [26, 27]. • Surrogate token training: Following Section IV-C, we optimize a single surrogate token sur∈ℝ1×Dx_sur ^1× D for the classification objective while keeping the pretrained backbones frozen. Surrogate token training emulates the proposed pipeline: the client selects task-relevant tokens via layer-selective attention rollout under threshold δsur _sur and transmits a bitstream compressed by an LIC model pretrained with λsur _sur, after which the server replaces the unselected positions with the surrogate token before inference. The token is trained for 2020 epochs, with the remaining settings following [39]. The selection map ∈0,1Ns∈\0,1\^N accompanies the encoded image data (^c,^)( X^c_S, Z) so that the server can both reconstruct the image and identify the positions to be replaced by the surrogate token. At our resolution, the selection map occupies only N=196N=196 bits per image, corresponding to less than 0.0040.004 bits per pixel ( bpp\,bpp), where the bit rate is computed as the average of sent bitsnumber of pixels sent bitsnumber of pixels over the evaluation set. We therefore omit the cost of s when computing the overall bit rate. V-B Rate–Accuracy Trade-Off In Fig. 5, we evaluate the trade-off between bit rate and classification accuracy of the proposed framework against three baseline families: prior token-oriented semantic communication frameworks, hand-crafted codecs, and LIC models. For a fair comparison, all baselines decode the transmitted representation into an image that is classified by the same pretrained server model, DeiT-I-Large, fixing the upper-bound accuracy under lossless transmission to 86.81 %86.81\,\%; the rate–accuracy curves thus differ only in how efficiently each method compresses and transmits the image. End-to-end frameworks, in contrast, jointly train a task-specific classifier with the transceiver, yielding a different accuracy ceiling and no standalone image, and thus are not directly comparable under this controlled protocol. The rate–accuracy trade-off of the proposed framework is characterized in the token-aligned LIC module by varying the token-selection threshold δ and switching among LIC models trained with different values of λ, where δ controls the number of selected tokens and λ determines the operating point of the LIC model. We sweep δ from 0.50.5 to 0.980.98 and λ from 0.00750.0075 to 22, where larger values of either generally yield higher bit rates and improved downstream performance. The remaining components are fixed during inference: (Ls,Le)=(7,12)(L_s,L_e)=(7,12) for layer-selective attention rollout, and a surrogate token trained with (δsur,λsur)=(0.35,0.03)( _sur, _sur)=(0.35,0.03). The effects of (Ls,Le)(L_s,L_e) and (δsur,λsur)( _sur, _sur) are further discussed in Section V-F and Section V-G, respectively. (a) Token-oriented semantic communication frameworks (b) Hand-crafted codecs (c) LIC frameworks Fig. 5: Trade-off between bit rate and classification accuracy of the proposed framework against (a) prior token-oriented semantic communication frameworks [33, 34], (b) hand-crafted codecs, and (c) LIC frameworks [26, 27, 28]. Server accuracy corresponds to the original downstream performance of DeiT-I-Large, i.e., 86.81 %86.81\,\%, attainable under lossless transmission at 24 bpp24\,bpp. Proposed denotes the achievable curve formed by connecting the Pareto-optimal operating points of the proposed framework. As shown in Fig. 5, the proposed framework achieves higher downstream accuracy at comparable bit rates to the token-oriented baselines, namely selected-patch transmission [33] and importance-aware quantization [34]. Since these baselines also exploit task relevance but transmit in the pixel domain, this gain comes mainly from the rate efficiency of token-aligned LIC. Against hand-crafted codecs (JPEG2000, WebP, and BPG), Fig. 5 shows a favorable trade-off below 2 bpp2\,bpp: the proposed framework attains 85.83 %85.83\,\% accuracy at 1.24 bpp1.24\,bpp, only 0.980.98 percentage points below the server accuracy, whereas WebP and BPG require about 1.57 bpp1.57\,bpp and 1.82 bpp1.82\,bpp to match it. A comparable advantage holds against task-agnostic LIC frameworks [26, 27, 28], where Fig. 5 shows higher accuracy at comparable bit rates below 2.5 bpp2.5\,bpp. Although the proposed selective transmission is not directly compatible with autoregressive context models such as ELIC [28], its implementation with the simpler mean-scale hyperprior still achieves a more favorable rate–accuracy trade-off than ELIC. This result highlights the benefit of task-relevance-guided latent selection over improving reconstruction-oriented entropy modeling alone. Overall, the gains against hand-crafted codecs and prior LIC frameworks confirm the benefit of incorporating task relevance, beyond optimizing reconstruction fidelity alone. V-C Classification Accuracy over Erasure Channel To evaluate the robustness of surrogate token substitution over unreliable channels, we measure the classification accuracy of the proposed framework on a packet-erasure channel. Because the LIC model uses arithmetic coding to compress both latent representations and hyperpriors, a single erasure may disrupt the synchronization of the entropy model between the encoder and decoder; we therefore packetize them and independently encode each packet to localize the impact of erasures. The hyperpriors, which provide the side information for latent decoding, are placed in a single packet assumed to be received without error. The latents are then greedily packed into the remaining packets in descending order of their per-latent self-information H^ic=−∑dlog2p(x^i,dc∣^)H_ x_i^c=- _d _2p( x^c_i,d Z). Each packet, constrained to a maximum transmission unit of 1460 bytes1460\,bytes, is then independently arithmetic-coded and transmitted over the erasure channel with erasure probability pep_e. Fig. 6: Classification accuracy over a packet-erasure channel. Original accuracy corresponds to the errorless downstream performance of the proposed framework, i.e., 85.92 %85.92\,\%, where pe=0p_e=0. The reported accuracy is averaged over 20 independent trials. We evaluate the classification accuracy of DeiT-I-Large on images compressed by the LIC model with λ=0.3λ=0.3 and transmitted under erasure probabilities pe∈0.05,0.1,0.15,0.2,0.3,0.4,0.5p_e∈\0.05,0.1,0.15,0.2,0.3,0.4,0.5\. The latents in erased packets are replaced by their predicted means, and the corresponding tokens are processed by one of three server-side inference strategies: direct-reconstruction inference, selected-patch inference, and surrogate token substitution. To isolate the effect of erasures, selective latent transmission is not applied, fixing the bit rate at 2.12 bpp2.12\,bpp. For reference, an image-level coding without packetization yields 2.11 bpp2.11\,bpp, indicating that packetization introduces a negligible overhead. As shown in Fig. 6, surrogate token substitution is consistently the most robust of the three strategies across the erasure probabilities. V-D Entropy-Aware Image Transmission Entropy-aware image transmission (EIT) further reduces the communication payload by allowing the client to classify sufficiently confident inputs locally without transmitting them to the server [33]. Specifically, EIT computes the min-entropy of the client-side predictive distribution, which is low for confident predictions, and invokes server-side inference only when the min-entropy is at least a threshold τ. Fig. 7: Trade-off between bit rate and classification accuracy of the proposed framework with and without entropy-aware image transmission (EIT), obtained over the sweep of (τ,δ,λ)(τ,δ,λ). Both are the achievable curve formed by connecting the Pareto-optimal operating points. We sweep τ from 0.10.1 to 11, together with reduced sets of δ and λ within the ranges of Section V-B. As shown in Fig. 7, EIT shifts the achievable curve toward lower bit rates: the framework with EIT attains 85.89 %85.89\,\% accuracy at 0.94 bpp0.94\,bpp, whereas approximately 1.24 bpp1.24\,bpp is required without EIT to attain comparable accuracy. Since the additional computation amounts to a single classification head, EIT provides a practical complement to the proposed framework, although it is not claimed as a contribution of this work. V-E Token-Aligned Learned Image Compression Fig. 8: Achieved rate–accuracy points of the proposed framework. Achievable accuracy is formed by connecting the Pareto-optimal operating points. Each solid line connects the operating points obtained with the same LIC model parameters while varying only δ. We first analyze the rate-control behavior of the proposed framework through its two hyperparameters δ and λ. Fig. 8 shows the operating points obtained by varying δ and switching among LIC models trained with different λ, whose connected achievable curve enables adaptive rate control under varying channel or latency conditions. Adjusting only δ at inference provides fine-grained control within a narrow bit-rate range without switching the LIC model, while five LIC models with λ∈0.03,0.1,0.2,0.4,1λ∈\0.03,0.1,0.2,0.4,1\ suffice to cover the range from 0.5 bpp0.5\,bpp to 2.5 bpp2.5\,bpp at near-optimal accuracy, keeping the memory cost of rate control low. Fig. 9: Trade-off between bit rate and classification accuracy for LIC models without and with mean imputation. Without mean imputation refers to the hyperprior-based LIC model [26] without mean imputation, whereas with mean imputation denotes the model augmented with mean hyperpriors [27], where the missing latent vectors are imputed using the predicted means. Both models are trained with λ=0.1λ=0.1. To assess the effect of mean imputation, we compare hyperprior-based LIC models without and with mean imputation [26, 27], using the same settings as the main experiments except that the surrogate token is trained in an LIC-agnostic manner for fairness. As shown in Fig. 9, the model with mean hyperpriors attains higher downstream accuracy. This gain does not come from the unsent regions, which are uniformly replaced by a surrogate token before inference, but from improved reconstruction of the selected regions: since LIC latents can influence neighboring patches, the selected patches are affected by adjacent unsent latents, and mean imputation alleviates this effect. V-F Layer-Selective Attention Rollout Fig. 10: Selected-patch inference accuracy under layer-selective attention rollout computed with different layer ranges (Ls,Le)(L_s,L_e). The end layer is fixed as Le=L_e=L, where L=12L=12 for DeiT-Tiny. We first identify the most task-relevant layer range (Ls,Le)(L_s,L_e), motivated by the noisy early-layer attentions observed in Fig. 4. Fixing Le=L_e=L and ablating the LIC module, we measure selected-patch inference accuracy in which DeiT-Tiny selects the top-6060, top-8080, or top-100100 tokens by layer-selective attention rollout and DeiT-I-Large classifies from the selected patches. As shown in Fig. 10, aggregating over the latter half of the layers, particularly with Ls=7L_s=7, yields the best performance in our main operating region. Fig. 11: Selected-patch inference accuracy using different attention-guided importance. Selected token rate is defined as the ratio of selected tokens to the total number of tokens. We then compare layer-selective attention rollout with (Ls,Le)=(7,12)(L_s,L_e)=(7,12) against other attention-guided importance methods under the same setting: last-layer attention [33], attention rollout [35], and gradient-weighted attention [36]. As shown in Fig. 11, layer-selective attention rollout attains the most favorable rate–accuracy trade-off in our main operating region, even without the additional gradient computation required by gradient-weighted attention. (a) DeiT-Small for client (b) DeiT-Base for client Fig. 12: Selected-patch inference accuracy using different attention-guided importance with the client-side ViT replaced by (a) DeiT-Small and (b) DeiT-Base, both having L=12L=12 transformer layers. Layer-selective attention rollout is computed over (Ls,Le)=(5,12)(L_s,L_e)=(5,12) and (7,12)(7,12), respectively. Moreover, we verify that the effectiveness of layer-selective attention rollout is not specific to the choice of client-side ViT by repeating the comparison with DeiT-Tiny replaced by DeiT-Small and DeiT-Base. Both backbones comprise L=12L=12 transformer layers, for which we set (Ls,Le)=(5,12)(L_s,L_e)=(5,12) and (7,12)(7,12), respectively, searched by the same procedure used for DeiT-Tiny. As shown in Fig. 12, layer-selective attention rollout consistently outperforms last-layer attention and attention rollout, and performs comparably to gradient-weighted attention on both backbones. Crucially, layer-selective attention rollout attains this accuracy with only a single forward pass, whereas the gradient computation required by gradient-weighted attention becomes increasingly impractical for these larger backbones—making layer-selective attention rollout the more practical choice at comparable accuracy. V-G Surrogate Token Substitution The surrogate token is trained by emulating the proposed pipeline under training hyperparameters δsur _sur and λsur _sur, which set the token-selection threshold and the LIC operating point during training, respectively; the detailed settings are given in Section V-A. Because these hyperparameters determine the quality of training-time inputs, they can affect both training stability and final accuracy. Fig. 13: Evaluation accuracy for surrogate tokens trained under different token-selection thresholds δsur _sur. We evaluate under the test-time thresholds δ∈0.6,0.7,0.79δ∈\0.6,0.7,0.79\. The annotations indicate the corresponding values of δsur _sur for each training token rate. We first determine δsur _sur by ablating the LIC module and training a separate surrogate token for each value of δsur _sur from 0.250.25 to 0.980.98. Fig. 13 evaluates these tokens at test-time thresholds δ∈0.6,0.7,0.79δ∈\0.6,0.7,0.79\ in our main operating region, showing that δsur=0.35 _sur=0.35 consistently attains near-optimal accuracy despite the mismatch between δ and δsur _sur. (a) Server-side inference strategies (b) LIC-aware vs. LIC-agnostic training Fig. 14: Trade-off between bit rate and classification accuracy of surrogate token substitution. (a) Comparison of server-side inference strategies, where the surrogate token is trained with (δsur,λsur)=(0.35,0.03)( _sur, _sur)=(0.35,0.03). (b) Comparison of surrogate tokens trained with and without the LIC module. Fixing δsur=0.35 _sur=0.35, we determine λsur=0.03 _sur=0.03 in the same manner. With these hyperparameters, Fig. 14 shows that surrogate token substitution outperforms both adaptation-free baselines. Furthermore, Fig. 14 shows that the LIC-aware surrogate token, trained with (δsur,λsur)=(0.35,0.03)( _sur, _sur)=(0.35,0.03), consistently outperforms the LIC-agnostic counterpart trained with δsur=0.35 _sur=0.35 alone, indicating that incorporating LIC awareness during training yields additional gains. VI Conclusion We proposed a modular, token-oriented semantic communication framework that coordinates pretrained client-side ViT, LIC, and server-side ViT models for client–server collaborative inference. To improve the rate–accuracy trade-off, we developed three technical components: (1) token-aligned learned image compression, which uses token-level task relevance to selectively transmit spatially aligned LIC latents; (2) layer-selective attention rollout, which efficiently estimates token relevance from a selected range of attention layers; and (3) learnable surrogate token substitution, which compensates for missing visual information at the server. Experimental results demonstrated that the proposed framework achieves a more favorable rate–accuracy trade-off than recent token-oriented semantic communication schemes, hand-crafted codecs, and learned image compression models. Overall, these results indicate that token-aligned compression of the input image, rather than direct compression of token representations, is a promising direction for modular token-level semantic communication in resource-constrained edge AI systems. References [1] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin (2017) Attention is all you need. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), p. 5998–6008. Cited by: §I. [2] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, et al. (2021) An image is worth 16x16 words: transformers for image recognition at scale. In Proc. Int. Conf. Learn. Representations (ICLR), Cited by: §I, §I, §I-C, §IV-A, §IV-B. [3] Y. Gong, Y. Chung, and J. Glass (2021) AST: audio spectrogram transformer. In Proc. Interspeech, p. 571–575. Cited by: §I. [4] A. Arnab, M. Dehghani, G. Heigold, C. Sun, M. Lucic, and C. Schmid (2021) ViViT: a video vision transformer. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV), p. 6836–6846. Cited by: §I. [5] G. Bertasius, H. Wang, and L. Torresani (2021) Is space-time attention all you need for video understanding?. In Proc. Int. Conf. Mach. Learn. (ICML), p. 813–824. Cited by: §I. [6] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021) Learning transferable visual models from natural language supervision. In Proc. Int. Conf. Mach. Learn. (ICML), p. 8748–8763. Cited by: §I. [7] C. Zhang, H. Zou, S. Lasaulce, W. Saad, M. Kountouris, and M. Bennis (2022) Goal-oriented communications for the IoT and application to data compression. IEEE Internet Things Mag. 5 (4), p. 58–63. Cited by: §I. [8] G. Shi, Y. Xiao, Y. Li, and X. Xie (2021) From semantic communication to semantic-aware networking: model, architecture, and open problems. IEEE Commun. Mag. 59 (8), p. 44–50. Cited by: §I. [9] Q. Lan, D. Wen, Z. Zhang, Q. Zeng, X. Chen, P. Popovski, and K. Huang (2021) What is semantic communication? A view on conveying meaning in the era of machine intelligence. J. Commun. Inf. Netw. 6 (4), p. 336–371. Cited by: §I. [10] D. Gündüz, Z. Qin, I. E. Aguerri, H. S. Dhillon, Z. Yang, A. Yener, K. K. Wong, and C. Chae (2023) Beyond transmitting bits: context, semantics, and task-oriented communications. IEEE J. Sel. Areas Commun. 41 (1), p. 5–41. Cited by: §I. [11] H. Xie and Z. Qin (2021) A lite distributed semantic communication system for internet of things. IEEE J. Sel. Areas Commun. 39 (1), p. 142–153. Cited by: §I. [12] Y. Kim, J. Shin, Y. Cassuto, and L. R. Varshney (2023) Distributed boosting classification over noisy communication channels. IEEE J. Sel. Areas Commun. 41 (1), p. 141–154. Cited by: §I. [13] Y. Kim, Y. Cassuto, and L. R. Varshney (2020) Distributed boosting classifiers over noisy channels. In Proc. Asilomar Conf. Signals, Syst. Comput., p. 1491–1496. Cited by: §I. [14] L. Qiao, M. B. Mashhadi, Z. Gao, R. Tafazolli, M. Bennis, and D. Niyato (2025) Token communications: a large model-driven framework for cross-modal context-aware semantic communications. IEEE Wireless Commun. Mag. 32 (5), p. 80–88. Cited by: §I, §I, §I-A, TABLE I. [15] B. Liu, L. Qiao, Y. Wang, Z. Gao, Y. Ma, K. Ying, and T. Qin (2025) Text-guided token communication for wireless image transmission. In Proc. IEEE/CIC Int. Conf. Commun. China (ICCC), p. 1–6. Cited by: §I. [16] J. Peng, H. Xing, Z. Xiao, L. Xu, and X. Lei (2025) Large model empowered multi-modal semantic communication with selective tokens for training. IEEE Signal Process. Lett. 32, p. 2967–2971. Cited by: §I. [17] A. Devoto, J. Pomponi, S. Petruzzi, P. Di Lorenzo, and S. Scardapane (2024) Adaptive semantic token selection for AI-native goal-oriented communications. In Proc. IEEE Global Commun. Conf. Workshop (GC Wkshps), p. 1–6. Cited by: §I. [18] A. Devoto, J. Pomponi, M. Merluzzi, P. Di Lorenzo, and S. Scardapane (2026) Adaptive semantic token communication for transformer-based edge inference. IEEE Trans. Mach. Learn. Commun. Netw. 4, p. 422–437. Cited by: §I. [19] H. Wei, W. Ni, W. Wang, W. Xu, D. Niyato, and P. Zhang (2026) Token communication in the era of large models: an information bottleneck-based approach. IEEE Wireless Commun. Lett. 15, p. 186–190. Cited by: §I. [20] S. Kornblith, M. Norouzi, H. Lee, and G. Hinton (2019) Similarity of neural network representations revisited. In Proc. Int. Conf. Mach. Learn. (ICML), p. 3519–3529. Cited by: §I. [21] A. van den Oord, O. Vinyals, and K. Kavukcuoglu (2017) Neural discrete representation learning. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), p. 6306–6315. Cited by: §I. [22] Q. Hu, G. Zhang, Z. Qin, Y. Cai, G. Yu, and G. Y. Li (2023) Robust semantic communications with masked VQ-VAE enabled codebook. IEEE Trans. Wireless Commun. 22 (12), p. 8707–8722. Cited by: §I, §I-A, TABLE I. [23] G. Zhang, Q. Hu, Z. Qin, Y. Cai, G. Yu, and X. Tao (2024) A unified multi-task semantic communication system for multimodal data. IEEE Trans. Commun. 72 (7), p. 4101–4116. Cited by: §I, §I-A, TABLE I. [24] Q. Yu, M. Weber, X. Deng, X. Shen, D. Cremers, and L. Chen (2024) An image is worth 32 tokens for reconstruction and generation. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), p. 128940–128966. Cited by: §I, §I-A, TABLE I. [25] J. Ballé, V. Laparra, and E. P. Simoncelli (2017) End-to-end optimized image compression. In Proc. Int. Conf. Learn. Representations (ICLR), Cited by: §I, §I-B. [26] J. Ballé, D. Minnen, S. Singh, S. J. Hwang, and N. Johnston (2018) Variational image compression with a scale hyperprior. In Proc. Int. Conf. Learn. Representations (ICLR), Cited by: §I, §I, §I-B, §I-B, §IV-A1, §IV-A, Fig. 5, Fig. 9, 1st item, §V-A, §V-B, §V-E, TABLE I. [27] D. Minnen, J. Ballé, and G. D. Toderici (2018) Joint autoregressive and hierarchical priors for learned image compression. In Proc. Adv. Neural Inf. Process. Syst. (NeurIPS), p. 10771–10780. Cited by: §I, §I, §I-B, §I-B, §IV-A1, §IV-A3, §IV-A, Fig. 5, Fig. 9, 1st item, §V-A, §V-B, §V-E. [28] D. He, Z. Yang, W. Peng, R. Ma, H. Qin, and Y. Wang (2022) ELIC: efficient learned image compression with unevenly grouped space-channel contextual adaptive coding. In Proc. IEEE Conf. Comput. Vis. Pattern Recognition (CVPR), p. 5708–5717. Cited by: §I, §I, §I-B, §IV-A, Fig. 5, §V-B. [29] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou (2021) Training data-efficient image transformers & distillation through attention. In Proc. Int. Conf. Mach. Learn. (ICML), p. 10347–10357. Cited by: §I, §I, Fig. 3, §IV-A, §IV-B, §V-A, TABLE I. [30] H. Touvron, M. Cord, and H. Jégou (2022) DeiT I: revenge of the ViT. In Proc. European Conf. Comput. Vis. (ECCV), p. 516–533. Cited by: §I, §I, §IV-A, §IV-C, §V-A, §V-A, TABLE I. [31] T. Liu, P. Li, Y. Gu, and P. Liu (2023) Efficient transformer inference for extremely weak edge devices using masked autoencoders. In Proc. IEEE Int. Conf. Commun. (ICC), p. 1718–1723. Cited by: 2nd item, §I, 1st item, §I-A, TABLE I, §IV-B. [32] T. Liu, P. Li, Y. Gu, P. Liu, and H. Wang (2024) Adaptive offloading of transformer inference for weak edge devices with masked autoencoders. ACM Trans. Sensor Netw.. Cited by: 2nd item, §I, 1st item, §I-A, TABLE I, §IV-B. [33] J. Im, N. Kwon, T. Park, J. Woo, J. Lee, and Y. Kim (2024) Attention-aware semantic communications for collaborative inference. IEEE Internet Things J. 11 (22), p. 37008–37020. Cited by: 2nd item, §I, §I, 1st item, §I-A, TABLE I, §IV-A, §IV-A, §IV-B, §IV-B, Fig. 5, §V-B, §V-D, §V-F. [34] J. Park, Y. Oh, Y. Kim, and Y. Jeon (2025) Vision transformer-based semantic communications with importance-aware quantization. IEEE Internet Things J. 12 (17), p. 35662–35677. Cited by: 2nd item, §I, §I, 1st item, §I-A, TABLE I, §IV-A, §IV-B, Fig. 5, §V-B. [35] S. Abnar and W. Zuidema (2020) Quantifying attention flow in transformers. In Proc. Annu. Meeting Assoc. Comput. Linguistics (ACL), p. 4190–4197. Cited by: 2nd item, 2nd item, §IV-B, §IV-B, §V-F. [36] H. Chefer, S. Gur, and L. Wolf (2021) Generic attention-model explainability for interpreting bi-modal and encoder-decoder transformers. In Proc. IEEE Int. Conf. Comput. Vis. (ICCV), p. 397–406. Cited by: 2nd item, 3rd item, §IV-B, §IV-B, §V-F. [37] M. Jia, L. Tang, B. Chen, C. Cardie, S. Belongie, B. Hariharan, and S. Lim (2022) Visual prompt tuning. In Proc. European Conf. Comput. Vis. (ECCV), p. 709–727. Cited by: §I-D, §IV-C. [38] H. Bao, L. Dong, S. Piao, and F. Wei (2022) BEiT: BERT pre-training of image transformers. In Proc. Int. Conf. Learn. Representations (ICLR), Cited by: §I-D. [39] K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick (2022) Masked autoencoders are scalable vision learners. In Proc. IEEE Conf. Comput. Vis. Pattern Recognition (CVPR), p. 16000–16009. Cited by: §I-D, 2nd item. [40] H. Chefer, S. Gur, and L. Wolf (2021) Transformer interpretability beyond attention visualization. In Proc. IEEE Conf. Comput. Vis. Pattern Recognition (CVPR), p. 782–791. Cited by: §IV-B. [41] J. E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, and W. Chen (2022) LoRA: low-rank adaptation of large language models. In Proc. Int. Conf. Learn. Representations (ICLR), Cited by: §IV-C. [42] B. X. B. Yu, J. Chang, H. Wang, L. Liu, S. Wang, Z. Wang, J. Lin, L. Xie, H. Li, Z. Lin, Q. Tian, and C. W. Chen (2024) Visual tuning. ACM Comput. Surv. 56 (12), p. 1–38. Cited by: §IV-C. [43] J. Deng, W. Dong, R. Socher, L. Li, K. Li, and L. Fei-Fei (2009) ImageNet: a large-scale hierarchical image database. In Proc. IEEE Conf. Comput. Vis. Pattern Recognition (CVPR), p. 248–255. Cited by: §V-A, TABLE I.