Paper deep dive
Adversarial Attacks on Deep OCR Systems
Wenbo Sun, Hongzong LI, Yanyun Wang, Jiahao MA, Shuxin Zhuang, Rong Feng, Shiqin Tang, Zi Liang
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 8/11/2026, 3:41:09 AM
Summary
This paper introduces the first pure black-box adversarial attack against generative OCR vision-language models, specifically targeting Deep-OCR. The method formulates the attack as a zeroth-order optimization problem using a bounded scalar loss based on sequence similarity between the original and perturbed output strings. It employs a random-direction finite-difference gradient estimator to maintain query efficiency independent of image dimensions, combined with an Adam optimizer and L-infinity projection to ensure imperceptible perturbations. Pilot experiments reveal severe qualitative failures in Deep-OCR, such as repetition and truncation, though controlled targeted rewriting remains challenging.
Entities (9)
Relation Signals (8)
Deep-OCR → istargetof → Black-box Adversarial Attack
confidence 95% · present... the first pure black-box adversarial attack against a generative OCR vision-language model... Pilot experiments on Deep-OCR validate the string-only attack
Black-box Adversarial Attack → usesalgorithm → Adam optimizer
confidence 95% · An Adam update with ell_infinity projection yields imperceptible perturbations
Black-box Adversarial Attack → usestechnique → Random-direction Finite-difference
confidence 92% · estimate the gradient with a random-direction finite-difference scheme
Black-box Adversarial Attack → usestechnique → Zeroth-Order Optimization
confidence 92% · recast the attack as a zeroth-order optimization problem
Zeroth-Order Optimization → usesmetric → Sequence Similarity
confidence 90% · driven by a bounded scalar loss defined directly on the string output via sequence similarity
Black-box Adversarial Attack → achievesobjective → Untargeted Degradation
confidence 88% · yields imperceptible perturbations for both untargeted and targeted objectives
Black-box Adversarial Attack → achievesobjective → Targeted Rewriting
confidence 88% · yields imperceptible perturbations for both untargeted and targeted objectives
Deep-OCR → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Deep-OCR (DeepSeek-OCR) advances document recognition by treating the visual modality as an optical compression medium, enabling long-context OCR at low token cost. However, its increased complexity may introduce new security vulnerabilities. In this paper, we present, to the best of our knowledge, the first pure black-box adversarial attack against a generative OCR vision-language model, where only the decoded string can be queried and no gradients, logits, or model internals are available. We recast the attack as a zeroth-order optimization problem driven by a bounded scalar loss defined directly on the string output via sequence similarity, and estimate the gradient with a random-direction finite-difference scheme whose query cost is independent of the image dimension. An Adam update with ell_infinity projection yields imperceptible perturbations for both untargeted and targeted objectives. Pilot experiments on Deep-OCR validate the string-only attack and evaluation pipeline and expose severe qualitative decoder failures, including repetition, truncation, and prompt leakage. They also show that controlled targeted rewriting remains substantially harder than untargeted degradation; we avoid claiming targeted success until the pre-registered evaluation is complete.
Tags
Links
- Source: https://arxiv.org/abs/2608.07636v1
- Canonical: https://arxiv.org/abs/2608.07636v1
Trouble viewing inline? Open PDF directly →
Full Text
43,111 characters extracted from source content.
Expand or collapse full text
Adversarial Attacks on Deep OCR Systems Wenbo Sun1, Hongzong LI211footnotemark: 1, Yanyun Wang3, Jiahao MA4, Shuxin Zhuang5,6, Rong Feng5,6, Shiqin Tang6, Zi Liang7 1Nanjing University of Aeronautics and Astronautics 2Northwestern Polytechnical University 3The Chinese University of Hong Kong 4The University of Hong Kong 5City University of Hong Kong 6Centre for Artificial Intelligence and Robotics, Chinese Academy of Sciences 7The Hong Kong Polytechnic University wenbosun@nuaa.edu.cn, lihongzong@nwpu.edu.cn, yanyunwang@se.cuhk.edu.hk, jiahao.ma@connect.hku.hk shuxin.zhuang, rongfeng3-c@my.cityu.edu.hk, shiqin.tang@cair-cas.org.hk, zi1415926.liang@connect.polyu.hk Equal contribution. Corresponding author. Abstract Deep-OCR (DeepSeek-OCR) advances document recognition by treating the visual modality as an optical compression medium, enabling long-context OCR at low token cost. However, its increased complexity may introduce new security vulnerabilities. In this paper, we present, to the best of our knowledge, the first pure black-box adversarial attack against a generative OCR vision-language model, where only the decoded string can be queried and no gradients, logits, or model internals are available. We recast the attack as a zeroth-order optimization problem driven by a bounded scalar loss defined directly on the string output via sequence similarity, and estimate the gradient with a random-direction finite-difference scheme whose query cost is independent of the image dimension. An Adam update with ℓ∞ _∞ projection yields imperceptible perturbations for both untargeted and targeted objectives. Pilot experiments on Deep-OCR validate the string-only attack and evaluation pipeline and expose severe qualitative decoder failures, including repetition, truncation, and prompt leakage. They also show that controlled targeted rewriting remains substantially harder than untargeted degradation; we avoid claiming targeted success until the pre-registered evaluation is complete. Adversarial Attacks on Deep OCR Systems Wenbo Sun1†thanks: Equal contribution., Hongzong LI211footnotemark: 1, Yanyun Wang3, Jiahao MA4, Shuxin Zhuang5,6, Rong Feng5,6, Shiqin Tang6, Zi Liang7†thanks: Corresponding author. 1Nanjing University of Aeronautics and Astronautics 2Northwestern Polytechnical University 3The Chinese University of Hong Kong 4The University of Hong Kong 5City University of Hong Kong 6Centre for Artificial Intelligence and Robotics, Chinese Academy of Sciences 7The Hong Kong Polytechnic University wenbosun@nuaa.edu.cn, lihongzong@nwpu.edu.cn, yanyunwang@se.cuhk.edu.hk, jiahao.ma@connect.hku.hk shuxin.zhuang, rongfeng3-c@my.cityu.edu.hk, shiqin.tang@cair-cas.org.hk, zi1415926.liang@connect.polyu.hk 1 Introduction Optical character recognition (OCR) has evolved from CTC-based sequence recognizers Shi et al. (2017) into large vision–language models (VLMs) that read a document page end-to-end and emit structured markup. This progression spans OCR-free document transformers such as Donut Kim et al. (2022) and Nougat Blecher et al. (2023), unified OCR systems such as GOT-OCR2.0 Wei et al. (2024), and general-purpose models such as Qwen2.5-VL Bai et al. (2025), InternVL Chen et al. (2024), and the GPT-4 family Achiam et al. (2024). Deep-OCR further treats vision as an optical compression medium, encoding thousands of textual tokens into hundreds of vision tokens Wei et al. (2025). Because OCR outputs feed databases, retrieval systems, and agents, corrupted transcriptions can affect financial or legal decisions. Most adversarial attacks assume a fixed label space and either differentiable logits Goodfellow et al. (2014); Madry et al. (2019); Carlini and Wagner (2017) or continuous query scores Chen et al. (2017); Ilyas et al. (2018). A deployed OCR-VLM instead returns only a variable-length string while JPEG encoding, dynamic tiling, file I/O, and decoding break the gradient path. This string-only setting creates three obstacles: no differentiable supervision, prohibitive coordinate-wise query cost, and no fixed “misclassification” event for defining success. We formulate both untargeted degradation and targeted rewriting as constrained zeroth-order optimization over a bounded sequence-similarity loss computed solely from decoded strings. Gaussian random-direction finite differences perturb all pixels simultaneously, reducing each update to 2q2q queries independent of image dimension; projected Adam Kingma and Ba (2015) keeps every iterate inside the pixel box and ℓ∞ _∞ budget. We instantiate the method on Deep-OCR because its open weights permit reproducible evaluation while its pipeline retains the non-differentiable stages of a deployed service. The method itself requires only an image-in/string-out oracle. Our pilot validates that a string-only loss can drive the full query and evaluation pipeline and reveals catastrophic qualitative failures on official demonstration pages. On the formal pilot, however, none of the nine completed untargeted pages crosses the strict ρ≤0.05ρ≤ 0.05 threshold, and controlled targeted rewriting is not achieved. We therefore separate these diagnostic results from the pre-registered final comparison in Section 5, rather than extrapolating from incomplete runs. • New Perspective on Attacking Generative OCR. We formalize a string-only threat model and a pure black-box attack supporting untargeted degradation and targeted rewriting. • Query-Efficient Attack Design. We introduce a bounded string loss, a random-direction estimator whose query cost is independent of image resolution, and a projected Adam update. 2 Background & Related Work 2.1 Deep-OCR CRNN Shi et al. (2017) couples a CNN feature extractor with an RNN sequence model for end-to-end recognition. Its CTC objective treats output tokens as independent, so it ignores language priors and accumulates semantic errors on long text. Transformer-OCR Li et al. (2021) replaces the recurrence with self-attention and captures global dependencies, at the cost of attention that scales quadratically with sequence length. Diffusion-OCR Fujitake (2023) learns the reverse mapping from a noisy sequence to the correct text, restoring the image before recognition and improving robustness on low-quality inputs. Deep-OCR Wei et al. (2025) uses the visual modality as a compression medium for text, encoding a page of thousands of textual tokens into hundreds of vision tokens and adding an optical forgetting mechanism for super-long contexts. Architecturally it inherits the unified end-to-end formulation of the OCR-2.0 line Wei et al. (2024), folding layout analysis, detection, and recognition into a single generative model. We take it as our target because this added complexity may also add attack surface. 2.2 Adversarial Attacks Ever since imperceptible perturbations were shown to flip the prediction of an otherwise accurate network Szegedy et al. (2013), a large body of work has studied how such perturbations can be constructed. The methods below assume white-box access, that is, the ability to differentiate through the model. FGSM Goodfellow et al. (2014) takes a single step of size ε along the sign of the input gradient: x′=x+ε⋅sign(∇xJ(θ,x,y))x =x+ ·sign( _xJ(θ,x,y)) (1) PGD Madry et al. (2019) iterates FGSM with a smaller step α and projects back onto the ϵε-ball after each step, accumulating gradient information within the budget: xi+1′=Projx,ϵxi′+α⋅sign(∇xi′L(f(xi′),y))x_i+1 =Proj_x,ε \x_i +α·sign ( _x_i L (f(x_i ),y ) ) \ (2) C&W Carlini and Wagner (2017) instead solves for the smallest perturbation that still causes misclassification, trading off distortion against an attack term: minδ(x,x+δ)+c⋅f(x+δ) _δD(x,x+δ)+c· f(x+δ) (3) EoT Athalye et al. (2018b) optimizes the perturbation in expectation over a distribution of input transformations, so that it survives the variations encountered in practice. Patch Attack Brown et al. (2017) confines the perturbation to a small but visible region that can be printed and placed in the physical world. 2.3 Black-box Attacks When gradients are unavailable, attacks are usually grouped by how much feedback the model returns. Transfer-based. The adversary trains a substitute model on queried input–output pairs and attacks it with a white-box method, hoping the perturbation transfers Papernot et al. (2016). Transferability is unreliable for a target as specialized as a document-parsing VLM, and building a substitute for it would itself require a large labeled corpus. Score-based. Given a continuous confidence score, the gradient can be estimated from function values alone. ZOO Chen et al. (2017) perturbs one coordinate at a time; NES-style Wierstra et al. (2014); Ilyas et al. (2018) and SPSA-style Spall (1992); Uesato et al. (2018) estimators instead perturb all coordinates along random directions, a family whose convergence is well characterized for Gaussian smoothing Nesterov and Spokoiny (2017). Such gradient-free evaluation is also the standard way to expose defenses that merely obfuscate gradients Athalye et al. (2018a). All of these methods, however, presuppose a scalar score that our target never exposes. Decision-based. Boundary Attack Brendel et al. (2018) and HopSkipJumpAttack Chen et al. (2019) weaken the assumption further and use only the predicted hard label. Even this is richer feedback than we assume: a hard label is one element of a known finite set, whereas the observable here is a variable-length string over a large vocabulary, for which no notion of “decision boundary” is defined. 2.4 Attacks on OCR and Vision-Language Models Prior attacks on OCR target modular pipelines with differentiable recognizers, either by perturbing rendered text images Song and Shmatikov (2018) or by embedding adversarial watermarks Chen and Xu (2020); both rely on gradients of a CTC or cross-entropy loss. Recent work on vision-language models shows that images can degrade captioning quality Zhao et al. (2023), jailbreak an aligned model Qi et al. (2023), or steer generation toward attacker-chosen text Bailey et al. (2023), but these attacks differentiate through open-weight surrogates or the victim itself. To the best of our knowledge, no prior work attacks a generative OCR model when the decoded string is the only observable. 3 Threat Model We formalize the adversary against a generative OCR vision-language model by specifying its goal, knowledge, and capability, following the standard convention in the adversarial machine learning literature Carlini and Wagner (2017); Madry et al. (2019). Throughout, we denote the target model by f, which maps an input image ∈[0,1]C×H×W x∈[0,1]^C× H× W to a decoded character string f()∈Σ∗f( x)∈ ^*, where Σ is the output vocabulary. Unlike a classifier, f produces a variable-length sequence generated autoregressively, so the attack surface is a discrete string rather than a fixed set of class logits. Adversary’s goal. We consider two objectives. In the untargeted setting, the adversary seeks a perturbed image ′ x whose decoded string f(′)f( x ) deviates as much as possible from the clean transcription 0=f() y_0=f( x), i.e. it degrades recognition fidelity and induces omissions, substitutions, or hallucinated content. In the targeted setting, the adversary chooses a string t y_t in advance and steers the output toward it, so that f(′)≈tf( x )≈ y_t. The targeted objective is strictly stronger and directly models integrity attacks such as tampering with the recognized content of contracts, invoices, or identity documents. Adversary’s knowledge. We assume a pure black-box setting, which is the most restrictive and realistic threat for a deployed OCR service accessed through an API. The adversary can only submit an image and observe the returned decoded string. Crucially, it has no access to model architecture, weights, gradients, output logits, token-level probabilities, attention maps, or any other internal state. This is markedly weaker than the white-box assumption of FGSM Goodfellow et al. (2014), PGD Madry et al. (2019), and C&W Carlini and Wagner (2017), and also weaker than score-based black-box attacks Chen et al. (2017); Ilyas et al. (2018) that require a continuous confidence score. It is even weaker than the decision-based setting Brendel et al. (2018); Chen et al. (2019), in which the adversary at least receives a hard label drawn from a known finite set: here the only observable is the final text string. Adversary’s capability. The adversary may add an additive perturbation =′− δ= x - x that is bounded in the ℓ∞ _∞ norm, ∥∞≤ϵ δ _∞≤ε, and must keep the result a valid image, ′∈[0,1]C×H×W x ∈[0,1]^C× H× W. The small budget ϵε enforces imperceptibility, so that the adversarial image looks visually identical to a human reader. The adversary is further constrained by a query budget: it can issue only a limited number of forward queries to f, reflecting the cost, latency, and rate limits of a real API. An effective attack must therefore be both imperceptible and query-efficient. 4 Method Our method casts the pure black-box attack as a constrained zeroth-order optimization problem driven entirely by the decoded string. We first define a bounded scalar loss over string outputs (§4.2), then estimate its gradient with a random-direction finite-difference scheme whose per-query cost is independent of the image dimension (§4.3), and finally apply an Adam update with ℓ∞ _∞ projection (§4.4). Figure 1 gives an overview of the resulting attack loop and of the information available to the adversary at each stage; the full procedure is summarized in Algorithm 1. current iterate ′ x draw q directions i∼(,d) u_i ( 0, I_d) probe pair Π(′±hi) ( x ± h\, u_i) serialize JPEG q95, 4:4:4 Deep-OCR (frozen) encoder → vision tokens → decoder strip layout markers result_ori.mmd → result.mmd sequence similarity ρ=2M/(|a|+|b|)ρ=2M/( a + b ) scalar loss ℓ±∈[0,1] ^±∈[0,1] aggregate ^=1q∑iℓ+−ℓ−2hi g= 1q _i ^+- ^-2h u_i Adam step, then project Π adversary endserver end (API)2q2q probe queriesper iteration no gradients, logits, token probabilities, or attention evaluate ℓ=ℒ(f(′),∗) =L(f( x ), y_ ), keep best ⋆ x , stop early if ℓ≤αℒ0 ≤ _0 only the decoded string f(′)f( x ) crosses the boundary; total cost (2q+1)T(2q+1)\,T queries, independent of d Figure 1: Overview of the proposed pure black-box attack. The adversary, on the left of the dashed boundary, never observes gradients, logits, token probabilities, or attention maps: the only signal returned by the frozen Deep-OCR service is a decoded Markdown string. Each iteration draws q random directions, spends 2q2q probe queries plus one evaluation query, turns the returned strings into a bounded scalar loss through the sequence-similarity ratio ρ of Eq. (5), aggregates the finite-difference estimate g of Eq. (8), and applies an Adam step followed by the projection Π of Eq. (13) onto the ℓ∞ _∞ ball intersected with [0,1]d[0,1]^d. Symbols match Algorithm 1. 4.1 Problem Formulation Let d=C×H×Wd=C× H× W be the input dimension. Given a clean image x, the attack solves min′ _ x ℒ(f(′),∗) \! (f( x ),\, y_ ) (4) s.t. ∥′−∥∞≤ϵ, x - x _∞≤ε, ′∈[0,1]d, x ∈[0,1]^d, where ∗=0 y_ = y_0 for the untargeted objective and ∗=t y_ = y_t for the targeted objective, and ℒL is the string-level loss defined below. Because f is only accessible as a black box that returns a string, ℒL is non-differentiable and its analytic gradient with respect to ′ x is unavailable. 4.2 String-level Loss Because the image is quantized to uint8uint8 and serialized as JPEG on each forward pass (§3), small perturbations risk being wiped out, an effect long known to blunt adversarial examples Dziugaite et al. (2016); Guo et al. (2018). The smoothing radius h must thus be wide enough to survive compression, and unlike unit-normalized sampling, our Gaussian directions i u_i provide sufficient per-coordinate magnitude for the symmetric probe pair ′±hi x ± h u_i to yield distinct observations. We require a scalar objective that can be computed solely from two strings. Let ρ(,)∈[0,1]ρ( a, b)∈[0,1] be a sequence-similarity ratio based on the total length M of the matching blocks between strings a and b, ρ(,)=2M||+||,ρ( a, b)\;=\; 2\,M a + b , (5) which equals 11 for identical strings and approaches 0 as they diverge. This is exactly the ratio computed by Ratcliff–Obershelp matching Ratcliff and Metzener (1988) and is cheap, bounded, and defined on raw text without any model internals. For the untargeted objective we set ℒunt(′)=ρ(f(′),0),L_unt( x )=ρ\! (f( x ),\, y_0 ), (6) so that minimizing ℒuntL_unt pushes the perturbed output away from the clean transcription. For the targeted objective we set ℒtar(′)=1−ρ(f(′),t),L_tar( x )=1-ρ\! (f( x ),\, y_t ), (7) so that minimizing ℒtarL_tar drives the output toward the attacker-chosen string. Both losses lie in [0,1][0,1], which stabilizes the finite-difference estimate and provides a natural stopping criterion: the attack is deemed successful once the loss falls below a small threshold τ. We stress that ρ acts purely as the attack signal; recognition quality is assessed separately with Levenshtein-based measures Levenshtein (1966), so that the reported degradation cannot be an artifact of optimizing and evaluating the very same quantity. ′ x (a) coordinate-wise2d2d queries per step′ x −^- g−∇ℒ- (b) random directions2q2q queries per step, q≪dq d[0,1]d[0,1]^d x∥∞≤ϵ δ _∞≤ stepΠ(′) ( x ) (c) projection Π : clip δ to ±ϵ±ε, then clip pixels to [0,1][0,1]; shaded area is feasible Figure 2: Geometry of the zeroth-order estimate and of the feasible-set projection. (a) Coordinate-wise finite differences probe one axis at a time and cost 2d2d queries per step, which is prohibitive for a full-resolution document image. (b) Our estimator perturbs all coordinates at once along q sampled directions; the resulting g of Eq. (8) aligns with the true gradient—which the adversary can never observe—at a cost of 2q2q queries that is independent of d. (c) A step may leave the feasible set, so the projection of Eq. (13) first clips the perturbation into the ℓ∞ _∞ ball around the clean image x and then clips the pixel values into [0,1]d[0,1]^d. 4.3 Random-Direction Zeroth-Order Gradient Estimation Since ∇′ℒ _ x L cannot be computed analytically, we estimate it from function values only, in the spirit of zeroth-order black-box attacks Chen et al. (2017); Ilyas et al. (2018) and of Gaussian-smoothed random gradient-free minimization Nesterov and Spokoiny (2017). Rather than perturbing one coordinate at a time—which costs O(d)O(d) queries per step and is prohibitive for a full-resolution document image—we use a random-direction finite-difference estimator. Figure 2 visualizes this behavior. At the current point ′ x we draw q random directions ii=1q\ u_i\_i=1^q, each sampled from a standard Gaussian, i∼(,d) u_i ( 0, I_d), and form the symmetric estimate ^=1q∑i=1qℒ(′+hi)−ℒ(′−hi)2hi, g\;=\; 1q _i=1^q L( x +h\, u_i)-L( x -h\, u_i)2h\, u_i, (8) where h>0h>0 is the smoothing radius. Each direction perturbs all coordinates simultaneously, so the estimator’s quality improves with q while the per-step query cost is 2q2q and is independent of the image dimension d. The queried points in Eq. (8) are always projected back into the feasible region (§4.4) before being passed to f, so every evaluation respects the ℓ∞ _∞ and pixel-range constraints. 4.4 Adam Update with ℓ∞ _∞ Projection Given the estimated gradient g, we take an Adam step Kingma and Ba (2015), which adapts the per-coordinate step size and greatly stabilizes optimization under the noisy zeroth-order estimate (see Figure 2(c)): t m_t =β1t−1+(1−β1)^, = _1 m_t-1+(1- _1) g, (9) t v_t =β2t−1+(1−β2)^⊙2, = _2 v_t-1+(1- _2) g 2, (10) ^t m_t =t/(1−β1t),^t=t/(1−β2t), = m_t/(1- _1^t), v_t= v_t/(1- _2^t), (11) ′ x ←′−η^t^t+ς, ← x -η\, m_t v_t+ , (12) where η is the learning rate, ⊙ denotes elementwise operations, and ς is a small constant. To satisfy the constraints of Eq. (4), we then apply the projection operator Π(′)=clip(+clip(′−,−ϵ,ϵ), 0, 1), ( x )=clip\! ( x+clip( x - x,\,-ε,\,ε),\;0,\;1 ), (13) which first clamps the perturbation into the ℓ∞ _∞ ball of radius ϵε and then clamps the pixel values into the valid range [0,1][0,1]. 4.5 Overall Algorithm Algorithm 1 combines the components above. Starting from the clean image, each iteration estimates the gradient with 2q2q black-box queries, performs one projected Adam update, and re-evaluates the loss; the best-so-far adversarial image is retained and returned, and the loop stops early once the loss drops below the threshold τ. The total query cost is approximately (2q+1)T(2q+1)\,T for T iterations and does not scale with the image resolution, making the attack practical under a tight query budget. Algorithm 1 Pure Black-box ZOO Attack on Deep-OCR 1:clean image x, model f, budget ϵε, learning rate η, radius h, directions q, iterations T, threshold τ 2:∗←0=f() y_ ← y_0=f( x) (untargeted) or t y_t (targeted) 3:′← x ← x; 0,0← m_0, v_0← 0; ℒ⋆←+∞L ←+∞ 4:for t=1t=1 to T do 5: ^← g← 0 6: for i=1i=1 to q do 7: sample i∼(,d) u_i ( 0, I_d) 8: ℓ+←ℒ(f(Π(′+hi)),∗) ^+ (f( ( x +h u_i)), y_ ) 9: ℓ−←ℒ(f(Π(′−hi)),∗) ^- (f( ( x -h u_i)), y_ ) 10: ^+=ℓ+−ℓ−2hi g += ^+- ^-2h\, u_i 11: end for 12: ^←^/q g← g/q 13: update t,t m_t, v_t and step ′←′−η^t/(^t+ς) x ← x -η\, m_t/( v_t+ ) 14: ′←Π(′) x ← ( x ) ⊳ Eq. (13) 15: ℓ←ℒ(f(′),∗) (f( x ), y_ ) 16: if ℓ<ℒ⋆ <L then ℒ⋆←ℓL ← ; ⋆←′ x ← x 17: end if 18: if ℓ<τ <τ then break ⊳ early stopping 19: end if 20:end for 21:return best adversarial image ⋆ x 5 Experiments 5.1 Experimental Setup Data and task split. We evaluate on a stratified pilot drawn from OmniDocBench Ouyang et al. (2025), which provides page-level text, formula, table, and reading-order annotations over heterogeneous document layouts. The local pool contains 20 English and four Chinese pages spanning academic articles, books, examinations, slides, textbooks, magazines, notes, reports, and newspapers. Following the identifiers used by our formal pilot runs, English pages are used for untargeted degradation and Chinese pages for targeted integrity case studies. This division keeps language fixed within each task: English supports comparable character- and word-level aggregate evaluation, while compact Chinese fields permit semantically meaningful substitutions with few Unicode characters. It is an operational design choice, not a claim that Chinese is intrinsically easier to attack. The final evaluation expands each split and fixes all page identifiers before running any attack. Target model and oracle. We use the official deepseek-ai/DeepSeek-OCR weights with Transformers 4.46.3 in bfloat16 inference mode. Every query uses base_size=1024, image_size=640, crop_mode=True, and the prompt <image> <|grounding|>Convert the document to markdown. The queried tensor is clipped to [0,1][0,1], quantized to 8-bit RGB, and serialized as JPEG quality 95 with 4:4:4 chroma before inference. Deep-OCR writes a raw result_ori.mmd, containing grounding coordinates, and a post-processed result.mmd. We consistently use the latter as f(′)f( x ): coordinate jitter would otherwise be counted as recognition error even when the decoded content is unchanged. A missing or empty result is assigned the worst loss and still consumes a query. Methods, controls, and budgets. Our method is denoted RD-ZOO throughout. The final comparison uses SPSA Spall (1992) under the same query budget, along with three non-optimized lower anchors (Gaussian noise, JPEG degradation, and Gaussian blur) and two destructive upper anchors (Gaussian noise at ϵ=64/255ε=64/255 and a blank white page). The clean output and ten repeated clean queries provide the accuracy reference and inference-noise floor, respectively. These controls distinguish optimization from generic model or codec instability; they are table placeholders until all matched runs finish. We report ϵ∈4,8,16/255ε∈\4,8,16\/255, with 8/2558/255 as the primary setting. RD-ZOO uses q=4q=4 Gaussian directions, T=20T=20 iterations, Adam learning rate 0.030.03, (β1,β2)=(0.9,0.999)( _1, _2)=(0.9,0.999), and a finite-difference radius selected from 10−4,1,2,4/255\10^-4,1,2,4\/255 by a pre-registered quantization test. One clean query, 2q2q probes, and one post-update evaluation per iteration yield at most 1+(2q+1)T=1811+(2q+1)T=181 calls per page. Every call, including an empty or failed decode, is charged. We return the best feasible iterate and stop when ℒt≤αℒ0L_t≤ _0 with α=0.05α=0.05. Randomized methods are repeated with three seeds on a fixed five-page subset. Evaluation metrics. Attack optimization observes only ρ, while final recognition quality is computed independently with character-level Levenshtein distance Levenshtein (1966) against the OmniDocBench reference. For untargeted attacks we report ASR, final ρ(f(′),0)ρ(f( x ), y_0), clean-to-adversarial CER, decoding-accuracy drop ΔDA , and successful-query median and interquartile range. A run succeeds iff its output is nonempty and ρ≤0.05ρ≤ 0.05. PSNR and SSIM Wang et al. (2004) measure perceptual quality; output-length ratio Rlen=|f(′)|/|0|R_len=|f( x )|/| y_0| and empty-output rate guard against counting a decoder crash as a useful attack. Targeted evaluation reports target similarity, specified-field hit rate, relative ASR ℒ≤0.05ℒ0L≤ 0.05L_0, and successful queries. WER, normalized edit distance, perturbation norms, repeated 4-gram ratio, total queries, and wall-clock time are retained for the appendix. 5.2 Results Pilot threshold sensitivity. Figure 3 reports the empirical ASR CDF reconstructed from the nine completed English untargeted pilot pages at ϵ=8/255ε=8/255. At the pre-registered threshold τ=0.05τ=0.05, pilot ASR is 0/9=0%0/9=0\%; the curve is included to expose threshold dependence rather than to claim a successful final attack. The repeated-clean noise floor is unavailable because the ten-query determinism experiment has not yet been run, and is therefore explicitly marked unavailable rather than estimated from attacked outputs. Figure 3: Pilot ASR sensitivity for the nine completed English untargeted RD-ZOO runs at ϵ=8/255ε=8/255. The vertical line marks τ=0.05τ=0.05. The clean-query noise floor is unavailable in this pilot. Qualitative failure modes. The official Deep-OCR demonstration pages are used only for qualitative pipeline validation and are excluded from the formal pilot. Figure 4 shows the strongest untargeted example: the image remains visually close while the output grows from 7,775 to 27,411 characters, ρ falls to 0.0571, and the decoder enters a repetition loop with prompt and grounding-token leakage. In the targeted show2 run, the requested phrase is never produced, although the output is truncated from 4,182 to 1,151 characters, illustrating the gap between destructive degradation and controlled rewriting. Clean OCR excerpt. Susan: When I am reading a story, the Latino children in my class just sit there. They look at me, but you can tell that they are not engaged in the story. Adversarial OCR excerpt. Lato children in my class just sit there. They look at me, but... is emitted repeatedly with growing punctuation, followed by <image>In<grounding>Convert... and malformed grounding tokens. Figure 4: Untargeted failure on the official show3 page. Original and adversarial inputs are visually close; the right panel visualizes the signed JPEG-domain residual at 10×10× gain (mid-gray is zero). The decoded output expands from 7,775 to 27,411 characters and enters a repetition loop, reducing similarity to the clean output to 0.0571. This demonstration page is excluded from quantitative evaluation. 5.3 Discussion What the pilot establishes. The current pilot validates the complete string-only query, accounting, and offline-evaluation pipeline, but it does not yet establish the final efficacy claim. In particular, all nine formal untargeted pilot pages miss the strict 0.050.05 success threshold, and the available Chinese targeted runs do not alter the decoded output. Reporting the threshold CDF alongside ASR prevents a single operating point from hiding this result. Final claims will be made only after completing the fixed splits, controls, and repeated seeds. Failure mechanisms. The official-page examples reveal four qualitatively different outcomes: plausible content substitutions, paragraph repetition, truncation or omission, and prompt/grounding-marker leakage. RlenR_len, empty-output rate, and repeated 4-gram ratio make these mechanisms auditable. In particular, nonempty output is required for ASR, preventing a blank decode from being misreported as a successful integrity attack. Language and targeted control. English untargeted and Chinese targeted results answer different questions and must not be pooled into a language comparison. Chinese was selected for compact, semantically consequential field edits, but the current evidence does not show that its orthography makes optimization easier. Moreover, field-level targeting imposes an unusually strict relative criterion: when only a few characters differ, a successful output must be almost identical to the full target page. Codec and constraint domain. JPEG serialization is part of the oracle and can make the maximum pixel difference after decoding exceed the pre-JPEG projected radius. We therefore report both tensor-domain constraint checks and post-JPEG perceptual metrics, state the domain beside every norm, and avoid describing post-codec L∞L_∞ as equal to ϵε. This distinction, together with the JPEG and blur anchors, is necessary to separate optimized vulnerability from ordinary codec sensitivity. 6 Conclusion and Future Work We studied adversarial robustness at the interface actually exposed by generative OCR services: an image is submitted and only a variable-length decoded string is returned. We formulated both untargeted degradation and targeted rewriting as constrained zeroth-order optimization using a bounded string-level loss, Gaussian random-direction finite differences, and projected Adam updates. The resulting query cost is independent of image dimension and requires neither gradients nor confidence scores. Our pilot validates this end-to-end attack and evaluation pipeline and reveals severe decoder failures, including repetition, truncation, and prompt leakage. At the same time, none of the completed formal pilot pages meets our strict untargeted threshold, and controlled targeted rewriting remains unsuccessful; these observations support the threat model and methodology, but not yet a general efficacy claim. Future work will complete the pre-registered matched evaluation across perturbation budgets, seeds, and document types, and will test whether the same string-only formulation transfers to other open and commercial OCR-VLMs. Promising directions include lower-dimensional or structured search, query-adaptive smoothing, and semantic objectives that remain computable from decoded text. Comparing against a white-box upper bound and evaluating adversarial training, randomized preprocessing, and output-consistency checks would further clarify how much robustness is lost specifically at the string-only interface. 7 Limitations Our evidence is currently limited to Deep-OCR and a small, incomplete pilot; the final matched controls and repeated-seed experiments are still pending. Consequently, the reported qualitative failures should not be interpreted as population-level attack rates or as evidence that the attack transfers across models, languages, or deployment pipelines. The method is also query-intensive despite its dimension-independent per-step cost, and its finite-difference signal can vanish after 8-bit quantization and JPEG encoding. The sequence-matching objective captures surface-form change rather than semantic harm, while targeted field rewriting is much harder than destructive degradation and was not achieved in the current pilot. Finally, ℓ∞ _∞, PSNR, and SSIM do not replace a human perceptual study; our threat model covers digital additive perturbations, not print–scan or other physical transformations. We do not evaluate adaptive defenses, proprietary rate limits, or a white-box performance ceiling. References J. Achiam, S. Adler, S. Agarwal, et al. (2024) GPT-4 technical report. External Links: 2303.08774 Cited by: §1. A. Athalye, N. Carlini, and D. Wagner (2018a) Obfuscated gradients give a false sense of security: circumventing defenses to adversarial examples. External Links: 1802.00420 Cited by: §2.3. A. Athalye, L. Engstrom, A. Ilyas, and K. Kwok (2018b) Synthesizing robust adversarial examples. External Links: 1707.07397 Cited by: §2.2. S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin (2025) Qwen2.5-vl technical report. External Links: 2502.13923 Cited by: §1. L. Bailey, E. Ong, S. Russell, and S. Emmons (2023) Image hijacks: adversarial images can control generative models at runtime. External Links: 2309.00236 Cited by: §2.4. L. Blecher, G. Cucurull, T. Scialom, and R. Stojnic (2023) Nougat: neural optical understanding for academic documents. External Links: 2308.13418 Cited by: §1. W. Brendel, J. Rauber, and M. Bethge (2018) Decision-based adversarial attacks: reliable attacks against black-box machine learning models. External Links: 1712.04248 Cited by: §2.3, §3. T. B. Brown, D. Mané, A. Roy, M. Abadi, and J. Gilmer (2017) Adversarial patch. External Links: 1712.09665 Cited by: §2.2. N. Carlini and D. Wagner (2017) Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium on Security and Privacy (SP), p. 39–57. External Links: Document Cited by: §1, §2.2, §3, §3. J. Chen, M. I. Jordan, and M. J. Wainwright (2019) HopSkipJumpAttack: a query-efficient decision-based attack. External Links: 1904.02144 Cited by: §2.3, §3. L. Chen and W. Xu (2020) Attacking optical character recognition (ocr) systems with adversarial watermarks. External Links: 2002.03095 Cited by: §2.4. P. Chen, H. Zhang, Y. Sharma, J. Yi, and C. Hsieh (2017) ZOO: zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security, AISec ’17, p. 15–26. External Links: Document Cited by: §1, §2.3, §3, §4.3. Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, B. Li, P. Luo, T. Lu, Y. Qiao, and J. Dai (2024) InternVL: scaling up vision foundation models and aligning for generic visual-linguistic tasks. External Links: 2312.14238 Cited by: §1. G. K. Dziugaite, Z. Ghahramani, and D. M. Roy (2016) A study of the effect of jpg compression on adversarial images. External Links: 1608.00853 Cited by: §4.2. M. Fujitake (2023) DiffusionSTR: diffusion model for scene text recognition. External Links: 2306.16707 Cited by: §2.1. I. J. Goodfellow, J. Shlens, and C. Szegedy (2014) Explaining and harnessing adversarial examples. External Links: 1412.6572 Cited by: §1, §2.2, §3. C. Guo, M. Rana, M. Cisse, and L. van der Maaten (2018) Countering adversarial images using input transformations. External Links: 1711.00117 Cited by: §4.2. A. Ilyas, L. Engstrom, A. Athalye, and J. Lin (2018) Black-box adversarial attacks with limited queries and information. In Proceedings of the 35th International Conference on Machine Learning, J. Dy and A. Krause (Eds.), Proceedings of Machine Learning Research, Vol. 80, p. 2137–2146. Cited by: §1, §2.3, §3, §4.3. G. Kim, T. Hong, M. Yim, J. Nam, J. Park, J. Yim, W. Hwang, S. Yun, D. Han, and S. Park (2022) OCR-free document understanding transformer. External Links: 2111.15664 Cited by: §1. D. P. Kingma and J. Ba (2015) Adam: a method for stochastic optimization. In 3rd International Conference on Learning Representations (ICLR), Cited by: §1, §4.4. V. I. Levenshtein (1966) Binary codes capable of correcting deletions, insertions, and reversals. Soviet Physics Doklady 10 (8), p. 707–710. Cited by: §4.2, §5.1. M. Li, T. Lv, J. Chen, L. Cui, Y. Lu, D. Florencio, C. Zhang, Z. Li, and F. Wei (2021) TrOCR: transformer-based optical character recognition with pre-trained models. External Links: 2109.10282 Cited by: §2.1. A. Madry, A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu (2019) Towards deep learning models resistant to adversarial attacks. External Links: 1706.06083 Cited by: §1, §2.2, §3, §3. Y. Nesterov and V. Spokoiny (2017) Random gradient-free minimization of convex functions. Foundations of Computational Mathematics 17 (2), p. 527–566. External Links: Document Cited by: §2.3, §4.3. L. Ouyang, Y. Qu, H. Zhou, J. Zhu, R. Zhang, Q. Lin, B. Wang, Z. Zhao, M. Jiang, X. Zhao, J. Shi, F. Wu, P. Chu, M. Liu, Z. Li, C. Xu, B. Zhang, B. Shi, Z. Tu, and C. He (2025) OmniDocBench: benchmarking diverse pdf document parsing with comprehensive annotations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p. 24838–24848. External Links: Document Cited by: §5.1. N. Papernot, P. McDaniel, I. Goodfellow, S. Jha, Z. B. Celik, and A. Swami (2016) Practical black-box attacks against machine learning. External Links: 1602.02697 Cited by: §2.3. X. Qi, K. Huang, A. Panda, P. Henderson, M. Wang, and P. Mittal (2023) Visual adversarial examples jailbreak aligned large language models. External Links: 2306.13213 Cited by: §2.4. J. W. Ratcliff and D. E. Metzener (1988) Pattern matching: the gestalt approach. Dr. Dobb’s Journal 13 (7), p. 46–72. Cited by: §4.2. B. Shi, X. Bai, and C. Yao (2017) An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (11), p. 2298–2304. External Links: Document Cited by: §1, §2.1. C. Song and V. Shmatikov (2018) Fooling ocr systems with adversarial text images. External Links: 1802.05385 Cited by: §2.4. J. C. Spall (1992) Multivariate stochastic approximation using a simultaneous perturbation gradient approximation. IEEE Transactions on Automatic Control 37 (3), p. 332–341. External Links: Document Cited by: §2.3, §5.1. C. Szegedy, W. Zaremba, I. Sutskever, J. Bruna, D. Erhan, I. Goodfellow, and R. Fergus (2013) Intriguing properties of neural networks. External Links: 1312.6199 Cited by: §2.2. J. Uesato, B. O’Donoghue, A. van den Oord, and P. Kohli (2018) Adversarial risk and the dangers of evaluating against weak attacks. External Links: 1802.05666 Cited by: §2.3. Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli (2004) Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13 (4), p. 600–612. External Links: Document Cited by: §5.1. H. Wei, C. Liu, J. Chen, J. Wang, L. Kong, Y. Xu, Z. Ge, L. Zhao, J. Sun, Y. Peng, C. Han, and X. Zhang (2024) General ocr theory: towards ocr-2.0 via a unified end-to-end model. External Links: 2409.01704 Cited by: §1, §2.1. H. Wei, Y. Sun, and Y. Li (2025) DeepSeek-ocr: contexts optical compression. External Links: 2510.18234 Cited by: §1, §2.1. D. Wierstra, T. Schaul, T. Glasmachers, Y. Sun, J. Peters, and J. Schmidhuber (2014) Natural evolution strategies. Journal of Machine Learning Research 15 (27), p. 949–980. Cited by: §2.3. Y. Zhao, T. Pang, C. Du, X. Yang, C. Li, N. Cheung, and M. Lin (2023) On evaluating adversarial robustness of large vision-language models. External Links: 2305.16934 Cited by: §2.4.