Paper deep dive
BrailleBench: Investigating Multi-Criteria Braille Comprehension in Large Language Models
Jinghan Zhang, Fengran Mo, Zhiyu Chen, Xiaoyan Han, Kunpeng Liu, Chang-Tien Lu
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 92%
Last extracted: 8/29/2026, 2:50:14 AM
Summary
The paper introduces BrailleBench, a benchmark designed to evaluate Large Language Models' (LLMs) comprehension and generation capabilities in Braille. It addresses the accessibility gap for blind and deafblind users by assessing LLMs on Braille-authored content (input), Braille expression (output), and end-to-end Braille interaction. The benchmark comprises 5,570 instances from five datasets (GSM8K, AIME 2024, CommonsenseQA, HotpotQA, 2WikiMultiHopQA) covering mathematics, commonsense, and multi-hop QA. It utilizes a deterministic pipeline via a custom Braille Toolkit based on liblouis to generate Unified English Braille (UEB) Grade 1 and Grade 2 representations without LLM-generated data, ensuring quality and preventing bias. Evaluation of six representative LLMs reveals a significant performance gap between print-English and Braille capabilities, with Grade 2 Braille showing particular fragility on the input side and asymmetric performance between understanding and expression.
Entities (14)
Relation Signals (12)
Jinghan Zhang → affiliatedwith → Clemson University
confidence 95% · Jinghan Zhang Affiliation: School of Computing Clemson University
BrailleBench → evaluates → Large Language Models
confidence 95% · BrailleBench, a benchmark for evaluating LLMs in Braille comprehension
BrailleBench → supportsstandards → Grade 1 Braille
confidence 94% · BrailleBench aligns 5,570 instances from five datasets... across English and Braille Grades 1 and 2.
BrailleBench → supportsstandards → Grade 2 Braille
confidence 94% · BrailleBench aligns 5,570 instances from five datasets... across English and Braille Grades 1 and 2.
BrailleBench → uses → Braille Toolkit
confidence 92% · the benchmark is built through a deterministic, expert-reviewed pipeline via a self-created Braille Toolkit
Braille Toolkit → basedon → liblouis
confidence 90% · We build a toolkit based on liblouis and fixed UEB tables
BrailleBench → includesdatasets → GSM8K
confidence 90% · we leverage the whole test set of GSM8K
BrailleBench → includesdatasets → HotpotQA
confidence 90% · For HotpotQA [35]
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Although Large language models (LLMs) mediate access to knowledge and computational assistance, their capabilities should benefit vulnerable groups in the same way. However, it is unclear whether existing AI systems are inclusive enough for blind and deafblind users to access the same functionality through Braille, whose indicators, contractions, and digital representations introduce distinct requirements for model comprehension. To this end, we introduce BrailleBench, a benchmark for evaluating LLMs in Braille comprehension from different Criteria. BrailleBench aligns 5,570 instances from five datasets, including mathematics, commonsense, and multi-hop question answering across English and Braille Grades 1 and 2. Different configurations are designed to understand whether the systems can comprehend Braille-authored content, express answers in Braille, and complete end-to-end Braille interaction. To ensure the quality and prevent evaluation bias, the benchmark is built through a deterministic, expert-reviewed pipeline via a self-created Braille Toolkit without using any data instances generated by LLMs. We evaluate six representative LLMs from various aspects. The results reveal a persistent gap between print-English capability and Braille accessibility. Braille understanding and expression are asymmetric, where Grade 2 is especially fragile on the input side compared to Grade 1, and fully Braille requests further reduce performance. The experimental observations provide valuable guidance for the development of future Braille AI systems. All related resources in BrailleBench are publicly available for future research.
Tags
Links
- Source: https://arxiv.org/abs/2608.27268v1
- Canonical: https://arxiv.org/abs/2608.27268v1
Trouble viewing inline? Open PDF directly →
Full Text
60,363 characters extracted from source content.
Expand or collapse full text
BrailleBench: Investigating Multi-Criteria Braille Comprehension in Large Language Models Jinghan Zhang Affiliation: School of Computing Clemson University Clemson, USA jinghaz@clemson.edu Fengran Mo Affiliation: Department of Computer Science Rochester Institute of Technology Rochester, USA frmvcs@rit.edu Zhiyu Chen Affiliation: Amazon.com, Inc. Seattle, USA zhiyuche@amazon.com Affiliation: Xiaoyan Han Affiliation: School of Computing Clemson University Clemson, USA xiaoyah@clemson.edu Kunpeng Liu Affiliation: School of Computing Clemson University Clemson, USA kunpenl@clemson.edu Chang-Tien Lu Affiliation: Department of Computer Science Virginia Tech Alexandria, USA ctlu@vt.edu Abstract Although Large language models (LLMs) mediate access to knowledge and computational assistance, their capabilities should benefit vulnerable groups in the same way. However, it is unclear whether existing AI systems are inclusive enough for blind and deafblind users to access the same functionality through Braille, whose indicators, contractions, and digital representations introduce distinct requirements for model comprehension. To this end, we introduce BrailleBench, a benchmark for evaluating LLMs in Braille comprehension from different Criteria. BrailleBench aligns 5,570 instances from five datasets, including mathematics, commonsense, and multi-hop question answering across English and Braille Grades 1 and 2. Different configurations are designed to understand whether the systems can comprehend Braille-authored content, express answers in Braille, and complete end-to-end Braille interaction. To ensure the quality and prevent evaluation bias, the benchmark is built through a deterministic, expert-reviewed pipeline via a self-created Braille Toolkit without using any data instances generated by LLMs. We evaluate six representative LLMs from various aspects. The results reveal a persistent gap between print-English capability and Braille accessibility. Braille understanding and expression are asymmetric, where Grade 2 is especially fragile on the input side compared to Grade 1, and fully Braille requests further reduce performance. The experimental observations provide valuable guidance for the development of future Braille AI systems. All related resources in BrailleBench are publicly available for future research. Index Terms: Braille comprehension, large language models, accessibility evaluation, benchmarks, inclusive AI I Introduction The development of large language models (LLMs) in the era of generative AI should be inclusive to benefit everyone. These LLM-based systems mediate access to knowledge and computational assistance [1], enabling common users to support their daily lives. However, it is unclear whether users can access the same functionality through disability-related reading and writing systems, such as Braille for blind and deaf-blind users. This is important for making AI systems truly inclusive. In terms of the interaction between blind users and AI systems, Braille is not merely an alternative rendering of speech or print. It serves as a private, editable input channel for precise technical data (e.g., natural language, code, mathematics, and notation) when speech interaction becomes impractical due to ambiguity or inefficiency. Supporting Braille in AI systems is therefore a question of interaction accessibility rather than simply adding another text format, which is underexplored in existing LLM-based systems. Earlier AI research on Braille has primarily addressed recognition and translation through rule-based, natural-language-processing, and neural approaches [2, 3, 4]; whether general-purpose LLMs can sustain task-oriented interaction through Braille remains underexplored. In this study, we aim to understand the capability of existing LLMs to comprehend Braille-authored messages by evaluating them under credible and practical criteria. To this end, we first identify three different scenarios with accessibility through a user’s reading and writing interface: (i) Braille to English to measure access to Braille-authored content, e.g., accessing reading, (i) English to Braille to measure accessible response generation, e.g., accessing writing, and (i) Braille to Braille to measure end-to-end interaction in Braille. Two complementary English Braille forms11 1 Different languages may have their own corresponding Braille systems, while our study focuses only on English-related Braille., Grade 1 [5] and Grade 2 [6] Braille, are included to distinguish basic uncontracted or contracted Braille capability from the higher level requirements of contracted Braille used in fluent reading. Grade 1 represents words without contractions, while Grade 2 replaces frequent words and letter groups with shorter but context-dependent signs, which requires more complex lexical and positional disambiguation. Intuitively, strong performance through original English-to-English interaction does not guarantee usable performance in Braille settings. The challenge for LLMs to comprehend Braille might lie in recognizing the Braille pattern in a consistent sequence rather than isolated character or word transcription. As shown in Figure 1, the way for blind people to access mathematical knowledge should be in an ASCII sequence rather than a structured equation in print English, which requires the LLMs to comprehend in the same way. Besides, different Braille standards in Grade 1 and Grade 2 will produce various semantic mapping results when given the same original input sequences. Since these capacities might not be involved in existing LLMs’ pre-training procedure, comprehensive Braille interactions cannot be guaranteed [7, 8]. To analyze such a capacity of LLMs, we construct BrailleBench, a reproducible and comprehensive evaluation framework to evaluate Braille comprehension, task completion, and accessible expression. Specifically, Braille comprehension and Braille generation are distinct and asymmetric capabilities. This is because, on the input side, a model should recover the intended linguistic and symbolic content. On the output side, it should follow the requested Grade representation rather than return print English as default and avoid code-switch [9]. To prevent the content in the benchmark from being generated by LLMs that would result in evaluation bias risk, we build a toolkit based on liblouis and fixed UEB tables to control the Braille expression quality. We evaluate six representative LLMs with detailed quantitative analysis. Our investigation shows that the existing LLMs perform in different implicit patterns when encountering a Braille environment, which should be related to the reasoning mechanisms [10, 11, 12] and whether Braille is included in the pre-training corpus. These findings provide valuable guidance for the development of future Braille AI systems. The whole benchmark22 2 BrailleBench: https://github.com/jinghanzhang1998/braillebench and toolkit33 3 Braille Toolkit: https://github.com/jinghanzhang1998/braille-toolkit for constructing Braille are available at our public repositories. (a) Mathematical expression Print EN Evaluate the limit as x approaches 00: limx→01+x+x2−1−x+x2sinx. _x→ 0 1+x+x^2- 1-x+x^2 x. G1 ,evaluate the limit as x approaches #j3 "<sqrt"<#a "6 x "6 x‘5#b"> - sqrt"<#a - x "6 x‘5#b">">_/ "<sin"<x">">4 (b) Natural-language question EN Who is older, Annie Morton or Terry Richardson? G1 ,who is older1 ,annie ,morton or ,terry ,richardson8 G2 ,:o is old1 ,annie ,morton or ,try ,ri*>dson8 Fig. 1: Two examples to illustrate the challenges in Braille-based interaction. (a) How print mathematics will be conveyed into Grade 1 Braille as a longer sequence of cells and indicators. (b) The same question will be expressed in different sequences in terms of Grade 1 and Grade 2 based on contractions. Our contributions are summarized as follows: 1. We construct a benchmark with 5,570 instances for Braille comprehension, which is crucial to facilitate inclusive interaction between blind users and AI systems and is overlooked in the literature. Both Grade 1 and Grade 2 Braille and different types of tasks are included. 2. We build a toolkit Braille Toolkit based on liblouis and fixed UEB tables to control the Braille expression quality, which prevents evaluation bias and ensures reproducibility. 3. We conduct a comprehensive evaluation of six representative LLMs. The analysis covers a range of scenarios to reveal the implicit pattern behind different LLMs for Braille comprehension, which is useful for the development of future Braille AI systems. I Related Work I-A Challenges of Braille Comprehension in Language Models Digital Braille can be presented to language models as Unicode patterns, dot-number notation, or Braille ASCII. Unicode assigns dedicated code points to Braille cells [13], but rare symbols remain poorly matched to English-centric tokenizers [14, 15]; dot notation expands each cell into several digits and delimiters, while rendered Braille additionally requires a visual encoder. Braille ASCII instead represents each six-dot cell with one printable character and is used in Braille Ready Format files [16]. It can pass through standard text APIs without rendering and retains compact ASCII sequences, although familiar characters must be interpreted as Braille cells rather than print English. This compatibility neither implies one model token per cell nor makes ASCII a universal user interface; it only reduces rendering and interface barriers while preserving the Braille-decoding problem. We therefore use Braille ASCII as the primary interface and test Unicode and dots as alternative surface forms. Unified English Braille (UEB) distinguishes uncontracted Grade 1 from contracted Grade 2 [17]. Grade 1 preserves words letter by letter and is commonly taught first [5]; Grade 2 replaces frequent words and letter groups with shorter signs and is widely used by experienced readers and in longer texts [6]. For LLMs, Grade 1 is longer but keeps spelling explicit, whereas Grade 2 is shorter but requires lexical and positional disambiguation. Evaluating both grades therefore separates sequence-length difficulty from the contextual difficulty of contractions while covering both spelling-explicit and fluent-reading use. Prior accessible-interaction systems also position blind users as active authors rather than passive recipients: BrailleSketch enables gesture-based Braille text entry, while ADCanvas combines a screen-reader-accessible editor with a multimodal LLM agent for blind and low-vision creators [3, 18]. These systems motivate evaluating both access to user-authored input and accessible model output. I-B LLM-Based Braille Systems Existing language-model work on Braille primarily studies translation, disambiguation, and visual recognition. Low-resource Chinese Braille systems address data scarcity and structural ambiguity through Braille-specific pre-training, joint translation–segmentation, dedicated tokenization and generation modules, or curriculum learning over synthetic and tone-omitted data [19, 20, 21, 22, 23]. Prompted LLMs have been used to resolve pronunciation choices in Taiwanese Mandarin and Japanese Braille [24, 25], while context-aware generation with automated back-translation targets text-to-Braille fidelity [26]. For technical content, Gross et al. [27] evaluate LLMs on bidirectional conversion between MathML and Nemeth or UEB. General-purpose models nevertheless show inconsistent comprehension and generation of Unicode Braille [28], and multimodal models can propagate optical transcription errors into downstream answers [29]. BrailleLLM [30] uses instruction tuning for translation, formula conversion, and Braille question answering; Korean experiments [31] similarly find task-specific supervised models more reliable than unadapted LLMs. Together, these studies establish the difficulty of zero-shot Braille processing and the value of adaptation, but largely evaluate translation, conversion, or recognition. Our BrailleBench instead holds the task and target fixed while separately measuring text-only Braille reading, Braille response generation, and end-to-end interaction in Grade 1 and Grade 2, without assuming Braille-specific model adaptation. I BrailleBench Fig. 2: Overview of the BrailleBench construction and evaluation pipeline. Five source datasets are adapted to seven English–Braille interaction protocols, normalized by task type, and deterministically transcribed into UEB Grade 1 and Grade 2 Braille with liblouis; the aligned Braille ASCII records are then validated, supplemented with Unicode and dot-number representations, and evaluated on LLMs using direction-aware scoring. To evaluate whether LLMs can comprehend Braille in task-oriented interactions, we construct the BrailleBench benchmark. The construction pipeline contains four stages: (i) Source Task Selection and Braille Evaluation Protocol; (i) Data Sample Construction; (i) Quality Control Mechanism; and (iv) Benchmarking on LLMs, as shown in Figure 2. I-A Source Task Selection and Braille Evaluation Protocol The data instances in BrailleBench are inherited from five widely used datasets, including grade-school arithmetic, competition mathematics, commonsense reasoning, and multi-hop question answering tasks, which are shown in Table I. There are a total of 5,570 source instances, where we leverage the whole test set of GSM8K [32], AIME 2024 [33], and CommonsenseQA [34]. For HotpotQA [35] and 2WikiMultiHopQA [36], the first 1,500 examples in each of the development sets are used to bound the total cost. Dataset Task Answer Format GSM8K Grade-school arithmetic Integer AIME 2024 Competition mathematics Integer CommonsenseQA Commonsense reasoning Multi-choice HotpotQA Multi-hop QA Answer span 2WikiMultiHopQA Multi-hop QA Answer span TABLE I: Composition of BrailleBench. Each data sample is aligned with English, Grade 1 Braille, and Grade 2 Braille formats. After determining the source tasks, the next step is to define the evaluation protocol in a Braille environment. Suppose ENEN, G1G_1, and G2G_2 denote English, Grade 1 Braille, and Grade 2 Braille, respectively; the benchmark can contain different input–output settings. Specifically, Gi→ENG_i→EN tests whether the model can understand and solve a Braille-authored problem. Conversely, EN→GiEN→G_i tests whether the model can produce an answer decoded in Braille without first-hand Braille input. Besides, Gi→GiG_i→G_i evaluate the end-to-end Braille interaction. In all configurations, the source input, target answer, and task evaluation metrics are consistent with the original EN→ENEN→EN setting, which serves as the reference in terms of the capability available to each evaluated model. In addition, Grade 1 and Grade 2 Braille will result in different difficulty for LLMs to comprehend because of their express rule. In Braille ASCII, each printable ASCII character represents one six-dot Braille cell, the basic 2×32× 3 dot-pattern unit of Braille. Thus, Grade 1 Braille yields highly informative cell sequences that maintain closer local correspondence with standard print orthography; within Braille ASCII, this may preserve lexical surface cues observed during pre-training. Conversely, Grade 2 achieves greater cell-level density, though its contractions overload individual symbols with broader semantic meaning, rendering interpretation highly contingent on positional constraints and local context. Furthermore, fewer cells do not necessarily imply fewer model tokens because segmentation of symbol-heavy strings is model-dependent. Then, the next question is how to construct Braille data instances based on the source instances. I-B Braille Data Instance Construction To construct Braille data instances from the English-based ones is non-trivial, since it is not a field-wise character substitution as translation task. For example, mathematical tasks contain LaTeX-based expressions whose visual structure must be made explicit before transcription. Besides, in natural language-based QA datasets, a dropped relation, reordered choice, or unresolved markup fragment can change the original semantics, where UEB indicators and Grade 2 contractions can then amplify that error during transcription. Thus, a task-aware pipeline is required to separate semantic normalization from Braille encoding. Given a source instance x, the pipeline constructs aligned representations xEN,xG1,xG2\x^EN,x^G_1,x^G_2\ in three stages: task-aware normalization, deterministic UEB transcription, and structured record assembly, which is described in the following. The same transformation is applied to every task-related field, including questions, answer choices, reference answers, alternative gold answers, and available solutions. Note that no LLM-generated Braille is used as context or ground truth. I-B1 Task-Aware Normalization For mathematical data, the pipeline deterministically linearizes common and LaTeX-based expressions into explicit plain text before transcription. Fractions, roots, superscripts, subscripts, relations, Greek letters, and answer boxes are converted while preserving their scope. For example, 34 would become (3)/(4). Presentation-only commands are removed, whereas unsupported commands remain visible as auditable residuals rather than being silently discarded. For example, x ?= y remains visible in the normalized text, while is recorded as a residual command for inspection or strict-mode rejection. For natural language data in QA tasks, dataset-specific adapters would map heterogeneous records into a common schema while preserving the question, labeled choices, multiple gold-answer aliases, and solution fields. I-B2 Deterministic UEB Transcription We transcribe each normalized field with liblouis version 3.32.0 [37], using the UEB tables en-ueb-g1.ctb and en-ueb-g2.ctb. We use a fixed translation engine, version, and pair of UEB tables to make the transcription deterministic and reproducible while eliminating model-dependent variation. Each Braille field is generated from its aligned normalized English field, and the original source field is not overwritten. In this way, the instance- and field-level correspondence across the English, Grade 1, and Grade 2 representations. I-B3 Braille Toolkit We further release Braille Toolkit, a benchmark-independent, open-source framework that generalizes this construction methodology beyond BrailleBench. Rather than serving as a thin wrapper around liblouis, it provides reusable task-aware adapters for researchers’ own mathematical and QA records, performs auditable normalization, transcribes each field in UEB Grade 1 and Grade 2, and emits synchronized Braille ASCII, Unicode Braille, and dot-number representations. Every converted record retains its original and normalized text, applied rules, residual markup, fallback status, and generated variants. The toolkit also provides fail-closed environment checks, back-translation, cross-format cell validation, and a model-output validator that preserves legal Braille ASCII punctuation, distinguishes print numerals from UEB numerals, and handles edge symbols that resemble Markdown syntax. It therefore supports the construction and evaluation of new UEB datasets without depending on BrailleBench data or model infrastructure. Its mathematical output is linearized content transcribed in English UEB rather than Nemeth Code. I-C Quality Control and Additional Representations For BrailleBench, we apply the toolkit’s quality controls at the environment, representation, and record levels. We first verify that the specified liblouis tables are available and reproduce known translations. We then back-translate each normalized field with the grade-matched table as a diagnostic. Since UEB back-translation can be context-sensitive, particularly for isolated mathematical symbols and Grade 2 contractions, disagreement is flagged for inspection rather than automatically treated as evidence of faulty forward transcription. The retained source text, normalization trace, residual markup, and fallback status make each flagged record auditable. We additionally verify cell-level consistency across Braille ASCII, Unicode Braille patterns, and dot-number notation: all three serializations must resolve to the same ordered Braille cells. Human experts review the generated transcriptions for fidelity and consistency after these automated checks. Together, deterministic generation, cross-representation validation, back-translation, and expert review prevent representation artifacts from being mistaken for model failures. Whenever Braille appears in the seven core settings, it is represented as Braille ASCII. The release additionally includes the aligned Unicode and dot-number forms generated from the same cells, enabling controlled evaluation without rebuilding the source data. Section IV-D uses these forms in a surface-format ablation that changes only the serialization while holding the grade, task content, requested answer, and scorer fixed. These additional settings test whether a failure reflects UEB processing more broadly or sensitivity to a particular model-facing representation. I-D Benchmarking on LLMs After construction and validation, a task-specific prompt renderer instantiates each source item under the seven settings. In the core benchmark, task and output instructions remain in English. For Braille-input conditions, only the question and, where applicable, answer choices are replaced by their grade-matched Braille ASCII forms. For Braille-output conditions, the prompt requests an answer in the corresponding grade. Models receive no demonstrations, decoding keys, or in-context translation examples, so paired settings differ only in the task-bearing representation and requested output. Task Adaptation. The prompt preserves the response convention of each source task. GSM8K requests step-by-step problem solving and a final numerical answer in ; AIME additionally preserves the requirement that the answer be an integer from 0 to 999. CommonsenseQA presents all labeled choices and requests only the selected answer text, while HotpotQA and 2WikiMultiHopQA request a short phrase or name. QA answers follow the marker “The answer is:”. FULLBR-G1G_1 and FULLBR-G2G_2, which also transcribe the English instruction while retaining English output, are evaluated separately as a prompt ablation rather than added to the core matrix. Braille-Output Validation. Raw English normalization cannot be applied directly to Braille ASCII because characters such as #, >, and ; are meaningful Braille cells. We first map Unicode or dot-number output, when present, to the corresponding Braille ASCII cells. A format-aware gate then identifies clear print-English responses; mathematical outputs additionally distinguish literal print numerals from valid UEB numerals. Remaining outputs are back-translated with the grade-matched liblouis table and evaluated against the English reference using the original task metric. Empty responses, refusals, and completed but unparseable answers receive zero credit. All models are audited with this same finalized extraction and scoring pipeline. IV Experiments IV-A Experimental Setup We conduct a series of comprehensive experiments to address multiple research questions. The details of the experimental setup are depicted as follows. Evaluation Metric and Datasets. The final answer is extracted using task-aware markers, including the last expression for mathematics and the explicit QA answer marker. GSM8K and AIME 2024 are scored with Math-Verify [38], following the general mathematical evaluation protocol of OpenCompass [39]. CommonsenseQA (CommQA), HotpotQA, and 2WikiMultiHopQA (2Wiki) use Exact Match (EM) and token-level F1 under FlashRAG normalization [40], taking the maximum score over valid answer aliases. The main experiments with EM as the reported metric for QA might be too restricted in Braille scenarios, and we thus also provide F1 and sub-EM as evaluation metrics to analyze whether evaluation bias might exist. For each non-English condition, Avg. Drop is the unweighted mean of the model’s EN→ENEN→EN score minus its score in that condition across the five datasets. Models. We evaluate six representative language models, including proprietary ones (Claude Opus 4.8 and Claude Haiku 4.5) and open-weight ones (Llama 3.3 70B Instruct [41], Llama 3.1 8B Instruct [42], and Qwen3 1.7B/32B [43]). Inference Protocol. All evaluations are closed-book, i.e., the models will not receive any demonstrations, retrieved context, or Braille decoding key. The maximum generation length is 4,096 tokens for AIME 2024 and 1,024 tokens for the other datasets, following general settings. Prompt construction, answer extraction, task metrics, and Braille-output validation follow the benchmark protocol described in Section I-D. IV-B RQ1: Can LLMs Understand Braille? Model GSM8K AIME 2024 CommonsenseQA HotpotQA 2Wiki Avg. Drop G1 G2 G1 G2 G1 G2 G1 G2 G1 G2 G1 G2 E→E\!→\!E Reference Opus 4.8 + CoT 97.4 93.3 88.8 42.7 56.1 – Claude Opus 4.8 97.3 100.0 88.6 40.0 46.7 – Claude Haiku 4.5 96.2 53.3 81.7 26.4 20.9 – Llama 3.3 70B 95.1 23.3 80.4 29.7 29.7 – Llama 3.1 8B 83.9 6.7 43.4 21.4 25.0 – Qwen3 32B 94.7 20.0 83.5 21.4 28.3 – Qwen3 1.7B 69.4 10.0 37.4 12.7 18.3 – E→GE\!→\!G Claude Opus 4.8 91.7 ↓ 5.6 86.0 ↓ 11.3 43.3 ↓ 56.7 53.3 ↓ 46.7 55.0 ↓ 33.6 46.5 ↓ 42.1 28.5 ↓ 11.5 25.3 ↓ 14.7 40.0 ↓ 6.7 27.3 ↓ 19.4 22.8 26.8 Claude Haiku 4.5 5.0 ↓ 91.2 0.1 ↓ 96.1 0.0 ↓ 53.3 0.0 ↓ 53.3 75.3 ↓ 6.4 72.5 ↓ 9.2 11.3 ↓ 15.1 12.9 ↓ 13.5 8.9 ↓ 12.0 11.0 ↓ 9.9 35.6 36.4 Llama 3.3 70B 4.9 ↓ 90.2 0.0 ↓ 95.1 3.3 ↓ 20.0 0.0 ↓ 23.3 50.0 ↓ 30.4 29.7 ↓ 50.7 11.1 ↓ 18.6 6.2 ↓ 23.5 12.1 ↓ 17.6 3.8 ↓ 25.9 35.4 43.7 Llama 3.1 8B 2.6 ↓ 81.3 0.0 ↓ 83.9 0.0 ↓ 6.7 0.0 ↓ 6.7 17.2 ↓ 26.2 17.4 ↓ 26.0 1.0 ↓ 20.4 0.9 ↓ 20.5 0.4 ↓ 24.6 0.5 ↓ 24.5 31.8 32.3 Qwen3 32B 6.1 ↓ 88.6 2.0 ↓ 92.7 3.3 ↓ 16.7 3.3 ↓ 16.7 57.8 ↓ 25.7 43.0 ↓ 40.5 2.5 ↓ 18.9 2.7 ↓ 18.7 2.2 ↓ 26.1 0.9 ↓ 27.4 35.2 39.2 Qwen3 1.7B 2.5 ↓ 66.9 0.0 ↓ 69.4 0.0 ↓ 10.0 0.0 ↓ 10.0 22.2 ↓ 15.2 26.5 ↓ 10.9 0.1 ↓ 12.6 0.1 ↓ 12.6 0.0 ↓ 18.3 0.0 ↓ 18.3 24.6 24.2 G→EG\!→\!E Claude Opus 4.8 89.8 ↓ 7.5 67.9 ↓ 29.4 90.0 ↓ 10.0 76.7 ↓ 23.3 86.9 ↓ 1.7 34.6 ↓ 54.0 38.8 ↓ 1.2 12.0 ↓ 28.0 49.5 ↑ 2.9 6.9 ↓ 39.8 3.5 34.9 Claude Haiku 4.5 35.6 ↓ 60.6 11.7 ↓ 84.5 3.3 ↓ 50.0 3.3 ↓ 50.0 30.9 ↓ 50.8 24.0 ↓ 57.7 25.6 ↓ 0.8 10.3 ↓ 16.1 23.9 ↑ 3.0 7.2 ↓ 13.7 31.8 44.4 Llama 3.3 70B 7.3 ↓ 87.8 3.7 ↓ 91.4 0.0 ↓ 23.3 0.0 ↓ 23.3 8.9 ↓ 71.5 4.1 ↓ 76.3 18.6 ↓ 11.1 3.0 ↓ 26.7 17.3 ↓ 12.4 0.8 ↓ 28.9 41.2 49.3 Llama 3.1 8B 4.6 ↓ 79.3 1.4 ↓ 82.5 0.0 ↓ 6.7 0.0 ↓ 6.7 0.2 ↓ 43.2 0.5 ↓ 42.9 10.9 ↓ 10.5 0.7 ↓ 20.7 7.1 ↓ 17.9 0.5 ↓ 24.5 31.5 35.5 Qwen3 32B 13.0 ↓ 81.7 7.7 ↓ 87.0 0.0 ↓ 20.0 3.3 ↓ 16.7 57.1 ↓ 26.4 17.4 ↓ 66.1 14.2 ↓ 7.2 1.3 ↓ 20.1 20.3 ↓ 8.0 0.9 ↓ 27.4 28.7 43.5 Qwen3 1.7B 2.4 ↓ 67.0 1.4 ↓ 68.0 0.0 ↓ 10.0 0.0 ↓ 10.0 9.6 ↓ 27.8 1.5 ↓ 35.9 7.1 ↓ 5.6 0.7 ↓ 12.0 8.3 ↓ 10.0 0.5 ↓ 17.8 24.1 28.7 G→G\!→\!G Claude Opus 4.8 86.3 ↓ 11.0 63.5 ↓ 33.8 73.3 ↓ 26.7 50.0 ↓ 50.0 23.3 ↓ 65.3 14.4 ↓ 74.2 42.3 ↑ 2.3 9.4 ↓ 30.6 57.8 ↑ 11.1 6.5 ↓ 40.2 17.9 45.8 Claude Haiku 4.5 14.5 ↓ 81.7 7.1 ↓ 89.1 0.0 ↓ 53.3 0.0 ↓ 53.3 15.2 ↓ 66.5 8.6 ↓ 73.1 21.1 ↓ 5.3 4.9 ↓ 21.5 21.6 ↑ 0.7 4.9 ↓ 16.0 41.2 50.6 Llama 3.3 70B 1.8 ↓ 93.3 0.2 ↓ 94.9 0.0 ↓ 23.3 0.0 ↓ 23.3 10.6 ↓ 69.8 4.3 ↓ 76.1 17.8 ↓ 11.9 1.9 ↓ 27.8 19.7 ↓ 10.0 1.1 ↓ 28.6 41.7 50.1 Llama 3.1 8B 0.5 ↓ 83.4 0.0 ↓ 83.9 0.0 ↓ 6.7 0.0 ↓ 6.7 0.1 ↓ 43.3 2.5 ↓ 40.9 4.1 ↓ 17.3 0.1 ↓ 21.3 4.5 ↓ 20.5 0.0 ↓ 25.0 34.2 35.6 Qwen3 32B 3.3 ↓ 91.4 1.6 ↓ 93.1 0.0 ↓ 20.0 0.0 ↓ 20.0 9.3 ↓ 74.2 1.1 ↓ 82.4 10.9 ↓ 10.5 1.4 ↓ 20.0 20.3 ↓ 8.0 3.5 ↓ 24.8 40.8 48.1 Qwen3 1.7B 0.0 ↓ 69.4 0.3 ↓ 69.1 0.0 ↓ 10.0 0.0 ↓ 10.0 2.5 ↓ 34.9 0.2 ↓ 37.2 1.5 ↓ 11.2 0.1 ↓ 12.6 1.3 ↓ 17.0 0.0 ↓ 18.3 28.5 29.4 TABLE I: Main experimental results organized by input–output direction. All Δ annotations continue to use the corresponding model’s standard EN-EN score. Bold denotes the best task score within each directional setting and grade. Across Language Settings and Models. As shown in Table I, introducing Braille leads to a broad performance decline across models and datasets. The strongest English models generally retain more capability in Braille, with Opus showing the most consistent performance, while smaller models tend to lose most of their original task capability. However, the ordering is not strictly determined by model scale: several models exhibit isolated strengths on particular QA tasks but fail to transfer them across task families. Mathematics shows the clearest separation between models, whereas QA performance is more variable and includes a small number of localized improvements that we examine separately. The three interaction directions further reveal that Braille understanding is not a single capability. Models may partially recover a Braille-authored question under G→EG\!→\!E yet fail to express an already known answer under E→GE\!→\!G. Requiring both abilities under G→G\!→\!G generally compounds these failures and produces the least reliable interaction. Across all three directions, Grade 2 is more challenging than Grade 1, particularly when contracted Braille must be interpreted on the input side. Overall, the results show that strong English task performance does not automatically transfer to Braille reading, accessible expression, or end-to-end interaction. Table I shows complete prompts for representative GSM8K and CommonsenseQA instances. Variant Model-Facing Prompt Gold GSM8K EN-EN Solve the following math problem step by step. Answer in plain English. Put your final numerical answer within . Problem: Janet’s ducks lay 16 eggs per day. She eats three for breakfast every morning and bakes muffins for her friends every day with four. She sells the remainder at the farmers’ market daily for $2 per fresh duck egg. How much in dollars does she make every day at the farmers’ market? 18 G1-EN The following question is written in Grade 1 (uncontracted) Braille ASCII notation. Solve the following math problem step by step. Answer in plain English. Put your final numerical answer within . Problem: ,janet’s ducks lay #af eggs per day4 ,she eats three for breakfast every morning and bakes muffins for her friends every day with four4 ,she sells the remainder at the farmers’ market daily for ‘s#b per fresh duck egg4 ,how much in dollars does she make every day at the farmers’ market8 18 G2-EN The following question is written in Grade 2 (contracted) Braille ASCII notation. Solve the following math problem step by step. Answer in plain English. Put your final numerical answer within . Problem: ,janet’s ducks lay #af e7s p "d4 ,%e eats ?ree = br1kfa/ e morn+ & bakes mu69s = h frs e "d ) f|r4 ,%e sells ! rema9d at ! f>ms’ m>ket daily = ‘s#b p fre% duck egg4 ,h m* 9 doll>s does %e make e "d at ! f>ms’ m>ket8 18 FULLBR-G1 ,solve the following math problem step by step4 ,answer in plain ,english4 ,put your final numerical answer within _*boxed_<_>4 ,problem3 ,janet’s ducks lay #af eggs per day4 ,she eats three for breakfast every morning and bakes muffins for her friends every day with four4 ,she sells the remainder at the farmers’ market daily for ‘s#b per fresh duck egg4 ,how much in dollars does she make every day at the farmers’ market8 18 FULLBR-G2 ,solve ! foll+ ma? problem /ep by /ep4 ,answ 9 pla9 ,5gli%4 ,put yr f9al numical answ )9 _*box$_<_>4 ,problem3 ,janet’s ducks lay #af e7s p "d4 ,%e eats ?ree = br1kfa/ e morn+ & bakes mu69s = h frs e "d ) f|r4 ,%e sells ! rema9d at ! f>ms’ m>ket daily = ‘s#b p fre% duck egg4 ,h m* 9 doll>s does %e make e "d at ! f>ms’ m>ket8 18 CommonsenseQA EN-EN Answer the following multiple-choice question. Answer in plain English. Write only the answer text after ’The answer is: ’. Question: A revolving door is convenient for two direction travel, but it also serves as a security measure at a what? Choices: A) bank B) library C) department store D) mall E) new york bank G1-EN The following question is written in Grade 1 (uncontracted) Braille ASCII notation. Answer the following multiple-choice question. Answer in plain English. Write only the answer text after ’The answer is: ’. Question: ,a revolving door is convenient for two direction travel1 but it also serves as a security measure at a what8 Choices: ,a"> bank ,b"> library ,c"> department store ,d"> mall ,e"> new york bank G2-EN The following question is written in Grade 2 (contracted) Braille ASCII notation. Answer the following multiple-choice question. Answer in plain English. Write only the answer text after ’The answer is: ’. Question: ,a revolv+ door is 3v5i5t = two direc;n travel1 b x al sves z a secur;y m1sure at a :at8 Choices: ,a"> bank ;,b"> libr>y ;,c"> de"p;t /ore ;,d"> mall ;,e"> new york bank FULLBR-G1 ,answer the following multiple-choice question4 ,answer in plain ,english4 ,write only the answer text after ’,the answer is3 ’4 ,question3 ,a revolving door is convenient for two direction travel1 but it also serves as a security measure at a what8 ,choices3 ,a"> bank ,b"> library ,c"> department store ,d"> mall ,e"> new york bank FULLBR-G2 ,answ ! foll+ multiple-*oice "q4 ,answ 9 pla9 ,5gli%4 ,write only ! answ text af ’,! answ is3 ’4 ,"q3 ,a revolv+ door is 3v5i5t = two direc;n travel1 b x al sves z a secur;y m1sure at a :at8 ,*oices3 ,a"> bank ;,b"> libr>y ;,c"> de"p;t /ore ;,d"> mall ;,e"> new york bank TABLE I: Representative GSM8K and CommonsenseQA prompt variants. FULLBR-G1 and FULLBR-G2 translate both the instruction and task content into Braille, retain English as the requested answer language, and are evaluated separately from the seven core configurations. Line breaks are collapsed for compact presentation. Performance in Grade 1 versus Grade 2. Across models and datasets, Grade 2 is consistently more difficult than Grade 1 when Braille must be read from the input. This difference is also reflected in end-to-end interaction, where the loss introduced by Grade 2 compounds with the requirement to generate a Braille answer. By contrast, the grade difference is smaller and less consistent when models only need to produce Braille from an English question. The gap is most visible on language-intensive QA tasks, while some mathematical settings exhibit a floor effect because several models already perform near zero under Grade 1. Overall, the results localize the main Grade 2 difficulty to accessing contracted input rather than simply producing a shorter Braille output. This difficulty cannot be explained by sequence length alone. Grade 2 uses fewer cells, but its contractions assign more lexical information to individual cells and make their interpretation dependent on position and context. As shown in Tables IV and Table V, models that attempt to process Grade 2 often produce longer decoding traces, incomplete answers, or empty responses despite receiving a shorter Braille sequence. Grade 2 therefore trades surface length for contextual ambiguity: local contraction errors can alter words, entities, and relations before task completion begins, and these errors become increasingly consequential as the input grows longer or requires multi-step understanding. Dataset EN G1 G2 G2→ Empty G2→ 2 Empty GSM8K 253 978 1,518 12.1 10.4 AIME 2024 1,289 2,214 2,555 20.0 23.3 CommonsenseQA 25 29 519 54.7 56.9 HotpotQA 88 516 910 64.9 72.2 2Wiki 163 588 940 71.6 78.4 TABLE IV: The average response length in non-empty examples for Opus under EN-EN, G1-EN, and G2-EN language settings. The right two columns report the percentage of empty output examples in G2-EN and G2-G2. Model GSM8K AIME CSQA Hotpot 2Wiki Claude Opus 4.8 6.0 2.0 20.5 10.4 5.8 Claude Haiku 4.5 2.2 1.0 17.8 4.3 3.1 Llama 3.3 70B 2.3 0.7 0.7 1.1 1.1 Llama 3.1 8B 1.7 1.2 3.8 4.0 4.2 Qwen3 32B 1.5 1.1 0.8 2.2 1.6 TABLE V: Ratio of average non-empty G2-EN response length to the same model’s EN-EN response length. Values below one indicate that the model produced less text under Braille input rather than externalizing a decode. Counterintuitive Results in Multi-hop QA Tasks. Intuitively, the English-English reference should be the upper bound for the results under Braille conditions. However, as shown in Table I, the Claude models exhibit performance improvement from Braille (Grade 1) to English and to Braille settings on two multi-hop QA datasets. We investigate the potential reasons, which are mainly attributed to two aspects. On one hand, as shown in Table VI, the Opus model tend response the question directly rather than conduct more steps of reasoning in the EN-EN setting, which is more “aggressive” compared to the setting with Grade 1 Braille. Thus, the model might not be able to produce a correct answer at once on this more challenging multi-hop QA dataset, while the model becomes more careful when involving Braille and thus has the higher chance to arrive at the correct answer, although with longer reasoning chains. On the other hand, Table VI compares strict EM with Sub-EM and F1 among different languages and CoT settings. We can see the potential bias in terms of the evaluation metrics. The manually activated CoT can further improve the results across all metrics, which confirms our previous conjecture. In addition, A more relaxed evaluation metric, i.e., Sub-EM and F1, can better reflect the performance gap between the intuitive upper bound and the results in Braille. Configuration Reasoning Activation Reasoning Deactivation Average Length EN-EN + CoT 99% 1% 278 EN-EN 57% 43% 163 G1-EN 97% 3% 588 G1-G1 99% 1% 790 Configuration EM Sub-EM F1 EN-EN + CoT 56.1 64.0 65.5 EN-EN 46.7 49.9 52.3 G1-EN 49.5 58.7 59.5 G1-G1 57.8 60.1 64.1 TABLE VI: Response activation and length of Opus on 2WikiMultiHopQA. IV-C RQ2: Can LLMs Understand a Pure Braille Request? In the Braille-input settings, the user question is written in Braille, while the task instruction remains in English to help the model identify the task and expected answer format, which might provide biased assistance. To test whether LLMs can understand a fully Braille input without any English instruction, we translate both the instruction and the problem into Grade 1 (FULLBR-G1) or Grade 2 (FULLBR-G2) Braille, respectively. Model GSM8K AIME CommQA HotpotQA 2Wiki G1-EN / FULLBR-G1 Opus 4.8 89 / 73 90 / 90 86 / 87 43 / 40 53 / 46 Haiku 4.5 34 / 5 3.3 / 6.7 28 / 14 36 / 21 34 / 19 Llama 3.3 70B 9 / 5 0 / 0 11 / 20 18 / 21 22 / 14 Llama 3.1 8B 6 / 7 0 / 0 0 / 0 11 / 12 13 / 20 Qwen3 32B 14 / 4 0 / 0 54 / 27 11 / 16 23 / 30 G2-EN / FULLBR-G2 Claude Opus 4.8 63 / 44 76.7 / 0 32 / 8 13 / 2 10 / 3 Claude Haiku 4.5 9 / 3 3.3 / 3.3 20 / 4 10 / 3 11 / 1 Llama 3.3 70B 3 / 2 0 / 0 4 / 6 2 / 3 0 / 1 Llama 3.1 8B 1 / 1 0 / 0 0 / 1 0 / 1 0 / 0 Qwen3 32B 7 / 5 3.3 / 6.7 19 / 4 0 / 0 1 / 0 TABLE VII: Effect of transcribing the task instruction into Braille across all five diagnostic subsets. Each cell reports G1-EN / FULLBR-G1 or G2-EN / FULLBR-G2 on the same 100-example (30-example on AIME 2024 prefix. Table VII show that transcribing both the instruction and the problem generally lowers performance, and the effect depends strongly on grade and model. Under FULLBR-G1, Opus retains substantial accuracy across all five tasks, while the other models decline from their mixed-prompt results or remain the same; occasional increases occur mainly at low absolute scores. Under FULLBR-G2, performance becomes uniformly weak: even Opus preserves meaningful accuracy only on GSM8K, and the remaining model–task combinations are close to the floor. These results show that the English instruction helps most models understand what task they should perform. Opus can still answer many FULLBR-G1 problems, showing that it can understand some requests written entirely in Grade 1 Braille, but FULLBR-G2 is difficult for every model. We next ask whether the same Braille content becomes easier or harder when it is represented in different digital formats. IV-D RQ3: How Expression Form of Braille Affect the Results? To separate knowledge of Braille rules from sensitivity to serialization, we render the same cells as Braille ASCII, Unicode patterns, and dot numbers while holding the grade, task instruction, requested English answer, and scorer fixed. Model GSM8K AIME CommQA HotpotQA 2Wiki Grade 1: ASCII / Unicode / Dots Opus 4.8 89/0/0 90/0/0 86/0/0 43/0/0 53/0/0 Haiku 4.5 34/48/1 3.3/10/0 28/19/0 36/24/0 34/29/0 Llama 3.3 70B 9/4/0 0/0/0 11/2/0 18/2/0 22/0/0 Llama 3.1 8B 6/0/1 0/0/0 0/0/0 11/0/0 13/0/0 Qwen3 32B 14/19/0 0/6.7/0 54/23/0 11/5/0 23/6/0 Grade 2: ASCII / Unicode / Dots Opus 4.8 63/0/0 76.7/0/0 32/0/0 13/0/0 10/0/0 Haiku 4.5 9/30/5 3.3/3.3/0 20/9/0 10/11/0 11/11/0 Llama 3.3 70B 3/0/2 0/0/0 4/1/0 2/0/0 0/0/0 Llama 3.1 8B 1/0/0 0/0/0 0/0/0 0/0/0 0/0/0 Qwen3 32B 7/9/0 3.3/3.3/0 19/11/0 0/0/0 1/1/1 TABLE VIII: Surface-format comparison on the diagnostic subsets. Each cell reports Braille ASCII / Unicode Braille / dot-number notation for the indicated grade. Table VIII shows that identical Braille cells produce markedly different results across surface representations. Braille ASCII gives the strongest and most stable performance across models, grades, and tasks. Unicode improves selected mathematics results for Haiku and Qwen3 32B, but usually underperforms ASCII elsewhere, while Opus scores zero on every evaluated task in this format. Dot notation is almost uniformly at the floor for both grades. These results show that a digital Braille representation is not a neutral rendering choice for current LLMs: performance depends on the serialized symbols through which the cells are presented. Unicode can be useful for particular model–task pairs, but its gains do not generalize, while expanding cells into dot-number sequences makes the input consistently harder rather than more explicit. Since choosing a more favorable surface form does not remove the broader understanding gap, we next test whether directly supplying the Braille mappings can repair it. i.e., improving the Braille context understanding. IV-E RQ4: Can an Explicit Braille Reference Help Context Understanding? To understand whether missing Braille mappings from difficulty applying them over a sequence, we prepend a liblouis-verified, grade-matched reference of letters, indicators, punctuation, and common contractions, without worked examples or answers. The Braille problem, English instruction and output, evaluation subset, and scorer remain fixed. Model GSM8K AIME CommQA HotpotQA 2Wiki Grade 1: no reference / with reference Opus 4.8 89.0/97.0 90.0/90.0 86.0/82.0 43.0/45.0 53.0/51.0 Haiku 4.5 34.0/73.0 3.3/23.3 28.0/37.0 36.0/39.0 34.0/37.0 Llama 3.3 70B 9.0/46.0 0.0/6.7 11.0/12.0 18.0/19.0 22.0/20.0 Llama 3.1 8B 6.0/11.0 0.0/0.0 0.0/1.0 11.0/8.0 13.0/12.0 Qwen3 32B 14.0/47.0 0.0/6.7 54.0/37.0 11.0/13.0 23.0/24.0 Grade 2: no reference / with reference Opus 4.8 63.0/30.0 76.7/6.7 32.0/2.0 13.0/0.0 10.0/1.0 Haiku 4.5 9.0/47.0 3.3/13.3 20.0/14.0 10.0/8.0 11.0/17.0 Llama 3.3 70B 3.0/8.0 0.0/3.3 4.0/11.0 2.0/0.0 0.0/0.0 Llama 3.1 8B 1.0/1.0 0.0/0.0 0.0/3.0 0.0/0.0 0.0/0.0 Qwen3 32B 7.0/13.0 3.3/0.0 19.0/6.0 0.0/0.0 1.0/0.0 TABLE IX: Ablation on the effectiveness of completed Braille-reference. Each cell compares the same subset without/with the supplied mapping. The results are shown in Table IX, where the explicit reference is most effective for Grade 1 mathematics. It substantially improves GSM8K for Haiku, Llama 3.3 70B, and Qwen3 32B, and also raises AIME accuracy for several of these models. The gains are smaller and inconsistent on the QA datasets, where some model–task pairs improve while others decline. Grade 2 is less effective: the reference helps selected weaker-model mathematics results, but offers no consistent benefit across QA and sharply reduces Opus performance. The contrast suggests that incomplete knowledge of local symbols is one source of Grade 1 failure, especially when recovering numbers and short expressions is sufficient to expose the mathematical problem. A static reference is less useful for longer linguistic inputs and Grade 2 contractions, where the correct expansion depends on lexical position and surrounding context rather than on a one-to-one lookup. Because access to the mapping does not guarantee successful sequence-level decoding, we next isolate where performance begins to break down across characters, numbers, words, and sentences. IV-F RQ5: Where Does Braille Understanding Break Down? To localize failures before downstream task completion, we remove the task and request direct Braille-to-English transcription of characters, numbers, Grade 1 words, contraction-bearing Grade 2 words, and sentences, yielding 446 items per model across two grades. Table X reports normalized exact match at all five scales and punctuation-insensitive sentence EM as an additional diagnostic. Model Characters Numbers G1 G2 Sentences Grade 1 Opus 4.8 97 97 100 98 40/88 Haiku 4.5 90 33 88 100 34/70 Llama 3.3 70B 92 13 100 98 8/56 Llama 3.1 8B 70 10 75 88 18/46 Qwen3 32B 68 37 100 100 30/72 Grade 2 Opus 4.8 92 100 90 72 2/28 Haiku 4.5 18 33 58 35 0/10 Llama 3.3 70B 35 10 70 23 0/4 Llama 3.1 8B 82 10 52 9 0/4 Qwen3 32B 13 23 58 16 0/4 TABLE X: Complete direct-decoding probe results. All cells report strict EM; sentence cells additionally report punctuation-insensitive EM after the slash. Grade 1 and Grade 2 denote the prompt encoding, while the two word columns denote the type of lexical item being transcribed. Table X shows a clear gap between recognizing local Grade 1 units and decoding complete sequences. Most models achieve high exact match on Grade 1 characters and words, yet all except Opus perform poorly on numbers, indicating difficulty maintaining the number indicator’s effect across subsequent cells. Sentence accuracy is much lower than word accuracy, although removing punctuation recovers a substantial portion of Grade 1 errors. The Grade 2 condition is harder at the lexical and sentence scales: contraction-bearing word accuracy falls, and strict sentence EM is zero for every model except Opus, with only limited recovery when punctuation is ignored. The scale-wise decline localizes the principal failure before downstream task completion. Current models often possess fragments of Braille knowledge, but number-state tracking, contextual contraction resolution, and the accumulation of local errors prevent that knowledge from supporting reliable sentence-level understanding. These diagnostic results clarify that downstream task failures can arise from sequence-level Braille decoding errors, rather than from insufficient task capability alone. V Conclusion and Future Work In this paper, we investigate whether LLM capabilities measured in common print English remain accessible through Braille. To this end, we propose BrailleBench, which contains English, UEB Grade 1, and UEB Grade 2 in different configurations to evaluate Braille comprehension from different aspects. We use a self-designed toolkit to produce Braille to ensure quality and avoid evaluation bias. Experiments on six representative LLMs show that English capability does not transfer reliably to Braille. Overall, some of the existing proprietary models might have fragments of Braille knowledge but lack robust Braille interaction capability. All experimental observations provide valuable guidance for the development of future Braille AI systems. In existing systems, the Braille-output validator is deliberately conservative but cannot perfectly distinguish lowercase print English from Grade 1 Braille ASCII when the byte strings coincide. Thus, they interpret its English-output rate as a lower bound and rely more heavily on numbers and contraction-bearing outputs to establish deliberate Braille encoding. Besides, API-hosted models expose neither tokenizer traces nor moderation decisions. Response length and empty-output patterns reveal observable costs and failures, but their internal causes remain hypotheses. To develop truly inclusive AI systems that can also benefit vulnerable groups, future work should investigate how to improve the adaptation for Braille tokenization, reasoning activation, and dealing with truncated outputs. Besides, specific evaluation in Braille scenarios should be specialized, e.g., error recovery, user-centered usability, and safety-aware mechanisms must be quantified. References [1] F. Mo, Z. Su, Y. Hui, J. Zhang, J. A. Sun, Z. Liu, C. Zhang, T. Sakai, and J. Nie (2026) Opendecoder: open large language model decoding to incorporate document quality in rag. In Proceedings of the ACM Web Conference 2026, p. 2252–2262. Cited by: §I. [2] S. A. Ali (2023) Artificial intelligence techniques to understand braille: a language for visually impaired individuals. In Handbook of research on artificial intelligence applications in literary works and social media, p. 254–276. Cited by: §I. [3] M. Li, M. Fan, and K. N. Truong (2017) BrailleSketch: a gesture-based text input method for people with visual impairments. In Proceedings of the 19th International ACM SIGACCESS Conference on Computers and Accessibility, p. 12–21. Cited by: §I, §I-A. [4] J. Zhang, X. Wang, Y. Jin, C. Chen, X. Zhang, and K. Liu (2024) Prototypical reward network for data-efficient rlhf. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 13871–13884. Cited by: §I. [5] International Council on English Braille (2024) The Rules of Unified English Braille. Third edition. Note: https://iceb.org/wp-content/uploads/2025/10/Rules-of-Unified-English-Braille-2024.pdfSection 2.5, Grades of Braille Cited by: §I, §I-A. [6] Royal National Institute of Blind People Contracted (Grade 2) Braille Explained. Note: https://w.rnib.org.uk/living-with-sight-loss/education-and-learning/braille-tactile-codes/contracted-grade-2-braille-explained/Education and learning guidance Cited by: §I, §I-A. [7] Z. Wang, J. Zhang, X. Zhang, K. Liu, P. Wang, and Y. Zhou (2025) Diversity-oriented data augmentation with large language models. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 22265–22283. Cited by: §I. [8] J. Zhang, F. Mo, T. C. Weerasooriya, R. Dai, X. Han, Y. Fu, D. Wang, and K. Liu (2026) StaRPO: stability-augmented reinforcement policy optimization. arXiv preprint arXiv:2604.08905. Cited by: §I. [9] A. Mohamed, Y. Zhang, M. Vazirgiannis, and G. Shang (2026) Lost in the mix: evaluating llm understanding of code-switched text. In Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 44922–44938. Cited by: §I. [10] J. Zhang, X. Wang, W. Ren, L. Jiang, D. Wang, and K. Liu (2025) Ratt: a thought structure for coherent and correct llm reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, p. 26733–26741. Cited by: §I. [11] J. Zhang, X. Wang, F. Mo, Y. Zhou, W. Gao, and K. Liu (2025) Entropy-based exploration conduction for multi-step reasoning. In Findings of the Association for Computational Linguistics: ACL 2025, p. 3895–3906. Cited by: §I. [12] J. Zhang, F. Mo, T. C. Weerasooriya, X. Ye, D. Wang, Y. Fu, and K. Liu (2026) Blind spot navigation in large language model reasoning with thought space explorer. In Findings of the Association for Computational Linguistics: EACL 2026, p. 3691–3707. Cited by: §I. [13] The Unicode Consortium The unicode standard, chapter 21: symbols, braille. Note: https://w.unicode.org/versions/Unicode17.0.0/core-spec/chapter-21/Unicode Standard Version 17.0 Cited by: §I-A. [14] M. Velayuthan and K. Sarveswaran (2025) Egalitarian language representation in language models: it all begins with tokenizers. In Proceedings of the 31st International Conference on Computational Linguistics, p. 5987–5996. Cited by: §I-A. [15] L. Ma, R. Zhang, Y. Han, S. Yu, Z. Wang, Z. Ning, J. Zhang, P. Xu, P. Li, W. Ju, et al. (2023) A comprehensive survey on vector database: storage and retrieval technique, challenge. arXiv preprint arXiv:2310.11703. Cited by: §I-A. [16] Library of Congress Braille ready format (BRF), family. Note: https://w.loc.gov/preservation/digital/formats/fdd/fdd000551.shtmlSustainability of Digital Formats Cited by: §I-A. [17] International Council on English Braille Unified English Braille (UEB). Note: https://iceb.org/publications/ueb/Overview and official publications Cited by: §I-A. [18] F. M. Li, M. X. Liu, C. L. Bennett, and S. K. Kane (2026) ADCanvas: accessible and conversational audio description authoring for blind and low vision creators. In Proceedings of the 2026 CHI Conference on Human Factors in Computing Systems, p. 1–21. Cited by: §I-A. [19] H. Yu, W. Su, L. Liu, J. Zhang, C. Cai, and C. Xu (2023) Pre-training model for low-resource chinese–braille translation. Displays 79, p. 102506. Cited by: §I-B. [20] T. Huang, W. Su, L. Liu, C. Cai, H. Yu, and Y. Yuan (2024) Improving braille–chinese translation with jointly trained and pre-trained language models. Displays 82, p. 102660. Cited by: §I-B. [21] D. Chen, C. Li, Z. Zhuang, L. Wei, and J. Liu (2026) An end-to-end chinese-braille translation method based on mt5: vocabulary expansion and structural enhancement. Displays, p. 103427. Cited by: §I-B. [22] A. Wu, Y. Yuan, Z. Xiao, and M. Zhang (2024) Vision-braille: a curriculum learning toolkit and braille-chinese corpus for braille translation. arXiv preprint arXiv:2407.06048. Cited by: §I-B. [23] X. Zhang, J. Zhang, F. Mo, D. Wang, Y. Fu, and K. Liu (2025) Leka: llm-enhanced knowledge augmentation. arXiv preprint arXiv:2501.17802. Cited by: §I-B. [24] D. Watty, M. Kitsunai, and S. Hsieh (2023) Prompt-based translation of chinese into taiwanese mandarin braille. In 2023 International Conference on Asian Language Processing (IALP), p. 56–61. Cited by: §I-B. [25] M. Kitsunai, D. Watty, and S. Hsieh (2024) Exploring the potential of prompt-based method for kanji-kana conversion in japanese braille translation. In Proceedings of the 30th Annual Meeting of the Association for Natural Language Processing, p. 2353–2357. External Links: Link Cited by: §I-B. [26] J. An, D. Kim, and K. Kang (2026) LLM-based system for enhanced text-to-braille translation: incorporating contextual awareness and automated verification. In Proceedings of the 41st ACM/SIGAPP Symposium on Applied Computing, p. 823–830. Cited by: §I-B. [27] M. Groß, D. Soiffer, and N. Soiffer (2026) The current state of ai for conversion of mathematics to and from braille. In International Conference on Computers Helping People with Special Needs, p. 146–155. Cited by: §I-B. [28] J. S. Daniel and A. Pal (2024) Impact of non-standard unicode characters on security and comprehension in large language models. arXiv preprint arXiv:2405.14490. Cited by: §I-B. [29] A. Karamolegkou, M. Nikandrou, G. Pantazopoulos, D. S. Villegas, P. Rust, R. Dhar, D. Hershcovich, and A. Søgaard (2025) Evaluating multimodal language models as visual assistants for visually impaired users. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), p. 25949–25982. Cited by: §I-B. [30] T. Huang, Z. Zhu, H. Xing, Z. Shao, Z. Yu, C. Yang, J. He, X. Liu, and J. Bu (2025) BrailleLLM: braille instruction tuning with large language models for braille domain tasks. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, p. 28589–28600. Cited by: §I-B. [31] A. Abdullah (2026) I’m sorry, but i can’t help with braille: revealing accessibility failures in state-of-the-art llms. arXiv preprint arXiv:2607.11893. Cited by: §I-B. [32] K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. (2021) Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. Cited by: §I-A. [33] Math-AI (2024) AIME 2024. Note: https://huggingface.co/datasets/math-ai/aime24Accessed: 2025-10-06 Cited by: §I-A. [34] A. Talmor, J. Herzig, N. Lourie, and J. Berant (2019) Commonsenseqa: a question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), p. 4149–4158. Cited by: §I-A. [35] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, R. Salakhutdinov, and C. D. Manning (2018) HotpotQA: a dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, p. 2369–2380. Cited by: §I-A. [36] X. Ho, A. D. Nguyen, S. Sugawara, and A. Aizawa (2020) Constructing a multi-hop qa dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, p. 6609–6625. Cited by: §I-A. [37] C. Egli, B. Frees, L. de Ruijter, and A. Yakuboy (2002) Liblouis: open-source braille translator and back-translator. Cited by: §I-B2. [38] Hugging Face (2025) Math-verify: a robust mathematical expression evaluation system. Note: https://github.com/huggingface/Math-VerifyVersion 0.9.0 Cited by: §IV-A. [39] M. Cao, K. Chen, H. Duan, Y. Fang, Z. Fei, T. Gao, G. Jiaye, M. Li, H. Liu, J. Liu, et al. (2026) OpenCompass: a universal evaluation platform for large language models. arXiv preprint arXiv:2605.19276. Cited by: §IV-A. [40] J. Jin, Y. Zhu, Z. Dou, G. Dong, X. Yang, C. Zhang, T. Zhao, Z. Yang, and J. Wen (2025) Flashrag: a modular toolkit for efficient retrieval-augmented generation research. In Companion Proceedings of the ACM on Web Conference 2025, p. 737–740. Cited by: §IV-A. [41] Meta AI (2024) Llama 3.3 70b instruct model card. Note: https://huggingface.co/meta-llama/Llama-3.3-70B-Instruct Cited by: §IV-A. [42] A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, et al. (2024) The llama 3 herd of models. arXiv preprint arXiv:2407.21783. Cited by: §IV-A. [43] Q. Team et al. (2025) Qwen3 technical report. arXiv preprint arXiv:2505.09388 6 (7), p. 13. Cited by: §IV-A.