Paper deep dive
LogiScope-VQA: Benchmarking Vision-Language Models for Logistics Hazard Identification in Industrial Scenarios
Hanjing Zhou, Mingze Yin, Ying Lian, Jun Ma, Chang-Yu Hsieh, Yanbing Zhou
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%
Last extracted: 9/10/2026, 3:21:17 AM
Summary
The paper introduces LogiScope-VQA, a benchmark designed to evaluate Large Multimodal Models (LMMs) on logistics hazard identification in industrial warehouse settings. The dataset comprises 2,476 images, 2,918 videos, and 10,274 VQA pairs sourced from real-world logistics parks, covering 18 core objects and 20 risk types. The benchmark assesses three dimensions: industrial element perception, warehouse knowledge understanding, and potential risk reasoning. Experiments show that even top proprietary models (GPT-5.5, Gemini-3.1-Pro, Claude-Opus-4.7) and open-source models (Qwen3.5-Plus) significantly lag behind human expert performance, highlighting challenges in fine-grained perception and risk reasoning, as well as pervasive security biases in LMMs.
Entities (11)
Relation Signals (10)
LogiScope-VQA â containsdatafrom â Cainiao
confidence 95% ¡ 3.5 million surveillance clips collected from the global intelligent logistics company Cainiao
LogiScope-VQA â coverstask â Industrial Element Perception
confidence 95% ¡ three principal themes: industrial element perception, warehouse knowledge understanding, and potential risk reasoning.
LogiScope-VQA â coverstask â Warehouse Knowledge Understanding
confidence 95% ¡ three principal themes: industrial element perception, warehouse knowledge understanding, and potential risk reasoning.
LogiScope-VQA â coverstask â Potential Risk Reasoning
confidence 95% ¡ three principal themes: industrial element perception, warehouse knowledge understanding, and potential risk reasoning.
LogiScope-VQA â evaluates â Qwen3.5-Plus
confidence 90% ¡ As exemplified by Qwen3.5-Plus, open-source LMMs closely track proprietary performance
LogiScope-VQA â evaluates â GPT-5.5
confidence 90% ¡ Extensive experiments unveil that even powerful proprietary models, including GPT-5.5... exhibit a significant gap
LogiScope-VQA â evaluates â Gemini 3.1 Pro
confidence 90% ¡ Extensive experiments unveil that even powerful proprietary models, including... Gemini-3.1-Pro... exhibit a significant gap
â â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Large Multimodal Models (LMMs) large-scale deployment in industrial warehouse settings specifically necessitates that models exhibit human-expert-level hazard-oriented perception, understanding, and reasoning capabilities. However, the scarcity of real industrial data, tightly coupled to commercial terms, significantly hampers further advancement. To bridge this gap, we curate LogiScope-VQA to investigate the practical applicability of mainstream LMMs in real-world logistics operations. LogiScope-VQA comprises 2,476 images and 2,918 videos primarily sourced from real-world logistics parks, along with 10,274 VQAs meticulously curated and validated by human annotators. Grounded in 18 core objects and 20 risk types, we devise 39 subtasks aligned with three principal themes: industrial element perception, warehouse knowledge understanding, and potential risk reasoning. Furthermore, we incorporate dynamic thinking-budget configurations and dual-dimensional risk bias analyses to elucidate the properties of LMMs. Extensive experiments unveil that even powerful proprietary models, including GPT-5.5, Gemini-3.1-Pro, and Claude-Opus-4.7, exhibit a significant gap relative to human performance. The unique challenge of jointly integrating perception, understanding, and reasoning for hazard identification poses substantial headroom for further improvement on LogiScope-VQA. We additionally reveal the pervasive security bias issue that impedes LLMs' practical deployment in real-world settings. The industrial dataset is publicly available under the CC BY-NC-SA 4.0 license.
Tags
Links
- Source: https://arxiv.org/abs/2609.09790v1
- Canonical: https://arxiv.org/abs/2609.09790v1
Trouble viewing inline? Open PDF directly â
Full Text
82,149 characters extracted from source content.
Expand or collapse full text
LogiScope-VQA: Benchmarking Vision-Language Models for Logistics Hazard Identification in Industrial Scenarios Hanjing Zhou 1,* , Mingze Yin 2,* , Ying Lian 1 , Jun Ma 1 , Chang-Yu Hsieh 2,â and Yanbing Zhou 1,â 1 Cainiao Group, Alibaba Group, 2 Zhejiang University, * Equal contributions. E-mail: zhj85393, mzyin256@gmail.com, â Corresponding authors. E-mail: kimhsieh@zju.edu.cn, tonychou.zyb@cainiao.com Large Multimodal Models (LMMs) large-scale deployment in industrial warehouse settings specifically necessitates that models exhibit human-expert-level hazard-oriented perception, understanding, and reasoning capabilities. However, the scarcity of real industrial data, tightly coupled to commercial terms, significantly hampers further advancement. To bridge this gap, we curate LogiScope-VQA to investigate the practical applicability of mainstream LMMs in real-world logistics operations. LogiScope- VQA comprises 2,476 images and 2,918 videos primarily sourced from real-world logistics parks, along with 10,274 VQAs meticulously curated and validated by human annotators. Grounded in 18 core objects and 20 risk types, we devise 39 subtasks aligned with three principal themes: industrial element perception, warehouse knowledge understanding, and potential risk reasoning. Furthermore, we incor- porate dynamic thinking-budget configurations and dual-dimensional risk bias analyses to elucidate the properties of LMMs. Extensive experiments unveil that even powerful proprietary models, including GPT-5.5, Gemini-3.1-Pro, and Claude-Opus-4.7, exhibit a significant gap relative to human performance. The unique challenge of jointly integrating perception, understanding, and reasoning for hazard identifi- cation poses substantial headroom for further improvement on LogiScope-VQA. We additionally reveal the pervasive security bias issue that impedes LLMsâ practical deployment in real-world settings. The industrial dataset is publicly available under the C BY-NC-SA 4.0 license. Keywords: Industrial Benchmark, Logistics Hazard Identification, Visual Question Answering 1. Introduction Large Multimodal Models (LMMs) have achieved remarkable success on a broad spectrum of question- answering tasks in naturalistic settings, showcasing strong perceptual and reasoning abilities as well as extensive embedded knowledge (Li et al., 2025; Liu et al., 2026; Zhou et al., 2025a). Nevertheless, the limited evaluation of LMMs in industrial scenarios has hindered the conversion of this potential into practically deployable systems and tools (Dou et al., 2026; Yue et al., 2024). Industrial environments pose unique challenges that are largely underexplored in current benchmarks, particularly due to safety-oriented task requirements and the relatively dense spatial distribution of targets. Logistics warehousing is a representative industrial scenario in which continuous monitoring and timely safety risk recognition are crucial for accident prevention, regulatory compliance, and efficient warehouse operations. Previous studies have begun to develop multimodal benchmarks for industrial logistics, but their applicability to real-world assessment remains constrained by two inherent obstacles: (i) In terms of evaluation depth, industrial logistics environments naturally raise concerns regarding commercial cost, privacy, safety, and liability. Consequently, IndustryEQA instead relies on physically simulated data produced within virtual environments. Collecting data from Š 2026 Alibaba. All rights reserved arXiv:2609.09790v1 [cs.CV] 9 Sep 2026 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios The order picker platform at the top is over 2m high and unoccupied. Is this correct? A. Yes B. No, the picker platform is not over 2m high C. No, the picker platform is occupied D. No, the picker platform is neither over 2m high nor unoccupied E. Target object not found F. Unable to determine Is the man in blue vest at bottom right wearing a safety helmet? A. Yes B. No C. Target object not found Who is the man in orange at the bottom? A. Driver B. Passenger C. Pedestrian D. Security inspector E. Unable to determine F. Target object not found Were the following images taken at the same location? A. Yes B. No C. Unable to determine Is PPE unnecessary in the warehouse? A. Yes B. No In frame 1, are the conveyors on the right and center unoccupied? A. Yes B. No What is the function of a truck cargo compartment? A. Providing vehicle power andpropulsion B. Securing wheels when parked C. Loading &transporting goods D. Providing driver navigation :Accordingtologisticsandwarehouseregulations,securityscreening machinesmustbecontinuouslymonitoredbydedicatedpersonnelduring operation.Staffarestrictlyprohibitedfromleavingtheirpost,switching stations,orengaginginunrelatedactivities.Doesthevideoshowany violationofthisrule? A. Violation detected B. No violation detected Accordingtologisticsandwarehouseregulations,anyopenflame, visiblesparks,orsmokeinstorageandoperationalareasisaseverehazard andstrictviolation.Unpermittedhotwork(e.g.,welding,cutting,grinding) isstrictlyprohibited.Doesthevideoshowanyviolationofthisrule? A. Violation detected B. No violation detected Accordingtologisticsandwarehouseregulations,allpersonnel mustevacuatethefacilitybeforethedailypower-offandclosure period.Nooneispermit-edtoremaininsideorcontinueany operations,maintenance,orinspections.Doesthevideoshowany violationofthisrule? A. Violation detected B. No violation detected Accordingtologisticsandwarehouseregulations,poweredforklift operatorsmustnotleavetheirseatuntilthevehicleisfullystopped,braked, andforksareloweredtotheground.Doesthevideoshowanyviolationofthis rule? A. Violation detected B. No violation detected Identifypotentialsafety hazardsinthecurrentscene Identifyallnon-compliant behaviorsinthecurrent operation,andthenprovide Assumeaseverecargo fireoccursduringlogistics operationsinthecurrent scene. Provide the immediate on-site response protocol and specify the emergency actions that personnel must take. the correct standardoperating procedure in a step-by-step format. and analyze the severe consequences that may arise iftimely intervention is not implemented. Figure 1|Overview diagram the LogiScope-VQA, evaluating LMMs in real-world industrial scenarios across three key competency dimensions: industrial element perception, warehouse knowledge understanding, and potential risk reasoning. (The full task taxonomy is provided in Appendix D.3.) real-world industrial environments for systematic evaluation remains a significant gap. (i) In terms of evaluation breadth, practical deployment of LMMs involves diverse safety-critical dimensions, including object perception, spatial reasoning, path planning, and action recognition. However, ARMBench and iSafetyBench are limited to evaluating specific tasks under a single question format. Comprehensive multi-task evaluation of logistics safety-oriented perception and reasoning capabilities has yet to be established. Therefore, the safety-oriented assessment of mainstream LMMs in real-world logistics scenarios remains an open question. To systematically evaluate LMMs for logistics safety inspection, we introduce LogiScope-VQA, an industrial benchmark consisting of 2,476 images, 2,918 videos, 1,0274 VQAs, 20 risk factors, and 18 logistics objects. As illustrated in Fig. 1, we adopt a progressive curriculum to broadly assess the 2 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios (c) Core object distribution(a) Data source distribution 5,394 Images&Videos 7.0% 48.6% 17.6% 23.5% 2.4% 0.8% 18 CoreObjects 32.0% 4.2% 4.6% 15.2% 6.7% 5.5% 7.8% 7.6% 5.0% 2.1% 1.9% 1.2% 1.0% 1.7% 1.4% 1.4% 0.5% 0.5% 8.2% 6.9% 8.1% 3.5% 4.4% 9.8% 7.0% 3.5% 3.5% 3.5% 3.5% 3.5% 3.5% 7.0% 7.0% 1.1% 4.4% 4.4% 3.5% 3.5% (b) Risktypedistribution 20 SafetyRisks Figure 2|Core statistical distributions in LogiScope-VQA, comprehensively covering 5,394 visual samples, 20 safety risks, and 18 core objects. deployment value of LMMs in real-world industrial scenarios. For industrial element perception, 3.5 million surveillance clips collected from the global intelligent logistics company Cainiao capture the dynamics and heterogeneous operational scenarios within logistics parks. For warehouse knowledge understanding, recruited logistics experts collectively devote 52 person-days to annotating essential factors, including spatial locations, object attributes, hazardous actions, etc. For potential risk reasoning, we additionally incorporate open-ended problems and an LLM-as-a-Judge evaluation protocol to assess LMMsâ reliability and fairness in sophisticated thinking. Coupled evaluation tasks pose fundamentally new challenges to LMMs in satisfying the demands of industrial logistics deployment. By conducting a holistic evaluation of 20 mainstream open-source and advanced proprietary LMMs, we summarize the key findings as follows: (a) As exemplified by Qwen3.5-Plus, open-source LMMs closely track proprietary performance and achieve parity with GPT-5.5 and Claude-Opus- 4.7. Nevertheless, LMMs still have a pronounced performance disparity relative to human experts, indicating substantial room for improvement. (b) Industrial scenarios pose unprecedented challenges to fine-grained visual perception. LMMs consistently perform below expectations on the fine-grained industrial element perception tasks. Open-source models attain average accuracies of 0.39 on FG- S and 0.36 on FG-C. (c) Tasks involving industrial element perception and warehouse knowledge understanding are predominantly driven by visual cues. In contrast, potential risk reasoning requires models proficient in sophisticated thinking to identify risk factors. (d) Mainstream LMMs exhibit a pervasive risk-averse bias when responding to risk-prediction queries. Proprietary commercial models tend to adopt conservative responses, whereas open-source models provide more permissive and candid reports. Only 25% of LMMs pass our fairness-criteria audit. Collectively, these findings emphasize the challenges intrinsic to the LogiScope-VQA benchmark and delineate directions for future research and model improvement. The curated benchmark bridges the last-mile from well-trained, simulation-validated models to practical application, providing actionable guidance for real-world readiness. For LMMs developed for large-scale industrial deployment, achieving strong performance on LogiScope-VQA is necessary to demonstrate holistic security expertise and expert-level perception and reasoning in operational environments. We hope the unique visual perception challenges, risk reasoning characteristics, and response bias tendencies introduced by LogiScope-VQA will provide useful insights for future work. 3 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios Step3 Visual Data Acquisition Internal Logistics Monitoring global smart logistics company real-world monitoring 3,050 images 4,050 videos object localization individual attributes Object Attribute Annotation QA filtering LogiScope-VQA sample cleaning safety manuals structured knowledge base basic annotators JSON schema perso n person person electriccarrier electricforklift Automated Question Generation Question-Answering Annotation specialized annotators 1,0274 QA pairs answer formulation cross-validation Logistics Dataset Assembly 1,0274 VQAs 2,476 images; 2,918videos 20 risk types 18coreobjects object attributes 15 pre-defined question templates placeholder population 1,1416questions What is the identity of the person in the orange vest at the top of the image? question stem refinement What is the identity of the person in the orange vest at the top of the image? A)Driver B)Passenger C)Pedestrian 1 2 3 45 1 4 s u b t a s k s 5 s u b t a s k s 2 0 s u b t a s k s r i s k r e a s o n i n g e l e m e n t p e r c e p t i o n k n o w l e d g e u n d e r s t a n d i n g targeted editing Figure 3|The curation pipeline of LogiScope-VQA, rigorously presenting the operational protocol alongside example outputs. 2. LogiScope-VQA 2.1. Overview of Logistics Benchmark We introduce the LogiScope-VQA benchmark, a novel benchmark meticulously curated to evaluate the perception and reasoning capabilities of foundation LMMs for logistics safety in real-world industrial scenarios. The visual corpus is derived from the warehouse surveillance of the global intelligent logistics company Cainiao, comprising 3.5 million raw clips that, after rigorous cleaning, filtering and augmentation, are distilled into 5,394 high-quality visual samples. Logistics questions are auto- matically generated from human-annotated object attributes and hand-crafted question templates, comprehensively covering single-choice, multiple-choice, and open-ended formats. Corresponding answers are authored, annotated, and cross-validated by recruited logistics experts through a hi- erarchical workflow, resulting in 10,274 high-quality VQA instances. Collectively, the benchmarkâs thorough data release and holistic experimental evaluation are underpinned by one year of surveil- lance across industrial warehouses worldwide, 52 person-days of annotation by logistics experts, and 324 million tokens consumed during LMM API inference. Grounded in diverse visual content and granular attribute annotations, our benchmark encom- passes single-choice, multiple-choice, and open-ended VQA problems. As illustrated in Fig. 2, we achieve comprehensive coverage of key elements in industrial logistics scenarios, spanning 18 core objects and 20 risk types. We aim to directly challenge LMMs to perceive logistics objects and integrate vertical knowledge to uncover latent industrial hazards. Eventually, LogiScope-VQA is designed to rigorously evaluate the essential skills of LMMs, spanning industrial element perception, warehouse knowledge understanding, and potential risk reasoning. Our benchmark proposes four key challenges for multimodal foundation models. Across diverse tasks, it requires LMMs to (i) perceive dense object distributions in industrial environments, (i) acquire and operationalize logistics-specific domain knowledge, (i) integrate multimodal information to perform reasoning, and (iv) anticipate and identify potential safety risks via sophisticated thinking. We aim for the curated benchmark to serve as a comprehensive LMM testbed for identifying logistics safety risks in industrial scenarios, thereby precisely evaluating the real-world deployment potential of mainstream foundation models. 2.2. Distinctive Positioning of LogiScope-VQA Distinct from existing benchmarks, we require LMMs to perform visual question-answering grounded in industrial contexts, enabling an intuitive assessment of practical operational value. The most 4 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios closely related prior efforts are IndustryEQA (Li et al., 2026) and iSafetyBench (Abdullah et al., 2025). However, IndustryEQA derives industrial data from simulated virtual environments, while iSafetyBench focuses solely on hazardous action recognition, constraining the breadth of visual elements and task coverage. In contrast, LogiScope-VQA primarily sources data from real-world logistics warehouses, a nearly tenfold size over the previous dataset, and conducts a comprehensive multi-dimensional evaluation across perception, understanding, and reasoning. 2.3. Dataset Curation Pipeline The construction of LogiScope-VQA followed a hierarchical curation pipeline that focused on data coverage and annotation quality, as illustrated in Fig. 3. In step 1, we aggregated one year of surveillance video data from Cainiaoâs global warehouse parks, amounting to 3.5 million raw clips. This massive corpus was then cleaned through video segmentation, camera-zone grouping, object detection pre-labeling and deduplication, yielding more than 7K high-quality visual samples as the initial seed data. Targeted editing is performed on approximately 900 samples with Nano Banana 2 (Gemini, 2023) to rebalance the distribution of rare risk types. (The detailed image editing prompt is presented in Appendix D.4.) These over 7K seed samples were subsequently filtered in step 4 based on VQA pair quality, retaining the final 5,394 visual samples. Concurrently, we identified core objects of interest within logistics safety and transformed their associated unstructured safety manuals into a structured schema. In step 2, basic annotators spent 12 person-days annotating 18 core objects and their basic attributes. Specifically, they first drew bounding boxes to identify designated objects (e.g., persons, electric forklifts) on the keyframes, and then labeled the corresponding attributes (e.g., frame location, gender). To ensure accuracy, a third-party audit was conducted, with substandard annotations returned for iterative correction. In step 3, the labeled object attributes were populated into the placeholders of the pre-defined 15 question templates to batch-generate initial VQA pairs, followed by a refinement operation. Such operation not only resolves referential ambiguity by adding specific modifiers and descriptive constraints to object names, but also improves sentence fluency by ensuring grammatical correctness and natural phrasing. In step 4, four specialized annotators leveraged their domain expertise to annotate ground-truth answers and simultaneously filter out low-quality VQA pairs. This three-round cross-validation process, which took 40 person-days in total, comprised: (i) initial answer annotation and filtering of ill-formed pairs (i.e., those with ambiguous referents, unnatural phrasing, or missing options), which retained 5,394 visual samples from the initial 7K seed samples; (i) independent re-annotation by a senior expert to accept concordant cases and adjudicate discordant cases with reference to the original labels to determine preferred answers; and (i) a stratified review that prioritized verifying cases with prior inter-round inconsistency. Consequently, approximately 90% of the data was retained as high-quality VQA pairs. In step 5, to ensure dataset diversity, we balanced the distribution of core objects, risk types, and hallucination rates by removing over-represented categories. Ultimately, the entire pipeline produced LogiScope- VQA, a comprehensive dataset comprising 10,274 high-quality VQA pairs. (Thorough details of the data processing pipeline are provided in Appendix D) 3. Experiment 3.1. Evaluation Setups LMM Baselines We evaluate the difficulty of LogiScope-VQA using diverse state-of-the-art LMMs as baselines and establish robust reference points for future research. For proprietary LMMs, six leading commercial models are selected: Claude-Sonnet-4.6 (Team., 2026b), Claude-Opus-4.7 (Team., 2026a), Gemini-3.1-Pro (Gemini, 2023), GPT-5.4, GPT-5.5 (OpenAI, 2025), and Qwen3.7-Plus QwenTeam 5 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios Table 1|Main results on LogiScope-VQA across 10 evaluation tasks. Abbr., FP-S: Fine-grained Percep- tion of Single-instance, FP-C: Fine-grained Perception of Cross-instance, CP: Coarse-grained Perception; WC: Warehouse Commonsense, SR: Spatial Relation, OR: Operator Role; PAC: Perimeter Access Con- trol, FM: Fire Monitoring, PSD: Personnel Safety Duty, EOC: Equipment Operation Compliance. The optimal proprietary and open-source models are highlighted inpurple andyellow , respectively. ModelOverall Industrial Element Perception Warehouse Knowledge Understanding Potential Risk Reasoning FP-S FP-CCPWCSRORPACFMPSD EOC Heuristics Baselines Random Choice0.330.24 0.22 0.320.31 0.15 0.070.37 0.46 0.46 0.47 Frequent Guess0.470.32 0.27 0.630.39 0.24 0.400.74 0.49 0.64 0.80 Proprietary Large Multimodal Models Claude-Sonnet-4.60.640.63 0.64 0.860.90 0.58 0.200.49 0.76 0.58 0.62 Claude-Opus-4.70.700.64 0.61 0.880.89 0.60 0.300.74 0.82 0.65 0.79 Gemini-3.1-Pro0.680.720.700.910.920.66 0.300.48 0.82 0.58 0.61 GPT-5.40.690.59 0.58 0.900.93 0.59 0.250.580.840.69 0.78 GPT-5.50.710.62 0.68 0.780.95 0.59 0.200.79 0.82 0.650.81 Qwen3.7-Plus0.660.67 0.64 0.890.95 0.560.400.48 0.83 0.53 0.69 Open-source Large Multimodal Models Kimi-K2-Thinking0.490.45 0.54 0.720.96 0.18 0.050.15 0.30 0.29 0.14 Kimi-K2.60.620.61 0.60 0.890.93 0.45 0.200.570.85 0.59 0.66 Qwen3-VL-8B0.500.48 0.44 0.890.87 0.09 0.050.42 0.73 0.57 0.65 Qwen3-VL-32B0.550.55 0.54 0.830.92 0.27 0.150.46 0.74 0.60 0.59 Qwen3-VL-235B0.640.64 0.59 0.890.91 0.55 0.250.43 0.73 0.60 0.67 Qwen3-VL-Plus0.340.53 0.45 0.760.91 0.15 0.050.20 0.57 0.30 0.13 Qwen3.5-Plus0.700.660.650.900.930.610.450.48 0.84 0.59 0.77 InternVL3.50.330.48 0.39 0.880.69 0.04 0.050.44 0.83 0.42 0.17 LLaVA-v1.6-7B0.500.34 0.30 0.480.60 0.21 0.050.85 0.62 0.690.82 MiMo-VL-7B-RL0.440.46 0.39 0.650.32 0.05 0.100.44 0.740.70 0.60 GLM-4.70.480.40 0.41 0.700.95 0.20 0.150.06 0.40 0.29 0.16 GLM-5.20.530.44 0.45 0.720.95 0.20 0.150.41 0.45 0.37 0.38 MiniMax-M2.10.430.47 0.49 0.540.93 0.19 0.150.50 0.43 0.40 0.60 MiniMax-M2.50.460.50 0.54 0.610.93 0.18 0.150.42 0.41 0.43 0.59 Human Performance Novice Student0.730.80 0.81 0.850.76 0.58 0.750.75 0.80 0.70 0.67 Logistics Expert0.950.970.970.961.000.990.950.930.950.950.91 (2026). These models represent the current upper bound of closed-source multimodal capabilities and serve as strong performance ceilings for LogiScope-VQA. For open-source LMMs, models spanning multiple architectural families and parameter scales are included: MiMo-VL-7B-RL (Team, 2025), GLM-5.2 (Z.ai, 2026), GLM-4.7 (Z.ai, 2025), Kimi-K2-Thinking (Kimi AI, 2025), Kimi-K2.6 (Kimi AI, 2026), InternVL-3.5 (InternVL, 2025), LLaVA-v1.6-7B (LLaVA, 2024), Qwen3.5-Plus (QwenTeam, 2026), and multiple Qwen3-VL variants (8B, 32B, 235B-A22B, Plus) (Bai et al., 2025). Moreover, we incorporate the Random Choice and Frequent Guess baselines to provide chance-level and prior-biased reference points. While computed in the standard manner for multiple-choice questions, the Random Choice accuracy is treated as zero for open-ended questions, given the extremely low probability that a solution randomly sampled from the vast candidate space would be consistent with the ground-truth answer. Meanwhile, the Frequent Guess baseline uses the most frequent canonical answer within each task category as the prediction for all open-ended questions. 6 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios Human Performance To assess holistic human performance, we recruited ten college students with no expertise in industrial logistics to represent the novice level, and one senior logistics specialist with over four years of on-site warehouse safety experience to represent the expert level. Human participants completed all questions independently based solely on professional knowledge, without external references or model outputs. This setup ensures that the reported performance reflects genuine domain expertise rather than test-taking strategies or external aids, serving as a reliable upper bound for model evaluation. (The detailed employment terms for human evaluators are provided in the Appendix E.2.) Decoding Settings For all problem instances, we use a unified prompt template that explicitly instructs the models to produce answers in a standardized format. For single-choice and multiple- choice questions, correctness is determined via direct keyword matching. For open-ended questions, we develop an LLM-as-a-Judge workflow to extract the key conclusion from each response for answer matching. Eventually, we adopt micro-averaged accuracy as the evaluation metric. All evaluated models are accessed via their official APIs or publicly available checkpoints. For each task, we repeat each test three times and report the average performance to ensure reliability and reproducibility. The sampling temperature for each model is set to its recommended or default value. (Detailed prompts and decoding setups are in Appendix E.1.) 3.2. Overall Results 3.2.1. Main Performance Analysis As reported in Tab. 1, proprietary models maintain an advantage, yet the gap with open-source models has essentially closed at the top, where the best open-source model now ties the strongest proprietary systems. These leading models have edged past the untrained human (novice student) while remaining far below the logistics expert, confirming that our benchmark measures learnable professional expertise rather than common intuition. The key finding, however, is that fine-grained perception in logistics safety scenarios constitutes a shared, industry-wide bottleneck. All top models cluster within a narrow, low band on FG-S and FG-C, falling short of even the novice human and trailing the expert by a wide margin. This is partly because surveillance images in these settings are typically wide-angle, low-resolution, and densely cluttered with numerous targets and backgrounds, posing a far greater challenge than the clean, high-quality images common in standard benchmarks. In stark contrast, on knowledge-intensive tasks such as AR and EOC, models already outperform untrained humans. This contrast reveals an interesting structural complementarity between human and machine capabilities. Humans naturally excel at spotting fine-grained visual details in logistics safety scenarios even without training, whereas models, despite their vast knowledge, often behave like a well-read scholar with poor eyesight. 3.2.2. In-depth Thinking Analysis As shown in Table 2 (evaluated on a subset of LogiScope-VQA), red, orange and blue cells present notable increases, moderate increases and noticeable drops, respectively. We can see that enabling thinking mode yields a mixed effect rather than a uniform improvement. Across models, the overall ranking remains largely stable, but the performance shift is highly task-dependent, suggesting that thinking mode interacts differently with perception-heavy and reasoning-heavy tasks. Industrial Element Perception is the least responsive dimension, while Warehouse Knowledge Understanding shows modest gains, which may be constrained by the perceptual layer itself. However, Potential Risk Reasoning benefits the most, especially for stronger models such as Qwen3.5-Plus. This is because the 7 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios process of observing the scene, matching safety rules and drawing inferences is reasoning-dominant, making it particularly well aligned with explicit chain-of-thought enabled by thinking mode. Building on this observation, we further probe whether the reasoning advantage persists as the thinking time budget increases for the same Potential Risk Reasoning VQA samples. Figure 4 indicates a clear saturation pattern across all models. At low budgets, performance degrades severely due to output truncation, while further gains become marginal beyond the standard budget. Therefore, to balance performance and token consumption, thinking mode should be selectively enabled depending on the taskâs primary reliance on visual perception or reasoning. And the time budget should be set near the saturation point. Table 2|Cross-task performance comparison with and without thinking mode. Colored cells indicate significant changes. Model Industrial Element Perception Warehouse Knowledge Understanding Potential Risk Reasoning non-thinkthinknon-thinkthinknon-thinkthink LLaVA-v1.6-7B0.280.30 â7.1%0.270.31 â14.8%0.650.71 â9.2% Kimi-K2.60.660.660.710.57 â19.7%0.680.74 â8.8% MiniMax-M2.10.140.10 â28.6%0.380.380.620.63 GLM-5.20.150.12 â20.0%0.390.390.500.54 â8.0% Qwen3-VL-235B0.630.640.660.690.700.71 Qwen3.5-Plus0.690.68 â1.5%0.720.740.720.80 â11.1% Qwen3.7-Plus0.690.690.710.68 â4.2%0.690.77 â11.6% 0.01x0.5x1x2x2.5x Time Budget 0% 20% 40% 60% 80% 100% 120% 140% 160% Score (% of standard) GLM-5.2 Kimi-K2.6 LLaVA-v1.6-7B MiniMax-M2.1 Qwen3.5-Plus Qwen3.7-Plus Figure 4|Impact of thinking time budget on reasoning-driven tasks. 3.2.3. Safety Risk Bias Analysis We investigate whether modelsâ safety risk judgments exhibit systematic bias or random noise with two experiments, as shown in Fig. 5. In panel (a), we quantify global directional skew via the indicator RBS, defined asRisk Bias Score= tanh log((FNR+í)/(FPR+í)) í , where FNR and FPR denote the false negative rate and false positive rate, andí=10 â8 andí=2. Positive, negative, and near-zero RBS indicate optimistic, conservative, and neutral judgments, respectively. Our evaluation reveals two critical findings. First, conservative bias dominates, with most models exhibiting a negative RBS. This systematic over-reporting tendency likely reflects a safety-first alignment strategy that prioritizes high recall, penalizing missed detections more heavily than false alarms. Second, proprietary and open-source models exhibit a marked performance divergence, suggesting a cognitive divide between genuine risk comprehension and superficial pattern matching. While top proprietary models cluster near the unbiased neutral zone, open-source ones scatter across the entire score range. This pattern implies that elite proprietary models may leverage event-level risk reasoning to accurately distinguish benign events from hazards, whereas less capable models lack such depth. Consequently, the latter likely rely on defensive over-reporting to mitigate uncertainty, manifesting as systemic conservative bias. Building on this global picture, we conduct a paired-image probe to examine how such bias behaves under explicit risk removal. Specifically, we evaluate models on 100 surveillance samples containing genuine hazards and a corresponding set of risk-free samples, captured by the same cameras at a different time period, to measure recall and false alarm rates, respectively. As depicted in panel (b), most models maintain a risk recall above 60% but their false alarm rates vary substantially, confirming the prevalence of conservative bias. Moreover, The models are distributed across three quadrants, with those in quadrant 2 exhibiting excessive sensitivity and high false alarm rates, whereas those in quadrant 4 underperform by missing genuine threats. In contrast, models in quadrant 1 achieve the optimal trade-off between recall and false alarm rates, yet they still remain far from human expert performance. Collectively, our results underscore that safety risk bias is a pervasive systemic issue, 8 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios Human Expert Qwen3.5-Plus GPT-5.5 Claude-Opus-4.7 MiniMax-M2.1 InternVL-3.5 (a) Statistical analysis of safety riskbias(b) Dual-dimensionalcomparison of risk recall False Alarm (25%, 75%) GLM-4.7 Kimi-K2 GLM-5.2 100% Gemini-3.1-Pro Claude-Sonnet-4.6 Qwen3.7-Plus Risk Recall GPT-5.4 MiniMax-M2.5 MiMo-VL-7B -0.1 +0.1 Figure 5|Empirical analysis of safety risk biases in mainstream LMMs. Panel (a) depicts holistic characterization of deviations in risk bias scores. Panel (b) provides detailed comparison across risk-recall and false-alarm dimensions. which may be attributed to an interplay between internal cognitive limitations and external factors such as training data and alignment protocols. 4. Challenges and Future Directions Evaluations on LogiScope-VQA expose two key limitations of current models. Firstly, fine-grained perception in logistics safety scenarios remains a major bottleneck, where models particularly struggle with low-resolution, wide-angle, heavily occluded and densely cluttered surveillance footage. To address this, future efforts should aim to boost modelsâ intrinsic visual capacity, achievable through strategies like optimizing pre-training for dense and low-resolution targets, or exploiting temporal information across video frames. Secondly, safety risk bias emerges as a significant yet overlooked issue, characterized by a prevalent unfairness, with most models heavily inclined to over-report risks. This problem deserves more attention from model developers, as industrial application models require fairness as much as accuracy. For future work, a crucial direction is to prevent such bias at the source, for example through balanced data curation or bias-aware training objectives. 5. Related Work 5.1. LMM Industrial Assessment Industrial logistics safety assessment provides a compelling testbed for studying the practical applica- bility of LMMs. Unlike natural-scene evaluation (Liu et al., 2024; Majumdar et al., 2024), industrial environments (Abdullah et al., 2025; Mitash et al., 2023) impose the integration of sophisticated object recognition, precise spatial reasoning, and safety-critical hazard identification. However, this research area remains heavily underexplored, as industrial data source are often constrained by commercial confidentiality. IndustryEQA (Li et al., 2026) is a pioneering study on logistics safety evaluation in warehouse scenarios. However, it contains only approximately 1.3 thousand physics- simulated samples generated in virtual environments. To fully alleviate this challenge, we release LogiScope-VQA, which consists of 10,274 VQA instances annotated by logistics experts. We expect the constructed benchmark to provide a direct and practical testbed for assessing the potential of LMMs in industrial applications. 5.2. Practical Application Benchmarks LMMs have emerged as a major frontier in AI research, offering a unified modeling framework that integrates visual perception, language modeling, and sophisticated reasoning (Alayrac et al., 2022; Liu 9 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios et al., 2023; Radford et al., 2021). Through vision-language pre-training, instruction tuning, and rein- forcement learning for reasoning, cutting-edge research has progressively advanced model capabilities across a broad spectrum of vertical domains. To keep pace with these advances, recent benchmarks have introduced targeted evaluations for reasoning-centric mathematical problem solving (Lu et al., 2024; Wang et al., 2026; Yue et al., 2024), knowledge-intensive medical diagnosis (Chen et al., 2024; Liu et al., 2025; Yin et al., 2026; Zhou et al., 2025b), and visually-grounded remote sensing tasks (Danish et al., 2025; Wang et al., 2025a,b). Logistics safety evaluation focuses on workflows, equipment states, human behaviors and safety regulations, framing an embodied risk-oriented QA task in real industrial settings with distinctive domain-specific challenges. LogiScope-VQA is the first dedicated evaluation suite for logistics scenarios, filling a critical gap in existing research. 6. Conclusion The proposed LogiScope-VQA represents a significant advance in assessing the capabilities of LMMs for logistics hazard identification in industrial scenarios. Concretely, we comprehensively evaluate the basic skills of industrial element perception, warehouse knowledge understanding, and potential risk reasoning. Extensive empirical evaluations reveal that LMMs fall short of expectations in industrial visual perception, and effective hazard identification hinges on coherent sophisticated reasoning. Moreover, our investigation uncovers a pervasive issue of security-related bias in current LMMs. These discrepant findings offer valuable insights for foundation model construction and practical deployment. In future work, we will further enrich the visual question-answering corpus and extend the benchmark for agentic industrial operations, continuously contributing robust resources to the multimodal evaluation research community. Acknowledgments We thank Yuan Zhou and other experts for their professional guidance on logistics safety knowledge and data annotation, and Qian Xu and her colleagues for their support in data annotation. We also thank the Qwen team for an insightful technical exchange, which deepened our understanding of the scarcity of logistics surveillance data for training foundation models. We have since shared a portion of our surveillance data with the team, in the hope that it may serve as a useful resource for future model development. References R. Abdullah, Y. S. Rawat, and S. Vyas. iSafetyBench: A video-language benchmark for safety in industrial environment. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1433â1442, 2025. J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716â23736, 2022. S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, et al. Qwen3-VL technical report. arXiv preprint arXiv:2511.21631, 2025, 2025. P. Chen, J. Ye, G. Wang, Y. Li, Z. Deng, W. Li, T. Li, H. Duan, Z. Huang, Y. Su, et al. GMAI-MMBench: A comprehensive multimodal evaluation benchmark towards general medical ai. Advances in Neural Information Processing Systems, 37:94327â94427, 2024. 10 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios M. Danish et al. GEOBench-VLM: Benchmarking vision-language models for geospatial tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. S. Dou, M. Zhang, Z. Yin, C. Huang, Y. Shen, J. Wang, J. Chen, Y. Ni, J. Ye, C. Zhang, et al. Cl-Bench: A benchmark for context learning. arXiv preprint arXiv:2602.03587, 2026. T. Gemini. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805, 2023, 2023. T. InternVL. InternVL3.5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025. Kimi AI. Introducing Kimi K2 thinking.https://w.kimi.com/blog/kimi-k2-thinking, 2025. Kimi AI. Kimi K2.6 tech blog: Advancing open-source coding.https://w.kimi.com/blog/ki mi-k2-6, 2026. Y. Li, Z. Liu, Z. Li, X. Zhang, Z. Xu, X. Chen, H. Shi, S. Jiang, X. Wang, J. Wang, et al. Perception, reason, think, and plan: A survey on large multimodal reasoning models. arXiv preprint arXiv:2505.04921, 2025. Y. Li, Y. Chen, A. Dao, L. Li, Z. Cai, Z. Tan, T. Chen, and Y. Kong. IndustryEQA: Pushing the frontiers of embodied question answering in industrial scenarios. In Advances in Neural Information Processing Systems, volume 38, 2026. B. Liu, K. Zou, L.-M. Zhan, Z. Lu, X. Dong, Y. Chen, C. Xie, J. Cao, X.-M. Wu, and H. Fu. GEMeX: A large-scale, groundable, and explainable medical vqa benchmark for chest x-ray diagnosis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21310â21320, 2025. H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning. Advances in Neural Information Processing Systems, 36:34892â34916, 2023. Y. Liu, H. Duan, Y. Zhang, B. Li, S. Zhang, W. Zhao, Y. Yuan, J. Wang, C. He, Z. Liu, et al. MMBench: Is your multi-modal model an all-around player? In Proceedings of the European Conference on Computer Vision, pages 216â233. Springer, 2024. Y. Liu, T. Qu, Z. Zhong, B. Peng, S. Liu, B. Yu, and J. Jia. VisionReasoner: Unified reasoning-integrated visual perception via reinforcement learning. In International Conference on Learning Representations, volume 2026, pages 94069â94086, 2026. T. LLaVA. LLaVA: v1.6. https://ollama.com/library/llava:v1.6, 2024. P. Lu, L. Xu, S. Mishra, L. Xia, Y. Nie, C. Zhu, M. Zhang, T. Goldstein, W. Y. Wang, and H. Hajishirzi. MathVista: Evaluating mathematical reasoning of foundation models in visual contexts. In The Twelfth International Conference on Learning Representations, 2024. A. Majumdar, A. Ajay, X. Zhang, P. Putta, S. Yenamandra, M. Henaff, S. Silwal, P. Mcvay, O. Maksymets, S. Arnaud, K. Yadav, Q. Li, B. Newman, M. Sharma, V. Berges, S. Zhang, P. Agrawal, Y. Bisk, D. Batra, M. Kalakrishnan, F. Meier, C. Paxton, S. Sax, and A. Rajeswaran. OpenEQA: Embodied question answering in the era of foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024. 11 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios C. Mitash, F. Wang, S. Lu, V. Terhuja, T. Garaas, F. Polido, and M. Nambi. ARMBench: An object-centric benchmark dataset for robotic manipulation. arXiv preprint arXiv:2303.16382, 2023. OpenAI. OpenAI GPT-5 system sard. arXiv preprint arXiv:2601.03267, 2025. QwenTeam. Qwen3.5: Towards native multimodal agents.https://qwen.ai/blog?id=qwen3.5, 2026. QwenTeam. Qwen3.7: Multimodal agent intelligence.https://qwen.ai/blog?id=qwen3.7-p lus, 2026. A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748â8763, 2021. A. C. Team. Introducing claude opus 4.7.https://w.anthropic.com/news/claude-opu s-4-7, 2026a. A. C. Team. Introducing claude sonnet 4.6.https://w.anthropic.com/news/claude-son net-4-6, 2026b. X. L.-C. Team. MiMo-VL technical report. arXiv preprint arXiv:2506.03569, 2025. F. Wang, M. Chen, X. He, Y.-F. Zhang, Y. Li, F. Liu, Z. Guo, Z. Hu, J. Wang, J. Xu, et al. OmniEarth-Bench: Towards holistic evaluation of earthâs six spheres and cross-spheres interactions with multimodal observational earth data. arXiv preprint arXiv:2505.23522, 2025a. F. Wang et al. XLRS-Bench: Could your multimodal llms understand extremely large ultra-high- resolution remote sensing imagery? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025b. X. Wang, M. Yin, Y. Zhao, G. Liu, and D. Li. LiveK12Bench: Have large multimodal models truly conquered high school-level examinations? arXiv preprint arXiv:2605.26781, 2026. M. Yin, Y. Zhu, J. Wu, J. Ma, H. Zhou, M. Li, Y. Zhou, J. Chen, T. Hou, J. Ye, et al. Caduceus: MoE foundation models for unifying biological and natural language. In Proceedings of the 32nd ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 2, pages 12715â12726, 2026. X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. MMMU: A massive multi-discipline multimodal understanding and reasoning benchmark for expert AGI. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556â9567, 2024. Z.ai. GLM-4.7: Advancing the coding capability. https://z.ai/blog/glm-4.7, 2025. Z.ai. GLM-5.2: Built for long-horizon tasks. https://z.ai/blog/glm-5.2, 2026. H. Zhou, M. Yin, D. Chen, J. Wu, and J. Chen. Group-On: Boosting one-shot segmentation with supportive query. In 2025 IEEE International Conference on Multimedia and Expo (ICME), pages 1â6. IEEE, 2025a. H. Zhou, M. Yin, W. Wu, M. Li, K. Fu, J. Chen, J. Wu, and Z. Wang. ProtCLIP: Function-informed protein multi-modal learning. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 22937â22945, 2025b. 12 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios A. Large Language Model Usage The conceptual innovation and motivation underlying this study are conceived independently by the human authors, without cognitive input from large language models. The manuscript is originally written by the human authors, and large language models are engaged only at the final stage to assist in polishing key academic terminology. During dataset construction, we primarily relied on proprietary real-world data, while approximately 20.5% of the dataset is generated by the large language model (i.e., Nano Banana 2 (Gemini, 2023)) to systematically enrich the diversity of risk factors. For experimental evaluation, we employ large language models exclusively through the official API endpoints provided by respective vendors. All humanâAI interactions are conducted in full compliance with all applicable terms of service and licensing conditions. B. Ethics Statement We are committed to responsible AI research and strict adherence to data privacy standards. For the private data in LogiScope-VQA, we have implemented rigorous anonymization protocols. Specifically, all privacy-sensitive contentâincluding clearly visible human faces, specific warehouse names and commercial identifiersâhas been blurred or masked. This process combines automated detection with manual verification, ensuring that no individual or proprietary entity can be identified. In contrast, since the synthetic data contains no real-world identities and the open-source data is already publicly available, both are released in their original format without additional anonymization. We confirm that the construction and release of LogiScope-VQA comply with relevant ethical guidelines and data protection regulations. C. Data Usage and Licensing LogiScope-VQA is released under the Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (C BY-NC-SA 4.0). It is strictly intended for academic and non-commercial research purposes only. Users are prohibited from using this data for any commercial activities, including but not limited to training commercial models, selling derived products, or integrating it into profit-generating services. Furthermore, any derivative works based on this dataset must be distributed under the same license. By accessing this dataset, users agree to adhere to these terms. D. Curation Details D.1. Hierarchical Curation Pipeline Visual Data Acquisition The raw surveillance corpus comprises 3.5 million video clips collected from Cainiaoâs global warehouse parks between April 15, 2025 and April 10, 2026. The data spans diverse operational scenarios, including high rack areas, loading docks, storage zones, light-duty picking shelves, office areas, etc. Notably, we declare that no surveillance data from the United States was included in this collection. To transform this massive raw corpus into a usable foundation, we implemented a rigorous cleaning process. Continuous videos were first segmented into clips or keyframes via fixed-interval sampling or risk-triggered alarm retrieval. These segments were then grouped by camera zones to align with specific operational areas. Subsequently, a pre-trained object detection model identified core objects, allowing us to filter out irrelevant background frames and balance object category distributions. The process concluded with deduplication, yielding over 7,000 high-quality, unique visual samples that serve as the foundational seed data for benchmark 13 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios Person ⢠attrs_object_quality [Clear, Normal, Blurry] ⢠attrs_object_occlusion [No Occlusion, Light, Moderate, Heavy] ⢠attrs_object_frame_location [Top, Bottom, Left, Right, Top-Left, Bottom-Left, Top-Right, Bottom-Right, Center] ⢠attrs_object_direction [Left, Right, Away from Camera, Facing Camera, No Direction, Away from Ground, Facing Ground, Unrecognizable, Other] ⢠attrs_object_status [Standing, Sitting, Lying, Bending, Squatting, Unrecognizable, Other] ⢠attrs_human_action [No Action, Picking, Climbing, Walking, Running, Driving, Getting Off, Getting On, Unrecognizable, Other] ⢠attrs_human_safety_hat [Yes, No, Unrecognizable] ⢠attrs_human_safety_belt [Yes, No, Unrecognizable] ⢠attrs_human_gender [Male, Female, Unrecognizable] ⢠attrs_human_upper_clothing_type [None, Short Sleeve, Long Sleeve, Vest, Unrecognizable, Other] ⢠attrs_human_upper_clothing_color [White, Black, Gray, Blue, Red, Orange, Green, Yellow, Cyan, Purple, Pink, Brown, Silver, Unrecognizable, Other] ⢠attrs_human_safety_hat_color [White, Black, Gray, Blue, Red, Orange, Green, Yellow, Cyan, Purple, Pink, Brown, Silver, Unrecognizable, Other] ⢠attrs_human_identity [Driver, Passenger, Pedestrian, Security Guard, Team Leader, Unrecognizable, Other] ⢠attrs_hoi_ground [Ground Level, Low Altitude, High Altitude, Unrecognizable] ⢠attrs_hoi_package [Picking, Stepping, Unrecognizable, Irrelevant] ⢠attrs_hoi_conveyor_belt [Stepping, Sitting, Leaning, Climbing, Lying, Other, Unrecognizable, Irrelevant] ⢠attrs_hoi_truck_compartment [Inside, Protruding, On Top, Climbing, Intruding Blind Spot, Unrecognizable, Irrelevant] ⢠attrs_hoi_rack [Climbing, Within High Rack Range, Unrecognizable, Irrelevant] Electric Forklift ⢠attrs_object_quality [Clear, Normal, Blurry] ⢠attrs_object_occlusion [No Occlusion, Light, Moderate, Heavy] ⢠attrs_object_frame_location [Top, Bottom, Left, Right, Top-Left, Bottom-Left, Top-Right, Bottom-Right, Center] ⢠attrs_object_color [White, Black, Gray, Blue, Red, Orange, Green, Yellow, Cyan, Purple, Pink, Brown, Silver, Unrecognizable, Other] ⢠attrs_object_direction [Left, Right, Away from Camera, Facing Camera, No Direction, Away from Ground, Facing Ground, Unrecognizable, Other] ⢠attrs_object_status [Stationary, Turning, Moving Straight, Unrecognizable, Other] ⢠attrs_ooi_powered_forklift_package [Fork Loaded, Over-height Load, Pushing Obstacle, Unrecognizable, Irrelevant] ⢠attrs_object_forks_height [Ground Level, Above Ground, Unrecognizable] ⢠attrs_object_forks_direction [Left, Right, Away from Camera, Facing Camera, No Direction, Unrecognizable, Other] ⢠attrs_object_forks_backward [Yes, No, Unrecognizable] Truck ⢠attrs_object_quality [Clear, Normal, Blurry] ⢠attrs_object_occlusion [No Occlusion, Light, Moderate, Heavy] ⢠attrs_object_frame_location [Top, Bottom, Left, Right, Top-Left, Bottom-Left, Top-Right, Bottom-Right, Center] ⢠attrs_object_color [White, Black, Gray, Blue, Red, Orange, Green, Yellow, Cyan, Purple, Pink, Brown, Silver, Unrecognizable, Other] ⢠attrs_object_direction [Left, Right, Away from Camera, Facing Camera, No Direction, Away from Ground, Facing Ground, Unrecognizable, Other] ⢠attrs_object_status [Stationary, Turning, Moving Forward, Reversing, Unrecognizable, Other] ⢠attrs_ooi_truck_compartment_package [Loaded, Empty, Unrecognizable, Irrelevant] ⢠attrs_ohi_truck_compartment_human [Person in Blind Spot, Unrecognizable, Irrelevant] Figure 6 | JSON schema examples for person, truck and electric forklift. 14 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios Determine the <imageAttr>presented in <image>. Identifypotentialsafetyhazardsinthecurrentscene andanalyzethesevereconsequencesthatmayariseif timelyinterventionisnotimplemented. Identifyallnon-compliantbehaviorsinthecurrent operation,andthenprovidethecorrectstandard operatingprocedureinastep-by-stepformat. Are the following images captured at the same location? Assume<incident>occursduringlogisticsoperations inthecurrentscene.Providetheimmediateon-site responseprotocolandspecifytheemergencyactions thatpersonnelmusttake. What is the <objectAttr>of <object>? What is the function of <object>? <object><condition>, is its <objectAttr> equal to <objectAttrValue>? Is the <objectAttr>of <object>equal to <objectAttrValue>? What is the relationship between <object1>and <object2>? Is the <objectAttr>of <object> <objectAttrValue1>or <objectAttrValue2>? Is it correct that there is a <relation> between <object1>and <object2>? What might happen given the <relation> between <object1>and <object2>? <object1><condition>, is it correct that there is a <relation>between it and <object2>? Do <object1>and <object2>have the same <objectAttr>? Figure 7 | Question templates for generating question stems. construction. With this visual foundation in place, a structured knowledge base from unstructured safety manuals became essential. Specifically, logistics specifications are parsed into hierarchical JSON schemas based on semantic tuples (Object,Attribute,Value). In this formulation,Object denotes one of the 18 core entities critical to warehouse safety monitoring, such as person, electric carrier and security scanner.Attributecaptures either the intrinsic state of an object (e.g., posture, PPE compliance) or its interaction relationship with other entities (e.g., human-truck proximity). Valueconsists of predefined enumerations derived from domain regulations, strictly constraining the attribute space. For instance, a valid tuple might be(person, attrs_hoi_rack, climbing), which explicitly encodes the specific interaction of a person climbing a rack. Some examples of the JSON schema are shown in Fig. 6. Object Attribute Annotation In Object Attribute Annotation, visual grounding was established via lightweight, cost-effective manual labeling. Specifically, annotators followed a "box-then-attribute" protocol: they first drew bounding boxes to localize core objects, and then marked basic attributes (e.g., frame location, occlusion status). For video samples, this process was applied exclusively to the single keyframe exhibiting the most salient risk characteristics, thereby maximizing annotation efficiency. Furthermore, to enrich semantic details, the "person" object received extended tags such as gender and upper-garment color. Crucially, all labels were constrained to predefined values aligned with the logistics knowledge base. To ensure final precision, a third-party quality audit was conducted, involving iterative sampling and correction cycles to rectify any remaining inaccuracies. Automated Question Generation This step transformed object attribute annotations into natural language queries through two sequential operations. First, we leveraged a library of 15 predefined question templates (Fig. 7) to generate raw question stems. Specifically, twelve multiple-choice templates were designed to assess cognitive understanding across dimensions such as holistic scene interpretation, single-object recognition, and multi-object interaction. The remaining three open- ended templates evaluated capabilities in safety risk perception, consequence prediction, hazard rectification, and emergency response planning. Based on these templates, we performed deterministic population by mapping extracted core objects and attribute values into the corresponding slots. Second, 15 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios Risk NameDefinition Warehouse Access Control Presence After Night Lockdown InaccordancewiththewarehousesafetyregulationâWarehouseAccessControlâ: Duringthedailypower-offandlockdownperiod,allpersonnelmustfinishtheirworkinadvanceandevacuatethewarehouse areaorderly;noonemayremainbehindorcontinuewarehousing,inspection,ordebuggingactivitiesforanyreason. Key Post Attendance Unattended Security Scanner Post InaccordancewiththewarehousesafetyregulationâKeyPostAttendanceâ: Adedicatedstaffmembermustmonitorthesecurityscannerandgoodsinrealtime.Theattendantmustnotleavethepost,swapduties, orengageinunrelatedactivitieswithoutauthorization. Personal Protective Equipment No Helmet in High-Rack Area InaccordancewiththewarehousesafetyregulationâPersonalProtectiveEquipmentâ: Allpersonnelenteringthehigh-rackstorageareaâwhetherforoperations,patrols,ortemporarypassageâmustproperlyweara standards-compliantsafetyhelmetwiththechinstrapfastened;thisareaposesrisksoffallinggoodsandbeamimpacts,andanyone withoutprotectiveequipmentisstrictlyforbiddentoenter. Elevated Work#1 Non-Designated Climbing Tool InaccordancewiththewarehousesafetyregulationâElevatedWorkâ: Allelevatedworkmustusedesignated,inspection-approvedtoolslikeaerialworkplatforms,inventorycages,orderpickers,A-frame ladders,ormobileladders;steppingonracks,cargoboxes,forkliftforks,orothernon-purpose-builtstructuresatheightisprohibited. Elevated Work#2 Multiple Riders on Climbing Device InaccordancewiththewarehousesafetyregulationâElevatedWorkâ: Dedicatedelevatingdevicessuchasliftingcagesandorderpickersarelimitedtoasingleoccupant;carryingtwoormorepersons simultaneouslyisstrictlyprohibited,asoverloadingsignificantlyreducesequipmentstabilityandincreasestheriskoftippingandfalls. Elevated Work#3 Elevated Work Without Protection InaccordancewiththewarehousesafetyregulationâElevatedWorkâ: WhenworkingatheightonanA-frameladderormobileladder,asafetyhelmetmustbewornproperly;whenusingmobileplatforms suchasaerialworkplatforms,inventorycages,ororderpickers,aprotectivehelmetisrequiredandasafetyharnessmustbesecuredto areliableanchorpointatalltimes. Fire Monitoring Open Flame or Smoke InaccordancewiththewarehousesafetyregulationâFireMonitoringâ: Inwarehouseandoperationareas,anyopenflame,visiblefirelight,flyingsparks,orsmokeemissionconstitutesaserioussafetyhazard andistreatedasaviolationupondiscovery.Unpermittedhotworkâsuchaswelding,cutting,orgrindingthatproduceshigh- temperaturesparksâisstrictlyprohibited. Dock Loading & Unloading#1 Personnel in Truck Danger Zone InaccordancewiththewarehousesafetyregulationâDockLoading&Unloadingâ: Whilethetruckismoving(includingstarting/reversing),noonemayenterthedangerzone:3mfront/rearand1msides.Thisarea coversblindspotsandposesseverecollision/crushingrisks.Driversmustcheckmirrors,cameras,andaids,proceedingslowlyonly afterconfirmingtheareaisclear. Dock Loading & Unloading#2 Personnel in Moving Truck Bed InaccordancewiththewarehousesafetyregulationâDockLoading&Unloadingâ: Noonemayremaininoronthecargobedwhilethetruckismoving.Afterloading/unloading,personnelmustevacuatetoasafearea, andthedrivermaymovethevehicleonlyafterconfirmingthebedisempty. Dock Loading & Unloading#3 Dock Climbing or Jumping InaccordancewiththewarehousesafetyregulationâDockLoading&Unloadingâ: Personnelmustgetonandoffthedockonlyviadesignatedpassagessuchasdedicatedstairs,ramps,ordocklevelers;climbingdock guardrails,crossingedgestructures,orjumpingfromthedockdowntothegroundorvehiclesisstrictlyprohibited.Suchbehaviorcan easilyleadtomissteps,slips,andfalls,causingseriouspersonalinjury. Forklift Driving Compliance#1 OverheightLoad on Forklift InaccordancewiththewarehousesafetyregulationâForkliftDrivingComplianceâ: Whenpoweredforklifts(electricforklifts,electricpallettrucks,semi-electricpallettrucks,palletstackers,orclamptrucks)travelwitha load,thestackedcargoheightmustnotexceed1.8metersortheupperlimitofthemast,soastopreventrolloverfromanexcessively highcenterofgravityorcollisionscausedbyblockedvision;cargostabilitymustbecheckedbeforeloadingtoensuresafeoperation. Forklift Driving Compliance#2 Loaded Forklift Not Reversing InaccordancewiththewarehousesafetyregulationâForkliftDrivingComplianceâ: Whenpoweredforklifts(electricforklifts,electricpallettrucks,semi-electricpallettrucks,palletstackers,orclamptrucks)travelwitha load,theforksmustfacebackwardwiththecargoheldtightagainstthemast,andthetruckmusttravelinreversetoensureclearvision andastablecenterofgravity;drivingforwardwithaloadisstrictlyprohibited. Forklift Driving Compliance#3 Insufficient Forklift Safety Distance InaccordancewiththewarehousesafetyregulationâForkliftDrivingComplianceâ: Duringpoweredforkliftoperations(electricforklifts,electricpallettrucks,semi-electricpallettrucks,palletstackers,orclamptrucks), non-operatorsmustmaintainasafedistance.Ifboundarylightsarepresent,donotenterthewarningzone.Otherwise,keepclearof2m front/rearand1msides. Forklift Driving Compliance#4 Forklift Pushing Cargo on Ground InaccordancewiththewarehousesafetyregulationâForkliftDrivingComplianceâ: Usingpoweredforklifts(electricforklifts,electricpallettrucks,semi-electricpallettrucks,palletstackers,orclamptrucks)topush cargoorobstaclesalongthegroundisstrictlyprohibited,assuchoperationcaneasilycauseequipmentlossofcontrol,cargotipping,or personnelbeingrunover;tomoveheavyobjects,usededicatedhandlingtoolsorplacethecargosecurelyontheforksbeforetransport. Forklift Driving Compliance#5 Extra Riders on Forklift InaccordancewiththewarehousesafetyregulationâForkliftDrivingComplianceâ: Poweredforklifts(electricforklifts,electricpallettrucks,semi-electricpallettrucks,palletstackers,orclamptrucks)mayonlybe operatedbyasinglelicenseddriver,andcarryinganyotherpersonisstrictlyprohibited;theforks,forkcarriage,oranypartofthetruck bodymustneverbeusedasaplatformforcarryingpeople. Forklift Driving Compliance#6 Forklift Forks Not Lowered InaccordancewiththewarehousesafetyregulationâForkliftDrivingComplianceâ: Whilepoweredforklifts(electricforklifts,palletstackers,orclamptrucks)aretravelingwithorwithoutaload,theforksmustbe loweredtoasafeheightofabout15cmabovetheground;travelingwithraisedforksisstrictlyprohibited. Forklift Driving Compliance#7 Operator Leaning Out of Cabin InaccordancewiththewarehousesafetyregulationâForkliftDrivingComplianceâ: Whileapoweredforklift(electricforklift,clamptruck,etc.)isinmotion,theoperatormuststayinsidethecabatalltimesandmustnot extendthehead,hands,torso,oranyotherbodypartoutofthecab.Suchbehaviorcaneasilycausescrapingcollisionswithracks,door frames,pipes,orotherequipment,resultinginseriouspersonalinjury. Forklift Driving Compliance#8 Dismounting a Moving Forklift InaccordancewiththewarehousesafetyregulationâForkliftDrivingComplianceâ: Operatorsmustnotleavetheseatuntilanypoweredforklift(e.g.,electric,pallet,stacker,clamp)isfullystoppedwithbrakesapplied. Fortemporaryabsence,stop,applythehandbrake,lowerforkstotheground,andensuretheequipmentissafebeforeleaving. Forklift Driving Compliance #9 Overloaded Inventory Cage InaccordancewiththewarehousesafetyregulationâForkliftDrivingComplianceâ: Whenaninventorycageismountedontheforkliftforks,thecageislimitedtoasingleoccupant;carryingmorethanonepersonis strictlyprohibited,asoverloadingsignificantlyreducesequipmentstabilityandincreasestheriskoftippingandfalls. Conveyor Operation Compliance Standing on Running Conveyor InaccordancewiththewarehousesafetyregulationâConveyorOperationComplianceâ: Whiletheconveyorisrunning,beingonitssurfaceisstrictlyprohibitedtoprevententanglementandinjury;formaintenance,the conveyormustbestopped,poweredoff,andlockedout,andpersonnelmayenteronlyafterconfirmingitisstationary. Figure 8 | Definitions of 20 risk in potential risk reasoning. 16 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios equipment operation compliance perimeter access control fire monitoring personnel safety duty a ttribute r ecognition object location counting o bject r ecognition action r ecognition o bject status color m oving direction consistency quality style scene attribute comparison interaction f ine - grained p erception [single - instance] f ine - grained p erception [cross - instance] corse- grained perception w arehouse commonsense s patial relation p hysical property object fu nction operator role o perator role common sense spatial relation fire monitoring personal protective equipment elevated work #1 elevated work #2 elevated work #3 key post attendance conveyor operation compliance forklift driving compliance #1 forklift driving compliance #2 forklift driving compliance #3 forklift driving compliance #4 forklift driving compliance #5 forklift driving compliance #6 forklift driving compliance #8 forklift driving compliance #7 forklift driving compliance #9 dock loading & unloading #1 dock loading & unloading #2 dock loading & unloading #3 warehouse access control Figure 9 | Full task taxonomy of LogiScope-VQA. System Prompt Role: Expert Logistics Security Image Editor. Task: Realistic modification for safety AI training. Strict Constraints: 1. Do NOT change the background warehouse environment. 2. Seamlessly blend the person with lighting and shadows. 3. Ensure physical plausibility (weight, contact points). 4. High resolution, photorealistic style. User Prompt Editing Instruction: Detect the person and edit to simulatea high-risk safety violation. Target Scenario: A worker is climbing barehanded up a fullyloaded racking shelf, one footon a horizontal beam, the otherfoot stepping upward, both hands gripping the upright column. Visual Details: The worker is back facing the camera,wearing a blue warehouse uniform, and is wearing a yellow hard hat but NO safety harness. Positioning: The worker is in the distance, at a high positionabove the ground. Height >2 meters from the ground. System Prompt [BaseConstraints] [EditInstruction] User Prompt [ImageDescription] [TargetScene(action)] [VisualDetails(orientation, clothing, safety gear)] [ToolDefinition(tool)] [PositionRequirement(position)] Figure 10 | Prompt template and example for targeted image editing. 17 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios we applied rule-driven linguistic refinement to enhance naturalness and precision. This refinement operation served two purposes. It synthesized multi-attribute constraints into natural pre-nominal modifiers to disambiguate and precisely localize target objects (e.g., refining âpersonâ with attributes to âthe woman in a blue upper garment in frame 3â). Meanwhile, it made minimal grammatical adjustments (e.g., quantifiers, prepositions or verbs) only when necessary, thereby preserving the structural integrity of the majority of queries. Question-Answering Annotation We performed Question-Answering Annotation to assign ground- truth answers and simultaneously filter out low-quality QA pairs. This process involved four specialized annotators and totaled 40 person-days of effort at a cost of $1,000. In the first round, two specialists divided the dataset (containing over 7,000 visual samples) equally and independently annotated their respective halves, producing initial labels for each VQA pair. During this round, they explicitly discarded VQA pairs with ambiguous referents, unnatural phrasing, or missing correct options, retaining only well-formed and verifiable pairs (corresponding to 5,394 final visual samples) for subsequent verification. In the second round, one senior expert independently re-annotated every VQA pair and compared the result with the first-round label. If the labels matched, the pair was tentatively accepted; if they disagreed, the expert re-evaluated the pair to adjudicate the discrepancy, either upholding the initial label or correcting it. In the third round, VQA pairs with consistent labels from the previous two rounds underwent a rapid check, while significant attention was devoted to verifying those pairs with prior disagreements. This intensive cross-validation ensured the precision of the final ground-truth answers. Dataset Assembly As is common in real-world scenarios, logistics safety surveillance data also exhibited a long-tail distribution, where the occurrences of core objects and safety hazards were highly uneven. To achieve a trade-off between diversity and balance, we rebalanced the VQA pairs across multiple dimensions, such as hallucination proportions, core object types and hazard categories, ultimately yielding LogiScope-VQA. D.2. Detailed Risk Definitions We provide precise definitions for 20 specific risks in Fig. 8, which are involved in the potential risk reasoning tasks. D.3. Full Task Taxonomy Fig. 9 presents the full task taxonomy. D.4. Prompt for Targeted Image Editing As illustrated in Fig. 10, our prompt template for targeted image editing employs multiple placeholders, accompanied by a concrete example demonstrating its usage. The diversity of resulting prompts stems from the rich set of allowable values for each placeholder, which we comprehensively enumerate in Fig. 11. 18 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios ElementAvailable Value action â Climbing barehanded up a loaded rack, feet on beams, hands gripping column, with no climbing aids nearby âĄStanding on scissor lift guardrail edge, leaning out, one hand holding rail while other reaches for distant goods â˘Standing on electric forklift tines without cage, one hand holding mast, other searching for items on shelf âŁStanding on top A-ladder step above safety line, body overreaching laterally to grab goods from adjacent racking â¤Straddling racking beam with legs dangling, one hand gripping column, other retrieving item from adjacent location âĽstanding on top of the operator cabin of an electric forklift, leaning toward the racking to inspect goods on the shelves âŚstanding on a handcart taller than a person's height, reaching for goods on the shelf â§standing on a wheeled mobile ladder platform with casters, retrieving goods from the shelf â¨standing alone on the platform of a single-occupancy order picker (man-up forklift) âŠstanding side by another worker, overcrowding the platform of a single-occupancy order picker (man-up forklift) âŞstanding inside a bright yellow, grid-walled inventory cage mounted directly onto the forks of an electric forklift âŤstanding on a high-level order picker vehicle, inspecting goods on the racking shelves âŹstanding side by another worker, overcrowding the cab of the same order picker vehicle âstanding on top of stacked cargo boxes, reaching for goods on the shelf âŽusing three empty pallets stacked vertically as a makeshift step ladder, standing on the top pallet and tiptoeing to retrievean item âŻstanding on the conveyor belt of an operating production line â°climbing over an operating production line conveyor belt âąa truck reversing toward the loading dock from 2-3 meters away, a worker standing directly behind the truck between the dock and the truck, only upper body visible, unaware of the reversing truck, at risk of being crushed â˛a truck reversing toward the loading dock from 2-3 meters away, a worker passing through the gap between the dock and the truck, only upper body visible, at risk of being crushed âła truck moving forward, a worker chasing the truck from behind to close the rear cargo door, at risk of being crushed orientation â back facing the camera âĄside profile facing left â˘side profile facing right âŁslightly looking up â¤facing the camera with head down âĽlooking down at feet âŚside profile facing left, leaning forward slightly â§side profile facing right, leaning backward slightly â¨squatting position facing left âŠsquatting position facing right âŞstanding with hands on hips âŤstanding with one hand on waist âŹbody tilting slightly to the left âbody tilting slightly to the right âŽhead tilted upward looking up âŻhead down focused on the ground clothing â wearing a blue warehouse uniform âĄwearing a yellow reflective vest â˘wearing an orange safety jacket âŁwearing gray workwear â¤wearing ablack hoodie and a fluorescent yellow reflective vest âĽwearing a blue reflective vest safetygear â wearing a yellow hard hat but NO safety harness âĄwearing a red hard hat with safety harness unhooked (dangling) â˘wearing both a hard hat and a safety harness âŁwearing a safety harness but NO hard hat, head fully exposed â¤without any hard hat, head fully exposed, and NO safety harness on body tool â electric forklift âa common logistics transport vehicle with two forks at the front for moving palletized goods; open-type driver cabin; metal mesh protective roof over the cabin âĄhandcart âa manually pushed logistics handling tool, usually with two or four wheels, similar to a large supermarket shopping cart â˘scissor lift âa logistics lift with an X-shaped hydraulic scissor mechanism and four-sided guardrailson all four sides âŁmobile ladder âa non-folding, straight wheeled ladder with guardrail handholds and castersat the bottom â¤A-frame ladder âa foldable A-shaped ladder, without wheels âĽracking shelf âa steel structure for 3D storage, composed of upright columns, horizontal beams and shelves, for multi-layer stacking of pallets or turnover boxes âŚorder picker âa dedicated industrial vehicle for order picking, with an operator platform and linked lifting device; the rideris the driver, no dedicated cabin, has a frame for safety harness, lifting column, and large motor, no forks â§cargo âgoods, packages or items to be transported, stored or handled in logistics, usually in cardboard or plastic boxes on pallets â¨empty pallet âa standard pallet without cargo, usually made of wood, plastic or metal, for subsequent loading and forklift handling âŠinventory cage âa cage with four-sided protective mesh mounted on forklift forks, with insertion holes at the bottom for forks, about half-person height to waist level âŞconveyor belt âa continuous transport system in logistics for moving goods along a production line âŤloading dock âthe height-difference platform between the warehouse floor and the truck parking area âŹtruck âa cargo vehicle parked at the loading dock area for loading and unloading position â far end, very high position, height above ground>4m âĄnear end, very high position, height above ground >4m â˘middle position, very high position, height above ground >4m âŁfar end, high position, height above ground >2m â¤near end, high position, height above ground >2m âĽdock area below (truck parking zone) âŚfar end, low position, height above ground â¤2m â§near end, low position, height above ground â¤2m â¨middle position, low position, height above ground â¤2m âŠconveyor belt marked area (indicated by red bounding box) Figure 11 | Values of elements for targeted image editing. 19 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios System Prompt YouarealogisticssafetyVQAassistant.Answerquestionsstrictlybasedonobservablevisualevidence.Logical reasoningispermitted,butdonotfabricatedetailsunsupportedbytheimage. Yourresponsemustconsistofexactlytwoparts,inthisorder: 1.**Reasoning**:Athoroughvisualanalysisandlogicaldeductionprocess. 2.**FinalAnswer**:Theformatofthispartdependsonthequestiontype: -Ifthequestionprovidesmultiple-choiceoptions(e.g.,A,B,C,D),outputaJSON-formattedlistcontaining theindices/lettersofallcorrectanswers(e.g.,["A"]or["A","C"]). -Ifthequestionisopen-ended(nooptionsprovided),outputaconcisetextualresponseusingprecisesafety terminologythatdirectlyaddressesthequery. Donotincludeanyadditionalcommentary,explanations,orformattingoutsidethesetwocomponents.Ensure thetransitionbetweenReasoningandFinalAnswerisclear. User Prompt [Question] A) [Option A] B) [Option B] [etc.] System Prompt Youarearigorousshort-answergradingexpert.Yourtaskistodeterminewhetherthe[StudentAnswer]is semanticallyequivalenttoorfullycoversthekeypointsofthe[ReferenceAnswer]. GradingRules: Pleasemakeyourjudgmentbasedonthefollowing: 1.KeyPointVerification:The[StudentAnswer]mustcontainallcriticalinformationpointsfromthe [ReferenceAnswer].Synonymsubstitution,sentencerestructuring,orreasonablesummarizationisacceptable. 2.FactualConsistencyCheck:The[StudentAnswer]mustnotcontainanyfactualerrorsthatcontradictthe [ReferenceAnswer].Ifanyfactualerrororlogicalcontradictionexists,assignascoreof0immediately. 3.HandlingRedundantInformation:Ifthe[StudentAnswer]includesadditionalcorrectinformationbeyond thescopeofthe[ReferenceAnswer],itshouldbeignoredandshallnotaffectthescore. 4.MissingPointDetermination:Ifanysinglekeypointisomitted,assignascoreof0,evenifallotherparts arecompletelycorrect. OutputFormat: -Outputonlyasingledigit:1forcorrect(allkeypointscoveredwithnoerrors),0forincorrect(missingkey points,factualerrors,orcontradictions). -Noexplanations,punctuationmarks,linebreaks,oranyothertext. UserPrompt [Original Question]: [question] [Reference Answer]: [answer] [Student Answer]: [pred] (a) Prompt for solving questions (b) Prompt for evaluating open-ended questions Figure 12 | Prompts for solving and evaluating questions. 20 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios E. Evaluation Details E.1. More Decoding Settings Fig. 12 illustrates two evaluation prompt templates. The left one is designed for solving multi-choices and open-ended questions, while the right one is utilized in the LLM-as-a-Judge workflow for a third- party LMM to assess the correctness of the modelâs responses (outputting âcorrectâ or âincorrectâ). In this judging workflow, the ground truth for short-answer questions typically comprises several core key points. A response is judged as âcorrectâ if it contains all these essential elements, regardless of any additional elaboration; conversely, the omission of any single key point results in an âincorrectâ classification. This zero-tolerance policy for omissions is designed to prioritize high recall in hazard detection, given the severe consequences of missed risks in industrial environments. Such a rigorous criterion serves as a stringent test, effectively validating a modelâs true capability and reliability in logistics safety surveillance. For improved transparency and reproducibility, Tab. 4 includes links to official documentation or model checkpoints of the evaluated LMMs, along with detailed inference configurations and the evaluation timeframe. E.2. Human Evaluation Protocol To obtain comprehensive human performance, we recruited two distinct groups of participants with contrasting expertise levels. The first group consisted of 10 undergraduate student volunteers who served as generalist evaluators. While possessing strong general common sense and basic visual recognition capabilities, they lacked specialized knowledge in logistics operations or safety protocols. These students collectively covered the entire benchmark by answering approximately 1,000 questions each over a period of two weeks, receiving a stipend of $15 per person. The second group comprised a single professional domain expert with at least four years of hands-on experience in logistics safety and surveillance. Unlike the generalists, this expert leveraged deep domain knowledge (e.g., familiarity with industrial equipment and safety protocols) to independently answer all questions in the benchmark. This expert evaluation also spanned two weeks and incurred a total cost of $700. The results labeled âNovice Studentâ and âLogistics Expertâ in Tab. 1 correspond to the performance of the student volunteers and logistics expert, respectively. E.3. Reliability of LLM-as-Judge Evaluation A reliable LLM-as-Judge evaluation must be both stable and aligned with human expert judgments. We therefore randomly sampled 100 questions from the open-ended VQA set and examined the judge along the following two dimensions: â˘Intra-judge: The same LLM judge evaluates each question 3Ă with temperature = 0.7. We report the agreement rate across the three runs. ⢠Inter-judge: Human experts independently judge the same subset. We report the agreement rate between the LLM judgeâs decisions and the human judgments. Table 3 | Stability and human alignment of LLM-as-a-Judge evaluation on 100 VQA pairs. DimensionMetricPACFMPSDEOC Overall Intra-judge Agreement rate 92.9% 95.1% 93.8% 91.6% 93.3% Inter-judge Agreement rate 85.3% 89.4% 87.9% 85.1% 87.2% 21 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios As illustrated in Tab. 3, intra-judge repeatability exceeds 93% and the LLM judgeâs agreement with human experts exceeds 87%. These results prove that the LLM judge is stable and aligned with human judgment. F. Visualization To facilitate a more intuitive understanding, we provide additional visualization examples. Fig. 13 displays visual-only examples (raw images). 22 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios (a) Real video: dock operations area (c) Real video: bulk storage area (g) Synthetic video: dock operations area (h) Synthetic video: conveyor area (e) Real video: high rack area (f) Real video: security screening area (d) Real video: nighttime scene (b) Real video: storage area (j) Synthetic images (i) Real images Figure 13|Examples of real and synthetic visual data, covering both daytime and nighttime scenes across various zones within logistics warehouses. 23 LogiScope-VQA : Benchmarking VLMs for Logistics Hazard Identification in Industrial Scenarios Table 4 | Inference configurations for proprietary (purple) and open-source (green) LMMs. GPT-5.4 & GPT-5.5 Link: https://openai.com/index/introducing-gpt-5-5/ Temperature: 0.2 | Max new tokens (think/non-think): 4096 / 256 | Eval date: June 30-July 15, 2026 Claude-Sonnet-4.6 & Opus-4.7 Sonnet Access: https://w.anthropic.com/claude/sonnet Opus Access: https://w.anthropic.com/claude/opus Temperature: 0.2 | Max new tokens (think/non-think): 4096 / 256 | Eval date: June 30-July 15, 2026 Gemini-3.1-Pro Link: https://deepmind.google/models/model-cards/gemini-3-1-pro/ Temperature: 0.2 | Max new tokens (think/non-think): 4096 / 256 | Eval date: June 30-July 15, 2026 Qwen3.7-Plus Link: https://qwen.ai/blog?id=qwen3.7-plus Temperature: 0.2 | Max new tokens (think/non-think): 4096 / 256 | Eval date: June 30-July 22, 2026 GLM-5.2 & GLM-4.7 Link: https://huggingface.co/zai-org/GLM-5.2 Max len: 32í | Batch size: 1 | backend: Pytorch Temperature: 0.2 | Max new tokens (think/non-think): 4096 / 256 | Eval date: July 5-22, 2026 MiMo-VL-7B-RL Link: https://huggingface.co/XiaomiMiMo/MiMo-VL-7B-RL Max len: 32í | Batch size: 1 | backend: Pytorch Temperature: 0.2 | Max new tokens (think/non-think): 4096 / 256 | Eval date: July 10-15, 2026 Kimi-K2-Thinking & Kimi-K2.6 Link: https://huggingface.co/moonshotai/Kimi-K2.6 Max len: 32í | Batch size: 1 | backend: Pytorch Temperature: 0.2 | Max new tokens (think/non-think): 4096 / 256 | Eval date: June 25-July 22, 2026 InternVL-3.5 Link: https://huggingface.co/OpenGVLab/InternVL3_5-8B Max len: 32í | Batch size: 1 | backend: Pytorch Temperature: 0.2 | Max new tokens (think/non-think): 4096 / 256 | Eval date: July 10-20, 2026 LLaVA-v1.6-7B Link: https://huggingface.co/llava-hf/llava-v1.6-mistral-7b-hf Max len: 32í | Batch size: 1 | backend: Pytorch Temperature: 0.2 | Max new tokens (think/non-think): 4096 / 256 | Eval date: July 10-20, 2026 Qwen Open-Source Suite Models: Qwen3.5-Plus, Qwen3.6-Flash, Qwen3-VL (8B, 30B-A3B, 32B, 235B-A22B, Plus) Qwen3 Access: https://huggingface.co/collections/Qwen/qwen3 Qwen3.5 Access: https://huggingface.co/collections/Qwen/qwen35 Qwen3.6 Access: https://huggingface.co/collections/Qwen/qwen36 Max len: 32í | Batch size: 1 | backend: Pytorch Temperature: 0.2 | Max new tokens (think/non-think): 4096 / 256 | Eval date: June 25-July 22, 2026 24