Paper deep dive
Graph-Based Recognition of Simulated Train-Driver States From Facial and Upper-Body Keypoints
Olivia Nocentini, Marta Lagomarsino, Gokhan Solak, Younggeol Cho, Qiyi Tong, Sara Zeynalpour, Marta Lorenzini, Alessandro Ledda, Arash Ajoudani
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 10/7/2026, 8:38:09 AM
Summary
This study presents a vision-based monitoring system for recognizing simulated train-driver states (alert, not-alert, and emergency) using a single front-facing RGB camera and a Graph Neural Network (GNN). The system extracts facial and skeletal keypoints to construct a whole-body graph, processed by a custom Directed-Graph Neural Network (DGNN). An ablation study compared skeletal-only, facial-only, and combined feature configurations, finding that combining facial and skeletal features yielded the highest accuracy (81% for three-class, 99% for binary alert/not-alert under light conditions). The authors also introduced a controlled RGB video dataset with 25 participants under three illumination conditions.
Entities (16)
Relation Signals (11)
Directed-Graph Neural Network → subclassof → Graph Neural Network
confidence 98% · This graph is then processed by a custom Directed-Graph Neural Network (DGNN)
Mediapipe → usedforextraction → Skeletal Keypoints
confidence 97% · From each video frame, we estimated the position of the upper body, employing the first 17 keypoints of Mediapipe.
Mediapipe Face Mesh → usedforextraction → Facial Keypoints
confidence 97% · we integrated an additional 478 facial keypoints from Mediapipe’s Face Mesh
Facial Keypoints → combinedwith → Skeletal Keypoints
confidence 96% · combining facial and skeletal features yields the highest accuracy
Bi-Temporal Convolution → partof → Directed-Graph Neural Network
confidence 95% · The Bi-Temporal Convolution (BiTC) layer then applies temporal convolutions
Graph-Temporal Convolution → partof → Directed-Graph Neural Network
confidence 95% · The model consists of two Graph-Temporal Convolution (GTC) blocks
Graph Neural Network → usedfor → Train-Driver States
confidence 95% · This study presents a vision-based monitoring system that relies solely on a single front-facing RGB camera and a graph neural network to classify simulated train-driver states
Ablation Study → →
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Driver fatigue poses a significant challenge to railway safety, with traditional systems like the dead-man switch offering limited and basic alertness checks. This study presents a vision-based monitoring system that relies solely on a single front-facing RGB camera and a graph neural network to classify simulated train-driver states into alert, not-alert, and an emergency class comprising acted emergency-like behaviours. To optimize input representations for the model, an ablation study was performed, comparing three feature configurations: skeletal-only, facial-only, and a combination of both. Experimental results show that combining facial and skeletal features yields the highest accuracy (81%) for the three-class model under the light condition, outperforming models that use only facial or skeletal features. Furthermore, the combination of facial and skeletal features achieves 99% accuracy in the alert/not alert classification in light condition. Additionally, we introduced a controlled RGB video dataset containing alert, not alert, and acted emergency-like behaviours recorded under three illumination conditions. These contributions represent a step toward passive and non-contact train-driver state recognition based on facial and upper-body dynamics.
Tags
Links
- Source: https://arxiv.org/abs/2610.07083v1
- Canonical: https://arxiv.org/abs/2610.07083v1
Trouble viewing inline? Open PDF directly →
Full Text
51,217 characters extracted from source content.
Expand or collapse full text
Date of publication x 00, 0000, date of current version x 00, 0000. Digital Object Identifier 10.1109/ACCESS.2024.0429000 Graph-Based Recognition of Simulated Train-Driver States From Facial and Upper-Body Keypoints OLIVIA NOCENTINI 1 , MARTA LAGOMARSINO 1 ,GOKHAN SOLAK 1 ,YOUNGGEOL CHO 2 ,QIYI TONG 1 , SARA ZEYNALPOUR 1 , MARTA LORENZINI 1 , ALESSANDRO LEDDA 3 , AND ARASH AJOUDANI 1 1 Human-Robot Interfaces and Interaction Laboratory, Istituto Italiano di Tecnologia, Via S.Quirico 19d, Genoa, Italy 2 Department of Bioengineering, Sir Michael Uren Hub Imperial College London 86 Wood Lane London W12 0BZ United Kingdom 3 Department of Technological Innovations and Safety of Plants, Products and Anthropic Settlements, Italian National Institute for Insurance against Accidents at Work (INAIL), Via R.Ferruzzi 38, Rome, Italy Corresponding author: Olivia Nocentini (e-mail: olivia.nocentini@iit.it). This work was supported by the Italian Workers’ Compensation Authority INAIL within the VIVA project. ABSTRACT Driver fatigue poses a significant challenge to railway safety, with traditional systems like the dead-man switch offering limited and basic alertness checks. This study presents a vision-based monitoring system that relies solely on a single front-facing RGB camera and a graph neural network to classify simulated train-driver states into alert, not-alert, and an emergency class comprising acted emergency-like behaviours. To optimize input representations for the model, an ablation study was performed, comparing three feature configurations: skeletal-only, facial-only, and a combination of both. Experimental results show that combining facial and skeletal features yields the highest accuracy (81%) for the three-class model under the light condition, outperforming models that use only facial or skeletal features. Furthermore, the combination of facial and skeletal features achieves 99% accuracy in the alert/not alert classification in light condition. Additionally, we introduced a controlled RGB video dataset containing alert, not alert, and acted emergency-like behaviours recorded under three illumination conditions. These contributions represent a step toward passive and non-contact train-driver state recognition based on facial and upper-body dynamics. INDEX TERMS driver fatigue monitoring, vision-based monitoring systems, behaviour-based systems, face and skeletal features, graph neural network, railway safety I. INTRODUCTION F ATIGUE is a critical safety concern in railway opera- tions, where long shifts and repetitive activities can sig- nificantly impair a driver’s alertness [1]. Despite regulations aimed at ensuring adequate rest for train drivers, fatigue- related incidents remain alarmingly common. According to research by the A Charitable Trust, one in eight drivers admits to falling asleep at the wheel, while nearly two- fifths have felt so tired that they feared they might nod off [2]. The risk is particularly high with modern automated trains operating on night shifts and monotonous routes [3]. While drivers remain essential for supervision, automation reduces their active engagement for long stretches, increasing the likelihood of inattention and drowsiness [4]. Fatigued drivers react more slowly to hazards and, in extreme cases, may fall asleep. Nighttime operation further heightens this risk because drowsiness typically increases during late-night hours [4]. Given these risks, effective fatigue management in railway operations is essential. A traditional safety mechanism in railway operations is the dead-man switch system, which requires the driver to press a pedal periodically to confirm alertness. If the pedal is not pressed within a specified time frame, an alarm is triggered, and the train may come to a halt. While this system ensures a basic level of vigilance, it also imposes a cognitive and physical burden on the driver, potentially leading to discomfort and operational inefficiency [5], [6]. To address this problem, increasingly advanced tech- nologies for drowsiness detection have been developed and can be grouped into three broad categories. Vehicle-based VOLUME 11, 20231 arXiv:2610.07083v1 [cs.CV] 5 Oct 2026 Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS systems infer alertness by monitoring driving performance indicators such as steering behavior and braking patterns [7]. Physiological-based systems use wearable sensors to capture signals like heart rate variability, muscle activity, and brainwaves [8]. Finally, behaviour-based systems rely on computer vision to analyze facial expressions, eye move- ments, head pose, yawning frequency, and skeletal posture [3]. The first approach primarily relies on vehicle driving data, including steering angle, speed, and acceleration. However, the effectiveness of such methods is limited in railway operations, where drivers interact with buttons and levers intermittently rather than continuously, as with car steering, making it difficult to assess alertness based on control inputs alone. In contrast, physiological signal-based methods provide a more direct and accurate assessment of the driver’s phys- ical condition. Despite their effectiveness, physiological measurement techniques often suffer from practical limi- tations. Conventional equipment often presents wearability constraints and is highly susceptible to signal noise, while medical-grade devices remain costly and less feasible for large-scale deployment. Given these limitations, behaviour-based monitoring has become a widely investigated alternative, as it enables pas- sive and non-contact observation of the driver without re- quiring wearable instrumentation or continuous access to vehicle-control signals. A substantial body of research has investigated vision-based driver fatigue and drowsiness mon- itoring [9]–[14]. To address the ethical and safety constraints of real-world road experiments, Wang et al. [10] developed a virtual driving system based on the CARLA simulation platform [15]. This system integrates a force-feedback device to simulate real driving interactions and employs a monocular camera to capture indicators such as Eye Aspect Ratio (EAR) [16], [17], Mouth Aspect Ratio (MAR) [18], and Percentage of Eyelid Closure Over the Pupil Over Time (PERCLOS) [19]. Using these features, a fatigue state classification model based on Random Forest was established, achieving high classification accuracy in the reported experiments. Albadawi et al. [20] developed a real-time drowsiness detection sys- tem using computer vision and machine learning to monitor operators’ eye patterns via a simple webcam. The system analyzes blink patterns based on neuroscience literature to assess alertness levels. It employs machine learning mod- els, specifically, Multilayer Perceptron, Random Forest, and Support Vector Machines (SVM), to process consecutive video frames. In their paper, the output of the network is represented by two classes (awake or asleep). In [21], the authors developed a Driver Assistance System focused on detecting driver drowsiness to prevent traffic accidents. The system analyzes 60-second sequences of facial images to assess alertness. Two approaches were tested: Recurrent and Convolutional Neural Network (R-CNN) and Deep Learn- ing with Fuzzy Logic to classify if the driver is awake or not awake. In [22], the authors developed a real-time, non- intrusive driver drowsiness detection (D) system using facial analysis and machine learning techniques. The study evaluates various algorithms, including K-Nearest Neighbors (KNN), SVM, convolutional neural networks (CNNs), and advanced computer vision models like YOLOv5, YOLOv8, and Faster R-CNN and output if the person is awake or not. However, the aforementioned approaches mostly focus on facial keypoints, neglecting the skeletal component, which provides valuable and stable information on train driver alertness. Notably, most existing studies have focused on passenger cars and freight vehicles, while research on fatigue detection in railway transportation remains relatively limited. However, the conditions and challenges of alertness detection for train drivers differ from those in road transportation. Unlike car drivers, who maintain relatively stable postures with their hands on the steering wheel, train operators are seated on spring-mounted chairs that introduce head oscillations due to track-induced vibrations. They also perform wider head movements, such as scanning long stretches of track, check- ing side panels and control screens, and monitoring signals outside the cabin. Compared with road-vehicle drivers, train operators exhibit different movement amplitudes and tem- poral patterns because they periodically inspect the track, side panels, control displays, and external signals rather than continuously steering a vehicle. Among the relatively limited studies addressing alertness monitoring in railway transportation, Avizzano et al. [3] developed an embedded computer vision system for train driver drowsiness detection. The system relies on a single camera installed in the cockpit to monitor gaze direction and eyelid blinking as visual indicators of reduced alertness. Moreover, another work, [23], introduced a multi-feature fatigue detection method designed for railway dispatchers, combining facial keypoints with body posture analysis to im- prove detection accuracy, even under challenging conditions such as facial occlusion or angle variations. This approach considers features such as eye closure duration, blinking patterns, yawning frequency, and posture-related behaviors to assess fatigue levels. The model employs an adaptive Bi-LSTM-SVM algorithm to integrate facial and behavioral data, outputting a three-level classification of fatigue. Beyond fatigue-oriented monitoring, Hayashi et al. [24] proposed a multimodal abnormal-sign detection system for professional road drivers. Their framework combines body and facial images with an ear-mounted pulse sensor, steering- angle measurements, and voice interaction. Separate modules detect a predefined set of cue signs, including touching the head or chest, one-arm hanging, prolonged eye closure, abnormal heart rate, and absence of steering input. The type, duration, and recurrence of the detected cues are subse- quently aggregated through a rule-based confidence level. Depending on this confidence level, the system can continue monitoring the driver, initiate a voice interaction to obtain additional information about the driver’s internal state, or trigger an intervention. Its system-level quantitative valida- 2VOLUME 11, 2023 Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS tion mainly addressed sleepiness, whereas pseudo-headache was presented as an illustrative simulated scenario rather than as a clinically validated medical condition. Their work demonstrated that driver monitoring has al- ready been extended beyond conventional fatigue detection to include behavioural and physiological cues potentially related to medical impairment. However, their system relies on multiple inputs, including visual data, a contact-based physiological sensor, vehicle-control information, and active voice confirmation. Although this multimodal configuration can provide richer information about the driver’s internal condition, it requires physical sensor placement, access to vehicle signals, and integration of multiple acquisition and interaction modules. In contrast, the present work investigates a complementary design trade-off by prioritising passive, non-contact, and non- wearable vision-based monitoring that relies on both facial and skeletal features. The proposed framework requires only a single front-facing RGB video stream and does not depend on wearable physiological sensors, train-control telemetry, or an explicit verbal response from the operator. Therefore, no device needs to be attached to the train driver during operation. This reduced sensing burden may facilitate the integration of the monitoring system into existing train cabins while limiting interference with the operator’s normal activ- ities. Moreover, rather than detecting independently defined cue signs and combining them through handcrafted rules, the proposed method directly learns the joint spatiotemporal evo- lution of facial and upper-body landmarks within a unified graph representation. Accordingly, the remaining research gap is the limited availability of railway-specific approaches that perform di- rect state recognition using only non-contact visual ob- servations. Moreover, existing railway-monitoring methods predominantly address binary awake–asleep classification or different levels of fatigue. Limited attention has there- fore been devoted to passive train-driver monitoring sys- tems that jointly model facial and upper-body dynamics and distinguish reduced alertness from acted emergency-like behaviours using a single RGB video stream. A further challenge is introduced by the operating con- ditions of railway vehicles. Train drivers may be exposed to abrupt illumination changes when entering or leaving tunnels, moving between outdoor and underground sections, or operating during nighttime conditions. Such variations can affect the reliability of facial and skeletal landmark es- timation and, consequently, the performance of downstream vision-based state-recognition models. This issue is partic- ularly relevant for a vision-only framework and therefore requires explicit evaluation under both stable and changing illumination conditions. Motivated by these considerations, this paper presents a graph-based vision framework for recognising train-driver states from the joint temporal evolution of facial and upper- body landmarks. The problem is formulated as a three-class classification task comprising alert, not alert, and emergency states. The latter class includes controlled behaviours such as pretending to faint, holding the head, or clutching the chest. The main contributions of this work are summarised as follows: 1) We propose a passive, non-contact, non-wearable, and vision-only train-driver monitoring framework based on a single front-facing RGB video stream and rely- ing on a graph neural network. The proposed system requires neither physiological instrumentation, train- control telemetry, nor active responses from the oper- ator. 2) We formulate train-driver monitoring as a direct three- class state-recognition problem comprising alert, not alert, and an acted emergency-like class. 3) We introduce a preliminary controlled RGB video dataset involving 25 participants and three driver-state classes recorded under three railway-relevant illumina- tion settings. The changing-illumination condition was designed to reproduce the visual variations that may occur when a train enters or exits a tunnel. 4) We evaluate the complementary contribution of facial and upper-body information through an ablation study comparing facial-only, skeletal-only, and combined face-and-upper-body graph representations in the three- class scenario. In addition, an occlusion-sensitivity anal- ysis is performed to investigate which facial and skeletal landmarks have the greatest influence on the model’s predictions. I. METHODOLOGY Figure 1 illustrates the overall flowchart of our graph-based monitoring framework for online train driver state assess- ment, analyzing whole-body 1 motion only using RGB im- ages. Facial and skeletal keypoints are extracted from each video frame to construct a graph. This graph is then processed by a custom Directed-Graph Neural Network (DGNN), which classifies the driver’s state into one of three categories: alert, not alert, or potentially experiencing an emergency condition. We collected a preliminary custom dataset of RGB videos simulating train driver operations under three common rail- way lighting conditions: light, dark, and sudden light/dark (see Section I-B for more details). This dataset enabled us to train the network on more realistic scenarios and conduct an ablation study alongside a model explainability analysis. These analyses aim to understand the relevance of different facial and skeletal keypoints in the network’s decision pro- cess and assess its sensitivity to keypoint occlusion. This paper proposes a rigorous pipeline to address the following research questions (RQs): RQ1. Can a vision and graph-based neural network effec- tively detect simulated train driver alertness and emer- gency states? 1 Throughout this paper, the term whole-body is used to present the combination of facial and skeletal features. VOLUME 11, 20233 Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS Mediapipe Face Mesh Skeleton Mediapipe Whole-body Graph Definition Alert Not alert Emergency Directed Graph Neural Network Graph with keypoints and edges Face keypoints Skeleton keypoints OUTPUT INPUT GTC DGN Block BatchNormalization 1D BiTemporal Convolution Dropout Global Pooling Fully Connected FIGURE 1. Flowchart of our monitoring system framework. The inputs are the images acquired by an RGB camera, from which facial and skeletal features are extracted. These keypoints are used to construct a whole-body graph, which is then processed by a custom DGNN to classify the user’s state into three categories: alert, not alert, or potentially experiencing an emergency condition. RQ2. Does considering whole-body movements improve driver state detection performance compared to using only facial or skeletal keypoints in the three-class sce- nario? A. ESTIMATION OF FACIAL AND SKELETAL KEYPOINTS IN VIDEO CAPTURES AND WHOLE-BODY GRAPH DEFINITION From each video frame, we estimated the position of the upper body, employing the first 17 keypoints of Mediapipe. We took only the upper part of the skeleton of Mediapipe since the train driver is usually seated on a chair in front of a dashboard. Since Mediapipe provides only few facial keypoints (corresponding to the positions of the left and right eyes, the ears, and the mouth), we integrated an additional 478 facial keypoints from Mediapipe’s Face Mesh [25]. This allowed us to better capture facial contours, eyebrows, eye openness, and mouth movements, which the skeleton of Mediapipe alone cannot detect. We selected the skeleton of Mediapipe and Mediapipe Face Mesh for features extraction because both are stable feature detectors for what concerns RGB images [26], [27]. More- over, since some keypoints were not consistently detected during the extraction process from the RGB videos, a mask- ing strategy was introduced to handle missing keypoints. Once the keypoints are extracted, we construct a graph to represent the whole-body posture and movements of the user. More specifically, facial and upper-body keypoints, represented by their 2D coordinates (x, y), serve as graph nodes, while bones define the edges that connect them. The keypoints are normalized with respect to the nose position across all three scenarios (face-only, skeleton-only, and face- and-skeleton keypoints). In Figure 2, samples of the graph defined by connecting the normalized keypoints through edges are reported for the three tested states: alert, not alert, and the simulated emergency. These graphs represent the input of our network. B. NEURAL NETWORK FOR ALERT, NOT ALERT, AND EMERGENCY STATE CLASSIFICATION 1) Network architecture The proposed architecture is a customised Directed Graph Neural Network (DGNN) that took inspiration from [28], designed to process structured spatiotemporal data by jointly modeling graph-based dependencies and temporal dynamics. The model consists of two Graph-Temporal Convolutional (GTC) blocks, each integrating a Directed Graph Network (DGN) block for relational learning and a Bi-Temporal Con- volution (BiTC) layer for temporal feature extraction. Each DGN block updates node and edge features based on directed graph connectivity matrices, using learned transformations to aggregate information from incoming and outgoing edges. This operation allows the model to capture structured in- teractions within the graph. The Bi-Temporal Convolution (BiTC) layer then applies temporal convolutions along the time dimension to both node and edge features, enhancing the model’s ability to learn temporal dependencies. The input data undergoes an initial batch normalization 4VOLUME 11, 2023 Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS Alert Graph Not Alert GraphEmergency Graph FIGURE 2. Samples of the defined graphs, which correspond to the input of our network, in the three tested users’ states: alert, not alert, and simulated emergency condition. The joints of Mediapipe Face Mesh are shown in red and the Mediapipe skeleton joints are drawn in green. The bones connecting the joints coming from Mediapipe Face Mesh are drawn in yellow, while the other edges related to the skeleton are drawn in orange. step, separately for node and edge features, ensuring stable training. The first GTC block expands the feature dimension, while the second block applies strided temporal convolutions (stride=2), reducing the temporal resolution and enabling hierarchical feature extraction. A dropout layer (p=0.3) is applied after each GTC block to mitigate overfitting. At the final stage, global pooling is applied to the learned node and edge features, followed by a fully connected layer that generates the output predictions. The model employs LeakyReLU activation throughout and uses a Label Smooth- ing Loss to enhance generalization and prevent overconfi- dence in predictions. Our customised model differs from the original implemen- tation of [28] in several key aspects, aimed at improving performance. First, we reduced the number of convolutional layers, creating a more compact architecture tailored to the relatively small size of our dataset. Second, we introduced dropout layers at specific stages to effectively mitigate over- fitting. Third, we replaced certain ReLU activation func- tions with LeakyReLU, allowing the model to better handle negative values and ensuring greater training stability. Ad- ditionally, we incorporated a label smoothing loss function (LabelSmoothingLoss) to enhance robustness when training on noisy datasets, such as ours, which contains inaccuracies in the detected keypoints due to varying lighting conditions during recording. 2) Network hyperparameters The customised DGNN was trained using the following hyperparameters: a learning rate of 0.001, 50 epochs, a batch size of 512, Adam as the optimizer, Label Smoothing of 0.1, and a lookback (i.e., how many frames the neural network looks back into the past) of 30. The hyperparameters were empirically tuned based on validation-set performance and FIGURE 3. The experimental setup consisted of users seated at a desk, following on-screen instructions to manipulate control levers, press dashboard buttons, and check side panels. An RGB camera continuously monitored the subject from the front. inspection of the validation curves. C. RELEVANCE OF FACIAL AND SKELETAL KEYPOINTS IN MODEL DECISION-MAKING 1) Ablation study We conducted an ablation study to assess the impact of different categories of keypoints on the performance of the proposed DGNN for train driver’s state monitoring. Specifi- cally, we trained three network variants using different input graphs, including: (1) whole-body keypoints (i.e., combined facial and skeletal keypoints, to examine their synergistic effect); (2) facial keypoints only, to determine whether the facial expressions and head movements alone suffice for state estimation, as commonly done in the literature; and (3) skele- tal keypoints only, to investigate the importance of upper- body posture and movements in isolation. This approach allowed us to isolate the contributions of facial and skeletal keypoints and determine how their integration enhances the model’s performance. 2) Occlusion Sensitivity Analysis This analysis aims to identify the most significant keypoints for the functioning of the proposed DGNN and accurate prediction generation. To achieve this, we employed the Occlusion Sensitivity Analysis, a method for understanding the importance of individual features for a deep network’s classification. The technique involves occluding (i.e., hiding or removing) one or more keypoints from the input data and observing how this alteration affects the model’s perfor- mance or predictions. If the occlusion of a specific keypoint causes a significant degradation in the model’s predictions, it can be concluded that the keypoint is particularly important for the network’s decision-making process. I. EXPERIMENTAL CAMPAIGN This section details the experimental setup, the dataset, and the analysis performed. A. SETUP The experimental setup (Fig. 3), involved participants seated on a chair in front of a table equipped with a computer VOLUME 11, 20235 Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS monitor and a Realsense D435i camera. This camera was positioned directly in front of the participant to capture RGB videos for dataset collection. The sensor operated at a frame rate of 30 fps, producing images with a resolution of 640×480 pixels. B. DATASET The artificial dataset, used for preliminary validation of our framework, consists of videos carried out at Human-Robot Interfaces and Interaction (HRII) Lab, Istituto Italiano di Tecnologia (IIT), in accordance with the Helsinki Declara- tion. The experimental protocol was approved by the ethics committee Azienda Sanitaria Locale Genovese N.3 (Protocol IIT_HRII_ERGOLEAN 156/2020) and participants were re- cruited from the student body and research staff at IIT. The dataset was divided into training, validation, and test sets, comprising data from 15, 5, and 5 subjects, respectively. In total, the dataset included 25 participants, consisting of 14 males and 11 females, with ages ranging from 23 to 36 years. Of these participants, 12 wore glasses, while 13 did not. For each user, we acquired 9 videos of two minutes and half each: the first 3 of them in which the subject is alert, the second 3 in which the subject is not alert, and the last 3 when the user is pretending to experience an emergency condition. Each condition (alert, not alert and emergency) is recorded in 3 different lighting conditions categorized as light, dark, and sudden light/dark, according to the intensity of the light created in the room. The light condition was defined as the room with 4 lights on, the dark condition was defined as the room without light, and the sudden light/dark condition was when the lights were turned on/off every 15 seconds to simulate entering/exiting a tunnel. During the alert phase, the subject was asked to watch a video created by the authors, depicting a train track view. This was designed to help the participant immerse themselves in the role of a train driver. Additionally, instructions appeared on the screen, guiding the participant to perform specific actions, such as moving levers, pressing buttons on the dashboard, or checking side panels. These prompts encouraged head and body movements, enhancing the system’s robustness to variations in head orientation and positioning relative to the camera. During the not alert phase, subjects were required to keep their eyes closed and pretend to be asleep. Finally, during the emergency phase, participants simulated experiencing a medical, as if suffering from a sudden illness. The emergency class does not represent clinically validated medical events. Rather, it consists of acted emergency-like behaviours (i.e pretending to faint or clutching their chest or putting both of their hands on the head) designed to provide a preliminary proxy class for abnormal driver conditions in a controlled ex- perimental setting. Importantly, across both the not alert and emergency phases, some subjects occasionally covered parts of their faces with their hands, creating self-occlusions. This added variability to the cues and made the scenarios more realistic by reflecting real conditions where facial features may be obscured. C. ANALYSIS 1) Accuracy Accuracy during neural network training measures the pro- portion of correct predictions to the total number of samples, calculated as: Accuracy = Number of Correct Predictions Total Number of Samples × 100%. (1) It evaluates how well the model predicts the true class (ground truth) across the dataset, providing a straightforward metric for performance assessment. Accuracy ranges from 0 to 100% and is particularly useful for classification tasks with balanced class distributions. 2) Confusion Matrices We computed the confusion matrices (i.e., a performance evaluation tool comparing predictions to actual labels) of the proposed DGNN using whole-body keypoints for state monitoring in three different lighting conditions for three and two class classification. Moreover, we computed the confusion matrices also for the only skeleton and only facial scenarios. IV. RESULTS To address the RQ1. and RQ2., we computed the metrics on the 5 test subjects ( accuracy, precision, recall and F1-Score) of the proposed DGNN model for three-class classification. The results using whole-body, facial-only, and skeletal-only keypoints are illustrated in Tables 1, 2, 3, while the per- formance of the alert/not alert classes is presented in Table 4.The best performance is observed when only two classes are considered (alert/not alert), with the test accuracy of 99%. The inclusion of the emergency class reduces accuracy, but the three-class whole-body model still performs well, outperforming models that rely solely on facial or skeletal keypoints. After the whole-body features, the best results are achieved using facial-only features, while skeletal-only features obtain the lowest performance. Moreover, the performance is highest in the full light con- dition, followed by the variable sudden light/dark condition in all the scenarios. To better understand class misclassifications under various lighting conditions, we analyzed the confusion matrices for our three-class model using facial and skeletal keypoints (see Figure 4). The analysis revealed that the emergency state is sometimes misclassified as the not alert condition. A video providing supplementary material for the paper, with a focus on the inference online, is available at this link: https://youtu.be/Cm-8flfn91E. Finally, to further address RQ2. on the individual and com- bined contribution of facial and skeleton keypoints to driver state estimation, we conducted an Occlusion Sensitivity Analysis on the proposed three-class whole-body model. The results, illustrated in Figures 5, show the relative importance 6VOLUME 11, 2023 Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS FIGURE 4. Confusion Matrices for the whole-body graph in the three-class conditions. of each keypoint for correct classification. Each bar in the plot represents a keypoint, with its height indicating its significance in distinguishing between the three driver states. V. DISCUSSION The results of this study highlight that a graph-based neural network using whole-body keypoints analysis was highly ef- fective for binary driver alertness classification and achieved an overall three-class accuracy of 78% for distinguishing among three classes: alert, not alert, and simulated emer- gency condition. Instead, using only skeletal or facial features was insufficient for appropriate three-class classification us- ing DGNN, particularly under the varying lighting condi- tions. The row-normalized confusion matrices provide further in- sight into the classification behaviour of the proposed model across the different lighting conditions. Overall, the model shows a strong ability to discriminate between the alert and not alert classes. In the global setting, 90.7% of the alert samples and 87.6% of the not alert samples are correctly classified, with only limited confusion between these two classes. This suggests that the model is able to effectively capture discriminative behavioural patterns associated with normal alertness and reduced alertness. In contrast, the emergency class represents the most chal- lenging condition to identify. In the global scenario, only 54.1% of the emergency samples are correctly classified, while 34.1% are misclassified as not alert. This misclassi- fication probably occurred because, during data collection, several participants adopted similar whole-body postures in both conditions. For instance, some subjects slumped for- ward or tilted their heads backwards both when experiencing an emergency event and when falling asleep, making it challenging for the network to distinguish between the two conditions. A key limitation of this study is that both the not alert and emergency classes were simulated in a controlled laboratory setup. Therefore, the proposed framework should be interpreted as a preliminary state-classification system for acted scenarios, not as a clinically validated detector of real fatigue or medical emergencies. Future work will require in- cab data, expert-informed scenario design, and real-world validation to assess operational generalization. Furthermore, VOLUME 11, 20237 Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS TABLE 1. Whole-body features metrics in general, light, dark, and sudden light/dark lighting conditions for the three class scenario Light ConditionAccuracy (%)Precision (%)Recall (%)F1-Score (%) Global78787778 Light81828180 Dark75757574 Sudden Light/Dark77767776 TABLE 2. Facial-only features metrics in general, light, dark, and sudden light/dark lighting conditions for the three class scenario Light ConditionAccuracy (%)Precision (%)Recall (%)F1-Score (%) Global75757575 Light78787877 Dark76757675 Sudden Light/Dark72717272 TABLE 3. Skeletal-only features metrics in general, light, dark, and sudden light/dark lighting conditions for the three class scenario Light ConditionAccuracy (%)Precision (%)Recall (%)F1-Score (%) Global61666160 Light64686464 Dark55615554 Sudden Light/Dark63686363 TABLE 4. Whole-body features metrics in general, light, dark, and sudden light/dark lighting conditions for the two-class scenario Light ConditionAccuracy (%)Precision (%)Recall (%)F1-Score (%) Global97979797 Light99999999 Dark94949494 Sudden Light/Dark96969696 the misclassification between emergency and not alert ex- plains why the test accuracy of the two-class model was higher compared to the three-class model (see Tables 4 and Table 1). When considering the different illumination settings, the best performance is obtained in the light condition, where both alert and not alert classes achieve high recognition rates, equal to 94.1% and 93.4%, respectively. The emer- gency class also reaches its highest value in this condition, with 56.2% of correctly classified samples. Conversely, the dark condition leads to the lowest performance, particularly for the emergency class, which decreases to 50.6%. This is because RGB cameras work well under well-lit conditions rather than in darkness. This can be explained not only by the intrinsic limitations of RGB cameras in low-light environments, but also by the fact that the keypoint extraction stage relies on Mediapipe models trained mainly on RGB datasets collected under favorable illumination conditions. Consequently, keypoint detection becomes less reliable in dark scenes, which negatively impacts the downstream clas- sification performance. Future works will investigate the use of thermal cameras to enhance the performance of the network in dark conditions. Finally, in the sudden light/dark condition, the model maintains good performance for the alert class, correctly classified in 91.8% of cases. However, a higher confusion is observed between the not alert and emergency classes, with 17.1% of not alert samples misclassified as emergency and 32.3% of emergency samples misclassified as not alert. This suggests that abrupt illumination changes may introduce additional variability in the extracted features, making the distinction between reduced alertness and emergency states more difficult. The Occlusion Sensitivity analysis showed that in the overall ranking (Figure 5), skeletal keypoints were among the most influential features, particularly keypoint 16, cor- responding to the right wrist, followed by keypoints 11 and 12, corresponding to the left and right shoulders, as well as keypoints 9 and 10, corresponding to the mouth region, and keypoint 15, corresponding to the left wrist. This suggests that the model strongly exploits posture-related information, especially shoulder configuration and hand/wrist position, together with facial-orientation or mouth-related cues, to discriminate the operator’s state. In conclusion, these find- ings indicate that skeletal landmarks related to shoulders and wrists play a central role in the model’s decision process, while facial landmarks provide complementary and more localized information. Regarding the impact of topology, the Occlusion Sensitivity Analysis provides information about the importance of in- dividual nodes but does not directly isolate the contribution of alternative edge configurations. A systematic comparison among different graph topologies, root definitions, and con- nectivity patterns was not performed in the present study and 8VOLUME 11, 2023 Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS 0100200300400500 keypoint index −0.25 0.00 0.25 0.50 0.75 1.00 1.25 Occlusion Value Total Occlusion sensitivity Skeleton (0-16) Face mesh (17-494) FIGURE 5. Results of the Occlusion Sensitivity Analysis revealing the significance of individual keypoints among the three classes for the whole-body configuration. would require a dedicated ablation analysis. We therefore identify topology comparison as a specific direction for fu- ture work. A direct quantitative comparison with existing approaches was not considered methodologically appropriate because the available studies generally rely on different datasets, acquisi- tion protocols, class definitions, and evaluation procedures. In particular, most previous driver-monitoring approaches focus on binary alert/not alert classification and are evaluated under a day/night or relatively stable illumination condition, whereas the present study considers three behavioural states and multiple challenging lighting scenarios. Consequently, the reported results are not directly comparable with those in the literature. Future works could explore Graph Attention Networks (GATs) or Vision Transformers (ViTs) to enhance feature selection by learning an adaptive attention map, focusing on the most discriminative facial and skeletal features for each state. Incorporating these techniques could lead to more robust and explainable human behavior recognition using only vision sensors, improving state classification accuracy while reducing reliance on redundant keypoints. Finally, we plan to expand the dataset (increasing robustly the number of subjects involved in the study) and include occlusions (hats, masks) and real-world artifacts like camera vibrations from train movement. VI. CONCLUSIONS Driver fatigue remains a critical challenge in railway safety, yet traditional monitoring systems, such as the dead-man switch, offer only rudimentary alertness checks. In this study, we proposed a vision-based monitoring system leveraging a customised Directed-Graph Neural Network (DGNN) and whole-body movement analysis to classify train driver states into alert, not alert, and simulated emergency conditions. To determine the most effective input representation for the network, we conducted an ablation study comparing three different keypoint configurations: skeletal-only, facial-only, and whole-body graph. Results indicate that the whole-body graph yields the highest performance, achieving a test ac- curacy of 81% for the three-class classification in the light condition. For the two-class classification, the accuracy is 99% in light condition. Furthermore, we introduced a controlled vision-based dataset containing alert, not-alert, and acted emergency-like behaviours in simulated train-driver scenarios. This study deliberately focused on standard RGB sensing to investigate a low-complexity and widely deployable vision-only config- uration. Future work will examine complementary thermal or near-infrared sensing to improve robustness under low-light conditions and will expand the acquired dataset. Moreover, we will leverage GATs or ViTs to enhance feature selection through adaptive attention maps and focus on the most rele- vant facial and body keypoints. This approach could improve classification accuracy, provide more interpretable human behavior recognition, and reduce reliance on redundant key- points. ACKNOWLEDGMENT This work was supported by the National Institute for In- surance against Accidents at Work INAIL within the VIVA project. REFERENCES [1] A. Filtness and A. Naweed, “Causes, consequences and countermeasures to driver fatigue in the rail industry: The train driver perspective,” Applied ergonomics, vol. 60, p. 12–21, 2017. [2] S. Hughes. The impact of fatigue on driving safety and the railway. [Online]. Available: https://dbesg.deutschebahn.com/esg/news/blog/blog_ driving-and-fatigue-8931050 [3] C. A. Avizzano, P. Tripicchio, E. Ruffaldi, A. Filippeschi, and J. M. Jacinto-Villegas, “Real-time embedded vision system for the watchfulness analysis of train drivers,” IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 1, p. 208–218, 2019. [4] C. Fan, S. Huang, S. Lin, D. Xu, Y. Peng, and S. Yi, “Types, risk factors, consequences, and detection methods of train driver fatigue and distrac- tion,” Computational intelligence and neuroscience, vol. 2022, no. 1, p. 8328077, 2022. [5] R. Foot and G. Doniol-Shaw, “Questions raised on the design of the “dead- man” device installed on trams,” Cognition, Technology & Work, vol. 10, no. 1, p. 41–51, 2008. [6] Q. Berdal, Q. Gadmer, and P. Richard, “Towards cooperative management of fatigue and vigilance in railway operations,” 2024. [7] M. Shahverdy, M. Fathy, R. Berangi, and M. Sabokrou, “Driver behavior detection and classification using deep convolutional neural networks,” Expert Systems with Applications, vol. 149, p. 113240, 2020. [8] L. Peppoloni, A. Filippeschi, E. Ruffaldi, and C. A. Avizzano, “A novel wearable system for the online assessment of risk for biomechanical load in repetitive efforts,” International Journal of Industrial Ergonomics, vol. 52, p. 1–11, 2016. [9] A. Joshi, S. Kyal, S. Banerjee, and T. Mishra, “In-the-wild drowsiness detection from facial expressions,” in 2020 IEEE intelligent vehicles symposium (IV). IEEE, 2020, p. 207–212. [10] W. Wang, H. Zhang, Q. Tong, W. Nie, X. Wang, Z. Wang, R. Lin, and H. Liu, “Fatigue detection based on multiple visual features in virtual driving system,” in IECON 2023-49th Annual Conference of the IEEE Industrial Electronics Society. IEEE, 2023, p. 1–6. [11] E. Perkins, C. Sitaula, M. Burke, and F. Marzbanrad, “Challenges of driver drowsiness prediction: The remaining steps to implementation,” IEEE Transactions on Intelligent Vehicles, vol. 8, no. 2, p. 1319–1338, 2022. VOLUME 11, 20239 Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS [12] V. U. Maheswari, R. Aluvalu, M. P. Kantipudi, K. K. Chennam, K. Kotecha, and J. R. Saini, “Driver drowsiness prediction based on multi- ple aspects using image processing techniques,” IEEE Access, vol. 10, p. 54 980–54 990, 2022. [13] N. N. Pandey and N. B. Muppalaneni, “Dumodds: Dual modeling ap- proach for drowsiness detection based on spatial and spatio-temporal features,” Engineering Applications of Artificial Intelligence, vol. 119, p. 105759, 2023. [14] D. Salem and M. Waleed, “Drowsiness detection in real-time via convolu- tional neural networks and transfer learning,” Journal of Engineering and Applied Science, vol. 71, no. 1, p. 122, 2024. [15] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun, “Carla: An open urban driving simulator,” in Conference on robot learning.PMLR, 2017, p. 1–16. [16] T. Soukupova and J. Cech, “Eye blink detection using facial landmarks,” in 21st computer vision winter workshop, Rimske Toplice, Slovenia, vol. 2, 2016. [17] J. Cech and T. Soukupova, “Real-time eye blink detection using facial landmarks,” Cent. Mach. Perception, Dep. Cybern. Fac. Electr. Eng. Czech Tech. Univ. Prague, p. 1–8, 2016. [18] G. Sikander and S. Anwar, “Driver fatigue detection systems: A review,” IEEE Transactions on Intelligent Transportation Systems, vol. 20, no. 6, p. 2339–2352, 2018. [19] D. F. Dinges and R. C. Grace, “Perclos: A valid psychophysiological measure of alertness as assessed by psychomotor vigilance,” 1998. [Online]. Available: https://api.semanticscholar.org/CorpusID:141471330 [20] Y. Albadawi, A. AlRedhaei, and M. Takruri, “Real-time machine learning- based driver drowsiness detection using visual features,” Journal of imag- ing, vol. 9, no. 5, p. 91, 2023. [21] E. Magán, M. P. Sesmero, J. M. Alonso-Weber, and A. Sanchis, “Driver drowsiness detection by applying deep learning techniques to sequences of images,” Applied Sciences, vol. 12, no. 3, p. 1145, 2022. [22] S. Essahraui, I. Lamaakal, I. El Hamly, Y. Maleh, I. Ouahbi, K. El Makkaoui, M. F. Bouami, P. Pławiak, O. Alfarraj, and A. A. Abd El-Latif, “Real-time driver drowsiness detection using facial analysis and machine learning techniques,” Sensors (Basel, Switzerland), vol. 25, no. 3, p. 812, 2025. [23] L. Chen and W. Zheng, “Research on railway dispatcher fatigue detection method based on deep learning with multi-feature fusion,” Electronics, vol. 12, no. 10, p. 2303, 2023. [24] H. Hayashi, M. Kamezaki, N. Oka, and S. Sugano, “Development of an abnormal sign detection system based on driver monitoring and voice in- teraction for preventing medical-condition-caused car accidents,” in 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC). IEEE, 2020, p. 1–8. [25] S. A. Jakhete and N. Kulkarni, “A comprehensive survey and evaluation of mediapipe face mesh for human emotion recognition,” in 2024 8th International Conference on Computing, Communication, Control and Automation (ICCUBEA). IEEE, 2024, p. 1–8. [26] J.-W. Kim, J.-Y. Choi, E.-J. Ha, and J.-H. Choi, “Human pose estimation using mediapipe pose and optimization method based on a humanoid model,” Applied sciences, vol. 13, no. 4, p. 2700, 2023. [27] C. Dong and G. Du, “An enhanced real-time human pose estimation method based on modified yolov8 framework,” Scientific Reports, 2024. [28] L. Shi, Y. Zhang, J. Cheng, and H. Lu, “Skeleton-based action recognition with directed graph neural networks,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, p. 7912– 7921. OLIVIA NOCENTINI received her B.S. degree in Biomedical Engineering from the University of Pisa, in 2014, and her M.S. degree in Robotics and Automation Engineering in 2018. She earned her Ph.D. in BioRobotics from Scuola Superiore Sant’Anna in 2023. She is currently a Postdoctoral Researcher at the Italian Institute of Technology, focusing on computer vision and human action recognition. MARTA LAGOMARSINO is a postdoctoral re- searcher at the Human-Robot Interfaces and In- teraction lab at IIT. She received the BD and MD in Biomedical and Robotics Engineering from the University of Genoa in 2018 and 2020, both with honors. She obtained her Ph.D. cum Laude in the Electronic, Information, and Bioengineering Department of Politecnico di Milano in 2023. Her research focuses on cognitive ergonomics as- sessment, human-robot collaboration, and mutual human-robot adaptation. GOKHAN SOLAK received the M.Sc. degree in computer engineering from Istanbul Techni- cal University, Istanbul, Turkey, in 2017, and the Ph.D. degree from Computer Science, Queen Mary University of London, London, U.K., in 2022. He is currently a Postdoctoral Researcher with the Italian Institute of Technology, Genoa, Italy. His research interests include robot learning, physical human-robot interaction, robotic grasp- ing and manipulation. YOUNGGEOL CHO was a postdoctoral re- searcher in the Human-Robot Interfaces and In- teraction (HRII) laboratory at the Istituto Italiano di Tecnologia, Italy. He received his M.S. and Ph.D. in Mechanical Engineering from the Korea Advanced Institute of Science and Technology (KAIST), Korea, in 2022. He received his B.S. in Mechanical Engineering from Yonsei University, Korea, in 2023. Currently, his research is in human gesture and intention recognition by multi-sensor fusion for human assistive devices. QIYI TONG is pursuing a Ph.D. at the Human- Robot Interfaces and Interaction Laboratory (HRII) of the Istituto Italiano di Tecnologia (IIT) and the Robotics and Intelligent Machines pro- gram at the University of Genova. He received his B.S. degree from Anhui Polytechnic University, China, in 2019, and his M.S. degree in Computer Science and Technology from Zhejiang University of Technology, China, in 2023. His research inter- ests include deep learning, computer vision, and human action recognition. SARA ZEYNALPOUR is a research technician at the Human-Robot Interfaces and Interaction lab, Istituto Italiano di Tecnologia. She obtained her M.Sc. in Data Science from Sapienza University of Rome and her B.Sc. in Mathematics. Her re- search interests include computer vision, particu- larly developing and optimizing algorithms for vi- sual perception and machine learning applications. 10VOLUME 11, 2023 Author et al.: Preparation of Papers for IEEE TRANSACTIONS and JOURNALS MARTA LORENZINI is a senior technician at the Human-Robot Interfaces and Interaction lab at IIT. She received the BD, MD, and Ph.D. cum Laude in the Electronic Information and Bioengineering Department of Politecnico di Milano, Italy, in 2014, 2016, and 2020, respectively. She was a postdoctoral researcher at the HRI² lab, IIT from 2020 to 2023. Her research interests include hu- man ergonomics estimation and assessment, as- sistive robotics, physical human-robot interaction, and feedback interfaces. ALESSANDRO LEDDA is a Senior Researcher at INAIL’s Department of Technological Inno- vations, working on occupational safety, innova- tive PPE, IoT, artificial intelligence, and emerging technologies for worker protection. He holds a Ph.D. in Building Engineering, Architecture and Construction. He has served as an expert for the Italian Ministry of Health on medical devices and participates in national technical, regulatory, and UNI standardization activities. ARASH AJOUDANI is the director of the Human-Robot Interfaces and Interaction labora- tory at IIT and a recipient of the ERC grants Ergo- Lean (2019) and Real-Move (2023). His research focuses on physical human-robot interaction, mo- bile manipulation, and adaptive control. VOLUME 11, 202311