Paper deep dive
I code or AI code: A comparative evaluation of AI-rated scores in classroom observations
Y. Fong, J. Xiang, T. Y. D. Chan, K. Lee, E. Y. H. Lau
Intelligence
Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 95%
Last extracted: 9/17/2026, 4:48:57 AM
Summary
This study evaluates the feasibility of using a Large Language Model (GPT-5) to score teacher-child interactions in early childhood classrooms using the Classroom Assessment Scoring System (CLASS) framework, benchmarked against human raters. Analyzing 87 video-recorded observations from 38 classrooms in Hong Kong, the research found greater convergence between AI and human ratings in the Emotional Support domain (specifically Quality of Feedback) but greater divergence in procedural or context-dependent domains like Classroom Organization and Instructional Support. The findings suggest AI-assisted scoring is suitable for preliminary screening and reflection rather than high-stakes evaluation.
Entities (9)
Relation Signals (9)
Yasmin Fong â affiliatedwith â The Education University of Hong Kong
confidence 99% · Authors Yasmin Fong... 1 Department of Early Childhood Education, The Education University of Hong Kong
GPT-5 â comparedwith â Human Raters
confidence 98% · AI-rated scores were then compared with human ratings by examining correlations and differences in mean scores
CLASS â hasdomain â Emotional Support
confidence 95% · The CLASS Pre-K framework comprises three overarching domains... The Emotional Climate domain captured the warmth...
CLASS â hasdomain â Classroom Organization
confidence 95% · The Classroom Organization domain focuses on how well the teacher structures behavior...
CLASS â hasdomain â Instructional Support
confidence 95% · The Instructional Support domain examines how teachers extend thinking...
Quality of Feedback â partof â Instructional Support
confidence 95% · The Instructional Support domain examines... through three corresponding dimensions (Concept Development, Quality of Feedback, and Language Modeling).
GPT-5 â usedfor â CLASS
confidence 95% · Using observation transcripts, the AI model was configured to apply the full Classroom Assessment Scoring System (CLASS) framework.
GPT-5 â â
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Classroom observations are widely recognized as a key tool for establishing benchmarks of education quality and guiding pedagogical improvement, yet they remain resource-intensive and dependent on trained observers. This study evaluated the feasibility of using a LLM (GPT-5 model) to score teacher-child interactions in early childhood classrooms, benchmarked against human raters. The study analyzed 87 video-recorded observations from 38 classrooms across 30 kindergartens in Hong Kong. Using observation transcripts, the AI model was configured to apply the full Classroom Assessment Scoring System (CLASS) framework. AI-rated scores were then compared with human ratings by examining correlations and differences in mean scores of the CLASS domains and dimensions. The results showed greater convergence between AI and raters for the Emotional Support domain and, in particular, the Quality of Feedback dimension, which captures how teachers use feedback to extend children's learning. Greater divergence emerged for interactions that were more procedural or context-dependent, particularly within the Classroom Organization and Instructional Support domains. These findings suggest that transcript-based AI scoring may capture some of the relative variation in teacher-child interactions but cannot yet reproduce calibrated human judgements consistently across the full CLASS framework. AI-assisted observation may therefore be more appropriate as a preliminary screening tool rather than as a replacement for trained observers, providing teachers with evidence for reflection rather than high-stakes evaluation. Future research should examine whether domain-specific training and incorporation of contextual and visual information can improve alignment between AI and human rated scores.
Tags
Links
- Source: https://arxiv.org/abs/2609.18274v1
- Canonical: https://arxiv.org/abs/2609.18274v1
PDF not stored locally. Use the link above to view on the source site.
Full Text
71,823 characters extracted from source content.
Expand or collapse full text
1 I Code or AI Code: A Comparative Evaluation of AI-rated Scores in Classroom Observations Authors Yasmin Fong 1 ykmfong@eduhk.hk ORCID: 0000-0002-4643-1258 Jane Xiang 1 xiang@eduhk.hk ORCID: 0009-0001-5836-7265 Tak-Yue Dickson Chan 1 ctakyue@eduhk.hk ORCID: 0009-0003-8079-9835 Kerry Lee 2* kerry.lee@yccece.edu.hk ORCID: 0000-0002-8296-8028 Eva Yi Hung Lau 1* evalau@eduhk.hk ORCID: 0000-0002-1176-0457 1 Department of Early Childhood Education, The Education University of Hong Kong, 10 Lo Ping Road, Tai Po, Hong Kong SAR, China 2 Yew Chung College of Early Childhood Education, 2 Tin Wan Hill Road, Tin Wan, Hong Kong SAR, China Corresponding authors (*) Correspondence concerning this article should be addressed to Eva Lau (evalau@eduhk.hk), Department of Early Childhood Education, The Education University of Hong, 10 Lo Ping Road, Tai Po, Hong Kong SAR, China, and/or to Kerry Lee (kerry.lee@yccece.edu.hk), Presidentâs Office, Yew Chung College of Early Childhood Education, 2 Tin Wan Hill Road, Tin Wan, Hong Kong SAR, China. Acknowledgements This study was supported by a project grant awarded to the corresponding author from the Hong Kong Jockey Club Charities Trust, Hong Kong. We thank all the research assistants and personnel for their data collection efforts, as well as all the teachers and children who participated in this study. The views expressed in this paper are the authorsâ and do not necessarily represent the views of their respective institutions. 2 I Code or AI Code: A Comparative Evaluation of AI-rated Scores in Classroom Observations Abstract Classroom observations are widely recognized as a key tool for establishing benchmarks of education quality and guiding pedagogical improvement, yet they remain resource-intensive and dependent on trained observers. This study evaluated the feasibility of using a LLM (GPT-5 model) to score teacher-child interactions in early childhood classrooms, benchmarked against human raters. The study analyzed 87 video-recorded observations from 38 classrooms across 30 kindergartens in Hong Kong. Using observation transcripts, the AI model was configured to apply the full Classroom Assessment Scoring System (CLASS) framework. AI-rated scores were then compared with human ratings by examining correlations and difference in mean scores of the CLASS domains and dimensions. The results showed greater convergence between AI and raters for the Emotional Support domain and, in particular, the Quality of Feedback dimension, which captures how teachers use feedback to extend children's learning. Greater divergence emerged for interactions that were more procedural or context-dependent, particularly within the Classroom Organization and Instructional Support domains. These findings suggest that transcript-based AI scoring may capture some of the relative variation in teacher-child interactions but cannot yet reproduce calibrated human judgements consistently across the full CLASS framework. AI-assisted observation may therefore be more appropriate as a preliminary screening tool rather than as a replacement for trained observers, providing teachers with evidence for reflection rather than high-stakes evaluation. Future research should examine whether domain-specific training and incorporation of contextual and visual information can improve alignment between AI and human rated scores. Keywords: AI in education; Classroom observation; teacher-child interaction; early childhood education 3 I Code or AI Code: A Comparative Evaluation of AI-rated Scores in Classroom Observations 1. Introduction Classroom observation is widely recognized as a critical approach for evaluating instructional quality and supporting educator professional development (e.g., Grimm et al., 2014), particularly through frameworks such as the Classroom Assessment Scoring System (CLASS; Pianta et al., 2008a), which focuses on the quality of teacher-child interactions. Despite their strong empirical grounding and widespread adoption, classroom observations are highly resource-intensive in terms of time, workload, funding, and scheduling availability (Broad, 2015). These demands are further exacerbated by substantial investments in managing rater training and calibration to ensure consistency (Matsumura et al., 2010; Cohen et al., 2003). As a result, there is a growing need for alternative or complementary approaches that can preserve the rigor of classroom observation while reducing its logistical and resource burdens. Recent advances in AI, particularly with large language models (LLMs) models, offer promising opportunities to address these challenges. While existing research on AI in education has largely focused on efficiency for studentâcentered assessment tasks (e.g., Moore & Lee, 2024), comparatively little empirical work has examined the application of AI to standardized classroom observation. Addressing this gap, the present study examines the methodological applicability of AI in conducting classroom observations by comparing AIârated and humanârated CLASS observation scores derived from videotaped classroom interactions. By analyzing both convergence and divergence in scoring patterns across domains and dimensions, this study seeks to evaluate the feasibility, reliability, and limitations of AIâassisted classroom observation, and to inform ongoing discussions about the appropriate role of AI in supporting scalable, evidenceâbased classroom quality assessment. 1.1 Value of Classroom Observations Classroom observations are a foundational pedagogical practice for assessing and enhancing educational quality worldwide. Beyond its evaluation function, systematic observations provide educators with concrete, evidence-based data about their instructional practices, creating meaningful opportunities for reflection and professional growth (Tarusha & Bushi, 2024; Siddiqua, 2019). There is evidence that informal classroom observation approaches, such as brief, timely, and intentional visits conducted throughout the year, can be beneficial to teachers (Ing, 2010). Through these observation processes, teachers can collaborate with trained professionals and peers to reflect on authentic classroom data within their unique educational contexts (Grimm et al., 2014). Participation in such professional and peer communities enables teachers to identify patterns in their practice, pose reflective questions about areas for improvement, and implement evidence-based instructional changes (Desimone et al., 2025; Wylie et al., 2025). Over time, this cumulative reflective practice supports the development of pedagogically sound strategies that enhance student learning outcomes (Hafen et al., 2015). However, the absence of standardized procedures and observer training often introduces subjective and procedural inconsistencies between teachers and their observers (Wang & Day, 2001), underscoring the need for standardized observation protocols to improve consistency and reliability. To address these needs, researchers have developed a range of standardized observational instruments to examine classroom quality from a more objective perspective. A prominent example is the Classroom Assessment Scoring System (CLASS; Pianta et al., 2008a), which emphasizes the quality of teacher-child interactions. Through instruments like CLASS, classroom quality can be observed through a grounded-theoretical lens, generating 4 empirical data that can inform formative reflection and summative evaluation of educator practices, thereby supporting instructional improvement and quality assurance (Bukhalenkova et al., 2023). In particular, the CLASS framework is grounded in developmental theories of process quality, which posit that childrenâs learning and development are shaped most directly by the quality of their proximal interactions with adults and peers (Bronfenbrenner & Morris, 1998; Pianta et al., 2002). CLASS operationalizes this perspective by systematically assessing observable features of teacher-child interactions, including teachersâ verbal instructions, emotional responsiveness, and nonverbal behaviors in everyday classroom settings. This structured approach to classroom observation has been extensively validated across diverse geographic and demographic contexts (e.g., Downer et al., 2012; Hu et al., 2016; Karuppiah, 2021; Pakarinen et al., 2010) and has been associated with teacher practices and child outcomes (Hamre et al., 2014; Hoang et al., 2019; Leyva et al., 2015). Moreover, CLASS has been widely integrated into teacher professional development and improvement initiatives, with evidence supporting its utility as a tool for reflective practice, coaching, and instructional support (e.g., Early et al., 2016; Pianta et al., 2008b; Zan & Donegan-Ritter, 2014). 1.2 Challenges in Resource Intensity for Classroom Observations Despite the well-established pedagogical value of classroom observations for supporting teacher professional development and evaluating program quality, their implementation poses substantial resource challenges for many education organizations. These challenges situate classroom observations within a broader set of constraints affecting teacher professional development, including time pressures, increased teacher workloads, limited funding, and restricted scheduling availability (e.g., Broad, 2015; Fairman et al., 2022), all of which may limit their prioritization in practice. As a result, although classroom observation is widely endorsed as evidence-based practice, its routine and sustained implementation are often difficult to prioritize in school settings. These logistical constraints may be resolved through video-based observation as an alternative to live observation (e.g., Borg, 2021). Compared with in-person observations, video recordings allow for classroom practices and dynamics to be captured remotely and reviewed multiple times, providing flexibility and analytic depth for observers (Ault et al., 2019). Video-based approaches may also reduce teachersâ anxiety associated with the physical presence of an observer, thereby supporting more naturalistic instructional behaviors and partially mitigating the Hawthorne effect, whereby teachers alter their practice when being observed (Mackey, 2017; Page, 2016). As such, video observations offer a less intrusive means of examining classroom processes and may yield more authentic representations of everyday instructional practices (Curby et al., 2016). However, video-based observation does not eliminate challenges associated with human raters. Observer fatigue and bias remain difficult to avoid, particularly in large-scale studies requiring prolonged coding sessions and repeated reliability checks. Prior research has demonstrated that fatigue-related drift can introduce increased variability and reduce measurement precision over time (Matsumura et al., 2010; Raudenbush et al., 2007). These cognitive and analytic demands continue to contribute to the overall resource burden of classroom observation when data collection is conducted remotely. Beyond fatigue, extensive rater calibration and training are required to ensure reliable and valid observation coding. Validated instruments, such as the CLASS measure, necessitate intensive initial training and ongoing reliability monitoring to maintain measurement validity (Pianta et al., 2008a; Hamre et al., 2013). This requirement reflects the inherently interpretative nature of classroom observation, which demands fineâgrained judgments about complex and dynamic 5 instructional interactions, often under time constraints (Bell et al., 2012). Sustaining reliability over time presents additional challenges, particularly in large-scale projects involving multiple raters, where continuous calibration and recalibration are necessary to maintain scoring consistency (Allen et al., 2013). These processes substantially increase the human, temporal, and financial resources required for classroom observations. Collectively, the demands associated with observation logistics, observer fatigue, intensive training, and calibration intensify the cumulative resource burden of classroom observation. These persistent challenges raise concerns about the long-term feasibility and scalability of traditional observational approaches, particularly in resource-constrained educational contexts, and underscore the need for complementary solutions that support reliable, efficient, and sustainable classroom-quality assessment. 1.3 Growing prominence of AI as a resource In recent years, AI has advanced rapidly, driven in part by the emergence of LLMs that enable increasingly sophisticated computational learning and human-like interaction (Kasneci et al., 2023). These developments have expanded AIâs capacity to simulate aspects of human learning, comprehension, problem-solving, decision-making, and creativity (e.g., Duan et al., 2019; Topol, 2019), positioning AI as a transformative technology with growing potential to alleviate resource constraints across sectors. In education, AI-powered systems, particularly conversational and generative tools, have gained prominence for their ability to support real-time problem-solving, generate high-quality instructional content, personalize learning processes, and provide rapid feedback to students and educators (e.g., Bettayeb et al., 2024). Consequently, much of the existing research has focused on technology-assisted teaching and learning, and on educational assessment practices, where AI has demonstrated considerable benefits such as increased efficiency, automated grading, feedback generation, and adaptive instructional support (Chen et al., 2022; Chu et al., 2025; Huang et al., 2023; Owan et al., 2023). As a result, AI and their LLM have been widely used to support student- centered evaluation activities, including assignment analysis, formative feedback, and reporting, thereby reducing educatorsâ workload (e.g., Liao et al., 2024; Liebenow et al., 2025). Despite their growing prominence, the application of AI in education has remained largely confined to task-based evaluations, with comparatively limited attention given to how AI-powered tools may be used in classroom evaluations to inform educators' instructional practices or professional development. One early attempt to address this gap was conducted by Shapsough and Zualkernan (2018), who examined the automation of classroom observation by testing machineâlearning algorithms within a mobile application. Using the Stallings Classroom Snapshot observation system (Stallings, 1977), their approach achieved 68.9% accuracy rate in labeling classroom activities from audio recordings. While these findings demonstrated the technical feasibility of automating certain aspects of classroom observation, the reported accuracy was relatively modest and achieved by only one of the 10 machine algorithms tested, highlighting both the promise and the limitations of early technologyâbased observational systems. Other work has approached AI-assisted observation from a different angle. For example, Singh et al. (2025) examined verbal exchanges between teachers and students in lecture settings, using AI to detect nuanced patterns in speech and verbal dynamics. Although this innovative work offered valuable insights into factors associated with lecture quality, it remained primarily focused on audio-based interactions and did not fully capture the broader dynamics of classroom settings, such as social relationships, curriculum structure, and instructional elements. Addressing the need for broader applicability, more recent work has moved beyond single-exchange analysis toward scaling AI-assisted observation across entire classroom 6 sessions. A recent pilot study by Li et al. (2026) explored the use of AI as a âteammateâ for scaling up classroom observations. Drawing on multiple LLM agents, the study developed a pipeline that transcribed classroom audio, contextualized the language, and evaluated it using an observational rubric. This approach achieved an 18-fold efficiency gain compared with traditional classroom observation methods involving human raters. While this was a notable advancement, the rubric was limited to binary coding (present/not present) across two subscales from the ECQRS-EC and SSTEW frameworks, which may constrain its broader applicability in capturing the complexity of classroom interactions. Collectively, these emerging studies demonstrate the growing potential of AI to support classroom evaluation. However, these studies remain constrained by limited scope, fragmented capture of interactions, and simplified rubric systems that warrant further empirical investigation. 1.4 The Current Study The present study contributes to the growing discourse on AI in education by examining the feasibility and reliability of LLM for scoring classroom observations. Classroom observation is a data-intensive process that depends heavily on trained human observers, which can limit the scalability of evidence-based classroom quality assessment in educational settings. In response to these challenges, this study investigates whether AI can perform the complex interpretive work traditionally undertaken by trained human observers when evaluating instructional interactions during classroom observation. Guided by this aim, the study addressed the following research question: To what extent do AI-rated scores align with human ratersâ scores in classroom observations? To answer this question, the CLASS measure was selected as the observational framework, given its status as a well-validated and standardized instrument for assessing classroom quality through measuring teacher-child interactions in early childhood settings (Hamre et al., 2014). Drawing on prior exploratory studies (Shapsough & Zualkernan, 2018; Singh et al., 2025), we hypothesized that AI could rate observation scores closely aligned with those of human raters in instructional language exchanges (i.e., convergence in Instructional Support and its dimensions). Given that the interactions within the Emotional Support and Classroom Organization domains, and their corresponding dimensions (Pianta et al., 2008a), may only be partially represented in transcripts, we were unsure of the alignment between AI and raters. 2. Method 2.1 Procedures The ethics review committee from the corresponding authorsâ university approved the current study before any data was obtained. Teachers, corresponding children, and parents of the respective kindergartens provided written consent for videotaping classroom activities throughout the childrenâs attendance on a typical kindergarten day. Participants who were filmed incidentally without consent were removed in post-production, and activities that would violate participantsâ privacy and protection (e.g., washroom breaks) were not filmed. Classroom observations were filmed by trained research assistants who adhered to a strict and standardized videotaping protocol. This study is part of a large-scale longitudinal project that focuses on community- based interventions to bridge developmental gaps among young children aged 3 to 6. The present study obtained video data from classroom observations conducted during the baseline year in 62 Hong Kong kindergartens, from August 2023 through the first year of kindergarten (K1), for children aged 3-4. Filming began at the start of each class session, when children entered the classroom, and continued through the end of the dayâs curriculum, depending on the preschool program and the participating classroomsâ schedules. The recordings captured a 7 range of typical curriculum activities, including activity corners, gross motor play, themed learning tasks, structured instructional lessons (e.g., language learning), and routine activities (e.g., snacks/mealtimes or transition to washroom breaks). 2.2 Sample This study drew subsample video recordings from 30 kindergartens, representing a range of annual tuition fees: low (government-subsidized or below HKD9,999, n = 16); mid- range (HKD10,000-19,999, n = 7); and high (HKD20,000 and above, n = 7). The sample was selected through a screening process based on predetermined criteria for audio-visual quality and coding feasibility. Videos were purposefully selected based on the criteria that classroom exchanges were clearly captured as videos were to be audio transcribed, classrooms with a single educator were prioritized over those with multiple educators to minimize overlapping speech and enhance the clarity of observational targets (excluding 83.24% of videos). In alignment with the CLASS framework (Pianta et al., 2008a), footage depicting childrenâs recess or unstructured free-play periods was excluded to maintain consistency with the measureâs coding requirements. Based on this screening process, only videos that met all the quality and relevance criteria were selected for subsequent coding and analysis. The finalized sample included video recordings from 38 K1 classroom teachers (47.4% female, 52.6% missing; average teaching experience of 10.89 years, SD = 7.42), totaling approximately 26.5 hours for analysis. The high percentage of missing demographic data was attributable to teacher non-reporting (see Table 1 for more details), though these variables are not considered central to the study. In total, there were 87 classroom observations, each lasting 15-20 minutes, with an average of two per classroom to capture variability across classroom activities. Table 1 Teacher demographics n Mean SD Gender Female 18 Missing 20 Age (years) 18 32.17 7.42 Missing 20 Teaching experience (years) 10.89 7.99 Missing 20 Education attainment 18 Up to Diploma Certificate 2 Bachelorâs Degree 12 Postgraduate or Masterâs Degree 3 Other Qualifications 1 Missing 20 2.3 Measures 2.3.1 The Classroom Assessment Scoring System Pre-K The Classroom Assessment Scoring System Pre-K (CLASS Pre-K; Pianta et al., 2008a) was used as the classroom observation measure to evaluate classroom quality. The CLASS Pre-K framework comprises three overarching domains derived from observed 8 dimensions, as shown in Figure 1. The Emotional Climate domain captured the warmth, responsiveness, and emotional tone in teacher-child relationships that help children feel safe to participate in the classroom through its four corresponding dimensions (Positive Climate, Negative Climate, Teacher Sensitivity, and Regard for Student Perspectives). The Classroom Organization domain focuses on how well the teacher structures behavior, time, and engagement to ensure learning runs smoothly through three corresponding dimensions (Behavior Management, Productivity, and Instructional Learning Formats). The Instructional Support domain examines how teachers extend thinking, provide feedback, and build on childrenâs vocabulary through meaningful interactions through three corresponding dimensions (Concept Development, Quality of Feedback, and Language Modeling). The domains provided a broad overview of teacher-child interaction quality, while a focus on the constituent dimensions allows for a more detailed examination of specific interactional practices. Domain scores were aggregated from individual dimension scores, which were rated from a 7-point Likert scale from (1) low to (7) high quality. Figure 1 CLASS Scale Overview Typically, the CLASS methodology requires between four and seven observations per classroom, which are averaged to provide a more representative estimate of overall educator- child interaction quality. This approach is primarily intended to evaluate classroom-level quality. However, as the primary aim of the present study was to examine the applicability and consistency of raters and AI during the coding process, ratings from individual 9 observation cycles were compared to more precisely evaluate similarities and differences between raters and AI-rated coding. 2.3.2 Human raters In the current study, the term âratersâ refers specifically to the performance and scoring of a group of eight human raters. All raters held at least a Bachelorâs Degree or were qualified early childhood education practitioners. These human raters completed a comprehensive training session on the CLASS framework led by a licensed and experienced CLASS Pre-K trainer. This comprehensive training covered the CLASS framework and observation logistics and included exemplar practice videos scored against the framework manual. Following the training, the trainer led a series of calibration rounds to ensure alignment with Hong Kongâs local context and to establish calibration and inter-rater reliability standards. The calibration process began with a first round in which all raters independently coded five identical classroom observation videos using the CLASS manual, followed by individualized feedback and a trainer-led group discussion to promote alignment in scoring. Raters then completed a second round of independent coding of five identical video observations and again received individualized feedback based on their scores. After achieving satisfactory alignment, the remaining 77 classroom observation videos were coded independently, with each observation randomly assigned to two different raters to ensure ongoing cross-checking of scores. Using dimension scores, the interrater reliability among the human raters was examined with weighted Cohenâs kappa (Îșw), yielding values ranging from .672 to .861, indicating moderate to substantial agreement with a certified trainer. While these values may be lower than typical thresholds, earlier studies of CLASS have noted minimum criteria of .60 (La Paro et al., 2004; Sandilos & DiPerna, 2011). Subsequently, a consolidated score, averaged across overlapping raters, was calculated for each observation and used for analysis. 2.3.3 AI Rater Within the scope of this study, âAIâ or âAI modelâ specifically refers to the performance and scoring rated by the Perplexity GPT-5.0 model (Perplexity, 2025), which served as the LLM for this comparative analysis. All AI training and coding occurred on August 7, 2025, 8:00 PM to August 8, 5:30 AM Pacific Time (PT), using the most up-to-date artificial intelligence model at the time of analysis. Given the difficulty of AI models in accurately interpreting video content at the time of this study, all classroom observation videos were transcribed for AI training and coding. Classroom observation transcriptions were completed by fluent or native bilingual (Chinese and English) research personnel who were not part of the CLASS coding process. Transcriptions were completed based on the classroom medium of instruction and the interaction between teachers and children, which was predominantly in Chinese. The AI model was trained using a prompt-engineering approach, a critical component for optimizing LLM learning and performance. Initial training prompts were developed by identifying essential instructions and requirements that guided how the model should learn and utilize the CLASS framework for scoring. These prompts were subsequently refined through interactive interactions with the AI model to improve clarity and effectiveness (see the detailed prompts in Appendix A). To parallel the training provided to human raters, the CLASS framework manual was provided to the AI model to establish an understanding of the observational framework and scoring procedures. 10 Following this, a series of structured training and calibration cycles was conducted using classroom observation notes and consolidated scores rated by the human raters. Each training cycle consisted of three steps (see Appendix B for more details): (i) a case study phase, in which the AI model was presented with two classroom observation transcripts, along with the human ratersâ consolidated scores and observation notes for alignment; (i) a practice phase, during with the AI independently coded two new observation transcripts; and (i) a feedback phase, in which consolidated scores and observations notes were shared with the AI to support reflective learning. This three-step cycle was repeated four times to strengthen the AIâs interpretability of the CLASS measure and alignment to the local context. Figure 2 summarizes this training procedure. Following completion of training and calibration, the AI model independently scored the remaining 71 classroom observation transcripts and rated scores for the respective CLASS dimensions for subsequent analysis. 11 Figure 2 Training and calibration procedure for Humans and AI 12 2.4 Data Analysis Before conducting analysis, all Negative Climate dimension scores were reversed (i.e., higher scores indicated a lower level of negativity in the observed classroom). Outliers in the CLASS dimension scores accounted for 0.423% of the data points for raters and 0.704% for AI, which were subsequently winsorized by replacing the corresponding data points with ï± 3 SD values for each dimension. Domain scores were then aggregated from their corresponding dimensions. Descriptive statistics were first computed to examine the scoring of both raters and AI at the CLASS domain and dimension levels. To examine alignment between human-rated scores and AI-rated scores, correlation analyses were conducted to assess convergence or divergence in scoring patterns across the domains and dimensions. Inter-rater reliability analysis was then used to evaluate overall agreement, followed by independent t-tests to examine mean differences between AI and ratersâ scores at both the domain and dimension levels. All quantitative analyses were conducted using IBM SPSS Statistics (Version 30). Our study examined how AI-rated scores compared with those of consolidated human raters in assessing classroom observations. 3. Results 3.1 Preliminary Results Table 2 presents the descriptive statistics for ratersâ and AI-rated scores across all domains and dimensions. For the Emotional Support domain, raters tended to score higher at both the domain and dimension levels than AI ratings of the corresponding domain scores and respective dimensions, with larger differences observed for the Positive Climate and Negative Climate dimensions. In the Classroom Organization domain, raters scored slightly lower than AI ratings. In general, their dimension scores were similar with minor differences, while the Behavior Management dimension showed the largest difference. Within the Instructional Support domain, raters consistently scored lower than AI ratings at both the domain and dimension level, with the Concept Development dimension showing the largest difference. Furthermore, the distributions of all measured variables were evaluated for normality by inspecting the unstandardized skewness (absolute value < 2) and kurtosis (absolute values < 3). Only the ratersâ Negative Climate dimension score showed a non-normal distribution, with extreme kurtosis. 13 Table 2 Descriptive statistics of raters (n = 71) and AI (n = 71) Raters AI Mean SD Skewness Kurtosis Mean SD Skewness Kurtosis CLASS Domains Emotional Support 5.490 .562 .124 -.234 4.685 .632 -.626 .215 Classroom Organization 5.121 .462 -.412 .264 5.134 .567 -.478 .240 Instructional Support 2.937 .878 .681 -.123 3.221 .799 .093 -.670 CLASS Dimensions Positive Climate 5.435 .793 -.736 1.461 4.280 .721 -.955 1.229 Negative Climate 1 6.956 .131 -3.034 8.293 5.870 .559 -.551 1.585 Teacher Sensitivity 5.178 .753 -.124 -.353 4.890 .838 -.533 -.075 Regard for Student Perspectives 4.385 1.039 -.047 -.579 3.690 .821 -.160 -.437 Behavior Management 5.308 .639 -.441 -.036 5.450 .672 -1.125 1.334 Productivity 5.338 .764 -.861 1.413 5.300 .705 -.743 .342 Instructional Learning Formats 4.713 .987 -.445 .393 4.650 .927 -.559 .496 Concept Development 2.425 1.072 .686 -.493 2.830 1.00 .262 -.703 Quality of Feedback 3.316 .957 .632 .383 3.510 .892 .164 -.110 Language Modelling 3.069 .859 .105 -.375 3.320 .770 -.250 .408 Note. 1 Reversed Score. 14 Figure 3 illustrates the score distribution of Negative Climate ratings for both AI and 1 raters. Given that Negative Climate was a reverse-scored dimension in which higher scores 2 indicated lower levels of negativity, ratersâ observation scores were heavily concentrated at 3 the scale ceiling, with 7 (n = 63) as the most common score, followed by scores between 6 4 and 7 (n = 8). This indicated raters generally perceived minimal negativity across observed 5 classrooms. In contrast, AI-rated scores showed a slightly larger distributional variance, with 6 scores predominantly at 6 (n = 51), followed by 5 (n = 13), 7 (n = 6), and 4 (n = 1). 7 8 Figure 3 9 Distribution of Negative Climate in Ratersâ and AI-rated scores 10 11 12 3.2 Scoring alignment between raters and AI 13 Correlation analysis was conducted to assess the associations in scores between raters 14 and AI across the three domains and ten dimensions, as shown in Table 3. Pearson 15 correlations were conducted on the domains, and to address the nonparametric assumptions, 16 Spearman correlations were conducted for the dimensions. 17 18 19 20 15 Table 3 1 Correlations between ratersâ and AI-rated scores 2 CLASS Framework Correlations 1 (r) Domains Emotional Support .406*** Classroom Organization .123 Instructional Support .195 Dimensions Positive Climate .438*** Negative Climate 2 .085 Teacher Sensitivity .313** Regard for Student Perspectives .296* Behavior Management .161 Productivity .029 Instructional Learning Formats .136 Concept Development .132 Quality of Feedback .269* Language Modelling .144 Note. 1 Pearson correlations for domains and Spearman correlations for the dimensions. 3 2 Reversed Score. Significant values: *p < .05, **p <.01, ***p <.001. 4 5 At the domain level, a statistically significant positive correlation emerged between 6 raters and AI on Emotional Support, indicating a modest degree of convergence. The 7 correlations for Classroom Organization and Instructional Support were not statistically 8 significant. At the dimension level, three of the four Emotional Support dimensions yielded 9 significant associations. Positive Climate showed the strongest alignment, followed by 10 Teacher Sensitivity and Regard for Student Perspectives, while the correlation for Negative 11 Climate was not significant. None of the three dimensions within the Classroom Organization 12 reached significance. The Quality of Feedback dimension was the only one to show a 13 positive, significant correlation, while the remaining Instructional Support dimensions were 14 non-significant. 15 Results from the inter-rater reliability between the consolidated ratersâ and AI-rated 16 scores for the CLASS dimensions indicated moderate agreement, with a weighted Cohenâs 17 kappa (Îșw) of 0.681 ([0.646, 0.716], p < .001). Based on the CLASS measurement protocol, 18 this reported statistic would be considered a reliable standard, with dimensions agreeing to 19 within 1 point (corresponding to Îșw â 0.60â0.75). Furthermore, examination of individual 20 ratersâ scores and AI-rated scores continued to indicate somewhat moderate agreement, with 21 weighted Cohenâs kappa (Îșw) values ranging from .515 to .712 (ps < .001). Subsequent 22 analyses examined further comparisons of the domain and dimension scores in detail. 23 Table 4 presents the results of the independent t-tests comparing ratersâ and AI-rated 24 scores at the domain- and dimension-levels of the CLASS framework. At the domain level, 25 significant differences were observed: raters tended to score higher in Emotional Support and 26 lower in Instructional Support than AI. No significant difference was found for the Classroom 27 Organization domain. 28 16 Table 4 1 Results from the independent t-tests 2 CLASS framework Mean Difference t-value Degrees of freedom p Cohenâs d Domain Level Emotional Support .806 8.002 140 < .001 1.343 Classroom Organization -.013 -.111 140 .912 .019 Instructional Support -.284 -2.041 140 .043 .343 Dimension Level Positive Climate 1.152 9.9075 140 < .001 1.523 Negative Climate 1 1.084 16.279 78.204 # < .001 2.732 Teacher Sensitivity .291 2.178 140 .031 .365 Regard for Student Perspectives .694 4.419 140 < .001 .742 Behavior Management -.149 -1.372 140 .172 .230 Productivity .041 .360 140 .720 .060 Instructional Learning Formats .069 .439 140 .662 .074 Concept Development -.406 -2.336 140 .021 .392 Quality of Feedback -.191 -1.229 140 .221 .206 Language Modelling -.255 -1.928 140 .056 .324 Note. 1 Reversed Score. Data are presented as mean (SD). Mean differences refer to the ratersâ 3 scores minus the AIâs scores. # Degrees of freedom adjusted due to violation of unequal 4 variances. 5 6 At the dimension level, significant differences between raters and AI were observed 7 across multiple dimensions. Across the dimensions of Positive Climate, Negative Climate, 8 Teacher Sensitivity, and Regard for Student Perspectives, raters scored significantly higher 9 than AI. This would suggest, at the mean level, a divergence of the Emotional Support 10 dimensions. No significant differences were observed for Behavior Management, 11 Productivity, and Instructional Learning Formats, suggesting no observed mean difference 12 between raters and AI in the Classroom Organization dimensions. On the other hand, 13 significant differences were observed in the Concept Development, but not in the Quality of 14 Feedback and Language Modeling dimensions. This suggests that a divergence within 15 specific Instructional Support dimensions, in which raters scored significantly lower than AI. 16 Overall, these findings suggest that there may be a domain- and dimension-specific pattern of 17 divergence between raters and that AI-rated scoring. 18 19 4. Discussion 20 The results presented above suggest that AI-rated scoring partially aligned with 21 human raters on the CLASS framework, but was domain-dependent and falls short of what 22 independent classroom observation would require. Overall, the agreement was moderate, 23 consistent with early reports on weighted kappa agreement (La Paro et al., 2004). AI-rated 24 scores were typically within 1-scale point of ratersâ scores, a level that the CLASS protocol 25 treats as an acceptable standard of reliability for observational assessment (Pianta et al., 26 2008a; Sandilos & DiPerna, 2011). Moderate agreement of this kind is sufficient for 27 17 observational applications when the goal is to support system-level monitoring or formative 1 feedback rather than individualized evaluation of classroom quality (Landis & Koch, 1977; 2 Cicchetti, 1994). Our findings partially confirm and partially refute our hypothesis, and this 3 pattern of convergence and divergence defines the current ceiling of AI-assisted observation. 4 5 4.1 Where AI aligns with raters: verbal and structured 6 Consistent with our hypothesis, convergence was strongest where classroom quality 7 was expressed through explicit and structured language, aligning with prior studies 8 (Shapsough & Zualkernan, 2018; Singh et al., 2025). The strength of the reported correlations 9 was comparable to earlier findings in which multimodal AI models approximated human 10 inter-rater reliability (Hou et al., 2024) suggesting promising consistency between human and 11 AI evaluations. In the Quality of Feedback dimension, correlations were significant and mean 12 differences were not, indicating minimal systematic bias. This is consistent with Whitehill 13 and LoCasale-Crouch (2024), who reported that LLM-based predictions approximated human 14 ratings in instructions-based support in the classroom when the construct operationalized 15 through explicitly verbal cues and with emerging evidence of AI-human convergence in 16 feedback evaluations (e.g., Barri & Hikmawan, 2026). The structured, exchange-based 17 indicators of this dimension (e.g., scaffolding, contingent responses) appeared to match the 18 pattern-recognition strengths of LLMs trained on large-scale text data (Mirchandani et al., 19 2023; Speer et al., 2026). This finding suggests that AI shows promising convergence in 20 evaluating teacher-child interactions related to contingent responding and may serve as an 21 informative indicator for teachers to reflect on their instructional practice. 22 Convergence was also evident in the Emotional Support domain and its dimensions 23 with significant correlations, a finding that exceeded our initial uncertainty about this domain. 24 The convergence suggests that AI captured the warm, positive verbal teacher-child 25 interactions that underpin a safe and warm classroom environment (Hamre & Pianta, 2005). 26 Yet, raters consistently assigned higher average scores. This difference likely reflected ratersâ 27 sensitivity to nuanced nonverbal and relational cues, such as facial expressions and tone 28 (Calvo & DâMello, 2010; Pianta et al., 2012) that were not accessible through transcripts. 29 However, it does not preclude the possibility of rater bias (Park et al., 2015). Such bias may 30 arise from prior expectations, subjective interpretations of emotional displays, or a tendency 31 to overvalue relational warmth in teaching contexts, thereby inflating observed differences. 32 Despite relying primarily on transcripts, AI still achieved consistency with raters, but the 33 persistent gap marks the limit of transcript-based scoring for this domain. 34 35 4.2 Where AI diverged: nonverbal, procedural, and contextual constructs 36 Divergence was most pronounced in the Classroom Organization and its dimensions, 37 indicating minimal evidence of alignment between raters and AI. This aligned with our 38 expectation of limited evidence for this domain and suggests that classroom management 39 poses challenges for automated observation. This difficulty likely stems from inherently 40 rapid, context-sensitive, and nonverbal aspects of classroom management practices (Li, 2006) 41 that would challenge transcript-based LLMs (Pang et al., 2023), particularly without 42 representation of spatial dynamics and movement of teachers and children within the 43 classroom environment (Elbaum et al., 2024). Accounting for these nuanced cues in future 44 LLM development may enable AI systems to better capture the complexity of classroom 45 management, thereby enhancing the validity of automated observations and aligning them 46 more closely with the multifaceted changes that occur in classroom environments. 47 The findings for Instructional Support were more inconclusive than our hypothesis 48 anticipated. The Concept Development and Language Modeling dimensions, which rely 49 18 heavily on explicit instructional language to capture pedagogical interactions (Hu et al., 1 2016), revealed low correlations between raters and AI. This misalignment with AI 2 consistently having scored higher than raters on both dimensions suggests that AI may have 3 identified additional, or incorrect, discourse features of instructional exchanges that raters 4 may have overlooked, especially when simultaneously evaluating multiple dimensions of 5 teaching quality (Cohen & Goldhaber, 2016). While this interpretation aligns with prior 6 studies demonstrating of the early capabilities of LLMs in detecting teacher discourse (Kelly 7 et al., 2018) and supports the view that classroom transcripts can provide sufficient evidence 8 to estimate global scores for instruction-focused exchanges (Whitehill & LoCasale-Crouch, 9 2024), potential of incorrect lens when observing discourse evidence. This is interesting, 10 given Quality of Feedback showed stronger convergence than the other two Instructional 11 Support dimensions could be due to these exchanges, in which the latter dimensions requiring 12 pedagogical focused strategies. This is consistent with evidence that well-trained AI models 13 falter when evaluating classroom dialogue beyond their training scope (Wang et al, 2024), 14 and points to the needed for targeted pedagogical training before AI can parallel human 15 observers in recognizing nuanced variations in teaching practices (e.g., Bell et al., 2019). The 16 partial support for our hypothesis is therefore instructive, indicating AI convergence where 17 constructs are explicit and structured, but not across the Instructional Support as a whole. 18 19 Against this backdrop, these findings reveal a specific structural pattern, that partially 20 supported the hypothesis of convergence between raters and AI in instructional language 21 exchanges (Instructional Support). On the other hand, the unspecified hypothesis on the 22 convergence of Emotional Support and Classroom Organization, yielded surprising results. 23 As agreement with human raters remains partial and conditional on the construct being 24 assessed, the present findings do not support using AI as an independent classroom 25 observation system but support the AI-assisted observation as complementary. 26 However, effective implementation requires further preparation. Teachers and 27 observers need further AI literacy training as part of professional development for classroom 28 observations (Riggs, 2025), while LLMs require targeted pedagogical training and calibration 29 approximate human observation (Wang & Chen, 2025). AI-assistance retains a clear value in 30 relieving the workload of classroom observation, which may also help mitigate occurrences 31 of rater fatigue when prolonged observation periods, repeated exposure to similar classroom 32 interactions, and sustained cognitive load diminish observersâ attentional accuracy and 33 consistency over time (Matsumura et al., 2010; Raudenbush et al., 2007). This positions AI to 34 serve as a first pass scoring and evidence-surfacing tool that directs human attention where it 35 is most needed. More importantly, our findings do not imply that AI should replace authentic 36 judgment but rather support the direction of a hybrid observation model in which AI could 37 enhance the efficacy, consistency, and scalability, while raters retain a central role in 38 interpreting emotional nuance and supporting professional learning. A hybrid model could 39 offer direct implications for how schools deploy AI-assisted observation, how teachers are 40 prepared to use it, and how future research should be prioritized. 41 42 5. Limitations and Future Directions 43 While our study provided innovative contributions on AI for classroom observations, 44 several limitations should be acknowledged, along with directions for future research. First, 45 due to the geographic and access constraints associated with AI tools at the time of data 46 collection, the study examined only one LLM (i.e., Perplexity GPT-5.0) using transcript-47 based classroom observations. While this approach allowed for an initial examination of AI 48 feasibility in classroom observation, reliance on a single model may limit the generalizability 49 19 of the findings. Future studies should incorporate comparisons across multiple LLMs to 1 enable more nuanced assessments of model performance and to determine whether observed 2 patterns are model-specific or generalizable across AI platforms. Moreover, recent advances 3 in multimodal LLM, such as Google Gemini (Gemini Team, 2023), demonstrate increasing 4 capacity to process video and audiovisual data. This suggests promising directions for AI-5 assisted classroom observation for continuous investigations. 6 Secondly, the studyâs contextual scope was limited to early childhood classrooms in 7 Hong Kong serving children aged 3-4 years. This developmental and cultural context may 8 place greater emphasis on certain aspects of teacher-child interactions, which could have 9 influenced the scoring patterns of both artificial and authentic raters. As a result, the findings 10 may not generalize to classroom observations serving older students, where instructional 11 interactions may be more cognitively and academically oriented (e.g., Tengberg et al., 2022). 12 Future research should extend this line of inquiry to primary and secondary education settings 13 to test the adaptability and effectiveness of AI systems across a broader range of instructional 14 contexts, age groups, and pedagogical demands. Moreover, our analysis sample was limited 15 to a single teacher per video due to transcription constraints, thereby excluding over 80% of 16 the full video sample. Although this reduced the representativeness of the sample and may 17 have limited the capture of classroom interaction variability, we do not expect it to 18 substantially affect our findings, as the study focused on the accuracy of AI-rated coding, 19 rather than the assessment of teachersâ classroom quality. Nevertheless, these factors may 20 limit the generalizability of our findings. Future research could address this by developing 21 scalable pipelines, similar to those described by Li et al. (2026), to support large-scale LLM-22 based observations. 23 Third, the present study relied exclusively on the CLASS measure as the 24 observational framework for comparison and used single observations. While CLASS is a 25 well-validated and widely used instrument, using a single scale and limited observation 26 rounds may limit the interpretability and stability of the findings. We recommend that future 27 studies increase the number of observational cycles to expand sample size and power for 28 multilevel analyses, enabling more robust examinations of reliability patterns and consistency 29 in classroom observations. Additionally, future research should also consider alternative 30 observational frameworks, such as the Early Childhood Environment Rating Scale (ECERS; 31 Harms et al., 1998) for assessing structural enablers in the classroom, or the International 32 Comparative Analysis of Learning and Teaching (ICALT; van der Lans et al., 2018) for 33 assessing teachersâ learning instruction in older grade classrooms. Doing so would examine 34 whether the performance is consistent across different conceptualizations of classroom 35 quality and observational protocols. 36 37 6. Conclusions and Implications 38 This study evaluated the methodological applicability of AI to support classroom 39 observation by comparing AIârated scores with those of trained human raters using the full 40 CLASS observational framework. The findings showed that AI can approximate human 41 judgment of an acceptable reliability for formative and system-level purposes, but only in a 42 specific CLASS dimension (Quality of Feedback) and domain (Emotional Support). 43 Divergence was evident when observations depended on nonverbal, procedural, and 44 contextual cues (Classroom Organization and Instructional Support domain). Our findings 45 conclude that AI-assisted observation can serve as a limited complementary to human 46 judgement. 47 From a practical perspective, the findings suggest that schools with limited human 48 resources may leverage AI as a reference or screening tool for classroom evaluation, 49 20 particularly when capacity for intensive human observation is constrained. AIâassisted 1 approaches can help alleviate resource demands in domains such as Emotional Support while 2 still informing instructional improvement, and they may be particularly helpful for teachers 3 who want to reflect on their feedback and prompting practices, indicators central to the 4 Quality of Feedback. For these benefits to be realized, AI-assisted observation should be 5 positioned as a complement to, not a replacement for, human raters, and teacher-facing tools 6 should be introduced with explicit training on the modelâs capabilities and limits. AI outputs 7 should be framed as evidence for reflection rather than as scores for evaluation. More 8 broadly, the integration of AI into classroom observation systems holds promise for providing 9 scalable, costâeffective monitoring of classroom quality, offering policymakers and 10 educational leaders new opportunities to support professional development and systemâlevel 11 improvement, particularly where transcript-based approaches can substitute for audiovisual 12 data raised from data-protection and ethical concerns. 13 From a research perspective, the findings highlight the need for further studies to 14 compare multiple AI models, incorporate multimodal data sources (e.g., video), and examine 15 diverse educational contexts to strengthen the validity and generalizability of AIâassisted 16 observation. Research should also evaluate whether AI-assisted workflows genuinely reduce 17 rater workload without compromising validity, and whether targeted training of LLMs on 18 early childhood classroom discourse closes the alignment gap in instruction-focused 19 dimensions. Continued collaboration between educational researchers and AI developers will 20 be essential to optimize AI systems for classroom observation and to ensure their responsible, 21 ethical, and effective use in supporting teaching and learning. Moving forward, research 22 should not be framing whether AI can replace classroom observation, but how AI-assisted 23 observations can be designed so that teachers can retain judgement where it matters and freed 24 from workload where AI can demonstrate reliability. 25 26 21 References 1 Allen, J., Gregory, A., Mikami, A., Lun, J., Hamre, B., & Pianta, R. (2013). Observations of 2 effective teacher-student interactions in secondary school classrooms: Predicting 3 student achievement with the Classroom Assessment Scoring System-4 Secondary. School Psychology Review, 42(1), 76â98. 5 https://doi.org/10.1080/02796015.2013.1208749 6 Ault, M. J., Spriggs, A. D., Bausch, M. E., & Courtade, G. R. (2019). Evaluation of remote 7 versus face-to-face observation of teacher candidates in an alternative certification 8 program. Rural Special Education Quarterly, 38(3), 124â136. 9 https://doi.org/10.1177/8756870519861030 10 Barri, M. A. F., & Hikmawan, R. (2026). Human-GenAI score alignment in rubric-11 constrained essay assessment: Procedural convergence without pedagogical 12 equivalence. Journal of Informatics and Vocational Education, 9(2), 13â22. 13 Bell, C. A., Dobbelaer, M. J., Klette, K., & Visscher, A. (2019). Qualities of classroom 14 observation systems. School Effectiveness and School Improvement, 30(1), 329. 15 https://doi.org/10.1080/09243453.2018.1539014 16 Bell, C. A., Gitomer, D. H., McCaffrey, D. F., Hamre, B. K., Pianta, R. C., & Qi, Y. (2012). 17 An argument approach to observation protocol validity. Educational 18 Assessment, 17(2-3), 62â87. https://doi.org/10.1080/10627197.2012.715014 19 Bettayeb, A. M., Abu Talib, M., Sobhe Altayasinah, A. Z., & Dakalbab, F. (2024). Exploring 20 the impact of ChatGPT: Conversational AI in education. Frontiers in Education, 9. 21 https://doi.org/10.3389/feduc.2024.1379796 22 Borg, S. (2021). Video-based observation in impact evaluation. Evaluation and Program 23 Planning, 89, 102007. https://doi.org/10.1016/j.evalprogplan.2021.102007 24 Broad, J. H. (2015). So many worlds, so much to do: Identifying barriers to engagement with 25 continued professional development for teachers in the further education and training 26 sector. London Review of Education, 13(1), 16â30. 27 https://doi.org/10.18546/LRE.13.1.03 28 Bronfenbrenner, U., & Morris, P. A. (1998). The ecology of developmental processes. In W. 29 Damon & R. M. Lerner (Eds.), Handbook of child psychology: Theoretical models of 30 human development (5th ed., p. 993â1028). John Wiley & Sons, Inc. 31 Bukhalenkova, D., Almazova, O., & Aslanova, M. (2023). Similarities and differences 32 between CLASS and ECERS-R estimates of educational environment 33 quality. Frontiers in Psychology, 14, 1253154. 34 https://doi.org/10.3389/fpsyg.2023.1253154 35 Calvo, R. A., & DâMello, S. (2010). Affect detection: An interdisciplinary review of models, 36 methods, and their applications. IEEE Transactions on Affective Computing, 1(1), 18â37 37. https://doi.org/10.1109/T-AFFC.2010.1. 38 Chen, X., Zou, D., Xie, H., Cheng, G., & Liu, C. (2022). Two decades of artificial 39 intelligence in education. Educational Technology & Society, 25(1), 28â47. 40 https://w.jstor.org/stable/48647028 41 Chu, Z., Wang, S., Xie, J., Zhu, T., Yan, Y., Ye, J., Zhong, A., Hu, X., Liang, J., Yu, P. S., & 42 Wen, Q. (2025). LLM agents for education: Advances and applications 43 (arXiv:2503.11733). arXiv. https://doi.org/10.48550/arXiv.2503.11733 44 Cicchetti, D. V. (1994). Guidelines, criteria, and rules of thumb for evaluating normed and 45 standardized assessment instruments in psychology. Psychological Assessment, 6(4), 46 284â290. https://doi.org/10.1037/1040-3590.6.4.284 47 22 Cohen, J., & Goldhaber, D. (2016). Building a more complete understanding of teacher 1 evaluation using classroom observations. Educational Researcher, 45(6), 378â387. 2 https://doi.org/10.3102/0013189X1665944 3 Cohen, D. K., Raudenbush, S. W., & Ball, D. L. (2003). Resources, instruction, and 4 research. Educational Evaluation and Policy Analysis, 25(2), 119â142. 5 https://doi.org/10.3102/0002831207312905 6 Curby, T. W., Johnson, P., Mashburn, A. J., & Carlis, L. (2016). Live versus video 7 observations: Comparing the reliability and validity of two methods of assessing 8 classroom quality. Journal of Psychoeducational Assessment, 34(8), 765â781. 9 https://doi.org/10.1177/0734282915627115 10 Desimone, L. M., Litke, E., & Snipes, J. (2025). Considerations in using classroom 11 observations to measure the effects of professional learning. In S. Kelly (Ed.), 12 Research handbook on classroom observation (p. 245â258). Edward Elgar 13 Publishing. 14 Downer, J. T., LĂłpez, M. L., Grimm, K. J., Hamagami, A., Pianta, R. C., & Howes, C. (2012). 15 Observations of teacher-child interactions in classrooms serving Latinos and dual 16 language learners: Applicability of the Classroom Assessment Scoring System in 17 diverse settings. Early Childhood Research Quarterly, 27(1), 21â32. 18 https://doi.org/10.1016/j.ecresq.2016.08.005 19 Duan, Y., Edwards, J. S., & Dwivedi, Y. (2019). Artificial intelligence for decision making in 20 the era of Big Data-evolution, challenges and research agenda. International Journal 21 of Information Management, 48. https://doi.org/10.1016/j.ijinfomgt.2019.01.021. 22 Elbaum, B., Perry, L. K., & Messinger, D. S. (2024). Investigating childrenâs interactions in 23 preschool classrooms: An overview of research using automated sensing 24 technologies. Early Childhood Research Quarterly, 66, 147â156. 25 https://doi.org/10.1016/j.ecresq.2023.10.005. 26 Fairman, J. C., Smith, D. J., Pullen, P. C., & Lebel, S. J. (2022). The challenge of keeping 27 teacher professional development relevant. Professional Development in Education, 28 49(2), 197â209. https://doi.org/10.1080/19415257.2020.1827010 29 Gemini Team. (2023). Gemini: A family of highly capable multimodal models. arXiv. 30 Grimm, E. D., Kaufman, T., & Doty, D. (2014). Rethinking Classroom 31 Observation. Educational Leadership, 71(8), 24â29. 32 Hafen, C. A., Hamre, B. K., Allen, J. P., Bell, C. A., Gitomer, D. H., & Pianta, R. C. (2015). 33 Teaching through interactions in secondary school classrooms: Revisiting the factor 34 structure and practical application of the Classroom Assessment Scoring System-35 Secondary. The Journal of Early Adolescence, 35(5-6), 651â680. 36 https://doi.org/10.1177/0272431614537117 37 Hamre, B. K., Pianta, R. C., Downer, J. T., DeCoster, J., Mashburn, A. J., Jones, S. M., 38 Brown, J. L., Cappella, E., Atkins, M., Rivers, S. E., Brackett, M. A., & Hamagami, 39 A. (2013). Teaching through interactions: Testing a developmental framework of 40 teacher effectiveness in over 4,000 classrooms. The Elementary School 41 Journal, 113(4), 461â487. https://doi.org/10.1086/669616 42 Hamre, B., Hatfield, B., Pianta, R., & Jamil, F. (2014). Evidence for general and domain-43 specific elements of teacher-child interactions: Associations with preschool childrenâs 44 development. Child Development, 85(3), 1257â1274. 45 https://doi.org/10.1111/cdev.12184 46 Harms, T., Clifford, R. M., & Cryer, D. (1998). Early Childhood Environment Rating Scale. 47 Teachers College Press, Columbia University, New York. 48 23 Hoang, N., Holopainen, L., & Siekkinen, M. (2019). Classroom quality profiles and 1 associations with childrenâs classroom engagement in Vietnamese 2 kindergartens. Learning Environments Research, 22(2), 193â208. 3 https://doi.org/10.1007/s10984-018-9273-1 4 Hou, R., FĂŒtterer, T., BĂŒhler, B., Bozkir, E., Gerjets, P., Trautwein, U., & Kasneci, E. (2024). 5 Automated assessment of encouragement and warmth in classrooms leveraging 6 multimodal emotional features and ChatGPT [Conference presentation]. International 7 Conference on Artificial Intelligence in Education, Switzerland. 8 https://doi.org/10.48550/arXiv.2404.15310 9 Hu, B. Y., Fan, X., Gu, C., & Yang, N. (2016). Applicability of the classroom assessment 10 scoring system in Chinese preschools based on psychometric evidence. Early 11 Education and Development, 27(5), 714â734. 12 https://doi.org/10.1080/10409289.2016.1113069 13 Huang, X., Zou, D., Cheng, G., Chen, X., & Xie, H. (2023). Trends, research issues and 14 applications of artificial intelligence in language education. Educational Technology 15 & Society, 26(1), 112â131. https://w.jstor.org/stable/10.2307/48707971 16 Ing, M. (2010). Using informal classroom observations to improve instruction. Journal of 17 Educational Administration, 48(3), 337â358. 18 https://doi.org/10.1108/09578231011041053. 19 Karuppiah, N. (2021). Enhancing the quality of teacher-child interactions in Singapore pre-20 school classrooms. Journal of Childhood, Education & Society, 2(1), 58â68. 21 https://doi.org/10.37291/2717638X.20212187 22 Kasneci, E., Sessler, K., KĂŒchemann, S., Bannert, M., Dementieva, D., Fischer, F., Gasser, U., 23 Groh, G., GĂŒnnemann, S., HĂŒllermeier, E., Krusche, S., Kutyniok, G., Michaeli, T., 24 Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., Stadler, M., & 25 Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large 26 language models for education. Learning and Individual Differences, 103, 102274. 27 https://doi.org/10.1016/j.lindif.2023.102274 28 Kelly, S., Olney, A. M., Donnelly, P., Nystrand, M., & DâMello, S. K. (2018). Automatically 29 measuring question authenticity in real-world classrooms. Educational 30 Researcher, 47(7), 451â464. https://doi.org/10.3102/0013189X18785613 31 La Paro, K. M., Pianta, R., C., & Stuhlman, M. (2004). The Classroom Assessment Scoring 32 System: Findings from the prekindergarten year. The Elementary School Journal, 33 104(5), 409â426. https://doi.org/10.1086/499760 34 Landis, J. R., & Koch, G. G. (1977). An application of hierarchical Kappa-type statistics in 35 the assessment of majority agreement among multiple observers. Biometrics, 363â36 374. https://w.jstor.org/stable/2529786. 37 Leyva, D., Weiland, C., Barata, M., Yoshikawa, H., Snow, C., Treviño, E., & Rolla, A. (2015). 38 Teacher-child Interactions in Chile and their associations with prekindergarten 39 outcomes. Child Development, 86(3), 781â799. https://doi.org/10.1111/cdev.12342 40 Li, X., Huang, R., Bao, Y., Jin, Y., Jiao, Y., & Hu, Q. (2026). When AI meeting early 41 childhood education: Large language models as assessment teammates in Chinese 42 preschools. (arXiv: arXiv:2603.24389). arXiv. 43 https://doi.org/10.48550/arXiv.2503.11733 44 Li, Y. L. (2006). Classroom organization: Understanding the context in which children are 45 expected to learn. Early Childhood Education Journal, 34(1), 37â43. 46 https://doi.org/10.1007/s10643-006-0120-0 47 Liao, X., Zhang, X., Wang, Z., & Luo, H. (2024). Design and implementation of an AIâ48 enabled visual report tool as formative assessment to promote learning achievement 49 24 and selfâregulated learning: An experimental study. British Journal of Educational 1 Technology, 55(3), 1253â1276. https://doi.org/10.1111/bjet.13424 2 Liebenow, L. W., Schmidt, F. T., Meyer, J., & Fleckenstein, J. (2025). Self-assessment 3 accuracy in the age of artificial intelligence: Differential effects of LLM-generated 4 feedback. Computers & Education, 105385. 5 https://doi.org/10.1016/j.compedu.2025.105385 6 Mackey, A. (2017). Classroom-based research. In S. Loewen, & M. Sato (Eds.), The 7 Routledge handbook of instructed second language acquisition (p. 541â561). 8 Routledge. https://doi.org/10.4324/9781315676968 9 Matsumura, L. C., Garnier, H. E., & Resnick, L. B. (2010). Implementing literacy coaching: 10 The role of school social resources. Educational Evaluation and Policy 11 Analysis, 32(2), 249â272. https://doi.org/10.3102/0162373710363743 12 Mirchandani, S., Xia, F., Florence, P., Ichter, B., Driess, D., Arenas, M. G., Rao, K., Sadigh, 13 D., & Zeng, A. (2023). Large language models as general pattern machines. arXiv. 14 https://doi.org/10.48550/arXiv.2307.04721 15 Moore, R. L., & Lee, S. S. (2024). Harnessing generative AI (GenAI) for automated feedback 16 in higher education: A systematic review. Online Learning Journal, 28(3), 82â104. 17 https://doi.org/ 10.24059/olj.v28i3.4593 18 Owan, V. J., Abang, K. B., Idika, D. O., Etta, E. O., & Bassey, B. A. (2023). Exploring the 19 potential of artificial intelligence tools in educational measurement and 20 assessment. Eurasia Journal of Mathematics, Science and Technology 21 Education, 19(8). https://doi.org/10.29333/ejmste/13428 22 Page, L. (2016). The impact of lesson observation on practice, professionalism and teacher 23 identity. In M. OâLeary (Ed.), Reclaiming Lesson Observation (p. 62â74). 24 Routledge. https://doi.org/10.4324/9781315621838 25 Pakarinen, E., Lerkkanen, M. K., Poikkeus, A. M., Kiuru, N., Siekkinen, M., Rasku-Puttonen, 26 H., & Nurmi, J. E. (2010). A validation of the Classroom Assessment Scoring System 27 in Finnish kindergartens. Early Education and Development, 21(1), 95â124. 28 https://doi.org/10.1080/10409280902858764 29 Pang, S., Lai, S., Zhang, A., Yang, Y., & Sun, D. (2023). Graph convolutional network for 30 automatic detection of teachersâ nonverbal behavior. Computers and Education: 31 Artificial Intelligence, 5, 100174. https://doi.org/10.1016/j.caeai.2023.100174 32 Park, Y.S., Chen, J., & Holtzman, S.L. (2015). Evaluating efforts to minimize rater bias in 33 scoring classroom observations. In T.J. Kane, K.A. Kerr & R.C. Pianta (Eds.), 34 Designing teacher evaluation systems: New guidance from the Measures of Effective 35 Teaching Project (p. 381â414). https://doi.org/10.1002/9781119210856.ch12 36 Pianta, R. C., Hamre, B. K., & Allen, J. P. (2012). Teacher-student relationships and 37 engagement: Conceptualizing, measuring, and improving the capacity of classroom 38 interactions. In S. L. Christenson, A. L. Reschly, & C. Wylie (Eds.), Handbook of 39 research on student engagement (p. 365â386). Boston, MA: Springer US. 40 Pianta, R. C., La Paro, K. M., & Hamre, B. K. (2008a). Classroom Assessment Scoring 41 Systemâą: Manual Pre-K. Paul H Brookes Publishing. 42 Pianta, R. C., La Paro, K. M., Payne, C., Cox, M. J., & Bradley, R. (2002). The relation of 43 kindergarten classroom environment to teacher, family, and school characteristics and 44 child outcomes. The Elementary School Journal, 102(3), 225â238. 45 https://w.jstor.org/stable/1002217 46 Pianta, R. C., Mashburn, A. J., Downer, J. T., Hamre, B. K., & Justice, L. (2008b). Effects of 47 web-mediated professional development resources on teacherâchild interactions in 48 25 pre-kindergarten classrooms. Early Childhood Research Quarterly, 23(4), 431â451. 1 https://doi.org/10.1016/j.ecresq.2008.02.001 2 Riggs, V. (2025). Impact of AI literacy professional development on teaching: Evaluating 3 changes in perceptions and classroom practices. American Journal of STEM 4 Education, 7, 37â50. https://doi.org/10.32674/rbajg012 5 Sandilos, L. E., & DiPerna, J. C. (2011). Interrater reliability of the Classroom Assessment 6 Scoring System-Pre-K (CLASS Pre-K). Journal of Early Childhood & Infant 7 Psychology, 7, 65â85. 8 Shapsough, S., & Zualkernan, I. (2018, October). Using machine learning to automate 9 classroom observation for low-resource environments. [Conference 10 presentation]. 2018 IEEE Global Humanitarian Technology Conference (GHTC 11 2018), San Jose, CA, USA. https://doi.org/10.1109/GHTC.2018.8601902 12 Siddiqua, A. (2019). Classroom observation as a tool for professional development. World 13 Journal of English Language, 9(1). https://doi.org/10.5430/wjel.v9n1p49 14 Singh, S., Singh, L., & Satsangee, N. (2025). Automated assessment of classroom interaction 15 based on verbal dynamics: A deep learning approach. SN Computer Science, 6(3). 16 https://doi.org/10.1007/s42979-025-03770-3 17 Speer, A. B., Delacruz, A. Y., Chawota, T. A., Perrotta, J., & Rudolph, C. W. (2026). 18 Unpacking the validity of open-ended personality assessments using fine-tuned large 19 language models. Organizational Research Methods, 0(0), 1â40. 20 https://doi.org/10.1177/10944281251413746 21 Stallings, J. (1977). Learning to look: A handbook on classroom observation and teaching 22 models. Wadsworth Publishing Company. 23 Tarusha, F., & Bushi, J. (2024). The role of classroom observation, its impact on improving 24 teacherâs teaching practices. European Journal of Theoretical and Applied 25 Sciences, 2(2), 718â723. https://doi.org/10.59324/ejtas.2024.2(2).63 26 Tengberg, M., Blikstad-Balas, M., & Roe, A. (2022). Missed opportunities of text-based 27 instruction: What characterizes learning of interpretation if strategies are not taught 28 and students not challenged? Teaching and Teacher Education, 115, 103698. 29 https://doi.org/10.1016/j.tate.2022.103698. 30 Topol, E. J. (2019). High-performance medicine: The convergence of human and artificial 31 intelligence. Nature Medicine, 25, 44â56. https://doi.org/10.1038/s41591-018-0300-7 32 Van der Lans, R. M., Van de Grift, W. J., & van Veen, K. (2018). Developing an instrument 33 for teacher feedback: Using the Rasch model to explore teachersâ development of 34 effective teaching strategies and behaviors. The Journal of Experimental Education, 35 86(2), 247â264. https://doi.org/10.1080/00220973.2016.1268086 36 Wang, D., & Chen, G. (2025). Evaluating the use of BERT and Llama to analyse classroom 37 dialogue for teachersâ learning of dialogic pedagogy. British Journal of Educational 38 Technology, 56(6), 2671â2704. https://doi.org/ 10.1111/bjet.13604 39 Wang, D., Tao, Y., & Chen, G. (2024). Artificial intelligence in classroom discourse: A 40 systematic review of the past decade. International Journal of Educational 41 Research, 123, 102275. https://doi.org/10.1016/j.ijer.2023.102275 42 Wang, W., & Day, C. (2001, Feb 27âMar 3). Issues and concerns about classroom 43 observation: Teachers' perspectives. [Conference presentation]. Annual Meeting of 44 the Teachers of English to Speakers of Other Languages (TESOL), St. Louis, MO, 45 United States. 46 Whitehill, J., & LoCasale-Crouch, J. (2024). Automated evaluation of classroom instructional 47 support with LLMs and BoWs: Connecting global predictions to specific 48 26 feedback. Journal of Educational Data Mining, 16(1), 34â60. 1 https://doi.org/10.5281/zenodo.10974824 2 Wylie, E. C., Lyon, C. J., & OÊŒDwyer, E. P. (2025). The benefits of peer teacher observations. 3 In S. Kelly (Ed.), Research handbook on classroom observation (p. 310â325). 4 Edward Elgar Publishing. 5 Zan, B., & Donegan-Ritter, M. (2014). Reflecting, coaching and mentoring to enhance 6 teacherâchild interactions in Head Start classrooms. Early Childhood Education 7 Journal, 42(2), 93â104. https://doi.org/10.1007/s10643-013-0592-7 8 9