Paper deep dive

Automatically Inferring Teachers' Geometric Content Knowledge: A Skills Based Approach

Ziv Fenigstein, Kobi Gal, Avi Segal, Osama Swidan, Inbal Israel, Hassan Ayoob

Year: 2026Venue: arXiv preprintArea: cs.CYType: PreprintEmbeddings: 45

Intelligence

Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%

Last extracted: 4/18/2026, 1:29:24 AM

Summary

This paper presents an automated approach for classifying teachers' Van Hiele geometric reasoning levels using Large Language Models (LLMs). By decomposing the Van Hiele model into 33 fine-grained reasoning skills, the authors developed a skills-aware classification framework using Retrieval-Augmented Generation (RAG) and Multi-Task Learning (MTL). The study demonstrates that integrating explicit skills information significantly improves classification accuracy compared to baseline models, providing a scalable, theory-grounded tool for teacher assessment.

Entities (5)

Multi-task learning · ai-method · 100%Retrieval-Augmented Generation · ai-method · 100%Van Hiele model · educational-theory · 100%Ziv Fenigstein · researcher · 100%Skills Dictionary · datasetresource · 95%

Relation Signals (3)

Skills Dictionary → improves → Van Hiele classification

confidence 98% · Results showed that for both methods, skills-aware variants significantly outperformed baselines

Retrieval-Augmented Generation → classifies → Van Hiele level

confidence 95% · The first method uses a Retrieval-Augmented Generation (RAG) pipeline that receives a new question-response pair as input... to diagnose the Van Hiele level

Multi-task learning → classifies → Van Hiele level

confidence 95% · The second classification method... fine-tunes an open-source LLM for Van Hiele classification.

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Assessing teachers' geometric content knowledge is essential for geometry instructional quality and student learning, but difficult to scale. The Van Hiele model characterizes geometric reasoning through five hierarchical levels. Traditional Van Hiele assessment relies on manual expert analysis of open-ended responses. This process is time-consuming, costly, and prevents large-scale evaluation. This study develops an automated approach for diagnosing teachers' Van Hiele reasoning levels using large language models grounded in educational theory. Our central hypothesis is that integrating explicit skills information significantly improves Van Hiele classification. In collaboration with mathematics education researchers, we built a structured skills dictionary decomposing the Van Hiele levels into 33 fine-grained reasoning skills. Through a custom web platform, 31 pre-service teachers solved geometry problems, yielding 226 responses. Expert researchers then annotated each response with its Van Hiele level and demonstrated skills from the dictionary. Using this annotated dataset, we implemented two classification approaches: (1) retrieval-augmented generation (RAG) and (2) multi-task learning (MTL). Each approach compared a skills-aware variant incorporating the skills dictionary against a baseline without skills information. Results showed that for both methods, skills-aware variants significantly outperformed baselines across multiple evaluation metrics. This work provides the first automated approach for Van Hiele level classification from open-ended responses. It offers a scalable, theory-grounded method for assessing teachers' geometric reasoning that can enable large-scale evaluation and support adaptive, personalized teacher learning systems.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

45,067 characters extracted from source content.

Expand or collapse full text

Automatically Inferring Teachers’ Geometric Content Knowledge: A Skills Based Approach Ziv Fenigstein 1[0009-0002-2237-1120] , Kobi Gal 1,2[0000-0001-7187-8572] , Avi Segal 1[0000-0003-1422-2598] , Osama Swidan 1[0000-0002-2689-7173] , Inbal Israel 1[0009-0005-3213-8922] , and Hassan Ayoob 1[0009-0005-7726-6602] 1 Ben-Gurion University, Israel zivfenig@post.bgu.ac.il, kobig@bgu.ac.il, avise@post.bgu.ac.il, osamas@bgu.ac.il, inbalalmasy@gmail.com, Hassan.ayoob@gmail.com 2 University of Edinburgh, U.K. kgal@ed.ac.uk Abstract.Assessing teachers’ geometric content knowledge is essential for geometry instructional quality and student learning, but difficult to scale. The Van Hiele model characterizes geometric reasoning through five hierarchical levels. Traditional Van Hiele assessmentrelies on manual expert analysis of open-ended responses. This process is time-consuming, costly, and prevents large-scale evaluation. This study develops an au- tomated approach for diagnosing teachers’ Van Hiele reasoning levels using large language models grounded in educational theory. Our central hypothesis is that integrating explicit skills information significantly im- proves Van Hiele classification. In collaboration with mathematics educa- tion researchers, we built a structured skills dictionary decomposing the Van Hiele levels into 33 fine-grained reasoning skills. Through a custom web platform, 31 pre-service teachers solved geometry problems, yield- ing 226 responses. Expert researchers then annotated each response with its Van Hiele level and demonstrated skills from the dictionary. Using this annotated dataset, we implemented two classification approaches: (1) retrieval-augmented generation (RAG) and (2) multi-task learning (MTL). Each approach compared a skills-aware variant incorporating the skills dictionary against a baseline without skills information. Re- sults showed that for both methods, skills-aware variants significantly outperformed baselines across multiple evaluation metrics. This work provides the first automated approach for Van Hiele level classification from open-ended responses. It offers a scalable, theory-grounded method for assessing teachers’ geometric reasoning that can enable large-scale evaluation and support adaptive, personalized teacher learning systems. Keywords:AI in teachers training · Geometric Reasoning · LLMs in Education The work is accepted for publication as a full paper (Main Track) at the 27th International Conference on Artificial Intelligence in Education (AIED 2026). arXiv:2604.13666v1 [cs.CY] 15 Apr 2026 2Z. Fenigstein et al. 1 Introduction Teachers’ content knowledge (CK) in mathematics has been shown to directly impact students’ learning outcomes [ 5,6,21]. Within the realm of Geometry, pre- service teachers often demonstrate lower competency than expected for effective instruction [ 2,18,22]. Understanding teachers’ content knowledge in this domain is therefore critical for both research and professional development. The Van Hiele model of geometric thought characterizes geometry content knowledge through five hierarchical levels of reasoning ranging from visualiza- tion to formal rigor [ 13]. Building on this framework, researchers have argued that Van Hiele assessment should focus on analyzing learners’ responses to open- ended questions rather than focus on the correctness of their answers [ 7,15]. Al- though widely used in education research, existing approaches for assessing Van Hiele levels rely primarily on manual response analysis, which is time-consuming, costly, and hard to scale [ 18,21,22]. This paper addresses this gap by using Large Language Models(LLMs) to classify teachers’ Van Hiele levels from their open-ended responses. The central hypothesis of our work was that integrating explicit skillsinformation would sig- nificantly improve Van Hiele classification. To this end, we constructed a struc- tured skills dictionary in collaboration with mathematicseducation researchers, decomposing the five Van Hiele levels into a total of 33 fine-grained reasoning skills that characterize each level. We first conducted a data collection study with 31 pre-service math teachers who provided open-ended responses to geometry problems through a custom web-based platform, yielding 226question-response pairs. Expert researchers in mathematics education annotated each pair with its Van Hiele level and identified which of that level’s associated skills from the skills dictionary were demonstrated in the response. We then developed several classification models that, given a question-response pair, diagnose the Van Hiele level reflected in the response rather than just surface correctness. We considered several approaches for classification, (1) retrieval-augmented generation (RAG), which incorporates skills by retrievingannotated examples with skills labels and including the skills dictionary in the model’s prompt, and (2) multi-task learning, which incorporates skills through attention mechanisms and an auxiliary skills prediction task alongside the primary Van Hiele level clas- sification. For each classification approach, we compared a skills-aware variant against a baseline without skills information, isolating the contribution of explicit skills modeling to classification performance. For both classification approaches, the skills-aware variants significantly outperformed the baseline variants on held out test-sets. To better understand the models’ behavior, we examined the mod- els’ sensitivity to skill definitions in the RAG approach, the contribution of in- dividual skill components in the multi-task learning approach, and classification difficulty patterns across Van Hiele levels. Our work makes several contributions. (1) it is the first automated approach for inferring Van Hiele levels from open-ended responses, addressing a scalability challenge in geometry education research; (2) it provides two novel classification approaches demonstrating that explicitly modeling teachers’ skills significantly Automatically Inferring Teachers’ Geometric Content Knowledge3 improves classification regardless of the modeling paradigm; and (3) it provides new data sources for researchers in collaboration with mathematics education researchers: a theoretically grounded skills dictionary and an annotated dataset of 226 question-response pairs from pre-service teachers.Our code, models, data, prompts, and skills dictionary are publicly available in GitHub 3 . 2 The Van Hiele Model of Geometric Reasoning The Van Hiele model [ 13] describes five hierarchical levels of geometric reasoning demonstrated by learners. Advancement through levels depends on mastery of the preceding levels. At the most basic level (Level 1, Visualization), learners identify shapes based on their overall appearance. For example, a student rec- ognizes a rectangle as visually distinct from a trapezoid. At Level 2 (Analysis), learners recognize and describe properties of geometric shapes. For example, a student might state that a square has four right angles and all sides of equal length. Level 3 (Informal Deduction) involves reasoning about relationships be- tween properties and between shape classes. For example, a student might con- clude that a rectangle is a type of parallelogram because it possesses all the defining properties of parallelograms. At Level 4 (Deduction), learners construct formal proofs based on definitions, axioms, and theorems. For example, a student can prove that the opposite angles of a parallelogram are congruent. The highest level (Level 5, Rigor) entails abstract reasoning in non-Euclidean or axiomatic systems. For example, a student analyzes how geometric properties, concepts, or proofs change when axioms are modified or when comparing Euclidean and non-Euclidean geometries [ 17]. Although the Van Hiele model was originally de- veloped to characterize geometric reasoning in school-aged students, its levels reflect instructional experience rather than age or maturation [7]. This makes it applicable to learners at any stage, including teachers. Mayberry [ 23] confirmed this empirically, showing that the Van Hiele hierarchy holds for undergraduate preservice teachers. Since teachers’ geometric content knowledge directly shapes their students’ learning opportunities [ 18,4], understanding teachers’ Van Hiele levels is essential for improving geometry instruction. The model has therefore been applied to teachers for a range of purposes: (1) inferring their content knowledge in geometry [ 18,22]; (2) measuring the relationship between their ge- ometric reasoning and student achievement [ 21]; and (3) studying the effects of pedagogical interventions on their content knowledge [ 2,25], and examining their ability to diagnose students’ geometric understanding [31]. All of these studies have relied on either manual expert assessment of open-ended responses or multiple-choice questionnaires. Manual assessment is atime-consuming and resource-intensive process that limits sample sizes and prevents large-scale longi- tudinal tracking of teacher development. Automating this classification enables researchers to scale up their studies significantly. Usiskin [ 29] developed a multiple-choice test where learners at a givenVan Hiele level were expected to correctly answer items at that level and all preced- 3 https://github.com/zivfenig/Van-Hiele-Level-Classification 4Z. Fenigstein et al. ing levels. However, researchers have critiqued this approach, arguing that Van Hiele levels comprise multiple distinct reasoning processes [ 7,11]. Therefore, ac- curate assessment requires evaluating the specific skills demonstrated in learners’ explanations, not merely correctness on multiple-choice items. This skills-based perspective directly motivates our modeling approach of decomposing each Van Hiele level into explicit, fine-grained reasoning skills. For example, consider the question:“Given a shape with four equal sides, is it necessarily a rhombus?”One student might respond,“No, maybe it is a square”. This reflects Level 2 reasoning (Analysis) demonstrating the Level 2 skill of describing shapes in terms of their properties but not the Level 3 skill of recognizing that squares are a special case of rhombus. Another student might say,“Yes, because a square is also a type of rhombus”, demonstrating Level 3 type reasoning (Informal Deduction) by explicitly understanding the inclusion relationship, that one shape class can be a subset of another. Building on this paradigm, our work develops a structured skills dictionary decomposing each Van Hiele level into explicit reasoning skills in collabora- tion with mathematics education researchers. This skill decomposition forms the foundation of our automated classification approach for answers to open-ended questions (the full skills dictionary available in our repository 4 ). 3 Related Work Recent advances have made LLMs increasingly effective for educational assess- ment. Henkel et al. [ 12] demonstrated that LLMs can achieve near-parity with human raters when grading open-ended reading comprehension responses, dis- tinguishing between fully correct, partially correct, andincorrect responses. Be- yond correctness, Lee et al. [ 19] utilized LLMs to classify middle-school writ- ten responses to science assessments into proficiency levels (Proficient, Devel- oping, and Beginning), finding that Chain-of-Thought prompting with explicit rubrics improved classification accuracy across proficiency levels. However, LLMs can struggle with nuanced reasoning assessment. Rachmatullah et al. [ 24] used LLMs to assess teachers’ Pedagogical Content Knowledge (PCK). They found that while LLMs achieved high reliability when scoring lesson plans, they strug- gled to assess teachers’ analyses of student misconceptions, relying heavily on keyword matching and therefore failed to accurately identify more nuanced re- sponses from teachers which did not include those keywords.This highlights the challenge of capturing nuanced reasoning. Our skills-aware approach addresses this by explicitly defining the reasoning patterns that distinguish between Van Hiele levels. While these studies demonstrate LLMs’ potential for educational as- sessment, to our knowledge no prior work has applied LLMs or machine learning to assess geometric reasoning under the Van Hiele framework. RAG has proven effective for classifying open-ended studentresponses. Fa- teen et al. [ 9] employed RAG to retrieve similar student answers as few-shot 4 https://github.com/zivfenig/Van-Hiele-Level-Classification Automatically Inferring Teachers’ Geometric Content Knowledge5 examples for automatic short-answer scoring, improving performance over base- line methods. Additionally, Jauhiainen and Guerra [ 16] utilized RAG to retrieve relevant reference material from scholarly articles for grading university exami- nation answers. We build on these approaches by augmenting retrieved examples with explicit skills annotations, providing pedagogically grounded context that guides classification beyond surface similarity. Multi-task learning is a well-established and effective paradigm across AI domains. Xu et al. [ 30] demonstrated that attention mechanisms informed by auxiliary task outputs improve performance by incorporating structured do- main knowledge in image classification and segmentation tasks. This approach has been successfully applied to education research. Huanget al. [ 14] showed that jointly learning multiple teacher-question classification tasks improves per- formance through shared semantic representations. An et al. [1] and Geden et al. [ 10] demonstrated that auxiliary objectives - such as option tracing or per- item prediction, lead to more accurate and stable student assessment. Inspired by these works, we apply multi-task learning where skills serve as an auxiliary prediction task alongside Van Hiele level classification. 4 Methodology In this section, we present our methodology to automatically classify teachers’ Van Hiele levels using skills-aware modeling approaches (see Figure 1). We first developed a comprehensive geometry question bank and structured skills dictio- nary grounded in Van Hiele theory. We then collected and annotated a dataset of teacher responses with expert-assigned Van Hiele levels and demonstrated skills. Using this annotated dataset we developed the two classification methods - one using retrieval-augmented generation, the other using multi-task learning - each compared against a baseline without skills information to isolate the contribu- tion of explicit skills modeling. We studied whether the skills-aware variant of both methods outperformed the baselines. Step 1: Design Resources Question Bank Development Skills Dictionary (Grounded in Van Hiele model) Step 2: Studies With Pre-Service Teachers Step 3: Data Collection && Annotation Question Answer Demonstrated Skills Van Hiele Level Step 4: Van Hiele Classification Models Method I: Retrieval-Augmented Classification Method I: Supervised Fine- Tuning Fig. 1.Overview of the proposed skills-based Van Hiele classification framework. 6Z. Fenigstein et al. 4.1 Question Bank and Skills Dictionary Development We developed a question bank building on problems from Usiskin’s Van Hiele geometry test, which consists entirely of closed-ended multiple-choice items [ 29]. We adapted items into open-ended formats to elicit richer explanations from teachers, promoting reasoning beyond recognition of fixed options. In total we constructed 59 different open-ended geometry problems. The questions addressed widely taught geometry topics such as quadrilaterals, angle relationships, triangle congruence, and similarity. Items varied in difficulty to elicit reasoning across the Van Hiele spectrum. Example of question included “List at least two properties that are shared by all squares but not shared by all rhombus, justify your choices”. We built a structured skills dictionary in collaboration with three mathemat- ics education researchers with expertise in geometry education and experience teaching and studying pre-service mathematics teachers. The skills dictionary was grounded in Crowley’s theoretical decomposition of VanHiele levels [ 7], which establishes that each level is characterized by specific observable reason- ing behaviors and distinct linguistic markers that can onlybe reliably identified from open-ended explanations. For example, Crowley explicitly identifies level- specific vocabulary and behaviors that directly correspondto skills in our dictio- nary, such as the use of logical connectives (“if. . . then”, “it follows that”) at the Informal Deduction level, and the ability to identify what is given versus what must be proved at the Deduction level. Drawing on this theoretical foundation and their pedagogical experience with pre-service teachers, the experts decom- posed each Van Hiele level into fine-grained reasoning skills that characterize that level, resulting in 33 distinct skills across the five levels. To collect teachers’ responses, we developed a custom web-based platform. The interface presents a sequence of problems from our question bank and teach- ers provide responses in free text, allowing them to articulate their full reason- ing process. For problems requiring geometric construction or manipulation, the platform embeds an interactive GeoGebra [ 26] applet alongside the question. The applet served only as a cognitive scaffold and was not usedin our model. 4.2 Data Collection and Annotation A total of 31 pre-service mathematics teachers from three regional teacher train- ing institutions participated in our study. All participants were enrolled in a geometry course. The study protocol was reviewed and approved by the Institu- tional Review Board (IRB). All participants provided informed consent before taking part in the study. The web-based platform described in Section 4.1served both for data collection and geometry practice, incorporating multiple-choice (MCQ) and open-ended problems. Each participant was assigned 20 problems randomly selected (at least 10 open-ended) from our question bank, including problems of varying difficulty designed to elicit responses across all Van Hiele levels. This ensured coverage of the full spectrum of geometric reasoning, from Automatically Inferring Teachers’ Geometric Content Knowledge7 basic visualization (Level 1) to Rigor (Level 5). The platform supported flexi- ble participation with no strict time limits and the abilityto save and resume work. For this study, we retained only open-ended responses; MCQ responses and empty submissions were excluded, yielding 226 valid question-response pairs. We employed a double-blind annotation protocol to ensure objective ground- truth labeling of Van Hiele levels. Two experts in mathematics education in- dependently reviewed every response and labeled it with theVan Hiele level without seeing the other’s labels. The inter-rater reliability was high (Cohen’s κ= 0.84) indicating substantial agreement between annotators’ initial labels. After assigning the Van Hiele level, the two experts collaboratively identified which skills from the skills dictionary were demonstrated in each response. This annotation process produced a dataset of 226 question-response pairs, each annotated with a Van Hiele level and demonstrated skills. The dataset contains responses across all five levels, with Levels 2 and 3most common and Levels 4 and 5 less represented, aligning with past studies [ 2,22]. 4.3 Van Hiele Level Classification Models We developed two methods to classify Van Hiele levels from question–response pairs. For each, we implemented a baseline without skills information and a skills-aware variant incorporating our structured skillsdictionary. All other com- ponents are identical across variants, isolating the effectof skills information. Method I: Retrieval-Augmented Classification.The first method uses a Retrieval-Augmented Generation (RAG) pipeline that receives a new question- response pair as input, retrieves similar pairs from our annotated dataset, and in- tegrates them into a prompt for LLM classification. Figure 2provides an overview of this retrieval-augmented classification architecture.Each question-response pair in the dataset is encoded separately usingmultilingual-e5-base 5 embed- ding model. All embeddings areL 2 -normalized to optimize retrieval within the RAG pipeline, enabling efficient and stable cosine similarity computation via normalized dot products [ 8]. For a new question-response pair, we construct a weighted query embedding assigning 80% weight to the response and 20% to the question.This weighting reflects the Van Hiele framework’s emphasis on reasoning expressed in responses rather than questions themselves (see discussion in Section 2). Cosine similarity is computed between the new question-response embeddings and the stored vec- tors, and the top-K (see Section 4.4for K) most similar examples are retrieved. In thebaseline variant(RAG without skills), each retrieved example includes the geometry question, the teacher’s response, and the corresponding Van Hiele level. The LLM is also supplied with a system prompt containing only the stan- dard Van Hiele level definitions. No skill annotations or skill definitions are provided to the model. In theskills-aware variant, retrieved examples addition- ally contain the annotated skills associated with each teacher’s response, and 5 https://huggingface.co/intfloat/multilingual-e5-base 8Z. Fenigstein et al. Question Response Embedding Q+A Similarity Search Van Hiele Definitions Skills Definitions Top K Retrieved Examples LLM Classifier Input Prompt Builder Labeled Dataset Predicted Van Hiele Level Output Question- Response Fig. 2.Retrieval-augmented generation architecture for Method I. the system prompt includes both Van Hiele level definitions and the full skills dictionary. All other components of the pipeline - including the input encoding, retrieval procedure, number of retrieved examples, language model, and inference settings - are the same for both variants. As a result, any observed performance differ- ences between the baseline and skills-aware versions can beattributed specifically to the presence of explicit skills information in the prompt. Method I: Multi-Task Learning.The second classification method (see Figure 3) fine-tunes an open-source LLM for Van Hiele classification.Similarly to Method I, the model receives a question-response pair as input and outputs the Van Hiele level. We implemented two variants: a baselinethat performs clas- sification without skills information during training, anda skills-aware variant that incorporates an auxiliary skills prediction task withVan Hiele classification. Both variants used the same base open-source LLM. Question Response Input Base Language Model (Gemma-3- 4B-it) LoRA Adapters Encoding Input Skills Embeddings × Encoded Input Representation Skills Attention Mechanism Input Representation + Attention Weights Auxiliary Task Head (Multi-Label Skills Head) Main Task Head (Classification Head) Combined Loss Predicted Van Hiele level Output Fig. 3.Multi-task learning architecture for Method I. In thebaseline variant, the question-response pair is encoded by the LLM and passed directly to a linear classification head that predicted the Van Hiele level. This variant does not use skills information during training. In theskills-aware variant, we augmented the baseline with a skills attention mechanism and an auxiliary skills prediction task. Like thebaseline, the question- response pair is first encoded by the LLM to produce an input representation. Automatically Inferring Teachers’ Geometric Content Knowledge9 Each Van Hiele skill from the skills dictionary is represented by a trainable em- bedding vector, initialized from text encodings of the skill descriptions using the multilingual-e5-basemodel. Attention weights are computed via dot prod- ucts between the LLM’s encoded input representation and theskill embeddings followed by a softmax normalization to produce a probability distribution over skills, quantifying how strongly the question-response pair aligns with each skill. This attention weights vector is then concatenated with theinput representation (encoded question-response pair). The enriched representation feeds two predic- tion heads in parallel: a primary Van Hiele level classification head (identical to the baseline) and an auxiliary skills prediction head that predicts which skills are demonstrated. The model is trained by optimizing a combined loss balancing the Van Hiele classification and auxiliary skills prediction tasks, weighted byλ: L total =L level +λ·L skills (1) whereL level is cross-entropy loss for Van Hiele level prediction, penalizing incor- rect classification across the five distinct levels, andL skills is binary cross-entropy loss for skills prediction, penalizing incorrect presence/absence predictions for each skill independently. See Section 4.4for implementation details. These components create complementary learning signals during training. The attention mechanism identifies which skills matter for each input, while the auxiliary prediction task ensures the model learns representations that encode these skills. The dual learning signals from the classification loss and the skills prediction loss shape both the LLM’s representations and the skills embeddings through backpropagation to better capture Van Hiele-specific reasoning patterns. At inference, both variants take the question-response pair as input and out- put the predicted Van Hiele level. The skills-aware variantalso computes skills predictions and attention weights internally, which can beaccessed for inter- pretability analysis if needed. As in Method I, all other components and hyper- parameters were the same for both variants. 4.4 Implementation Settings For Method I, all retrieval-augmented variants useGemini-2.0-Flash[ 27], cho- sen for long-context processing. Retrieval usesK= 5examples, as prior work shows this optimizes citation recall [ 20]. Generation uses temperature 0.0 for deterministic predictions [3]. Variants differ only in prompt content: the base- line includes Van Hiele level definitions and retrieved examples annotated with levels, while the skills-aware variant additionally includes the skills dictionary and skill annotations. Gemini 2.0 Flash was accessed via Vertex AI on Google Cloud Platform. For Method I, both fine-tuning variants useGemma-3-4B-IT[ 28], a mid- sized instruction-tuned model balancing capacity and efficiency for limited-data supervision. We apply LoRA (rank 16), which performed best empirically, keep- ing base weights frozen. To mitigate overfitting risk given the limited dataset size,we apply dropout in LoRA adapters (0.05) and in the Van Hiele classification head (0.25), with 10Z. Fenigstein et al. learning rate2×10 −4 and weight decay 0.05. Training runs up to 30 epochs with early stopping on macro F1 (patience=4). The skills-aware variant uses auxiliary loss weightλ= 0.5, selected empirically. Both variants were trained on an NVIDIA RTX 6000 GPU. Additional technical configurationscan be found in our GitHub repository 6 . 5 Experiments And Results We evaluated both methods by comparing their baseline and skills-aware vari- ants on classifying Van Hiele levels in our collected dataset. We used five-fold cross-validation with a fixed random seed (42) for reproducibility. The dataset was partitioned into five independent folds while preserving the Van Hiele level distribution. Each fold served as a test set in turn, with theremaining four folds used for training and retrieval (for the RAG system). Withineach training set, 15% was held out as a validation set for parameter tuning and early stopping. We report macro-averaged and weighted-averaged F1 scores (F1-macro, F1- weighted) to assess standard classification performance. F1-macro measures how consistently the model performs across Van Hiele levels by assigning equal weight to each class, while F1-weighted reflects overall performance by accounting for the empirical label distribution. To capture the ordinal structure of the Van Hiele hierarchy, we additionally report Mean Absolute Error (MAE), penalizing larger ordinal misclassifications, and Quadratic Weighted Kappa (QWK), which ac- counts for chance agreement; QWK ranges from−1to1, where values close to 1indicate stronger agreement. Figure 4shows the average results for RAG classification (Method I) and Multi-task learning methods (Method I), comparing baseline (gray bars) against skills-aware (blue bars) classification across five-fold test sets. For both methods, the skills-aware models significantly outperformed the relevant baseline in all measures. We also note that the standard deviations of the skills-aware variants in both methods were consistently lower compared to their respective baselines, suggesting the models are more stable than the baseline variants. A series of paired t-tests confirmed that the improvements ofMethod I are statistically significant across all metrics: F1-macro(t(4)=2.909, p=0.0437), F1-weighted (t(4)=3.008, p=0.0396), MAE (t(4)=-5.2, p=0.0065), and QWK (t(4)=3.465, p=0.0257). With respect to Method I, statistical significance was obtained for F1-macro (t(4)=2.908, p=0.043), MAE (t(4)=-3.836, p=0.0185) and QWK (t(4)=3.874, p=0.018), but not for F1-weighted measure. A possible reason for this is the small data set (226 question-responsepairs). 5.1 Sensitivity Analysis We performed several sensitivity analyses to study how variations in skill-related information influence model behavior. With respect to Method I, we wanted to 6 https://github.com/zivfenig/Van-Hiele-Level-Classification Automatically Inferring Teachers’ Geometric Content Knowledge11 Fig. 4.Average results across 5-fold cross-validation comparingbaseline and skills- aware variants for RAG (Method I) and MTL (Method I). Numbers above bars show mean scores; lines within the bars indicate standard deviation. For MAE measure, lower is better. validate the performance gain of the skills-aware variant to selecting theright skills for therightquestion-response pair, as opposed to simply providing the model with additional context. To this end, we compared the baseline and the skills-aware variants to a “noisy skills” variant. In this setup, we randomly shuf- fled the skills definitions, so that each skill label was assigned a definition of another skill. All other components of the skill-aware method and the empirical methodology remained the same. Table 1shows that the noisy skills variant ex- Table 1.Skills sensitivity analysis for Method I. Values are reported as mean±stan- dard deviation across five cross-validation folds. VariantF1-macroF1-weightedQWKMAE Baseline (No Skills)0.624±0.0930.678±0.0950.625±0.0350.47±0.09 Skills-Aware0.695±0.090.736±0.080.721±0.0540.376±0.06 Noisy Skills0.61±0.1220.635±0.1170.673±0.0930.495±0.156 hibited significant performance degradation across all metrics compared to the skills-aware, and even performed worse than baseline. Thisdemonstrates that performance gains depend on the model’s ability to utilize correctly aligned and pedagogically meaningful skills information, rather thansimply benefiting from additional contextual input or increased prompt length. For Method I, we quantified the individual contributions ofthe skills atten- tion mechanism and auxiliary skills prediction head by isolating each compo- nent. We compared two partial variants: (1)Attention-Guidedvariant, trained with the skills-based attention mechanism and the Van Hieleclassification head, without the auxiliary skills prediction task; (2)Skills-Supervisedvariant, trained with the auxiliary skills prediction head and the Van Hiele classification head, but without the skills-based attention mechanism. Table 2compares the full ap- 12Z. Fenigstein et al. proachFull Model (Method I)(skills-aware variant) to theAttention-Guided, theSkills-Supervisedvariants and theBaseline (No Skills)variant. Table 2.Skills component analysis for Method I. Values are reported as mean± standard deviation across five cross-validation folds. VariantF1-macroF1-weightedQWKMAE Baseline0.646±0.0820.657±0.0920.586±0.1570.523±0.119 Attention-Guided0.676±0.0580.679±0.0660.632±0.1290.487±0.086 Skills-Supervised0.652±0.0630.653±0.0880.581±0.0890.558±0.119 Full Model (Method I)0.725±0.0430.725±0.0530.717±0.1030.403±0.102 Results show the attention-guided variant clearly outperforms the baseline, while skills supervision alone yields limited gains. The full model performs best, indicating the components are complementary: the auxiliary task encourages the encoder to capture patterns aligned with pedagogical skills and Van Hiele levels, and the attention mechanism leverages these signals to focus on skills- relevant aspects of responses, producing more informativerepresentations for level classification. 5.2 Error Analysis To identify which Van Hiele levels are most difficult to classify, we analyzed per- level performance by aggregating predictions across all cross-validation folds. Both methods achieve high accuracy on Level 4. A possible reason is that Level 4 (Deduction) responses involve constructing formal proofs, which are character- ized by structured patterns that are easier for models to learn and identify. The skills dictionary likely contributes to this: Level 4 skills such as identifying given information versus what must be proved, and constructing formal logical arguments, map directly onto textual patterns that models can detect reliably. In addition, for both methods, Level 2 accuracy is around 60%, with most misclassifications occurring as Level 3. This demonstratesthe difficulty of dis- tinguishing between these adjacent levels, where differences in responses may be subtle and evident only in specific linguistic details (see example in Section 2). Level 5 is challenging for both methods but shows markedly different per- formance: Method I achieves only 31% accuracy rate while Method I achieves 69%. In both cases, predictions are distributed broadly across multiple levels rather than concentrated near the true level, indicating persistent classification difficulty. This likely stems from limited Level 5 representation in our dataset. The performance gap between the methods at Level 5 can be attributed to their fundamental difference in approach: Method I relies on retrieving similar ex- amples from the dataset, but with only 7% of responses at Level 5, similarity search rarely surfaces relevant examples, leaving the model without meaningful context for classification. Method I, by contrast, learns representations through training, enabling it to capture Level 5 patterns from sparse supervision. Automatically Inferring Teachers’ Geometric Content Knowledge13 6 Discussion and Conclusion This work demonstrates that Large Language Models can effectively infer teach- ers’ Van Hiele reasoning levels when guided by structured pedagogical infor- mation. Our central hypothesis was that integrating explicit skills information improves Van Hiele classification. We tested this with two distinct approaches: Retrieval-Augmented Generation and Multi-Task Learning.In both cases, skills- aware variants significantly outperformed baselines without skills information across multiple evaluation metrics. We hypothesize that the skills dictionary helps models identify diagnostic patterns that distinguish Van Hiele levels. These include the use of logical language like "if...then" or "therefore," or linking prop- erties across shape families. By defining reasoning patterns characteristic of each level, the skills help models attend to features that differentiate between levels. Our research has practical implications for mathematics education research and professional development. By automating Van Hiele assessment - tradition- ally constrained by manual expert evaluation - our approachenables researchers to study geometric reasoning development at scale across large teacher cohorts. Automated assessment could also enable adaptive professional development sys- tems that dynamically adjust content based on teachers’ current Van Hiele lev- els and skill profiles. Additionally, extending the proposed models to output demonstrated skills alongside Van Hiele levels would create fine-grained diag- nostic profiles for detecting strengths and gaps, enabling targeted interventions that address specific reasoning weaknesses. Several limitations should be noted. First, the dataset of 226 responses is relatively small, particularly for higher Van Hiele levels, which may limit model generalizability. Second, our research was conducted witha fixed set of 59 ques- tions, and models may not generalize to new problems; extending this approach to arbitrary geometry questions remains an important challenge for future work. Third, in contrast to Van Hiele levels, skills were identified through expert con- sensus rather than independent annotation; while our experiments validate their utility for Van Hiele classification, independent annotation with inter-rater reli- ability would strengthen confidence in skill labels. Beyond geometric reasoning, this work demonstrates a potentially gener- alizable approach: decomposing hierarchical learning frameworks into explicit, fine-grained skills that guide automated assessment. Whilewe validated this methodology for Van Hiele levels, the principle may extend to other structured frameworks such as Bloom’s Taxonomy or subject-specific reasoning models. Such approaches bridge AI capabilities with pedagogical theory by ensuring models assess learning using the same constructs educatorsuse. Validating this skills-based methodology remains important future work. Acknowledgments.This study was funded in part by Israeli Ministry of Science and Technology grant number 7774. Disclosure of Interests.The authors have no competing interests to declare that are relevant to the content of this article. 14Z. Fenigstein et al. References 1. An, S., Kim, J., Kim, M., Park, J.: No task left behind: Multi-task learning of knowledge tracing and option tracing for better student assessment. Proceedings of the AAAI Conference on Artificial Intelligence36(4), 4424–4431 (Jun 2022). https://doi.org/10.1609/aaai.v36i4.20364 2. Armah, R.B., Cofie, P.O., Okpoti, C.A.: Investigating theeffect of Van Hiele phase-based instruction on pre-service teachers’ geometric thinking. Interna- tional Journal of Research in Education and Science4(1), 314–330 (2018), https://eric.ed.gov/?id=EJ1169856 3. Bajan, C., Lambard, G.: Exploring the expertise of large language models in ma- terials science and metallurgical engineering. Digital Discovery4, 500–512 (2025). https://doi.org/10.1039/D4D00319E 4. Beswick, K., Goos, M.: Measuring pre-service primary teachers’ knowledge for teaching mathematics. Mathematics Teacher Education and Development14(2), 70–90 (2012) 5. Campbell, P.F., Malkus, N.N.: The impact of elementary mathematics coaches on student achievement. The elementary school journal111(3), 430–454 (2011). https://doi.org/10.1086/657654 6. Copur-Gencturk, Y., Li, J., Cohen, A.S., Orrill, C.H.: The impact of an interactive, personalized computer-based teacher professional development program on student performance: A randomized controlled trial. Computers & education210(2024). https://doi.org/10.1016/j.compedu.2023.104963 7. Crowley, M.L.: The van hiele model of the development of geometric thought. Learning and teaching geometry, K-121, 1–16 (1987) 8. Douze, M., Guzhva, A., Deng, C., Johnson, J., Szilvasy, G., Mazaré, P.E., Lomeli, M., Hosseini, L., Jégou, H.: The faiss library. IEEE Transactions on Big Data p. 1–17 (2025). https://doi.org/10.1109/TBDATA.2025.3618474 9. Fateen, M., Wang, B., Mine, T.: Beyond scores: A modular rag-based sys- tem for automatic short answer scoring with feedback. IEEE Access12(2024). https://doi.org/10.1109/ACCESS.2024.3508747 10. Geden, M., Emerson, A., Rowe, J., Azevedo, R., Lester, J.: Predictive stu- dent modeling in educational games with multi-task learning. Proceedings of the AAAI Conference on Artificial Intelligence34(01), 654–661 (Apr 2020). https://doi.org/10.1609/aaai.v34i01.5406 11. Gutiérrez, A., Jaime, A., Fortuny, J.M.: An alternativeparadigm to evaluate the acquisition of the van hiele levels. Journal for Research inMathematics Education 22(3), 237–251 (1991) 12. Henkel, O., Hills, L., Roberts, B., McGrane, J.: Can llmsgrade open response reading comprehension questions? an empirical study usingthe roars dataset. In- ternational journal of artificial intelligence in education35(2), 651–676 (2025). https://doi.org/10.1007/s40593-024-00431-z 13. van Hiele, P.: The child’s thought and geometry. ERIC Archive (translated from original Dutch) (1959), originally presented at the OEEC conference, Sèvres, 1957 14. Huang, G.Y., Chen, J., Liu, H., Fu, W., Ding, W., Tang, J.,Yang, S., Li, G., Liu, Z.: Neural multi-task learning for teacher question detection in online classrooms. In: International Conference on Artificial Intelligence inEducation. p. 269–281. Springer (2020). https://doi.org/10.1007/978-3-030-52237-7_22 15. Jaime, A., Gutiérrez, A.: A model of test design to assessthe van hiele levels. In: Proceedings of the 18th PME Conference. vol. 3, p. 41–48. Pme (1994) Automatically Inferring Teachers’ Geometric Content Knowledge15 16. Jauhiainen, J.S., Guerra, A.G.: Evaluating students’ open-ended written responses with llms: Using the rag framework for gpt-3.5, gpt-4, claude-3, and mistral-large (2024),https://arxiv.org/abs/2405.05444 17. Jupri, A.: Using the van hiele theory to analyze primary school teachers’ written work on geometrical proof problems. Journal of Physics: Conference Series1013(1), 012117 (may 2018). https://doi.org/10.1088/1742-6596/1013/1/012117 18. Kurt-Birel, G., Deniz, S., Önel, F.: Analysis of primaryschool teachers’ knowledge of geometry. International Electronic Journal of Elementary Education12(4), 303– 309 (2020). https://doi.org/10.26822/iejee.2020459459 19. Lee, G.G., Latif, E., Wu, X., Liu, N., Zhai, X.: Applying large language models and chain-of-thought for automatic scoring. Com- puters and Education: Artificial Intelligence6, 100213 (2024). https://doi.org/https://doi.org/10.1016/j.caeai.2024.100213 20. Leto, A., Aguerrebere, C., Bhati, I., Willke, T., Tepper, M., Vo, V.A.: Toward optimal search and retrieval for rag (2024), https://arxiv.org/abs/2411.07396 21. Lumbre, A.P., Beltran-Joaquin, M.N., Monterola, S.L.C.: Relationship between mathematics teachers’ van hiele levels and students’ achievement in geometry. In- ternational Journal of Studies in Education and Science (IJSES)4(2), 113–123 (2023) 22. Manero, V., Arnal-Bailera, A.: Understanding proof practices of pre-service math- ematics teachers in geometry. Mathematics Teaching-Research Journal13(3), 99– 130 (2021) 23. Mayberry, J.: The van hiele levels of geometric thought in undergraduate preser- vice teachers. Journal for research in mathematics education14(1), 58–69 (1983). https://doi.org/10.5951/jresematheduc.14.1.0058 24. Rachmatullah, A., Tayde, S., Alozie, N., et al.: Exploring large language model’s ca- pabilities in identifying science teacher pck using lessonplans and open-ended ques- tions. Disciplinary and Interdisciplinary Science Education Research8(3) (2026). https://doi.org/10.1186/s43031-025-00151-x 25. Swafford, J.O., Jones, G.A., Thornton, C.A.: Increased knowledge in geometry and instructional practice. Journal for Research in Mathematics Education28(4), 467–483 (1997). https://doi.org/10.5951/jresematheduc.28.4.0467 26. Tamam, B., Dasari, D.: The use of geogebra software in teaching mathe- matics. Journal of Physics: Conference Series1882(1), 012042 (may 2021). https://doi.org/10.1088/1742-6596/1882/1/012042 27. Team, G., et al.: Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805 (2023). https://doi.org/10.48550/arXiv.2312.11805 28. Team, G., et al.: Gemma 3 technical report. arXiv preprint arXiv:2503.19786 (2025). https://doi.org/10.48550/arXiv.2503.19786 29. Usiskin, Z.: Van hiele levels and achievement in secondary school geometry. cdassg project. ERIC (1982) 30. Xu, M., Huang, K., Qi, X.: Multi-task learning with context-oriented self-attention for breast ultrasound image classification and segmentation. In: 2022 IEEE 19th International Symposium on Biomedical Imaging (ISBI). p. 1–5 (2022). https://doi.org/10.1109/ISBI52829.2022.9761685 31. Yi, M., Flores, R., Wang, J.: Examining the influence of van hiele theory-based instructional activities on elementary preservice teachers’ geometry knowledge for teaching 2-d shapes. Teaching and Teacher Education91, 103038 (2020). https://doi.org/https://doi.org/10.1016/j.tate.2020.103038

Instant research discovery