Paper deep dive
Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation using Knowledge Graphs and Large Language Models
Mehmet Can ĹakiroÄlu, H. Altay GĂźvenir, Kamer Kaya
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 96%
Last extracted: 4/14/2026, 2:24:49 AM
Summary
This paper introduces a novel framework for generating multiple-choice questions (MCQs) with interpretable difficulty estimation by integrating knowledge graphs (KGs) and large language models (LLMs). The system constructs KGs from input documents, generates MCQs using graph-based subgraphs, and computes a unified difficulty score based on nine interpretable signals (graph topology, semantic similarity, and linguistic features) calibrated against a new human-annotated dataset.
Entities (5)
Relation Signals (3)
Large Language Model â constructs â Knowledge Graph
confidence 95% ¡ Our approach uses an LLM to construct a KG from input documents
Difficulty Estimation â evaluates â Multiple-Choice Question
confidence 95% ¡ For each MCQ, nine difficulty signals are computed and combined into a unified difficulty score
Knowledge Graph â usedfor â Multiple-Choice Question
confidence 95% ¡ Our approach uses an LLM to construct a KG from input documents, from which MCQs are then systematically generated.
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Generating multiple-choice questions (MCQs) with difficulty estimation remains challenging in automated MCQ-generation systems used in adaptive, AI-assisted education. This study proposes a novel methodology for generating MCQs with difficulty estimation from the input documents by utilizing knowledge graphs (KGs) and large language models (LLMs). Our approach uses an LLM to construct a KG from input documents, from which MCQs are then systematically generated. Each MCQ is generated by selecting a node from the KG as the key, sampling a related triple or quintuple -- optionally augmented with an extra triple -- and prompting an LLM to generate a corresponding stem from these graph components. Distractors are then selected from the KG. For each MCQ, nine difficulty signals are computed and combined into a unified difficulty score using a data-driven approach. Experimental results demonstrate that our method generates high-quality MCQs whose difficulty estimation is interpretable and aligns with human perceptions. Our approach improves automated MCQ generation by integrating structured knowledge representations with LLMs and a data-driven difficulty estimation model.
Tags
Links
- Source: https://arxiv.org/abs/2604.10748v1
- Canonical: https://arxiv.org/abs/2604.10748v1
Trouble viewing inline? Open PDF directly â
Full Text
69,419 characters extracted from source content.
Expand or collapse full text
Generating Multiple-Choice Knowledge Questions with Interpretable Difficulty Estimation using Knowledge Graphs and Large Language Models Mehmet Can ĹakiroÄlu 1 H. Altay GĂźvenir 1 Kamer Kaya 2,3â 1 Computer Engineering Department, Bilkent University, Ankara, Turkey 2 Faculty of Engineering and Natural Sciences, Sabanci University 3 VERIM, Center of Excellence in Data Analytics, Sabanci University Abstract Generating multiple-choice questions (MCQs) with difficulty estimation remains chal- lenging in automated MCQ-generation systems used in adaptive, AI-assisted education. This study proposes a novel methodology for generating MCQs with difficulty estimation from the input documents by utilizing knowledge graphs (KGs) and large language models (LLMs). Our approach uses an LLM to construct a KG from input documents, from which MCQs are then systematically generated. Each MCQ is generated by selecting a node from the KG as the key, sampling a related triple or quintupleâoptionally augmented with an extra tripleâand prompting an LLM to generate a corresponding stem from these graph compo- nents. Distractors are then selected from the KG. For each MCQ, nine difficulty signals are computed and combined into a unified difficulty score using a data-driven approach. Exper- imental results demonstrate that our method generates high-quality MCQs whose difficulty estimation is interpretable and aligns with human perceptions. Our approach improves au- tomated MCQ generation by integrating structured knowledge representations with LLMs and a data-driven difficulty estimation model. Keywords: multiple-choice question generation; difficulty estimation; interpretability; knowl- edge graph; large language models. 1 Introduction Multiple-choice questions are a common form of assessment used in educational settings due to their utility in standardized testing, self-assessment, and adaptive learning systems. A multiple- choice question typically consists of a question or statement known as the âstem" followed by several possible answers. One of these answers is the correct one, referred to as the âkey", while the others are incorrect, known as âdistractors". An example multiple-choice question whose parts are annotated can be seen in Figure 1. The effectiveness of a multiple-choice question depends on the quality of its stem, key, and distractors, as they must be clear, concise, and appropriately challenging for the intended audience. Automated multiple-choice question gener- ation offers the potential to reduce the time and effort required for manual question generation in education while maintaining quality and relevance. â Corresponding author: kaya@sabanciuniv.edu 1 arXiv:2604.10748v1 [cs.CL] 12 Apr 2026 Figure 1: An example multiple-choice question. Even though multiple-choice questions are well-known and widely used, automatically gen- erating relevant, high-quality, and diverse multiple-choice knowledge questions with realistic difficulty estimation remains a challenging task. Existing methods either rely on handcrafted rules, which lack scalability, or black-box deep-learning approaches, which struggle with inter- pretability. An effective MCQ-generation system must not only generate high-quality questions but also provide an interpretable difficulty estimation aligned with human perception. Fur- thermore, another challenge lies in the cost of creating an appropriate dataset, as it requires extensive human annotation. To address such challenges, we propose a novel approach that combines knowledge graphs (KGs) with large language models (LLMs) for generating multiple-choice questions with in- terpretable difficulty estimation. The integration of LLMs enables advanced natural language understanding and generation capabilities, while the knowledge graph serves as a structured backbone for representing factual information and entity relationships. An illustrative example with a small knowledge graph is presented in Figure 2, where entities, their types, and semantic relationships are encoded as nodes and edges. Figure 2: An illustrative example of a knowledge graph representing entities, their types, and the semantic relationships between them. Our method begins by building a knowledge graph (KG) from the input text using an LLM. Multiple-choice questions (MCQs) are then systematically generated from this graph. For each MCQ, a key node serving as the correct answer is selected, along with a relevant triple or quin- tuple from the KG, which may be supplemented with an extra triple for additional context. These structured graph elements are then converted into a question stem by the LLM, and suitable distractors are also selected from the KG. To ensure explainability in difficulty esti- mation, we compute several intuitive and interpretable difficulty signals, including graph- and network-analytics-based metrics, embedding-based semantic similarities, and linguistic features, and combine them into a unified difficulty score in a data-driven manner by leveraging the hu- man evaluation data we collect. Furthermore, beyond its technical aspects, our framework has 2 substantial potential for enhancing educational practices, as it can be integrated into intelli- gent tutoring systems, adaptive testing platforms, and digital learning environments to deliver personalized assessments. Overall, the novel contributions of this work are summarized below: ⢠We propose a novel end-to-end framework for generating multiple-choice questions from textual sources. The framework constructs a knowledge graph from the input text, gen- erates MCQs from the resulting graph, and estimates their difficulty in an interpretable manner. Our hybrid approach combines the structural knowledge of KGs with the natural language generation capabilities of LLMs to produce high-quality MCQs. ⢠We construct a novel dataset comprising MCQs, their corresponding knowledge subgraphs, and empirically derived difficulty labels based on an average of 38 independent human re- sponses per question. To the best of our knowledge, this is the first dataset that jointly includes MCQs, their underlying knowledge subgraphs, and empirical difficulty labels de- rived from broad human evaluation. ⢠We introduce nine interpretable, data-driven difficulty estimation signals spanning graph topology, textual semantics, and linguistic complexity, thereby capturing both structural and semantic properties of the generated MCQs. Their effectiveness is assessed through four regression models and an ablation study. ⢠Our data and code are available to enable reproduction of this study and further research. 1 The remainder of this paper is organized as follows: Section 2 discusses related work in ques- tion generation, MCQ generation, difficulty modeling, and the usage of knowledge graphs and LLMs in these settings. Section 3 details our methodology, including knowledge graph construc- tion, multiple-choice question generation, and difficulty estimation. Section 4 details the dataset construction process, including the human evaluation protocol used to obtain empirical difficulty labels, and presents key statistics characterizing the resulting dataset. Section 5 presents the experimental results and analysis, and Section 6 discusses them. Finally, Section 7 concludes the paper, points out key limitations, and outlines directions for future research accordingly. 2 Background and Related Work Automated question generation has seen significant advancements with various approaches ex- ploring different aspects of the problem. A review of related works on this topic, including their improvement aspects and limitations, is conducted with comparisons to our own. 2.1 Graph- and Ontology-based Question Generation Reddy et al. [2017] propose a method that employs recurrent neural networks (RNNs) rather than templates, as in their prior work, for question verbalization, thereby enhancing robustness and reducing the need for manual effort. However, the authors do not discuss the difficulty level of the question. Instead, they directly focus on the question text (stem) generation with the correct answer (key). Elsahar et al. [2018] present an encoder-decoder architecture that leverages textual contexts and a copy mechanism to address the challenge of the inability to generate questions for unseen predicates and entity types of previous works, hence enabling zero-shot question generation. Yet, Elsahar et al. restrict their task to generating natural language questions given a single input triple instead of multiple triples. They also do not emphasize parameters such as difficulty. 1 https://github.com/kamerkaya/Generating-Multiple-Choice-Knowledge-Questions-from-KGs 3 Later, Chen et al. [2023] propose a bidirectional Graph2Seq (Graph-to-Sequence) model to encode the KG subgraph to take advantage of the information provided by the graph structure. Then, they use an RNN decoder with a node-level copying mechanism to generate the question based on the output of a GNN-based (Graph Neural Network) encoder they propose. However, Chen et al. do not control or investigate the difficulty of the generated questions. Li et al. [2023] take advantage of a pre-trained LLM, GPT-2, to obtain a comprehensive semantic context, and then they construct a graph using the entities identified within this context. Then, they utilize an answer-aware graph attention network (GAT) to update it based on the constructed graph to generate the question. However, the constructed graph is not a KG and its edges do not represent semantic relationships; instead, they represent the co-occurrence of two entities in the same paragraph. The authors also do not discuss the difficulty of the generated question. 2.1.1 Approaches Taking Difficulty into Account Alsubait et al. [2016] generate difficulty-controllable multiple-choice questions by utilizing on- tologies to measure the similarity between the distractors and the key, based on the number of common properties they share in an ontology. They handle the stem generation with a template-based approach based on the ontology. Seyler et al. [2017] provide a KG-based question generation technique that first selects an entity from the KG and then generates a SPARQL (SPARQL Protocol and RDF Query Lan- guage) query that specifies that node uniquely and converts the query to a natural language question with preset templates referred to as question verbalization. For difficulty estimation, their approach requires a question-answer corpus with annotated difficulties to train a classifier on. They propose generating distractors by relaxing their original queryâs constraints to retrieve more than one unique answer. The effect of the distractors on the questionâs difficulty is defined with the confusion metric computed via the difficulty classifier trained on the Q&A corpus. Kumar et al. [2019] study on generating complex, multi-hop questions that require reasoning across multiple KG triples, hence accepting subgraphs and multiple triples as its input, as opposed to a single triple. They present a Transformer-based model that also takes the difficulty into account. This work is similar to ours as it shares the goal of enabling the generation of multi-hop questions with controllable difficulty levels. However, Kumar et al. define difficulty exclusively based on two NER-dependent factors: (1) confidence scores for entity-mention linking and (2) the selectivity of entity surface forms, without incorporating structural or semantic properties of the question or the subgraph as a whole. Kusuma et al. [2022] propose an ontology-based question generation framework that in- troduces a combination of taxonomy ontology and sentence ontology, referred to as knowledge ontology. Given a textual input, their system constructs an ontology and generates various types of questions using query templates. Both the type and difficulty of each question are determined manually by ontology engineers and domain experts. While their approach supports multiple question formats and provides difficulty annotations, it relies on template-based generation and does not incorporate automated difficulty estimation. Bi et al. [2024] introduce DiffQG, a difficulty-controllable single-answer question generation model with no distractors that produces natural language questions from a given KG subgraph with a specified difficulty level. Their approach uses a mixture-of-experts module to learn soft templates for different difficulty levels, enhancing the diversity of question phrasing without rely- ing on manually crafted templates. They also incorporate a disentanglement module that isolates the KG triples relevant to the target difficulty, which enables counterfactual reasoningâtraining the model on perturbed subgraphs to reinforce a causal link between the difficulty label and the 4 question features. Additionally, the authors propose a difficulty estimation mechanism called AutoDE that considers eight difficulty signals and the use of those signals by normalizing and linearly combining them. We follow a similar approach to estimate the difficulty with differ- ent signals, yet the assumption of linearity can be misleading, and to solve that, we propose a data-driven methodology while employing both linear and non-linear models. Moreover, they propose the counterfactual reasoning approach to further enable the generation of questions of different difficulty levels given the same input subgraph, with the underlying idea that the given subgraph should not dictate the difficulty of the question. On the contrary, we think the topol- ogy of the given subgraph and the information within should be the main determining factor for difficulty, especially in knowledge questions. The broader challenge of text-based question difficulty prediction is systematically reviewed by AlKhuzaey et al. [2023], who highlight the major role of linguistic features and the need for standardized datasets to enable meaningful comparisons between models. Zhu et al. [2024] quantify difficulty as the average similarity of each distractor to the correct answer. In their system, an MCQ can be generated in multiple versions: by selecting distractors that are very similar to the answer, the questionâs difficulty score increases, whereas using more dissimilar distractors yields an easier question. They implement an algorithm to automatically pick a set of distractors such that the resulting questionâs difficulty falls into a specified range or level. If impossible, they automatically lower the targeted level by one. To improve the accuracy of this difficulty metric, the authors refine the similarity calculation with topological weights, effectively incorporating knowledge from the structure of a semantic network to better judge how âcloseâ a distractor is to the answer. This approach allows for explicit difficulty control: the same question stem can be paired with easier or harder sets of options, producing tiered versions of the question. One limitation is that this method predominantly captures only one aspect of difficulty: distractor ambiguity. It does not account for other factors like the intrinsic complexity of the questionâs content. Additionally, while the difficulty calculation is interpretable, they do not investigate a comprehensive list of signals that might affect the difficulty of an MCQ. Wei and Hao [2024] introduce KGNN-ADP, a KG-enhanced neural network designed for predicting the difficulty of an MCQ. Their framework leverages a domain-specific KG to extract relevant knowledge points for a given question and assess difficulty along two primary axes: (1) knowledge difficulty, computed by analyzing the semantic proximity and structural weight of the matched knowledge points in the graph, and (2) option difficulty, modeled using both the semantic similarity between distractors and the answer and the internal semantic divergence within the full question. While this work shares our emphasis on difficulty estimation grounded in KGs, it differs from ours in scope: their model predicts difficulty for existing MCQs rather than generating new ones by solely focusing on the absolute difficulty of MCQs, and not on the broader task of MCQ generation from raw text or graphs. The prior research also extends to specialized topics. For example, an ontology-based ap- proach by Leo et al. [2019] has been applied in the medical field to generate multi-term, case- based multiple-choice questions (MCQs) from a medical ontology, which shows the value of structured knowledge in niche areas. Also, a hybrid framework by Kumar et al. [2023] combines machine-learning and semantic techniques to create different types of MCQ stems for technical fields, which are then assessed against the cognitive levels of Bloomâs Taxonomy. Additionally, the synergy between knowledge graphs and large language models has recently gained attention for related tasks as well, such as question answering, where combining struc- tured KGs with LLMs improves performance [Huang et al., 2025, Jiang et al., 2024]. 5 2.2 Non-graph-based Approaches Vachev et al. [2022] present Leaf, a system for the end-to-end generation of MCQs from educa- tional texts. The goal is to ease quiz/exam creation for instructors by inputting course material and outputting factual MCQs. The system uses neural models to perform the sub-tasks of question and distractor generation: it fine-tunes a Transformer-based text-to-text model (T5) on Q&A pairs and trains another model to produce realistic distractors. Given a passage and a target answer, the former produces a question that becomes the input of the latter. This system demonstrates that large-scale pre-trained models can be harnessed to automate MCQ creation from unstructured text, achieving good fluency and relevance in both questions and distractors. Leaf, however, does not incorporate question difficulty; all the questions are intended to be high- quality, but there is no control on or estimation of the difficulty. Furthermore, Leaf relies on raw text input and does not utilize structured KGs; as such, it may struggle with the coverage of specific relationships or while providing explanations. On the contrary, our work aims to estimate and explain the difficulty of each question generated. Zeng et al. [2024] propose RTRL, a deep question generation framework that combines a relation-aware transformer with reinforcement learning for deep question generation. RTRL introduces an encoder that embeds answer distances and words, and uses a BiLSTM to build answer-contextualized representations. RTRL also models both explicit linguistic relations and implicit semantic relations via its relation-aware transformer. The model is fine-tuned using reinforcement learning with sentence-level rewards to optimize evaluation metrics, thereby ad- dressing training-evaluation mismatch. Although RTRL significantly improves performance in deep question generation, it does not focus on multiple-choice question generation or difficulty modeling; its focus remains on deep question generation. The research in this field has also moved beyond English-language applications, with works like Johnson et al. [2024] demonstrating a parallel construction method to generate questions for Spanish textbooks by leveraging an existing English-based system. 2.2.1 Approaches Taking Difficulty into Account Gao et al. [2019] explores difficulty control while generating open-ended questions for reading comprehension. They use text passages, where the input is a sentence from a reading compre- hension article plus a target answer phrase, and the output is a question that asks about that answer at a specified difficulty level. A dataset with difficulty tags (over 70k questions split into easy and hard), enabling the authors to train an end-to-end sequence-to-sequence question generation model with difficulty, is leveraged. The results showed that their model can indeed tailor the generated questions to the requested difficulty without losing quality, as the questions remain fluent and answerable while matching the difficulty specification. Gao et al. thereby demonstrate the feasibility of controlling question difficulty. However, their work is limited to open-ended Q&A pairs in text and a binary difficulty distinction, since there is no provision for generating distractors as in MCQs. Cheng et al. [2021] advances Gao et al. [2019]âs approach by focusing on the reasoning complexity of questions. Noting that the prior method offered little interpretability on difficulty, they redefine the difficulty as the number of inference hops or reasoning steps required to answer it. This strategy offers better interpretability and stronger logical controllability than treating difficulty as a latent label. However, the approach is applied to free-text Q&A settings and generates single-answer questions; it does not handle MCQs. Moreover, it restricts the definition of difficulty to only a single parameter: the number of reasoning steps required. 6 2.3 A Brief Comparison with the Literature Our work distinguishes itself from the literature by integrating easy-to-achieve structured knowl- edge with auto-generated knowledge graphs and large language models to generate high-quality MCQs with interpretable difficulty estimation. Unlike prior QG work, we leverage a knowledge graph with an LLM, using the KG to ground the question in verifiable facts and the LLM to produce fluent questions by taking advantage of its pre-trained capabilities. This combination enables us to cover complex knowledge that might require reasoning while maintaining nat- ural language fluency and diversity by leveraging LLMs, a synergy not explored in previous systems to the best of our knowledge. Furthermore, our difficulty estimation system provides justifications for a questionâs difficulty rating based on the defined difficulty signals, offering interpretability. Lastly, we explicitly calibrate our difficulty predictions with the dataset we collect from human participants, ensuring that the difficulty levels correspond to the real world. This human-in-the-loop calibration means that our systemâs idea of difficulty is not just an as- sumption but reflects actual ease or struggle observed among the participants. Through this novel integration of KGs, LLMs, and a human-calibrated approach, our work delivers a more comprehensive solution for generating multiple-choice questions and estimating their difficulty. A comparison of the literature above, including this work, is presented in Table 1. 3 Materials and Methods In this section, we present the methodology adopted in our study, which consists of three main components: KG construction, MCQ generation, and difficulty estimation. Figure 3 provides an overview of the proposed framework. Given a corpus of factual information, we first con- struct a structured knowledge graph that captures the entities and their semantic relationships. This graph serves as the foundation for generating candidate MCQs by selecting appropriate subgraphs and transforming them into MCQs. Finally, we estimate the difficulty of each MCQ using a combination of graph-based, embedding-based semantic similarities, and linguistic fea- tures. Each stage of the framework is elaborated in the following subsections. 3.1 Knowledge Graph Construction Given a set of input text documents, we construct a knowledge graph (KG) to serve as the foundation for multiple-choice question generation. To extract structured information from given documents, we apply a large language model (LLM) to identify salient factual statements and convert them into subjectâpredicateâobject triples. These triples capture entityârelationâentity structures that represent core knowledge units within the text. The procedure is summarized in Algorithm 1, which outlines how each document is processed into a graph document and subsequently integrated into a unified KG stored in a Neo4j 2 database. We begin by constructing the knowledge graph (KG) from a curated collection of textual documents. In this work, we use Wikipediaâs top-100 most-viewed articles 3 , which we automat- ically retrieved using a custom scraping script. The raw content is processed to extract factual knowledge using GPT-4o 4 , a large language model provided by OpenAI. To transform unstructured text into structured graph representations, we employ LLMGraph- Transformer from LangChain 5 . This tool ingests a list of documents and using the underlying LLM, it extracts subjectâpredicateâobject triples to represent key factual relationships. These 2 https://neo4j.com 3 https://en.wikipedia.org/wiki/Wikipedia:Popular_pages#Top-100_list 4 https://platform.openai.com/docs/models/gpt-4o 5 https://w.langchain.com 7 Table 1: Comparison of the presented literature, including the proposed study. InputQG ModelMul.-hop QGMCQ Gen.DifficultyModel-ingHumanCalib.in Diff.Mod. Reddy et al. [2017] KGRNN-based (seq2seq)â-- Elsahar et al. [2018] KG triple and re- lated texts Encoder-decoderwith copy mechanism â-- Vachev et al. [2022] TextTransformer-based (T5)ââ-- Chen et al. [2023] KGBidirectional graph2seq model with copy mecha- nism ââ-- Li et al. [2023] Text/auto- generated entity co-occurrence graph GPT-2,answer-aware GAT, and multi-head attentiongeneration module ââ-- Zeng et al. [2024] TextRelation-awaretrans- former with reinforcement learning ââ-- Alsubait et al. [2016] OntologyTemplate-basedââDifficulty control via key/dis- tractor similarities based on number of common properties â Seyler et al. [2017] KGSPARQL queries and templates ââDifficulty estimation with logis- tic regression â(Ind.) Kumar et al. [2019] KGTransformer-basedââDifficulty control and estima- tion with custom formula â Gao et al. [2019] TextSeq2seq encoder-decoder based on LSTMs with copy mechanism âDifficulty control by initializing the hidden state of the decoder w.r.t. the required difficulty â(Ind.) Kusuma et al. [2022] Text, and its auto constructed ontology SPARQL queries and templates ââManual via ontology engineers & domain experts â Bi et al. [2024] KGSoft-templates with MoE, and counterfactual rea- soning ââDifficulty control and estima- tion via linear modeling of pro- posed signals â Zhu et al. [2024] Semantic Net- work Rule-basedââDifficulty estimation and dis- crete control based on key/dis- tractor similarities by also inte- grating topological weights â Weiand Hao [2024] Existing MCQ and KG ---Difficulty prediction with the proposed KG-enhanced neural network based on Bi-LSTM â This workText/auto-gen. KG Knowledge graph and LLM-based âDifficulty estimation with both linear and non-linear models based on signals â 8 Figure 3: Overview of the proposed framework. triples are subsequently stored in a Neo4j graph database. Thus, the KG captures both the semantic entities, their types, and the relations between them, which are later leveraged to generate MCQs. The KG construction pipeline is designed to be schema-free, accommodating a wide variety of relations and entity types extracted from natural language. To facilitate downstream tasks, we also compute a node centrality metric, specifically degree centrality, and node embeddings for each entity in the graph using FastRP by Chen et al. [2019], which later serve as useful components for some of the difficulty signals. The entire construction process is automated, resulting in a rich and semantically meaningful representation of the source content. 3.2 Multiple-Choice Question Generation Once the knowledge graph is constructed, we systematically generate MCQs by selecting high- centrality nodes as key candidates. Specifically, in this work, we choose the top-40 most central nodes in the KG based on degree centrality and attempt to generate four MCQs per node. The process that defines the generation of a single MCQ is outlined in Algorithm 2. Each MCQ is generated from either a triple (single-hop question), a triple with additional context from an extra triple, a quintuple (double-hop question), or a quintuple with additional context from an extra triple. For each selected node (key), a related triple or quintuple is selected from the KG. Optionally, an extra triple may be added to provide additional context. 9 Algorithm 1 Knowledge Graph Construction 1: Input: Set of documents D =doc 1 ,doc 2 ,...,doc n 2: for each doc i âD do 3: graphDocument i â LLMGraphTransformer(doc i )⡠Extract nodes with types and relationships using an LLM, specifically with LangChainâs LLMGraphTransformer 4: end for 5: G âgraphDocument 1 ,...,graphDocument n ⡠Aggregate all documents 6: constructAndSaveKG(G)⡠Parse and persist the graph in Neo4j Figure 4: The overview of the knowledge graph construction process from input documents using an LLM and integration into a Neo4j graph database. These structured elements are passed to a prompt-based question generation module powered by GPT-4o to create the question stem. The prompts are crafted to follow the style of well-crafted trivia questions, inspired by formats such as those used in the quiz show Who Wants to Be a Millionaire?. Figure 6 presents illustrative examples of each subgraph type alongside potential question stems that could be generated from them. Distractors are selected from the KG using a breadth-first search (BFS) strategy, ensuring they belong to the same semantic type as the key and are progressively distant in graph depth. In rare cases, the KG does not contain a sufficient number of appropriate distractors for a given key. In such instances, the MCQ generation attempt is aborted to preserve the quality of the generated MCQ. In our experiments, this limitation accounts for a total of 156 finalized MCQs instead of the originally targeted 160. At this stage, having obtained the key, the question stem, and the distractors, the construc- tion of the MCQ is complete. To ensure the quality and correctness of the generated MCQ, we perform a post-processing step for validation using a large language model. Each distrac- tor option is independently evaluated by prompting the LLM with the question stem and the respective distractor. If the model identifies any distractor as a potentially correct answer, the MCQ is marked as invalid. This automated validation process serves as a safeguard against ambiguities or inconsistencies in the question formulation, ensuring that each MCQ has a single, unambiguous correct answer. Only those questions that pass this validation step are retained for subsequent analysis. In cases where an MCQ fails the validation step, a retry mechanism is triggered to attempt regeneration. The process revisits the same key entity and initiates a new question generation sequence, potentially selecting a different supporting triple, quintuple, or extra triple, and gener- ating a new set of distractors. This mechanism is repeated up to a predefined number of retries to ensure robustness and increase the likelihood of producing a valid MCQ. If all attempts fail, the MCQ is discarded from the final dataset to maintain overall quality. Finally, each MCQ is saved with a set of metadata features referred to as difficulty signals, which are later utilized for difficulty estimation. All signals are normalized into the range [0, 1] using min-max normalization to ensure consistency and comparability across different feature types. These signals serve as interpretable proxies for MCQ difficulty and are further investigated and explained in Section 3.3, where we also describe how they are integrated for a unified difficulty estimation model. 10 Algorithm 2 Multiple-Choice Question Generation 1: Input: kg⡠Knowledge graph constructed in Algorithm 1 2: Input: keyNode⡠Selected node to serve as the correct answer 3: tripleOrQuintuple â Sample a triple or quintuple of keyNode 4: useExtraTriple â Set to True or False 5: if useExtraTriple then 6: extraTriple â Retrieve another triple of keyNode 7: associatedSubgraph â tripleOrQuintuple, extraTriple 8: else 9: associatedSubgraph â tripleOrQuintuple 10: end if 11: prompt â constructPrompt(keyNode, associatedSubgraph) 12: questionStem â LLM(prompt) 13: maxDepth â 5⡠Maximum BFS depth for distractor search 14: distractors â generateDistractors(kg, keyNode, associatedSubgraph, maxDepth) 15: if validate(questionStem, keyNode, distractors) then 16: difficultySignals â Compute the defined difficulty signals 17: mcq â MCQ(questionStem, keyNode, distractors) 18: save(mcq, difficultySignals) 19: end if Figure 5: The overview of the MCQ generation process: The pipeline samples an associated subgraph from the KG given the key, prompts an LLM to generate a question stem from it, retrieves proper graph-based distractors, computes difficulty signals of the generated MCQ after validation, and saves the resulting MCQ with its computed difficulty signals. 3.3 Difficulty Estimation To estimate the difficulty of each generated MCQ, we extract and analyze a set of nine in- terpretable signals, each capturing a distinct aspect of the questionâs structure, semantics, or linguistic complexity. These signals detailed here serve both as standalone indicators and as input features to a regression model for predicting difficulty scores. Reasoning This binary signal distinguishes between questions that are derived from a single triple (single-hop reasoning) and those constructed from a quintuple (double-hop reasoning). Double-hop questions are generally more cognitively demanding, as they require the integration of multiple relational facts. Figure 7 illustrates this distinction. Extra Triple This binary signal captures whether an additional supporting triple is incorpo- rated, providing further context. The inclusion of an extra triple may enrich the given knowledge inside the MCQ, but also potentially increase its complexity. An example is shown in Figure 8. Distractor Depth This signal measures the average depth of distractors from the key entity in the knowledge graph using breadth-first traversal. The farther the distractors are from the key entity, the less semantically related they tend to be, which typically reduces the difficulty. This structural notion is visualized in Figure 9. 11 Figure 6: Illustrative examples of each subgraph type alongside potential question stems that could be generated from them. The green node represents the key in each case. Node Embedding Similarity We compute the average cosine similarity between the node embeddings of the distractors and the key, using FastRP embeddings, denoted as e n . This is expressed in (1). SignalNodeEmbedSim = 1 3 3 X i=1 cos (e n (node(d i )),e n (node(key)))(1) Higher similarity implies more plausible distractors, increasing the MCQâs difficulty. Text Embedding Similarity This signal captures the semantic similarity between each dis- tractor and the question stem in the embedding space. To contextualize the score, it is normal- ized by the similarity between the key and the stem. The computation is defined in (2). SignalTextEmbedSim = 1 3 P 3 i=1 cos (e t (text(d i )),e t (stem)) cos (e t (text(key)),e t (stem)) (2) 12 Figure 7: Illustration of single-hop and double-hop reasoning for MCQ generation. The green node represents the key. Figure 8: Illustration of MCQ generation with and without an extra triple. The green node represents the key. Here, e t denotes the text embedding function, specifically OpenAIâs text-embedding-3-large 6 model. Degree Centrality The degree centrality of a node v is defined as the number of incoming and outgoing edges it has. The formal definition is given in (3). SignalDegreeCentrality = 1 |V s | X vâV s deg(v)(3) The signal takes the average degree centrality of all nodes involved in the associated knowledge subgraph. Higher centrality often implies more well-known concepts. Readability We compute the Flesch Reading Ease score, denoted as F, for the question stem, following the formulation proposed by Flesch [1948]. This score is used as the readability signal to quantify linguistic complexity. The computation is provided in (4). 6 https://platform.openai.com/docs/models/text-embedding-3-large 13 Figure 9: Visualization of candidate distractor depths. The green node represents the key. F = 206.835â 1.015¡ # words # sentences â 84.6¡ # syllables # words (4) Above Largest Gap Count We compute the cosine similarity between the question stem and each of the four distractor options, and sort the resulting similarity scores in descending order: sim 1 ⼠sim 2 ⼠sim 3 ⼠sim 4 . We then calculate the differences between adjacent similarity values and identify the position of the largest gap, drawing inspiration from the Maximum Gap Thresholding technique proposed by SoykĂśk and GĂźvenir [2025]. The signal is defined as the number of distractors that appear before the largest gap in the sorted list, as formalized in (5). SignalAboveLargestGapCount = arg max iâ1,2,3 (sim i â sim i+1 )(5) This ordinal signal captures how many distractors are grouped more closely in semantic similarity to the stem, potentially influencing the difficulty of identifying the correct answer. LLM Extra Fact This binary signal indicates whether the generated question stem includes factual content not directly inferable from the associated knowledge subgraph. Such additions may be unintended hallucinations by the LLM and typically increase the difficulty or ambiguity of the question. This signal is automatically generated for each MCQ by prompting an LLM to compare the question stem and the associated knowledge subgraph. All signals are normalized to the [0, 1] interval using min-max normalization to ensure scale compatibility. Together, they serve as interpretable and complementary indicators of MCQ difficulty. Moving on, we explore how these signals are leveraged to train a supervised model for estimating difficulty scores. Table 2 summarizes the nine proposed signals. 3.3.1 Difficulty Estimation Model After computing the defined difficulty signals for each validated MCQ, we train a supervised regression model to estimate the MCQ difficulty. The target variable is the empirical difficulty score, approximated by the observed incorrect response rate collected from users. This score lies within the [0, 1] interval, where higher values correspond to more difficult questions. The proposed difficulty signals are model-agnostic and can be straightforwardly used as input features in any regression framework. We evaluate several standard regression models, specifically LinearRegression, RandomForestRegressor, and GradientBoostingRegressor from the 14 Table 2: Overview of MCQ-generation signals and their descriptions. SignalDescription ReasoningA binary indicator denoting whether the MCQ was derived from a triple or a quintuple. Extra TripleA binary indicator representing the use of an extra triple for additional context. Distractor DepthThe average graph distance between the key and each distractor. Node Embedding Similar- ity The average cosine similarity between the key and distractors based on their graph node embeddings. Text Embedding Similarity The average cosine similarity between the distractors and the question stem, divided by the cosine similarity between the key and the stem. Degree CentralityThe average degree centrality of the entities involved in the question stem, re- flecting their graph-level prominence. ReadabilityThe Flesch Reading Ease score of the question stem, indicating its linguistic complexity. Above Largest Gap Count An ordinal indicator capturing the number of distractor options that precede the largest semantic gap when cosine similarities between the question stem and each distractor are sorted in descending order. LLM Extra FactA binary flag indicating whether the question stem introduces factual content not explicitly present in the selected subgraph. scikit-learn 7 library, as well as XGBRegressor from the XGBoost 8 library. Among these, the XGBRegressor yields the best performance in our experiments. All models are trained using an 80/20 split, and we report evaluation metrics including root mean squared error (RMSE), mean squared error (MSE), mean absolute error (MAE), and coefficient of determination (R 2 score) for each. The resulting regression model provides a continuous-valued difficulty score for each MCQ, which we refer to as its estimated difficulty. In addition to enabling quantitative difficulty estimation, this approach allows us to interpret the relative importance of each difficulty signal through feature importance analysis. 4 Dataset To the best of our knowledge, there exists no available dataset that jointly provides (i) the KG or subgraph used to construct a multiple-choice question (MCQ), (i) the MCQ itself, and (i) a corresponding difficulty label. Given that our difficulty estimation framework relies on both the structural and semantic characteristics of the source knowledge graph as well as the inherent properties of the MCQ itself, we constructed a dedicated dataset for this task. We began by collecting textual data from Wikipediaâs top-100 most popular articles, which we automatically retrieved using a custom scraping script. These documents were then pro- cessed using LangChainâs LLMGraphTransformer with OpenAIâs GPT-4o model to construct a knowledge graph. The resulting KG captures semantic relationships and entity types, and serves as the foundation for MCQ generation. To generate MCQs, we selected the top-40 most central nodes in the graph using degree centrality and attempted to generate four MCQs per node using our methodology. This yielded a total of 156 MCQs. We presented these MCQs to a diverse pool of human participants, and for each MCQ, we received approximately 38 responses on average. This approach enhances the reliability of the ground truth difficulty labels by reducing subjectivity and averaging out individual biases, as each MCQâs ground truth label depends on the responses of approximately 38 participants. 7 https://scikit-learn.org/ 8 https://xgboost.readthedocs.io/ 15 For each MCQ, we computed the incorrect answer rate, the proportion of participants who answered the question incorrectly, and used it as the empirical ground truth difficulty score. Ad- ditionally, participants were asked to rate how much they liked each question on a standardized scale. These ratings were averaged to yield a quality metric for each MCQ. The mean liking score across all MCQs was 66%, indicating a generally favourable reception. Also, a negative correlation of â0.49 was observed between average liking scores and difficulty, suggesting that participants tended to prefer easier questions. The histogram in Figure 10 shows the distribution of incorrect answer rates across the dataset. The mean incorrect response rate was 0.34, indicating a moderate overall difficulty level. The distribution is slightly skewed toward easier questions, likely due to the intentional selection of high-centrality nodes in the knowledge graph, derived from Wikipediaâs most popular articles, to ensure familiarity and contextual relevance for participants. Figure 10: Histogram of the incorrect answer rates of generated MCQs. This dataset offers a unique resource where each data point consists of the source knowledge subgraph, the generated MCQ, and an empirically derived difficulty label as the ground truth for the difficulty score. It enables modeling the difficulty using both KGâbased and MCQâbased signal sources, effectively addressing our initial requirement. 5 Experimental Results and Analysis This section presents the experimental evaluation of our proposed difficulty estimation frame- work. All experiments were conducted on a MacBook Pro equipped with an Apple M3 chip, integrated GPU, and 16 GB of unified memory. The system was running macOS with ARM64 architecture. We assess the effectiveness of multiple regression models using standard performance metrics and provide an analysis of predictive accuracy and feature importance. 5.1 Evaluation Metrics To evaluate the performance of the regression models, we report the following metrics: 16 ⢠Root Mean Squared Error (RMSE): Measures the square root of the average squared differences between predicted and actual values. RMSE penalizes larger errors more than smaller ones. ⢠Mean Absolute Error (MAE): Represents the average absolute difference between predicted and actual values, offering a more interpretable error measure. ⢠Coefficient of Determination (R 2 ): Indicates the proportion of variance in the actual difficulty scores explained by the predicted scores. ⢠Spearmanâs Rank Correlation (Ď): Measures the strength and direction of the mono- tonic relationship between predicted and actual difficulty scores. This metric is particularly useful for evaluating how well the predicted rankings of MCQs align with the ground-truth rankings, thus reflecting the modelâs ability to preserve the relative ordering in terms of MCQ difficulty. These metrics collectively provide a comprehensive view of model accuracy, robustness, and consistency in ranking, which are important for assessing the quality of difficulty predictions. 5.2 Model Selection and Results Given the limited size of our dataset, we restrict our analysis to classical regression models that are well-suited for small-to-moderate-sized datasets. Specifically, we evaluate LinearRegression, RandomForestRegressor, GradientBoostingRegressor, and XGBRegressor, leveraging scikit-learn and XGBoost libraries for implementation. Table 3 summarizes the performance of these models. Table 3: Performance comparison of regression models on MCQ difficulty estimation. (The target variable, MCQ difficulty, is a continuous value in the range [0, 1].) ModelRMSE MAE R 2 Spearmanâs Ď Linear Regression0.140.12 0.4662.4% Random Forest Regressor0.140.12 0.4164.2% Gradient Boosting Regressor 0.150.13 0.3446.4% XGBoost Regressor0.13 0.11 0.5264.3% During our analysis, we identified an outlier MCQ instance in the dataset with an empirically derived difficulty score of 0.975, the highest in the dataset and visibly higher than all others, as can be seen in Figure 10. This particular instance was found to contain two semantically valid answer choices, which were not detected by either our KG-based or LLM-based validation mechanisms during generation. Moreover, the distractor labeled as incorrect appeared more plausible to users than the intended correct answer, leading to unusually high confusion and, consequently, an out-of-distribution empirical difficulty score. Although this is a single instance, we conducted our experiments both including and excluding it (the code provided under Data availability reproduces both variants), and the results demonstrated that excluding this faulty instance had minimal impact on the overall model performances, as expected given it is only one data point. The results reported in this paper are based on the version of the dataset with this MCQ excluded. 5.3 Prediction Accuracy Visualization Figures 11a, 11b, 11c, and 11d visualize the predicted vs. actual difficulty scores for each regression model. The red dashed line represents the ideal case where predicted and actual values are equal. 17 (a) Linear Regression(b) Random Forest Regressor (c) Gradient Boosting Regressor(d) XGBoost Regressor Figure 11: Predicted vs actual difficulty for each regression model. 5.4 Feature Importance Analysis To gain insight into which signals contribute most to the difficulty prediction, we examine the feature importances learned by the XGBoost Regressor, which yielded the best performance. The feature importance scores shown in Figure 12 are derived from the feature_importances_ attribute of the trained XGBoost Regressor. This metric quantifies the contribution of each input feature to the modelâs predictions by measuring the average gain in performance (the amount of reduction in loss) when a feature is used for splitting across all trees in the ensemble. Features with higher scores are those that, on average, contribute more to accurate predictions. The results are presented in Figure 12. To further interpret the model predictions, we use SHAP (SHapley Additive exPlanations) values [Lundberg and Lee, 2017]. We compute SHAP values for the trained XGBoost Regressor using the method proposed by Lundberg et al. [2020], and visualize the results using a summary plot, as shown in Figure 13. The plot shows both the magnitude and direction of each featureâs impact across instances. Red points represent higher feature values, and blue points represent lower values, enabling a detailed visual of how individual features affect the modelâs output across varying input instances. As seen in Figure 13, the top two features and the last feature remain the same as in Figure 12. Only minor changes are observed in the intermediate rankings, which indicates an overall similarity. 18 Figure 12: Feature importance scores from XGBoost Regressor. 5.5 Ablation Study To evaluate the contribution of each individual signal to the difficulty estimation task, we con- ducted an ablation study in which we trained the XGBoost Regressor, our best-performing model, after excluding one signal at a time. The results are presented in Figure 14, where each plot shows the predicted vs. actual difficulty scores along with the metrics for the model trained without a specific feature. We observed that the absence of three particular signals (Reasoning, Degree Centrality, and Above Largest Gap Count) did not degrade performance compared to our original best-performing model that used all the signals (RMSE = 0.13, MAE = 0.11). This observation motivated a sec- ond round of ablation experiments, in which we trained models excluding pairwise combinations of these three features. Among the three possible 2-feature exclusion combinations, the configuration excluding Rea- soning and Above Largest Gap Count achieved improved performance, with RMSE = 0.12 and MAE = 0.10, thereby surpassing our previous best-performing model. The other two combi- nations produced RMSE = 0.14 and MAE = 0.12, which were inferior to the baseline. For completeness, we also excluded all three features simultaneously, resulting in RMSE = 0.14 and MAE = 0.12, again worse than the baseline. The results of these experiments are presented in Figure 15. Moreover, the results of the new best-performing model, compared with the previous best, are shown in Table 4. Table 4: Comparison of baseline and improved XGBoost Regressor configurations. ModelRMSE MAE R 2 Spearmanâs Ď XGBRegressor - All signals 0.130.11 0.5264.3% XGBRegressor - Best0.12 0.10 0.5866.1% 19 Figure 13: SHAP summary plot for the XGBoost Regressor. Each point corresponds to a SHAP value for a feature and an instance. 6 Discussion Our experiments show that all evaluated regression models produce reasonably strong perfor- mance given the size of the dataset, with the XGBoost Regressor consistently outperforming the others across all evaluation metrics. The R 2 score of 0.52 and Spearmanâs Ď of 64.3% for the baseline XGBoost configuration indicate a strong correspondence between predicted and actual difficulty scores, confirming the viability of our proposed difficulty estimation framework. The ablation study results provide deeper insights into the role of individual signals. The absence of certain signals (Reasoning, Degree Centrality, and Above Largest Gap Count) did not reduce performance compared to the full-feature baseline. More importantly, removing one specific pair of features (Reasoning and Above Largest Gap Count) led to a new best-performing model with RMSE = 0.12, MAE = 0.10, R 2 = 0.58, Spearmanâs Ď = 64.3% and surpassing the original configuration. Furthermore, the feature importance and SHAP analyses offer interpretable evidence regard- ing the contributions of individual signals. The observed results indicate that semantic features play a central role in difficulty estimation. Among them, Text Embedding Similarity emerges as the most influential signal, highlighting the critical role of semantic alignment between the distractors and the question stem, with respect to the alignment between the key and the stem. Notably, Readability, a relatively simple and long-established linguistic measure, also proves to be highly effective, reaffirming its continued relevance. Graph-based features such as Node Embedding Similarity also demonstrate meaningful contributions, supporting the utility of incor- porating structural information from the underlying knowledge graph into difficulty estimation. In contrast, LLM Extra Fact exhibits relatively minimal influence, suggesting that its inclusion serves more as a precautionary indicator than a key predictive signal. These findings demonstrate the viability of our approach for performing interpretable and accurate difficulty estimation on the MCQs generated from the input textual sources through our framework, using features grounded in both the underlying knowledge graph and the MCQs themselves. 20 Figure 14: Single-feature ablation study results. Beyond the methodological contributions and discussions, our findings have practical im- plications for multiple application domains. In adaptive learning systems, automated MCQ generation with accurate and interpretable difficulty estimation can enable the dynamic selec- tion of questions tailored to a learnerâs proficiency, thereby improving engagement and learning outcomes. Moreover, while the experiments were conducted on a dataset of limited size and domain scope, the underlying methodology is not inherently tied to any specific subject area. The combination of semantic, structural, and linguistic signals, together with a systematic feature evaluation process, makes the framework adaptable to other domains where multiple-choice knowledge questions are used or applicable. The same approach could be applied to different areas. This adaptability, together with the strong performance demonstrated in our experiments, underlines the potential for broader applicability beyond the specific setting examined in this work. This study demonstrates that interpretable, high-performing difficulty estimation for auto- matically generated MCQs is achievable through a combination of semantic, structural, and linguistic features, with careful attention to feature evaluation and selection. The proposed framework offers a foundation for practical deployment in real-world educational and assess- ment contexts, and it opens multiple pathways for future research. 21 (a) 2-feature ablation experiment.(b) 2-feature ablation experiment.* (c) 2-feature ablation experiment.(d) 3-feature ablation experiment. Figure 15: Extended ablation study results for 2-feature and 3-feature exclusions. (a) Excluded: Reasoning and DegreeCentrality. No improvement. (b) Excluded: Reasoning and AboveLargestGapCount. *Best-performing exp. (c) Excluded: DegreeCentrality and AboveLargestGapCount. No improvement. (d) Excluded: Reasoning, DegreeCentrality, and AboveLargestGapCount. No improvement. 7 Conclusion and Future Work This study introduces a novel framework for generating multiple-choice knowledge questions with interpretable difficulty estimation by integrating knowledge graphs (KGs), which provide structured representations of factual knowledge, with large language models (LLMs), which offer advanced natural language understanding and generation capabilities. Starting from unstruc- tured textual sources, we construct a KG that captures factual relationships, generate MCQs from selected subgraphs, and compute a suite of interpretable signals that reflect both structural and linguistic aspects of each MCQ. These signals are then used as input features to train regres- sion models for predicting empirical difficulty scores, demonstrating reasonably strong alignment with human evaluations. Our experimental results confirm the viability of this approach, showing that difficulty can be effectively estimated using features derived from both the underlying KG and the MCQs themselves. Among the evaluated models, XGBoost achieved the best performance across all 22 metrics, and feature importance analysis further supported the value of combining multiple types of signals. One key limitation of our work lies in the size of the collected dataset. Due to practical constraints, the number of MCQs and corresponding human responses was relatively limited. This restricted our ability to further explore more complex models requiring large volumes of training data, such as deep neural architectures. Future work could aim to scale data collection to support the training and evaluation of different and more sophisticated models. Furthermore, while the current dataset primarily targets common knowledge, expanding it to incorporate specialized knowledge domains represents a promising direction for future work. Additionally, exploring user modeling and personalized difficulty estimation offers another valuable direction for advancement. Moreover, although failures in detecting invalid question generation attempts through either the KG-based or LLM-based validation mechanisms are currently rare, with only one such in- stance occurring in this work, future work could incorporate cross-model validation with multiple models to further reduce the likelihood of such cases. Overall, our work establishes a comprehensive framework that generates multiple-choice knowledge questions given textual inputs and estimates their difficulty in an interpretable and accurate manner. By leveraging large language models for language understanding and genera- tion capabilities and knowledge graphs for structured representation, our approach transforms raw texts into high-quality MCQs and provides interpretable difficulty estimation grounded in both semantic (graph-based and text-based) and linguistic characteristics. This end-to-end ca- pability also opens new pathways for research at the intersection of knowledge representation, educational technology, and natural language understanding and generation. This approach has strong potential to support personalized learning and adaptive assessment in educational settings by enabling the generation of high-quality MCQs with interpretable difficulty levels. Author contributions:: Conceptualization, H.A.G. and K.K.; methodology, M.C.Ĺ.; soft- ware, M.C.Ĺ.; validation, M.C.Ĺ. and H.A.G.; data curation, M.C.Ĺ.; writingâoriginal draft preparation, M.C.Ĺ., H.A.G., and K.K; writingâreview and editing, M.C.Ĺ., H.A.G., and K.K; visualization, M.C.Ĺ., H.A.G., and K.K; supervision, H.A.G.; project administration, H.A.G. and K.K.; funding acquisition, H.A.G. and K.K. All authors have read and agreed to the pub- lished version of the manuscript. Funding: This research was supported by the European Union under the Horizon Europe Research and Innovation Action (RIA) project titled Customized Games and Routes for Cultural Heritage and Arts (Grant Agreement No. 101094428). The authors gratefully acknowledge the funding provided by the European Commission, which enabled the development of this study. The views and opinions expressed are those of the authors and do not necessarily reflect those of the European Union or the granting authority. Institutional Review: This study, related to our dataset collection experiment, has been approved by the Bilkent University IRB with approval number 662 on 01/03/2025. Abbreviations: The following abbreviations are used in this manuscript: 23 MCQMultiple-choice question KGKnowledge graph LLMLarge language model RNNRecurrent neural network GNNGraph neural network GATGraph attention network SPARQLSPARQL Protocol and RDF Query Language NERNamed entity recognition BFSBreadth-first search MoEMixture of experts BiLSTMBidirectional long short-term memory RTRLRelation-aware transformer with reinforcement learning KGNN-ADP Knowledge graph neural network-based adaptive difficulty prediction AutoDEAutomatic difficulty estimation FastRPFast random projection RMSERoot mean squared error MSEMean squared error MAEMean absolute error R 2 Coefficient of determination SHAPSHapley Additive exPlanations References Samah AlKhuzaey, Floriana Grasso, Terry R. Payne, and Valentina Tamma. Text-based question difficulty prediction: A systematic review of automatic approaches. International Journal of Artificial Intelligence in Education, 34(3):862â914, September 2023. ISSN 1560-4306. doi: 10.1007/s40593-023-00362-1. URL http://dx.doi.org/10.1007/s40593-023-00362-1. Tahani Alsubait, Bijan Parsia, and Ulrike Sattler. Ontology-based multiple choice question generation. KI - KĂźnstliche Intelligenz, 30(2):183â188, 2016. doi: 10.1007/s13218-015-0405-9. URL https://doi.org/10.1007/s13218-015-0405-9. Sheng Bi, Jianyu Liu, Zeyi Miao, and Qizhi Min. Difficulty-controllable question generation over knowledge graphs: A counterfactual reasoning approach. Information Processing & Manage- ment, 61(4):103721, 2024. ISSN 0306-4573. doi: https://doi.org/10.1016/j.ipm.2024.103721. URL https://w.sciencedirect.com/science/article/pii/S0306457324000815. Haochen Chen, Syed Fahad Sultan, Yingtao Tian, Muhao Chen, and Steven Skiena. Fast and accurate network embeddings via very sparse random projection. In Proceedings of the 28th ACM International Conference on Information and Knowledge Management, CIKM â19, page 399â408, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450369763. doi: 10.1145/3357384.3357879. URL https://doi.org/10.1145/3357384. 3357879. Yu Chen, Lingfei Wu, and Mohammed J. Zaki. Toward subgraph-guided knowledge graph question generation with graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, pages 1â12, 2023. doi: 10.1109/TNNLS.2023.3264519. Yi Cheng, Siyao Li, Bang Liu, Ruihui Zhao, Sujian Li, Chenghua Lin, and Yefeng Zheng. Guiding the growth: Difficulty-controllable question generation through step-by-step rewriting. In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, Proceedings of the 59th 24 Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5968â 5978, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/v1/ 2021.acl-long.465. URL https://aclanthology.org/2021.acl-long.465/. Hady Elsahar, Christophe Gravier, and Frederique Laforest. Zero-shot question generation from knowledge graphs for unseen predicates and entity types. In Marilyn Walker, Heng Ji, and Amanda Stent, editors, Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 218â228, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18653/v1/N18-1020. URL https://aclanthology.org/ N18-1020. R. Flesch. A new readability yardstick. Journal of Applied Psychology, 32:221â233, 1948. doi: 10.1037/h0057532. Yifan Gao, Lidong Bing, Wang Chen, Michael R. Lyu, and Irwin King. Difficulty controllable generation of reading comprehension questions, 2019. URL https://arxiv.org/abs/1807. 03586. Wenyu Huang, Guancheng Zhou, Mirella Lapata, Pavlos Vougiouklis, Sebastien Montella, and Jeff Z. Pan. Prompting large language models with knowledge graphs for question answer- ing involving long-tail facts. Knowledge-Based Systems, 324:113648, 2025. ISSN 0950-7051. doi: https://doi.org/10.1016/j.knosys.2025.113648. URL https://w.sciencedirect.com/ science/article/pii/S095070512500694X. Boran Jiang, Yuqi Wang, Yi Luo, Dawei He, Peng Cheng, and Liangcai Gao. Reasoning on efficient knowledge paths: Knowledge graph guides large language model for domain question answering. In 2024 IEEE International Conference on Knowledge Graph (ICKG), pages 142â 149, 2024. doi: 10.1109/ICKG63256.2024.00026. Benny G. Johnson, Rachel Van Campenhout, Bill Jerome, MarĂa Fernanda Castro, Ro- drigo Bistolfi, and Jeffrey S. Dittel. Automatic question generation for spanish textbooks: Evaluating spanish questions generated with the parallel construction method. Interna- tional Journal of Artificial Intelligence in Education, April 2024. ISSN 1560-4306. doi: 10.1007/s40593-024-00394-1. URL http://dx.doi.org/10.1007/s40593-024-00394-1. Archana Praveen Kumar, Ashalatha Nayak, Manjula Shenoy K, Chaitanya, and Kaustav Ghosh. A novel framework for the generation of multiple choice question stems using semantic and machine-learning techniques. International Journal of Artificial Intelligence in Education, 34 (2):332â375, March 2023. ISSN 1560-4306. doi: 10.1007/s40593-023-00333-6. URL http: //dx.doi.org/10.1007/s40593-023-00333-6. Vishwajeet Kumar, Yuncheng Hua, Ganesh Ramakrishnan, Guilin Qi, Lianli Gao, and Yuan- Fang Li. Difficulty-controllable multi-hop question generation from knowledge graphs. In Chiara Ghidini, Olaf Hartig, Maria Maleshkova, VojtÄch SvĂĄtek, Isabel Cruz, Aidan Hogan, Jie Song, Maxime Lefrançois, and Fabien Gandon, editors, The Semantic Web â ISWC 2019, pages 382â398, Cham, 2019. Springer International Publishing. ISBN 978-3-030-30793-6. Selvia Ferdiana Kusuma, Daniel Oranova Siahaan, and Chastine Fatichah. Automatic ques- tion generation with various difficulty levels based on knowledge ontology using a query template. Knowledge-Based Systems, 249:108906, 2022. ISSN 0950-7051. doi: https: 25 //doi.org/10.1016/j.knosys.2022.108906. URL https://w.sciencedirect.com/science/ article/pii/S0950705122004336. J. Leo, G. Kurdi, N. Matentzoglu, B. Parsia, U. Sattler, S. Forge, G. Donato, and W. Dowling. Ontology-based generation of medical, multi-term mcqs. International Journal of Artificial Intelligence in Education, 29(2):145â188, January 2019. ISSN 1560-4306. doi: 10.1007/ s40593-018-00172-w. URL http://dx.doi.org/10.1007/s40593-018-00172-w. Zhenping Li, Zhen Cao, Pengfei Li, Yong Zhong, and Shaobo Li. Multi-hop question generation with knowledge graph-enhanced language model. Applied Sciences, 13(9), 2023. ISSN 2076- 3417. doi: 10.3390/app13095765. URL https://w.mdpi.com/2076-3417/13/9/5765. Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In Proceedings of the 31st International Conference on Neural Information Processing Sys- tems, NIPSâ17, page 4768â4777, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860964. Scott M. Lundberg, Gabriel Erion, Hugh Chen, Alex DeGrave, Jordan M. Prutkin, Bala Nair, Ronit Katz, Jonathan Himmelfarb, Nisha Bansal, and Su-In Lee. From local explanations to global understanding with explainable ai for trees. Nature machine intelligence, 2, 2020. doi: 10.1038/s42256-019-0138-9. URL https://par.nsf.gov/biblio/10167481. Sathish Reddy, Dinesh Raghu, Mitesh M. Khapra, and Sachindra Joshi. Generating natural language question-answer pairs from a knowledge graph using a RNN based question gener- ation model. In Mirella Lapata, Phil Blunsom, and Alexander Koller, editors, Proceedings of the 15th Conference of the European Chapter of the Association for Computational Lin- guistics: Volume 1, Long Papers, pages 376â385, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https://aclanthology.org/E17-1036. Dominic Seyler, Mohamed Yahya, and Klaus Berberich. Knowledge questions from knowledge graphs. In Proceedings of the ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR â17, page 11â18, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450344906. doi: 10.1145/3121050.3121073. URL https://doi.org/ 10.1145/3121050.3121073. Irmak TĂźrkĂśz SoykĂśk and H. Altay GĂźvenir. Multi-label multi-modal classification of movie scenes. Knowledge-Based Systems, 318:113459, 2025. ISSN 0950-7051. doi: https://doi.org/ 10.1016/j.knosys.2025.113459. URL https://w.sciencedirect.com/science/article/ pii/S0950705125005064. Kristiyan Vachev, Momchil Hardalov, Georgi Karadzhov, Georgi Georgiev, Ivan Koychev, and Preslav Nakov. Leaf: Multiple-choice question generation. In Advances in Informa- tion Retrieval: 44th European Conference on IR Research, ECIR 2022, Stavanger, Nor- way, April 10â14, 2022, Proceedings, Part I, page 321â328, Berlin, Heidelberg, 2022. Springer-Verlag. ISBN 978-3-030-99738-0. doi: 10.1007/978-3-030-99739-7\_41. URL https://doi.org/10.1007/978-3-030-99739-7_41. Lai Wei and Guo-Sheng Hao. Knowledge graph based absolute difficulty prediction of multiple choice question. In 2024 9th International Conference on Electronic Technology and Informa- tion Science (ICETIS), pages 720â727, 2024. doi: 10.1109/ICETIS61828.2024.10593775. Hongwei Zeng, Bifan Wei, and Jun Liu. Rtrl: Relation-aware transformer with reinforce- ment learning for deep question generation. Knowledge-Based Systems, 300:112120, 2024. 26 ISSN 0950-7051. doi: https://doi.org/10.1016/j.knosys.2024.112120. URL https://w. sciencedirect.com/science/article/pii/S0950705124007548. Junjie Zhu, Dongfeng Liu, and Silun Chen. Multiple-choice question generation and difficulty calculations based on semantic similarity. Neural Computing and Applications, pages 1â13, 12 2024. doi: 10.1007/s00521-024-10671-8. 27