Paper deep dive
From Untamed Black Box to Interpretable Pedagogical Orchestration: The Ensemble of Specialized LLMs Architecture for Adaptive Tutoring
Nizam Kadir
Intelligence
Status: succeeded | Model: google/gemini-3.1-flash-lite-preview | Prompt: intel-v1 | Confidence: 97%
Last extracted: 3/26/2026, 2:14:29 AM
Summary
The paper introduces the Ensemble of Specialized LLMs (ES-LLMs) architecture, a neuro-symbolic approach for adaptive tutoring that decouples pedagogical decision-making from natural language generation. By using a deterministic, rule-based orchestrator guided by Bayesian Knowledge Tracing (BKT) and specialized agents, the system enforces pedagogical constraints (e.g., 'attempt-before-hint') and improves efficiency, resolving the 'Mastery Gain Paradox' where monolithic LLMs inflate performance through over-assistance.
Entities (5)
Relation Signals (3)
ES-LLMs â utilizes â BKT
confidence 100% · guided by an interpretable Bayesian Knowledge Tracing (BKT) student model.
Orchestrator â coordinates â Pedagogical Agents
confidence 95% · The Orchestrator coordinates six core specialized agents
ES-LLMs â outperforms â Monolithic LLMs
confidence 95% · The architecture significantly outperformed monolithic baselines across all seven dimensions
Cypher Suggestions (0)
No Cypher suggestions yet.
Abstract
Abstract:Monolithic Large Language Models (LLMs) used in educational dialogue often behave as "black boxes," where pedagogical decisions are implicit and difficult to audit, frequently violating instructional constraints by providing answers too early. We introduce the Ensemble of Specialized LLMS (ES-LLMS) architecture that separates decision-making from wording. Pedagogical actions are selected by a deterministic rules-based orchestrator coordinating specialized agents covering tutoring, assessment, feedback, scaffolding, motivation and ethics-guided by an interpretable Bayesian Knowledge Tracing (BKT) student model. An LLM renderer surface-realizes the chosen action in natural language. This design emphasizes reliability and controllability: constraints such as "attempt-before-hint" and hint caps are enforced as explicit rules, and the system logs per-turn agent traces and constraint checks. Validation of pedagogical quality via human expert reviewers (N=6) and a multi-LLM-as-Judge panel (six state-of-the-art models) showed that ES-LLMs were preferred in 91.7% and 79.2% of cases, respectively. The architecture significantly outperformed monolithic baselines across all seven dimensions, particularly in Scaffolding & Guidance, and Trust & Explainability. Furthermore, a Monte Carlo simulation (N=2,400) exposed a "Mastery Gain Paradox," where monolithic tutors inflated short-term performance through over-assistance. In contrast, ES-LLMs achieved 100% adherence to pedagogical constraints (e.g., attempt-before-hint) and a 3.3x increase in hint efficiency. Operationally, ES-LLMs reduced costs by 54% and latency by 22% by utilizing stateless prompts. We conclude that structural decoupling is essential for transforming stochastic models into trustworthy, verifiable and resource-efficient pedagogical agents.
Tags
Links
- Source: https://arxiv.org/abs/2603.23990v1
- Canonical: https://arxiv.org/abs/2603.23990v1
Trouble viewing inline? Open PDF directly â
Full Text
39,495 characters extracted from source content.
Expand or collapse full text
From Untamed Black Box to Interpretable Pedagogical Orchestration: The Ensemble of Specialized LLMs Architecture for Adaptive Tutoring Nizam Kadir 1[0000-0002-6725-1133] 1 Singapore University of Technology and Design, Singapore 487372, Singapore nizam_kadir@mymail.sutd.edu.sg Abstract. Monolithic Large Language Models (LLMs) used in educational dia- logue often behave as "black boxes," where pedagogical decisions are implicit and difficult to audit, frequently violating instructional constraints by providing answers too early. We introduce the Ensemble of Specialized LLMs (ES-LLMs) architecture that separates decision-making from wording. Pedagogical actions are selected by a deterministic rules-based orchestrator coordinating specialized agentsâcovering tutoring, assessment, feedback, scaffolding, motivation and ethicsâguided by an interpretable Bayesian Knowledge Tracing (BKT) student model. An LLM renderer surface-realizes the chosen action in natural language. This design emphasizes reliability and controllability: constraints such as âat- tempt-before-hintâ and hint caps are enforced as explicit rules, and the system logs per-turn agent traces and constraint checks. Validation of pedagogical qual- ity via human expert reviewers (N=6) and a multi-LLM-as-Judge panel (six state- of-the-art models) showed that ES-LLMs were preferred in 91.7% and 79.2% of cases, respectively. The architecture significantly outperformed monolithic base- lines across all seven dimensions, particularly in Scaffolding & Guidance, and Trust & Explainability. Furthermore, a Monte Carlo simulation (N=2,400) ex- posed a "Mastery Gain Paradox," where monolithic tutors inflated short-term per- formance through over-assistance. In contrast, ES-LLMs achieved 100% adher- ence to pedagogical constraints (e.g., attempt-before-hint) and a 3.3x increase in hint efficiency. Operationally, ES-LLMs reduced costs by 54% and latency by 22% by utilizing stateless prompts. We conclude that structural decoupling is es- sential for transforming stochastic models into trustworthy, verifiable and re- source-efficient pedagogical agents. Keywords: Intelligent Tutoring Systems, Large Language Models, Multi-agent Tutors. 1 Introduction Large Language Models (LLMs) have solved the âFluency Problemâ in educational dialogue but exacerbated the âControl Problemâ [1]. In safety-critical tutoring, mono- lithic tutors often violate pedagogical constraintsâgiving away answers too earlyâ due to the inherent bias of general-purpose LLMs toward frictionless user satisfaction 2 N. Kadir [2]. We term this the Mastery Gain Paradox: a phenomenon where users achieve high short-term performance metrics (via over-assistance) while their latent mastery stag- nates or degrades. Classical Intelligent Tutoring Systems (ITS) prioritize theoretically grounded deci- sion logic (e.g., Bayesian Knowledge Tracing (BKT) [3]) but lack generative flexibil- ity. To address this gap, this paper investigates the following central research problem: How can we harness the conversational fluency of LLMs for adaptive tutoring while strictly guaranteeing adherence to pedagogical constraints and procedural fairness? We propose Ensemble of Specialized LLMs (ES-LLMs), a hybrid neuro-symbolic archi- tecture that treats the tutor not as a monolithic tool, but as a team of coordinated spe- cialists. Importantly, the term "ensemble" here refers to the systematic coordination of multiple deterministic, rule-based pedagogical policies (e.g., assessment, scaffolding, ethics) computing in parallel, rather than an ensemble of expensive stochastic LLM calls. By decoupling the deterministic pedagogical decision from the single-call LLM surface realization, ES-LLMs restores auditability and trust [1]. 2 Related Work 2.1 From Rule-Based ITS to Generative Tutors Classical ITS like Cognitive Tutors [4, 5] use model tracing and BKT [3] to deliver personalized, rule-based instruction with proven learning gains [6, 7]. Performance Fac- tors Analysis (PFA) [8] and neural alternatives exist but lack the transparency needed for real-time safety orchestration. Recent work explores fine-tuning LLMs for feedback [9] and teachable agents [10]. However, even optimized monolithic models struggle with negative constraints (e.g., âdo not reveal the answerâ). Multi-agent approaches suggest decomposing roles but often leave the logic implicit in prompts. ES-LLMs makes these roles deterministic and rule-governed. 2.2 Simulation and Automated Evaluation Methods Simulation-Based Mastery Learning (SBML) [11] posits that mastery in simulation (T1) is a prerequisite for translational outcomes in real-world practice (T2) and down- stream impact (T3). We apply this framework by utilizing Monte Carlo simulations with synthetic students to stress-test reliability as validation of pedagogical rigor. Tra- ditional metrics like BLEU are poor for tutoring. Emerging âLLM-as-judgeâ methods [12] and Panels of LLMs (PoLL) [13] offer scalability, but human expert review re- mains the gold standard. 2.3 Procedural Fairness and Interpretability Fairness in AIED [14] requires more than demographic parity; it necessitates process consistency (procedural fairness), which interprets as architectural safety in our frame- work. It is also worth noting that fairness ties into the interpretability of the tutor: an interpretable system allows stakeholders to detect if it is treating some students differ- ently [15]. By maintaining a rule-based decision layer, educators can audit and adjust From Untamed Black Box to Interpretable Pedagogical Orchestration 3 the policies to be fair (for instance, by adding rules that ensure all students receive mo- tivational feedback after a certain number of errors, to avoid neglecting quieter stu- dents). As we progress to real-world deployment, we plan to incorporate privacy-pre- serving collection of demographic data (with consent) to directly evaluate fairness met- rics and mitigate any biases. For this paperâs scope, we demonstrate how fairness is considered via design (agents like EthicsBot) and via offline cohort evaluations. 3 System Architecture and Implementation Fig. 1 illustrates the architecture of our Ensemble of Specialized LLMs (ES-LLMs) tutoring system. The design is a pipeline with multiple stages, integrating data-driven modeling and rule-based decision making. Conceptually, ES-LLMs can be mapped to a triarchic tutoring blueprintâExpert Model (domain/solution knowledge), Learner Model (BKT mastery), and Tutor Model (agent policies and orchestrator)âwith the LLM restricted to surface realization. Fig. 1. Architecture of the ES-LLMs adaptive tutoring system. A BKT student model produces mastery posteriors that drive specialized agent proposals, while a deterministic rules-based or- chestrator enforces sequencing and constraints (e.g., attempt-before-hint and hint caps) and rec- ords per-turn traces. An LLM renderer is used for wording of the selected action; all decisions and constraint checks are logged for system-level evaluation. 4 N. Kadir 3.1 Data Ingestion and Feature Engineering The system ingests student interaction logs from ASSISTments 2017 [16] tutoring in- teractions (N=942,816) and computes a set of pedagogically relevant features for each attempt. These features include rolling accuracy (the studentâs recent success rate on a skill), hint usage counts and time since last hint, time-on-task durations, number of op- portunities practiced for each skill, and other indicators of possible frustration (such as âwheel-spinningâ when many attempts are made without success). The features are up- dated after each student action and stored for use by the student model. This prepro- cessing ensures that the tutor has a rich, contextual state representation at each turn. 3.2 Student Modeling The deployed tutor uses per-skill BKT as its runtime student model [3]. BKT maintains a mastery posterior for each skill (N=102) and updates this belief after each student attempt (e.g., correct/incorrect, with optional slip/guess and learning parameters). Agents and the orchestrator consult these mastery estimates to decide when to provide feedback, hints, scaffolding, or progression. 3.3 Pedagogical Agents: An Ensemble of Specialized LLMs The Orchestrator coordinates six core specialized agents (detailed in Table 1a) using a strict priority hierarchy: Safety First (EthicsBot filters harmful content), Assessment Second (AssessmentBot tracks mastery state), and Pedagogy Third (ScaffoldBot and MotivatorBot provide support). To ensure reliability, the ensemble operates in two stages. First, a deterministic decision layer evaluates the learner's state using strict rules and knowledge tracing (BKT) to select the single active agent, effectively muting the others. This modular design ensures that pedagogical strategy (e.g., scaffolding) is never overridden by generative fluency. Second, the selected agent is instantiated via a single, tailored LLM call. By structurally isolating each agent's logic before generation, the architecture creates a 'mixture of prompts' capable of specialized expertise without the latency or hallucination risks of uncontrolled agent interaction. Table 1b summarizes the supporting architecture modules in the ES-LLMs pipeline, including the meta-orchestrator, pedagogical policy, domain expert, affect detector, and LLM renderer, linking each component to its operational function within the system. Table 1a. Core Pedagogical Agents in ES-LLMs, their theoretical foundation, and implementa- tion rules Agent Role Evidence Basis Encoding in System AssessmentBot Mastery estima- tion Knowledge tracing / BKT [3, 17, 18], assessing learning [19] Updates after each at- tempt From Untamed Black Box to Interpretable Pedagogical Orchestration 5 FeedbackBot Corrective feed- back Feedback effective- ness [21, 22] Emits CONFIRM/ NUDGE/ REMEDIATE from correctness and mas- tery ScaffoldBot Hinting/ scaf- folding Scaffolding in tu- toring [23, 24] Emits HINT_ MIN/ MED/ FULL from er- ror windows and hint caps MotivatorBot Affective sup- port Motivation and af- fect dynamics [25, 26] Encouragement for low confidence or error streaks EthicsBot Guardrails Help-seeking and gaming constraints [27, 28] Enforces attempt- be- fore- hint and max- hint rules TutorBot Progression/ next item Mastery learning and time-to-learn [29, 30, 31] Chooses NEXT when mastery exceeds threshold Table 1b. Supporting Architecture Modules and their roles in the system pipeline Modules Role Evidence basis Encoding in system Meta-Orches- trator Decision coor- dination ITS architectures and effectiveness [5, 6, 7] Coordinates modules, orders actions, aggre- gates into one response Pedagogical Policy Action sequenc- ing Rule-based ITS policies [5, 20] Orders agents by cor- rectness, mastery, af- fect, streaks Affect Detector Affect state Affective dynamics in learning [26] Heuristic affect infer- ence from error pat- terns Domain Expert Client Content hints Human tutoring ex- planations [23, 32] Short content hint/ex- plain injected into feedback/scaffold Renderer (LLM) Wording and aggregation Feedback should be specific and con- cise [21, 22] One LLM call to sur- face-realize aggregated decisions 3.4 The Orchestrator The Orchestrator (or Meta-Orchestrator) functions as a deterministic, rule-based policy manager that arbitrates candidate actions from all agents to determine execution order. 6 N. Kadir We implement a priority hierarchy based on Subsumption Architecture [33], a robot- ics design pattern that manages competing goals through hierarchical suppression. In this educational adaptation, higher-priority layers suppress lower-priority ones to en- sure pedagogical safety: Safety (EthicsBot) > Assessment > Feedback > Scaffolding > Motivation. For instance, if EthicsBot detects a constraint violation (e.g., student attempt count = 0), it outputs a DENY_HINT action that immediately suppresses Scaf- foldBot's HINT_FULL proposal. Conversely, if no safety rules are violated and the student's BKT mastery probability exceeds a threshold (pL > 0.95), TutorBot triggers a NEXT_PROBLEM action. The Orchestrator's output is a structured representation of the final tutor turn, passed as an ordered list of decisions to a single aggregation step for surface realization. 3.5 LLM-based Renderer The system invokes an LLM to generate the dialogue text for the tutorâs turn. Instead of one LLM handling everything end-to-end, we use a single aggregation prompt that combines the ordered agent decisions into one concise response. In our prototype, the default realizer configuration uses gpt-4o-mini with temperature 0.3 and max_to- kens=120 for short, stable surface realizations; these parameters are part of the runtime configuration and are logged per run. The realizer prompt is structured as (i) a system role instruction (âcompose a single response from the ordered decisions; do not change the decisionsâ) and (i) a user message containing the ordered decisions plus a compact, sanitized context (skill, mastery posterior, attempt counts, constraint state, and relevant dialogue history). The key is that the LLM does not decide what pedagogical action to take; it only phrases the chosen actions in one response. If an API key is unavailable, the system falls back to deterministic wording templates. 3.6 Delivery and Interface The finalized tutor message (or sequence of messages) is delivered to the student. In our implementation, we built a simple chat-style web interface where the conversation between the student and the tutor (ES-LLMs or baseline) is displayed. Each tutor turn includes badges indicating which agents were activated (for the researchersâ debugging, this would be hidden from actual students). The system also logs all interactions and model outputs for analysis. User Interface Design. Fig. 2 shows the web-based tutoring interface through which students interact with the ES-LLMs system. Unlike black-box LLM tutors where decision-making is opaque, the ES-LLMs interface makes the system's pedagogical rea- soning visible to researchers, educators, and learners. The interface follows established design principles for intelligent tutoring systems, presenting a clean, distraction-free environment that focuses learner attention on the mathematical problem at hand. The layout consists of three primary zones: (1) a problem presentation area displaying the current mathematics question, (2) a student input field with a confidence slider (1-5 scale) where learners enter their answers, and (3) a tutor response area where the system provides feedback, scaffolding, or encouragement based on the orchestrator's pedagog- ical decisions. From Untamed Black Box to Interpretable Pedagogical Orchestration 7 (a) main tutoring interface (b) visible agent badges Fig. 2. The ES-LLMs interface: (a) student input and (b) interpretable agent orchestration 4 Methodology and Experimental Design 4.1 Research Questions We compare ES-LLMs to a monolithic baseline across two primary dimensions: (1) Does decoupling decision-making from generation resolve the "Mastery Gain Paradox" by improving pedagogical constraint adherence and hint efficiency? (2) Quality and Operational Efficiency: How does ES-LLMs compare to monolithic tutors in expert- rated pedagogical quality (across seven dimensions) and in computational resource ef- ficiency (latency and token cost)? 4.2 Baseline (Direct LLM) The baseline is a single-call, monolithic tutor that uses the exact same LLM (gpt-4o- mini) and decoding settings as ES-LLMs' renderer and agent calls. We intentionally selected a lightweight, standard instruction-tuned model (gpt-4o-mini) for both systems to isolate the impact of our orchestrator architecture from the inherent advanced rea- soning capabilities of frontier models (e.g., o1). The baseline was allowed a higher max_tokens limit of 300 to ensure its verbosity was not artificially truncated. It receives the same problem context and dialogue history but makes and phrases pedagogical de- cisions end-to-end in a single prompt. 4.3 Tutoring Scenarios and Dialogue Generation To reflect authentic learner behaviors, we used the ASSISTments 2017 dataset [16] (students N=1,709) in k-means clustering across three dimensions (skill difficulty, in- teraction sequence, and learner profile). We identified 8 distinct interaction signatures (e.g. Clean Correct, Hint Abuse, Deep Struggle). We crossed this with 3 difficulty tiers 8 N. Kadir to construct a controlled testbed of 24 fixed scenarios anchored to the skills in the Com- mon Core State Standards for Mathematics (C-SSM) for middle school (typically ages 11-14). 4.4 Monte Carlo Simulation with Synthetic Students We conducted a large-scale Monte Carlo simulation (N=2,400). We instantiated Syn- thetic Student Agents based on BKT parameters (P(L 0 ), P(T), P(S), P(G)) derived from the ASSISTments 2017 dataset [16]. We defined four learner archetypes (Strug- gling, Low, Average, High) using k-means clustering and ran 600 stochastic simula- tions per archetype. In each run, student parameters were perturbed by Gaussian noise to ensure robustness. This setup allows us to measure Procedural Fairness; verifying that the system's treatment (constraints) remains consistent despite stochastic variations in student behavior, and to statistically compare the learning efficiency of ES-LLMs against the baseline. We prioritized computational metrics from these 2,400 runs to evaluate the system's structural integrity. Drawing on the framework of Simulation- Based Mastery Learning (SBML) [11], we utilized synthetic student agents to ensure that the tutorâs pedagogical policy holds under stochastic variation. The simulation serves as a T1 validation of the 'Deliberate Practice' conditions required for effective learning. 4.5 Pedagogical Quality Evaluation by Human Experts and LLM Judges Human experts consisting of educators and educational researchers (N=6) evaluated the paired tutoring dialogues. We utilized a rigorous, double-blind 5-point Likert rubric adapted to assess seven critical pedagogical dimensions: Adaptivity, Scaffolding & Guidance, Ethical Reasoning, Engagement, Feedback Quality, Tone & Style, and Trust & Explainability. As a secondary signal, we conducted multi-LLMs-as-Judge evalua- tions using six state-of-the-art LLMs (Gemini 3 Pro, Qwen3-Max, Kimi K2, DeepSeek- V3, Grok-2, ChatGPT-5.2). Each model was prompted to act as an expert educator and assessed the 24 paired interactions using the exact same rubric applied by human eval- uators, capturing both quantitative scores and qualitative reasoning. 5 Results 5.1 Computational Metrics: The Efficiency-Fidelity Trade-off Monte Carlo simulations (Table 2) reveal key differences across pedagogical fidel- ity and resource efficiency dimensions. The Mastery Gain Paradox and Constraint Adherence. The simulations reveal a diagnostic failure we label the Mastery Gain Paradox: the baseline achieved a higher mastery gain (D = +0.70 vs. +0.40), but only by providing 12x more hints (6.8 vs. 0.44 per dialogue). The baseline defaulted to giving answers early to minimize friction, whereas ES-LLMs achieved 100% adher- ence to the attempt-before-hint constraint, encouraging productive struggle [7]. In From Untamed Black Box to Interpretable Pedagogical Orchestration 9 addition, ES-LLMs has higher hint efficiency (which is calculated as the mastery gain per hint). Table 2. Computational Metrics from Monte Carlo Simulation (N=2,400). Significance via Wil- coxon Signed-Rank Test. Metric Type Baseline Ï ES-LLMs Ï Wilcoxon p Mastery Gain (BKT) 0.70 0.19 0.40 0.42 < 0.001*** Constraint Adherence 62.4% 22.1 100.0% 0.0 < 0.001*** Hint Efficiency 0.10 0.05 0.33 0.38 <0.001*** Latency (ms) 800 150 625 110 <0.05* Tokens 1300 220 590 95 <0.001*** Resource Efficiency. Beyond pedagogical gains, ES-LLMs demonstrates substan- tial computational advantages. By separating deterministic decision logic from natural language generation, the architecture achieves 54% cost reduction (590 vs. 1,300 to- kens per turn, p<0.001) and 22% latency improvement (625ms vs. 800ms, p<0.05). The efficiency stems from stateless NLG promptsâagents pass only decision-relevant context (action type, skill state, mastery level) rather than full conversation history, re- ducing input tokens 2.2Ă. At scale, this translates to $1.41 savings per 1,000 sessions while maintaining 100% constraint adherence, demonstrating that pedagogical correct- ness and operational efficiency are complementary, not competing, design goals. 5.2 Unanimous Superiority in Pedagogical Quality Six human experts and six LLM judges evaluated the 24 scenario pairs. Out of the 144 total comparative evaluations (6 x 24 items), the ES-LLMs architecture was selected as superior in 91.7% (n=132) and 79.2% of cases (n=114) by humans and LLM judges respectively. This is compared to 6.9% (n=10) and 13.2% for the baseline (n=19) re- spectively. Tied outcomes were 1.4% (n=2) and 7.6% (n=11) respectively. Table 3 shows aggregate dimensional scores. ES-LLMs significantly outperformed baseline on all 7 dimensions for both human experts and LLM judges. Qualitative Themes. Thematic analysis of evaluator reasoningâfrom both human experts and LLM judgesârevealed three critical pedagogical distinctions: Prevention of "Gaming the System". ES-LLMs consistently detected and mitigated hint abuse (e.g., students spamming "idk" or clicking hints without reading), withhold- ing answers until effort was demonstrated. Human experts noted this in 75% of cases (n=18), with one educator commenting: "Dialogue A [ES-LLMs] correctly detected 'idk' spam and withheld the answerâ. The baseline provided answers immediately, en- abling gaming strategies. Deep Remediation vs. Surface Loops. ES-LLMs employed deeper remediation strat- egies (using analogies like money vs temperature), while the baseline tended to repeat the same formulaic advice. Human experts highlighted this in 58% of evaluations (n=14), noting: "Dialogue A [ES-LLMs] changed strategies with a concrete pizza 10 N. Kadir analogy when the student was deeply stuck". LLM judges concurred, with three models (DeepSeek-V3, Kimi K2, ChatGPT-5.2) rating ES-LLMs higher on Scaffolding & Guidance (mean difference: +1.99 human, +1.84 LLM, p<.001). Table 3. Human Expert and Multi-LLMs-as-Judge Aggregate Scores comparing ES-LLMs and Baseline. Ratings are on a 1â5 Likert scale across 24 tutoring dialogue pairs. Positive differences indicate ES-LLMs superiority. All differences are statistically significant (p<.01 for Ethical Rea- soning and Tone and Style; p<.001 for all other dimensions). Dimension ES-LLMs Mean Baseline Mean Difference LLMs Humans LLMs Humans LLMs Humans Adaptivity 4.82 4.62 3.55 2.96 +1.27 +1.66 Scaffolding & Guidance 4.78 4.67 2.94 2.68 +1.84 +1.99 Ethical Reasoning 4.95 4.88 4.10 4.01 +0.85 +0.87 Engagement 4.86 4.29 3.42 3.12 +1.44 +1.17 Feedback Quality 4.89 4.67 3.78 3.59 +1.11 +1.08 Tone and Style 5.00 4.76 4.20 4.15 +0.80 +0.61 Trust & Explainability 4.70 4.67 3.10 2.80 +1.60 +1.87 Ethical Reasoning. Both evaluator types rated systems highly on safety, but human experts emphasized ES-LLMs' avoidance of misleading feedback in 46% of cases (n=11): "Dialogue B [baseline] said the answer was 'close' when completely wrong. Dialogue A [ES-LLMs] never misled". Despite strong performance by both systems (ES-LLMs: 4.88, Baseline: 4.01), ES-LLMs maintained an edge (Î=+0.87, p<.01). Efficiency Trade-off. In a minority of cases where the baseline outperformed ES- LLMs (e.g., 3+0, 1+1) human experts noted the orchestration overhead: "A difficulty assessment layer could have bypassed unnecessary scaffolding." Three LLM judges also flagged these edge cases, suggesting future work should include adaptive orches- tration depth. 5.3 Interpretable Constraint Enforcement: From Opaque to Transparent Fig. 4a shows how the interface transforms invisible algorithmic decisions into trans- parent feedback. When a student triggers a gaming detector, the system displays a "HINT DENIED" badge with an explicit pedagogical rationale ("I need to see you try first..."). This visualizes the system's deterministic logic, converting a potential "black- box" refusal into an interpretable educational moment that reinforces the attempt-be- fore-hint protocol. The "Team Player" Ensemble in Action. A 6-turn deep remediation sequence illustrates the orchestratorâs strategic escalation of support (see Fig. 4b). The progres- sion of agent badges (from REMEDIATE DEEP to HINT FULL) reveals how the system dynamically shifts specialists in response to persistent struggle. The dialogue also high- lights the interleaving of MotivatorBotâs encouragement alongside ScaffoldBotâs in- structional guidance. By explicitly labeling which agent contributes to each turn, the interface demonstrates the collaborative nature of the ensemble, providing a level of transparency and context-awareness that opaque monolithic models lack. From Untamed Black Box to Interpretable Pedagogical Orchestration 11 (a) Constraint enforcement (b) Deep remediation sequence Fig. 4. Transparency in action: (a) HINT_DENIED rationale and (b) agent escalation traces 6 Discussion Our architecture targets a reliability gap in LLM tutoring: monolithic tutors can be prompt-sensitive and difficult to constrain, while educational settings often require con- sistent enforcement of pedagogical rules. Below we discuss implications and limitations of an orchestrated, multi-agent ES-LLMs design. 6.1 Implications for Adaptive Tutor Design A core implication of ES-LLMs is that pedagogical policy can be externalized as ex- plicit, testable rules. This allows for the rigorous enforcement of 'minimum passing standards' and 'deliberate practice' protocols, which are identified as the essential com- ponents of simulation-based mastery learning [11]. Unlike monolithic models where these standards are implicit and probabilistic, ES-LLMs guarantees the rigorous prac- tice environment necessary for translational outcomes. Another implication is the maintainability and extensibility of such a system. The modular ES-LLMs design means one can improve or swap out components inde- pendently for example, replacing the BKT student model with a more advanced model 12 N. Kadir (e.g., deep knowledge tracing) without changing the orchestration or wording layers. In contrast, the monolithic approach is a single entity: if it is not performing well, the primary recourse is prompt engineering or fine-tuning, which are less transparent than adjusting an explicit rule or threshold in a multi-agent system. This modular approach aligns with good software engineering (and MLOps) principles. In production, a multi- agent tutor may also be easier to troubleshoot: if responses are off, one can inspect whether the issue was with the student model estimate, a bot's rule, or the LLM phras- ing; with a monolith, it is often unclear why a decision was made. 6.2 Broader Implications: The Policy-Generation Decoupling Paradigm Beyond intelligent tutoring, the ES-LLMs architecture illustrates a generalizable solu- tion for high-stakes AI applications: the structural decoupling of decision-making (policy) from surface realization (generation). This pattern offers a robust alternative to purely probabilistic models in domains like healthcare and customer operations, where strict adherence to clinical protocols or escalation logic is non-negotiable. By externalizing control logic into a deterministic orchestrator, the system replaces malle- able prompt-based guardrails with architectural constraints. This approach successfully reconciles the tension between the necessary rigidity of rule-based compliance and the conversational fluency of Large Language Models, providing a template for trustwor- thy AI alignment. 6.3 Alignment Between AI and Human Evaluation One interesting next step is the concept of calibrating LLM evaluators for educational criteria. Perhaps an LLM could be fine-tuned on a dataset of human ratings of tutor responses to become a more accurate âeducation-specific judge.â There is emerging work on improving LLM-evaluator prompts and calibration for better human alignment [12], and our study can contribute data to that effort in the tutoring domain. In practice, a combination of human and LLM evaluation may yield the best efficiency and relia- bility: use LLM judges to triage obvious failures and monitor regressions but use peri- odic human audits to ensure pedagogical quality is truly high [12]. 6.4 Fairness and Ethical Auditability We define fairness in AI tutoring not merely as demographic equity, but as behavioral consistencyâensuring uniform pedagogical support across varying skill levels [14]. The ES-LLMs architecture achieves Safety by Design by offloading constraint en- forcement to a deterministic orchestrator. This eliminates the "stochastic unfairness" inherent in monolithic LLMs, where identical student inputs can yield varying levels of support due to model probability. Our simulations confirmed that ES-LLMs maintained 100% protocol compliance across all behavioral cohorts (e.g., high-mastery vs. gaming behaviors), whereas the baseline exhibited high variance that could disadvantage strug- gling learners. Furthermore, unlike "black box" tutors, the interpretable decision layer allows for granular auditing of policy enforcement, providing a necessary foundation for future demographic equity studies in real-world deployments. From Untamed Black Box to Interpretable Pedagogical Orchestration 13 6.5 Limitations Several limitations must be acknowledged. First, and primarily, we report quantitative efficacy metrics based on Monte Carlo synthetic students rather than real-world learn- ing outcomes. While these simulations successfully establish architectural reliability (e.g., verifying 100% constraint adherence and procedural fairness under massive sto- chastic variation), synthetic learners governed by idealized BKT parameters do not per- fectly mirror the messy cognitive and affective realities of human classroomsâsuch as unpredictable slips or attention drift. As emphasized by Simulation-Based Mastery Learning frameworks [11], this computational validation constitutes a necessary Phase I trial to ensure system safety before deployment. Controlled, in-classroom studies with human learners are required to establish true educational effectiveness. Second, our evaluation scope was restricted to foundational, single-problem mathe- matics tutoring. Findings may not generalize to open-ended domains or multi-session interactions, and future applications must assess the architecture's capacity to orches- trate scaffolding for semantically complex, multi-step word problems. Third, we com- pared ES-LLMs against a standard prompt-only baseline; incorporating stronger base- lines tailored with few-shot or self-critique methods might alter the comparative results. Finally, our multi-agent architecture is inherently more complex than a monolithic so- lution, introducing potential points of failure and requiring more engineering resources to maintain. 6.6 Future Work Future research will prioritize a user study with real students to validate learning out- comes and engagement. In the taxonomy of Simulation-based Mastery Learning (SBML) [11], the current study establishes strong T1 outcomes (simulation-based reli- ability and constraint adherence). The next phase of research aims to establish T2 trans- lational outcomes (improved problem-solving practices in the classroom) and ulti- mately T3 outcomes (long-term mastery retention). The simulation results provide the necessary confidence that the system is safe and rigorous enough for this transition. We also plan to explore adaptive orchestration policies that move beyond static rules, potentially using reinforcement learning to fine-tune agent sequencing. Since the baseline outperformed ES-LLMs on trivial problems, we propose adding a difficulty assessment layer. This would route simple items directly to the LLM for immediate confirmation while reserving full orchestration for challenging tasks, improving effi- ciency without sacrificing rigor. Improving the LLM phrasing layer is another key goal. We intend to fine-tune models on high-quality tutor phrasing to enhance tone and brevity while ensuring the LLM remains strictly a wording engine. 7 Conclusion We introduced ES-LLMs, an architecture that decouples deterministic pedagogical de- cisions from generative wording. This structural separation ensures auditable tutoring, addressing the reliability risks of monolithic models. Human and LLM-as-Judge 14 N. Kadir evaluations confirmed that ES-LLMs significantly outperform baselines in pedagogical quality. Furthermore, simulations (N=2,400) exposed a "Mastery Gain Paradox" where monolithic tutors inflate performance through over-assistance. In contrast, ES-LLMs achieved 100% constraint adherence and a 3.3x increase in hint efficiency. Operation- ally, the architecture reduced costs by 54% and latency by 22%. Future work will trans- late these simulation-based gains to controlled studies with human learners. Acknowledgments. We would like to thank Assoc. Prof. Dorien Herremans, Dr Nachamma Sockalingam, Mr. Yang Xingyu and Mr. Syed Adeel Ahmed Mustaq Ahamed for their involve- ment in this work. We also extend our deepest gratitude to the six human expert reviewers for their time and essential insights in evaluating the pedagogical quality of the system. Disclosure of Interests / Ethics Approval. The author has no competing interests to declare that are relevant to the content of this article. This study was approved by the Singapore University of Technology and Design (SUTD) Institutional Review Board under reference code S-25-782. Data Availability and Reproducibility. The pedagogical ruleset, evaluation rubrics, and sample scenario data are open-source and available at: https://github.com/nizamkadirteach/aied2026-es- llms. References 1. Kadir, N.: Machine Pedagogical Intelligence: The Cognitive and Computational Architec- tures of Automated Instruction. 1st edn. AcademiaX Press (2025) 2. Scarlatos, A., et al.: Training LLM-based Tutors to Improve Student Learning Outcomes in Dialogues. In: AIED 2025 Proceedings (2025) 3. Corbett, A.T., Anderson, J.R.: Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 253â278 (1995) 4. Anderson, J.R., Conrad, F.G., Corbett, A.T.: Skill acquisition and the LISP tutor. Cognitive Science 13(4), 467â505 (1989) 5. Anderson, J.R., Corbett, A.T., Koedinger, K.R., Pelletier, R.: Cognitive Tutors: Lessons Learned. Journal of the Learning Sciences 4(2), 167â207 (1995) 6. Kulik, J.A., Fletcher, J.D.: Effectiveness of Intelligent Tutoring Systems. Review of Educa- tional Research 86(1), 42â78 (2016) 7. VanLehn, K.: The Relative Effectiveness of Human Tutoring, Intelligent Tutoring Systems, and Other Tutoring Systems. Educational Psychologist 46(4), 197â221 (2011) 8. Pavlik, P.I., Cen, H., Koedinger, K.R.: Performance Factors AnalysisâA New Alternative to Knowledge Tracing. In: Proc. of AIED 2009, p. 531â538 (2009) 9. Scarlatos, A., Smith, D., Woodhead, S., Lan, A.: Improving the Validity of Automatically Generated Feedback via Reinforcement Learning. In: Artificial Intelligence in Education, p. 280â294. Springer, Cham (2024) 10. Ma, Q., Shen, H., Koedinger, K., Wu, S.T.: How to Teach Programming in the AI Era? Using LLMs as a Teachable Agent for Debugging. In: Artificial Intelligence in Education, p. 265â279. Springer, Cham (2024) 11. McGaghie, W.C., Issenberg, S.B., Barsuk, J.H., Wayne, D.B.: A critical review of simula- tion-based mastery learning with translational outcomes. Medical Education 48(4), 375â385 (2014) From Untamed Black Box to Interpretable Pedagogical Orchestration 15 12. Liu, Y., et al.: G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2023) 13. Verga, P., et al.: Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv:2404.18796 (2024) 14. Khalil, M., Prinsloo, P., Slade, S.: Fairness, Trust, Transparency, Equity, and Responsibility in Learning Analytics. Journal of Learning Analytics 10(1), 1â7 (2023) 15. Deck, L., Schoeffer, J., De-Arteaga, M., KĂŒhl, N.: A Critical Survey on Fairness Benefits of Explainable AI. In: Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, p. 1579â1595. ACM (2024) 16. Heffernan, N.T., Heffernan, C.L.: The ASSISTments Ecosystem: Building a Platform that Brings Scientists and Teachers Together for Minimally Invasive Research on Human Learn- ing and Teaching. International Journal of Artificial Intelligence in Education 24(4), 470â 497 (2014) 17. Dai, M., et al.: Knowledge tracing: A review of available technologies. Journal of Educa- tional Technology Development and Exchange 14(1), 1â24 (2021) 18. Lu, Y., et al.: Advanced knowledge tracing: A review of models and applications. Journal of Educational Data Mining 16(1), 1â32 (2024) 19. Black, P., Wiliam, D.: Assessment and Classroom Learning. Assessment in Education: Prin- ciples, Policy & Practice 5(1), 7â74 (1998) 20. Corbett, A.T.: Modeling student knowledge: Cognitive tutors in high school and college. User Modeling and User-Adapted Interaction 10(2â3), 81â108 (2000) 21. Hattie, J., Timperley, H.: The Power of Feedback. Review of Educational Research 77(1), 81â112 (2007) 22. Shute, V.J.: Focus on Formative Feedback. Review of Educational Research 78(1), 153â189 (2008) 23. Wood, D., Bruner, J.S., Ross, G.: The role of tutoring in problem solving. Journal of Child Psychology and Psychiatry 17(2), 89â100 (1976) 24. van de Pol, J., Volman, M., Beishuizen, J.: Scaffolding in TeacherâStudent Interaction: A Decade of Research. Educational Psychology Review 22(3), 271â296 (2010) 25. Deci, E.L., Ryan, R.M.: The âWhatâ and âWhyâ of Goal Pursuits: Human Needs and the Self-Determination of Behavior. Psychological Inquiry 11(4), 227â268 (2000) 26. D'Mello, S., Graesser, A.: Dynamics of affective states during complex learning. Learning and Instruction 22(2), 145â157 (2012) 27. Aleven, V., Roll, I., McLaren, B.M., Koedinger, K.R.: Help Helps, But Only So Much: Re- search on Help Seeking with Intelligent Tutoring Systems. International Journal of Artificial Intelligence in Education 26(1), 205â223 (2016) 28. Baker, R.S.J.d., de Carvalho, A.M.J.B., Raspat, J., Aleven, V., Corbett, A.T., Koedinger, K.R.: Educational Software Features that Encourage and Discourage âGaming the Systemâ. In: Artificial Intelligence in Education. IOS Press (2009) 29. Bloom, B.S.: Learning for mastery. Evaluation Comment 1(2), 1â12 (1968) 30. Carroll, J.B.: A model of school learning. Teachers College Record 64(8), 723â733 (1963) 31. Guskey, T.R.: Formative classroom assessment and Benjamin S. Bloom: Theory, research, and implications. Phi Delta Kappan 86(9), 623â629 (2005) 32. Chi, M.T.H., Siler, S.A., Jeong, H., Yamauchi, T., Hausmann, R.G.: Learning from human tutoring. Cognitive Science 25(4), 471â533 (2001) 33. Brooks, R.: A robust layered control system for a mobile robot. IEEE Journal of Robotics and Automation 2(1), 14â23 (1986)