Paper deep dive

Responsible Institutional Analytics: Interpreting Bias with AI Support

Francielle Marques, Ariel Ortiz-Beltrán, Ishari Amarasinghe, Davinia Hernández-Leo

Year: 2026Venue: arXiv preprintArea: cs.HCType: PreprintEmbeddings: 92

Intelligence

Status: succeeded | Model: Gemma-4-26B-A4B | Prompt: intel-v1 | Confidence: 91%

Last extracted: 10/7/2026, 8:52:14 AM

Summary

This paper introduces FACTRIA, a framework for identifying biasing factors in Institutional Analytics (IA) dashboards across four dimensions: analytics pipeline, institutional context, course-level characteristics, and demographics. The authors developed a generative AI chatbot that uses FACTRIA to prompt users to reflect on these factors during data interpretation. A qualitative study with stakeholders demonstrated that the AI-guided approach helps users recognize overlooked biases and supports more responsible, context-aware interpretation of institutional data.

Entities (10)

FACTRIA · framework · 98%Institutional Analytics · field · 95%Pompeu Fabra University · organization · 95%Generative AI Chatbot · system · 92%Analytics Pipeline · bias-category · 90%Course-Level Characteristics · bias-category · 90%Demographics · bias-category · 90%Institutional Context · bias-category · 90%Transition Network Analysis · method · 88%DELICATE · framework · 85%

Relation Signals (8)

FACTRIA → hascomponent → Analytics Pipeline

confidence 95% · FACTRIA framework that organizes potential biasing factors across four areas: the analytics pipeline...

FACTRIA → hascomponent → Institutional Context

confidence 95% · FACTRIA framework that organizes potential biasing factors across four areas: ... institutional context...

FACTRIA → hascomponent → Course-Level Characteristics

confidence 95% · FACTRIA framework that organizes potential biasing factors across four areas: ... course-level characteristics...

FACTRIA → hascomponent → Demographics

confidence 95% · FACTRIA framework that organizes potential biasing factors across four areas: ... and demographics.

Generative AI Chatbot → usesframework → FACTRIA

confidence 92% · We used the FACTRIA framework as input to a generative-AI chatbot designed to prompt users to reflect on these factors

Generative AI Chatbot → supports → Responsible Interpretation

confidence 90% · Findings indicated that combining a structured framework with AI-based guidance can enhance context-aware, responsible interpretation of institutional data.

Institutional Analytics → informs → Decision-making

confidence 85% · Institutional Analytics (IA) dashboards inform decision-making in higher education

Cypher Suggestions (0)

No Cypher suggestions yet.

Abstract

Abstract:Institutional Analytics (IA) dashboards inform decision-making in higher education, yet data limitations, constraints in analytical techniques, and missing contextual information often affect their interpretation. To support more responsible interpretation of IA, we introduce FACTRIA, a framework that organizes potential biasing factors across four areas: the analytics pipeline, institutional context, course-level characteristics, and demographics. We used the FACTRIA framework as input to a generative-AI chatbot designed to prompt users to reflect on these factors while analyzing IA. A qualitative study with stakeholders, drawing on four authentic IA cases, and a transition network analysis showed that the chatbot prompted participants to recognize how overlooked factors influenced their initial interpretation. Findings indicated that combining a structured framework with AI-based guidance can enhance context-aware, responsible interpretation of institutional data.

PDF

Open source PDF →Open local PDF →

Trouble viewing inline? Open PDF directly →

Full Text

89,470 characters extracted from source content.

Expand or collapse full text

Journal of Universal Computer Science, vol. 33, no. 6 (2027), 2987-2989 submitted: 26/05/2026, accepted: 1/10/2026, appeared: n.d C BY 4.0 Responsible Institutional Analytics: Interpreting Bias with AI Support Francielle Marques (Pompeu Fabra University, Barcelona, Catalonia https://orcid.org/0000-0002-1992-8728, francielle.marques@upf.edu) Ariel Ortiz-Beltrán (Pompeu Fabra University, Barcelona, Catalonia https://orcid.org/0000-0003-1522-2362, ariel.ortiz@upf.edu) Ishari Amarasinghe (Pompeu Fabra University, Barcelona, Catalonia https://orcid.org/0000-0003-2960-4804, ishari.amarasinghe@upf.edu) Davinia Hernández-Leo (Pompeu Fabra University, Barcelona, Catalonia https://orcid.org/0000-0003-0548-7455, davinia.hernandez-leo@upf.edu) Abstract:Institutional Analytics (IA) dashboards inform decision-making in higher education, yet data limitations, constraints in analytical techniques, and missing con- textual information often affect their interpretation. To support more responsible in- terpretation of IA, we introduce FACTRIA, a framework that organizes potential bias- ing factors across four areas: the analytics pipeline, institutional context, course-level characteristics, and demographics. We used the FACTRIA framework as input to a generative-AI chatbot designed to prompt users to reflect on these factors while ana- lyzing IA. A qualitative study with stakeholders, drawing on four authentic IA cases, and a transition network analysis showed that the chatbot prompted participants to recognize how overlooked factors influenced their initial interpretation. Findings in- dicated that combining a structured framework with AI-based guidance can enhance context-aware, responsible interpretation of institutional data. Keywords:Institutional Analytics, Generative AI, Transition Network Analysis, Dash- boards Categories:I.2, L.3, L.3.1 DOI:10.3897/jucs.200690 1Introduction Institutional Analytics (IA) is a field that helps higher education institutions report data and insights to inform and improve decision-making related to teach- ing and learning ( Yanosky & Arroway2015). Drawing on indicators from sources such as learning management systems, student performance, and student satis- faction surveys, it enables stakeholders (academic managers, educators) to make evidence-based decisions to enhance educational quality and student outcomes (Paz & Cazella2019). IA includes Learning Analytics (LA)—which examines arXiv:2610.07205v1 [cs.HC] 5 Oct 2026 2988 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support learners’ interactions, engagement, and progression within courses—while ex- tending these insights across programs, departments, and cohorts (Jones2019) to inform institutional systems, strategic planning, and stakeholders’ actions. As IA matures in higher education, research has shown that responsible in- terpretation depends not only on the indicators provided in IA dashboards but on users’ awareness of the potential biases embedded in the analytical context and processes (Williamson & Kizilcec2022). Biases can emerge from analyti- cal choices, characteristics of the underlying data, the data context, and how analysts frame or communicate results; each of these factors can influence how stakeholders interpret patterns and act on them (Olteanu et al.2019). Respon- sible interpretation, therefore, requires careful attention to sense-making, trans- parency about assumptions and limitations, and explicit consideration of factors that clarify findings, support comparability, and keep interpretations appropri- ately bounded by the data. Recent educational AI research further suggests that conversational scaffolding and reflective prompting can support users’ metacog- nitive engagement when interacting with complex information environments, al- though such systems require careful design to avoid reinforcing automation bias or uncritical acceptance of generated content (Shi et al.2026) Indeed, some frameworks have already been proposed for the adoption of LA in institutions that call for responsible considerations. For example, (Šimić et al. 2025) shows the importance of considering factors such as teaching, learning, and assessment, as well as data management and analytics, ethics, privacy, legal is- sues, and people competencies in the implementation of IA. Another framework, proposed byDrachsler & Greller(2016), developed a checklist, named DELI- CATE, with eight action points that institutional managers and decision-makers should consider to facilitate the trusted implementation of IA. Furthermore, Hernández-Leo et al.(2019) proposed Analytics Layers for Learning Design, which identifies different data sources relevant to informed, data-driven decision- making in educational design. However, these frameworks remain generic and do not identify or address the specific sources of potential bias in the data context and analysis that may affect their responsible interpretation. This paper addresses this gap by examining how to support stakeholders in interpreting IA responsibly, with particular attention to biasing factors that arise from the data context and analytical process. It proposes extending IA dashboards with mechanisms that surface existing knowledge about these factors and with a scaffolding layer that guides users towards a factor-aware, responsible interpretation. Specifically, the paper introduces a framework that integrates the currently fragmented knowledge on biasing factors in IA-FACTRIA (Framework of Fac- tors in the Analytical process and the data Context for Responsible Institutional Analytics)—and presents a conversational-agent–based system that leverages GenAI with FACTRIA-aware retrieval-augmented generation. Rather than of- fering interpretations or recommending decisions, the FACTRIA-aware chatbot prompts users to reflect on data quality, assumptions, comparability, and other potential biasing factors that may shape the interpretation of IA. We illustrate the framework’s applicability with four real IA cases at a higher education insti- tution and assess the system’s use with key stakeholders using transition network analysis (Saqr et al.2025). Throughout this paper, we use responsible interpretation of IA to refer to the process of drawing conclusions from visualizations while accounting for the Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 2989 factors that condition what those visualizations can legitimately support. We op- erationalize it through the FACTRIA framework (Section3), whose dimensions specify the factors to consider when interpreting a visualization. The remainder of this paper is organized as follows. Section2overviews the background of this work. Section3the FACTRIA Framework. Section4presents the methods. Section5presents the results. Section6discusses the results, Sec- tion7discusses the limitations, and Section8presents our conclusions. 2Background IA has become not only highly relevant but also increasingly useful in higher education, supporting evidence-informed decision-making around teaching qual- ity, student experience, and institutional effectiveness (Bichsel2012). However, prior studies show that effective use of IA depends on understanding the sen- sitivity of its indicators arising from both the analytical process and the data context (Radchenko2020,Bedggood & Donovan2012). For example, educa- tional research identifies student satisfaction metrics as key IA indicators for evaluating teaching quality, yet also cautions that these metrics must be inter- preted within context, as they may be influenced by demographic or contextual factors—such as gender (Flegl & Andrade Rosas2019,Marques et al.2024) or institutional policy conditions (e.g., those associated with COVID-19) (Clayson 2009,Ortiz-Beltrán et al.2022). Aspects of the learning design are also relevant when interpreting indicators of student outcomes (Rienties & Toetenel2016,Li et al.2016). In particular, visual dashboards can help teachers gain insight into their students’ knowledge, making these outcomes more interpretable and actionable (Calvo Morata et al. 2019). Additionally, research highlights that satisfaction with teaching materials, assessments, and workload correlates positively with the overall teaching expe- rience (Li et al.2016,Marques et al.2025). In IA studies focusing on retention, predictive models have shown that dropout risks vary by discipline—being higher in Engineering and Architecture—and are influenced by personal habits, insti- tutional experiences, and educational goals, necessitating stakeholder-inclusive agendas (Llauró et al.2021,De Silva et al.2022). Moreover, research indicates that LA dashboards often exceed users’ interpre- tive capabilities, in part because they do not explicitly articulate the analytical- process factors that users must consider for responsible interpretation (Lem et al. 2014,Pozdniakov et al.2025,Susnjak et al.2022). When factors remain implicit, users may overlook limitations, misjudge comparability, or draw conclusions that are insuﬀiciently grounded in the underlying evidence. Thus, effective IA tools require not only clear visualizations but also support that helps users attend to the analytical and contextual conditions required for responsible interpretation. The literature similarly emphasizes the value of evaluation-focused approaches that help interpretation lead to responsible ac- tion (e.g., the Analytics4Action Evaluation Framework, validated at scale) while keeping claims proportionate to evidence (Rienties et al.2016,2018). Consistent with accounts that expose modeling choices rather than treating algorithms as substitutes for judgment (Atkisson & Wiley2011,Weil et al.2023). Regarding AI-powered tools for improving visualization readability,Yan et al. (2024) developed VizChat, a prototype chatbot that generates AI-driven expla- nations for data visualizations. The system combines multimodal generative AI 2990 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support with retrieval-augmented generation to produce contextually relevant explana- tions and insights, as shown in a case study. Similarly,Yang et al.(2025) em- ployed multi-agent retrieval-augmented generation to generate explanations for learning analytics dashboards, tailoring them to user needs and enhancing teach- ers’ comprehension of classroom discourse analysis. In developing the FACTRIA framework, we drew not only on AI features but also on knowledge of the factors that hinder responsible interpretation of IA visualizations. Additionally, existing frameworks for responsible analytics address related concerns from different perspectives. Governance frameworks such as DELI- CATE (Drachsler & Greller2016) and SHEILA (Tsai et al.2018) operate pri- marily at the institutional level, specifying principles related to issues such as consent and transparency. Approaches to visualization literacy, such as CALVI (Ge et al.2023), focus on graphical features that may mislead readers. In con- trast, the FACTRIA framework specifies the factors that readers should consider when drawing conclusions from an institutional analytics (IA) visualization. Un- like these previous approaches, FACTRIA is operationalized at the moment of interpretation, providing the basis for our FACTRIA-aware chatbot. 3FACTRIA framework Figure 1: FACTRIA: A framework of Factors in the Analytical process and the data Context for Responsible IA. Building on prior literature, two researchers identified biasing factors that hinder the interpretation of IA and organized them into four FACTRIA cat- egories: analytics pipeline, institutional, course, and demographic factors (see below, AppendixIV, and Figure1). Existing studies emphasize not only the breadth of these factors but also the complexity of their interrelations. For example, course-level aggregations shape the indicators made available for analysis, while demographic distributions interact with these aggregations and with underlying model assumptions. Such interdependencies highlight the utility of a framework, such as FACTRIA, that Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 2991 foregrounds the multi-layered nature of biasing influences and supports their consideration in the design and interpretation of IA. Thus, aligned with the aim of this paper, the framework does not prescribe how decisions should be made. Instead, it provides a structured, evidence-informed scaffold that guides stakeholders in reflecting critically on the analytical and con- textual conditions that shape their interpretations. Its purpose is to help users surface implicit assumptions, question comparability, and recognize factors that may shape interpretation—rather than to offer definitive conclusions or recom- mendations. Proposing a framework, such as FACTRIA, contributes to the IA field in three key ways. First, it provides a mechanism for synthesizing and integrat- ing currently fragmented knowledge, offering a clearer and more comprehensive account of the factors influencing IA interpretations. Second, it supports the formulation of better-grounded hypotheses and more rigorous empirical investi- gations by making these factors explicit. Third, it enables the development of tools, for instance, dashboards that scaffold users’ interpretation processes by drawing attention to relevant contextual or analytical considerations. Finally, FACTRIA is a framework, i.e., it is intended as a flexible and ex- tensible theoretical construct. As IA practices evolve and new evidence emerges, additional biasing factors may be incorporated. The framework thus serves as an adaptable foundation for advancing responsible IA. 3.1Analytics Pipeline Factors The IA pipeline represents the sequence of processes used to apply analytics to address specific problems. It includes both methodological and technical stages of data analysis, where factors that affect interpretation can emerge at any point, underscoring the importance of understanding each stage. Data qualityCollected data often suffers from intrinsic biases, such as miss- ing values or inconsistencies common in educational contexts (Olteanu et al. 2019,Gaftandzhieva et al.2023). Additionally, the level of aggregation signif- icantly impacts results (Radchenko2020); while high-level summaries protect privacy, they may obscure granular nuances such as discrimination (Mar- ques et al.2024).Gaftandzhieva et al.(2023) highlights that higher educa- tion institutions need to consider the limitations when applying data-driven decision-making methods. Especially regarding data quality, data collected in the educational field can be inconsistent, incomplete, or unavailable. Types of data analysis and visualizationThis factor involves the selection of analytical techniques and visualizations, along with their underlying as- sumptions. For instance, selecting models that account for non-linear rela- tionships between learning design and outcomes is crucial for accuracy (Ortiz- Beltrán et al.2025). Regarding the assumptions underlying data analysis, Marques et al.(2024) illustrates that, in specific contexts (such as the analy- sis of gender bias), it is essential to clearly define what constitutes gender bias to ensure the analysis is meaningful and appropriately interpreted. De Silva et al. (2022) emphasizes the importance of considering context when employ- ing evidence-based approaches, as well as the need for humans to evaluate and interpret the analysis before implementing any intervention. 2992 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3.2Institutional Factors Institutional factors reflect the context of Higher Education Institutions, which are crucial for fair, informed, analytics-based decisions. Academic UnitsDifferences may be found by analyzing structural and demo- graphic characteristics of academic departments and centers. For instance, the Engineering center may exhibit different student outcomes than the Hu- manities center (Marques et al.2024). The department gender composition, which refers to the distribution of professors by gender within each depart- ment, can also have impacts that require a contextualized interpretation (Aragón et al.2023,Rosen2018). Educational FrameworkDifferences in institutional teaching practices, such as learning design guidelines, preferred teaching methodologies, assessment strategies, and constraints imposed by departmental or faculty norms, shape the context in which any institutional analysis is produced. As a result, understanding the educational framework in which the data originates is essential for making fair, contextualized interpretations of IA (Wise & Jung 2019). Modality of the UniversityModality influences teaching methods, assessment strategies, student engagement, and the way data is interpreted (Ortiz-Beltrán et al.2025,Usher et al.2021). Different modalities generate various data types: in a traditional university, data comes from classrooms, LMS, and tools, while in an online university, data is mainly from digital activities supported by LMS. IA interpretation should account for modality differ- ences by highlighting potential coverage gaps across courses or units based on the data sources. 3.3Course Factors Course level, Class sizeCharacteristics regarding the course, such as class size, course level (e.g, first year, second year), may influence students’ out- comes. For example,Radchenko(2020) found biases in student satisfaction scores using a large American dataset, specifically with respect to course fac- tors such as class size, which penalizes larger classes. In Marques et al.(2025), the authors show that factors beyond instructors’ control, such as class size and course level, influence satisfaction scores, but also identify cases that go beyond expectations and call for a balanced, responsible interpretation. Learning DesignPrevious studies have shown that learning design decisions significantly affect student satisfaction and performance (e.g., ( Thai et al. 2017) for the case of flipped classroom designs). The AL4LD framework also shows interactions between learning design factors and learning analytics Hernández-Leo et al.(2019). Students’ workloadResearch has shown that students’ perceived workload in a course correlates with other key course indicators, making it an important factor to consider—both for its potential biasing effects and for identifying cases that deviate from expected patterns—when interpreting IA (Marques et al.2025). This factor also interacts with other elements of the educational context, particularly aspects of the learning design (Laurillard2013). Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 2993 Student’s performanceExpected or achieved grades have been shown to in- fluence satisfaction ratings: students anticipating or receiving higher grades tend to evaluate instructors more positively (Badri et al.2006,Garrouste & Le Saout2020). Although research has shown that the overall effect size is small and counterexamples exist, performance remains a relevant factor to consider when interpreting IA indicators. 3.4Demographic Factors Demographic factors influence IA, as unintended biases may exist, affecting data interpretation and use. Incorporating demographic awareness helps institutions reduce biases, supporting fair decision-making. GenderGender may introduce potential unconscious biases in IA.Marques et al.(2024) highlights the importance of considering this factor and its interactions with factors such as age and academic context, taking into ac- count both the student’s gender and the professors’ gender. As with other factors, these influences are not deterministic but call for careful, responsible consideration. For example, while some studies report that female professors consistently receive lower evaluations, especially from male students (Mengel et al.2018,Boring2017), others find no significant gender-based differences in teaching quality or satisfaction when specific instructional criteria are applied (Suarman2014,Arrona-Palacios et al.2020). AgeInstructor age has been identified as a predictor of student evaluations, often correlating with perceived teaching effectiveness (Radmacher & Martin 2001). This factor also interacts with gender, with evidence showing that careful consideration should be given to analytics of female instructors who may be penalized more frequently across different age ranges compared to their male counterparts in some contexts (Marques et al.2024). LanguageLanguage variations and accents can shape how IA indicators are interpreted. For example, instructors with non-native accents may receive lower evaluations despite no measurable differences in student learning out- comes (Silaj et al.2024). Linguistic and cultural contexts may also trigger biases linked to nationality or expected grades, which can intersect with gender and cultural prejudices (Fan et al.2019,Al-Issa & Sulieman2007). 2994 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 4Methods This work is grounded in the FACTRIA framework introduced in Section3. We used the FACTRIA framework in two ways. First, participants read a summary of the framework before the interpretation task to become familiar with the factors relevant to responsible interpretation. Second, the framework shapes the chatbot implementation, where each FACTRIA factor is encoded in the chatbot as users examine the cases with support. The remainder of this section describes the experimental design in detail. 4.1Study Design The study used a qualitative think-aloud interview design (Leighton2017), record- ing participants as they completed a structured task and verbalized their thoughts. Participants received neutral instructions to speak naturally, with minimal facil- itator reminders. Each participant first completed the consent form, then reviewed the FAC- TRIA framework (AppendixIV) (5 minutes) before engaging in two sessions with a dashboard depicting two IA cases (Selected, from a total of four cases, for their relevance to participants’ stakeholder contexts. Further details provided in4.2). For a given case, each participant (1) first interpreted an IA visualization with- out the chatbot and then (2) interpreted the same visualization with FACTRIA- aware chatbot guidance (20 minutes). The order of conditions (with or without the FACTRIA-aware chatbot) was not counterbalanced, as our aim was to as- sess the added value of the framework and the FACTRIA-aware chatbot as an additional scaffold rather than to compare order effects. We conducted all ses- sions in a distraction-free environment and recorded participants’ comments and on-screen dashboard activity to support a comprehensive analysis. At the end, participants completed the System Usability Scale (SUS) questionnaire (Brooke et al.1996) (Figure2). We conducted a cross-case analysis of all participants and IA cases to address the research question:To what extent does a chatbot leveraging FACTRIA-aware retrieval-augmented generation support stakeholders in interpreting IA responsi- bly? Figure 2: Study Design 4.1.1Institutional Analytics Cases The activity presented four real, context-relevant IA visualizations. We selected the cases to differ in data structure, analytical focus, and FACTRIA interpretive Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 2995 requirements. The visualizations originated in previous institutional studies that the participants had not previously encountered. –Case 1 presented student satisfaction with teaching, aggregated by contex- tual factors (see Figure3). –Case 2 depicted the relationship among students’ academic performance (measured by grades), teaching methodologies, and perceived workload (see Figure4). –Case 3 illustrated a social comparison across three courses characterized by higher-than-usual levels of student interaction within an LMS platform (see Figure5). –Case 4 displayed patterns in LMS configuration practices across different teaching methodologies (see Figure6). Figure 3: Case 1: Distribution of student satisfaction scores with teaching, aggregated by professor gender, student gender, and faculty. Each dot represents a professor’s mean satisfaction score ( Marques et al.2024). 2996 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support Figure 4: Case 2: Distribution of the performance (Weighted Grades) by five of the most used Active Teaching Methodologies: Lectures (LECT), Project-based learning (PJBL), Problem-based learning (PRBL), Flipped classroom (FLIP), and Case studies (CASE). Courses are ranked by workload satisfaction, and the top 20% and bottom 20% are highlighted based on their ratings. ( Marques et al. n.d.) Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 2997 Figure 5: Case 3: Comparison of three similar high-interaction courses (A, B, C) against the institutional average on a common 0–1 scale covering 4 student outcomes (students’ performance and 3 satisfaction indicators), 7 LMS usage variables at course and Task—Activity or Resource (A/R)—level, 4 planned task types, 3 social designs, and 2 task modalities. ( Ortiz-Beltrán et al.n.d.) Figure 6: Case 4: Diversity of LMS course configurations using the Shannon Diversity Index (0–2.5), where higher values indicate a wider mix of activity types. Results are grouped by teaching methodology (LECT, PJBL, PRBL, FLIP, CASE) and compare courses with high versus low workload satisfaction (“Top Courses” vs. “Bottom Courses”). ( Marques et al.n.d.) 2998 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 4.2Participants The participants (4 male, 7 female) included professors, academic managers, and technicians from a Spanish university. TableIsummarizes these participants’ anonymized profiles. Table I: Participant Profiles. In bold are the participants and selected narratives. ParticipantRole and ExperienceUse of Institutional Data P1 (Case 3 & 4) LMS support specialist with 10+ years of experience. Frequently reviews LMS data. P2(Case 1 & Case 4) Education specialist in teaching innovation and training with 10+ years of support experience. Checks institutional data occasionally. P3 (Case 1 & 2) Teaching quality data support specialist with 20+ years of experience. Frequently reviews institutional data. P4 (Case 1 & 4) Education specialist in teaching innovation with 16+ years of support experience. Frequently reviews institutional data. P5 (Case 2 & 3) LMS and pedagogy support manager with 2+ years of experience. Frequently reviews institutional data. P6(Case 2 & Case 3) Academic leader in teaching quality and innovation, with 15+ years teaching and 4+ years management experience. Frequently reviews institutional data. P7(Case 1& Case 2) Academic coordinator with 25+ years teaching and 6+ years managing an undergraduate program. Periodically reviews institutional data related to teaching quality. P8 (Case 3 & 4) Lecturer with 15+ years of teaching experience. Checks teaching satisfaction data occasionally. P9 (Case 1 & 4) Lecturer with 10+ years of teaching experience. Checks teaching satisfaction data occasionally. P10(Case 1 & Case 2) Lecturer with 5+ years of teaching experience. Checks teaching satisfaction data occasionally. P11 (Case 1 & 3) Lecturer with 6+ years of teaching experience. Checks teaching satisfaction data occasionally. Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 2999 4.3FACTRIA-Aware Chatbot Assistant The FACTRIA-aware chatbot is a web-based, LLM-based system that facili- tates responsible interpretation of IA, grounded in FACTRIA through retrieval- augmented generation. The dashboard interface (Figure7) integrates visualiza- tion with the chatbot, while its interaction logic (Figure8) uses a deterministic state machine to connect users’ initial interpretations to the framework and emphasize relevant factors. The system then cycles through FACTRIA factors related to the visualization, providing prompts that guide reflection towards the visible data (for more details, see appendixA). Figure 7: User interface of the chatbot 4.4Data Collection and Analysis The experiments were documented using the IA Dashboard, capturing both screen activity and audio. Two researchers annotated the text data collected dur- ing participants’ interactions with the FACTRIA-aware chatbot. We developed a coding scheme using a bottom-up approach (see TableVI). Two researchers first coded the transcripts independently (Cohen’s k = 0.76, indicating substan- tial agreement). The two researchers resolved disagreements through discussion. We translated participant quotations into English where necessary and lightly edited hesitations and repetitions for readability without altering their substan- tive meaning. To determine how the FACTRIA framework and the chatbot assisted users in the responsible interpretation of IA, we used transition network analysis (TNA) (Saqr et al.2025) to examine interactions between users and the FACTRIA- aware chatbot, as TNA is a method for visualizing and understanding transition patterns using a probabilistic graph model.TNA was performed for each case ( 3000 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support Figure 8: Flowchart describing the conversational flow between user and the chatbot see appendixB) and across all cases (see Figure9) to visualize how chatbot-based assistance facilitated responsible interpretation of IA. In addition, to understand how different participants engage with the responsible interpretation of IA with the FACTRIA-aware chatbot support, we present four narratives that reflect variation across four IA cases and distinct participant profiles. 5Results Through interactions with the FACTRIA-aware chatbot, participants identified the relevant factors for respective cases (3 factors for Case 1, 5 for Case 2, 5 for Case 3, and 4 for Case 4), responding to the chatbot’s prompts and ultimately producing a final summary for their analyses. Responses to the post-activity questionnaire (see appendixV) indicated that most participants (10 out of 11) rated the chatbot highly (4 or 5 out of 5), for providing meaningful support for contextualizing the data.In the post-survey questionnaire, Participants also reported moderate confidence when using the system, although responses varied (“I felt very confident using the system”,3.7± 1.23). P2 described using the system as challenging and gave its usefulness for responsible interpretation a lower rating (2 out of 5). Another participant (P8) found the system useful but described the prompts astoo generic, expressing a preference for more direct guidance. Participants with prior experience using chatbots for similar tasks (e.g., P1, P3, P9) reported positive interactions, a pattern that is consistent with the con- fidence scores. However, the chatbot’s ability to provide on-demand definitions Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3001 of key concepts needed to read the visualizations was especially important for less-experienced users. P6’s observation that, without immediate explanations for terms such as “weighted grade” or “Shannon index,” they would have “lost five minutes just figuring out what to do” illustrates how on-demand definitions reduced the initial interpretive effort required to use the visualization. This is also consistent with the post-survey results, which suggest that participants gen- erally did not feel they required substantial technical support to use the system (“I think that I would need the support of a technical person to be able to use this system”,2.27±1.45), even though on-demand explanations remained im- portant for understanding domain-specific concepts. The overall SUS score was 70.84±16.39, which is above the conventional threshold for good usability. Figure9shows the transition network used to examine the overall struc- ture of chatbot–user interactions across all cases. For readability, the figure in- cludes only transitions with relative probability above 0.25. The most prominent chatbot-coded nodes wereRI.Reflection(appeared 46 times),RI.Interpretation (20),PI.Explanation(26), andA.Validation(40), while the most prominent user- coded node wasPI.SharingInsight(84) (see appendixVIfor further information about the coding scheme). This distribution shows that assistant prompts and explanatory moves structured the interaction, while participants subsequently generated interpretations. Notably, the recurrence ofPI.SharingInsight, includ- ing its self-loop, shows that users often remained in an interpretive mode once they began articulating their own reasoning. Figure 9: Transition Network Analysis considering all cases. Node size reflects the prominence of coded actions in the overall network, and edge labels represent relative transition probabilities between coded states. 3002 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support Additionally, TableIIpresents descriptive statistics for the number of ad- dressed sub-factors (without and with the FACTRIA-aware chatbot) under three scenarios. The first scenario (Case X + Case Y) includes all collected data across the study, treating each case addressed by a participant as an independent ob- servation(n= 22). We computed the paired-samples t-test, which showed a sta- tistically significant difference between without and with the FACTRIA-aware chatbot,p<.001. andCohensd= 2.1530, indicating a very large effect size. The second scenario (1° Case) restricts the analysis to the first case each par- ticipant was assigned(n= 11), to control potential learning effects that could arise from participants completing a second case. In this second scenario, we also found statistically significant differences, withp<.001andCohensd= 4.4947, indicating a very large effect size. In the third scenario, we considered only the data from the second case assigned to participants, also with statistically signif- icant differences withn= 11,p<.001. andCohensd= 1.7866. In all three scenarios observed in TableII, the use of the FACTRIA-aware chatbot increased the average number of sub-factors considered in each case. In both cases, time constraints prevented participants from addressing all sub-factors related to the visualization using the FACTRIA-aware chatbot. Table I: Descriptive statistics for the number of sub-factors addressed without and with the FACTRIA-aware chatbot, computed across all cases (Case X + Case Y), using only participants’ first assigned case (1 o Cases) and using only participants’ second assigned case (2 o Cases). Case X + Case Y 1 o Cases 2 o Cases Without With Without With Without With Average1.413.451.73 3.55 1.09 3.36 Standard deviation 0.800.860.65 0.82 0.83 0.92 Variance0.630.740.42 0.67 0.69 0.85 Median1.504.002.00 4.00 1.00 4.00 Finally, we present exemplary narratives covering all four IA cases and high- lighting different participant profiles (see bold in TableI). These narratives help illustrate how each individual makes sense of their experiences. 5.1Case 1 5.1.1P7 – Illustrative example The participant’s first interpretation focused almost entirely on the clearly visible gender differences in the visualization across academic degrees. The explanation centered on the pattern that female professors appeared to receive lower evalua- tions than male professors. They highlighted greater discrepancies in Translation and Economics, and smaller ones in the Humanities, Law, and Engineering fac- ulties. Much of this reasoning relied directly on the quartile distributions shown in the visualization, especially the lower quartiles: Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3003 “In most degrees female professors receive worse scores than male professors... the differences are more evident in the lower quartiles.” (PI.SharingInsight) At this stage, the participant interpreted the visualization as a relatively direct portrayal of gender bias, without yet invoking data quality issues, ag- gregation concerns, response rates, or institutional differences across academic units. By interacting with the FACTRIA-aware chatbot, the participant began ac- knowledging analytics pipeline factors—especially data quality—and, when prompted about small sample sizes, reconsidered the limitations of the aggregated indica- tors. As the FACTRIA-aware chatbot introduced demographic factors, the partic- ipant recognized the need to consider student–professor gender interactions but remained cautious in the interpretation, recognizing the need for further inquiry: “I would be cautious without knowing the sample sizes... and how the students’ gender interacts with the professors’ gender.” (PI.Caution) Although the participant did not initially reference other framework com- ponents, such as institutional differences (e.g., departmental composition), the subsequent chatbot’s prompts helped them develop a more conditional, context- aware interpretation. This shift—from straightforward readings to considering how multiple factors interact—marks a move from descriptive analysis towards responsibility-oriented inquiry. The conversation also highlighted analytics pipeline considerations, including variability in sample size and the representativeness of aggregated statistics, alongside the demographic dynamics underlying student evaluations. 5.2Case 2 5.2.1P10 – Illustrative example At the start of the session, the participant focused on the chart’s most visually salient elements. Their reasoning emphasized apparent group differences with- out yet considering how those differences were produced. The participant initially treated the box-plot position as direct evidence of course quality: “It looks like the bottom group is performing worse, because their box is lower... I guess those courses are simply weaker.” At this stage, the participant treated the visual pattern as evidence of weaker courses, without considering analytics pipeline factors such as aggregation or course factors such as class size and workload. After the FACTRIA-aware chat- bot clarified that the visualization showed course-level aggregates rather than individual student data, the participant replaced the individual-level assumption with a class-size hypothesis: “Ah, so this isn’t each student... It’s a summary of the whole course? Then maybe the variation has to do with how many students are in each class.” 3004 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support This marked one of the first moments in which the participant integrated analytics pipeline factors—specifically, aggregation and data transformation— into their interpretation. When the FACTRIA-aware chatbot prompted the participant to consider course-related factors—especially workload, learning design, and class size—the participant shifted again and recognized that course structure could influence satisfaction and performance indicators: “If a course has a much heavier workload or a big group of students, that could affect the scores. I didn’t think about that when I first looked at it.” This response incorporated course factors—workload and class size—as alterna- tive explanations for the observed differences. Compared with the initial con- clusion that the courses were “simply weaker,” the participant’s interpretation became more conditional and context-aware. 5.3Case 3 5.3.1P6 – Illustrative example The participant’s first reading of the visualization focused on comparing courses A, B, and C with the institutional average. They noted which indicators ap- peared “high” or “low” relative to the black reference line and used this to infer strengths and areas for improvement. Their interpretation emphasized visible differences in methodology, subject–course interaction, and LMS-related indica- tors. For example, the participant described Course B as showing “lower values in methodology” relative to the reference line(PI.SharingInsight) The participant approached the visualization descriptively, using the ob- served differences to identify possible areas for course improvement. Although they expressed uncertainty about labels such as “Diversity index” and “social group,” their initial interpretation did not examine analytics pipeline factors related to indicator definition and data quality. The first shift occurred when the FACTRIA-aware chatbot asked how unclear indicator definitions might influence interpretation. The participant then recog- nized that unclear definitions—a data quality issue within the analytics pipeline— limited the conclusions they could draw: “I’m not fully sure about the meaning or the potential impact of this indica- tor.” (Express Doubts) This response shows how an analytics pipeline concern qualified the participant’s initial comparison: interpreting the indicator values required clearer definitions of how the underlying analysis defined the measures. As the FACTRIA-aware chatbot shifted to course factors, specifically learning design and workload, the participant reconsidered the interpretation regarding the meaning of the high LMS interaction levels. The participant connected in- teraction patterns to workload, recognizing that course design choices shape the indicators: Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3005 “Course A seems to have the highest interaction, and it also has the highest value of workload.” (PI.SharingInsight) This progression connected the LMS interaction indicators to course factors, particularly learning design and workload, rather than treating them as direct measures of course quality. Institutional and demographic factors did not enter the participant’s reasoning, showing the boundaries of the interpretive shift. 5.4Case 4 5.4.1P2 – Illustrative example At the outset, the participant struggled to identify the purpose of the visualiza- tion, noting confusion both about the graphic and the meaning of its labels. The participant interpreted the chart as comparing satisfaction levels across teaching methodologies, rather than as showing Shannon diversity of LMS task configu- ration for courses with high and low workload satisfaction. This misalignment shaped their early reasoning, as they attempted to connect each methodology with its expected level of platform usage: “I’m not clear about the objective of the graph... I thought it was comparing how much each methodology uses the platform.” Based on their pedagogical understanding, the participant expected Flipped Classroom courses to use more platform resources than lectures and therefore regarded the lower diversity values as inconsistent with that expectation. The participant also questioned the inclusion of “lectures” as a methodology and showed uncertainty about methodology labels, the Shannon Index scale, and the meaning of the top/bottom course groups. Overall, the participant grounded the interpretation in visible patterns, but an incorrect premise and unclear indica- tor definitions constrained it, without yet considering analytics pipeline factors related to metric definition and data representation. As the FACTRIA-aware chatbot introduced factor-based prompts, the par- ticipant gradually shifted from a satisfaction-based interpretation towards the in- tended focus: LMS task diversity and its connection to learning design structures. The first shift occurred when the participant explicitly requested clarification of the Shannon Index(RI.AskingForClarification). The chatbot’s explanation pro- vided the definitional clarity needed to reconsider the visualization. This signaled a move towards analytics pipeline factors, particularly data defi- nitions and metric assumptions. As the FACTRIA-aware chatbot elaborated on data quality and aggregation, the participant understood that the equal weight- ing of all resources and activities could affect how the index represents diversity: “If you apply a more complex methodology, you will need to configure more tasks in the LMS.” (PI.SharingInsight) The participant increasingly acknowledged that the portrayal of the data, rather than the index itself, was influencing their interpretation. This response con- nected their reasoning to analytics pipeline factors concerning data representa- tion and types of analysis. 3006 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support When the FACTRIA-aware chatbot shifted to course factors, the participant moved towards aligning the chart with their pedagogical expertise. They high- lighted how task variety in Flipped Classroom should reflect multiple channels and formats to support preparation before class: “Tasks should be as complete as possible, across different channels, with dif- ferent application possibilities for students.” (PI.SharingInsight) The participant then connected two FACTRIA course factors, learning design and student workload, noting that methodologies like Flipped Classroom inher- ently require careful consideration of student workload in their design: “Methodologies such as FLIP have as a key factor the consideration of the workload in their design.” (Reflection) Despite this progress, institutional factors did not emerge in this exchange, de- limiting the scope of the participant’s revised interpretation. The participant’s initial reading, shaped by uncertainty about the graph’s objective, treated the methodology bars as indicators of satisfaction rather than as indicators of task di- versity. Through the chatbot’s guidance, the participant engaged with analytics pipeline factors (data definitions, data quality, aggregation limits) and course factors (learning design structures, workload implications, task completeness), progressively realigning their responsible interpretation with the visualization’s scope. 6Discussion The study findings show that stakeholders often begin their interpretation of IA with narrow observations that overlook key analytical and contextual factors. Before interacting with the FACTRIA-aware chatbot, participants referenced an average of1.50sub-factors (SD= 0.80) considering both cases, even though each case required considering multiple dimensions, from data quality and aggre- gation to course design and demographics. Since all participants had read the framework at the start of the experiment, familiarity alone was evidently not suﬀicient to prompt broad consideration in their initial interpretations. We eval- uated the effect of FACTRIA-aware chatbot scaffolding under three scenarios (both cases combined, first cases only, and second cases only) and in each the chatbot surfaced sub-factors participants had not considered on their own. This confirms concerns in the literature about the need for approaches that strengthen users’ awareness of potential biases embedded in both the analytical context and the interpretive process (Williamson & Kizilcec2022,Olteanu et al.2019). Another finding relates to how the FACTRIA-aware chatbot, design—particularly its emphasis on retrieval-augmented, framework-bound prompting—enabled par- ticipants to integrate consideration of biasing factors into their interpretations. Participants who initially relied heavily on visual cues (e.g., quartile differences in Case 1 or box-plot positions in Case 2) began, through chatbot prompts, to re-examine assumptions related to sample size, aggregation level, indicator definitions, learning-design structures, or demographic interactions. Across all participants and the four IA cases, the FACTRIA-aware chatbot supported a clear shift towards more reflective, context-aware, conditional interpretations. Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3007 The qualitative data coding and transition network analysis demonstrated that the FACTRIA-aware chatbot guided the user towards a responsible interpreta- tion by asking questions that encourage reasoning (RI.Interpretation (20 times), RI.Factors (16 times), RI.Reflection (46 times), RI.Explanation (26 times)) in- stead of providing answers. This finding aligns with previous study findings showing that theory-informed structures and timely, context-aware guidance can improve users’ ability to interpret complex analytics (Paulsen & Lindsay2024, Han et al.2021). This finding also complements work on AI-mediated assistance for educational dashboards (Yan et al.2024,Yang et al.2025), while extending that work to the institutional-level analytics context and emphasizing critical reflection rather than explanation alone. In terms of implications for the implementation and use of IA systems, this study highlights the value of guiding stakeholders through relevant interpre- tive factors when working with IA. The findings suggest that frameworks such as FACTRIA can help connect dashboard indicators with the broader contex- tual considerations needed for sound institutional decision-making. For practice, the results point to the usefulness of embedding factor-aware guidance directly within analytics tools. Providing contextual cues at the point of interpreta- tion can help prevent premature conclusions and promote more transparent, evidence-aligned decisions. Integrating such support into dashboards or training can strengthen institutional capacity for responsible analytics use. 7Limitations Two aspects of our study design should be considered when interpreting the findings. First, the sample included 11 participants, and each engaged with two of the four cases. While the participants represented genuine user profiles for the case under study, the experimental conditions resulted in a limited number of observations per case. Thus, essentially we treated the study as exploratory and emphasize case-level results rather than comparative. Our study’s value lies in providing a first computer-supported approach to support scaffolding in the responsible interpretation in institutional analytics. A second limitation is that the previous knowledge about the framework was not controlled: all participants first completed the task having read the framework but without chatbot support and then completed it with FACTRIA- aware chatbot assistance. This design reflects the study’s focus on the role of the chatbot in guiding participants’ reflection, rather than on their prior knowledge of the factors included in the FACTRIA framework. We therefore examined how participants’ reflection evolved across their interactions with the chatbot in two subsequent cases, while acknowledging that their familiarity with FACTRIA was also expected to increase by the second case. Although this aspect of the experimental design was considered in the analysis and interpretation of the results, future work should examine alternative experimental conditions that allow the effects of chatbot support to be more clearly distinguished from those of increasing familiarity with the framework. The interpretation of the results should also account for possible automa- tion bias, over-reliance of users on chatbot guidance, and known limitations of the think-aloud methodology, including reactivity effects and the diﬀiculty of inferring genuine understanding from verbalized reasoning. Verbalizing interpre- tations may influence how participants approach the task, while differences in 3008 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support the detail of their responses may reflect their ability or willingness to articulate their reasoning rather than their underlying understanding. We therefore treat these data as evidence of participants’ expressed sense-making rather than as a direct measure of comprehension. Nevertheless, the think-aloud methodology strengthened the exploratory study by providing rich qualitative insights. Given these limitations, we present this work as a proof of concept. FACTRIA remains a framework that may require adaptation across cases and institutional contexts. Although the sample was relatively small, the study captures a range of authentic perspectives from a diverse set of stakeholders spanning from teachers, managers, and technicians who are directly involved in institutional data interpretation, decision-making, and taking subsequent actions at the wider institution level. Within a single study, reaching such a diverse range of stakeholders is challeng- ing due to practical constraints, particularly those associated with participants’ schedules and availability. Nevertheless, our study successfully generated empir- ical findings across these stakeholder groups, providing an important starting point for the field of institutional analytics to consider how LLMs can support the responsible interpretation of IA. Future research should expand and refine FACTRIA by testing its adapt- ability across a wider range of IA cases and examining how additional sources of bias can be incorporated as institutional analytics practices and technolo- gies evolve. Researchers should also investigate the GenAI component, given the known risks of oversimplification, occasional inaccuracies, and inconsistent alignment with institutional contexts (Martínez-Maldonado2020,Holmes et al. 2022). Applying the FACTRIA approach across a broader range of universities would further strengthen the generalizability of the findings. The findings from our study also suggest more specific directions for future research. Differences among participants indicate a need to examine how users internalize this type of scaffolding over time and whether adaptive—for example, experience-sensitive— or role-specific prompting enhances its impact. Further research could also test how different forms of explanation affect users’ sense-making when working with complex IA. 8Conclusion This study provides an initial but meaningful response to a key gap in IA: the lack of structured support to help stakeholders consider the factors that shape respon- sible IA interpretation. The study offers an original contribution by synthesizing biasing factors in the FACTRIA framework that adds the needed specificity to discussions of responsible interpretation—and showing how a GenAI chatbot can operationalize this framework to scaffold users’ reflective sense-making. To assess the framework’s applicability, we employed a qualitative research design involving stakeholders responsible for institutional decision-making, authentic interpretive cases, and transition network analysis. Looking ahead, further re- search is warranted—both to address the limitations of the present study and to investigate the new directions it opens. Collectively, this work provides a conceptual and empirical foundation for developing more transparent, fair, and contextually sound interpretive practices in IA. Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3009 Acknowledgments The work was done in collaboration with UPF services (Strategy, Data and Quality Service, and the Data Protection Oﬀicer) through the UPF Learning Lab initiative. Funding This work has been supported by PID2023-146692OB-C33 and 2023-26 CEX2021- 001195-M granted by MICIU/AEI/10.13039/501100011033. D. Hernández-Leo (Serra Húnter) also acknowledges the support of the AGAUR ICREA Academia programme. 3010 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support References Al-Issa, A. & Sulieman, H. (2007), ‘Student evaluations of teaching: perceptions and biasing factors’,QualityAssuranceinEducation15(3), 302–317. Aragón, O. R., Pietri, E. S. & Powell, B. A. (2023), ‘Gender bias in teaching evaluations: the causal role of department gender composition’,Proceedings oftheNationalAcademyofSciences120(4). Arrona-Palacios, A., Okoye, K., Camacho-Zuñiga, C., Hammout, N., Luttmann- Nakamura, E., Hosseini, S. & Escamilla, J. (2020), ‘Does professors’ gender impact how students evaluate their teaching and the recommendations for the best professor?’,Heliyon6(10), e05313. Atkisson, M. & Wiley, D. A. (2011), Learning analytics as interpretive practice: Applying westerman to educational intervention,in ‘Proceedings of the 1st International Conference on Learning Analytics & Knowledge (LAK)’, p. 22– 23. Badri, M. A., Abdulla, M., Kamali, M. A. & Dodeen, H. (2006), ‘Identifying potential biasing variables in student evaluation of teaching in a newly ac- credited business program in the uae’,InternationalJournalofEducational Management20(1), 43–59. Bedggood, R. E. & Donovan, J. D. (2012), ‘University performance evaluations: what are we really measuring?’,StudiesinHigherEducation37(7), 825–842. Bichsel, J. (2012), ‘Analytics in higher education benefits, barriers, progress, and recommendations (research report)’,Louisville,CO:EDUCAUSECenterfor AppliedResearch . URL:https://library.educause.edu/ /media/files/library/2012/6/ers1207.pdf Boring, A. (2017), ‘Gender biases in student evaluations of teaching’,Journalof publiceconomics145, 27–41. Brooke, J. et al. (1996), ‘Sus-a quick and dirty usability scale’,Usability evaluationinindustry189(194), 4–7. Calvo Morata, A., Alonso Fernández, C., Pérez Colado, I. J., Freire Morán, M., Martínez Ortiz, I. & Fernández Manjón, B. (2019), ‘Improving teacher game learning analytics dashboards through ad-hoc development’. Clayson, D. E. (2009), ‘Student evaluations of teaching: Are they related to what students learn?: A meta-analysis and review of the literature’, Journal ofMarketingEducation31(1), 16–30. De Silva, L. M. H., Chounta, I.-A., Rodríguez-Triana, M. J., Roa, E. R., Gram- berg, A. & Valk, A. (2022), ‘Toward an institutional analytics agenda for addressing student dropout in higher education: An academic stakeholders’ perspective.’,JournalofLearningAnalytics9(2), 179–201. Drachsler, H. & Greller, W. (2016), Privacy and analytics: it’s a delicate issue a checklist for trusted learning analytics,in ‘Proceedings of the sixth interna- tional conference on learning analytics & knowledge’, p. 89–98. Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3011 Fan, Y., Shepherd, L. J., Slavich, E., Waters, D., Stone, M., Abel, R. & Johnston, E. L. (2019), ‘Gender and cultural bias in student evaluations: Why represen- tation matters’,PloSone14(2), e0209749. Flegl, M. & Andrade Rosas, L. A. (2019), ‘Do professor’s age and gender matter or do students give higher value to professors’ experience?’,QualityAssurance inEducation27(4), 511–532. Gaftandzhieva, S., Hussain, S., Hilcenko, S., Doneva, R. & Boykova, K. (2023), ‘Data-driven decision making in higher education institutions: State-of-play’, InternationalJournalofAdvancedComputerScienceandApplications14(6). Garrouste, M. & Le Saout, R. (2020), ‘Good teaching and good grades. can you buy pedagogy?’,AnnalsofEconomicsandStatistics139, 29–60. Ge, L. W., Cui, Y. & Kay, M. (2023), Calvi: Critical thinking assessment for lit- eracy in visualizations, in ‘Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems’, CHI ’23, Association for Computing Machin- ery, New York, NY, USA. URL:https://doi.org/10.1145/3544548.3581406 Han, J., Chen, F. & Kitto, K. (2021), ‘Learning analytics dashboards for adaptive support in face-to-face collaborative argumentation’,Computers&Education 163, 104099. Heffernan, T. (2023), ‘Abusive comments in student evaluations of courses and teaching: the attacks women and marginalised academics endure’,Higher Education85(1), 225–239. Hernández-Leo, D., Martinez-Maldonado, R., Pardo, A., Muñoz-Cristóbal, J. A. & Rodríguez-Triana, M. J. (2019), ‘Analytics for learning design: A layered framework and tools’,BritishJournalofEducationalTechnology50(1), 139– 152. Holmes, W., Bialik, M. & Fadel, C. (2022), ‘Ethics of ai in education: Towards a community-wide framework’,BritishJournalofEducationalTechnology 53(2), 550–566. Jones, K. M. L. (2019), ‘Learning analytics and higher education: A proposed model for adoption’,InternationalJournalofEducationalTechnologyin HigherEducation16(1), 1–16. Laurillard, D. (2013),Teachingasadesignscience:Buildingpedagogicalpatterns forlearningandtechnology, Routledge. Leighton, J. P. (2017),Usingthink-aloudinterviewsandcognitivelabsin educationalresearch, Oxford University Press. Lem, S., Onghena, P., Verschaffel, L. & Van Dooren, W. (2014), ‘Experts’ misin- terpretation of box plots – a dual processing approach’,PsychologicaBelgica 54(4), 395–405. Li, N., Marsh, V. & Rienties, B. (2016), ‘Modelling and managing learner satis- faction: Use of learner feedback to enhance blended and online learning expe- rience’,DecisionSciencesJournalofInnovativeEducation14(2), 216–242. 3012 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support Llauró, A., Fonseca, D., Villegas, E., Aláez, M. & Romero, S. (2021), Educa- tional data mining application for improving the academic tutorial sessions, and the reduction of early dropout in undergraduate students,in ‘Ninth In- ternational Conference on Technological Ecosystems for Enhancing Multicul- turality (TEEM’21)’, TEEM’21, Association for Computing Machinery, New York, NY, USA, p. 212–218. Mangaroska, K. & Giannakos, M. (2019), ‘Learning analytics for learning design: A systematic literature review of analytics-driven design to enhance learning’, IEEETrans.Learn.Technol.12(4), 516–534. Marques, F., Hernández-Leo, D. & Castillo, C. (2025), ‘Beyond bias in student satisfaction surveys: exploring the role of grades and satisfaction with the learn- ing design’,JournalofNewApproachesinEducationalResearch14(1), 1–13. Marques, F., Hernández-Leo, D. & Castillo, C. (2024), ‘Measuring gender bias in student satisfaction in higher education: a cross-department study’,Cogent Education11(1), 2375183. Marques, F., Ortiz-Beltrán, A. & Hernandez-Leo, D. (n.d.), Behind the extremes: Leveraging institutional analytics to bridge learning design and student out- comes. Manuscript under review. Martínez-Maldonado, R. (2020), Human-centred learning analytics: Four chal- lenges in dashboard design, in ‘Proceedings of the 10th International Confer- ence on Learning Analytics & Knowledge (LAK ’20)’, p. 100–105. Mengel, F., Sauermann, J. & Zölitz, U. (2018), ‘Gender Bias in Teaching Evalu- ations’,JournaloftheEuropeanEconomicAssociation17(2), 535–566. Olteanu, A., Castillo, C., Diaz, F. & Kıcıman, E. (2019), ‘Social data: Biases, methodological pitfalls, and ethical boundaries’, Frontiersinbigdata2, 13. Ortiz-Beltrán, A., Hernández-Leo, D. & Amarasinghe, I. (2022), ‘Surviving and thriving: How changes in teaching modalities influenced student satisfac- tion before, during and after covid-19’,AustralasianJournalofEducational Technology p. 72–88. Ortiz-Beltrán, A., Marques, F. & Hernández-Leo, D. (2025), Exploring the complex analytics interplay of lms design, usage, academic outcomes, and perceived workload: A case study,in K. Tammets, S. Sosnovsky, R. Ferreira Mello, G. Pishtari & T. Nazaretsky, eds, ‘Two Decades of TEL. From Lessons Learnt to Challenges Ahead’, Springer Nature Switzerland, Cham, p. 212–217. Ortiz-Beltrán, A., Marques, F. & Hernandez-Leo, D. (n.d.), Multidimensional institutional analytics (mia): Turning institutional data into design-oriented hypotheses. Manuscript under review. Paulsen, C. & Lindsay, J. (2024), ‘Learning analytics dashboards are increas- ingly becoming about learning and not just analytics: A systematic review’, EducationandInformationTechnologies . Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3013 Paz, F. J. & Cazella, S. C. (2019), ‘Academic analytics: a systematic review of literature’,InternationalJournalofDevelopmentResearch9(11), 31710– 31716. Pozdniakov, S., Martínez-Maldonado, R., Tsai, Y.-S., Echeverría, V., Swiecki, Z. & Gašević, D. (2025), ‘Investigating the effect of visualization literacy and guidance on teachers’ dashboard interpretation’,JournalofLearningAnalytics 12(1), 367–390. Radchenko, N. (2020), ‘Biases in student evaluations of teaching: An american case study’,IZADiscussionPaper . Radmacher, S. A. & Martin, D. J. (2001), ‘Identifying significant predictors of student evaluations of faculty through hierarchical regression analysis’,The JournalofPsychology135(3), 259–268. Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K. & Murphy, S. (2016), Analytics4action evaluation framework: A review of evidence-based learning analytics interventions at the open university uk, Technical report, The Open University. Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K. & Murphy, S. (2018), Analytics4action evaluation framework—a review of evidence-based learning analytics interventions at the open university uk, Technical report, The Open University. Rienties, B. & Toetenel, L. (2016), ‘The impact of learning design on student be- haviour, satisfaction and performance: A cross-institutional comparison across 151 modules’,ComputersinHumanBehavior60, 333–341. Rosen, A. S. (2018), ‘Correlations, trends and potential biases among publicly accessible web-based student evaluations of teaching: a large-scale study of ratemyprofessors. com data’,Assessment&EvaluationinHigherEducation 43(1), 31–44. Saqr, M., López-Pernas, S., Törmänen, T., Kaliisa, R., Misiejuk, K. & Tikka, S. (2025), Transition network analysis: A novel framework for modeling, visualiz- ing, and identifying the temporal patterns of learners and learning processes, in ‘Proceedings of the 15th international learning analytics and knowledge conference’, p. 351–361. Shi, Y., Yu, K., Dong, Y. & Chen, F. (2026), ‘Large language models in educa- tion: A systematic review of empirical applications, benefits, and challenges’, ComputersandEducation:ArtificialIntelligence10, 100529. Silaj, K. M., Frangiyyeh, A. & Paquette-Smith, M. (2024), ‘The impact of mul- timedia design and the accent of the instructor on student learning and eval- uations of teaching’,AppliedCognitivePsychology38(1), e4143. Šimić, D., Begičević Ređep, N., Rako, S., Kadoić, N., Van Petegem, W., Rien- ties, B., Cabrera Lanzo, N., Eichhorn, M., Guàrdia, L., Kučina Softić, S. & Tillmann, A. (2025), ‘HELA-CMM: capability maturity model for adoption of learning analytics in higher education’,Internationaljournalofeducational technologyinhighereducation22(1), 25. 3014 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support Suarman (2014), ‘Gender differences on students satisfaction: The role of teach- ing quality in higher education’,Middle-EastJournalofScientificResearch 21(9), 1434–1441. Susnjak, T., Ramaswami, G. S. & Mathrani, A. (2022), ‘Learning analytics dashboard: A tool for providing actionable insights to learners’,International JournalofEducationalTechnologyinHigherEducation19, 12. Thai, N. T. T., De Wever, B. & Valcke, M. (2017), ‘The impact of a flipped classroom design on learning performance in higher education: Looking for the best “blend” of lectures and guiding questions with feedback’,Computers &Education107, 113–126. Tsai, Y.-S., Moreno-Marcos, P. M., Tammets, K., Kollom, K. & Gašević, D. (2018), Sheila policy framework: informing institutional strategies and pol- icy processes of learning analytics,in ‘Proceedings of the 8th International Conference on Learning Analytics and Knowledge’, LAK ’18, Association for Computing Machinery, New York, NY, USA, p. 320–329. URL:https://doi.org/10.1145/3170358.3170367 Usher, M., Hershkovitz, A. & Forkosh‐Baruch, A. (2021), ‘From data to actions: Instructors’ decision making based on learners’ data in online emergency re- mote teaching’,52(4), 1338–1356. Weil, D., Kendall, C. & Snyder, R. (2023), ‘A modern framework for institutional analytics’, EDUCAUSEReview(Online) . Williamson, K. & Kizilcec, R. (2022), A review of learning analytics dashboard research in higher education: Implications for justice, equity, diversity, and inclusion,in ‘LAK22: 12th International Learning Analytics and Knowledge Conference’, LAK22, Association for Computing Machinery, New York, NY, USA, p. 260–270. Wise, A. F. & Jung, Y. (2019), ‘Teaching with analytics: Towards a situated model of instructor decision-making’,JournalofLearningAnalytics6(2), 53– 69. Yan, L., Zhao, L., Echeverria, V., Jin, Y., Alfredo, R., Li, X., Gaševi’c, D. & Martinez-Maldonado, R. (2024), Vizchat: enhancing learning analytics dashboards with contextualised explanations using multimodal generative ai chatbots,in ‘International conference on artificial intelligence in education’, Springer, p. 180–193. Yang, C., Wang, D. & Chen, G. (2025), Chat-lad: Enhancing teacher under- standing of learning analytics dashboard with ai-empowered explanations, in ‘Proceedings of the Twelfth ACM Conference on Learning @ Scale’, L@S ’25, Association for Computing Machinery, New York, NY, USA, p. 197–201. Yanosky, R. & Arroway, P. (2015), ‘The analytics landscape in higher education’, Louisville,CO:EducaseCenterforAnalysisandResearch . Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3015 Table I: Summary of studies that influenced the development of FACTRIA framework Reference Description Radchenko ( 2020 ) Described the aspects that may influence student satisfaction and the effects that result in biases. The author argues that various factors,such as gender and course level, influence data collection on student satisfaction. Marques et al. ( 2024 ) Described the importance of considering reliability to analyze the data collection. Also investigated gender biases by studying differentfactors such as age, level of the course, and academic centers. Clayson ( 2009 ) Examined previous research on the connection between learning and student satisfaction. The findings indicated that, on average, there isonly a weak relationship between the two variables, depending on factors such as context, discipline, course type, and instructor. Ortiz-Beltrán et al. ( 2022 ) Evaluated how changes in teaching modalities influenced student satisfaction scores before, during, and after COVID-19. Rosen ( 2018 ) Investigated gender and the impact of teaching evaluations across various academic departments and disciplines. Aragón et al. ( 2023 ) Investigated the impact of department gender composition on student satisfaction scores. Rienties & Toetenel ( 2016 ) Studied the impact of learning design to understand student behavior and performance. Li et al. ( 2016 ) Identified that learning design has a significant influence on the overall perception of a course. Mangaroska & Giannakos ( 2019 ) In a literature review, it was identified that connecting learning design with learning analytics is important for aligning both concepts togather insights and improve decisions in learning design. Hernández-Leo et al. ( 2019 ) Proposed a layered framework that considers analytics layers for learning design. Marques et al. ( 2025 ) Studied the interplay between student satisfaction measurements, workload, and performance. Heffernan ( 2023 ) Demonstrated the influence of the gender of the professor on student satisfaction. Flegl & Andrade Rosas ( 2019 ) Analyzed the influence of age on student satisfaction. Al-Issa & Sulieman ( 2007 ) Identified that cultural and demographic factors potentially influence student satisfaction. Fan et al. ( 2019 ) Demonstrated that gender and cultural biases are present in student satisfaction surveys, finding biases against women and non-nativeEnglish speakers. 3016 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support Table IV: Institutional Analytics Framework Factor Description of the factor Sub-factor Description of the sub-factor Examples AnalyticsPipeline Factors Process related to theinstitutional analyticsapproach applied in adetermined problem. Data quality Processes related to gathering, preparing,and structuring data for analysis (pre-processing). Data collection, feature extraction, sample,data reliability, level of data aggregation. Types of data analysis Approaches and assumptions used to interpret and extract insights from data. Selection of analysis techniques, visualization, analytical context, assumptions of the data analyses. InstitutionalFactors Factors related to theinstitutional level in HigherEducation Institutions. Academic centers Structural and demographic characteristicsof academic departments and centers. Academic centers (e.g., Economics,Engineering, Humanities); departmentgender composition (distribution ofprofessors by gender in each department). Educational framework Guidelines for teachers in higher education. Teaching methodologies framework. Modality of theuniversity Refers to the different ways a universityoperates. Brick-and-mortar university or onlineuniversity. Course Factors Factors related to the courselevel in Higher EducationInstitutions. Course level, class size Aspects regarding the course. Class size, level of the course. Learning design Learning design decisions made in thecourse. Content, assessments, schedule. Students’ workload Students’ satisfaction with workload. Students’ workload. Student performance Student performance. Student performance, such as grades. DemographicFactors Factors related to demographicinformation of students andprofessors. Gender Aspects regarding the influence of gender. Gender of the students and gender of the professors. Age Aspects regarding the influence of age. Age of the professor. Language Aspects regarding the influence oflanguage. Language of the course. Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3017 Table V: System Usability Survey No. Item 1 Did the system help you to get a more responsible interpretation? 2 I think that I would like to use this system frequently 3 I found the system unnecessarily complex 4 I thought the system was easy to use 5 I think that I would need the support of a technical person to be able to use this system 6 I found the various functions in this system were well integrated 7 I thought there was too much inconsistency in this system 8 I would imagine that most people would learn to use this system very quickly 9 I found the system very diﬀicult to use 10 I felt very confident using the system11 I needed to learn a lot of things before I could get going with this system 3018 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support Table VI: Coding Scheme for user interactions. Code SubCode Definition and Examples Agreement Confirmation A consensus or aﬀirmation regarding a specific statement or piece of information. User: “Yes, of course, the score could not be representative if the questionnaire is filled by a few students.” Assistant: “Yes, the ‘Bottom Courses’ are those with the lowest student enrollment.” Validation Acknowledgment that the provided information is accurate. Assistant: “That is an interesting observa- tion about how student and professor gender might relate to satisfaction scores.”; “Your interpretation effectively explores how the gender of both students and professors might influence satisfaction results across the faculties.” Disagreement When there is a disagreement between the user and the assistant regarding a statement or pieceof information. User: “I don’t think the academic unit influences, but the course factors.”; “there is no infor- mation on the course level.” Express Doubts When there is ambiguity/doubts about the meaning of a statement or piece of information foreither side. User: “I don’t understand what you mind”; “The visualization is comparing the shannon index by teaching methodologies, but for me it is not clear the objective of the visualization.” Provide Information (PI) Clarification When either side clarifies a statement or piece of information. Assistant: “Data Quality examines whether the aggregated scores accurately represent the full range of individual satisfaction responses.” ; User: “just faculty types.” Explanation When there is an explanation from either side about a statement or piece of information. Assis- tant: “The way a methodology is implemented certainly affects the relationship between workload and student performance.”; User: “On the other hand for the explorative purposes it is indeed helpful such presentation.” Directing Attention When there is a statement or piece of information guiding the user in a different direction of rea-soning. Assistant: “It is helpful to start by observing the main elements presented in the visualization.”; “The visualization provides both individual data points (swarm plot) and aggregated summaries (box plot).” Reminder When either side reminds the other about a statement or piece of information. Assistant: “While instrument testing is crucial, remember that the visualization relies on mean satisfaction scores, which are av- erages of many individual student responses.”; “It is important to remember that the Shannon Index calculation relies on the quality of the underlying data regarding resources and activities.” Caution When either side expresses caution about a piece of information or statement. User: “However these results should be interpreted cautiously as there are outliers include in the analysis and seems like the number of female professors evaluated dont match the male professors evaluated”; “Acknowledging the potential for bias across different contexts is crucial when assessing Data Quality.” Asking for Info When there’s a question about a piece of a statement or information. User: “Do you mean that the median cannot be representative if, for example, the sample size is small?”; “How can it be that the average in some cases is lower than the individuals marks?” Continued on next page Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3019 Code SubCode Definition and Examples Sharing Insight When there’s an observation, thought, or knowledge about a piece of a statement or information.User: “Economics faculty: female students seem to be more satisfied with the teaching they received from female teachers, and display a higher variability in the teaching received from male teachers.” ; Assistant: “Different teaching methodologies often require varying levels of student effort and time commitment.” Statement When there’s a piece of information. Assistant: “Your interpretation relies on the assumption that the data accurately reflects the course quality.” ; User: “Having both results (individual scores and the group average) is very helpful” Summary When there’s a summary of the conversation. Assistant: “Here is a summary for Performance and Work- load by Active Teaching Methodology. This analysis of teaching methodologies, performance, and workload re- quires careful consideration of several factors for a responsible interpretation.”; “Here is a summary for Diver- sity of LMS configuration by Teaching Methodology. This visualization compares the variety of LMS resources and activities, measured by the Shannon Index, across Traditional Teaching and Active Learning methodologies.” Request Information (RI) Factors When there’s a clear question or information regarding the framework’s factors. Assistant: “Now that we have considered the data itself, how might the specific context of Demographic Factors — Gender in- fluence your interpretation of the satisfaction differences?”; “How could the specific Learning Design of these methodologies influence whether students feel the workload is justified by the results?” Framework When there’s a question about the framework. Assistant: “Considering the framework, how might the interaction between the student’s gender and the professor’s gender affect the resulting satisfaction scores?” Reflection When a question guides either side through a line of reasoning. Assistant: “How might this specific workload difference influence the performance gap you identified for FLIP?” ; User: “So that it can be a bias?” Interpretation When a question directs either side along an interpretation. Assistant: “How do you think the Learn- ing Design, such as the type of tasks or LMS usage, might influence these outcomes, and how might this relate to the Student’s workload?” ; User: “It can be that one type of methodology has been more implemented in one department and less in other?” Directing Attention When a question leads either side to a new line of reasoning. Assistant: “Now, examine the workload data for FLIP: is there a notable difference in the reported workload between the top and bottom students?”; “Now, let’s shift our focus to the Analytics Pipeline Factors: how might the Data Quality influence the reliabil- ity of these performance and workload measurements?” Visualization When there’s a direct question about the visualization. Assistant: “How does the use of a weighted av- erage specifically impact how you interpret the performance differences between the methodologies shown in the visualization?”; “What does the height or spread of the boxes suggest to you about the variety of LMS configura- tions within each teaching methodology?” Asking for Clarification When there’s a question asking for clarification. User: “Well, if data have been standardized, the mean should be calculated once it has been done, isn’t it?”; “Lectures has more index than others in the top courses, why is it?” Continued on next page 3020 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support Code SubCode Definition and Examples Reflection When there is reasoning based on the interpretation of the visualization. User: “But at the end of the course, I think good students evaluate methodologies that allowed them to increase their knowledge.”; “In my opinion, project-based learning methodologies are hands-on learning situations and experiences, usually made in group, so team-work is a big factor influencing the outcome.” Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3021 Table VII: Number of sub-factors addressed without and with the use of the FACTRIA-aware chatbot. ParticipantAssigned case (max number of sub-factors)WithoutWith P1 CASE 3 (5 sub-factors)13 CASE 4 (4 sub-factors)04 P2 CASE 1 (4 sub-factors)24 CASE 4 (4 sub-factors)14 P3 CASE 1 (4 sub-factors)12 CASE 2 (5 sub-factors)03 P4 CASE 1 (4 sub-factors)24 CASE 4 (4 sub-factors)04 P5 CASE 2 (5 sub-factors)24 CASE 3 (5 sub-factors)11 P6 CASE 2 (5 sub-factors)35 CASE 3 (5 sub-factors)13 P7 CASE 1 (4 sub-factors)13 CASE 2 (5 sub-factors)23 P8 CASE 3 (5 sub-factors)13 CASE 4 (4 sub-factors)14 P9 CASE 1 (4 sub-factors)24 CASE 4 (4 sub-factors)24 P10 CASE 1 (4 sub-factors)23 CASE 2 (5 sub-factors)24 P11 CASE 1 (4 sub-factors)24 CASE 3 (5 sub-factors)23 3022 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support AFACTRIA-Aware Chatbot: Technical Details Each API request incorporates three structured elements: (1) a<CONTEXT>block with the visualization’s title and descriptive text explaining what the chart rep- resents, the underlying data, the variables displayed, and relevant methodolog- ical notes (e.g., standardization procedures, data aggregation methods); (2) a <FACTOR_STATE>block specifying the current factor from FACTRIA; and (3) a <FRAMEWORK>block containing the FACTRIA entries for that case. These XML- like tags mark the injected content as the authoritative source for the response. The system assembles context through direct injection. For each case, two re- searchers identified the FACTRIA entries relevant to the visualization in advance, and the system supplies those entries, the case description, and the tabular values underlying the chart verbatim in every request. This fixed, auditable evidence set therefore bounds generation instead of relying on a query-time search. Three mechanisms limit unsupported statements: the injected blocks restrict response content; the model may reference numerical values only when they appear in the injected data; and near-deterministic decoding reduces variability. We did not measure adherence quantitatively, and we discuss the residual risk among the limitations. The interaction advances through a deterministic state machine, and an aux- iliary classification call selects each transition. The call classifies the first user message as INITIAL_INTERPRETATION, GREETING_OR_SMALLTALK, or READY_BUT_VAGUE. It then classifies each subsequent message, relative to the factor under discussion, as SUBSTANTIVE (advance to the next factor), EXPLORING (re-prompt on the same factor), CLARIFICATION (hand over to the clarification agent), or END (proceed to the summary). The clarifica- tion agent returns a two- to three-sentence definition of at most 80 words. Five prompted components therefore make up the system: an opening-intent classifier, a turn classifier, a clarification agent, the primary agent, and the summarization agent. The primary agent follows an instruction set designed to elicit user reflection. We instruct the primary agent to respond in at most two short sentences (50 words) and avoid technical jargon. The agent never provides direct interpreta- tions of the visualization. We also instruct it to redirect users who overlook or avoid a factor. After the dialogue ends, the summarization agent generates a 150-word sum- mary for practitioners. It receives the full conversation and the relevant FAC- TRIA factors, producing a critical yet constructive synthesis that highlights the factors discussed and gently corrects misunderstandings. The tone remains pro- fessional and accessible, framing limitations through expressions such as “a re- sponsible interpretation could help us understand...”. A JSON schema constrains the output and records, for each factor discussed, a coverage label (thorough, par- tial, or surface) and the insight the user reached, together with the factors not reached and a short narrative. All components use Google’s Gemini API (gemini-2.0-flash) with con- servative settings (temperature = 0.2, top-p = 0.9) to promote coherence and reduce variability. We cap the conversational agent at 200 tokens to maintain brevity and allocate up to 2,048 tokens to the summarization agent for adequate synthesis. The interface operates in English, Spanish, and Catalan, and we in- struct all agents to reply in the participant’s language. The prompts and the Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3023 FACTRIA entries used by the system are available athttps://doi.org/10.5281/ zenodo.23082579. 3024 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support BTransition Network Analysis This section presents case-specific transition networks. Each network includes transitions observed at least twice with a probability greater than 0.2. B.1Case 1 Figure10shows that the assistant guides the user towards responsible interpreta- tion through prompts coded asRI.Reflection,RI.Interpretation, andRI.Factors; some corresponding transitions have a probability ofP= 1. The user subse- quently shares insights (PI.SharingInsight) or expresses agreement (A.Confirmation). Figure 10: Transition Network Analysis Case 1. Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3025 B.2Case 2 Figure11shows a more diverse set of user–assistant interactions. The user con- tributes actions coded asPI.Caution,Reflection, andPI.SharingInsight, while the assistant contributes prompts coded asRI.Reflection,RI.Visualization, and RI.Factors. The higher-probability transitions indicate thatPI.Explanationfol- lows assistantRI.Reflection, while userReflectionfollowsPI.SharingInsight. Figure 11: Transition Network Analysis Case 2. 3026 Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support B.3Case 3 Figure12shows diverse interactions between the user and assistant, consistent with guidance rather than answer provision. The user contributes actions coded asPI.SharingInsight,RI.AskForClarification,ExpressDoubts, andPI.Caution. The assistant asks reflective questions coded asRI.Factors,RI.Interpretation, andRI.Reflection, and explains or validates user contributions throughPI.Explanation andA.Validation. Figure 12: Transition Network Analysis Case 3. Marques et al., Responsible Institutional Analytics: Interpreting Bias with AI Support 3027 B.4Case 4 Figure13shows that prompts coded asRI.Factors,RI.Interpretation, andRI.Reflection precede the user’sPI.SharingInsight. FollowingRI.Reflection, the assistant also contributesPI.Explanation,PI.Clarification, orPI.SharingInsight. When the user expresses doubt or requests clarification (ExpressDoubts;RI.AskForClarification), the assistant responds withPI.Clarificationfollowed by anotherRI.Reflection prompt. Figure 13: Transition Network Analysis Case 4.

Instant research discovery