Generating Complex Questions from Domain Ontologies

Abstract

Designing and constructing pedagogical tests that equitably measure various skills across different students is a challenging task. The quality and validity of assessments are heavily reliant on the quality of the questions included. Traditional test development methods rely on manual effort, which can be time-consuming, labour-intensive and inconsistent, leading to variability in question quality. This challenge is further compounded by the advent of online learning platforms that require a large and diverse pool of questions, making manual creation and review impractical. Furthermore, for effective assessments, questions must be calibrated with difficulty levels before being incorporated into exams. However, difficulty calibration is another challenge that complicates questions development. In recent years, Automatic Question Generation (AQG) has emerged as a powerful tool for effortlessly generating assessment questions in massive numbers with a minimal level of human intervention. Ontologies have been used as a semantic knowledge source to generate questions automatically. However, most questions that have previously been generated from the use of ontologies have been criticised for their simplicity and lack of cognitive engagement. Furthermore, many existing question generation frameworks primarily focus on the technical aspects, but they lack a strong theoretical foundation. This highlights the need to enrich existing question types by generating more complex questions that cover both a broader and deeper understanding of the associated knowledge and that require more complex reasoning skills than that which is necessary for recalling simple facts. In this thesis, we present a novel ontology-based question generation approach designed to facilitate the creation of complex educational questions, which require larger knowledge coverage and higher cognitive processes. Our method leverages the concept of Query Graphs, a graph-like structure capable of representing natural language queries through appropriate mappings. We propose the use of Query Graphs as a formalism for representing templates that incorporate multiple ontology-based constraints to elevate the level of reasoning required to answer the questions. We demonstrate that this approach is indeed effective and aligns with the assessment of educational experts. To further support the plausibility of our computational framework, we shed light on its consistency with theories from education and cognitive psychology. This provides a solid theoretical foundation that ensures that questions are generated according to principled methods that are grounded in the theory of learning and cognition. The proposed approach is agnostic to the choice of different subject areas or knowledge domains, and independent of the question format. Therefore, the approach proposed is highly general and applicable to a variety of contexts. Being the primary source of knowledge, ontologies have an impact on the effectiveness of the quality of the questions generated. Therefore, we examine how different ontologies perform when applied to the same question generating task. An expert-based study was conducted, leading to the identification of ontology evaluation metrics designed to assess the suitability of domain ontologies for successful use in AQG. These metrics facilitate the reuse of existing ontologies and reduces the need to develop new ones from scratch, thereby lowering the cost of implementing AQG models.