Modeling Learners’ Cognition and Metacognition In Data Science Problem Solving

Abstract

In today's age of big data, data scientists —a group of specialists trained to process and analyze large, complex amounts of data and interpret and communicate results to inform decision-making processes in interdisciplinary fields —are in high demand. A competent data scientist should be fluent in declarative knowledge (knowing facts) about data science methods, procedural knowledge (knowing how to do things) of executing the methods, and conditional knowledge (knowing when and why to apply specific methods) to address problems effectively. In addition to these cognitive competencies, a data scientist needs to be equipped with metacognitive capabilities (thinking about thinking) to effectively plan, monitor, and regulate the problem-solving processes. As AI tools like ChatGPT become increasingly capable of performing routine, well-specified tasks, training data science learners to be equipped with higher-order Data Science Problem Solving (DSPS) skills and the relevant metacognitive competency becomes critically urgent. Despite the urgent need for DSPS training, current data science education programs often focus on declarative and procedural knowledge in the cognitive domain but lack explicit coaching in high-order problem-solving and related metacognitive skills. Furthermore, the traditional mentorship approach with experienced data scientists that guide practice and make expert thinking visible, inspired by “cognitive apprenticeship,” struggles to scale due to a limited supply of qualified mentors. Consequently, this demand and supply gap may create potential equity concerns, as the scarce mentoring resources will likely be unevenly distributed. In light of these challenges, alternative approaches are needed. This dissertation is built upon an existing case-based learning tool, Caselet that focusing on the training of DSPS that require critical thinking and metacognitive skills besides declarative and procedural knowledge. Additionally, by tracking and modeling learners' cognitive and metacognitive progress, this work lays the foundation for a future student model that could be a key component of an intelligent tutoring system (ITS). An ITS is a computer-based system designed to provide personalized and adaptive instruction to learners, mimicking the guidance and support typically offered by human tutors. This kind of system could potentially address the second challenge, namely, the shortage of qualified mentors. Leveraging machine learning and learning analytics to model and track learners' cognition and metacognition in DSPS tasks will be achieved through three primary objectives: (1) building models of cognitive and metacognitive processes while learners are engaged in DSPS activities and (2) exploring the role of cognition and metacognition in predicting learning outcomes. By understanding the cognitive and metacognitive processes employed by learners during DSPS tasks and elucidating the connection between learners' cognition and metacognition, this dissertation aims to inform the design of student model within ITS to facilitate large-scale individualized DSPS learning and, ultimately, enhance the education of competent data science problem solvers in the era of AI.