TO D OR NOT TO D: SOME METHODOLOGICAL AND ESTIMATION PERSPECTIVES ON THE D-SCORING METHOD OF MEASUREMENT

Abstract

For more than a century, educational and psychological measurements have relied on two primary measurement frameworks throughout the measurement development process: classical test theory (CTT) and item response theory (IRT). A new measurement framework was proposed by Dimitrov and adopted by the National Center of Assessment (NCA) in Saudi Arabia, known as the D-scoring Method (DSM). Due to its simplicity and transparency, the interest of psychometricians in DSM has been increasing, especially in Saudi Arabia. The aim of this study was to investigate some of the psychometric features of the DSM. Studies have been conducted on the DSM, but further work is still needed. This study included two goals. The first goal was to investigate the accuracy of two DSM models, the rational function model with two parameters (RFM2) and the rational function model with three parameters (RFM3), under four different sample sizes (200, 300, 500, 1,000) and three different test lengths (10, 20, 40) using three different estimation methods: the classical approach, the maximum likelihood estimation (MLE), and the Markov chain Monte Carlo estimation (MCMC). The second goal was to thoroughly compare the DSM, CTT, and IRT. A simulation study was conducted to examine the first goal. The results showed that the RFM2 under the MCMC estimation only required 200 examinees and 10 items test to provide a good level of accuracy. Furthermore, the results showed that the RFM3 under the MCMC estimation only required 200 examinees and 40 items test to provide a good level of accuracy. In contrast, the RFM2 and RFM3 under classical and MLE approaches failed to provide a reasonable level of accuracy. An empirical study was conducted to target the second goal. The results showed high comparability between the DSM, CTT, and IRT. The results of this study showed that the DSM framework is a promising framework given its simplicity. The implications and limitations of the results of this study are discussed as well.