A Comparison Between the Use of Latent D-Scoring Method Models and Item Response Theory Models with Respect to Item Fit and Person Recovery Parameter

Abstract

Psychometricians at an organization named the Education and Training Evaluation Commission (ETEC) developed a new test scoring method called the latent D-scoring method (DSM-L) where it is believed that the new method itself is much easier and more efficient to use compared to the Item Response Theory (IRT) method. However, there are no studies that compared the methods directly regarding item fit and person parameter recovery to understand if the new method (DSM-L) provides better item fit and person parameter recovery compared to the IRT. Therefore, the purpose of this study was to compare the use of latent D-scoring method (DSM-L) models (RFM1 and RFM2) and Item Response Theory (IRT) models (1-PL and 2-PL) with respect to item fit and person parameter recovery across different test data from the ETEC. It was found that both the DSM-L and IRT methods show similar results under both Mean Absolute Error (MAE) and Mean Absolute Difference (MAD) item fit statistics methods across the different test data sets from the ETEC. Also, I found that larger sample sizes and test lengths were associated with improved item fit under the MAE item fit statistics across the different methods. I also found that a two-parameter model type (RFM2 and 2-PL models) shows lower mean MAE item fit statistics across the test data sets compared to a one-parameter model type (RFM1 and 1-PL models). Finally, the DSM-L RFM2 model showed a slightly lower RMSE rate compared to the RFM1 model. Also, both RFM1 and RFM2 models showed acceptable rate of bias (bias equal to zero). On the other hand, the IRT 1-PL model showed a lower RMSE rate compared to the 2-PL model, and 1-PL model showed an acceptable rate of bias (bias approximately equal to zero). With the study outcomes in mind, the psychometricians at the ETEC can use either method (DSM-L or IRT) if the goal is to have good item fit under the MAE item fit statistics since both methods showed similar MAE results. I also recommend using the two-parameter model type based on MAE item fit statistics because it shows lower mean MAE compared to using the one parameter model type. On the other hand, if the goal is to have lower RMSE rates and acceptable rates of bias (bias approximately equal to zero) under IRT I recommend using the 1-PL model. Also, if the goal is to have lower RMSE rate under the DSM-L I recommend using the RFM2 model, and if the goal is to have an acceptable rate of bias under the DSM-L I recommend using either the RFM1 or RFM-2 models.