Assessing the Performance of Model Fit Indices in Multilevel Structural Equation Modeling: A Comparison of the Standard Approach, Level-Specific Evaluation, and Equivalence Testing

Abstract

Multilevel Structural Equation Modeling (MSEM) is a method suitable for analyzing data with multi-level structure. This method is particularly useful for exploring relationships across different levels of analysis. The objective of this dissertation was to assess the performance of model fit indices in MSEM using a standard approach (SA), level-specific (LS) evaluation, and equivalence testing (ET) methods. Monte Carlo Simulations were implemented to contrast the performance of common fit indices in each method in MSEM under three design factors: sample size (SS), intraclass correlation coefficient (ICC), and specification model (SM). Additionally, the effectiveness of SA, LS, and ET approaches were evaluated using real-world data, utilizing emotional intelligence as a personality state dataset.

The results demonstrated that the SA model fits, assessed using CFI and RMSEA, effectively identified the correct specification model (CSM) and rejected the measurement misspecification model (MMM) across all SSs and ICCs. However, these fit indices failed to detect the structure misspecification model (SMM). Furthermore, the LS model fits, including CFILSW, CFILSB, RMSEALSW, and RMSEALSB, successfully retained the CSM and rejected the MMM. Only the CFILSW was sensitive to detecting the misfit of SMM when the ICC was 0.3. In the examination of ET for the CSM, the T-size (CFIETWt, RMSEAETWt, and RMSEAETBt) indicated excellent fit across all SSs and ICCs, while the CFIETBt showed excellent fit with an ICC of 0.1 and close fit with an ICC of 0.3. All T-sizes measures except RMSEAETBt, rejected MMM, while only CFIETBt consistently detected SMM across all SSs and ICCs. ANOVA results indicated that specification model (SM) had the most significant effect on the performance of fit indices, followed by ICC and SS. Real-world data analysis supported these findings, highlighting the limitations of traditional fit indices and emphasizing the need for comprehensive evaluation methods to detect model misspecifications accurately. Although no single approach performed successfully in all scenarios, a combined approach, especially LS and ET, using multiple indices and methodologies is recommended for a more robust and accurate assessment of model fit in MSEM.