Industrial AI-based Workload Performance Analytics: Applications to Mixed Reality Multitasking

Abstract

Immersive technologies such as augmented and virtual reality are increasingly integrated into our daily lives. As this digital transformation progresses, understanding human reactions to these technologies becomes crucial, particularly in the context of human factors engineering, which prioritizes human safety and well-being. Mixed reality (MR), which blends the physical and virtual worlds, introduces new multitasking possibilities but also presents challenges. One critical aspect is the impact of MR multitasking on human workload, a key performance measure. This research employs an Industrial AI approach, combining traditional machine learning with advanced pre-trained models to develop predictive models for estimating human workload in MR environments. An experiment was conducted in which participants multitasked between a physical and a digital task within a defined timeframe. Workload data, collected via the NASA Task Load Index (NASA-TLX), was used alongside synthetic data generated by a Generative Adversarial Network (GAN) to create an enriched dataset. The combined real and synthetic data were then used to train predictive models, enhancing accuracy. To improve workload prediction, this study integrates pre-trained models such as BERT (Bidirectional Encoder Representations from Transformers) from large language models (LLMs) and CLIP (Contrastive Language-Image Pretraining) from computer vision applications, alongside traditional machine learning techniques like regression and neural networks. Evaluation using the Root Mean Square Error (RMSE) metric demonstrates that the proposed hybrid models incorporating transfer learning and pre-trained models significantly outperformed conventional methods. The deviations between actual and predicted values were minimal, indicating a more reliable workload estimation. This dissertation advances knowledge in human factors engineering by addressing a critical gap in workload prediction within MR multitasking environments. The findings provide insights into human-computer interaction in complex digital settings. Organizations implementing MR technologies can leverage these predictive models to better understand worker workload and optimize conditions for well-being and efficiency.