Multimodal Data Fusion to Track Students’ Distress during Educational Gameplay

Authors

  • Jewoong Moon The University of Alabama
  • Fengfeng Ke Florida State University
  • Zlatko Sokolikj Florida State University
  • Ibrahim Dahlstrom-Hakki Technical Education Research Centers (TERC)

DOI:

https://doi.org/10.18608/jla.2022.7631

Keywords:

educational game, distress, educational data mining, multimodal data fusion

Abstract

Using multimodal data fusion techniques, we built and tested prediction models to track middle-school student distress states during educational gameplay. We collected and analyzed 1,145 data instances, sampled from a total of 31 middle-school students’ audio- and video-recorded gameplay sessions. We conducted data wrangling with student gameplay data from multiple data sources, such as individual facial expression recordings and gameplay logs. Using supervised machine learning, we built and tested candidate classifiers that yielded an estimated probability of distress states. We then conducted confidence-based data fusion that averaged the estimated probability scores from the unimodal classifiers with a single data source. The results of this study suggest that the classifier with multimodal data fusion improves the performance of tracking distress states during educational gameplay, compared to the performance of unimodal classifiers. The study finding suggests the feasibility of multimodal data fusion in developing game-based learning analytics. Also, this study proposes the benefits of optimizing several methodological means for multimodal data fusion in educational game research.

References

Alam, M. R., Bennamoun, M., Togneri, R., & Sohel, F. (2015). A confidence-based late fusion framework for audio-visual biometric identification. Pattern Recognition Letters, 52, 65–71. https://doi.org/10.1016/j.patrec.2014.10.006

Asbell-Clarke, J., Rowe, E., Almeda, V., Edwards, T., Bardar, E., Gasca, S., Baker, R. S., & Scruggs, R. (2021). The development of students’ computational thinking practices in elementary- and middle-school classes using the learning game, Zoombinis. Computers in Human Behavior, 115, 106587. https://doi.org/10.1016/j.chb.2020.106587

Baker, R. S., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive-affective states during interactions with three different computer-based learning environments. International Journal of Human–Computer Studies, 68(4), 223–241. https://doi.org/10.1016/j.ijhcs.2009.12.003

Baltrušaitis, T., Robinson, P., & Morency, L. P. (2016, March). OpenFace: An open source facial behavior analysis toolkit. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV) (pp. 1–10). IEEE. https://doi.org/10.1109/WACV.2016.7477553

Beck, J., & Rodrigo, M. M. T. (2014). Understanding wheel spinning in the context of affective factors. In S. Trausan-Matu, K. E. Boyer, M. Crosby, & K. Panourgia (Eds.), Proceedings of the 12th International Conference on Intelligent Tutoring Systems (ITS 2014), 5–9 June 2014, Honolulu, HI, USA (pp. 162–167). Springer. https://doi.org/10.1007/978-3-319-07221-0_20

Biglan, A. (1991). Distressed behavior and its context. The Behavior Analyst, 14(2), 157–169. https://doi.org/10.1007/BF03392566

Bosch, N., Chen, Y., & D’Mello, S. (2014). It’s written on your face: Detecting affective states from facial expressions while learning computer programming. In S. Trausan-Matu, K. E. Boyer, M. Crosby, & K. Panourgia (Eds.), Proceedings of the 12th International Conference on Intelligent Tutoring Systems (ITS 2014), 5–9 June 2014, Honolulu, HI, USA (pp. 39–44). Springer. https://doi.org/10.1007/978-3-319-07221-0_5

Bosch, N., D’Mello, S., Baker, R., Ocumpaugh, J., Shute, V., Ventura, M., Wang, L., & Zhao, W. (2015). Automatic detection of learning-centered affective states in the wild. Proceedings of the 20th International Conference on Intelligent User Interfaces (IUI ’15), 29 March–1 April 2015, Atlanta, GA, USA (pp. 379–388). ACM Press. https://doi.org/10.1145/2678025.2701397

Bowman, N. D., & Tamborini, R. (2012). Task demand and mood repair: The intervention potential of computer games. New Media & Society, 14(8), 1339–1357. https://doi.org/10.1177/146144481245042

Chrousos, G. P., Loriaux, D. L., & Gold, P. W. (2013). Mechanisms of physical and emotional stress (Vol. 245). Springer Science & Business Media.

Chango, W., Cerezo, R., & Romero, C. (2021). Multi-source and multimodal data fusion for predicting academic performance in blended learning university courses. Computers & Electrical Engineering, 89, 106908. https://doi.org/10.1016/j.compeleceng.2020.106908

Chen, F., Cui, Y., & Chu, M. W. (2020). Utilizing game analytics to inform and validate digital game-based assessment with evidence-centered game design: A case study. International Journal of Artificial Intelligence in Education, 30(3), 481–503. https://doi.org/10.1007/s40593-020-00202-6

DeFalco, J. A., Rowe, J. P., Paquette, L., Georgoulas-Sherry, V., Brawner, K., Mott, B. W., Baker, R. S., & Lester, J. C. (2018). Detecting and addressing frustration in a serious game for military training. International Journal of Artificial Intelligence in Education, 28(2), 152–193. https://doi.org/10.1007/s40593-017-0152-1

Emerson, A., Cloude, E. B., Azevedo, R., & Lester, J. (2020). Multimodal learning analytics for game‐based learning. British Journal of Educational Technology, 51(5), 1505–1526. https://doi.org/10.1111/bjet.12992

Eseryel, D., Law, V., Ifenthaler, D., Ge, X., & Miller, R. (2014). An investigation of the interrelationships between motivation, engagement, and complex problem solving in game-based learning. Journal of Educational Technology & Society, 17(1), 42–53. http://hdl.handle.net/20.500.11937/26368

Fernández, A., Garcia, S., Herrera, F., & Chawla, N. V. (2018). SMOTE for learning from imbalanced data: Progress and challenges, marking the 15-year anniversary. Journal of Artificial Intelligence Research, 61, 863–905. https://doi.org/10.1613/jair.1.11192

Filsecker, M., & Hickey, D. T. (2014). A multilevel analysis of the effects of external rewards on elementary students’ motivation, engagement and learning in an educational game. Computers & Education, 75, 136–148. https://doi.org/10.1016/j.compedu.2014.02.008

Fultz, J., Schaller, M., & Cialdini, R. B. (1988). Empathy, sadness, and distress: Three related but distinct vicarious affective responses to another’s suffering. Personality and Social Psychology Bulletin, 14(2), 312–325. https://doi.org/10.1177/0146167288142009

Giannopoulos, P., Perikos, I., & Hatzilygeroudis, I. (2018). Deep learning approaches for facial emotion recognition: A case study on FER-2013. In I. Hatzilygeroudis & V. Palade (Eds.), Advances in hybridization of intelligent methods (pp. 1–16). Springer. https://doi.org/10.1007/978-3-319-66790-4_1

Goodfellow, I. J., Erhan, D., Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., Zhou, Y., Ramaiah, C., Feng, F., Li, R., Wang, X., Athanasakis, D., Shawe-Taylor, J., Milakov, M., Park, J., ... & Bengio, Y. (2013, November). Challenges in representation learning: A report on three machine learning contests. ICML 2013 Workshop on Challenges in Representation Learning (pp. 117–124). Springer. https://doi.org/10.48550/arXiv.1307.0414

Gupta, H., Varshney, H., Sharma, T. K., Pachauri, N., & Verma, O. P. (2021). Comparative performance analysis of quantum machine learning with deep learning for diabetes prediction. Complex & Intelligent Systems, 8, 3073–3087. https://doi.org/10.1007/s40747-021-00398-7

Hamari, J., Shernoff, D. J., Rowe, E., Coller, B., Asbell-Clarke, J., & Edwards, T. (2016). Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning. Computers in Human Behavior, 54, 170–179. https://doi.org/10.1016/j.chb.2015.07.045

Henderson, N. L., Rowe, J. P., Mott, B. W., Brawner, K., Baker, R., & Lester, J. C. (2019). 4D affect detection: Improving frustration detection in game-based learning with posture-based temporal data fusion. In S. Isotani, E. Millán, A. Ogan, P. Hastings, B. McLaren, & R. Luckin (Eds.), Proceedings of the 20th International Conference on Artificial Intelligence in Education (AIED 2019), 25–29 June 2019, Chicago, IL, USA (pp. 144–156). Springer. https://doi.org/10.1007/978-3-030-23204-7_13

Henderson, N., Rowe, J., Paquette, L., Baker, R. S., & Lester, J. (2020). Improving affect detection in game-based learning with multimodal data fusion. In I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, E. Millán (Eds.), Proceedings of the 21st International Conference on Artificial Intelligence in Education (AIED 2020), 6–10 July 2020, Ifrane, Morocco (pp. 228–239). Springer. https://doi.org/10.1007/978-3-030-52237-7_19

Hofmann, M., & Klinkenberg, R. (Eds.). (2016). RapidMiner: Data mining use cases and business analytics applications. CRC Press.

Jadhav, A., Pramod, D., & Ramanathan, K. (2019). Comparison of performance of data imputation methods for numeric dataset. Applied Artificial Intelligence, 33(10), 913–933. https://doi.org/10.1080/08839514.2019.1637138

Kessler, R. C., Andrews, G., Colpe, L. J., Hiripi, E., Mroczek, D. K., Normand, S. L., Walters, E. E., & Zaslavsky, A. M. (2002). Short screening scales to monitor population prevalences and trends in non-specific psychological distress. Psychological Medicine, 32(6), 959–976. https://doi.org/10.1017/s0033291702006074

LePine, J. A., LePine, M. A., & Jackson, C. L. (2004). Challenge and hindrance stress: Relationships with exhaustion, motivation to learn, and learning performance. Journal of Applied Psychology, 89(5), 883–891. https://doi.org/10.1037/0021-9010.89.5.883

Liu, T., & Israel, M. (2022). Uncovering students’ problem-solving processes in game-based learning environments. Computers & Education, 182, 104462. https://doi.org/10.1016/j.compedu.2022.104462

Malarvizhi, R., & Thanamani, A. S. (2012). K-nearest neighbor in missing data imputation. International Journal of Engineering Research and Development, 5(1), 5–7. https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.375.925&rep=rep1&type=pdf

Mouri, K., Suzuki, F., Shimada, A., Uosaki, N., Yin, C., Kaneko, K., & Ogata, H. (2019). Educational data mining for discovering hidden browsing patterns using non-negative matrix factorization. Interactive Learning Environments, 29(7), 1176-1188. https://doi.org/10.1080/10494820.2019.1619594

Mu, T., Jetten, A., & Brunskill, E. (2020). Towards suggesting actionable interventions for wheel-spinning students. Conference on Educational Data Mining (EDM2020), 10–13 July 2020, Online (pp. 183–193). International Educational Data Mining Society. https://educationaldatamining.org/files/conferences/EDM2020/papers/paper_201.pdf

Munshi, A., Rajendran, R., Ocumpaugh, J., Biswas, G., Baker, R. S., & Paquette, L. (2018, July). Modeling learners’ cognitive and affective states to scaffold SRL in open-ended learning environments. Proceedings of the 26th Conference on User Modeling, Adaptation and Personalization (UMAP 2018), 8–11 July 2018, Singapore (pp. 131–138). ACM Press. https://doi.org/10.1145/3209219.3209241

Naylor, R. (2020). Key factors influencing psychological distress in university students: The effects of tertiary entrance scores. Studies in Higher Education, 47(3), 630–642. https://doi.org/10.1080/03075079.2020.1776245

Ochoa, X., & Worsley, M. (2016). Augmenting learning analytics with multimodal sensory data. Journal of Learning Analytics, 3(2), 213–219. https://doi.org/10.18608/jla.2016.32.10

Owen, V. E., Roy, M. H., Thai, K. P., Burnett, V., Jacobs, D., Keylor, E., & Baker, R. S. (2019). Detecting wheel-spinning and productive persistence in educational games. In C. F. Lynch, A. Merceron, M. Desmarais, & R. Nkambou (Eds.), Proceedings of the 12th International Conference on Educational Data Mining (EDM2019), 2–5 July 2019, Montréal, Quebec, Canada (pp. 378–383). International Educational Data Mining Society. https://files.eric.ed.gov/fulltext/ED599202.pdf

Plass, J. L., Homer, B. D., Pawar, S., Brenner, C., & MacNamara, A. P. (2019). The effect of adaptive difficulty adjustment on the effectiveness of a game to develop executive function skills for learners of different ages. Cognitive Development, 49, 56–67. https://doi.org/10.1016/j.cogdev.2018.11.006

Rodrigo, M. M. T. (2011). Dynamics of student cognitive-affective transitions during a mathematics game. Simulation & Gaming, 42(1), 85–99. https://doi.org/10.1177/1046878110361513

Rudland, J. R., Golding, C., & Wilkinson, T. J. (2020). The stress paradox: How stress can be good for learning. Medical Education, 54(1), 40–45. https://doi.org/10.1111/medu.13830

Rowe, E., Almeda, M. V., Asbell-Clarke, J., Scruggs, R., Baker, R., Bardar, E., & Gasca, S. (2021). Assessing implicit computational thinking in Zoombinis puzzle gameplay. Computers in Human Behavior, 120, 106707. https://par.nsf.gov/servlets/purl/10061932

Silvervarg, A., Haake, M., & Gulz, A. (2018). Perseverance is crucial for learning: “OK! but can I take a break?” In C. P. Rosé et al. (Eds.), Proceedings of the 19th International Conference on Artificial Intelligence in Education (AIED 2018), 27–30 June 2018, London, UK (pp. 532–544). Lecture Notes in Computer Science vol. 10947. Springer. https://doi.org/10.1007/978-3-319-93843-1_39

Shute, V. J., D’Mello, S., Baker, R., Cho, K., Bosch, N., Ocumpaugh, J., Ventura, M., & Almeda, V. (2015). Modeling how incoming knowledge, persistence, affective states, and in-game progress influence student learning from an educational game. Computers & Education, 86, 224–235. https://doi.org/10.1016/j.compedu.2015.08.001

Taub, M., Azevedo, R., Bradbury, A. E., & Mudrick, N. V. (2020). Self-regulation and reflection during game-based learning. In J. L. Plass, R. E. Mayer, & B. D. Homer (Eds.), Handbook of game-based learning (pp. 239–262). The MIT Press.

Verma, V., Rheem, H., Amresh, A., Craig, S. D., & Bansal, A. (2020, December). Predicting real-time affective states by modeling facial emotions captured during educational video game play. In I. Marfisi-Schottman, F. Bellotti, L. Hamon, & R. Klemke (Eds.), Proceedings of the International Conference on Games and Learning Alliance (GALA 2020), 1–3 December 2020, La Spezia, Italy. Lecture Notes in Computer Science vol. 12517. (pp. 447–452). Springer. https://doi.org/10.1007/978-3-030-63464-3_45

Yeh, Y. C., Lai, G. J., Lin, C. F., Lin, C. W., & Sun, H. C. (2015). How stress influences creativity in game-based situations: Analysis of stress hormones, negative emotions, and working memory. Computers & Education, 81, 143–153. https://doi.org/10.1016/j.compedu.2014.09.011

Downloads

Published

2022-11-20

How to Cite

Moon, J., Ke, F., Sokolikj, Z., & Dahlstrom-Hakki, . I. (2022). Multimodal Data Fusion to Track Students’ Distress during Educational Gameplay. Journal of Learning Analytics, 1-13. https://doi.org/10.18608/jla.2022.7631

Issue

Section

Research Papers

Most read articles by the same author(s)