Game-Based Learning Prediction Model Construction
Towards Validated Stealth Assessment Implementation
DOI:
https://doi.org/10.18608/jla.2025.8105Keywords:
serious games, learning analytics, game-based learning, logging system, computational model, conceptual model and frameworks, machine learning, prediction model, educational data mining, performance measurement, research paperAbstract
Game-based learning (GBL) is increasingly recognized as an effective tool for teaching diverse skills, particularly in science education, due to its interactive, engaging, and motivational qualities, along with timely assessments and intelligent feedback. However, more empirical studies are needed to facilitate its wider application in school curricula. A significant challenge is designing and implementing valid in-game assessments crucial for measuring student progress and providing reliable references for instructors’ intervention decisions. Stealth assessment, guided by the evidence-centred design (ECD) framework, offers a promising solution but requires more specific guidelines for full effectiveness. In this study, we present a granular, framework-supported pipeline to systematically implement stealth assessments in a GBL environment. This pipeline involves constructing an ECD framework, generating features, selecting appropriate models, preprocessing data, evaluating model performance, and conducting model inference on a black-box computational model. We validate the effectiveness of this pipeline by assessing the performance of these computational models and identifying distinct behavioural patterns between high and low performers. Our analysis highlights potential areas for improvement in the design of stealth assessments within digital games for learning. Furthermore, we discuss the generalizability of the proposed pipeline and outline limitations for future research to address.
References
Abrahart, R. J., & See, L. (2000). Comparing neural network and autoregressive moving average techniques for the provision of continuous river flow forecasts in two contrasting catchments. Hydrological Processes, 14(11-12), 2157–2172. https://doi.org/10.1002/1099-1085(20000815/30)14:11/12%3C2157::AID-HYP57%3E3.0.CO;2-S
Agudo-Peregrina, A. F., Iglesias-Pradas, S., Conde-Gonzalez, M. A., & Hernandez-Garcıa, A. (2014). Can we predict success from log data in VLEs? Classification of interactions for learning analytics and their relation with performance in VLE-supported F2F and online learning. Computers in Human Behavior, 31, 542–550. https://doi.org/10.1016/j.chb.2013.05.031
Akram, B., Min, W., Wiebe, E., Mott, B., Boyer, K. E., & Lester, J. (2018). Improving stealth assessment in game-based learning with LSTM-based analytics. In K. E. Boyer & M. Yudelson (Eds.), Proceedings of the 11th International Conference on Educational Data Mining (EDM 2018), 15–18 July 2018, Buffalo, New York, USA (pp. 208–218). EDM. https://educationaldatamining.org/files/conferences/EDM2018/EDM2018_Preface_TOC_Proceedings.pdf
Albert, J. H. (1988). Computational methods using a Bayesian hierarchical generalized linear model. Journal of the American Statistical Association, 83(404), 1037–1044. https://doi.org/10.1080/01621459.1988.10478698
Alonso-Fernandez, C., Calvo-Morata, A., Freire, M., Martınez-Ortiz, I., & Manjon, B. F. (2021). Data science meets standardized game learning analytics. In T. Klinger, C. Kollmitzer, & A. Pester (Eds.), Proceedings of the 2021 IEEE Global Engineering Education Conference (EDUCON 2021), 21–23 April 2021, Vienna, Austria (pp. 1546–1552). IEEE. https://doi.org/10.1109/EDUCON46332.2021.9454134
Alonso-Fernandez, C., Freire, M., Martınez-Ortiz, I., & Fernandez-Manjon, B. (2021). Improving evidence-based assessment of players using serious games. Telematics and Informatics, 60, 101583. https://doi.org/10.1016/j.tele.2021.101583
Amory, A. (2007). Game object model version II: A theoretical framework for educational game development. Educational Technology Research and Development, 55(1), 51–77. https://doi.org/10.1007/s11423-006-9001-x
Arnab, S., Lim, T., Carvalho, M. B., Bellotti, F., de Freitas, S., Louchart, S., Suttie, N., Berta, R., & De Gloria, A. (2015). Mapping learning and game mechanics for serious games analysis. British Journal of Educational Technology, 46(2), 391–411. https://doi.org/10.1111/bjet.12113
Ayzenberg, Y., Hernandez Rivera, J., & Picard, R. (2012). FEEL: Frequent EDA and event logging—a mobile social interaction stress monitoring system. In CHI EA 2012: CHI ’12 Extended Abstracts on Human Factors in Computing Systems, 5–10 May 2012, Austin, Texas, USA (pp. 2357–2362). ACM. https://doi.org/10.1145/2212776.2223802
Baker, R. S., Corbett, A. T., Koedinger, K. R., & Wagner, A. Z. (2004). Off-task behavior in the cognitive tutor classroom: When students “game the system.” In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI 2004), 24–29 April 2004, Vienna, Austria (pp. 383–390). ACM. https://doi.org/10.1145/985692.985741
Baker, R. S., D’Mello, S. K., Rodrigo, M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223–241. https://doi.org/10.1016/j.ijhcs.2009.12.003
Belland, B. R., Walker, A. E., & Kim, N. J. (2017). A Bayesian network meta-analysis to synthesize the influence of contexts of scaffolding use on cognitive outcomes in STEM education. Review of Educational Research, 87(6), 1042–1081. https://doi.org/10.3102/0034654317723009
Bernecker, K., & Ninaus, M. (2021). No pain, no gain? Investigating motivational mechanisms of game elements in cognitive tasks. Computers in Human Behavior, 114, 106542. https://doi.org/10.1016/j.chb.2020.106542
Beserra, V., Nussbaum, M., & Oteo, M. (2019). On-task and off-task behavior in the classroom: A study on mathematics learning with educational video games. Journal of Educational Computing Research, 56(8), 1361–1383. https://doi.org/10.1177/0735633117744346
Bird, K. A., Castleman, B. L., Mabel, Z., & Song, Y. (2021). Bringing transparency to predictive analytics: A systematic comparison of predictive modeling methods in higher education. AERA Open, 7, 23328584211037630. https://doi.org/10.1177/23328584211037630
Breiman, L. (2001). Random forests. Machine Learning, 45, 5–32. https://doi.org/10.1023/A:1010933404324
Breuer, J. S., & Bente, G. (2010). Why so serious? On the relation of serious games and learning. Eludamos: Journal for Computer Game Culture, 4(1), 7–24. https://doi.org/10.7557/23.6111
Buda, M., Maki, A., & Mazurowski, M. A. (2018). A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks, 106, 249–259. https://doi.org/10.1016/j.neunet.2018.07.011
Butcher, B., & Smith, B. J. (2020). Feature engineering and selection: A practical approach for predictive models: By Max Kuhn and Kjell Johnson. Boca Raton, FL: Chapman & Hall/CRC Press, 2019, xv + 297 pp., $79.95(H), ISBN: 978-1-13-807922-9 [Review]. The American Statistician, 74(3), 308–309. https://doi.org/10.1080/00031305.2020.1790217
Calvo-Morata, A., Rotaru, D. C., Alonso-Fernandez, C., Freire-Moran, M., Martınez-Ortiz, I., & Fernandez-Manjon, B. (2020). Validation of a cyberbullying serious game using game analytics. IEEE Transactions on Learning Technologies, 13(1), 186–197. https://doi.org/10.1109/TLT.2018.2879354
Cardia da Cruz, L., Sierra-Franco, C. A., Silva-Calpa, G. F. M., & Barbosa Raposo, A. (2020). A self-adaptive serious game for eye-hand coordination training. In X. Fang (Ed.), HCI in games. HCII 2020. Lecture notes in computer science (pp. 385–397, Vol. 12211). Springer International Publishing. https://doi.org/10.1007/978-3-030-50164-8_28
Carpenter, D., Emerson, A., Mott, B. W., Saleh, A., Glazewski, K. D., Hmelo-Silver, C. E., & Lester, J. C. (2020). Detecting off-task behavior from student dialogue in game-based collaborative learning. In I. I. Bittencourt, M. Cukurova, K. Muldner, R. Luckin, & E. Millan (Eds.), Proceedings of the 21st International Conference on Artificial Intelligence in Education, Part I (AIED 2020), 6–10 July 2020, Ifrane, Morocco (pp. 55–66). ACM. https://doi.org/10.1007/978-3-030-52237-7_5
Carvalho, M. B., Bellotti, F., Berta, R., De Gloria, A., Sedano, C. I., Hauge, J. B., Hu, J., & Rauterberg, M. (2015). An activity theory-based model for serious games analysis and conceptual design. Computers & Education, 87, 166–181. https://doi.org/10.1016/j.compedu.2015.03.023
Cerra, P. P., Alvarez, H. F., Parra, B. B., & Cordera, P. I. (2022). Effects of using game-based learning to improve the academic performance and motivation in engineering studies. Journal of Educational Computing Research, 60(7), 1663–1687. https://doi.org/10.1177/07356331221074022
Champion, C., & Elkan, C. (2017). Visualizing the consequences of evidence in Bayesian networks. arXiv preprint arXiv:1707.00791. https://arxiv.org/abs/1707.00791
Chang, C.-C., Liang, C., Chou, P. - N., & Lin, G. - Y. (2017). Is game-based learning better in flow experience and various types of cognitive load than non-game-based learning? Perspective from multimedia and media richness. Computers in Human Behavior, 71, 218–227. https://doi.org/10.1016/j.chb.2017.01.031
Chen, C.-H., Law, V., & Huang, K. (2019). The roles of engagement and competition on learner’s performance and motivation in game-based science learning. Educational Technology Research and Development, 67(4), 1003–1024. https://doi.org/10.1007/s11423-019-09670-7
Chen, F., Cui, Y., & Chu, M. W. (2020). Utilizing game analytics to inform and validate digital game-based assessment with evidence-centered game design: A case study. International Journal of Artificial Intelligence in Education, 30(4), 481–503. https://doi.org/10.1007/s40593-020-00202-6
Cloude, E. B., Dever, D. A., Wiedbusch, M. D., & Azevedo, R. (2020). Quantifying scientific thinking using multichannel data with Crystal Island: Implications for individualized game-learning analytics. Frontiers in Education, 5, 21. https://doi.org/10.3389/feduc.2020.572546
Cram er, H. (1999). Mathematical methods of statistics. Princeton University Press. https://press.princeton.edu/books/paperback/9780691005478/mathematical-methods-of-statistics
Dabbous, M., Kawtharani, A., Fahs, I., Hallal, Z., Shouman, D., Akel, M., Rahal, M., & Sakr, F. (2022). The role of game-based learning in experiential education: Tool validation, motivation assessment, and outcomes evaluation among a sample of pharmacy students. Education Sciences, 12(7), 434. https://doi.org/10.3390/educsci12070434
David Des Armier Jr., C. E. S., & Skrabut, S. (2016). Using game elements to increase student engagement in course assignments. College Teaching, 64(2), 64–72. https://doi.org/10.1080/87567555.2015.1094439
De Freitas, S., & Oliver, M. (2006). How can exploratory learning with games and simulations within the curriculum be most effectively evaluated? Computers & Education, 46(3), 249–264. https://doi.org/10.1016/j.compedu.2005.11.007
Dhal, P., & Azad, C. (2022). A comprehensive survey on feature selection in the various fields of machine learning. Applied Intelligence, 52(4), 4543–4581. https://doi.org/10.1007/s10489-021-02550-9
Dıaz-Ramırez, J. (2020). Gamification in engineering education—An empirical assessment on learning and game performance. Heliyon, 6(9), e04972. https://doi.org/10.1016/j.heliyon.2020.e04972
Dunn, O. J. (1964). Multiple comparisons using rank sums. Technometrics, 6(3), 241–252. https://doi.org/10.1080/00401706.1964.10490181
Elrahman, S. M. A., & Abraham, A. (2013). A review of class imbalance problem. Journal of Network and Innovative Computing, 1, 332–340. https://cspub-jnic.org/index.php/jnic/article/view/42
Emerson, A., Cloude, E. B., Azevedo, R., & Lester, J. (2020). Multimodal learning analytics for game-based learning. British Journal of Educational Technology, 51(5), 1505–1526. https://doi.org/10.1111/bjet.12992
Fadila, N. N., Saidah, K., & Wendha, D. D. N. (2023). Development of interactive multimedia based on educational games of plant parts and their functions to improve student learning outcomes. Jurnal Pijar Mipa, 18(5), 666–669. https://doi.org/10.29303/jpm.v18i5.5472
Fang, Y., Li, T., Huynh, L., Christhilf, K., Roscoe, R. D., & McNamara, D. S. (2023). Stealth literacy assessments via educational games. Computers, 12(7), 130. https://doi.org/10.3390/computers12070130
Feng, X., & Yamada, M. (2019). Effects of game-based learning on informal historical learning: A learning analytics approach. In M. Chang, H. - J. So, L.-H. Wong, J.-L. Shih, & F.- Y. Yu (Eds.), Proceedings of the 27th International Conference on Computers in Education (ICCE 2019), 2–6 December 2019, Kenting, Taiwan (pp. 505–514). APSCE. https://doi.org/10.58459/icce.2019.496
Fisher, R. A. (1935). The logic of inductive inference. Journal of the Royal Statistical Society, 98(1), 39–82. https://www.jstor.org/stable/2342435
Fogarty, J. A. (2006). Constructing and evaluating sensor-based statistical models of human interruptibility [Doctoral dissertation, Carnegie Mellon University] [AAI3241594]. http://reports-archive.adm.cs.cmu.edu/anon/usr/ftp/hcii/CMU-HCII-06-100.pdf
Fokides, E., Atsikpasi, P., Kaimara, P., & Deliyannis, I. (2019). Factors influencing the subjective learning effectiveness of serious games. Journal of Information Technology Education: Research, 18, 437–466. https://doi.org/10.28945/4441
Gee, J. P. (2003). What video games have to teach us about learning and literacy. Computers in Entertainment (CIE), 1(1), 20. https://doi.org/10.1145/950566.950595
Georgiadis, K., van Lankveld, G., Bahreini, K., & Westera, W. (2019). Learning analytics should analyse the learning: Proposing a generic stealth assessment tool. In Proceedings of the 2019 IEEE Conference on Games (CoG), 20–23 August 2019, London, UK (pp. 1–8). IEEE. https://doi.org/10.1109/CIG.2019.8847960
Georgiadis, K., van Lankveld, G., Bahreini, K., & Westera, W. (2021). On the robustness of stealth assessment. IEEE Transactions on Games, 13(2), 180–192. https://doi.org/10.1109/TG.2020.3020015
Gibson, D., & Clarke-Midura, J. (2015). Some psychometric and design implications of game-based learning analytics. In P. Isaıas, J. M. Spector, D. Ifenthaler, & D. G. Sampson (Eds.), E-learning systems, environments and approaches: Theory and implementation (pp. 247–261). Springer International Publishing. https://doi.org/10.1007/978-3-319-05825-2 17
Glass, G. V. (1966). Note on rank biserial correlation. Educational and Psychological Measurement, 26(3), 623–631. https://doi.org/10.1177/001316446602600307
Gobel, S., de Carvalho Rodrigues, A., Mehm, F., & Steinmetz, R. (2009). Narrative game-based learning objects for story-based digital educational games. Narrative, 14, 16. https://www.kom.tu-darmstadt.de/papers/GdMS09 533.pdf
Gobel, S., & Mehm, F. (2013). Personalized, adaptive digital educational games using narrative game-based learning objects. In K. Bredl & W. Bosche (Eds.), Serious games and virtual worlds in education, professional development, and healthcare (pp. 74–84). IGI Global. https://doi.org/10.4018/978-1-4666-3673-6.ch005
Goggins, S. P., Gallagher, M., Laffey, J., & Amelung, C. (2010). Social intelligence in completely online groups—toward social prosthetics from log data analysis and transformation. In Proceedings of the 2010 IEEE Second International Conference on Social Computing (SocialCom 2010), 20–22 August 2010, Minneapolis, Minnesota, USA (pp. 500–507). IEEE. https://doi.org/10.1109/SocialCom.2010.79
Goggins, S. P., Galyen, K., Petakovic, E., & Laffey, J. M. (2016). Connecting performance to social structure and pedagogy as a pathway to scaling learning analytics in MOOCs: An exploratory study. Journal of Computer Assisted Learning, 32(3), 244–266. https://doi.org/10.1111/jcal.12129
Goggins, S. P., Laffey, J., & Amelung, C. (2011). Context aware CSCL: Moving toward contextualized analysis. In H. Spada, G. Stahl, N. Miyake, & N. Law (Eds.), Connecting Computer-Supported Collaborative Learning to Policy and Practice: CSCL2011 Conference Proceedings. Volume II—Short Papers & Posters, 4–8 July 2011, Hong Kong, China. International Society of the Learning Sciences. https://doi.org/10.22318/cscl2011.591
Gomez, M. J., Ruiperez-Valiente, J. A., & Garcıa Clemente, F. J. (2023). A framework to support interoperable game-based assessments as a service (GBAaaS): Design, development, and use cases. Software: Practice and Experience, 53(11), 2222–2240. https://doi.org/10.1002/spe.3254
Grover, S., Basu, S., Bienkowski, M., Eagle, M., Diana, N., & Stamper, J. (2017). A framework for using hypothesis-driven approaches to support data-driven learning analytics in measuring computational thinking in block-based programming environments. ACM Transactions on Computing Education, 17(3). https://doi.org/10.1145/3105910
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., & Pedreschi, D. (2018). A survey of methods for explaining black box models. ACM Computing Surveys, 51(5). https://doi.org/10.1145/3236009
Gunter, G., Kenny, R. F., & Vick, E. H. (2006). A case for a formal design paradigm for serious games. The Journal of the International Digital Media and Arts Association, 3(1), 93–105. https://www.academia.edu/29030853/A_Case_ for_a_Formal_Design_Paradigm_for_Serious_Games_iDMAa_and_IMS_conference
Guo, X., Yin, Y., Dong, C., Yang, G., & Zhou, G. (2008). On the class imbalance problem. In M. Guo, L. Zhao, & L. Wang (Eds.), Proceedings of the 2008 Fourth International Conference on Natural Computation (ICNC), 18–20 October 2008, Jinan, China (pp. 192–201, Vol. 4). IEEE. https://doi.org/10.1109/ICNC.2008.871
Gupta, A., Carpenter, D., Min, W., Rowe, J. P., Azevedo, R., & Lester, J. C. (2021). Multimodal multi-task stealth assessment for reflection-enriched game-based learning. In Multimodal Artificial Intelligence in Education at the 22nd International Conference on Artificial Intelligence in Education (MAIED@ AIED), 14 June 2021, Utrecht, Netherlands (pp. 93–102). CEUR. https://ceur-ws.org/Vol-2902/paper9.pdf
Guyon, I., & Elisseeff, A. (2003). An introduction to variable and feature selection (L. P. Kaelbling, Ed.). Journal of Machine Learning Research, 3, 1157–1182. https://dl.acm.org/doi/10.5555/944919.944968
Haas, L., & Tussey, J. (2022). Equity and engagement through digital storytelling and game-based learning. In Research anthology on developments in gamification and game-based learning (pp. 1451–1472). IGI Global. https://doi.org/10.4018/978-1-7998-5770-9.ch013
Hauge, J. B., Berta, R., Fiucci, G., Manjon, B. F., Padron-Napoles, C., Westra, W., & Nadolski, R. (2014). Implications of learning analytics for serious game design. In Proceedings of the 2014 IEEE 14th International Conference on Advanced Learning Technologies (ICALT 2014), 7–10 July 2014, Athens, Greece (pp. 230–232). IEEE. https://doi.org/10.1109/ICALT.2014.73
Heine, C. (2021). Towards modeling visualization processes as dynamic Bayesian networks. IEEE Transactions on Visualization and Computer Graphics, 27(2), 1000–1010. https://doi.org/10.1109/TVCG.2020.3030395
Heinemann, B., Ehlenz, M., Gorzen, S., & Schroeder, U. (2022). xAPI made easy: A learning analytics infrastructure for interdisciplinary projects. International Journal of Online & Biomedical Engineering, 18(14). https://doi.org/10.3991/ijoe.v18i14.35079
Henderson, N., Acosta, H., Min, W., Mott, B., Lord, T., Reichsman, F., Dorsey, C., Wiebe, E., & Lester, J. (2022). Enhancing stealth assessment in game-based learning environments with generative zero-shot learning. In A. Mitrovic & N. Bosch (Eds.), Proceedings of the 15th International Conference on Educational Data Mining (EDM 2022), 24–27 July 2022, Durham, UK (pp. 171–182). International Educational Data Mining Society. https://educationaldatamining.org/edm2022/proceedings/2022.EDM-long-papers.15/index.html
Henderson, N., Kumaran, V., Min, W., Mott, B., Wu, Z., Boulden, D., Lord, T., Reichsman, F., Dorsey, C., Wiebe, E., & Lester, J. (2020). Enhancing student competency models for game-based learning with a hybrid stealth assessment framework. In A. N. Rafferty, J. Whitehill, C. Romero, & V. Cavalli-Sforza (Eds.), Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020), 10–13 July 2020, online (pp. 92–103). International Educational Data Mining Society. https://educationaldatamining.org/files/conferences/EDM2020/papers/paper 158.pdf
Hicks, D. (2021). Stealth assessment: Teacher’s perceptions of how digital-based educational games influence teaching and learning in the middle school mathematics classroom [Doctoral dissertation, Piedmont College]. https://www.proquest.com/openview/a10b0dc17805485238724aa3376201b3/1
Hooshyar, D., Ahmad, R. B., Yousefi, M., Fathi, M., Horng, S.-J., & Lim, H. (2016). Applying an online game-based formative assessment in a flowchart-based intelligent tutoring system for improving problem-solving skills. Computers & Education, 94, 18–36. https://doi.org/10.1016/j.compedu.2015.10.013
Huang, Y.- M., & Du, S. - X. (2005). Weighted support vector machine for classification with uneven training class sizes. In Proceedings of the 2005 International Conference on Machine Learning and Cybernetics (ICMLC 2005), 18–21 August 2005, Guangzhou, China (pp. 4365–4369, Vol. 7). IEEE. https://doi.org/10.1109/ICMLC.2005.1527706
Jalota, C., & Agrawal, R. (2021). Feature selection algorithms and student academic performance: A study. In D. Gupta, A. Khanna, S. Bhattacharyya, A. E. Hassanien, S. Anand, & A. Jaiswal (Eds.), International conference on innovative computing and communications. Advances in intelligent systems and computing (pp. 317–328, Vol. 1165). Springer. https://doi.org/10.1007/978-981-15-5113-0 23
Jeon, H., He, H., Wang, A., & Spooner, S. (2023). Modeling student performance in game-based learning environments. arXiv preprint arXiv:2309.13429. https://arxiv.org/abs/2309.13429
Kang, J., Liu, M., & Qu, W. (2017). Using gameplay data to examine learning behavior patterns in a serious game. Computers in Human Behavior, 72, 757–770. https://doi.org/10.1016/j.chb.2016.09.062
Kaur, H., Pannu, H. S., & Malhi, A. K. (2019). A systematic review on imbalanced data challenges in machine learning: Applications and solutions. ACM Computing Surveys (CSUR), 52(4). https://doi.org/10.1145/3343440
Ke, F., Parajuli, B., & Smith, D. (2019). Assessing game-based mathematics learning in action. In D. Ifenthaler & Y. J. Kim (Eds.), Game-based assessment revisited (pp. 213–227). Springer International Publishing. https://doi.org/10.1007/978-3-030-15569-8_11
Ke, F., & Shute, V. (2015). Design of game-based stealth assessment and learning support. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement (pp. 301-318). Springer International Publishing. https://doi.org/10.1007/978-3-319-05834-4 13
Keerin, P. (2021). A comparative study of missing value imputation methods for education data. In Proceedings of the 29th International Conference on Computers in Education (ICCE 2021), 22–26 November 2021, online. Asia-Pacific Society for Computers in Education. https://library.apsce.net/index.php/ICCE/article/view/4233
Kim, Y. J., Ruiperez-Valiente, J. A., Philip, T., Louisa, R., & Klopfer, E. (2019). Towards a process to integrate learning analytics and evidence-centered design for game-based assessment. In Companion Proceedings of the Ninth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, Arizona, USA (pp. 204–205). SoLAR. https://www.solaresearch.org/wp-content/uploads/2019/08/LAK19_Companion_Proceedings.pdf
Kruskal, W. H., & Wallis, W. A. (1952). Use of ranks in one-criterion variance analysis. Journal of the American Statistical Association, 47(260), 583–621. https://doi.org/10.1080/01621459.1952.10483441
Kuhn, M. (2019). The caret package. https://topepo.github.io/caret/available-models.html
Laffey, J. M., Griffin, J., Sigoloff, J., Lander, S., Sadler, T., Goggins, S., Kim, S. M., Wulff, E., & Womack, A. J. (2017). Mission HydroSci: A progress report on a transformational role playing game for science learning. In Proceedings of the 12th International Conference on the Foundations of Digital Games (FDG 2017), 14–17 August 2017, Hyannis, Massachusetts, USA. ACM. https://doi.org/10.1145/3102071.3106354
Lee, J. Y., Donkers, J., Jarodzka, H., & van Merrienboer, J. J. (2019). How prior knowledge affects problem-solving performance in a medical simulation game: Using game-logs and eye-tracking. Computers in Human Behavior, 99, 268–277. https://doi.org/10.1016/j.chb.2019.05.035
Li, F.-Y., Hwang, G.- J., Chen, P. -Y., & Lin, Y.-J. (2021). Effects of a concept mapping-based two-tier test strategy on students’ digital game-based learning performances and behavioral patterns. Computers & Education, 173, 104293. https://doi.org/10.1016/j.compedu.2021.104293
Li, N., Kidzinski, Ł., Jermann, P., & Dillenbourg, P. (2015). MOOC video interaction patterns: What do they tell us? In
G. Conole, T. Klobucar, C. Rensing, J. Konert, & E. Lavoue (Eds.), Design for teaching and learning in a networked world. EC-TEL 2015. Lecture notes in computer science (pp. 197–210, Vol. 9307). Springer International Publishing. https://doi.org/10.1007/978-3-319-24258-3_15
Lin, Y. -C., Hsieh, Y. - H., Hou, H.- T., & Wang, S.-M. (2019). Exploring students’ learning and gaming performance as well as attention through a drill-based gaming experience for environmental education. Journal of Computers in Education, 6(3), 315–334. https://doi.org/10.1007/s40692-019-00130-y
Liu, M., Cai, Y., Han, S., & Shao, P. (2022). Understanding student navigation patterns in game-based learning. Journal of Learning Analytics, 9(3), 50–74. https://doi.org/10.18608/jla.2022.7637
Loh, C. S., Li, I.- H., & Sheng, Y. (2016). Comparison of similarity measures to differentiate players’ actions and decision making profiles in serious games analytics. Computers in Human Behavior, 64, 562–574. https://doi.org/10.1016/j.chb.2016.07.024
Loh, C. S., & Sheng, Y. (2015). Measuring the (dis-) similarity between expert and novice behaviors as serious games analytics. Education and Information Technologies, 20(1), 5–19. https://doi.org/10.1007/s10639-013-9263-y
Loh, C. S., Sheng, Y., & Ifenthaler, D. (2015). Serious games analytics: Theoretical framework. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement (pp. 3–29). Springer International Publishing. https://doi.org/10.1007/978-3-319-05834-4_1
Lu, W., Griffin, J., Sadler, T. D., Laffey, J., & Goggins, S. P. (2023). Serious game analytics by design: Feature generation and selection using game telemetry and game metrics—toward predictive model construction. Journal of Learning Analytics, 10(1), 168–188. https://doi.org/10.18608/jla.2023.7681
Lundberg, S., & Lee, S. -I. (2017). A unified approach to interpreting model predictions. arXiv preprint arXiv:1705.07874. https://arxiv.org/abs/1705.07874
Ma, J., Han, X., Yang, J., & Cheng, J. (2015). Examining the necessary condition for engagement in an online learning environment based on learning analytics approach: The role of the instructor. The Internet and Higher Education, 24, 26–34. https://doi.org/10.1016/j.iheduc.2014.09.005
Mann, H. B., & Whitney, D. R. (1947). On a test of whether one of two random variables is stochastically larger than the other. The Annals of Mathematical Statistics, 18(1), 50–60. https://www.jstor.org/stable/2236101
Manzano-Leon, A., Camacho-Lazarraga, P., Guerrero, M. A., Guerrero-Puerta, L., Aguilar-Parra, J. M., Trigueros, R., & Alias,
A. (2021). Between level up and game over: A systematic literature review of gamification in education. Sustainability, 13(4), 2247. https://doi.org/10.3390/su13042247
Marron, J. S., Todd, M. J., & Ahn, J. (2007). Distance-weighted discrimination. Journal of the American Statistical Association, 102(480), 1267–1271. https://doi.org/10.1198/016214507000001120
Martınez-Mones, A., Harrer, A., & Dimitriadis, Y. (2011). An interaction-aware design process for the integration of interaction analysis into mainstream CSCL practices. In S. Puntambekar, G. Erkens, & C. Hmelo-Silver (Eds.), Analyzing interactions in CSCL: Methods, approaches and issues (pp. 269–291). Springer US. https://doi.org/10.1007/978-1-4419-7710-6_13
Maryani, I., & Hidayat, N. (2019). Interactive game: A step to reduce science learning difficulties of elementary school students. In R. N. D. Irmawati, D. Sulisworo, E. Arroyo, I. Tokoro, R. C. I. Prahmana, & A. Azhari (Eds.), Proceedings of the First International Conference on Progressive Civil Society (ICONPROCS 2019), 19 February 2019, Yogyakarta, Indonesia (pp. 100–103). Atlantis Press. https://doi.org/10.2991/iconprocs-19.2019.20
Min, W., Frankosky, M. H., Mott, B. W., Rowe, J. P., Smith, A., Wiebe, E., Boyer, K. E., & Lester, J. C. (2020). DeepStealth: Game-based learning stealth assessment with deep neural networks. IEEE Transactions on Learning Technologies, 13(2), 312–325. https://doi.org/10.1109/TLT.2019.2922356
Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence-centered design. ETS Research Report Series, 2003(1), i–29. https://doi.org/10.1002/j.2333-8504.2003.tb01908.x
Molnar, C., Casalicchio, G., & Bischl, B. (2020). Interpretable machine learning—a brief history, state-of-the-art and challenges. In I. Koprinska, M. Kamp, A. Appice, C. Loglisci, L. Antonie, A. Zimmermann, R. Guidotti, O. Ozgobek, R. P. Ribeiro, R. Gavalda, J. Gama, L. Adilova, Y. Krishnamurthy, P. M. Ferreira, D. Malerba, I. Medeiros, M. Ceci, G. Manco, E. Masciari, . . . J. A. Gulla (Eds.), ECML PKDD 2020 workshops. ECML PKDD 2020. Communications in computer and information science (pp. 417–431, Vol. 1323). Springer International Publishing. https://doi.org/10.1007/978-3-030-65965-3_28
Moore, G. R., & Shute, V. J. (2017). Improving learning through stealth assessment of conscientiousness. In A. Marcus-Quinn & T. Hourigan (Eds.), Handbook on digital learning for K–12 schools (pp. 355–368). Springer International Publishing. https://doi.org/10.1007/978-3-319-33808-8 21
Mouri, K., Okubo, F., Shimada, A., & Ogata, H. (2016, July). Bayesian network for predicting students’ final grade using e-book logs in university education. In J. M. Spector, C.- C. Tsai, D. G. Sampson, Kinshuk, R. Huang, N. - S. Chen, & P. Resta (Eds.), Proceedings of the 2016 IEEE 16th International Conference on Advanced Learning Technologies (ICALT 2016), 25–28 July 2018, Austin, Texas, USA (pp. 85–89). IEEE. https://doi.org/10.1109/ICALT.2016.27
Nguyen, H. A., Hou, X., Stamper, J., & McLaren, B. M. (2020). Moving beyond test scores: Analyzing the effectiveness of a digital learning game through learning analytics. In A. N. Rafferty, J. Whitehill, C. Romero, & V. Cavalli-Sforza (Eds.), Proceedings of the 13th International Conference on Educational Data Mining (EDM 2020), 10–13 July 2020, online. International Educational Data Mining Society. https://educationaldatamining.org/files/conferences/EDM2020/papers/paper_182.pdf
Niemela, M., Karkkainen, T., Ayramo, S., Ronimus, M., Richardson, U., & Lyytinen, H. (2020). Game learning analytics for understanding reading skills in transparent writing system. British Journal of Educational Technology, 51(6), 2376–2390. https://doi.org/10.1111/bjet.12916
Nietfeld, J. L. (2020). Predicting transfer from a game-based learning environment. Computers & Education, 146, 103780. https://doi.org/10.1016/j.compedu.2019.103780
Owen, V. E., & Baker, R. S. (2020). Fueling prediction of player decisions: Foundations of feature engineering for optimized behavior modeling in serious games. Technology, Knowledge and Learning, 25(2), 225–250. https://doi.org/10.1007/s10758-018-9393-9
Park, H.-S., & Cho, S.- B. (2010). Building mobile social network with semantic relation using Bayesian NeTwork-based Life-log Mining. In Proceedings of the 2010 IEEE Second International Conference on Social Computing (SocialCom 2010), 20–22 August 2010, Minneapolis, Minnesota, USA (pp. 401–406). IEEE. https://doi.org/10.1109/SocialCom.2010.64
Perez-Colado, I. J., Perez-Colado, V. M., Martinez-Ortiz, I., Freire, M., & Fernandez-Manjon, B. (2022). Using e-learning standards to improve serious game deployment and evaluation. In I. Kallel, H. M. Kammoun, & L. Hsairi (Eds.), Proceedings of the 2022 IEEE Global Engineering Education Conference (EDUCON 2022), 28–31 March 2022, Tunis, Tunisia (pp. 2077–2083). IEEE. https://doi.org/10.1109/EDUCON52537.2022.9766573
Rahimi, M., Moradi, H., Vahabie, A. - h., & Kebriaei, H. (2023). Continuous reinforcement learning-based dynamic difficulty adjustment in a visual working memory game. arXiv preprint arXiv:2308.12726. https://arxiv.org/abs/2308.12726
Ray, P., Reddy, S. S., & Banerjee, T. (2021). Various dimension reduction techniques for high dimensional data analysis: A review. Artificial Intelligence Review, 54(5), 3473–3515. https://doi.org/10.1007/s10462-020-09928-0
Reeves, T., Romine, W., Laffey, J., Sadler, T., & Goggins, S. (2020). Distance learning through game-based 3D virtual learning environments: Mission Hydro Science. Evaluation report for Mission HydroSci. https://files.eric.ed.gov/fulltext/ED605283.pdf
Ribeiro, M. T., Singh, S., & Guestrin, C. (2016). “Why should I trust you?”: Explaining the predictions of any classifier. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 2016), 13–17 August 2016, San Francisco, California, USA (pp. 1135–1144). ACM. https://doi.org/10.1145/2939672.2939778
Rohmani, R., & Pambudi, N. (2023). A critical review of educational games as a tool for strengthening digital literacy. International Journal of Multidisciplinary: Applied Business and Education Research, 4(5), 1483–1493. https://doi.org/10.11594/ijmaber.04.05.10
Romero, M., Usart, M., & Ott, M. (2015). Can serious games contribute to developing and sustaining 21st century skills? Games and Culture, 10(2), 148–177. https://doi.org/10.1177/1555412014548919
Romine, W. L., Sadler, T. D., & Wulff, E. P. (2017). Conceptualizing student affect for science and technology at the middle school level: Development and implementation of a measure of affect in science and technology (MAST). Journal of Science Education and Technology, 26(5), 534–545. https://doi.org/10.1007/s10956-017-9697-x
Rosydiana, E. A., Sudjimat, D. A., & Utama, C. (2023). The effect of digital learning media using scratch game based learning on student problem solving skills. Jurnal Penelitian Pendidikan IPA, 9(11), 10010–10015. https://doi.org/10.29303/jppipa.v9i11.4876
Rowe, E., Asbell-Clarke, J., Baker, R. S., Eagle, M., Hicks, A. G., Barnes, T. M., Brown, R. A., & Edwards, T. (2017). Assessing implicit science learning in digital games. Computers in Human Behavior, 76, 617–630. https://doi.org/10.1016/j.chb.2017.03.043
Rowe, J. P., McQuiggan, S. W., Robison, J. L., & Lester, J. C. (2009). Off-task behavior in narrative-centered learning environments. In V. Dimitrova, R. Mizoguchi, B. du Boulay, & A. Graesser (Eds.), Frontiers in artificial intelligence and applications. Ebook Volume 200: Artificial intelligence in education (pp. 99–106). IOS Press. https://doi.org/10.3233/978-1-60750-028-5-99
Sabourin, J. L., Shores, L. R., Mott, B. W., & Lester, J. C. (2013). Understanding and predicting student self-regulated learning strategies in game-based learning environments. International Journal of Artificial Intelligence in Education, 23(1), 94–114. https://doi.org/10.1007/s40593-013-0004-6
Sao Pedro, M. A., Baker, R. S. J. d., & Gobert, J. D. (2012). Improving construct validity yields better models of systematic inquiry even with less information. In D. Hutchison, T. Kanade, J. Kittler, J. M. Kleinberg, F. Mattern, J. C. Mitchell, M. Naor, O. Nierstrasz, C. Pandu Rangan, B. Steffen, M. Sudan, D. Terzopoulos, D. Tygar, M. Y. Vardi, G. Weikum, J. Masthoff, B. Mobasher, M. C. Desmarais, & R. Nkambou (Eds.), User modeling, adaptation, and personalization. UMAP 2012. Lecture notes in computer science (pp. 249–260, Vol. 7379). Springer. https://doi.org/10.1007/978-3-642-31454-4_21
Schardosim Simao, J. P., Mellos Carlos, L., Saliah-Hassane, H., da Silva, J. B., & da Mota Alves, J. B. (2018). Model for recording learning experience data from remote laboratories using xAPI. In Proceedings of the 2018 XIII Latin American Conference on Learning Technologies (LACLO 2018), 1–5 October 2018, Sao Paulo, Brazil (pp. 450–457). IEEE. https://doi.org/10.1109/LACLO.2018.00081
Seaton, J. X., Chang, M., & Graf, S. (2019). Integrating a learning analytics dashboard in an online educational game. In A. Tlili & M. Chang (Eds.), Data analytics approaches in educational games and gamification systems (pp. 127–138). Springer Singapore. https://doi.org/10.1007/978-981-32-9335-9_7
Serrano-Laguna, A., Martınez-Ortiz, I., Haag, J., Regan, D., Johnson, A., & Fernandez-Manjon, B. (2017). Applying standards to systematize learning analytics in serious games. Computer Standards & Interfaces, 50, 116–123. https://doi.org/10.1016/j.csi.2016.09.014
Serrano-Laguna, A., Torrente, J., Moreno-Ger, P., & Fernandez-Manjon, B. (2014). Application of learning analytics in educational videogames. Entertainment Computing, 5(4), 313–322. https://doi.org/10.1016/j.entcom.2014.02.003
Sevcenko, N., Ninaus, M., Wortha, F., Moeller, K., & Gerjets, P. (2021). Measuring cognitive load using in-game metrics of a serious simulation game. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.572437
Seyderhelm, A. J., & Blackmore, K. L. (2023). How hard is it really? Assessing game-task difficulty through real-time measures of performance and cognitive load. Simulation & Gaming, 54(3), 294–321. https://doi.org/10.1177/10468781231169910
Shohel, M. M. C., Ashrafuzzaman, M., Naomee, I., Tanni, S. A., & Azim, F. (2022). Game-based teaching and learning in higher education: Challenges and prospects. In C.-A. Lane (Ed.), Handbook of research on acquiring 21st century literacy skills through game-based learning (pp. 78–106). IGI Global. https://doi.org/10.4018/978-1-7998-7271-9.ch005
Shoukry, L., Gobel, S., & Steinmetz, R. (2014). Learning analytics and serious games: Trends and considerations. In Proceedings of the 2014 ACM International Workshop on Serious Games (SeriousGames 2014), 7 November 2014, Orlando, Florida, USA (pp. 21–26). ACM. https://doi.org/10.1145/2656719.2656729
Shute, V., Ke, F., & Wang, L. (2017). Assessment and adaptation in games. In P. Wouters & H. van Oostendorp (Eds.), Instructional techniques to facilitate learning and motivation of serious games (pp. 59–78). Springer International Publishing. https://doi.org/10.1007/978-3-319-39298-1 4
Shute, V., & Ventura, M. (2013). Stealth assessment: Measuring and supporting learning in video games. The MIT Press. https://doi.org/10.7551/mitpress/9589.001.0001
Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795
Shute, V. J. (2011a). Stealth assessment in computer-based games to support learning. In S. Tobias & J. D. Fletcher (Eds.), Computer games and instruction (pp. 503–523). Information Age Publishing.
Shute, V. J. (2011b). Stealth assessment in computer-based games to support learning. In S. Tobias & J. D. Fletcher (Eds.), Computer games and instruction (pp. 503–524, Vol. 55). IAP Information Age Publishing. https://psycnet.apa.org/record/2011-11269-020
Shute, V. J., & Rahimi, S. (2021). Stealth assessment of creativity in a physics video game. Computers in Human Behavior, 116, 106647. https://doi.org/10.1016/j.chb.2020.106647
Shute, V. J., Wang, L., Greiff, S., Zhao, W., & Moore, G. (2016). Measuring problem solving skills via stealth assessment in an engaging video game. Computers in Human Behavior, 63, 106–117. https://doi.org/10.1016/j.chb.2016.05.047
Siddique, A., Jan, A., Majeed, F., Qahmash, A. I., Quadri, N. N., & Wahab, M. O. A. (2021). Predicting academic performance using an efficient model based on fusion of classifiers. Applied Sciences, 11(24), 11845. https://doi.org/10.3390/app112411845
Smith, G., Shute, V., & Muenzenberger, A. (2019). Designing and validating a stealth assessment for calculus competencies. Journal of Applied Testing Technology, 20(S1), 52–59. https://www.jattjournal.net/index.php/atp/article/view/142702
Snow, E. L., Allen, L. K., & McNamara, D. S. (2015). The dynamical analysis of log data within educational games. In C. S. Loh, Y. Sheng, & D. Ifenthaler (Eds.), Serious games analytics: Methodologies for performance measurement, assessment, and improvement (pp. 81–100). Springer International Publishing. https://doi.org/10.1007/978-3-319-05834-4 4
Strukova, S., Ruiperez-Valiente, J. A., & Marmol, F. G. (2023). Adapting knowledge inference algorithms to measure geometry competencies through a puzzle game. ACM Transactions on Knowledge Discovery from Data, 18(1). https://doi.org/10.1145/3614436
Student. (1908). The probable error of a mean. Biometrika, 6(1), 1–25. https://doi.org/10.2307/2331554
Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53–55. https://doi.org/10.5116/ijme.4dfb.8dfd
Udeozor, C., Abegao, F. R., & Glassey, J. (2024). Measuring learning in digital games: Applying a game-based assessment framework. British Journal of Educational Technology, 55(3), 957–991. https://doi.org/10.1111/bjet.13407
Ventura, M., & Shute, V. (2013). The validity of a game-based assessment of persistence. Computers in Human Behavior, 29(6), 2568–2572. https://doi.org/10.1016/j.chb.2013.06.033
Vidakis, N., Barianos, A. K., Trampas, A. M., Papadakis, S., Kalogiannakis, M., & Vassilakis, K. (2020). In-game raw data collection and visualization in the context of the “ThimelEdu” educational game. In H. C. Lane, S. Zvacek, & J. Uhomoibhi (Eds.), Computer supported education. CSEDU 2019. Communications in computer and information science (pp. 629–646, Vol. 1220). Springer International Publishing. https://doi.org/10.1007/978-3-030-58459-7 30
Wang, L. H., Chen, B., Hwang, G. J., Guan, J. Q., & Wang, Y. Q. (2022). Effects of digital game-based STEM education on students’ learning achievement: A meta-analysis. International Journal of STEM Education, 9(1), 26. https://doi.org/10.1186/s40594-022-00344-0
Wang, L., Shute, V., & Moore, G. R. (2015). Lessons learned and best practices of stealth assessment. International Journal of Gaming and Computer-Mediated Simulations (IJGCMS), 7(4), 66–87. https://doi.org/10.4018/IJGCMS.2015100104
Wang, M., & Zheng, X. (2021). Using game-based learning to support learning science: A study with middle school students. The Asia-Pacific Education Researcher, 30(2), 167–176. https://doi.org/10.1007/s40299-020-00523-z
Wen, C.-T., Chang, C. -J., Chang, M.- H., Fan Chiang, S.-H., Liu, C.- C., Hwang, F.- K., & Tsai, C.- C. (2018). The learning analytics of model-based learning facilitated by a problem-solving simulation game. Instructional Science, 46(6), 847–867. https://doi.org/10.1007/s11251-018-9461-5
Westera, W., Nadolski, R., & Hummel, H. (2014). Serious gaming analytics: What students’ log files tell us about gaming and learning. International Journal of Serious Games, 1(2), 35–50. https://journal.seriousgamessociety.org/index.php/IJSG/article/view/9/pdf 56
Widaman, K. F., & Helm, J. L. (2023). Exploratory factor analysis and confirmatory factor analysis. In H. Cooper, M. N. Coutanche, L. M. McMullen, A. T. Panter, D. Rindskopf, & K. J. Sher (Eds.), APA handbook of research methods in psychology: Data analysis and research publication (2nd edition, pp. 379–410). American Psychological Association. https://doi.org/10.1037/0000320-017
Wongvorachan, T., He, S., & Bulut, O. (2023). A comparison of undersampling, oversampling, and SMOTE methods for dealing with imbalanced classification in educational data mining. Information, 14(1), 54. https://doi.org/10.3390/info14010054
Xing, W., Wadholm, B., & Goggins, S. (2014). Learning analytics in CSCL with a focus on assessment: An exploratory study
of activity theory-informed cluster analysis. In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (LAK 2014), 24–28 March 2014, Indianapolis, Indiana, USA (pp. 59–67). ACM. https://doi.org/10.1145/2567574.2567587
Yang, D., Zargar, E., Adams, A. M., Day, S. L., & Connor, C. M. (2021). Using interactive e-book user log variables to track reading processes and predict digital learning outcomes. Assessment for Effective Intervention, 46(4), 292–303. https://doi.org/10.1177/1534508420941935
Yeo, I.-K., & Johnson, R. A. (2000). A new family of power transformations to improve normality or symmetry. Biometrika, 87(4), 954–959. https://doi.org/10.1093/biomet/87.4.954
Yu, J., Ma, W., Moon, J., & Denham, A. (2022). Developing a stealth assessment system using a continuous conjunctive model. Journal of Learning Analytics, 9(3), 11–31. https://doi.org/10.18608/jla.2022.7639
Zhang, Q., & Rutherford, T. (2022). Grade 5 students’ elective replay after experiencing failures in learning fractions in an educational game: When does replay after failures benefit learning? In Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK 2022), 21–25 March 2022, online (pp. 98–106). ACM. https://doi.org/10.1145/3506860.3506873
Zheng, A., & Casari, A. (2018). Feature engineering for machine learning: Principles and techniques for data scientists (1st edition). O’Reilly Media. https://www.oreilly.com/library/view/feature-engineering-for/9781491953235/
Zhu, S., Guo, Q., & Yang, H. H. (2023). Beyond the traditional: A systematic review of digital game-based assessment for students’ knowledge, skills, and affections. Sustainability, 15(5), 4693. https://doi.org/10.3390/su15054693
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
