Serious Game Analytics by Design: Feature Generation and Selection Using Game Telemetry and Game Metrics

Toward Predictive Model Construction




serious games, learning analytics, game-based learning environment, logging system, trace data, educational data mining, performance measurement, research paper


The construction of prediction models reflecting players’ learning performance in serious games currently faces various challenges for learning analytics. In this study, we design, implement, and field test a learning analytics system for a serious game, advancing the field by explicitly showing which in-game features correspond to differences in learner performance. We then deploy and test a system that provides instructors with clear signals regarding student learning and progress in the game, which instructors could depend upon for interventions. Within the study, we examined, coded, and filtered a substantial gameplay corpus, determining expertise in the game. Mission HydroSci (MHS) is a serious game that teaches middle-school students water science. Using our logging system, designed and implemented along with game design and development, we captured around 60 in-game features from the gameplay of 373 students who completed Unit 3 of MHS in its first field test. We tested eight hypotheses during the field test and presented this paper’s results to participating teachers. Our findings reveal several features with statistical significance that will be critical for creating a validated prediction model. We discuss how this work will help future research establish a framework for designing analytics systems for serious games and advancing gaming design and analytics theory.


All, A., Nuñez Castellar, E. P., & Van Looy, J. (2015). Towards a conceptual framework for assessing the effectiveness of digital game-based learning. Computers & Education, 88, 29–37.

Alvarez, J., Plantec, J.-Y., Vermeulen, M., & Kolski, C. (2017). RDU Model dedicated to evaluate needed counsels for Serious Game projects. Computers & Education, 114, 38–56.

Barab, S. A., Gresalfi, M., & Ingram-Goble, A. (2010). Transformational play: Using games to position person, content, and context. Educational Researcher, 39(7), 525–536.

Barzilai, S., & Blau, I. (2014). Scaffolding game-based learning: Impact on learning achievements, perceived learning, and game experiences. Computers & Education, 70, 65–79.

Boyle, E. A., Hainey, T., Connolly, T. M., Gray, G., Earp, J., Ott, M., Lim, T., Ninaus, M., Ribeiro, C., & Pereira, J. (2016). An update to the systematic literature review of empirical evidence of the impacts and outcomes of computer games and serious games. Computers & Education, 94, 178–192.

Buckingham Shum, S., & Deakin Crick, R. (2016). Learning analytics for 21st century competencies. Journal of Learning Analytics, 3(2), 6–21.

Caballero-Hernández, J. A., Palomo-Duarte, M., & Dodero, J. M. (2017). Skill assessment in learning experiences based on serious games: A systematic mapping study. Computers & Education, 113, 42–60.

Carvalho, M. B., Bellotti, F., Berta, R., De Gloria, A., Sedano, C. I., Hauge, J. B., Hu, J., & Rauterberg, M. (2015). An activity theory-based model for serious games analysis and conceptual design. Computers & Education, 87, 166–181.

Chaves, R. O., von Wangenheim, C. G., Furtado, J. C. C., Oliveira, S. R. B., Santos, A., & Favero, E. L. (2015). Experimental evaluation of a serious game for teaching software process modeling. IEEE Transactions on Education, 58(4), 289–296.

Chittaro, L., & Buttussi, F. (2015). Assessing knowledge retention of an immersive serious game vs. a traditional education method in aviation safety. IEEE Transactions on Visualization and Computer Graphics, 21(4), 529–538.

Clarke-Midura, J., Rosenheck, L., & Groff, J. (2015). Designing games for learning and assessment: The radix endeavor. In Proceedings of the 14th International Conference on Interaction Design and Children (IDC 2015), 21–24 June 2015, Boston, MA, USA (pp. 343–346). ACM.

Cutumisu, M., Blair, K. P., Chin, D. B., & Schwartz, D. L. (2015). Posterlet: A game-based assessment of children’s choices to seek feedback and to revise. Journal of Learning Analytics, 2(1), 49–71.

Fernández, C. A. (2016). Gaming learning analytics for serious games [eprint], Universidad Complutense Madrid.

Fridenson-Hayo, S., Berggren, S., Lassalle, A., Tal, S., Pigat, D., Meir-Goren, N., O’Reilly, H., Ben-Zur, S., Bölte, S., Baron-Cohen, S., & Golan, O. (2017). “Emotiplay”: A serious game for learning about emotions in children with autism: Results of a cross-cultural evaluation. European Child & Adolescent Psychiatry, 26(8), 979–992.

Fulmore, Y. (2015). Video games and the customization of learning: Interactive narratives as a promising design framework for crafting inclusive educational environments. In Proceedings of the Third Conference on GenderIT (GenderIT 2015), 24 April 2015, Philadelphia, PA, USA (pp. 32–39). ACM.

Gauthier, A., & Jenkinson, J. (2018). Designing productively negative experiences with serious game mechanics: Qualitative analysis of game-play and game design in a randomized trial. Computers & Education, 127, 66–89.

Goggins, S. P., Gallagher, M., Laffey, J., & Amelung, C. (2010). Social intelligence in completely online groups—Toward social prosthetics from log data analysis and transformation. In IEEE Second International Conference on Social Computing (SocialCom 2010), 20–22 August 2010, Minneapolis, MN, USA (pp. 500–507). IEEE.

Goggins, S. P., Galyen, K., & Laffey, J. (2010). Network analysis of trace data for the support of group work: Activity patterns in a completely online course. In Proceedings of the 16th ACM International Conference on Supporting Group Work (GROUP 2010), 7–10 November 2010, Sanibel Island, FL, USA (pp. 107–116).

Goggins, S., Galyen, K., Petakovic, E., & Laffey, J. (2016). Connecting performance to social structure and pedagogy as a pathway to scaling learning analytics in MOOCs: An exploratory study. Journal of Computer Assisted Learning, 32(3), 244–266.

Graves, I., McDonald, N., & Goggins, S. P. (2016). Sifting signal from noise: A new perspective on the meaning of tweets about the “big game.” New Media & Society, 18(2), 293–312.

Hauge, J. B., Berta, R., Fiucci, G., Manjón, B. F., Padrón-Nápoles, C., Westra, W., & Nadolski, R. (2014). Implications of learning analytics for serious game design. In IEEE 14th International Conference on Advanced Learning Technologies (ICALT 2014), 7–9 July 2014, Athens, Greece (pp. 230–232). IEEE.

Kang, J., Liu, M., & Qu, W. (2017). Using gameplay data to examine learning behavior patterns in a serious game. Computers in Human Behavior, 72, 757–770.

Kapur, M. (2008). Productive failure. Cognition and Instruction, 26(3), 379–425.

Kiili, K., Moeller, K., & Ninaus, M. (2018). Evaluating the effectiveness of a game-based rational number training—In-game metrics as learning indicators. Computers & Education, 120, 13–28.

Laffey, J., Griffin, J., Sigoloff, J., Sadler, T., Goggins, S., Womack, A., Wulff, E., & Lander, S. (n.d.). Mission HydroSci: Meeting learning standards through gameplay [Unpublished manuscript]. College of Education & Human Development, University of Missouri.

Liu, M., Lee, J., Kang, J., & Liu, S. (2016). What we can learn from the data: A multiple-case study examining behavior patterns by students with different characteristics in using a serious game. Technology, Knowledge and Learning, 21(1), 33–57.

Loh, C. S., & Li, I.-H. (2015). Predicting the competency improvement for serious games analytics: Action-sequences, game grids, PLS DA and JMP [Paper presentation]. Discovery Summit 2015, San Diego, CA, USA.

Loh, C. S., Li, I.-H., & Sheng, Y. (2016). Comparison of similarity measures to differentiate players’ actions and decisionmaking profiles in serious games analytics. Computers in Human Behavior, 64, 562–574.

Loh, C. S., & Sheng, Y. (2014). Maximum similarity index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games. Computers in Human Behavior, 39, 322–330.

Loh, C. S., Sheng, Y., & Li, I.-H. (2015). Predicting expert–novice performance as serious games analytics with objectiveoriented and navigational action sequences. Computers in Human Behavior, 49, 147–155.

Martin, C. K., Nacu, D., & Pinkard, N. (2016). Revealing opportunities for 21st century learning: An approach to interpreting user trace log data. Journal of Learning Analytics, 3(2), 37–87.

Palmquist, S. D., & Crowley, K. (2007). Studying dinosaur learning on an island of expertise. In R. Goldman, R. Pea, B. Barron, & S. J. Derry (Eds.), Video research in the learning sciences (pp. 271–286). Routledge.

Renninger, K. A. (2012). Interest and its development. In R. M. Ryan (Ed.), The Oxford Handbook of Human Motivation (pp. 167–187). Oxford University Press.

Renninger, K. (2010). Working with and cultivating the development of interest, self-efficacy, and self-regulation. In D. D. Preiss & R. J. Sternberg (Eds.), Innovations in educational psychology: Perspectives on learning, teaching and human development (pp. 107–138). Springer.

Renninger, K., Cai, M., Lewis, M., Adams, M., & Ernst, K. (2011). Motivation and learning in an online, unmoderated, mathematics workshop for teachers. Educational Technology Research and Development, 59(2), 229–247.

Serrano-Laguna, A., Martínez-Ortiz, I., Haag, J., Regan, D., Johnson, A., & Fernandez-Manjon, B. (2017). Applying standards to systematize learning analytics in serious games. Computer Standards & Interfaces, 50, 116–123.

Shute, V., & Ventura, M. (2013). Stealth assessment: Measuring and supporting learning in video games. The MIT Press.

Towne, T. J., Boot, W. R., & Ericsson, K. A. (2016). Understanding the structure of skill through a detailed analysis of individuals’ performance on the Space Fortress game. Acta Psychologica, 169, 27–37.

van Leeuwen, A. (2015). Learning analytics to support teachers during synchronous CSCL: Balancing between overview and overload. Journal of Learning Analytics, 2(2), 138–162.

Ventura, M., & Shute, V. (2013). The validity of a game-based assessment of persistence. Computers in Human Behavior, 29(6), 2568–2572.




How to Cite

Lu, W., Griffin, J., Sadler, T. D., Laffey, J., & Goggins, S. P. (2023). Serious Game Analytics by Design: Feature Generation and Selection Using Game Telemetry and Game Metrics: Toward Predictive Model Construction. Journal of Learning Analytics, 10(1), 168-188.