The Scope of Multimodal Learning Analytics in K–8
A Systematic Review
DOI:
https://doi.org/10.18608/jla.2025.8505Keywords:
multimodal data, multimodal learning analytics, learning analytics, K-8, research paperAbstract
This scoping review aims to provide an overview of how multimodal learning analytics has been applied in K–8 research and offers methodological insights and recommendations to bridge the gap between theory and practice. We identified 14 peer-reviewed empirical studies published between 2011 and 2023 through searches in relevant databases and reviews of journals that publish multimodal learning analytics–related research and publications from leading researchers in the field of multimodal learning analytics. The results revealed that 1) the application of multimodal learning analytics in K–8 research is varied; 2) even though studies explicitly stated how multimodal data were collected and analyzed, they rarely mentioned the data fusion techniques and its process, and ethics and transparency; and 3) multimodal learning analytics has the potential to improve learning outcomes, but more guidance and shared understanding are needed to conduct sound research using multimodal learning analytics. Implications and recommendations are provided.
References
Abrahamson, D., Worsley, M., Pardos, Z. A., & Ou, L. (2022). Learning analytics of embodied design: Enhancing synergy. International Journal of Child–Computer Interaction, Article 100409. https://doi.org/10.1016/j.ijcci.2021.100409
Alwahaby, H., Cukurova, M., Papamitsiou, Z., & Giannakos, M. (2022). The evidence of impact and ethical considerations of multimodal learning analytics: A systematic literature review. In M. Giannakos, D. Spikol, D. Di Mitri, K. Sharma, X. Ochoa, & R. Hammad (Eds.), The multimodal learning analytics handbook (pp. 289–325). Springer. https://doi.org/10.1007/978-3-031-08076-0_12
Andrade, A. (2017). Understanding student learning trajectories using multimodal learning analytics within an embodied-interaction learning environment. In I. Molenaar, X. Ochoa, & S. Dawson (Eds.), Understanding, informing and improving learning with data: LAK’17 conference proceedings: The seventh international learning analytics & knowledge conference (pp. 70–79). ACM Press. http://dx.doi.org/10.1145/3027385.3027429
Andrade, A., Delandshere, G., & Danish, J. A. (2016). Using multimodal learning analytics to model student behaviour: A systematic analysis of epistemological framing. Journal of Learning Analytics, 3(2), 282–306. http://dx.doi.org/10.18608/jla.2016.32.14
Aslan, S., Agrawal, A., Alyuz, N., Chierichetti, R., Durham, L. M., Manuvinakurike, R., Okur, E., Sahay, S., Sharma, S., Sherry, J., Raffa, G., & Nachman, L. (2022). Exploring kid space in the wild: A preliminary study of multimodal and immersive collaborative play-based learning experiences. Educational Technology Research and Development, 70(1), 205–230. https://doi.org/10.1007/s11423-021-10072-x
Boehmke, B., & Greenwell, B. M. (2020). Hands-on machine learning with R. CRC Press. https://bradleyboehmke.github.io/HOML/
Beardsley, M., Martínez Moreno, J., Vujovic, M., Santos, P., & Hernández‐Leo, D. (2020). Enhancing consent forms to support participant decision making in multimodal learning data research. British Journal of Educational Technology, 51(5), 1631–1652. https://doi.org/10.1111/bjet.12983
Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220–238. http://dx.doi.org/10.18608/jla.2016.32.11
Camacho, V. L., de la Guía, E., Olivares, T., Flores, M. J., & Orozco-Barbosa, L. (2020). Data capture and multimodal learning analytics focused on engagement with a new wearable IoT approach. IEEE Transactions on Learning Technologies, 13(4), 704–717. https://doi.org/10.1109/TLT.2020.2999787
Chango, W., Lara, J. A., Cerezo, R., & Romero, C. (2022). A review on data fusion in multimodal learning analytics and educational data mining. WIREs Data Mining and Knowledge Discovery, 12(4), Article e1458. https://doi.org/10.1002/widm.1458
Chejara, P. (2020). Multimodal learning analytics for understanding collocated collaboration in authentic classroom settings. In M. Chang, D. G. Sampson, R. Huang, D. Hooshyar, N.-S. Chen, Kinshuk, & M. Pedaste (Eds.), Proceedings: IEEE 20th international conference on advanced learning technologies: ICALT 2020 (pp. 390–392). IEEE. https://doi.org/10.1109/ICALT49669.2020.00123
Crescenzi‐Lanna, L. (2020). Multimodal learning analytics research with young children: A systematic review. British Journal of Educational Technology, 51(5), 1485–1504. https://doi.org/10.1111/bjet.12959
Cukurova, M., Giannakos, M., & Martinez-Maldonado, R. (2020). The promise and challenges of multimodal learning analytics. British Journal of Educational Technology, 51(5), 1441–1449. https://doi.org/10.1111/bjet.13015
Cukurova, M., Kent, C., & Luckin, R. (2019). Artificial intelligence and multimodal data in the service of human decision-making: A case study in debate tutoring. British Journal of Educational Technology, 50(6), 3032–3046 https://doi.org/10.1111/bjet.12829
Cukurova, M., Luckin, R., & Baines, E. (2018). The significance of context for the emergence and implementation of research evidence: The case of collaborative problem-solving. Oxford Review of Education, 44(3), 322–337. https://doi.org/10.1080/03054985.2017.1389713
Di Mitri, D., Schneider, J., Specht, M., & Drachsler, H. (2018). From signals to knowledge: A conceptual model for multimodal learning analytics. Journal of Computer Assisted Learning, 34(4), 338–349. https://doi.org/10.1111/jcal.12288
Echeverria, V., Martinez-Maldonado, R., Buckingham Shum, S., Chiluiza, K., Granda, R., & Conati, C. (2018). Exploratory versus explanatory visual learning analytics: Driving teachers’ attention through educational data storytelling. Journal of Learning Analytics, 5(3), 72–97. http://dx.doi.org/10.18608/jla.2018.53.6
Emerson, A., Cloude, E. B., Azevedo, R., & Lester, J. (2020). Multimodal learning analytics for game‐based learning. British Journal of Educational Technology, 51(5), 1505–1526. https://doi.org/10.1111/bjet.12992
Eradze, M., Rodríguez Triana, M. J., & Laanpere, M. (2020). Context-aware multimodal learning analytics taxonomy. In M. Scheffel, V. Kovanović, N. Pinkwart, & K. Verbert (Eds.), Shaping the future of the field: Companion proceedings of 10th international conference learning analytics & knowledge LAK20 (pp. 533–538). SoLAR. https://www.solaresearch.org/wp-content/uploads/2020/06/LAK20_Companion_Proceedings.pdf
Giannakos, M. (2023). Theory and intermediate-level knowledge in multimodal learning analytics. In K. Bartimote, S. K. Howard, & D. Gašević (Eds.), Theory informing and arising from learning analytics (pp. 87–104). Springer. https://doi.org/10.1007/978-3-031-60571-0_6
Giannakos, M., & Cukurova, M. (2023). The role of learning theory in multimodal learning analytics. British Journal of Educational Technology, 54(5), 1246–1267. https://doi.org/10.1111/bjet.13320
Lee-Cultura, S., Sharma, K., & Giannakos, M. (2022). Children’s play and problem-solving in motion-based learning technologies using a multi-modal mixed methods approach. International Journal of Child–Computer Interaction, 31, Article 100355. https://doi.org/10.1016/j.ijcci.2021.100355
Martinez-Maldonado, R., Echeverria, V., Fernandez Nieto, G., & Buckingham Shum, S. (2020). From data to insights: A layered storytelling approach for multimodal learning analytics. In R. Bernhaupt, F. Mueller, D. Verweij, J. Andres, J. McGrenere, A. Cockburn, I. Avellino, A. Goguey, P. Bjørn, S. Zhao, B. P. Samson, & R. Kocielnik (Eds.), Proceedings of the 2020 CHI conference on human factors in computing systems (pp. 1–15). ACM Press. https://doi.org/10.1145/3313831.3376148
Min, W., Park, K., Wiggins, J., Mott, B., Wiebe, E., Boyer, K. E., & Lester, J. (2019). Predicting dialogue breakdown in conversational pedagogical agents with multimodal LSTMs. In S. Isotani, E. Millán, A. Ogan, P. Hastings, B. McLaren, & R. Luckin (Eds.), Artificial intelligence in education: 20th international conference, AIED 2019, Chicago, IL, USA, June 25–29, 2019, proceedings, part II (pp. 195–200). Springer. https://doi.org/10.1007/978-3-030-23207-8_37
Moon, J., Ke, F., Sokolikj, Z., & Dahlstrom-Hakki, I. (2022). Multimodal data fusion to track students’ distress during educational gameplay. Journal of Learning Analytics, 9(3), 75–87. https://doi.org/10.18608/jla.2022.7631
Mu, S., Cui, M., & Huang, X. (2020). Multimodal data fusion in learning analytics: A systematic review. Sensors, 20(23), Article 6856. https://doi.org/10.3390/s20236856
Munn, Z., Peters, M. D., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18(143). https://doi.org/10.1186/s12874-018-0611-x
Nasir, J., Kothiyal, A., Bruno, B., & Dillenbourg, P. (2021). Many are the ways to learn identifying multi-modal behavioural profiles of collaborative learning in constructivist activities. International Journal of Computer-Supported Collaborative Learning, 16(4), 485–523. https://doi.org/10.1007/s11412-021-09358-2
Ocak, C., & Caskurlu, S. (2024, April 14). In-service teachers’ perception of developing a critical lens toward technology integration [Roundtable presentation]. AERA Annual Meeting, Philadelphia, PA, United States.
Ocak, C., Caskurlu, S., & Dai, C.-P. (2023, October 15–19). Developing a checklist as a guide to conduct multimodal learning analytics research [Paper presentation]. AECT International Convention, Orlando, FL, United States.
Ocak, C., Kopcha, T. J., & Dey, R. (2023). An AI-enhanced pattern recognition approach to temporal and spatial analysis of children’s embodied interactions. Computers and Education: Artificial Intelligence, Article 100146. https://doi.org/10.1016/j.caeai.2023.100146
Olsen, J. K., Sharma, K., Rummel, N., & Aleven, V. (2020). Temporal analysis of multimodal data to predict collaborative learning outcomes. British Journal of Educational Technology, 51(5), 1527–1547. https://doi.org/10.1111/bjet.12982
Ouhaichi, H., Spikol, D., & Vogel, B. (2023). Research trends in multimodal learning analytics: A systematic mapping study. Computers and Education: Artificial Intelligence, 4, Article 100136. https://doi.org/10.1016/j.caeai.2023.100136
Ouyang, F., Dai, X., & Chen, S. (2022). Applying multimodal learning analytics to examine the immediate and delayed effects of instructor scaffoldings on small groups’ collaborative programming. International Journal of STEM Education, 9, Article 45. https://doi.org/10.1186/s40594-022-00361-z
Oviatt, S. (2018). Ten opportunities and challenges for advancing student-centered multimodal learning analytics. In S. K. D’Mello, P. Georgiou, S. Scherer, E. M. Provost, M. Soleymani, & M. Worsley (Eds.), ICMI’18: Proceedings of the 20th ACM International Conference on Multimodal Interaction (pp. 87–94). ACM Press. https://doi.org/10.1145/3242969.3243010
Papavlasopoulou, S., Sharma, K., Melhart, D., Schellekens, J., Lee-Cultura, S., Giannakos, M. N., & Yiannakakis, G. N. (2021). Investigating gaze interaction to support children’s gameplay. International Journal of Child–Computer Interaction, 30, Article 100349. https://doi.org/10.1016/j.ijcci.2021.100349
Prinsloo, P., Slade, S., & Khalil, M. (2023). Multimodal learning analytics—In‐between student privacy and encroachment: A systematic review. British Journal of Educational Technology, 54(6), 1566–1586. https://doi.org/10.1111/bjet.13373
Sharma, K., & Giannakos, M. (2020). Multimodal data capabilities for learning: What can multimodal data tell us about learning? British Journal of Educational Technology, 51(5), 1450–1484. https://doi.org/10.1111/bjet.12993
Shin, H., Kim, B., & Gweon, G. (2020). Guessing or solving? Exploring the use of motion features from educational game logs. In R. Bernhaupt, F. Mueller, D. Verweij, J. Andres, J. McGrenere, A. Cockburn, I. Avellino, A. Goguey, P. Bjørn, S. Zhao, B. P. Samson, & R. Kocielnik (Eds.), CHI’20: Extended abstracts of the 2020 CHI conference on human factors in computing systems (LBW344). ACM Press. https://doi.org/10.1145/3334480.3383005
Siemens, G., & Gašević, D. (2012). Guest editorial: Learning and knowledge analytics. Journal of Educational Technology & Society, 15(3), 1–2. https://www.proquest.com/docview/1287024909
Spikol, D., Ruffaldi, E., & Cukurova, M. (2017). Using multimodal learning analytics to identify aspects of collaboration in project-based learning. In B. K. Smith, M. Borge, E. Mercier, & K. Y. Lim (Eds.), Making a difference: Prioritizing equity and access in CSCL, 12th international conference on computer supported collaborative learning (CSCL) 2017 (Vol. 1, pp. 263–270). International Society of the Learning Sciences. https://repository.isls.org/handle/1/240
Vujovic, M., Hernández‐Leo, D., Tassani, S., & Spikol, D. (2020). Round or rectangular tables for collaborative problem solving? A multimodal learning analytics study. British Journal of Educational Technology, 51(5), 1597–1614. https://doi.org/10.1111/bjet.12988
Wang, H., Barone, G., & Smith, A. (2024). Current and future role of data fusion and machine learning in infrastructure health monitoring. Structure and Infrastructure Engineering, 20(12), 1853–1882. https://doi.org/10.1080/15732479.2023.2165118
Worsley, M. (2014). Multimodal learning analytics as a tool for bridging learning theory and complex learning behaviours. In X. Ochoa, M. Worsley, K. Chiluiza, & S. Luz (Eds.), MLA’14 proceedings of the 2014 ACM multimodal learning analytics workshop and grand challenge (pp. 1–4). ACM Press. http://dx.doi.org/10.1145/2666633.2666634
Worsley, M., Martinez-Maldonado, R., & D’Angelo, C. (2021). A new era in multimodal learning analytics: Twelve core commitments to ground and grow MMLA. Journal of Learning Analytics, 8(3), 10–27. https://doi.org/10.18608/jla.2021.7361
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
