Multimodal Learning Analytics to Inform Learning Design: Lessons Learned from Computing Education




Multimodal learning analytics, Learning design, Physiological measures, Debugging, Predictive modelling


Programming is a complex learning activity that involves coordination of cognitive processes and affective states. These aspects are often considered individually in computing education research, demonstrating limited understanding of how and when students learn best. This issue confines researchers to contextualize evidence-driven outcomes when learning behaviour deviates from pedagogical intentions. Multimodal learning analytics (MMLA) captures data essential for measuring constructs (e.g., cognitive load, confusion) that are posited in the learning sciences as important for learning, and cannot effectively be measured solely with the use of programming process data (IDE-log data). Thus, we augmented IDE-log data with physiological data (e.g., gaze data) and participants’ facial expressions, collected during a debugging learning activity. The findings emphasize the need for learning analytics that are consequential for learning, rather than easy and convenient to collect. In that regard, our paper aims to provoke productive reflections and conversations about the potential of MMLA to expand and advance the synergy of learning analytics and learning design among the community of educators from a post-evaluation design-aware process to a permanent monitoring process of adaptation.


Ahonen, L., Cowley, B., Torniainen, J., Ukkonen, A., Vihavainen, A., & Puolamäki, K. (2016). Cognitive collaboration found in cardiac physiology: Study in classroom environment. PloS One, 11(7), e0159178.

Altadmri, A., & Brown, N. C. (2015). 37 million compilations: Investigating novice programming mistakes in large-scale student data. Proceedings of the 46th ACM Technical Symposium on Computer Science Education (SIGCSE ’15), 4–7 March 2015, Kansas City, MO, USA (pp. 522–527). New York, NY: ACM.

Andreassi, J. L. (2010). Psychophysiology: Human behavior and physiological response. Hove, East Sussex, UK: Psychology Press.

Arapakis, I., Lalmas, M., & Valkanas, G. (2014). Understanding within-content engagement through pattern analysis of mouse gestures. Proceedings of the 23rd ACM International Conference on Information and Knowledge Management (CIKM ’14), 3–7 November 2014, Shanghai, China (pp. 1439–1448). New York, NY: ACM.

Azevedo, R. (2015). Defining and measuring engagement and learning in science: Conceptual, theoretical, methodological, and analytical issues. Educational Psychologist, 50(1), 84–94.

Baker, R. S., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223–241.

Bakharia, A., Corrin, L., De Barba, P., Kennedy, G., Gašević, D., Mulder, R., … Lockyer, L. (2016). A conceptual framework linking learning design with learning analytics. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 329–338). New York: ACM.

Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L.-P. (2018). Openface 2.0: Facial behavior analysis toolkit. Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), 15–19 May 2018, Xi’an, China (pp. 59–66). Washington, DC: IEEE Computer Society.

Baumeister, R. F., & Vohs, K. D. (2004). Handbook of self-regulation: Research, theory, and applications. New York, NY: The Guilford Press.

Beattie, V., Collins, B., & McInnes, B. (1997). Deep and surface learning: A simple or simplistic dichotomy? Accounting Education, 6(1), 1–12.

Beatty, J. (1982). Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91(2), 276.

Bednarik, R. (2012). Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations. International Journal of Human–Computer Studies, 70(2), 143–155.

Bednarik, R., Busjahn, T., Gibaldi, A., Ahadi, A., Bielikova, M., Crosby, M., … Lister, R. (2020). EMIP: The eye movements in programming dataset. Science of Computer Programming, 198, article 102520.

Begel, A. (2016). Fun with software developers and biometrics: Invited talk. Proceedings of the 1st International Workshop on Emotion Awareness in Software Engineering (SEmotion ’16), 17 May 2016, Austin, TX, USA (pp. 1–2). New York, NY: ACM.

Blikstein, P. (2013). Multimodal learning analytics. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 102–106). New York, NY: ACM.

Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220–238.

Blikstein, P., Worsley, M., Piech, C., Sahami, M., Cooper, S., & Koller, D. (2014). Programming pluralism: Using learning analytics to detect patterns in the learning of computer programming. Journal of the Learning Sciences, 23(4), 561–599.

Bruce, C., Buckingham, L., Hynd, J., McMahon, C., Roggenkamp, M., & Stoodley, I. (2004). Ways of experiencing the act of learning to program: A phenomenographic study of introductory programming students at university. Journal of Information Technology Education: Research, 3(1), 145–160.

Brunken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in multimedia learning. Educational Psychologist, 38(1), 53–61.

Buettner, R. (2013). Cognitive workload of humans using artificial intelligence systems: Towards objective measurement applying eye-tracking technology. Annual Conference on Artificial Intelligence (pp. 37–48). Lecture notes in Computer Science, vol. 8077.

Busjahn, T., Schulte, C., Sharif, B., Begel, A., Hansen, M., Bednarik, R., … Antropova, M. (2014). Eye tracking in computing education. Proceedings of the 10th Annual Conference on International Computing Education Research (ICER ’14), 11–13 August 2014, Glasgow, UK (pp. 3–10). New York, NY: ACM.

Carter, A. S., & Hundhausen, C. D. (2015). The design of a programming environment to support greater social awareness and participation in early computing courses. Journal of Computing Sciences in Colleges, 31(1), 143–153.

Chan, M. C. E., Ochoa, X., & Clarke, D. (2020). Multimodal learning analytics in a laboratory classroom Machine Learning Paradigms (pp. 131–156): Springer.

Corrin, L., De Barba, P., Lockyear, L., Gašević, D., Williams, D., Dawson, S., … Bakharia, A. (2016). Completing the loop: Returning meaningful learning analytic data to teachers. Retrieved from

Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3), 241–250.

Crk, I., & Kluthe, T. (2014). Toward using alpha and theta brain waves to quantify programmer expertise. Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2014), 26–30 August 2014, Chicago, IL, USA. Washington, DC: IEEE Computer Society.

Cukurova, M., Giannakos, M., & Martinez‐Maldonado, R. (2020). The promise and challenges of multimodal learning analytics. British Journal of Educational Technology, 51(5), 1441–1449.

D’Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145–157.

D’Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, 153–170.

Dillenbourg, P. (2013). Design for classroom orchestration. Computers & Education, 69, 485–492.

Dörner, D., & Funke, J. (2017). Complex problem solving: What it is and what it is not. Frontiers in Psychology, 8, 1153.

Echeverria, V., Martinez-Maldonado, R., & Buckingham Shum, S. (2019). Towards collaboration translucence: Giving meaning to multimodal group data. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), 4–9 May 2019, Glasgow, Scotland, UK (Paper No. 39). New York, NY: ACM.

Edwards, S. H., & Perez-Quinones, M. A. (2008). Web-CAT: Automatically grading programming assignments. ACM SIGCSE Bulletin, 40(3), 328.

Ekman, P. (1993). Facial expression and emotion. American Psychologist, 48(4), 384–392.

Ekman, P., & Friesen, E. (1978). Facial action coding system: A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press.

Ekman, P., Freisen, W. V., & Ancoli, S. (1980). Facial signs of emotional experience. Journal of Personality and Social Psychology, 39(6), 1125-1134.

Fernández-Delgado, M., Cernadas, E., Barro, S., & Amorim, D. (2014). Do we need hundreds of classifiers to solve real world classification problems? The Journal of Machine Learning Research, 15(1), 3133–3181.

Fritz, T., Begel, A., Müller, S. C., Yigit-Elliott, S., & Züger, M. (2014). Using psycho-physiological measures to assess task difficulty in software development. Proceedings of the 36th International Conference on Software Engineering (ICSE 2014), 31 May–7 June 2014, Hyderabad, India (pp. 402–413). New York, NY: ACM.

Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71.

Gašević, D., Kovanović, V., & Joksimović, S. (2017). Piecing the learning analytics puzzle: A consolidated model of a field of research and practice. Learning: Research and Practice, 3(1), 63–78.

Giannakos, M. N., Sharma, K., Pappas, I. O., Kostakos, V., & Velloso, E. (2019). Multimodal data as a means to understand the learning experience. International Journal of Information Management, 48, 108–119.

Grover, S., Bienkowski, M., Tamrakar, A., Siddiquie, B., Salter, D., & Divakaran, A. (2016). Multimodal analytics to study collaborative problem solving in pair programming. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 516–517). New York: ACM.

Hernández‐Leo, D., Martinez‐Maldonado, R., Pardo, A., Muñoz‐Cristóbal, J. A., & Rodríguez‐Triana, M. J. (2019). Analytics for learning design: A layered framework and tools. British Journal of Educational Technology, 50(1), 139–152.

Hollender, N., Hofmann, C., Deneke, M., & Schmitz, B. (2010). Integrating cognitive load theory and concepts of human–computer interaction. Computers in Human Behavior, 26(6), 1278–1288.

Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., & Van de Weijer, J. (2011). Eye tracking: A comprehensive guide to methods and measures. Oxford, UK: Oxford University Press.

Hundhausen, C. D., Olivares, D. M., & Carter, A. S. (2017). IDE-based learning analytics for computing education: A process model, critical review, and research agenda. ACM Transactions on Computing Education (TOCE), 17(3), 11, 1–26.

Ihantola, P., Sorva, J., & Vihavainen, A. (2014). Automatically detectable indicators of programming assignment difficulty. Proceedings of the 15th Annual Conference on Information Technology Education (SIGITE ’14), 15–18 October 2014, Atlanta, GA, USA (pp. 33–38). New York, NY: ACM.

Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S. H., … Rivers, K. (2015). Educational data mining and learning analytics in programming: Literature review and case studies. Proceedings of the 2015 ITiCSE on Working Group Reports (ITICSE-WGR ’15), 4–8 July 2015, Vilnius, Lithuania (pp. 41–63). New York, NY: ACM.

Jadud, M. C. (2006). Methods and tools for exploring novice compilation behaviour. Proceedings of the 2nd International Workshop on Computing Education Research (ICER ’06), 9–10 September 2006, Canterbury, UK (pp. 73–84). New York, NY: ACM.

Järvelä, S., Malmberg, J., Haataja, E., Sobocinski, M., & Kirschner, P. A. (2019). What multimodal data can tell us about the students’ regulation of their learning process. Learning and Instruction (in press, corrected proof).

Kaller, C. P., Rahm, B., Bolkenius, K., & Unterrainer, J. M. (2009). Eye movements and visuospatial problem solving: Identifying separable phases of complex cognition. Psychophysiology, 46(4), 818–830.

Kevic, K., Walters, B. M., Shaffer, T. R., Sharif, B., Shepherd, D. C., & Fritz, T. (2015). Tracing software developers’ eyes and interactions for change tasks. Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015), 30 August–4 September 2015, Bergamo, Italy (pp. 202–213). New York, NY: ACM.

Kirschner, P. A. (2017). If it ain’t there, it’s broke! Retrieved from

Kress, G. (2001). Multimodal teaching and learning: The rhetorics of the science classroom. London, UK: A&C Black.

Kurland, D. M., Pea, R. D., Clement, C., & Mawby, R. (1986). A study of the development of programming ability and thinking skills in high school students. Journal of Educational Computing Research, 2(4), 429–458.

Lockyer, L., & Dawson, S. (2012). Where learning analytics meets learning design. In S. Buckingham Shum, D. Gašević, & R. Ferguson (Eds.), Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ʼ12), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 14–15). New York, NY: ACM.

Mangaroska, K., & Giannakos, M. N. (2018). Learning analytics for learning design: A systematic literature review of analytics-driven design to enhance learning. IEEE Transactions on Learning Technologies, 12(4), 516–534. 10.1109/TLT.2018.2868673

Mangaroska, K., Sharma, K., Giannakos, M., Træteberg, H., & Dillenbourg, P. (2018). Gaze-driven design insights to amplify debugging skills: A learner-centred analysis approach. Journal of Learning Analytics, 5(3), 98–119.

Marion, B., Impagliazzo, J., St. Clair, C., Soroka, B., & Whitfield, D. (2007). Assessing computer science programs: What have we learned? Proceedings of the 38th ACM Technical Symposium on Computer Science Education (SIGCSE 07), 7–10 March 2007, Covington, KY, USA (pp. 131–132). New York, NY: ACM.

Martinez-Maldonado, R., Mangaroska, K., Schulte, J., Elliott, D., Axisa, C., & Shum, S. B. (2020). Teacher tracking with integrity: What indoor positioning can reveal about instructional proxemics. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(1), Article No. 22.

May, J. G., Kennedy, R. S., Williams, M. C., Dunlap, W. P., & Brannan, J. R. (1990). Eye movement indices of mental workload. Acta Psychologica, 75(1), 75–89.

Mayer, R. E. (2010). Unique contributions of eye-tracking research to the study of learning with graphics. Learning and Instruction, 20(2), 167–171.

McCauley, R., Fitzgerald, S., Lewandowski, G., Murphy, L., Simon, B., Thomas, L., & Zander, C. (2008). Debugging: A review of the literature from an educational perspective. Computer Science Education, 18(2), 67–92.

McDaniel, B., D’Mello, S., King, B., Chipman, P., Tapp, K., & Graesser, A. (2007). Facial features for affective state detection in learning environments. In D. S. McNamara & G. Trafton (Eds.), Proceedings of the 29th Annual Conference of the Cognitive Science Society (CogSci 2007), 1–4 August 2007, Nashville, TN, USA (pp. 467–472). Austin, TX: Cognitive Science Society.

Melero, J., Hernández‐Leo, D., Sun, J., Santos, P., & Blat, J. (2015). How was the activity? A visualization support for a case of location‐based learning design. British Journal of Educational Technology, 46(2), 317–329.

Müller, S. C. (2015). Measuring software developers’ perceived difficulty with biometric sensors. Proceedings of the 37th International Conference on Software Engineering (ICSE ’15), 16–24 May 2015, Florence, Italy (Vol. 2, pp. 887–890). New York, NY: ACM. 10.1109/ICSE.2015.284

Nathan, M. J., & Wagner Alibali, M. (2010). Learning sciences. Wiley Interdisciplinary Reviews: Cognitive Science, 1(3), 329–345.

Neelen, M., & Kirschner, P. (2017). Where are the learning sciences in learning analytics research? Retrieved from

O’Grady, M. J. (2012). Practical problem-based learning in computing education. ACM Transactions on Computing Education (TOCE), 12(3), 1–16.

Ochoa, X., Domínguez, F., Guamán, B., Maya, R., Falcones, G., & Castells, J. (2018). The RAP system: Automatic feedback of oral presentation skills using multimodal analysis and low-cost sensors. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 360–364). New York, NY: ACM.

Ochoa, X. (2017). Multimodal learning analytics. In C. Lang, G. Siemens, A. Wise & D. Gašević (Eds.), The Handbook of Learning Analytics (1st Ed.) (pp. 129–141). Society for Learning Analytics Research.

Ochoa, X., & Worsley, M. (2016). Augmenting learning analytics with multimodal sensory data. Journal of Learning Analytics, 3(2), 213–219.

Ogan, A. (2019). Reframing classroom sensing: Promise and peril. Interactions, 26(6), 26–32. Retrieved from

Olsen, A. (2012, March 20). The Tobii I-VT fixation filter: Algorythm description. Tobii Technology. Retrieved from

Oviatt, S., Grafsgaard, J., Chen, L., & Ochoa, X. (2018, October). Multimodal learning analytics: Assessing learners’ mental state during the process of learning. In S. Oviatt, B. Schuller & P. R. Cohen (Eds.), The handbook of multimodal-multisensor interfaces: Signal processing, architectures, and detection of emotion and cognition (Vol. 2) (pp. 331–374). New York, NY: ACM Books.

Parnin, C. (2011). Subvocalization: Toward hearing the inner thoughts of developers. Proceedings of the IEEE 19th International Conference on Program Comprehension (ICPC 2011) 22–24 June 2011, Kingston, ON, Canada (pp. 197–200). New York, NY: ACM.

Perkins, D. N., Hancock, C., Hobbs, R., Martin, F., & Simmons, R. (1986). Conditions of learning in novice programmers. Journal of Educational Computing Research, 2(1), 37–55.

Perrotta, C. (2013). Do school‐level factors influence the educational benefits of digital technology? A critical analysis of teachers’ perceptions. British Journal of Educational Technology, 44(2), 314–327.

Persico, D., & Pozzi, F. (2015). Informing learning design with learning analytics to improve teacher inquiry. British Journal of Educational Technology, 46(2), 230–248.

Prieto, L. P., Sharma, K., Kidzinski, Ł., Rodríguez‐Triana, M. J., & Dillenbourg, P. (2018). Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data. Journal of Computer Assisted Learning, 34(2), 193–203.

Reimann, P. (2016). Connecting learning analytics with learning research: The role of design-based research. Learning: Research and Practice, 2(2), 130–142.

Rienties, B., Nguyen, Q., Holmes, W., & Reedy, K. (2017). A review of ten years of implementation and research in aligning learning design with learning analytics at the Open University UK. Interaction Design and Architecture(s), 33, 134–154.

Rodríguez-Triana, M. J., Prieto, L. P., Martínez-Monés, A., Asensio-Pérez, J. I., & Dimitriadis, Y. (2018). The teacher in the loop: Customizing multimodal learning analytics for blended learning. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 417–426). New York, NY: ACM.

Schmitz, M., Van Limbeek, E., Greller, W., Sloep, P., & Drachsler, H. (2017). Opportunities and challenges in using learning analytics in learning design. Proceedings of the 12th European Conference on Technology Enhanced Learning (EC-TEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 209–223). Lecture Notes in Computer Science, vol. 10474. Springer.

Schulte, C., Magenheim, J., Müller, K., & Budde, L. (2017). The design and exploration cycle as research and development framework in computing education. Proceedings of the 2017 IEEE Global Engineering Education Conference (EDUCON 2017), 25–28 April 2017, Athens, Greece (pp. 867–876). Washington, DC: IEEE Computer Society.

Sharma, K., Mangaroska, K., Trætteberg, H., Lee-Cultura, S., & Giannakos, M. (2018). Evidence for programming strategies in university coding exercises. Proceedings of the 13th European Conference on Technology Enhanced Learning (EC-TEL 2018), 3–5 September 2018, Leeds, UK (pp. 326–339). Lecture Notes in Computer Science, vol. 11082. Cham, Switzerland: Springer.

Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14.

Snow, R. E., Corno, L., & Jackson III, D. N. (1996). Individual differences in affective and conative functions. In D. C. Berliner & R. C. Calfee (Eds.), Handbook of educational psychology (pp. 243-310). New York, NY: Macmillan Library Reference USA.

Turkle, S., & Papert, S. (1992). Epistemological pluralism and the revaluation of the concrete. Journal of Mathematical Behavior, 11(1), 3–33.

Van Harmelen, M., & Workman, D. (2012). Analytics for learning and teaching. CETIS Analytics Series, 1(3), 1–40.

Van Merriënboer, J. J., Kirschner, P. A., & Kester, L. (2003). Taking the load off a learner’s mind: Instructional design for complex learning. Educational Psychologist, 38(1), 5–13.

Wiley, K. J., Dimitriadis, Y., Bradford, A., & Linn, M. C. (2020). From theory to action: Developing and evaluating learning analytics for learning design. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 569–578). New York, NY: ACM.

Wise, A. F., & Jung, Y. (2019). Teaching with Analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69.

Worsley, M. (2014). Multimodal learning analytics as a tool for bridging learning theory and complex learning behaviors. Proceedings of the 2014 ACM Workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA ’14), 12–16 November 2014, Istanbul, Turkey (pp. 1–4). New York, NY: ACM.

Worsley, M., & Blikstein, P. (2013). Towards the development of multimodal action based assessment. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 94–101). New York: ACM.

Worsley, M., & Blikstein, P. (2015). Leveraging multimodal learning analytics to differentiate student learning strategies. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 360–367). New York, NY: ACM.




How to Cite

Mangaroska, K., Sharma, K. ., Gasevic, D. ., & Giannakos, M. . (2020). Multimodal Learning Analytics to Inform Learning Design: Lessons Learned from Computing Education. Journal of Learning Analytics, 7(3), 79-97.



Special Section: Learning Design and Learning Analytics

Most read articles by the same author(s)

1 2 3 4 > >>