Multimodal Learning Analytics to Inform Learning Design: Lessons Learned from Computing Education

Authors

DOI:

https://doi.org/10.18608/jla.2020.73.7

Keywords:

Multimodal learning analytics, Learning design, Physiological measures, Debugging, Predictive modelling

Abstract

Programming is a complex learning activity that involves coordination of cognitive processes and affective states. These aspects are often considered individually in computing education research, demonstrating limited understanding of how and when students learn best. This issue confines researchers to contextualize evidence-driven outcomes when learning behaviour deviates from pedagogical intentions. Multimodal learning analytics (MMLA) captures data essential for measuring constructs (e.g., cognitive load, confusion) that are posited in the learning sciences as important for learning, and cannot effectively be measured solely with the use of programming process data (IDE-log data). Thus, we augmented IDE-log data with physiological data (e.g., gaze data) and participants’ facial expressions, collected during a debugging learning activity. The findings emphasize the need for learning analytics that are consequential for learning, rather than easy and convenient to collect. In that regard, our paper aims to provoke productive reflections and conversations about the potential of MMLA to expand and advance the synergy of learning analytics and learning design among the community of educators from a post-evaluation design-aware process to a permanent monitoring process of adaptation.

References

Ahonen, L., Cowley, B., Torniainen, J., Ukkonen, A., Vihavainen, A., & Puolamäki, K. (2016). Cognitive collaboration found in cardiac physiology: Study in classroom environment. PloS One, 11(7), e0159178. https://dx.doi.org/10.1371/journal.pone.0159178

Altadmri, A., & Brown, N. C. (2015). 37 million compilations: Investigating novice programming mistakes in large-scale student data. Proceedings of the 46th ACM Technical Symposium on Computer Science Education (SIGCSE ’15), 4–7 March 2015, Kansas City, MO, USA (pp. 522–527). New York, NY: ACM. https://dx.doi.org/10.1145/2676723.2677258

Andreassi, J. L. (2010). Psychophysiology: Human behavior and physiological response. Hove, East Sussex, UK: Psychology Press.

Arapakis, I., Lalmas, M., & Valkanas, G. (2014). Understanding within-content engagement through pattern analysis of mouse gestures. Proceedings of the 23rd ACM International Conference on Information and Knowledge Management (CIKM ’14), 3–7 November 2014, Shanghai, China (pp. 1439–1448). New York, NY: ACM. https://dx.doi.org/10.1145/2661829.2661909

Azevedo, R. (2015). Defining and measuring engagement and learning in science: Conceptual, theoretical, methodological, and analytical issues. Educational Psychologist, 50(1), 84–94. https://dx.doi.org/10.1080/00461520.2015.1004069

Baker, R. S., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223–241. https://dx.doi.org/10.1016/j.ijhcs.2009.12.003

Bakharia, A., Corrin, L., De Barba, P., Kennedy, G., Gašević, D., Mulder, R., … Lockyer, L. (2016). A conceptual framework linking learning design with learning analytics. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 329–338). New York: ACM. https://dx.doi.org/10.1145/2883851.2883944

Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L.-P. (2018). Openface 2.0: Facial behavior analysis toolkit. Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), 15–19 May 2018, Xi’an, China (pp. 59–66). Washington, DC: IEEE Computer Society. https://dx.doi.org/10.1109/FG.2018.00019

Baumeister, R. F., & Vohs, K. D. (2004). Handbook of self-regulation: Research, theory, and applications. New York, NY: The Guilford Press.

Beattie, V., Collins, B., & McInnes, B. (1997). Deep and surface learning: A simple or simplistic dichotomy? Accounting Education, 6(1), 1–12. https://dx.doi.org/10.1080/096392897331587

Beatty, J. (1982). Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91(2), 276. https://dx.doi.org/10.1037/0033-2909.91.2.276

Bednarik, R. (2012). Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations. International Journal of Human–Computer Studies, 70(2), 143–155. https://dx.doi.org/10.1016/j.ijhcs.2011.09.003

Bednarik, R., Busjahn, T., Gibaldi, A., Ahadi, A., Bielikova, M., Crosby, M., … Lister, R. (2020). EMIP: The eye movements in programming dataset. Science of Computer Programming, 198, article 102520. https://dx.doi.org/10.1016/j.scico.2020.102520

Begel, A. (2016). Fun with software developers and biometrics: Invited talk. Proceedings of the 1st International Workshop on Emotion Awareness in Software Engineering (SEmotion ’16), 17 May 2016, Austin, TX, USA (pp. 1–2). New York, NY: ACM. https://dx.doi.org/10.1145/2897000.2897007

Blikstein, P. (2013). Multimodal learning analytics. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 102–106). New York, NY: ACM. https://dx.doi.org/10.1145/2460296.2460316

Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220–238. https://dx.doi.org/10.18608/jla.2016.32.11

Blikstein, P., Worsley, M., Piech, C., Sahami, M., Cooper, S., & Koller, D. (2014). Programming pluralism: Using learning analytics to detect patterns in the learning of computer programming. Journal of the Learning Sciences, 23(4), 561–599. https://dx.doi.org/10.1080/10508406.2014.954750

Bruce, C., Buckingham, L., Hynd, J., McMahon, C., Roggenkamp, M., & Stoodley, I. (2004). Ways of experiencing the act of learning to program: A phenomenographic study of introductory programming students at university. Journal of Information Technology Education: Research, 3(1), 145–160. https://dx.doi.org/10.28945/294

Brunken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in multimedia learning. Educational Psychologist, 38(1), 53–61. https://dx.doi.org/10.1207/S15326985EP3801_7

Buettner, R. (2013). Cognitive workload of humans using artificial intelligence systems: Towards objective measurement applying eye-tracking technology. Annual Conference on Artificial Intelligence (pp. 37–48). Lecture notes in Computer Science, vol. 8077. https://dx.doi.org/10.1007/978-3-642-40942-4_4

Busjahn, T., Schulte, C., Sharif, B., Begel, A., Hansen, M., Bednarik, R., … Antropova, M. (2014). Eye tracking in computing education. Proceedings of the 10th Annual Conference on International Computing Education Research (ICER ’14), 11–13 August 2014, Glasgow, UK (pp. 3–10). New York, NY: ACM. https://dx.doi.org/10.1145/2632320.2632344

Carter, A. S., & Hundhausen, C. D. (2015). The design of a programming environment to support greater social awareness and participation in early computing courses. Journal of Computing Sciences in Colleges, 31(1), 143–153.

Chan, M. C. E., Ochoa, X., & Clarke, D. (2020). Multimodal learning analytics in a laboratory classroom Machine Learning Paradigms (pp. 131–156): Springer. https://dx.doi.org/10.1007/978-3-030-13743-4_8

Corrin, L., De Barba, P., Lockyear, L., Gašević, D., Williams, D., Dawson, S., … Bakharia, A. (2016). Completing the loop: Returning meaningful learning analytic data to teachers. Retrieved from https://research.monash.edu/en/publications/completing-the-loop-returning-meaningful-learning-analytic-data-t

Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3), 241–250. https://dx.doi.org/10.1080/1358165042000283101

Crk, I., & Kluthe, T. (2014). Toward using alpha and theta brain waves to quantify programmer expertise. Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2014), 26–30 August 2014, Chicago, IL, USA. Washington, DC: IEEE Computer Society. https://dx.doi.org/10.1109/embc.2014.6944840

Cukurova, M., Giannakos, M., & Martinez‐Maldonado, R. (2020). The promise and challenges of multimodal learning analytics. British Journal of Educational Technology, 51(5), 1441–1449. https://dx.doi.org/10.1111/bjet.13015

D’Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145–157. https://dx.doi.org/10.1016/j.learninstruc.2011.10.001

D’Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, 153–170. https://dx.doi.org/10.1016/j.learninstruc.2012.05.003

Dillenbourg, P. (2013). Design for classroom orchestration. Computers & Education, 69, 485–492. https://dx.doi.org/10.1016/j.compedu.2013.04.013

Dörner, D., & Funke, J. (2017). Complex problem solving: What it is and what it is not. Frontiers in Psychology, 8, 1153. https://dx.doi.org/10.3389/fpsyg.2017.01153

Echeverria, V., Martinez-Maldonado, R., & Buckingham Shum, S. (2019). Towards collaboration translucence: Giving meaning to multimodal group data. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), 4–9 May 2019, Glasgow, Scotland, UK (Paper No. 39). New York, NY: ACM. https://dx.doi.org/10.1145/3290605.3300269

Edwards, S. H., & Perez-Quinones, M. A. (2008). Web-CAT: Automatically grading programming assignments. ACM SIGCSE Bulletin, 40(3), 328. https://dx.doi.org/10.1145/1384271.1384371

Ekman, P. (1993). Facial expression and emotion. American Psychologist, 48(4), 384–392. https://dx.doi.org/10.1037/0003-066X.48.4.384

Ekman, P., & Friesen, E. (1978). Facial action coding system: A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press.

Ekman, P., Freisen, W. V., & Ancoli, S. (1980). Facial signs of emotional experience. Journal of Personality and Social Psychology, 39(6), 1125-1134. https://dx.doi.org/10.1037/h0077722

Fernández-Delgado, M., Cernadas, E., Barro, S., & Amorim, D. (2014). Do we need hundreds of classifiers to solve real world classification problems? The Journal of Machine Learning Research, 15(1), 3133–3181.

Fritz, T., Begel, A., Müller, S. C., Yigit-Elliott, S., & Züger, M. (2014). Using psycho-physiological measures to assess task difficulty in software development. Proceedings of the 36th International Conference on Software Engineering (ICSE 2014), 31 May–7 June 2014, Hyderabad, India (pp. 402–413). New York, NY: ACM. https://dx.doi.org/10.1145/2568225.2568266

Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. https://dx.doi.org/10.1007/s11528-014-0822-x

Gašević, D., Kovanović, V., & Joksimović, S. (2017). Piecing the learning analytics puzzle: A consolidated model of a field of research and practice. Learning: Research and Practice, 3(1), 63–78. https://dx.doi.org/10.1080/23735082.2017.1286142

Giannakos, M. N., Sharma, K., Pappas, I. O., Kostakos, V., & Velloso, E. (2019). Multimodal data as a means to understand the learning experience. International Journal of Information Management, 48, 108–119. https://dx.doi.org/10.1016/j.ijinfomgt.2019.02.003

Grover, S., Bienkowski, M., Tamrakar, A., Siddiquie, B., Salter, D., & Divakaran, A. (2016). Multimodal analytics to study collaborative problem solving in pair programming. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 516–517). New York: ACM. https://dx.doi.org/10.1145/2883851.2883877

Hernández‐Leo, D., Martinez‐Maldonado, R., Pardo, A., Muñoz‐Cristóbal, J. A., & Rodríguez‐Triana, M. J. (2019). Analytics for learning design: A layered framework and tools. British Journal of Educational Technology, 50(1), 139–152. https://dx.doi.org/10.1111/bjet.12645

Hollender, N., Hofmann, C., Deneke, M., & Schmitz, B. (2010). Integrating cognitive load theory and concepts of human–computer interaction. Computers in Human Behavior, 26(6), 1278–1288. https://dx.doi.org/10.1016/j.chb.2010.05.031

Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., & Van de Weijer, J. (2011). Eye tracking: A comprehensive guide to methods and measures. Oxford, UK: Oxford University Press.

Hundhausen, C. D., Olivares, D. M., & Carter, A. S. (2017). IDE-based learning analytics for computing education: A process model, critical review, and research agenda. ACM Transactions on Computing Education (TOCE), 17(3), 11, 1–26. https://dx.doi.org/10.1145/3105759

Ihantola, P., Sorva, J., & Vihavainen, A. (2014). Automatically detectable indicators of programming assignment difficulty. Proceedings of the 15th Annual Conference on Information Technology Education (SIGITE ’14), 15–18 October 2014, Atlanta, GA, USA (pp. 33–38). New York, NY: ACM. https://dx.doi.org/10.1145/2656450.2656476

Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S. H., … Rivers, K. (2015). Educational data mining and learning analytics in programming: Literature review and case studies. Proceedings of the 2015 ITiCSE on Working Group Reports (ITICSE-WGR ’15), 4–8 July 2015, Vilnius, Lithuania (pp. 41–63). New York, NY: ACM. https://dx.doi.org/10.1145/2858796.2858798

Jadud, M. C. (2006). Methods and tools for exploring novice compilation behaviour. Proceedings of the 2nd International Workshop on Computing Education Research (ICER ’06), 9–10 September 2006, Canterbury, UK (pp. 73–84). New York, NY: ACM. https://dx.doi.org/10.1145/1151588.1151600

Järvelä, S., Malmberg, J., Haataja, E., Sobocinski, M., & Kirschner, P. A. (2019). What multimodal data can tell us about the students’ regulation of their learning process. Learning and Instruction (in press, corrected proof). https://dx.doi.org/10.1016/j.learninstruc.2019.04.004

Kaller, C. P., Rahm, B., Bolkenius, K., & Unterrainer, J. M. (2009). Eye movements and visuospatial problem solving: Identifying separable phases of complex cognition. Psychophysiology, 46(4), 818–830. https://dx.doi.org/10.1111/j.1469-8986.2009.00821.x

Kevic, K., Walters, B. M., Shaffer, T. R., Sharif, B., Shepherd, D. C., & Fritz, T. (2015). Tracing software developers’ eyes and interactions for change tasks. Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015), 30 August–4 September 2015, Bergamo, Italy (pp. 202–213). New York, NY: ACM. https://dx.doi.org/10.1145/2786805.2786864

Kirschner, P. A. (2017). If it ain’t there, it’s broke! Retrieved from https://3starlearningexperiences.wordpress.com/2017/09/05/if-it-aint-there-its-broken/

Kress, G. (2001). Multimodal teaching and learning: The rhetorics of the science classroom. London, UK: A&C Black.

Kurland, D. M., Pea, R. D., Clement, C., & Mawby, R. (1986). A study of the development of programming ability and thinking skills in high school students. Journal of Educational Computing Research, 2(4), 429–458. https://dx.doi.org/10.2190%2FBKML-B1QV-KDN4-8ULH

Lockyer, L., & Dawson, S. (2012). Where learning analytics meets learning design. In S. Buckingham Shum, D. Gašević, & R. Ferguson (Eds.), Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ʼ12), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 14–15). New York, NY: ACM. https://dx.doi.org/10.1145/2330601.2330609

Mangaroska, K., & Giannakos, M. N. (2018). Learning analytics for learning design: A systematic literature review of analytics-driven design to enhance learning. IEEE Transactions on Learning Technologies, 12(4), 516–534. https://dx.doi.org/ 10.1109/TLT.2018.2868673

Mangaroska, K., Sharma, K., Giannakos, M., Træteberg, H., & Dillenbourg, P. (2018). Gaze-driven design insights to amplify debugging skills: A learner-centred analysis approach. Journal of Learning Analytics, 5(3), 98–119. https://dx.doi.org/10.18608/jla.2018.53.7

Marion, B., Impagliazzo, J., St. Clair, C., Soroka, B., & Whitfield, D. (2007). Assessing computer science programs: What have we learned? Proceedings of the 38th ACM Technical Symposium on Computer Science Education (SIGCSE 07), 7–10 March 2007, Covington, KY, USA (pp. 131–132). New York, NY: ACM. https://dx.doi.org/10.1145/1227310.1227358

Martinez-Maldonado, R., Mangaroska, K., Schulte, J., Elliott, D., Axisa, C., & Shum, S. B. (2020). Teacher tracking with integrity: What indoor positioning can reveal about instructional proxemics. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(1), Article No. 22. https://dx.doi.org/10.1145/3381017

May, J. G., Kennedy, R. S., Williams, M. C., Dunlap, W. P., & Brannan, J. R. (1990). Eye movement indices of mental workload. Acta Psychologica, 75(1), 75–89. https://dx.doi.org/10.1016/0001-6918(90)90067-P

Mayer, R. E. (2010). Unique contributions of eye-tracking research to the study of learning with graphics. Learning and Instruction, 20(2), 167–171. https://dx.doi.org/10.1016/j.learninstruc.2009.02.012

McCauley, R., Fitzgerald, S., Lewandowski, G., Murphy, L., Simon, B., Thomas, L., & Zander, C. (2008). Debugging: A review of the literature from an educational perspective. Computer Science Education, 18(2), 67–92. https://dx.doi.org/10.1080/08993400802114581

McDaniel, B., D’Mello, S., King, B., Chipman, P., Tapp, K., & Graesser, A. (2007). Facial features for affective state detection in learning environments. In D. S. McNamara & G. Trafton (Eds.), Proceedings of the 29th Annual Conference of the Cognitive Science Society (CogSci 2007), 1–4 August 2007, Nashville, TN, USA (pp. 467–472). Austin, TX: Cognitive Science Society. http://csjarchive.cogsci.rpi.edu/Proceedings/2007/docs/p467.pdf

Melero, J., Hernández‐Leo, D., Sun, J., Santos, P., & Blat, J. (2015). How was the activity? A visualization support for a case of location‐based learning design. British Journal of Educational Technology, 46(2), 317–329. https://dx.doi.org/10.1111/bjet.12238

Müller, S. C. (2015). Measuring software developers’ perceived difficulty with biometric sensors. Proceedings of the 37th International Conference on Software Engineering (ICSE ’15), 16–24 May 2015, Florence, Italy (Vol. 2, pp. 887–890). New York, NY: ACM. https://dx.doi.org/ 10.1109/ICSE.2015.284

Nathan, M. J., & Wagner Alibali, M. (2010). Learning sciences. Wiley Interdisciplinary Reviews: Cognitive Science, 1(3), 329–345. https://dx.doi.org/10.1002/wcs.54

Neelen, M., & Kirschner, P. (2017). Where are the learning sciences in learning analytics research? Retrieved from https://3starlearningexperiences.wordpress.com/2017/10/17/where-are-the-learning-sciences-in-learning-analytics-research/

O’Grady, M. J. (2012). Practical problem-based learning in computing education. ACM Transactions on Computing Education (TOCE), 12(3), 1–16. https://dx.doi.org/10.1145/2275597.2275599

Ochoa, X., Domínguez, F., Guamán, B., Maya, R., Falcones, G., & Castells, J. (2018). The RAP system: Automatic feedback of oral presentation skills using multimodal analysis and low-cost sensors. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 360–364). New York, NY: ACM. https://dx.doi.org/10.1145/3170358.3170406

Ochoa, X. (2017). Multimodal learning analytics. In C. Lang, G. Siemens, A. Wise & D. Gašević (Eds.), The Handbook of Learning Analytics (1st Ed.) (pp. 129–141). Society for Learning Analytics Research. https://dx.doi.org/10.18608/hla17

Ochoa, X., & Worsley, M. (2016). Augmenting learning analytics with multimodal sensory data. Journal of Learning Analytics, 3(2), 213–219. https://dx.doi.org/10.18608/jla.2016.32.10

Ogan, A. (2019). Reframing classroom sensing: Promise and peril. Interactions, 26(6), 26–32. Retrieved from https://interactions.acm.org/archive/view/november-december-2019/reframing-classroom-sensing

Olsen, A. (2012, March 20). The Tobii I-VT fixation filter: Algorythm description. Tobii Technology. Retrieved from https://www.tobiipro.com/siteassets/tobii-pro/learn-and-support/analyze/how-do-we-classify-eye-movements/tobii-pro-i-vt-fixation-filter.pdf

Oviatt, S., Grafsgaard, J., Chen, L., & Ochoa, X. (2018, October). Multimodal learning analytics: Assessing learners’ mental state during the process of learning. In S. Oviatt, B. Schuller & P. R. Cohen (Eds.), The handbook of multimodal-multisensor interfaces: Signal processing, architectures, and detection of emotion and cognition (Vol. 2) (pp. 331–374). New York, NY: ACM Books. https://dx.doi.org/10.1145/3107990.3108003

Parnin, C. (2011). Subvocalization: Toward hearing the inner thoughts of developers. Proceedings of the IEEE 19th International Conference on Program Comprehension (ICPC 2011) 22–24 June 2011, Kingston, ON, Canada (pp. 197–200). New York, NY: ACM. https://dx.doi.org/10.1109/ICPC.2011.49

Perkins, D. N., Hancock, C., Hobbs, R., Martin, F., & Simmons, R. (1986). Conditions of learning in novice programmers. Journal of Educational Computing Research, 2(1), 37–55. https://dx.doi.org/10.2190%2FGUJT-JCBJ-Q6QU-Q9PL

Perrotta, C. (2013). Do school‐level factors influence the educational benefits of digital technology? A critical analysis of teachers’ perceptions. British Journal of Educational Technology, 44(2), 314–327. https://dx.doi.org/10.1111/j.1467-8535.2012.01304.x

Persico, D., & Pozzi, F. (2015). Informing learning design with learning analytics to improve teacher inquiry. British Journal of Educational Technology, 46(2), 230–248. https://dx.doi.org/10.1111/bjet.12207

Prieto, L. P., Sharma, K., Kidzinski, Ł., Rodríguez‐Triana, M. J., & Dillenbourg, P. (2018). Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data. Journal of Computer Assisted Learning, 34(2), 193–203. https://dx.doi.org/10.1111/jcal.12232

Reimann, P. (2016). Connecting learning analytics with learning research: The role of design-based research. Learning: Research and Practice, 2(2), 130–142. https://dx.doi.org/10.1080/23735082.2016.1210198

Rienties, B., Nguyen, Q., Holmes, W., & Reedy, K. (2017). A review of ten years of implementation and research in aligning learning design with learning analytics at the Open University UK. Interaction Design and Architecture(s), 33, 134–154.

Rodríguez-Triana, M. J., Prieto, L. P., Martínez-Monés, A., Asensio-Pérez, J. I., & Dimitriadis, Y. (2018). The teacher in the loop: Customizing multimodal learning analytics for blended learning. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 417–426). New York, NY: ACM. https://dx.doi.org/10.1145/3170358.3170364

Schmitz, M., Van Limbeek, E., Greller, W., Sloep, P., & Drachsler, H. (2017). Opportunities and challenges in using learning analytics in learning design. Proceedings of the 12th European Conference on Technology Enhanced Learning (EC-TEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 209–223). Lecture Notes in Computer Science, vol. 10474. Springer. https://dx.doi.org/10.1007/978-3-319-66610-5_16

Schulte, C., Magenheim, J., Müller, K., & Budde, L. (2017). The design and exploration cycle as research and development framework in computing education. Proceedings of the 2017 IEEE Global Engineering Education Conference (EDUCON 2017), 25–28 April 2017, Athens, Greece (pp. 867–876). Washington, DC: IEEE Computer Society. https://dx.doi.org/10.1109/EDUCON.2017.7942950

Sharma, K., Mangaroska, K., Trætteberg, H., Lee-Cultura, S., & Giannakos, M. (2018). Evidence for programming strategies in university coding exercises. Proceedings of the 13th European Conference on Technology Enhanced Learning (EC-TEL 2018), 3–5 September 2018, Leeds, UK (pp. 326–339). Lecture Notes in Computer Science, vol. 11082. Cham, Switzerland: Springer. https://dx.doi.org/10.1007/978-3-319-98572-5_25

Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14. https://dx.doi.org/10.3102%2F0013189X015002004

Snow, R. E., Corno, L., & Jackson III, D. N. (1996). Individual differences in affective and conative functions. In D. C. Berliner & R. C. Calfee (Eds.), Handbook of educational psychology (pp. 243-310). New York, NY: Macmillan Library Reference USA.

Turkle, S., & Papert, S. (1992). Epistemological pluralism and the revaluation of the concrete. Journal of Mathematical Behavior, 11(1), 3–33.

Van Harmelen, M., & Workman, D. (2012). Analytics for learning and teaching. CETIS Analytics Series, 1(3), 1–40.

Van Merriënboer, J. J., Kirschner, P. A., & Kester, L. (2003). Taking the load off a learner’s mind: Instructional design for complex learning. Educational Psychologist, 38(1), 5–13. https://dx.doi.org/10.1207/S15326985EP3801_2

Wiley, K. J., Dimitriadis, Y., Bradford, A., & Linn, M. C. (2020). From theory to action: Developing and evaluating learning analytics for learning design. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 569–578). New York, NY: ACM. https://dx.doi.org/10.1145/3375462.3375540

Wise, A. F., & Jung, Y. (2019). Teaching with Analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69. https://dx.doi.org/10.18608/jla.2019.62.4

Worsley, M. (2014). Multimodal learning analytics as a tool for bridging learning theory and complex learning behaviors. Proceedings of the 2014 ACM Workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA ’14), 12–16 November 2014, Istanbul, Turkey (pp. 1–4). New York, NY: ACM. https://dx.doi.org/10.1145/2666633.2666634

Worsley, M., & Blikstein, P. (2013). Towards the development of multimodal action based assessment. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 94–101). New York: ACM. https://dx.doi.org/10.1145/2460296.2460315

Worsley, M., & Blikstein, P. (2015). Leveraging multimodal learning analytics to differentiate student learning strategies. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 360–367). New York, NY: ACM. https://dx.doi.org/10.1145/2723576.2723624

Downloads

Published

2020-12-17

How to Cite

Mangaroska, K., Sharma, K. ., Gasevic, D. ., & Giannakos, M. . (2020). Multimodal Learning Analytics to Inform Learning Design: Lessons Learned from Computing Education. Journal of Learning Analytics, 7(3), 79-97. https://doi.org/10.18608/jla.2020.73.7

Issue

Section

Special Section: Learning Design and Learning Analytics

Most read articles by the same author(s)

<< < 1 2 3 > >>