Multimodal Learning Analytics to Inform Learning Design: Lessons Learned from Computing Education
Keywords:Multimodal learning analytics, Learning design, Physiological measures, Debugging, Predictive modelling
Programming is a complex learning activity that involves coordination of cognitive processes and affective states. These aspects are often considered individually in computing education research, demonstrating limited understanding of how and when students learn best. This issue confines researchers to contextualize evidence-driven outcomes when learning behaviour deviates from pedagogical intentions. Multimodal learning analytics (MMLA) captures data essential for measuring constructs (e.g., cognitive load, confusion) that are posited in the learning sciences as important for learning, and cannot effectively be measured solely with the use of programming process data (IDE-log data). Thus, we augmented IDE-log data with physiological data (e.g., gaze data) and participants’ facial expressions, collected during a debugging learning activity. The findings emphasize the need for learning analytics that are consequential for learning, rather than easy and convenient to collect. In that regard, our paper aims to provoke productive reflections and conversations about the potential of MMLA to expand and advance the synergy of learning analytics and learning design among the community of educators from a post-evaluation design-aware process to a permanent monitoring process of adaptation.
Ahonen, L., Cowley, B., Torniainen, J., Ukkonen, A., Vihavainen, A., & Puolamäki, K. (2016). Cognitive collaboration found in cardiac physiology: Study in classroom environment. PloS One, 11(7), e0159178. https://dx.doi.org/10.1371/journal.pone.0159178
Altadmri, A., & Brown, N. C. (2015). 37 million compilations: Investigating novice programming mistakes in large-scale student data. Proceedings of the 46th ACM Technical Symposium on Computer Science Education (SIGCSE ’15), 4–7 March 2015, Kansas City, MO, USA (pp. 522–527). New York, NY: ACM. https://dx.doi.org/10.1145/2676723.2677258
Andreassi, J. L. (2010). Psychophysiology: Human behavior and physiological response. Hove, East Sussex, UK: Psychology Press.
Arapakis, I., Lalmas, M., & Valkanas, G. (2014). Understanding within-content engagement through pattern analysis of mouse gestures. Proceedings of the 23rd ACM International Conference on Information and Knowledge Management (CIKM ’14), 3–7 November 2014, Shanghai, China (pp. 1439–1448). New York, NY: ACM. https://dx.doi.org/10.1145/2661829.2661909
Azevedo, R. (2015). Defining and measuring engagement and learning in science: Conceptual, theoretical, methodological, and analytical issues. Educational Psychologist, 50(1), 84–94. https://dx.doi.org/10.1080/00461520.2015.1004069
Baker, R. S., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human-Computer Studies, 68(4), 223–241. https://dx.doi.org/10.1016/j.ijhcs.2009.12.003
Bakharia, A., Corrin, L., De Barba, P., Kennedy, G., Gašević, D., Mulder, R., … Lockyer, L. (2016). A conceptual framework linking learning design with learning analytics. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 329–338). New York: ACM. https://dx.doi.org/10.1145/2883851.2883944
Baltrusaitis, T., Zadeh, A., Lim, Y. C., & Morency, L.-P. (2018). Openface 2.0: Facial behavior analysis toolkit. Proceedings of the 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), 15–19 May 2018, Xi’an, China (pp. 59–66). Washington, DC: IEEE Computer Society. https://dx.doi.org/10.1109/FG.2018.00019
Baumeister, R. F., & Vohs, K. D. (2004). Handbook of self-regulation: Research, theory, and applications. New York, NY: The Guilford Press.
Beattie, V., Collins, B., & McInnes, B. (1997). Deep and surface learning: A simple or simplistic dichotomy? Accounting Education, 6(1), 1–12. https://dx.doi.org/10.1080/096392897331587
Beatty, J. (1982). Task-evoked pupillary responses, processing load, and the structure of processing resources. Psychological Bulletin, 91(2), 276. https://dx.doi.org/10.1037/0033-2909.91.2.276
Bednarik, R. (2012). Expertise-dependent visual attention strategies develop over time during debugging with multiple code representations. International Journal of Human–Computer Studies, 70(2), 143–155. https://dx.doi.org/10.1016/j.ijhcs.2011.09.003
Bednarik, R., Busjahn, T., Gibaldi, A., Ahadi, A., Bielikova, M., Crosby, M., … Lister, R. (2020). EMIP: The eye movements in programming dataset. Science of Computer Programming, 198, article 102520. https://dx.doi.org/10.1016/j.scico.2020.102520
Begel, A. (2016). Fun with software developers and biometrics: Invited talk. Proceedings of the 1st International Workshop on Emotion Awareness in Software Engineering (SEmotion ’16), 17 May 2016, Austin, TX, USA (pp. 1–2). New York, NY: ACM. https://dx.doi.org/10.1145/2897000.2897007
Blikstein, P. (2013). Multimodal learning analytics. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 102–106). New York, NY: ACM. https://dx.doi.org/10.1145/2460296.2460316
Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220–238. https://dx.doi.org/10.18608/jla.2016.32.11
Blikstein, P., Worsley, M., Piech, C., Sahami, M., Cooper, S., & Koller, D. (2014). Programming pluralism: Using learning analytics to detect patterns in the learning of computer programming. Journal of the Learning Sciences, 23(4), 561–599. https://dx.doi.org/10.1080/10508406.2014.954750
Bruce, C., Buckingham, L., Hynd, J., McMahon, C., Roggenkamp, M., & Stoodley, I. (2004). Ways of experiencing the act of learning to program: A phenomenographic study of introductory programming students at university. Journal of Information Technology Education: Research, 3(1), 145–160. https://dx.doi.org/10.28945/294
Brunken, R., Plass, J. L., & Leutner, D. (2003). Direct measurement of cognitive load in multimedia learning. Educational Psychologist, 38(1), 53–61. https://dx.doi.org/10.1207/S15326985EP3801_7
Buettner, R. (2013). Cognitive workload of humans using artificial intelligence systems: Towards objective measurement applying eye-tracking technology. Annual Conference on Artificial Intelligence (pp. 37–48). Lecture notes in Computer Science, vol. 8077. https://dx.doi.org/10.1007/978-3-642-40942-4_4
Busjahn, T., Schulte, C., Sharif, B., Begel, A., Hansen, M., Bednarik, R., … Antropova, M. (2014). Eye tracking in computing education. Proceedings of the 10th Annual Conference on International Computing Education Research (ICER ’14), 11–13 August 2014, Glasgow, UK (pp. 3–10). New York, NY: ACM. https://dx.doi.org/10.1145/2632320.2632344
Carter, A. S., & Hundhausen, C. D. (2015). The design of a programming environment to support greater social awareness and participation in early computing courses. Journal of Computing Sciences in Colleges, 31(1), 143–153.
Chan, M. C. E., Ochoa, X., & Clarke, D. (2020). Multimodal learning analytics in a laboratory classroom Machine Learning Paradigms (pp. 131–156): Springer. https://dx.doi.org/10.1007/978-3-030-13743-4_8
Corrin, L., De Barba, P., Lockyear, L., Gašević, D., Williams, D., Dawson, S., … Bakharia, A. (2016). Completing the loop: Returning meaningful learning analytic data to teachers. Retrieved from https://research.monash.edu/en/publications/completing-the-loop-returning-meaningful-learning-analytic-data-t
Craig, S., Graesser, A., Sullins, J., & Gholson, B. (2004). Affect and learning: An exploratory look into the role of affect in learning with AutoTutor. Journal of Educational Media, 29(3), 241–250. https://dx.doi.org/10.1080/1358165042000283101
Crk, I., & Kluthe, T. (2014). Toward using alpha and theta brain waves to quantify programmer expertise. Proceedings of the 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC 2014), 26–30 August 2014, Chicago, IL, USA. Washington, DC: IEEE Computer Society. https://dx.doi.org/10.1109/embc.2014.6944840
Cukurova, M., Giannakos, M., & Martinez‐Maldonado, R. (2020). The promise and challenges of multimodal learning analytics. British Journal of Educational Technology, 51(5), 1441–1449. https://dx.doi.org/10.1111/bjet.13015
D’Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145–157. https://dx.doi.org/10.1016/j.learninstruc.2011.10.001
D’Mello, S., Lehman, B., Pekrun, R., & Graesser, A. (2014). Confusion can be beneficial for learning. Learning and Instruction, 29, 153–170. https://dx.doi.org/10.1016/j.learninstruc.2012.05.003
Dillenbourg, P. (2013). Design for classroom orchestration. Computers & Education, 69, 485–492. https://dx.doi.org/10.1016/j.compedu.2013.04.013
Dörner, D., & Funke, J. (2017). Complex problem solving: What it is and what it is not. Frontiers in Psychology, 8, 1153. https://dx.doi.org/10.3389/fpsyg.2017.01153
Echeverria, V., Martinez-Maldonado, R., & Buckingham Shum, S. (2019). Towards collaboration translucence: Giving meaning to multimodal group data. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), 4–9 May 2019, Glasgow, Scotland, UK (Paper No. 39). New York, NY: ACM. https://dx.doi.org/10.1145/3290605.3300269
Edwards, S. H., & Perez-Quinones, M. A. (2008). Web-CAT: Automatically grading programming assignments. ACM SIGCSE Bulletin, 40(3), 328. https://dx.doi.org/10.1145/1384271.1384371
Ekman, P. (1993). Facial expression and emotion. American Psychologist, 48(4), 384–392. https://dx.doi.org/10.1037/0003-066X.48.4.384
Ekman, P., & Friesen, E. (1978). Facial action coding system: A technique for the measurement of facial movement. Palo Alto, CA: Consulting Psychologists Press.
Ekman, P., Freisen, W. V., & Ancoli, S. (1980). Facial signs of emotional experience. Journal of Personality and Social Psychology, 39(6), 1125-1134. https://dx.doi.org/10.1037/h0077722
Fernández-Delgado, M., Cernadas, E., Barro, S., & Amorim, D. (2014). Do we need hundreds of classifiers to solve real world classification problems? The Journal of Machine Learning Research, 15(1), 3133–3181.
Fritz, T., Begel, A., Müller, S. C., Yigit-Elliott, S., & Züger, M. (2014). Using psycho-physiological measures to assess task difficulty in software development. Proceedings of the 36th International Conference on Software Engineering (ICSE 2014), 31 May–7 June 2014, Hyderabad, India (pp. 402–413). New York, NY: ACM. https://dx.doi.org/10.1145/2568225.2568266
Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. https://dx.doi.org/10.1007/s11528-014-0822-x
Gašević, D., Kovanović, V., & Joksimović, S. (2017). Piecing the learning analytics puzzle: A consolidated model of a field of research and practice. Learning: Research and Practice, 3(1), 63–78. https://dx.doi.org/10.1080/23735082.2017.1286142
Giannakos, M. N., Sharma, K., Pappas, I. O., Kostakos, V., & Velloso, E. (2019). Multimodal data as a means to understand the learning experience. International Journal of Information Management, 48, 108–119. https://dx.doi.org/10.1016/j.ijinfomgt.2019.02.003
Grover, S., Bienkowski, M., Tamrakar, A., Siddiquie, B., Salter, D., & Divakaran, A. (2016). Multimodal analytics to study collaborative problem solving in pair programming. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 516–517). New York: ACM. https://dx.doi.org/10.1145/2883851.2883877
Hernández‐Leo, D., Martinez‐Maldonado, R., Pardo, A., Muñoz‐Cristóbal, J. A., & Rodríguez‐Triana, M. J. (2019). Analytics for learning design: A layered framework and tools. British Journal of Educational Technology, 50(1), 139–152. https://dx.doi.org/10.1111/bjet.12645
Hollender, N., Hofmann, C., Deneke, M., & Schmitz, B. (2010). Integrating cognitive load theory and concepts of human–computer interaction. Computers in Human Behavior, 26(6), 1278–1288. https://dx.doi.org/10.1016/j.chb.2010.05.031
Holmqvist, K., Nyström, M., Andersson, R., Dewhurst, R., Jarodzka, H., & Van de Weijer, J. (2011). Eye tracking: A comprehensive guide to methods and measures. Oxford, UK: Oxford University Press.
Hundhausen, C. D., Olivares, D. M., & Carter, A. S. (2017). IDE-based learning analytics for computing education: A process model, critical review, and research agenda. ACM Transactions on Computing Education (TOCE), 17(3), 11, 1–26. https://dx.doi.org/10.1145/3105759
Ihantola, P., Sorva, J., & Vihavainen, A. (2014). Automatically detectable indicators of programming assignment difficulty. Proceedings of the 15th Annual Conference on Information Technology Education (SIGITE ’14), 15–18 October 2014, Atlanta, GA, USA (pp. 33–38). New York, NY: ACM. https://dx.doi.org/10.1145/2656450.2656476
Ihantola, P., Vihavainen, A., Ahadi, A., Butler, M., Börstler, J., Edwards, S. H., … Rivers, K. (2015). Educational data mining and learning analytics in programming: Literature review and case studies. Proceedings of the 2015 ITiCSE on Working Group Reports (ITICSE-WGR ’15), 4–8 July 2015, Vilnius, Lithuania (pp. 41–63). New York, NY: ACM. https://dx.doi.org/10.1145/2858796.2858798
Jadud, M. C. (2006). Methods and tools for exploring novice compilation behaviour. Proceedings of the 2nd International Workshop on Computing Education Research (ICER ’06), 9–10 September 2006, Canterbury, UK (pp. 73–84). New York, NY: ACM. https://dx.doi.org/10.1145/1151588.1151600
Järvelä, S., Malmberg, J., Haataja, E., Sobocinski, M., & Kirschner, P. A. (2019). What multimodal data can tell us about the students’ regulation of their learning process. Learning and Instruction (in press, corrected proof). https://dx.doi.org/10.1016/j.learninstruc.2019.04.004
Kaller, C. P., Rahm, B., Bolkenius, K., & Unterrainer, J. M. (2009). Eye movements and visuospatial problem solving: Identifying separable phases of complex cognition. Psychophysiology, 46(4), 818–830. https://dx.doi.org/10.1111/j.1469-8986.2009.00821.x
Kevic, K., Walters, B. M., Shaffer, T. R., Sharif, B., Shepherd, D. C., & Fritz, T. (2015). Tracing software developers’ eyes and interactions for change tasks. Proceedings of the 10th Joint Meeting on Foundations of Software Engineering (ESEC/FSE 2015), 30 August–4 September 2015, Bergamo, Italy (pp. 202–213). New York, NY: ACM. https://dx.doi.org/10.1145/2786805.2786864
Kirschner, P. A. (2017). If it ain’t there, it’s broke! Retrieved from https://3starlearningexperiences.wordpress.com/2017/09/05/if-it-aint-there-its-broken/
Kress, G. (2001). Multimodal teaching and learning: The rhetorics of the science classroom. London, UK: A&C Black.
Kurland, D. M., Pea, R. D., Clement, C., & Mawby, R. (1986). A study of the development of programming ability and thinking skills in high school students. Journal of Educational Computing Research, 2(4), 429–458. https://dx.doi.org/10.2190%2FBKML-B1QV-KDN4-8ULH
Lockyer, L., & Dawson, S. (2012). Where learning analytics meets learning design. In S. Buckingham Shum, D. Gašević, & R. Ferguson (Eds.), Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ʼ12), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 14–15). New York, NY: ACM. https://dx.doi.org/10.1145/2330601.2330609
Mangaroska, K., & Giannakos, M. N. (2018). Learning analytics for learning design: A systematic literature review of analytics-driven design to enhance learning. IEEE Transactions on Learning Technologies, 12(4), 516–534. https://dx.doi.org/ 10.1109/TLT.2018.2868673
Mangaroska, K., Sharma, K., Giannakos, M., Træteberg, H., & Dillenbourg, P. (2018). Gaze-driven design insights to amplify debugging skills: A learner-centred analysis approach. Journal of Learning Analytics, 5(3), 98–119. https://dx.doi.org/10.18608/jla.2018.53.7
Marion, B., Impagliazzo, J., St. Clair, C., Soroka, B., & Whitfield, D. (2007). Assessing computer science programs: What have we learned? Proceedings of the 38th ACM Technical Symposium on Computer Science Education (SIGCSE 07), 7–10 March 2007, Covington, KY, USA (pp. 131–132). New York, NY: ACM. https://dx.doi.org/10.1145/1227310.1227358
Martinez-Maldonado, R., Mangaroska, K., Schulte, J., Elliott, D., Axisa, C., & Shum, S. B. (2020). Teacher tracking with integrity: What indoor positioning can reveal about instructional proxemics. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4(1), Article No. 22. https://dx.doi.org/10.1145/3381017
May, J. G., Kennedy, R. S., Williams, M. C., Dunlap, W. P., & Brannan, J. R. (1990). Eye movement indices of mental workload. Acta Psychologica, 75(1), 75–89. https://dx.doi.org/10.1016/0001-6918(90)90067-P
Mayer, R. E. (2010). Unique contributions of eye-tracking research to the study of learning with graphics. Learning and Instruction, 20(2), 167–171. https://dx.doi.org/10.1016/j.learninstruc.2009.02.012
McCauley, R., Fitzgerald, S., Lewandowski, G., Murphy, L., Simon, B., Thomas, L., & Zander, C. (2008). Debugging: A review of the literature from an educational perspective. Computer Science Education, 18(2), 67–92. https://dx.doi.org/10.1080/08993400802114581
McDaniel, B., D’Mello, S., King, B., Chipman, P., Tapp, K., & Graesser, A. (2007). Facial features for affective state detection in learning environments. In D. S. McNamara & G. Trafton (Eds.), Proceedings of the 29th Annual Conference of the Cognitive Science Society (CogSci 2007), 1–4 August 2007, Nashville, TN, USA (pp. 467–472). Austin, TX: Cognitive Science Society. http://csjarchive.cogsci.rpi.edu/Proceedings/2007/docs/p467.pdf
Melero, J., Hernández‐Leo, D., Sun, J., Santos, P., & Blat, J. (2015). How was the activity? A visualization support for a case of location‐based learning design. British Journal of Educational Technology, 46(2), 317–329. https://dx.doi.org/10.1111/bjet.12238
Müller, S. C. (2015). Measuring software developers’ perceived difficulty with biometric sensors. Proceedings of the 37th International Conference on Software Engineering (ICSE ’15), 16–24 May 2015, Florence, Italy (Vol. 2, pp. 887–890). New York, NY: ACM. https://dx.doi.org/ 10.1109/ICSE.2015.284
Nathan, M. J., & Wagner Alibali, M. (2010). Learning sciences. Wiley Interdisciplinary Reviews: Cognitive Science, 1(3), 329–345. https://dx.doi.org/10.1002/wcs.54
Neelen, M., & Kirschner, P. (2017). Where are the learning sciences in learning analytics research? Retrieved from https://3starlearningexperiences.wordpress.com/2017/10/17/where-are-the-learning-sciences-in-learning-analytics-research/
O’Grady, M. J. (2012). Practical problem-based learning in computing education. ACM Transactions on Computing Education (TOCE), 12(3), 1–16. https://dx.doi.org/10.1145/2275597.2275599
Ochoa, X., Domínguez, F., Guamán, B., Maya, R., Falcones, G., & Castells, J. (2018). The RAP system: Automatic feedback of oral presentation skills using multimodal analysis and low-cost sensors. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 360–364). New York, NY: ACM. https://dx.doi.org/10.1145/3170358.3170406
Ochoa, X. (2017). Multimodal learning analytics. In C. Lang, G. Siemens, A. Wise & D. Gašević (Eds.), The Handbook of Learning Analytics (1st Ed.) (pp. 129–141). Society for Learning Analytics Research. https://dx.doi.org/10.18608/hla17
Ochoa, X., & Worsley, M. (2016). Augmenting learning analytics with multimodal sensory data. Journal of Learning Analytics, 3(2), 213–219. https://dx.doi.org/10.18608/jla.2016.32.10
Ogan, A. (2019). Reframing classroom sensing: Promise and peril. Interactions, 26(6), 26–32. Retrieved from https://interactions.acm.org/archive/view/november-december-2019/reframing-classroom-sensing
Olsen, A. (2012, March 20). The Tobii I-VT fixation filter: Algorythm description. Tobii Technology. Retrieved from https://www.tobiipro.com/siteassets/tobii-pro/learn-and-support/analyze/how-do-we-classify-eye-movements/tobii-pro-i-vt-fixation-filter.pdf
Oviatt, S., Grafsgaard, J., Chen, L., & Ochoa, X. (2018, October). Multimodal learning analytics: Assessing learners’ mental state during the process of learning. In S. Oviatt, B. Schuller & P. R. Cohen (Eds.), The handbook of multimodal-multisensor interfaces: Signal processing, architectures, and detection of emotion and cognition (Vol. 2) (pp. 331–374). New York, NY: ACM Books. https://dx.doi.org/10.1145/3107990.3108003
Parnin, C. (2011). Subvocalization: Toward hearing the inner thoughts of developers. Proceedings of the IEEE 19th International Conference on Program Comprehension (ICPC 2011) 22–24 June 2011, Kingston, ON, Canada (pp. 197–200). New York, NY: ACM. https://dx.doi.org/10.1109/ICPC.2011.49
Perkins, D. N., Hancock, C., Hobbs, R., Martin, F., & Simmons, R. (1986). Conditions of learning in novice programmers. Journal of Educational Computing Research, 2(1), 37–55. https://dx.doi.org/10.2190%2FGUJT-JCBJ-Q6QU-Q9PL
Perrotta, C. (2013). Do school‐level factors influence the educational benefits of digital technology? A critical analysis of teachers’ perceptions. British Journal of Educational Technology, 44(2), 314–327. https://dx.doi.org/10.1111/j.1467-8535.2012.01304.x
Persico, D., & Pozzi, F. (2015). Informing learning design with learning analytics to improve teacher inquiry. British Journal of Educational Technology, 46(2), 230–248. https://dx.doi.org/10.1111/bjet.12207
Prieto, L. P., Sharma, K., Kidzinski, Ł., Rodríguez‐Triana, M. J., & Dillenbourg, P. (2018). Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data. Journal of Computer Assisted Learning, 34(2), 193–203. https://dx.doi.org/10.1111/jcal.12232
Reimann, P. (2016). Connecting learning analytics with learning research: The role of design-based research. Learning: Research and Practice, 2(2), 130–142. https://dx.doi.org/10.1080/23735082.2016.1210198
Rienties, B., Nguyen, Q., Holmes, W., & Reedy, K. (2017). A review of ten years of implementation and research in aligning learning design with learning analytics at the Open University UK. Interaction Design and Architecture(s), 33, 134–154.
Rodríguez-Triana, M. J., Prieto, L. P., Martínez-Monés, A., Asensio-Pérez, J. I., & Dimitriadis, Y. (2018). The teacher in the loop: Customizing multimodal learning analytics for blended learning. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 417–426). New York, NY: ACM. https://dx.doi.org/10.1145/3170358.3170364
Schmitz, M., Van Limbeek, E., Greller, W., Sloep, P., & Drachsler, H. (2017). Opportunities and challenges in using learning analytics in learning design. Proceedings of the 12th European Conference on Technology Enhanced Learning (EC-TEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 209–223). Lecture Notes in Computer Science, vol. 10474. Springer. https://dx.doi.org/10.1007/978-3-319-66610-5_16
Schulte, C., Magenheim, J., Müller, K., & Budde, L. (2017). The design and exploration cycle as research and development framework in computing education. Proceedings of the 2017 IEEE Global Engineering Education Conference (EDUCON 2017), 25–28 April 2017, Athens, Greece (pp. 867–876). Washington, DC: IEEE Computer Society. https://dx.doi.org/10.1109/EDUCON.2017.7942950
Sharma, K., Mangaroska, K., Trætteberg, H., Lee-Cultura, S., & Giannakos, M. (2018). Evidence for programming strategies in university coding exercises. Proceedings of the 13th European Conference on Technology Enhanced Learning (EC-TEL 2018), 3–5 September 2018, Leeds, UK (pp. 326–339). Lecture Notes in Computer Science, vol. 11082. Cham, Switzerland: Springer. https://dx.doi.org/10.1007/978-3-319-98572-5_25
Shulman, L. S. (1986). Those who understand: Knowledge growth in teaching. Educational Researcher, 15(2), 4–14. https://dx.doi.org/10.3102%2F0013189X015002004
Snow, R. E., Corno, L., & Jackson III, D. N. (1996). Individual differences in affective and conative functions. In D. C. Berliner & R. C. Calfee (Eds.), Handbook of educational psychology (pp. 243-310). New York, NY: Macmillan Library Reference USA.
Turkle, S., & Papert, S. (1992). Epistemological pluralism and the revaluation of the concrete. Journal of Mathematical Behavior, 11(1), 3–33.
Van Harmelen, M., & Workman, D. (2012). Analytics for learning and teaching. CETIS Analytics Series, 1(3), 1–40.
Van Merriënboer, J. J., Kirschner, P. A., & Kester, L. (2003). Taking the load off a learner’s mind: Instructional design for complex learning. Educational Psychologist, 38(1), 5–13. https://dx.doi.org/10.1207/S15326985EP3801_2
Wiley, K. J., Dimitriadis, Y., Bradford, A., & Linn, M. C. (2020). From theory to action: Developing and evaluating learning analytics for learning design. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 569–578). New York, NY: ACM. https://dx.doi.org/10.1145/3375462.3375540
Wise, A. F., & Jung, Y. (2019). Teaching with Analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69. https://dx.doi.org/10.18608/jla.2019.62.4
Worsley, M. (2014). Multimodal learning analytics as a tool for bridging learning theory and complex learning behaviors. Proceedings of the 2014 ACM Workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA ’14), 12–16 November 2014, Istanbul, Turkey (pp. 1–4). New York, NY: ACM. https://dx.doi.org/10.1145/2666633.2666634
Worsley, M., & Blikstein, P. (2013). Towards the development of multimodal action based assessment. Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (LAK ’13), 8–12 April 2013, Leuven, Belgium (pp. 94–101). New York: ACM. https://dx.doi.org/10.1145/2460296.2460315
Worsley, M., & Blikstein, P. (2015). Leveraging multimodal learning analytics to differentiate student learning strategies. Proceedings of the 5th International Conference on Learning Analytics and Knowledge (LAK ʼ15), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 360–367). New York, NY: ACM. https://dx.doi.org/10.1145/2723576.2723624
How to Cite
Copyright (c) 2020 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) license that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).