Advancing Multimodal Collaboration Analytics
A Scoping Review
DOI:
https://doi.org/10.18608/jla.2025.8625Keywords:
multimodal collaboration analytics, MMCA, collaborative learning, scoping review, research paperAbstract
The advent of advanced technology has opened new horizons for studying collaborative learning, although ambiguity remains in the classification and rationale for combining modalities in multimodal collaboration analytics (MMCA). Addressing this gap is crucial for the progression of collaborative learning practices and research. This review critically examines and classifies the modalities employed in MMCA studies while elucidating the rationales for their combined use and their resulting empirical contributions to collaborative learning research. A scoping review of 36 empirical studies informs the development of a framework for classifying modalities used in MMCA. We also review the rationales underlying the use of different combinations of modalities and how MMCA literature contributes to our understanding of collaboration. The review results in a definitional framework comprising five categories: auditory, visual, physiological, kinesthetic, and tactile. The findings reveal diverse arrangements of modalities. We find that the underlying rationales for combining modalities are based on technical, practical/pedagogical, methodological, or theoretical premises, which lead to different empirical contributions. Conducting MMCA research is motivated by the need for a holistic comprehension of learner behaviours, interactions, and cognitive processes during collaboration, transcending the limitations of single modalities in isolation. These findings offer both a theoretical and practical guidepost for enhancing MMCA research and applications.
References
Andriessen, J., & Baker, M. (2020). On collaboration: Personal, educational and societal arenas. Sense-Brill.
Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616
Blikstein, P., & Worsley, M. (2016). Multimodal learning analytics and education data mining: Using computational technologies to measure complex learning tasks. Journal of Learning Analytics, 3(2), 220–238. https://doi.org/10.18608/jla.2016.32.11
Booth, A., Sutton, A., Clowes, M., & Martyn-St James, M. (2022). Systematic approaches to a successful literature review (3rd ed.). SAGE Publications.
Buckingham Shum, S., Echeverria, V., & Martinez-Maldonado, R. (2019). The multimodal matrix as a quantitative ethnography methodology. In B. Eagan, M. Misfeldt, & A. Siebert-Evenstone (Eds.), Advances in quantitative ethnography: First international conference, ICQE 2019, Madison, WI, USA, October 20–22, 2019, proceedings (pp. 26–40). Springer Cham. https://doi.org/10.1007/978-3-030-33232-7_3
Bunt, H., Beun, R.-J., & Borghuis, T. (1998). Multimodal human–computer communication: Systems, techniques, and experiments. Springer Berlin, Heidelberg. https://doi.org/10.1007/BFb0052309
Chejara, P., Prieto, L. P., Ruiz-Calleja, A., Rodríguez-Triana, M. J., Shankar, S. K., & Kasepalu, R. (2020). Quantifying collaboration quality in face-to-face classroom settings using MMLA. In A. Nolte, C. Alvarez, R. Hishiyama, I.-A. Chounta, M. J. Rodríguez-Triana, & T. Inoue (Eds.), Collaboration technologies and social computing: 26th international conference, CollabTech 2020, Tartu, Estonia, September 8–11, 2020, proceedings (pp. 159–166). Springer Cham. https://doi.org/10.1007/978-3-030-58157-2_11
Chua, Y. H. V., Dauwels, J., & Tan, S. C. (2019). Technologies for automated analysis of co-located, real-life, physical learning spaces: Where are we now? In C. Brooks, R. Ferguson, & U. Hoppe (Program chairs), Learning Analytics to Promote Inclusion and Success: Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 11–20). ACM Press. https://doi.org/10.1145/3303772.3303811
Echeverria, V., Martinez-Maldonado, R., & Buckingham Shum, S. (2019). Towards collaboration translucence: Giving meaning to multimodal group data. In S. Brewster, G. Fitzpatrick, A. Cox, & V. Kostakos (Chairs), CHI 2019: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Paper 39). ACM Press. https://doi.org/10.1145/3290605.3300269
Fernandez Nieto, G. M., Kitto, K., Buckingham Shum, S., & Martinez-Maldonado, R. (2022). Beyond the learning analytics dashboard: Alternative ways to communicate student data insights combining visualisation, narrative and storytelling. In A. F. Wise, R. Martinez-Maldonado, & I. Hilliger (Program chairs), Learning Analytics for Transition, Disruption and Social Change: The Twelfth International Conference on Learning Analytics & Knowledge (pp. 219–229). ACM Press. https://doi.org/10.1145/3506860.3506895
Fernandez-Nieto, G. M., Echeverria, V., Buckingham Shum, S., Mangaroska, K., Kitto, K., Palominos, E., Axisa, C., & Martinez-Maldonado, R. (2021). Storytelling with learner data: Guiding student reflection on multimodal team data. IEEE Transactions on Learning Technologies, 14(5), 695–708. https://doi.org/10.1109/TLT.2021.3131842
Huang, K., Bryant, T., & Schneider, B. (2019). Identifying collaborative learning states using unsupervised machine learning on eye-tracking, physiological and motion sensor data. In C. F. Lynch, A. Merceron, M. Desmarais, & R. Nkambou (Eds.), Proceedings of the 12th International Conference on Educational Data Mining (pp. 318–323).
Giannakos, M., & Cukurova, M. (2023). The role of learning theory in multimodal learning analytics. British Journal of Educational Technology, 54(5), 1246–1267. https://doi.org/10.1111/bjet.13320
Di Mitri, D., Schneider, J., Specht, M., & Drachsler, H. (2018). From signals to knowledge: A conceptual model for multimodal learning analytics. Journal of Computer Assisted Learning, 34(4), 338–349. https://doi.org/10.1111/jcal.12288
Fritzsch, B. (Ed.). (2021). The senses: A comprehensive reference (2nd Ed.). Elsevier.
Graesser, A. C., Greiff, S., Stadler, M., & Shubeck, K. T. (2020). Collaboration in the 21st century: The theory, assessment, and teaching of collaborative problem solving. Computers in Human Behavior, 104, Article 106134. https://doi.org/10.1016/j.chb.2019.09.010
Kim, Y.-J., & Yoo, J.-H. (2020). The utilization of debriefing for simulation in healthcare: A literature review. Nurse Education in Practice, 43, Article 102698. https://doi.org/10.1016/j.nepr.2020.102698
Kim, T., Chang, A., Holland, L., & Pentland, A. S. (2008). Meeting mediator: Enhancing group collaboration using sociometric feedback. In B. Begole & D. W. McDonald (General chairs), Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (pp. 457–466). ACM Press. https://doi.org/10.1145/1460563.1460636
Ma, Y., Celepkolu, M., & Boyer, K. E. (2022). Detecting impasse during collaborative problem solving with multimodal learning analytics. In A. F. Wise, R. Martinez-Maldonado, & I. Hilliger (Program chairs), Learning Analytics for Transition, Disruption and Social Change: The Twelfth International Learning Analytics & Knowledge Conference (pp. 45–55). ACM Press. https://doi.org/10.1145/3506860.3506865
Mao, A., Mason, W., Suri, S., & Watts, D. J. (2016). An experimental study of team size and performance on a complex task. PLoS ONE, 11(4), Article e0153048. https://doi.org/10.1371/journal.pone.0153048
Madan, A., Caneel, R., & Pentland, A. S. (2004). GroupMedia: Distributed multi-modal interfaces. Proceedings of the 6th International Conference on Multimodal Interfaces (pp. 309–316). ACM Press. https://doi.org/10.1145/1027933.1027983
Malmberg, J., Järvelä, S., Holappa, J., Haataja, E., Huang, X., & Siipo, A. (2019). Going beyond what is visible: What multichannel data can reveal about interaction in the context of collaborative learning? Computers in Human Behavior, 96, 235–245. https://doi.org/10.1016/j.chb.2018.06.030
Martinez-Maldonado, R., Dimitriadis, Y., Martinez-Monés, A., Kay, J., & Yacef, K. (2013). Capturing and analyzing verbal and physical collaborative learning interactions at an enriched interactive tabletop. International Journal of Computer-Supported Collaborative Learning, 8(4), 455–485. https://doi.org/10.1007/s11412-013-9184-1
Martinez-Maldonado, R., Echeverria, V., Fernandez Nieto, G., & Buckingham Shum, S. (2020). From data to insights: A layered storytelling approach for multimodal learning analytics. In R. Bernhaupt, F. Mueller, D. Verwrij, J. Andres, J. McGrenere, A. Cockburn, I. Avellino, A. Goguey, P. Bjørn, S. Zhao, B. P. Samson, & R. Kocielnik (Chairs), CHI’20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Paper 21). ACM Press. https://doi.org/10.1145/3313831.3376148
Martinez-Maldonado, R., Gašević, D., Echeverria, V., Fernandez Nieto, G., Swiecki, Z., & Buckingham Shum, S. (2021). What do you mean by collaboration analytics? A conceptual model. Journal of Learning Analytics, 8(1), 126–153. https://doi.org/10.18608/jla.2021.7227
Martinez-Maldonado, R., Kay, J., Buckingham Shum, S., & Yacef, K. (2019). Collocated collaboration analytics: Principles and dilemmas for mining multimodal interaction data. Human–Computer Interaction, 34(1), 1–50. https://doi.org/10.1080/07370024.2017.1338956
Martinez-Maldonado, R., Power, T., Hayes, C., Abdiprano, A., Vo, T., Axisa, C., & Buckingham Shum, S. (2017). Analytics meet patient manikins: Challenges in an authentic small-group healthcare simulation classroom. In A. F. Wise, P. H. Winne, G. Lynch, X. Ochoa, I. Molenaar, S. Dawson, & M. Hatala (Chairs), Understanding, Informing and Improving Learning with Data: The Seventh International Learning Analytics & Knowledge Conference (pp. 90–94). ACM Press. https://doi.org/10.1145/3027385.3027401
Marlow, S., Bisbey, T., Lacerenza, C., & Salas, E. (2018). Performance measures for health care teams: A review. Small Group Research, 49(3), 306–356. https://doi.org/10.1177/1046496417748196
Moher, D., Liberati, A., Tetzlaff, J., Altman, D. G., & The PRISMA Group*. (2009). Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Medicine, 6(7), Article e1000097. https://doi.org/10.1371/journal.pmed.1000097
Muukkonen, H., Lakkala, M., Lahti-Nuuttila, P., Ilomäki, L., Karlgren, K., & Toom, A. (2020). Assessing the development of collaborative knowledge work competence: Scales for higher education course contexts. Scandinavian Journal of Educational Research, 64(7), 1071–1089. https://doi.org/10.1080/00313831.2019.1647284
Munn, Z., Peters, M. J. D., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18, Article 143. https://doi.org/10.1186/s12874-018-0611-x
Müller, P., Huang, M. X., & Bulling, A. (2018). Detecting low rapport during natural interactions in small groups from non-verbal behaviour. In S. Berkovsky, Y. Hijikata, J. Rekimoto, M. Burnett, M. Billinghurst, & A. Quigley (Chairs), IUI 2018: Proceedings of the 23rd International Conference on Intelligent User Interfaces (pp. 153–164). ACM Press. https://doi.org/10.1145/3172944.3172969
Nakano, Y. I., Nihonyanagi, S., Takase, Y., Hayashi, Y., & Okada, S. (2015). Predicting participation styles using co-occurrence patterns of nonverbal behaviours in collaborative learning. In Z. Zhang, P. Cohen, D. Bohus, R. Horaud, & H. Meng (Eds.), ICMI’15: Proceedings of the 2015 ACM International Conference on Multimodal Interaction (pp. 91–98). ACM Press. https://doi.org/10.1145/2818346.2820764
Neubauer, C., Woolley, J., Khooshabeh, P., & Scherer, S. (2016). Getting to know you: A multimodal investigation of team behavior and resilience to stress. In Y. I. Nakano, E. André, T. Nishida, L.-P. Morency, C. Busso, & C. Pelachaud (Chairs), ICMI’16: Proceedings of the 18th ACM International Conference on Multimodal Interaction (pp. 193–200). ACM Press. https://doi.org/10.1145/2993148.2993195
Noroozi, O., Alikhani, I., Järvelä, S., Kirschner, P. A., Juuso, I., & Seppänen, T. (2019). Multimodal data to design visual learning analytics for understanding regulation of learning. Computers in Human Behavior, 100, 298–304. https://doi.org/10.1016/j.chb.2018.12.019
Noël, R., Miranda, D., Cechinel, C., Riquelme, F., Primo, T. T., & Munoz, R. (2022). Visualizing collaboration in teamwork: A multimodal learning analytics platform for non-verbal communication. Applied Sciences, 12(15), Article 7499. https://doi.org/10.3390/app12157499
Ochoa, X., Chiluiza, K., Granda, R., Falcones, G., Castells, J., & Guamán, B. (2018). Multimodal transcript of face-to-face group-work activity around interactive tabletops. In A. Pardo, K. Bartimote, G. Lynch, S. Buckingham Shum, R. Ferguson, A. Merceron, & X. Ochoa (Eds.), Companion Proceedings of the 8th International Conference on Learning Analytics & Knowledge. SoLAR.
Olsen, J. K., Sharma, K., Rummel, N., & Aleven, V. (2020). Temporal analysis of multimodal data to predict collaborative learning outcomes. British Journal of Educational Technology, 51(5), 1527–1547. https://doi.org/10.1111/bjet.12982
Ouhaichi, H., Spikol, D., & Vogel, B. (2021). MBOX: Designing a flexible IoT multimodal learning analytics system. In M. Chang, N.-S. Chen, D. G. Sampson, & A. Tlili (Eds.), 2021 International Conference on Advanced Learning Technologies (ICALT) (pp. 122–126). IEEE. https://doi.org/10.1109/ICALT52272.2021.00044
Oviatt, S., & Cohen, A. (2014). Written activity, representations and fluency as predictors of domain expertise in mathematics. In A. A. Salah, J. Cohn, B. Schuller, O. Aran, L.-P. Morency, & P. R. Cohen (Chairs), ICMI’14: Proceedings of the 2014 International Conference on Multimodal Interaction (pp. 10–17). ACM Press. https://doi.org/10.1145/2663204.2663245
Praharaj, S., Scheffel, M., Drachsler, H., & Specht, M. (2021). Literature review on co-located collaboration modeling using multimodal learning analytics: Can we go the whole nine yards? IEEE Transactions on Learning Technologies, 14(3), 367–385. https://doi.org/10.1109/TLT.2021.3097766
Peng, S., & Nagao, K. (2021). Recognition of students’ mental states in discussion based on multimodal data and its application to educational support. IEEE Access, 9, 18235–18250. https://doi.org/10.1109/ACCESS.2021.3054176
Praharaj, S., Scheffel, M., Drachsler, H., & Specht, M. (2018). Multimodal analytics for real-time feedback in co-located collaboration. In V. Pammer-Schindler, M. Pérez-Sanagustín, H. Drachsler, R. Elferink, & M. Scheffel (Eds.), Lifelong technology-enhanced learning: 13th European conference on technology enhanced learning, EC-TEL 2018, Leeds, UK, September 3–5, 2018, proceedings (pp. 187–201). Springer Cham. https://doi.org/10.1007/978-3-319-98572-5_15
Quek, F., McNeill, D., Bryll, R., Duncan, S., Ma, X.-F., Kirbas, C., McCullough, K. E., & Ansari, R. (2002). Multimodal human discourse: Gesture and speech. ACM Transactions on Computer–Human Interaction, 9(3), 171–193. https://doi.org/10.1145/568513.568514
Reilly, J. M., & Schneider, B. (2019). Predicting the quality of collaborative problem solving through linguistic analysis of discourse. In C. F. Lynch, A. Merceron, M. Desmarais, & R. Nkambou (Eds.), Proceedings of the 12th International Conference on Educational Data Mining (EDM 2019) (pp. 149–157). International Educational Data Mining Society. https://educationaldatamining.org/edm2019/proceedings/
Reimann, P., Markauskaite, L., & Bannert, M. (2014). e-Research and learning theory: What do sequence and process mining methods contribute? British Journal of Educational Technology, 45(3), 528–540. https://doi.org/10.1111/bjet.12146
Roschelle, J., & Teasley, S. D. (1995). The construction of shared knowledge in collaborative problem solving. In C. O’Malley (Ed.), Computer supported collaborative learning (pp. 69–97). NATO ASI Series, vol 128. Springer. https://doi.org/10.1007/978-3-642-85098-1_5
Scherer, S., Weibel, N., Morency, L.-P., & Oviatt, S. (2012). Multimodal prediction of expertise and leadership in learning groups. MLA’12: Proceedings of the 1st International Workshop on Multimodal Learning Analytics (Article 1). ACM Press. https://doi.org/10.1145/2389268.2389269
Schneider, B., & Pea, R. (2015). Does seeing one another’s gaze affect group dialogue? A computational approach. Journal of Learning Analytics, 2(2), 107–133. https://doi.org/10.18608/jla.2015.22.9
Schneider, B., Sung, G., Chng, E., & Yang, S. (2021). How can high-frequency sensors capture collaboration? A review of the empirical links between multimodal metrics and collaborative constructs. Sensors, 21(24), Article 8185. https://doi.org/10.3390/s21248185
Sinclair, A. J., & Schneider, B. (2021). Linguistic and gestural coordination: Do learners converge in collaborative dialogue? In S. Hsiao, S. Sahebi, F. Bouchet, & J.-J. Vie (Eds.), Proceedings of the 14th International Conference on Educational Data Mining (pp. 431–438). International Educational Data Mining Society. https://files.eric.ed.gov/fulltext/ED615472.pdf
Spikol, D., Ruffaldi, E., & Cukurova, M. (2017). Using multimodal learning analytics to identify aspects of collaboration in project-based learning. In B. K. Smith, M. Borge, E. Mercier, & K. Y. Lim (Eds.), Making a difference: Prioritizing equity and access in CSCL, 12th International Conference on Computer Supported Collaborative Learning (CSCL) 2017 (pp. 263–270). International Society of the Learning Sciences. https://repository.isls.org/handle/1/240
Spikol, D., Ruffaldi, E., Landolfi, L., & Cukurova, M. (2017). Estimation of success in collaborative learning based on multimodal learning analytics features. In M. Chang, N.-S. Chen, R. Huang, Kinshuk, D. G. Sampson, & R. Vasiu (Eds.), The 17th IEEE International Conference on Advanced Learning Technologies (ICALT 2017) (pp. 269–273). IEEE. https://doi.org/10.1109/ICALT.2017.122
Spikol, D., Ruffaldi, E., Dabisias, G., & Cukurova, M. (2018). Supervised machine learning in multimodal learning analytics for estimating success in project-based learning. Journal of Computer Assisted Learning, 34(4), 366–377. https://doi.org/10.1111/jcal.12263
Stewart, A. E. B., Keirn, Z., & D’Mello, S. K. (2021). Multimodal modeling of collaborative problem-solving facets in triads. User Modelling and User-Adapted Interaction, 31(4), 713–751. https://doi.org/10.1007/s11257-021-09290-y
Sturm, J., Herwijnen, O. H., Eyck, A., & Terken, J. (2007). Influencing social dynamics in meetings through a peripheral display. In K. Mase, D. Massaro, K. Takeda, D. Roy, & A. Potamianos (Chairs), ICMI’07: Proceedings of the Ninth International Conference on Multimodal Interfaces (pp. 263–270). ACM Press. https://doi.org/10/b3fcjc
Sun, C., Shute, V. J., Stewart, A., Yonehiro, J., Duran, N., & D’Mello, S. (2020). Towards a generalized competency model of collaborative problem solving, Computers & Education, 143, Article 103672. https://doi.org/10.1016/j.compedu.2019.103672
Theobald, E. J., Eddy, S. L., Grunspan, D. Z., Wiggins, B. L., & Crowe, A. J. (2017). Student perception of group dynamics predicts individual performance: Comfort and equity matter. PLoS ONE, 12(7), Article e0181336. https://doi.org/10.1371/journal.pone.0181336
Turk, M. (2014). Multimodal interaction: A review. Pattern Recognition Letters, 36, 189–195. https://doi.org/10.1016/j.patrec.2013.07.003
Viswanathan, S. A., & VanLehn, K. (2018). Using the tablet gestures and speech of pairs of students to classify their collaboration. IEEE Transactions on Learning Technologies, 11(2), 230–242. https://doi.org/10/gds4fq
Vrzakova, H., Amon, M. J., Stewart, A., Duran, N. D., & D’Mello, S. K. (2020). Focused or stuck together: Multimodal patterns reveal triads’ performance in collaborative problem solving. In C. Rensing, H. Drachsler, V. Kovanović, N. Pinkwart, M. Scheffel, & K. Verbert (Chairs), Celebrating 10 years of LAK: Shaping the Future of the Field: The Tenth International Conference on Learning Analytics & Knowledge (pp. 295–304). ACM Press. https://doi.org/10.1145/3375462.3375467
Worsley, M., & Blikstein, P. (2014). Deciphering the practices and affordances of different reasoning strategies through multimodal learning analytics. In X. Ochoa, M. Worsley, K. Chiluiza, & S. Luz (Chairs), MLA’14: Proceedings of the 2014 ACM Multimodal Learning Analytics Workshop and Grand Challenge (pp. 21–27). ACM Press. https://doi.org/10.1145/2666633.2666637
Wise, A. F., Knight, S., & Buckingham Shum, S. (2021). Collaborative learning analytics. In U. Cress, C. Rosé, A. F. Wise & J. Oshima (Eds.), International handbook of computer-supported collaborative learning (pp. 425–443). Springer Cham. https://doi.org/10.1007/978-3-030-65291-3_23
Zhao, L., Swiecki, Z., Gašević, D., Yan, L., Dix, S., Jaggard, H., Wotherspoon, R., Osborne, A., Li, X., Alfredo, R., & Martinez-Maldonado, R. (2023). METS: Multimodal learning analytics of embodied teamwork learning. In I. Hilliger, H. Khosravi, B. Rienties, & S. Dawson (Chairs), Towards Trustworthy Learning Analytics: The Thirteenth International Conference on Learning Analytics & Knowledge (pp. 186–196). ACM Press. https://doi.org/10.1145/3576050.3576076
Zhao, L., Yan, L., Gašević, D., Dix, S., Jaggard, H., Wotherspoon, R., Alfredo, R., Li, X., & Martinez-Maldonado, R. (2022). Modelling co-located team communication from voice detection and positioning data in healthcare simulation. In A. F. Wise, R. Martinez-Maldonado, & I. Hilliger (Chairs), Learning Analytics for Transition, Disruption and Social Change: The Twelfth International Conference on Learning Analytics & Knowledge (pp. 370–380). ACM Press. https://doi.org/10.1145/3506860.3506935
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
