Charting the Development of Collaboration Skills Through Collaborative Learning Analytics Systems
DOI:
https://doi.org/10.18608/jla.2025.8523Keywords:
collaboration skills, collaborative learning analytics, feedback, multimodal learning analytics, research paperAbstract
Collaboration skills are fundamental to effective collaborative learning, career success, and responsible citizenship. Collaborative learning analytics (CLA) systems hold significant potential in helping students develop these skills by automatically collecting group interaction data, analyzing skill levels, and providing actionable feedback so students can reflect, practise, and improve. Previously, most collaborative feedback systems have focused on improving collaborative processes rather than serving as instructional systems for developing collaboration skills over time. To identify what is needed to navigate toward this new type of tool, our paper proposes an interdisciplinary framework that serves as a guiding compass for designing and evaluating such systems. Through an extensive literature review, we evaluate 15 selected systems through the lens of each element of this framework. We map out the current state of the field and identify four major gaps that need to be addressed to transition from systems that support collaboration to systems that support the development of collaboration skills. These gaps are unexplored collaboration skills, lack of validated indicators, limited modelling techniques, and pedagogical feedback design. Finally, we propose a set of corresponding research agendas to bridge these gaps, providing a forward-looking roadmap for designing effective and actionable CLA systems for collaboration skills development.
References
Bachour, K., Kaplan, F., & Dillenbourg, P. (2010). An interactive table for supporting participation balance in face-to-face collaborative learning. IEEE Transactions on Learning Technologies, 3(3), 203–213. https://doi.org/10.1109/TLT.2010.18
Balaam, M., Fitzpatrick, G., Good, J., & Harris, E. (2011, May). Enhancing interactional synchrony with an ambient display. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (SIGCHI 2011), 7–12 May 2011, Vancouver, British Columbia, Canada (pp. 867–876). ACM. https://doi.org/10.1145/1978942.1979070
Bergstrom, T., & Karahalios, K. (2007). Conversation Clock: Visualizing audio patterns in co-located groups. In R. Sprague (Ed.), Proceedings of the 2007 40th Annual Hawaii International Conference on System Sciences (HICSS 2007), 3–6 January, 2007, Waikoloa, Hawaii, USA (p. 78). IEEE. https://doi.org/10.1109/HICSS.2007.151
Bodie, G. D., Vickery, A. J., Cannava, K., & Jones, S. M. (2015). The role of “active listening” in informal helping conversations: Impact on perceptions of listener helpfulness sensitivity and supportiveness and discloser emotional improvement. Western Journal of Communication, 79(2), 151–173. https://doi.org/10.1080/10570314.2014.943429
Boothe, M., Yu, C., Lewis, A., & Ochoa, X. (2022). Towards a pragmatic and theory-driven framework for multimodal collaboration feedback. In Proceedings of the 12th International Conference on Learning Analytics and Knowledge (LAK 2022), 21–25 March 2022, online (pp. 507–513). ACM. https://doi.org/10.1145/3506860.3506898
Cronbach, L. J., & Meehl, P. E. (1955). Construct validity in psychological tests. Psychological Bulletin, 52(4), 281. https://doi.org/10.1037/h0040957
Cukurova, M., Luckin, R., Mill ´an, E., & Mavrikis, M. (2018). The NISPI framework: Analysing collaborative problem-solving from students’ physical interactions. Computers & Education, 116, 93–109. https://doi.org/10.1016/j.compedu.2017.08.007
D’Angelo, S., & Gergle, D. (2016). Gazed and confused: Understanding and designing shared gaze for remote collaboration. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016), 7–12 May 2016, San Jose, California, USA (pp. 2492–2496). ACM. https://doi.org/10.1145/2858036.2858499
Di Mitri, D., Schneider, J., Specht, M., & Drachsler, H. (2018). From signals to knowledge: A conceptual model for multimodal learning analytics. Journal of Computer Assisted Learning, 34(4), 338–349. https://doi.org/10.1111/jcal.12288
DiMicco, J. M., Pandolfo, A., & Bender, W. (2004). Influencing group participation with a shared display. In Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (CSCW 2004), 6–10 November 2004, Chicago, Illinois, USA (pp. 614–623). ACM. https://doi.org/10.1145/1031607.1031713
D’Mello, S. K., Duran, N., Michaels, A., & Stewart, A. E. B. (2024). Improving collaborative problem-solving skills via automated feedback and scaffolding: A quasi-experimental study with CPSCoach 2.0. In J. Masthoff (Ed.), User modeling and user-adapted interaction (pp. 1087–1125, Vol. 34). Springer Nature Link. https://doi.org/10.1007/s11257-023-09387-6
Echeverria, V., Martinez-Maldonado, R., Chiluiza, K., & Shum, S. B. (2017). DBCollab: Automated feedback for face-to-face group database design. In Proceedings of the 25th International Conference on Computers in Education (ICCE 2017), Main Conference Proceedings, 4–8 December 2017, Christchurch, New Zealand. APSCE. https://library.apsce.net/index.php/ICCE/article/view/2240
Echeverria Barzola, V. I. (2020). Designing feedback for collocated teams using multimodal learning analytics [PhD]. University of Technology Sydney. https://hdl.handle.net/10453/140936
Giannakos, M., & Cukurova, M. (2023). The role of learning theory in multimodal learning analytics. British Journal of Educational Technology, 54(5), 1246–1267. https://doi.org/10.1111/bjet.13320
Hao, J., Liu, L., Kyllonen, P., Flor, M., & von Davier, A. A. (2019). Psychometric considerations and a general scoring strategy for assessments of collaborative problem solving. ETS Research Report Series, 2019(1), 1–17. https://doi.org/10.1002/ets2.12276
Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487
Hu, L., & Chen, G. (2021). A systematic review of visual representations for analyzing collaborative discourse. Educational Research Review, 34, 100403. https://doi.org/10.1016/j.edurev.2021.100403
Janssen, J., & Bodemer, D. (2013). Coordinated computer-supported collaborative learning awareness and awareness tools. Educational Psychologist, 48(1), 40–55. https://doi.org/10.1080/00461520.2012.749153
Jarvela, S., & Hadwin, A. F. (2013). New frontiers: Regulating learning in CSCL. Educational Psychologist, 48(1), 25–39. https://doi.org/10.1080/00461520.2012.748006
Jung, Y., & Wise, A. F. (2024). Probing actionability in learning analytics: The role of routines, timing, and pathways. In Proceedings of the 14th International Conference on Learning Analytics and Knowledge (LAK 2024), 18–22 March 2024, Kyoto, Japan (pp. 871–877). ACM. https://doi.org/10.1145/3636555.3636914
Karahalios, K., & Bergstrom, T. (2009). Social mirrors as social signals: Transforming audio into graphics. IEEE Computer Graphics and Applications, 29(5), 22–32. https://doi.org/10.1109/MCG.2009.85
Kim, T., Chang, A., Holland, L., & Pentland, A. S. (2008). Meeting mediator: Enhancing group collaboration using sociometric feedback. Proceedings of the 2008 ACM Conference on Computer Supported Cooperative Work (CSCW 2008), 8–12 November 2008, San Diego, California, USA, 457–466. https://doi.org/10.1145/1460563.1460636
Koh, E., Tee, Y.-H., Suresh, D., & Caleon, I. (2020). A digital formative assessment intervention for cultivating teamwork skills. In M. Gresalfi & I. S. Horn (Eds.), The interdisciplinarity of the learning sciences, 14th international conference of the learning sciences (ICLS) 2020 (pp. 1597–1600, Vol. 3). International Society of the Learning Sciences (ISLS). https://repository.isls.org//handle/1/6375
Kulyk, O., Wang, J., & Terken, J. (2005). Real-time feedback on nonverbal behaviour to enhance social dynamics in small group meetings. In S. Renals & S. Bengio (Eds.), Machine learning for multimodal interaction. MLMI 2005. Lecture notes in computer science (Vol. 3869). Springer. https://doi.org/10.1007/11677482_13
Lai, E., DiCerbo, K., & Foltz, P. (2017). Skills for today: What we know about teaching and assessing collaboration (tech. rep.). Pearson. https://www.pearson.com/content/dam/one-dot-com/one-dot-com/global/Files/efficacy-and-research/skills-for-today/Collaboration-FullReport.pdf
Lim, L., Dawson, S., Joksimovic, S., & Gasevic, D. (2019). Exploring students’ sensemaking of learning analytics dashboards: Does frame of reference make a difference? In Proceedings of the Ninth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, Arizona, USA (pp. 250–259). ACM. https://doi.org/10.1145/3303772.3303804
Lipnevich, A. A., Berg, D. A. G., & Smith, J. K. (2016). Toward a model of student response to feedback. In G. T. L. Brown & L. R. Harris (Eds.), Handbook of human and social conditions in assessment (pp. 169–185). Routledge. https://doi.org/10.4324/9781315749136
Liu, O. L., Ling, G., & Fishtein, D. (2021). Charting the skills for the future: Perspectives from employers and higher education institutions. Chinese/English Journal of Educational Measurement and Evaluation, 2(2). https://doi.org/10.59863/VNWR4406
Lyndgaard, S. F., & Kanfer, R. (2024). Interpersonal, intrapersonal, and cognitive tactics: A thematic analysis of adults’ 21st century learning management. Journal of the Learning Sciences, 33(1), 175–211. https://doi.org/10.1080/10508406.2024.2303777
Meier, A., Spada, H., & Rummel, N. (2007). A rating scheme for assessing the quality of computer-supported collaboration processes. International Journal of Computer-Supported Collaborative Learning, 2(1), 63–86. https://doi.org/10.1007/s11412-006-9005-x
Narciss, S. (2013). Designing and evaluating tutoring feedback strategies for digital learning environments on the basis of the Interactive Tutoring Feedback Model. Digital Education Review, 23, 7–26. https://www.learntechlib.org/p/131614/
Narciss, S., & Huth, K. (2004). How to design informative tutoring feedback for multimedia learning. In H. M. Niegemann, D. Leutner, & R. Brunken (Eds.), Instructional design for multimedia learning: Proceedings of the Fifth International Workshop of SIG 6 Instructional Design of the European Association for Research on Learning and Instruction (EARLI 2002), 27–29 June 2002, Erfurt, Germany (Vol. 181195). Waxmann.
Ochoa, X. (2022). Chapter 6: Multimodal learning analytics: Rationale, process, examples, and direction. In C. Lang, G. Siemens, A. F. Wise, D. Gasevic, & A. Merceron (Eds.), The handbook of learning analytics (pp. 54–65). SoLAR. https://doi.org/10.18608/hla22.006
Ochoa, X., Echeverria, V., Carrillo, G., Heredia, V., & Chiluiza, K. (2023). Supporting online collaborative work at scale: A mixed-methods study of a learning analytics tool. In Proceedings of the 10th ACM Conference on Learning @ Scale (L@S 2023), 20–22 July 2023, Copenhagen, Denmark (pp. 237–247). ACM. https://doi.org/10.1145/3573051.3596165 OECD. (2017, August). PISA 2015 assessment and analytical framework: Science, reading, mathematic, financial literacy and collaborative problem solving (tech. rep.). OECD. https://doi.org/10.1787/9789264281820-en
Ofstedal, K., & Dahlberg, K. (2009). Collaboration in student teaching: Introducing the collaboration self-assessment tool. Journal of Early Childhood Teacher Education, 30(1), 37–48. https://doi.org/10.1080/10901020802668043
Orona, G., Liu, O. L., & Arum, R. (2023). The language of 21st century skills: Next directions for closing the skills gap between employers and postsecondary graduates. Chinese/English Journal of Educational Measurement and Evaluation, 4(2). https://doi.org/10.59863/OIVI3767
Panadero, E., & Lipnevich, A. A. (2022). A review of feedback models and typologies: Towards an integrative model of feedback elements. Educational Research Review, 35, 100416. https://doi.org/10.1016/j.edurev.2021.100416
Praharaj, S., Scheffel, M., Drachsler, H., & Specht, M. (2018). Multimodal analytics for real-time feedback in co-located collaboration. In V. Pammer-Schindler, M. P ´erez-Sanagust´ın, H. Drachsler, R. Elferink, & M. Scheffel (Eds.), Lifelong technology-enhanced learning. EC-TEL 2018. Lecture notes in computer science (pp. 187–201, Vol. 11082). Springer. https://doi.org/10.1007/978-3-319-98572-5_15
Praharaj, S., Scheffel, M., Drachsler, H., & Specht, M. (2021). Literature review on co-located collaboration modeling using multimodal learning analytics—can we go the whole nine yards? IEEE Transactions on Learning Technologies, 14(3), 367–385. https://doi.org/10.1109/TLT.2021.3097766
Sarmiento, J. P., Campos, F., & Wise, A. (2020). Engaging Students as Co-Designers of Learning Analytics. In Companion Proceedings of the 10th International Learning Analytics and Knowledge Conference (LAK 2020), 23–27 March, Frankfurt, Germany (pp. 29–32). SoLAR. https://www.solaresearch.org/wp-content/uploads/2020/06/LAK20_Companion_Proceedings.pdf
Scalise, K., Bernbaum, D. J., Timms, M., Harrell, S. V., Burmester, K., Kennedy, C. A., & Wilson, M. (2007). Adaptive technology for e-learning: Principles and case studies of an emerging field. Journal of the American Society for Information Science and Technology, 58(14), 2295–2309. https://doi.org/10.1002/asi.20701
Schneider, B., & Pea, R. (2013). Using eye-tracking technology to support visual coordination in collaborative problem-solving groups. In N. Rummel, M. Kapur, M. Nathan, & S. Puntambekar (Eds.), To see the world and a grain of sand: Learning across levels of space, time, and scale: CSCL 2013 conference proceedings—full papers & symposia (pp. 406–413, Vol. 1). International Society of the Learning Sciences. https://repository.isls.org/handle/1/1935
Schneider, B., Sharma, K., Cuendet, S., Zufferey, G., Dillenbourg, P., & Pea, R. (2018). Leveraging mobile eye-trackers to capture joint visual attention in co-located collaborative learning groups. International Journal of Computer-Supported Collaborative Learning, 13(3), 241–261. https://doi.org/10.1007/s11412-018-9281-2
Schneider, B., Sung, G., Chng, E., & Yang, S. (2021). How can high-frequency sensors capture collaboration? A review of the empirical links between multimodal metrics and collaborative constructs. Sensors, 21(24), 8185. https://doi.org/10.3390/s21248185
Singh Dubey, R., Paul, J., & Tewari, V. (2022). The soft skills gap: A bottleneck in the talent supply in emerging economies. The International Journal of Human Resource Management, 33(13), 2630–2661. https://doi.org/10.1080/09585192.2020.1871399
Snyder, H. (2019). Literature review as a research methodology: An overview and guidelines. Journal of Business Research, 104, 333–339. https://doi.org/10.1016/j.jbusres.2019.07.039
Soller, A., Martinez, A., Jermann, P., & Muehlenbrock, M. (2005). From mirroring to guiding: A review of state of the art technology for supporting collaborative learning. Journal of Artificial Intelligence in Education, 15(4), 261–290. https://dl.acm.org/doi/abs/10.5555/1434935.1434937
Starr, E. L., Reilly, J. M., & Schneider, B. (2018). Toward using multi-modal learning analytics to support and measure collaboration in co-located dyads. In J. Kay & R. Luckin (Eds.), Rethinking learning in the digital age: Making the learning sciences count, 13th international conference of the learning sciences (ICLS) 2018 (Vol. 1). International Society of the Learning Sciences. https://repository.isls.org//handle/1/888
Stevens, M. J., & Campion, M. A. (1994). The knowledge, skill and ability requirements for teamwork: Implications for human resource management. Journal of Management, 20(2), 503–530. https://doi.org/10.1016/0149-2063(94)90025-6
Tausch, S., Hausen, D., Kosan, I., Raltchev, A., & Hussmann, H. (2014). Groupgarden: Supporting brainstorming through a metaphorical group mirror on table or wall. In Proceedings of the Eighth Nordic Conference on Human-Computer Interaction: Fun, Fast, Foundational (NordCHI 2014), 26–30 October 2014, Helsinki, Finland (pp. 541–550). ACM. https://doi.org/10.1145/2639189.2639215
Tausch, S., Ta, S., & Hussmann, H. (2016). A comparison of cooperative and competitive visualizations for co-located collaboration. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI 2016), 7–12 May 2016, San Jose, California (pp. 5034–5039). ACM. https://doi.org/10.1145/2858036.2858072
Terken, J., & Sturm, J. (2010). Multimodal support for social dynamics in co-located meetings. Personal and Ubiquitous Computing, 14(8), 703–714. https://doi.org/10.1007/s00779-010-0284-x
Van Leeuwen, A., Knoop-van Campen, C. A. N., Molenaar, I., & Rummel, N. (2021). How teacher characteristics relate to how teachers use dashboards: Results from two case studies in K–12. Journal of Learning Analytics, 8(2), 6–21. https://doi.org/10.18608/jla.2021.7325
Wise, A. F., Knight, S., & Buckingham Shum, S. (2021). Collaborative learning analytics. In U. Cress, C. Rose, A. F. Wise, & J. Oshima (Eds.), International handbook of computer-supported collaborative learning (pp. 425–443). Springer International Publishing. https://doi.org/10.1007/978-3-030-65291-3_23
Wise, A. F., & Shaffer, D. W. (2015). Why theory matters more than ever in the age of big data. Journal of Learning Analytics, 2(2), 5–13. https://doi.org/10.18608/jla.2015.22.2
Wong-Villacres, M., Granda, R., Ortiz, M., & Chiluiza, K. (2016). Exploring the impact of a tabletop-generated group work feedback on students’ collaborative skills. In R. Mart´ınez Maldonado & D. Hernandez Leo (Eds.), Proceedings of the First International Workshop on Learning Analytics across Physical and Digital Spaces (CrossLAK 2016) co-located with the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 58–64). CEUR. https://ceur-ws.org/Vol-1601/CrossLAK16Paper11.pdf
Worsley, M., & Ochoa, X. (2020). Towards collaboration literacy development through multimodal learning analytics. In Companion Proceedings of the 10th International Learning Analytics and Knowledge Conference (LAK 2020), 23–27 March, Frankfurt, Germany (pp. 585–595). SoLAR. https://www.solaresearch.org/wp-content/uploads/2020/06/LAK20_Companion_Proceedings.pdf
Worthington, D. L., & Bodie, G. (Eds.). (2017). The sourcebook of listening research: Methodology and measures. John Wiley & Sons, Inc. https://doi.org/10.1002/9781119102991
Zhu, X., Shui, H., & Chen, B. (2023). The synthesis lab: Empowering collaborative learning in higher education through knowledge synthesis. In CSCW ’23 Companion: Companion Publication of the 2023 Conference on Computer Supported Cooperative Work and Social Computing, 14–18 October 2023, Minneapolis, Minnesota, USA (pp. 245–248). ACM. https://doi.org/10.1145/3584931.3606996
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
