Enriching Multimodal Data

A Temporal Approach to Contextualize Joint Attention in Collaborative Problem-Solving





collaborative problem-solving, temporality, temporal analysis, multimodal learning analytics, collaboration analytics, CSCL, research paper


Collaboration is a complex, multidimensional process; however, details of how multimodal features intersect and mediate group interactions have not been fully unpacked. Characterizing and analyzing the temporal patterns based on multimodal features is a challenging yet important work to advance our understanding of computer-supported collaborative learning (CSCL). This paper highlights the affordances, as well as the limitations, of different temporal approaches in terms of analyzing multimodal data. To tackle the remaining challenges, we present an empirical example of multimodal temporal analysis that leverages multi-level vector autoregression (mlVAR) to identify temporal patterns of the collaborative problem-solving (CPS) process in an immersive astronomy simulation. We extend previous research on joint attention with a particular focus on the added value from a multimodal, temporal account of the CPS process. We incorporate verbal discussion to contextualize joint attention, examine the sequential and contemporaneous associations between them, and identify significant differences in temporal patterns between low- and high-achieving groups. Our paper does the following: 1) creates interpretable multimodal group interaction patterns, 2) advances understanding of CPS through examination of verbal and non-verbal interactions, and 3) demonstrates the added value of a complete account of temporality including both duration and sequential order.


Bakeman, R., & Gottman, J. M. (1997). Observing interaction: An introduction to sequential analysis (2nd ed.). Cambridge University Press. https://doi.org/10.1017/CBO9780511527685

Baker, R. S. J. d., D’Mello, S. K., Rodrigo, M. M. T., & Graesser, A. C. (2010). Better to be frustrated than bored: The incidence, persistence, and impact of learners’ cognitive–affective states during interactions with three different computer-based learning environments. International Journal of Human–Computer Studies, 68(4), 223–241. https://doi.org/10.1016/j.ijhcs.2009.12.003

Barron, B. (2003). When smart groups fail. Journal of the Learning Sciences, 12(3), 307–359. https://doi.org/10.1207/S15327809JLS1203_1

Bringmann, L. F., Vissers, N., Wichers, M., Geschwind, N., Kuppens, P., Peeters, F., Borsboom, D., & Tuerlinckx, F. (2013). A network approach to psychopathology: New insights into clinical longitudinal data. PLoS ONE, 8(4), e60188. https://doi.org/10.1371/journal.pone.0060188

Brohinsky, J., Marquart, C., Wang, J., Ruis, A. R., & Shaffer, D. W. (2021). Trajectories in epistemic network analysis. In A. R. Ruis & S. B. Lee (Eds.), Advances in quantitative ethnography (pp. 106–121). Springer. https://doi.org/10.1007/978-3-030-67788-6_8

Chang, C.-J., Chang, M.-H., Chiu, B.-C., Liu, C.-C., Fan Chiang, S.-H., Wen, C.-T., Hwang, F.-K., Wu, Y.-T., Chao, P.-Y., Lai, C.-H., Wu, S.-W., Chang, C.-K., & Chen, W. (2017). An analysis of student collaborative problem solving activities mediated by collaborative simulations. Computers & Education, 114, 222–235. https://doi.org/10.1016/j.compedu.2017.07.008

Chango, W., Lara, J. A., Cerezo, R., & Romero, C. (2022). A review on data fusion in multimodal learning analytics and educational data mining. WIREs Data Mining and Knowledge Discovery, 12(4), e1458. https://doi.org/10.1002/widm.1458

Chen, B., Knight, S., & Wise, A. F. (2018). Critical issues in designing and implementing temporal analytics. Journal of Learning Analytics, 5(1). https://doi.org/10.18608/jla.2018.53.1

Chen, B., Resendes, M., Chai, C. S., & Hong, H.-Y. (2017). Two tales of time: Uncovering the significance of sequential patterns among contribution types in knowledge-building discourse. Interactive Learning Environments, 25(2), 162–175. https://doi.org/10.1080/10494820.2016.1276081

Chiu, M. M. (2000). Group problem-solving processes: Social interactions and individual actions. Journal for the Theory of Social Behaviour, 30(1), 26–49. https://doi.org/10.1111/1468-5914.00118

Csanadi, A., Eagan, B., Kollar, I., Shaffer, D. W., & Fischer, F. (2018). When coding-and-counting is not enough: Using epistemic network analysis (ENA) to analyze verbal data in CSCL research. International Journal of Computer-Supported Collaborative Learning, 13(4), 419–438. https://doi.org/10.1007/s11412-018-9292-z

Cukurova, M., Giannakos, M., & Martinez-Maldonado, R. (2020). The promise and challenges of multimodal learning analytics. British Journal of Educational Technology, 51(5), 1441–1449. https://doi.org/10.1111/bjet.13015

Diederich, M., Kang, J., Kim, T., & Lindgren, R. (2021). Developing an in-application shared view metric to capture collaborative learning in a multi-platform astronomy simulation. Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA (pp. 173–183). ACM Press. https://doi.org/10.1145/3448139.3448156

Dillenbourg, P., Baker, M., Blaye, A., & O’Malley, C. (1996). The evolution of research on collaborative learning. In E. Spada & P. Reiman (Eds.), Learning in humans and machine: Towards an interdisciplinary learning science (pp. 189–211). Oxford, Elsevier. http://infoscience.epfl.ch/record/33761

Dillenbourg, P., Järvelä, S., & Fischer, F. (2009). The evolution of research on computer-supported collaborative learning: From design to orchestration (pp. 3–19). https://doi.org/10.1007/978-1-4020-9827-7_1

Echeverria, V., Martinez-Maldonado, R., & Buckingham Shum, S. (2019). Towards collaboration translucence: Giving meaning to multimodal group data. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), 4–9 May 2019, Glasgow, Scotland, UK (Paper no. 39). ACM Press. https://doi.org/10.1145/3290605.3300269

Emerson, A., Cloude, E. B., Azevedo, R., & Lester, J. (2020). Multimodal learning analytics for game‐based learning. British Journal of Educational Technology, 51(5), 1505–1526. https://doi.org/10.1111/bjet.12992

Epskamp, S., Deserno, M. K., Bringmann, L. F., & Veenman, M. (2021). mlVAR: Multi-level vector autoregression (0.5) [Computer software]. https://CRAN.R-project.org/package=mlVAR

Epskamp, S., Waldorp, L. J., Mõttus, R., & Borsboom, D. (2018). The Gaussian graphical model in cross-sectional and time-series data. Multivariate Behavioral Research, 53(4), 453–480. https://doi.org/10.1080/00273171.2018.1454823

Fiore, S. M., Rosen, M. A., Smith-Jentsch, K. A., Salas, E., Letsky, M., & Warner, N. (2010). Toward an understanding of macrocognition in teams: Predicting processes in complex collaborative contexts. Human Factors, 52(2), 203–224. https://doi.org/10.1177/0018720810369807

Helske, J., & Helske, S. (2021). seqHMM: Mixture hidden Markov models for social sequence data and other multivariate, multichannel categorical time series (1.2.0) [Computer software]. https://CRAN.R-project.org/package=seqHMM

Janssen, J., Kirschner, F., Erkens, G., Kirschner, P. A., & Paas, F. (2010). Making the black box of collaborative learning transparent: Combining process-oriented and cognitive load approaches. Educational Psychology Review, 22(2), 139–154. https://doi.org/10.1007/s10648-010-9131-x

Kang, J., An, D., Yan, L., & Liu, M. (2019). Collaborative problem-solving process in a science serious game: Exploring group action similarity trajectory. In C. F. Lynch, A. Merceron, M. Desmarais, & R. Nkambou (Eds.), Proceedings of the 12th International Conference on Educational Data Mining (EDM2019), 2–5 July 2019, Montréal, Quebec, Canada (pp. 336–341). International Educational Data Mining Society.

Kapur, M., Voiklis, J., & Kinzer, C. K. (2008). Sensitivities to early exchange in synchronous computer-supported collaborative learning (CSCL) groups. Computers & Education, 51(1), 54–66. https://doi.org/10.1016/j.compedu.2007.04.007

Karumbaiah, S., Andres, J. M. A. L., Botelho, A. F., Baker, R. S., & Ocumpaugh, J. S. (2018). The implications of a subtle difference in the calculation of affect dynamics. In J. C. Yang, M. Chang, L.-H. Wong, & M. T. Rodrigo (Eds.), Proceedings of the 26th International Conference on Computers in Education (ICCE 2018), 26–30 November 2018, Manila, Philippines (pp. 29–38). Asia-Pacific Society for Computers in Education. http://icce2018.ateneo.edu/wp-content/uploads/2018/12/C1-04.pdf

Knight, S., Wise, A. F., & Chen, B. (2017). Time for change: Why learning analytics needs temporal analysis. Journal of Learning Analytics, 4(3), 7–17. https://doi.org/10.18608/jla.2017.43.2

Lämsä, J., Hämäläinen, R., Koskinen, P., Viiri, J., & Lampi, E. (2021). What do we do when we analyse the temporal aspects of computer-supported collaborative learning? A systematic literature review. Educational Research Review, 33, 100387. https://doi.org/10.1016/j.edurev.2021.100387

Martinez, R., Yacef, K., Kay, J., Al-Qaraghuli, A., & Kharrufa, A. (2011). Analysing frequent sequential patterns of collaborative learning activity around an interactive tabletop. In M. Pechenizkiy et al. (Eds.), Proceedings of the 4th Annual Conference on Educational Data Mining (EDM2011), 6–8 July 2011, Eindhoven, Netherlands (pp. 111–120). International Educational Data Mining Society. https://educationaldatamining.org/EDM2011/wp-content/uploads/proc/edm11_proceedings.pdf

Martinez-Maldonado, R., Dimitriadis, Y., Martinez-Monés, A., Kay, J., & Yacef, K. (2013). Capturing and analyzing verbal and physical collaborative learning interactions at an enriched interactive tabletop. International Journal of Computer-Supported Collaborative Learning, 8(4), 455–485. https://doi.org/10.1007/s11412-013-9184-1

Martinez-Maldonado, R., Kay, J., Buckingham Shum, S., & Yacef, K. (2019). Collocated collaboration analytics: Principles and dilemmas for mining multimodal interaction data. Human–Computer Interaction, 34(1), 1–50. https://doi.org/10.1080/07370024.2017.1338956

Mercier, E., Vourloumi, G., & Higgins, S. (2017). Student interactions and the development of ideas in multi-touch and paper-based collaborative mathematical problem solving. British Journal of Educational Technology, 48(1), 162–175. https://doi.org/10.1111/bjet.12351

Molenaar, I., & Wise, A. F. (2022). Temporal aspects of learning analytics: Grounding analyses in concepts of time. In C. Lang, G. Siemens, A. F. Wise, D. Gašević, & A. Merceron (Eds.), Handbook of learning analytics (pp. 66–76). https://doi.org/10.18608/hla22.007

Mushtaq, R. (2011, August 17). Augmented Dickey Fuller test. Econometrics. https://doi.org/10.2139/ssrn.1911068

Nizam, S. S. M., Zainal Abidin, R., Che Hashim, N., Lam, M. C., Arshad, H., & Abd Majid, N. A. (2018). A review of multimodal interaction technique in augmented reality environment. International Journal on Advanced Science, Engineering and Information Technology, 8(4–2), 1460–1469. https://doi.org/10.18517/ijaseit.8.4-2.6824

Noroozi, O., Alikhani, I., Järvelä, S., Kirschner, P. A., Juuso, I., & Seppänen, T. (2019). Multimodal data to design visual learning analytics for understanding regulation of learning. Computers in Human Behavior, 100, 298–304. https://doi.org/10.1016/j.chb.2018.12.019

Nystrand, M., Wu, L. L., Gamoran, A., Zeiser, S., & Long, D. A. (2003). Questions in time: Investigating the structure and dynamics of unfolding classroom discourse. Discourse Processes, 35(2), 135–198. https://doi.org/10.1207/S15326950DP3502_3

Phon, D. N. E., Ali, M. B., & Halim, N. D. A. (2014). Collaborative augmented reality in education: A review. 2014 International Conference on Teaching and Learning in Computing and Engineering, 11–13 April 2014, Kuching, Malaysia (pp. 78–83). IEEE Computer Society. https://doi.org/10.1109/LaTiCE.2014.23

Planey, J., Rajarathinam, R. J., Mercier, E., & Lindgren, R. (2023). Gesture-mediated collaboration with augmented reality headsets in a problem-based astronomy task. International Journal of Computer-Supported Collaborative Learning, 18(2), 259–289. https://doi.org/10.1007/s11412-023-09398-w

Praharaj, S., Scheffel, M., Drachsler, H., & Specht, M. (2021). Literature review on co-located collaboration modeling using multimodal learning analytics: Can we go the whole nine yards? IEEE Transactions on Learning Technologies, 14(3), 367–385. https://doi.org/10.1109/TLT.2021.3097766

Reimann, P. (2009). Time is precious: Variable- and event-centred approaches to process analysis in CSCL research. International Journal of Computer-Supported Collaborative Learning, 4(3), 239–257. https://doi.org/10.1007/s11412-009-9070-z

Sarmiento, J. W., & Stahl, G. (2008). Extending the joint problem space: Time and sequence as essential features of knowledge building. Proceedings of the 8th International Conference on the Learning Sciences (ICLS ’08), 24–28 June 2008, Utrecht, Netherlands (Vol. 3, pp. 295–302). International Society of the Learning Sciences. http://www.gerrystahl.net/pub/icls2008johann.pdf

Schneider, B., Worsley, M., & Martinez-Maldonado, R. (2021). Gesture and gaze: Multimodal data in dyadic interactions. In U. Cress, C. Rosé, A. F. Wise, & J. Oshima (Eds.), International Handbook of Computer-Supported Collaborative Learning (pp. 625–641). Springer. https://doi.org/10.1007/978-3-030-65291-3_34

Shaffer, D. W., Hatfield, D., Svarovsky, G. N., Nash, P., Nulty, A., Bagley, E., Frank, K., Rupp, A. A., & Mislevy, R. (2009). Epistemic network analysis: A prototype for 21st-century assessment of learning. International Journal of Learning and Media, 1(2), 33–53. https://doi.org/10.1162/ijlm.2009.0013

Sharma, K., & Giannakos, M. (2020). Multimodal data capabilities for learning: What can multimodal data tell us about learning? British Journal of Educational Technology, 51(5), 1450–1484. https://doi.org/10.1111/bjet.12993

Siposova, B., & Carpenter, M. (2019). A new look at joint attention and common knowledge. Cognition, 189, 260–274. https://doi.org/10.1016/j.cognition.2019.03.019

Stahl, G. (2006). Group cognition: Computer support for building collaborative knowledge. The MIT Press. https://doi.org/10.7551/mitpress/3372.001.0001

Syczewska, E. M. (2010). Empirical power of the Kwiatkowski-Phillips-Schmidt-Shin test. In Working Papers 45, Department of Applied Econometrics, Warsaw School of Economics. https://ideas.repec.org/p/wse/wpaper/45.html

Tan, S. C., Wang, X., & Li, L. (2022). The development trajectory of shared epistemic agency in online collaborative learning: A study combing network analysis and sequential analysis. Journal of Educational Computing Research, 59(8), 1655–1681. https://doi.org/10.1177/07356331211001562

Teasley, S. D., & Roschelle, J. (1993). Constructing a joint problem space: The computer as a tool for sharing knowledge. In S. P. Lajoie & S. J. Derry (Eds.), Computers as Cognitive Tools (pp. 229–258). Routledge.

Volet, S., Vauras, M., Salo, A.-E., & Khosa, D. (2017). Individual contributions in student-led collaborative learning: Insights from two analytical approaches to explain the quality of group outcome. Learning and Individual Differences, 53, 79–92. https://doi.org/10.1016/j.lindif.2016.11.006

Worsley, M. (2014). Multimodal learning analytics as a tool for bridging learning theory and complex learning behaviors. Proceedings of the 2014 ACM Workshop on Multimodal Learning Analytics Workshop and Grand Challenge (MLA ’14), 12–16 November 2014, Istanbul, Turkey (pp. 1–4). ACM Press. https://doi.org/10.1145/2666633.2666634

Zheng, J., Xing, W., & Zhu, G. (2019). Examining sequential patterns of self- and socially shared regulation of STEM learning in a CSCL environment. Computers & Education, 136, 34–48. https://doi.org/10.1016/j.compedu.2019.03.005

Zhou, G., Moulder, R., Sun, C., & D’Mello, S. (2022). Investigating temporal dynamics underlying successful collaborative problem solving behaviors with multilevel vector autoregression. In A. Mitrovic & N. Bosch (Eds.), Proceedings of the 15th International Conference on Educational Data Mining (EDM2022), 24–27 July 2022, Durham, UK (pp. 290–301). International Educational Data Mining Society. https://doi.org/10.5281/zenodo.6853137

Zhou, Y., & Kang, J. (2022). Characterizing joint attention dynamics during collaborative problem-solving in an immersive astronomy simulation. In A. Mitrovic & N. Bosch (Eds.), Proceedings of the 15th International Conference on Educational Data Mining (EDM2022), 24–27 July 2022, Durham, UK (pp. 406–413). International Educational Data Mining Society. https://doi.org/10.5281/zenodo.6852988




How to Cite

Zhou, Y., & Kang, J. (2023). Enriching Multimodal Data: A Temporal Approach to Contextualize Joint Attention in Collaborative Problem-Solving. Journal of Learning Analytics, 1-15. https://doi.org/10.18608/jla.2023.7989



Research Papers