A Learning Analytics Approach to Monitoring the Quality of Online One-to-One Tutoring

Authors

  • Mutlu Cukurova University College London https://orcid.org/0000-0001-5843-4854
  • Madiha Khan-Galaria University College London
  • Eva Millán Universidad de Málaga Avda
  • Rose Luckin University College London

DOI:

https://doi.org/10.18608/jla.2022.7411

Keywords:

online one-to-one tutoring, learning analytics, sequential pattern mining, decision trees, research paper

Abstract

One-to-one online tutoring provided by human tutors can improve students’ learning outcomes. However, monitoring the quality of such tutoring is a significant challenge. In this paper, we propose a learning analytics approach to monitoring online one-to-one tutoring quality. The approach analyzes teacher behaviours and classifies tutoring sessions into those that are effective and those that are not effective. More specifically, we use sequential behaviour pattern mining to analyze tutoring sessions using the CM-SPAM algorithm and classify tutoring sessions into effective and less effective using the J-48 and JRIP decision tree classifiers. To show the feasibility of the approach, we analyzed data from 2,250 minutes of online one-to-one primary math tutoring sessions with 44 tutors from eight schools. The results showed that the approach can classify tutors’ effectiveness with high accuracy (F measures of 0.89 and 0.98 were achieved). The results also showed that effective tutors present significantly more frequent hint provision and proactive planning behaviours than their less-effective colleagues in these online one-to-one sessions. Furthermore, effective tutors sequence their monitoring actions with appropriate pauses and initiations of students’ self-correction behaviours. We conclude that the proposed approach is feasible for monitoring the quality of online one-to-one primary math tutoring sessions.

 

References

Amiri, M., & Ghonsooly, B. (2015). The relationship between English learning anxiety and the students’ achievement on examinations. Journal of Language Teaching and Research, 6(4), 855–865. http://dx.doi.org/10.17507/jltr.0604.20

Anania, J. (1983). The influence of instructional conditions on student learning and achievement. Evaluation in Education: An International Review Series, 7, 3–76. https://doi.org/10.1016/0191-765X(83)90002-2

Andres, J. M. A. L., Ocumpaugh, J., Baker, R. S., Slater, S., Paquette, L., Jiang, Y., … Biswas, G. (2019). Affect sequences and learning in Betty’s Brain. In Proceedings of the Ninth International Conference on Learning Analytics & Knowledge (LAK 2019), 4–8 March 2019, Tempe, Arizona, USA (pp. 383–390). New York: ACM. https://doi.org/10.1145/3303772.3303807

Bell, A. (1993). Principles for the design of teaching. Educational Studies in Mathematics, 24(1), 5–34. https://doi.org/10.1007/BF01273293

Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in Education: Principles, Policy & Practice, 5(1), 7–74. https://doi.org/10.1080/0969595980050102

Boroujeni, M. S., & Dillenbourg, P. (2019). Discovery and temporal analysis of MOOC study patterns. Journal of Learning Analytics, 6(1), 16–33. https://doi.org/10.18608/jla.2019.61.2

Butler, A. C., Karpicke, J. D., & Roediger III, H. L. (2007). The effect of type and timing of feedback on learning from multiple-choice tests. Journal of Experimental Psychology: Applied, 13(4), 273. https://doi.org/10.1037/1076-898X.13.4.273

Camacho, V. L., de la Guía, E., Olivares, T., Flores, M. J., & Orozco-Barbosa, L. (2020). Data capture and multimodal learning analytics focused on engagement with a new wearable IoT approach. IEEE Transactions on Learning Technologies, 13(4), 704–717. https://doi.org/10.1109/TLT.2020.2999787

Carter Jr., R. A., Rice, M., Yang, S., & Jackson, H. A. (2020). Self-regulated learning in online learning environments: strategies for remote learning. Information and Learning Sciences, 121(5/6), 321–329. https://doi.org/10.1108/ILS-04-2020-0114

Chi, M. T., Bassok, M., Lewis, M. W., Reimann, P., & Glaser, R. (1989). Self-explanations: How students study and use examples in learning to solve problems. Cognitive Science, 13(2), 145–182. https://doi.org/10.1207/s15516709cog1302_1

Chi, M. T., De Leeuw, N., Chiu, M.-H., & LaVancher, C. (1994). Eliciting self-explanations improves understanding. Cognitive Science, 18(3), 439–477. https://doi.org/10.1016/0364-0213(94)90016-7

Cukurova, M., Mavrikis, M., Luckin, R., Clark, J., & Crawford, C. (2017). Interaction analysis in online maths human tutoring: The case of third space learning. In Proceedings of the International Conference on Artificial Intelligence In Education (AIED 2017), 28 June–1 July 2017, Wuhan, China (pp. 636-643). Cham: Springer. https://doi.org/10.1007/978-3-319-61425-0_80

Cullinane, C., & Montacute, R. (2020). COVID-19 and Social Mobility Impact Brief #1: School Shutdown. London, UK: Sutton Trust. Retrieved from https://eric.ed.gov/?id=ED605807

D’Mello, S., & Graesser, A. (2012). Dynamics of affective states during complex learning. Learning and Instruction, 22(2), 145–157. https://doi.org/10.1016/j.learninstruc.2011.10.001

Donald, C., Blake, A., Girault, I., Datt, A., & Ramsay, E. (2009). Approaches to learning design: Past the head and the hands to the HEART of the matter. Distance Education, 30(2), 179–199. https://doi.org/10.1080/01587910903023181

Fournier-Viger, P., Gomariz, A., Campos, M., & Thomas, R. (2014). Fast vertical mining of sequential patterns using co-occurrence information. In Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining (PAKDD 2014), 13–16 May 2014, Tainan, Taiwan (pp. 40–52). Cham: Springer. https://doi.org/10.1007/978-3-319-06608-0_4

Gitinabard, N., Heckman, S., Barnes, T., & Lynch, C. F. (2019). What will you do next? A sequence analysis on the student transitions between online platforms in blended courses. arXiv:1905.00928. Retrieved from https://arxiv.org/abs/1905.00928

Graesser, A. C. (1993). Questioning Mechanisms during Tutoring, Conversation, and Human-Computer Interaction (Tech. Rep.). Memphis State University Department of Psychology. https://apps.dtic.mil/sti/pdfs/ADA266420.pdf

Graesser, A. C., Person, N., Harter, D., & Tutoring Research Group. (2000). Teaching tactics in autotutor. Modelling human teaching tactics and strategies. International Journal of Artificial Intelligence in Education, 11, 1020–1029. http://users.sussex.ac.uk/~bend/its2000/graesser.pdf

Graesser, A. C., Person, N. K., & Magliano, J. P. (1995). Collaborative dialogue patterns in naturalistic one-to-one tutoring. Applied Cognitive Psychology, 9(6), 495–522. https://doi.org/10.1002/acp.2350090604

Hacker, D. J., Dole, J. A., Ferguson, M., Adamson, S., Roundy, L., & Scarpulla, L. (2015). The short-term and maintenance effects of self-regulated strategy development in writing for middle school students. Reading & Writing Quarterly, 31(4), 351–372. https://doi.org/10.1080/10573569.2013.869775

Hastie, M., Chen, N.-S., & Kuo, Y.-H. (2007). Instructional design for best practice in the synchronous cyber classroom. Journal of Educational Technology & Society, 10(4), 281–294. Retrieved from https://www.j-ets.net/collection/published-issues/10_4

Hodges, C., Moore, S., Lockee, B., Trust, T., & Bond, A. (2020). The difference between emergency remote teaching and online learning. Educause Review, 27. Retrieved from https://er.educause.edu/articles/2020/3/the-difference-between-emergency-remote-teaching-and-online-learning

Hwang, G.-J., Hsu, T.-C., Lai, C.-L., & Hsueh, C.-J. (2017). Interaction of problem-based gaming and learning anxiety in language students’ English listening performance and progressive behavioral patterns. Computers & Education, 106, 26–42. https://doi.org/10.1016/j.compedu.2016.11.010

Johnson, G. M., & Bratt, S. E. (2009). Technology education students: E‐tutors for school children. British Journal of Educational Technology, 40(1), 32-41. https://doi.org/10.1111/j.1467-8535.2007.00805.x

Khan-Galaria, M., Cukurova, M., & Luckin, R. (2020). A framework for exploring the impact of tutor practices on learner self-regulation in online environments. In Proceedings of the International Conference on Artificial Intelligence in Education (AIED 2020), 6–10 July 2020, Ifrane, Morocco (pp. 135–139). Cham: Springer. https://doi.org/10.1007/978-3-030-52240-7_25

Kinnebrew, J. S., Loretz, K. M., & Biswas, G. (2013). A contextualized, differential sequence mining method to derive students’ learning behavior patterns. Journal of Educational Data Mining, 5(1), 190–219. https://doi.org/10.5281/zenodo.3554617

Kopp, B., Matteucci, M. C., & Tomasetto, C. (2012). E-tutorial support for collaborative online learning: An explorative study on experienced and inexperienced e-tutors. Computers & Education, 58(1), 12–20. https://doi.org/10.1016/j.compedu.2011.08.019

Kulik, J. A., & Kulik, C.-L. C. (1988). Timing of feedback and verbal learning. Review of Educational Research, 58(1), 79–97. https://doi.org/10.3102/00346543058001079

Lefevre, D., & Cox, B. (2017). Delayed instructional feedback may be more effective, but is this contrary to learners’ preferences? British Journal of Educational Technology, 48(6), 1357–1367. https://doi.org/10.1111/bjet.12495

Lepper, M. R., & Woolverton, M. (2002). The wisdom of practice: Lessons learned from the study of highly effective tutors. In J. Aronson (ed.), Improving Academic Achievement (pp. 135–158). Elsevier. https://doi.org/10.1016/B978-012064455-1/50010-5

Lin, X., Schwartz, D. L., & Hatano, G. (2005). Toward teachers’ adaptive metacognition. Educational Psychologist, 40(4), 245–255. https://doi.org/10.1207/s15326985ep4004_6

Malekian, D., Bailey, J., & Kennedy, G. (2020). Prediction of students’ assessment readiness in online learning environments: The sequence matters. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, Frankfurt, Germany (pp. 382–391). New York: ACM. https://doi.org/10.1145/3375462.3375468

Martinez-Maldonado, R., Yacef, K., Kay, J., Kharrufa, A., & Al-Qaraghuli, A. (2011). Analysing frequent sequential patterns of collaborative learning activity around an interactive tabletop. In Proceedings of the Fourth International Educational Data Mining Conference (EDM 2011), 6–8 July 2011, Eindhoven, Netherlands. Retrieved from https://educationaldatamining.org/EDM2011/wp-content/uploads/proc/edm11_proceedings.pdf

Matcha, W., Gasevic, D., Jovanovic, J., Pardo, A., Lim, L., Maldonado-Mahauad, J., … Tsai, Y.-S. (2020). Analytics of learning strategies: Role of course design and delivery modality. Journal of Learning Analytics, 7(2), 45–71. https://doi.org/10.18608/jla.2020.72.3

Mathan, S. A., & Koedinger, K. R. (2002). An empirical assessment of comprehension fostering features in an intelligent tutoring system. In Proceedings of the International Conference on Intelligent Tutoring Systems (ITS 2002), 2–7 June 2002, Biarritz, France, and San Sebastian, Spain (pp. 330–343). Springer. https://doi.org/10.1007/3-540-47987-2_37

Merrill, D. C., Reiser, B. J., Ranney, M., & Trafton, J. G. (1992). Effective tutoring techniques: A comparison of human tutors and intelligent tutoring systems. The Journal of the Learning Sciences, 2(3), 277–305. https://doi.org/10.1207/s15327809jls0203_2

Narciss, S., & Huth, K. (2004). How to design informative tutoring feedback for multimedia learning. In H. M. Niegemann, D. Leutner, and R. Brünken (eds.), Instructional Design for Multimedia Learning, pp. 181–195. Kornwestheim, Germany: Waxmann Buecher. Retrieved from https://studierplatz2000.tu-dresden.de/toolkit/presentations/CD/Literatur/Publikationen/ID_nahu.pdf

Ozdagoglu, G., Oztas, G. Z., & Cagliyangil, M. (2019). An application framework for mining online learning processes through event-logs. Business Process Management Journal, 25(5), 860–886. https://doi.org/10.1108/BPMJ-10-2017-0279

Pardos, Z. A., Baker, R. S., San Pedro, M. O., Gowda, S. M., & Gowda, S. M. (2013). Affective states and state tests: Investigating how affect throughout the school year predicts end of year learning outcomes. In Proceedings of the Third International Conference on Learning Analytics and Knowledge (LAK 2013), 8–12 April 2013, Leuven, Belgium (pp. 117–124). New York: ACM. https://doi.org/10.1145/2460296.2460320

Pekrun, R., Goetz, T., Titz, W., & Perry, R. P. (2002). Academic emotions in students’ self-regulated learning and achievement: A program of qualitative and quantitative research. Educational Psychologist, 37(2), 91–105. https://doi.org/10.1207/S15326985EP3702_4

Plass, J. L., Heidig, S., Hayward, E. O., Homer, B. D., & Um, E. (2014). Emotional design in multimedia learning: Effects of shape and color on affect and learning. Learning and Instruction, 29, 128–140. https://doi.org/10.1016/j.learninstruc.2013.02.006

Prieto, L. P., Sharma, K., Dillenbourg, P., & Jesús, M. (2016). Teaching analytics: Towards automatic extraction of orchestration graphs using wearable sensors. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 148–157). New York: ACM. https://doi.org/10.1145/2883851.2883927

Prieto, L. P., Sharma, K., Kidzinski, Ł., Rodríguez‐Triana, M. J., & Dillenbourg, P. (2018). Multimodal teaching analytics: Automated extraction of orchestration graphs from wearable sensor data. Journal of Computer Assisted Learning, 34(2), 193–203. https://doi.org/10.1111/jcal.12232

Rapanta, C., Botturi, L., Goodyear, P., Guàrdia, L., & Koole, M. (2020). Online university teaching during and after the Covid-19 crisis: Refocusing teacher presence and learning activity. Postdigital Science and Education, 2, 923–945. https://doi.org/10.1007/s42438-020-00155-y

Riel, J., Lawless, K. A., & Brown, S. W. (2018). Timing matters: Approaches for measuring and visualizing behaviours of timing and spacing of work in self-paced online teacher professional development courses. Journal of Learning Analytics, 5(1), 25–40. https://doi.org/10.18608/jla.2018.51.3

Ritter, S., Anderson, J. R., Koedinger, K. R., & Corbett, A. (2007). Cognitive tutor: Applied research in mathematics education. Psychonomic Bulletin & Review, 14, 249–255. https://doi.org/10.3758/BF03194060

Ryan, R. M., & Deci, E. L. (2009). Promoting self-determined school engagement: Motivation, learning, and well-being. In K. R. Wenzel & A. Wigfield (Eds.), Educational Psychology Handbook Series. Handbook of Motivation at School (pp. 171–195). Routledge/Taylor & Francis Group. https://psycnet.apa.org/record/2009-24219-009

Saint, J., Gasevic, D., Matcha, W., Uzir, N. A., & Pardo, A. (2020). Combining analytic methods to unlock sequential and temporal patterns of self-regulated learning. In Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK 2020), 23–27 March 2020, Frankfurt, Germany (pp. 402–411). New York: ACM. https://doi.org/10.1145/3375462.3375487

Schmidt, H. G., Dolmans, D., Gijselaers, W. H., & Des Marchais, J. E. (1995). Theory-guided design of a rating scale for course evaluation in problem-based curricula. Teaching and Learning in Medicine: An International Journal, 7(2), 82–91. https://doi.org/10.1080/10401339509539719

Shield, M., & Dole, S. (2013). Assessing the potential of mathematics textbooks to promote deep learning. Educational Studies in Mathematics, 82, 183–199. https://doi.org/10.1007/s10649-012-9415-9

Shorrocks-Taylor, D., & Hargreaves, M. (1999). Making it clear: A review of language issues in testing with special reference to the national curriculum mathematics tests at key stage 2. Educational Research, 41(2), 123–136. https://doi.org/10.1080/0013188990410201

Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795

Sleeman, D., Kelly, A. E., Martinak, R., Ward, R. D., & Moore, J. L. (1989). Studies of diagnosis and remediation with high school algebra students. Cognitive Science, 13(4), 551–568. https://doi.org/10.1016/0364-0213(89)90023-2

Torgerson, C., Ainsworth, H., Buckley, H., Hampden-Thompson, G., Hewitt, C., Humphry, D., … Torgerson, D. (2016). Affordable Online Maths Tuition: Evaluation Report and Executive Summary. Education Endowment Foundation. Retrieved from https://eric.ed.gov/?id=ED581116

Walsh, J. A., & Sattes, B. D. (2016). Quality Questioning: Research-Based Practice to Engage Every Learner. Corwin Press. ISBN: 1 74101 442 5. https://books.google.co.uk/books?id=lG51DQAAQBAJ&lr=

Wang, Y., Li, T., Geng, C., & Wang, Y. (2019). Evaluating student learning effect based on process mining. In Proceedings of the Second International Conference on Applied Informatics (ICAI 2019), 7–9 November 2019, Madrid, Spain (pp. 59–72). Springer. https://doi.org/10.1007/978-3-030-32475-9_5

Wittwer, J., Nückles, M., Landmann, N., & Renkl, A. (2010). Can tutors be supported in giving effective explanations? Journal of Educational Psychology, 102(1), 74. https://doi.org/10.1037/a0016727

Wittwer, J., Nückles, M., & Renkl, A. (2008). Is underestimation less detrimental than overestimation? The impact of experts’ beliefs about a layperson’s knowledge on learning and question asking. Instructional Science, 36, 27–52. https://doi.org/10.1007/s11251-007-9021-x

Wong, J., Khalil, M., Baars, M., de Koning, B. B., & Paas, F. (2019). Exploring sequences of learner activities in relation to self-regulated learning in a massive open online course. Computers & Education, 140, 103595. https://doi.org/10.1016/j.compedu.2019.103595

Wylie, R., & Chi, M. T. (2014). 17—The self-explanation principle in multimedia learning. In R. E. Mayer (ed.), The Cambridge Handbook of Multimedia Learning (pp. 413–432). Cambridge University Press. https://doi.org/10.1017/CBO9781139547369.021

Zheng, J., Xing, W., & Zhu, G. (2019). Examining sequential patterns of self- and socially shared regulation of STEM learning in a CSCL environment. Computers & Education, 136, 34–48. https://doi.org/10.1016/j.compedu.2019.03.005

Downloads

Published

2022-05-01

How to Cite

Cukurova, M., Khan-Galaria, M., Millán, E., & Luckin, R. (2022). A Learning Analytics Approach to Monitoring the Quality of Online One-to-One Tutoring. Journal of Learning Analytics, 1-16. https://doi.org/10.18608/jla.2022.7411

Issue

Section

Research Papers