Use of Learning Analytics in K–12 Mathematics Education
Systematic Scoping Review of the Impact on Teaching and Learning
DOI:
https://doi.org/10.18608/jla.2024.8299Keywords:
K-12 education, learning analytics, data-based decision-making (DBDM), analytics for learners, teaching, learning, research paperAbstract
The generation, use, and analysis of educational data comes with many promises and opportunities, especially where digital materials allow usage of learning analytics (LA) as a tool in data-based decision-making (DBDM). However, there are questions about the interplay between teachers, students, context, and technology. Therefore, this paper presents an exploratory systematic scoping review to investigate findings regarding LA usage in digital materials, teaching, and learning in K–12 mathematics education. In all, 3,654 records were identified, of which 19 studies met all the inclusion criteria. Results show that LA research in mathematics education is an emerging field where applications of LA are used in many contexts across many curricula content and standards of K–12 mathematics education, supporting a wide variety of teacher data use. Teaching with DBDM is mainly focused on supervision and guidance and LA usage had a generally positive effect on student learning with high-performing students benefiting most. We highlight a need for further research to develop knowledge of LA usage in classroom practice that considers both teacher and student perspectives in relation to design and affordances of digital learning systems. Finally, we propose a new class of LA, which we define as guiding analytics for learners, which harnesses the potential of LA for promoting achievement and independent learning.
References
Aguerrebere, C., He, H., Kwet, M., Laakso, M.-J., Lang, C., Marconi, C., Prince-Dennis, D., & Zhang, H. (2022). Global perspectives on learning analytics in K12 education. In C. Lang, G. Siemens, A. F. Wise, D. Gašević, & A. Merceron (Eds.), The handbook of learning analytics (2nd ed., pp. 223–231). SoLAR. https://doi.org/10.18608/hla22
Aleven, V., Blankestijn, J., Lawrence, L., Nagashima, T., & Taatgen, N. (2022). A dashboard to support teachers during students’ self-paced AI-supported problem-solving practice. In I. Hilliger, P. J. Muñoz-Merino, T. De Laet, A. Ortega-Arranz, & T. Farrell (Eds.), Educating for a new future: Making sense of technology-enhanced learning adoption: 17th European conference on technology enhanced learning, EC-TEL 2022, Toulouse, France, September 12–16, 2022, proceedings (pp. 16–30). Springer Cham. https://doi.org/10.1007/978-3-031-16290-9_2
Anderson, L. W., Krathwohl, D. R., Airasian, P. W., Cruikshank, K. A., Mayer, R. E., Pintrich, P. R., Raths, J., & Wittrock, M. C. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. Addison Wesley Longman.
Arksey, H., & O’Malley, L. (2005). Scoping studies: Towards a methodological framework. International Journal of Social Research Methodology, 8(1), 19–32. https://doi.org/10.1080/1364557032000119616
Barrus, A. (2013). Does self-regulated learning-skills training improve high-school students’ self-regulation, math achievement, and motivation while using an intelligent tutor? [Unpublished doctoral dissertation]. Arizona State University. https://keep.lib.asu.edu/items/151688
Blum, W., Artigue, M., Mariotti, M. A., Sträßer, R., & van den Heuvel-Panhuizen, M. (Eds.) (2019). European traditions in didactics of mathematics. SpringerOpen.
Boyatzis, R. E. (1998). Transforming qualitative information: Thematic analysis and code development. Sage Publications.
Campos, F. C., Ahn, J., DiGiacomo, D. K., Nguyen, H., & Hays, M. (2021). Making sense of sensemaking: Understanding how K–12 teachers and coaches react to visual analytics. Journal of Learning Analytics, 8(3), 60–80. https://doi.org/10.18608/jla.2021.7113
Chen, C.-M., & Chen, M.-C. (2009). Mobile formative assessment tool based on data mining techniques for supporting web-based learning. Computers & Education, 52(1), 256–273. https://doi.org/10.1016/j.compedu.2008.08.005
Cen, H., Koedinger, K. R., & Junker, B. (2007). Is over practice necessary? Improving learning efficiency with the cognitive tutor through educational data mining. Proceedings of the 2007 Conference on Artificial Intelligence in Education (AIED-2007), 9–13 July 2007, Los Angeles, CA, USA (pp. 511–518). IOS Press.
Colquhoun, H. L., Levac, D., O’Brien, K. K., Straus, S., Tricco, A. C., Perrier, L., Kastner, M., & Moher, D. (2014). Scoping reviews: Time for clarity in definition, methods, and reporting. Journal of Clinical Epidemiology, 67(12), 1291–1294. https://doi.org/10.1016/j.jclinepi.2014.03.013
Confrey, J., Toutkoushian, E., & Shah, M. (2019). A validation argument from soup to nuts: Assessing progress on learning trajectories for middle-school mathematics. Applied Measurement in Education, 32(1), 23–42. https://doi.org/10.1080/08957347.2018.1544135
Consoli, T., Désiron, J., & Cattaneo, A. (2023). What is “technology integration” and how is it measured in K–12 education? A systematic review of survey instruments from 2010 to 2021. Computers & Education, 197, 104742. https://doi.org/10.1016/j.compedu.2023.104742
Datnow, A., Park, V., & Kennedy-Lewis, B. (2013). Affordances and constraints in the context of teacher collaboration for the purpose of data use. Journal of Educational Administration, 51(3), 341–362. https://doi.org/10.1108/09578231311311500
Dickinson, A. R., & Hui, D. (2009). Enhancing intelligence, English and math competencies in the classroom via e@Leader integrated online edutainment gaming and assessment. In D. Russell (Ed.), Cases on collaboration in virtual learning environments: Processes and interactions (pp. 263–283). Information Science Reference.
Dignath, C., Buettner, G., & Langfeldt, H.-P. (2008). How can primary school students learn self-regulated learning strategies most effectively? A meta-analysis on self-regulation training programmes. Educational Research Review, 3(2), 101–129. https://doi.org/10.1016/j.edurev.2008.02.003
Du, X., Yang, J., Shelton, B. E., Hung, J.-L., & Zhang, M. (2021). A systematic meta-review and analysis of learning analytics research. Behaviour & Information Technology, 40(1), 49–62. https://doi.org/10.1080/0144929X.2019.1669712
European Commission. (2019). Key competences for lifelong learning. Publications Office of the European Union. https://data.europa.eu/doi/10.2766/569540
Faber, J. M., Luyten, H., & Visscher, A. J. (2017). The effects of a digital formative assessment tool on mathematics achievement and student motivation: Results of a randomized experiment. Computers & Education, 106, 83–96. https://doi.org/10.1016/j.compedu.2016.12.001
Filderman, M. J., Toste, J. R., Didion, L., & Peng, P. (2022). Data literacy training for K–12 teachers: A meta-analysis of the effects on teacher outcomes. Remedial and Special Education, 43(5), 328–343. https://doi.org/10.1177/07419325211054208
Gough, D., Oliver, S., & Thomas, J. (2017). An introduction to systematic reviews (2nd ed.). SAGE.
Hase, A., & Kuhl, P. (2024). Teachers’ use of data from digital learning platforms for instructional design: A systematic review. Education Technology Research and Development, 72(4), 1925–1945. https://doi.org/10.1007/s11423-024-10356-y
Hattie, J., & Yates, G. (2014). Visible learning and the science of how we learn. Routledge.
Hawn, A. (2019a). Study 1: Exploring teachers’ online usage of student testing data. In Data-wary, value-driven: Teacher attitudes, efficacy, and online access for data-based decision making (pp. 91–167) [Unpublished doctoral dissertation]. Columbia University. https://api.semanticscholar.org/CorpusID:182074848
Hawn, A. (2019b). Study 2: Connecting teacher roles and data use attitudes to online behaviours. In Data-wary, value-driven: Teacher attitudes, efficacy, and online access for data-based decision making (pp. 186–278) [Unpublished doctoral dissertation]. Columbia University. https://api.semanticscholar.org/CorpusID:182074848
Hillmayr, D., Ziernwald, L., Reinhold, F., Hofer, S. I., & Reiss, K. M. (2020). The potential of digital tools to enhance mathematics and science learning in secondary schools: A context-specific meta-analysis. Computers & Education, 153, 103897. https://doi.org/10.1016/j.compedu.2020.103897
Hoogland, I., Schildkamp, K., van der Kleij, F., Heitink, M., Kippers, W., Veldkamp, B., & Dijkstra, A. M. (2016). Prerequisites for data-based decision making in the classroom: Research evidence and practical illustrations. Teaching and Teacher Education, 60, 377–386. https://doi.org/10.1016/j.tate.2016.07.012
Jormanainen, I. & Sutinen, E. (2012). Using data mining to support teacher’s intervention in a robotics class. 2012 IEEE Fourth International Conference on Digital Game and Intelligent Toy Enhanced Learning (DIGITEL), 27–30 March 2012, Takamatsu, Japan (pp. 39–46). https://doi.org/10.1109/DIGITEL.2012.14.
Kalloo, V., & Mohan, P. (2011a). An investigation into mobile learning for high school mathematics. International Journal of Mobile and Blended Learning, 3(3), 59–76. https://doi.org/10.4018/jmbl.2011070105.
Kalloo, V., & Mohan, P. (2011b). Correlation between student performance and use of an mLearning application for high school mathematics. Proceedings of the 2011 IEEE 11th International Conference on Advanced Learning Technologies (ICALT 2011), 6–8 July 2011, Athens, GA, USA (pp. 174–178). IEEE. https://doi.org/10.1109/ICALT.2011.57
Koehler, M. J., & Mishra, P. (2009). What is technological pedagogical content knowledge? Contemporary Issues in Technology and Teacher Education, 9(1). https://citejournal.org/volume-9/issue-1-09/general/what-is-technological-pedagogicalcontent-knowledge
Krauss, S., Brunner, M., Kunter, M., Baumert, J., Blum, W., Neubrand, M., & Jordan, A. (2008). Pedagogical content knowledge and content knowledge of secondary mathematics teachers. Journal of Educational Psychology, 100(3), 716–725. https://doi.org/10.1037/0022-0663.100.3.716
Lang, C., Siemens, G., Wise, A. F., Gašević, D., & Merceron, A. (Eds.). (2022). The handbook of learning analytics (2nd ed.). SoLAR. https://doi.org/10.18608/hla22
Lin, C.-P., & Yang, S.-Y. (2021). Multiple scaffolds used to support self-regulated learning in elementary mathematics classrooms. International Journal of Online Pedagogy and Course Design, 11(4), 1–19. http://doi.org/10.4018/IJOPCD.2021100101
Larrabee Sønderlund, A., Hughes, E., & Smith, J. (2019). The efficacy of learning analytics interventions in higher education: A systematic review. British Journal of Educational Technology, 50(5), 2594–2618. https://doi.org/10.1111/bjet.12720.
Mandinach, E. B., & Abrams, L. M. (2022). Data literacy and learning analytics. In C. Lang, G. Siemens, A. F. Wise, D. Gašević, & A. Merceron (Eds.), The handbook of learning analytics (2nd ed., pp. 196–204). SoLAR. https://doi.org/10.18608/hla22
Martins, R. M., Berge, E., Milrad, M., & Masiello, I. (2019). Visual learning analytics of multidimensional student behavior in self-regulated learning. In M. Scheffel, J. Broisin, V. Pammer-Schindler, A. Ioannou, & J. Schneider (Eds.), Transforming learning with meaningful technologies: 14th European conference on technology enhanced learning, EC-TEL 2019, Delf, The Netherlands, September 16–19, 2019, proceedings (pp. 737–741). Springer. https://doi.org/10.1007/978-3-030-29736-7_78
Masiello, I., Mohseni, Z., Palma, F., Nordmark, S., Augustsson, H., & Rundquist, R. (2024). A current overview of the use of learning analytics dashboards. Education Sciences, 14(1), 82. https://doi.org/10.3390/educsci14010082
McHugh, M. L. (2012). Interrater reliability: The kappa statistic. Biochemia Medica, 22(3), 276–282. https://doi.org/10.11613/BM.2012.031
Molenaar, I., Horvers, A., Dijkstra, R., & Baker, R. S. (2020). Personalized visualizations to promote young learners’ SRL: The learning path app. Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 330–339). ACM Press. https://doi.org/10.1145/3375462.3375465
Molenaar, I., & Knoop-van Campen, C. A. N. (2019). How teachers make dashboard information actionable. IEEE Transactions on Learning Technologies, 12(3), 347–355. https://doi.org/10.1109/TLT.2018.2851585
Moltudal, S. H., Krumsvik, R. J., & Høydal K. L. (2022). Adaptive learning technology in primary education: Implications for professional teacher knowledge and classroom management. Frontiers in Education, 7, 830536. https://doi.org/10.3389/feduc.2022.830536
Mora, T., Escardíbul, J.-O., & Di Pietro, G. (2018). Computers and students’ achievement: An analysis of the one laptop per child program in Catalonia. International Journal of Educational Research, 92, 145–157. https://doi.org/10.1016/j.ijer.2018.09.013
Munn, Z., Peters, M. D. J., Stern, C., Tufanaru, C., McArthur, A., & Aromataris, E. (2018). Systematic review or scoping review? Guidance for authors when choosing between a systematic or scoping review approach. BMC Medical Research Methodology, 18(1), 143. https://doi.org/10.1186/s12874-018-0611-x
National Council of Teachers of Mathematics. (2000). Principles and standards for school mathematics. National Council of Teachers of Mathematics. https://www.nctm.org/Standards-and-Positions/Principles-and-Standards/
Ottestad, G., & Guðmundsdóttir, G. B. (2018). Information and communication technology policy in primary and secondary education in Europe. In J. Voogt, G. Knezek, R. Christensen, & K.-W. Lai (Eds.), Second handbook of information technology in primary and secondary education (pp. 1343–1362). Springer. https://doi.org/10.1007/978-3-319-71054-9_92
Peters, M. D. J., Godfrey, C., McInerney, P., Munn, Z., Tricco, A. C., & Khalil, H. (2020). Chapter 11: Scoping reviews. In E. Aromataris & Z. Munn (Eds.), JBI reviewer’s manual (pp. 407–452). JBI. https://doi.org/10.46658/JBIMES-20-12
Qushem, U. B., Christopoulos, A., & Laakso, M.-J. (2022). Learning management system analytics on arithmetic fluency performance: A skill development case in K6 education. Multimodal Technologies Interaction, 6(8), 61. https://doi.org/10.3390/mti6080061
Ramli, I. S. M., Maat, S. M., & Khalid, F. (2019). Learning analytics in mathematics: A systematic review. International Journal of Academic Research in Progressive Education and Development, 8(4), 436–449. https://doi.org/10.6007/IJARPED/v8-i4/6563
Rodríguez-Martínez, J. A., González-Calero, J. A., del Olmo-Muñoz, J., Arnau, D., & Tirado-Olivares, S. (2023). Building personalised homework from a learning analytics based formative assessment: Effect on fifth-grade students’ understanding of fractions. British Journal of Educational Technology, 54(1), 76–97. https://doi.org/10.1111/bjet.13292
Sahin, M., & Ifenthaler, D. (2021). Visualizations and dashboards for learning analytics: A systematic literature review. In M. Sahin & D. Ifenthaler (Eds.), Visualizations and dashboards for learning analytics (pp. 3–22). Springer Cham. https://doi.org/10.1007/978-3-030-81222-5_1
Scherer, R., Siddiq, F., & Tondeur, J. (2019). The technology acceptance model (TAM): A meta-analytic structural equation modeling approach to explaining teachers’ adoption of digital technology in education. Computers & Education, 128, 13–35. https://doi.org/10.1016/j.compedu.2018.09.009
Schildkamp, K. (2019). Data-based decision-making for school improvement: Research insights and gaps. Educational Research, 61(3), 257–273. https://doi.org/10.1080/00131881.2019.1625716
Schildkamp, K., Karbautzki, L., & Vanhoof, J. (2014). Exploring data use practices around Europe: Identifying enablers and barriers. Studies in Educational Evaluation, 42, 15–24. http://dx.doi.org/10.1016/j.stueduc.2013.10.007
Schildkamp, K., & Kuiper, W. (2010). Data-informed curriculum reform: Which data, what purposes, and promoting and hindering factors. Teaching and Teacher Education, 26(3), 482–496. https://doi.org/10.1016/j.tate.2009.06.007
Schwarz, B. B., Prusak, N., Swidan, O., Livny, A., Gal, K., & Segal, A. (2018). Orchestrating the emergence of conceptual learning: A case study in a geometry class. International Journal of Computer-Supported Collaborative Learning, 13(2), 189–211. https://doi.org/10.1007/s11412-018-9276-z.
Selwyn, N. (2016). Is technology good for education? Polity Press.
Siemens, G., & Baker, R. S. J. d. (2012). Learning analytics and educational data mining: Towards communication and collaboration. Proceedings of the 2nd International Conference on Learning Analytics and Knowledge (LAK ’12), 29 April–2 May 2012, Vancouver, BC, Canada (pp. 252–254). ACM Press. https://doi.org/10.1145/2330601.2330661
Stecker, P. M., & Foegen, A. (2022). Developing an online system to support algebra progress monitoring: Teacher use and feedback. Frontiers in Education, 7. https://doi.org/10.3389/feduc.2022.944836
Taraghi, B., Ebner, M., Saranti, A., & Schön, M. (2014). On using Markov chain to evidence the learning structures and difficulty levels of one digit multiplication. Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (LAK ’14). 24–28 March 2014, Indianapolis, IN, USA (pp. 68–72). ACM Press. https://doi.org/10.1145/2567574.2567614
Tricco, A. C., Lillie, E., Zarin, W., O’Brien, K. K., Colquhoun, H., Levac, D., Moher, D., Peters, M. D. J., Horsley, T., Weeks, L., Hempel, S., Akl, E. A., Chang, C., McGowan, J., Stewart, L., Hartling, L., Aldcroft, A., Wilson, M. G., Garritty, C., … Straus, S. E. (2018). PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation. Annals of Internal Medicine, 169(7), 467–473. https://doi.org/10.7326/M18-0850
Utterberg Modén, M. (2021). Teaching with digital mathematics textbooks: Activity theoretical studies of data-driven technology in classroom practices [doctoral dissertation]. University of Gothenburg. http://hdl.handle.net/2077/69472
Utterberg Modén, M., Tallvid, M., Lundin, J., & Lindström, B. (2021). Intelligent tutoring systems: Why teachers abandoned a technology aimed at automating teaching processes. Proceedings of the 54th Hawaii International Conference on System Sciences (HICSS-54), 5–8 January 2021, Grand Wailea, Maui, HI, USA (pp. 1538–1547). IEEE Computer Society. https://doi.org/10.24251/HICSS.2021.186
van der Kleij, F. M., Vermeulen, J. A., Schildkamp, K., & Eggen, T. J. H. M. (2015). Integrating data-based decision making, assessment for learning and diagnostic testing in formative assessment. Assessment in Education: Principles, Policy & Practice, 22(3), 324–343. https://doi.org/10.1080/0969594X.2014.999024
van Laar, E., van Deursen, A. J. A. M., van Dijk, J. A. G. M., & de Haan, J. (2017). The relation between 21st-century skills and digital skills: A systematic literature review. Computers in Human Behavior, 72, 577–588. https://doi.org/10.1016/j.chb.2017.03.010
van Leeuwen, A., Teasley, S. D., & Wise, A. F. (2022). Teacher and student facing learning analytics. In C. Lang, G. Siemens, A. F. Wise, D. Gašević, & A. Merceron (Eds.), The handbook of learning analytics (2nd ed., pp. 130–140). SoLAR. https://doi.org/10.18608/hla22
Viberg, O., Khalil, M., & Baars, M. (2020). Self-regulated learning and learning analytics in online learning environments: A review of empirical research. Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 524–533). ACM Press. https://doi.org/10.1145/3375462.3375483
Wang, G., Chen, X., Zhang, D., Kang, Y., Wang, F., & Su, M. (2022). Development and application of an intelligent assessment system for mathematics learning strategy among high school students: Take Jianzha County as an example. Sustainability, 14(19), 12265. https://doi.org/10.3390/su141912265
Wang, M.-H., Wang, C.-S., Lee, C.-S., Lin, S.-W., & Hung, P.-H. (2014). Type-2 fuzzy set construction and application for adaptive student assessment system. Proceedings of the 2014 IEEE International Conference on Fuzzy Systems (FUZZ-IEEE 2014), 6–11 July 2014, Beijing, China (pp. 888–894). https://doi.org/10.1109/FUZZ-IEEE.2014.6891894
Webster, M. D. (2017). Questioning technological determinism through empirical research. Symposion, 4(1), 107–125. https://doi.org/10.5840/symposion2017416
Wise, A. F., Zhao, Y., & Hausknecht, S. N. (2014). Learning analytics for online discussions: Embedded and extracted approaches. Journal of Learning Analytics, 1(2), 48‐71. https://doi.org/10.18608/jla.2014.12.4
Wohlfart, O., & Wagner, I. (2023). Teachers’ role in digitalizing education: An umbrella review. Education Technology Research and Development, 71(2), 339–365. https://doi.org/10.1007/s11423-022-10166-0
Yackel, E., & Cobb, P. (1996). Sociomathematical norms, argumentation, and autonomy in mathematics. Journal for Research in Mathematics Education, 27(4), 458–477. https://doi.org/10.2307/749877
Yang, K.-H., & Chen, H.-H. (2023). What increases learning retention: Employing the prediction-observation-explanation learning strategy in digital game-based learning. Interactive Learning Environments, 31(6), 3898–3913. https://doi.org/10.1080/10494820.2021.1944219
Yang, K.-H., & Lu, B.-C. (2021). Towards the successful game-based learning: Detection and feedback to misconceptions is the key. Computers & Education, 160, 104033. https://doi.org/10.1016/j.compedu.2020.104033
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution 4.0 International License.
