Analytics of Learning Strategies: Role of Course Design and Delivery Modality

Authors

  • Wannisa Matcha School of Informatics, University of Edinburgh, and Faculty of Communication Sciences, Prince of Songkla University
  • Dragan Gašević Faculty of Information Technology, Monash University, and School of Informatics, University of Edinburgh https://orcid.org/0000-0001-9265-1908
  • Nora'ayu Ahmad Uzir School of Informatics, University of Edinburgh, and Universiti Teknologi MARA https://orcid.org/0000-0002-9589-2800
  • Jelena Jovanović Faculty of Organizational Sciences, University of Belgrade
  • Abelardo Pardo University of South Australia
  • Lisa Lim University of South Australia
  • Jorge Maldonado-Mahauad Pontificia Universidad Cat´ólica de Chile, and Universidad de Cuenca https://orcid.org/0000-0003-1953-390X
  • Sheridan Gentili Teaching Innovation Unit, University of South Australia
  • Mar Pérez-Sanagustín Pontificia Universidad católica de Chile, and Université Paul Sabatier Toulouse III, Institute de Recherche en Informatique de Toulouse (IRIT)
  • Yi-Shan Tsai School of Informatics, University of Edinburgh

DOI:

https://doi.org/10.18608/jla.2020.72.3

Keywords:

learning strategies, learning tactics, self-regulated learning, data mining, modality, course design

Abstract

Generalizability of the value of methods based on learning analytics remains one of the big challenges in the field of learning analytics. One approach to testing generalizability of a method is to apply it consistently in different learning contexts. This study extends a previously published work by examining the generalizability of a learning analytics method proposed for detecting learning tactics and strategies from trace data. The method was applied to the datasets collected in three different course designs and delivery modalities, including flipped classroom, blended learning, and massive open online course. The proposed method combines process mining and sequence analysis. The detected learning strategies are explored in terms of their association with academic performance. The results indicate the applicability of the proposed method across different learning contexts. Moreover, the findings contribute to the understanding of the learning tactics and strategies identified in the trace data: learning tactics proved to be responsive to the course design, whereas learning strategies were found to be more sensitive to the delivery modalities than to the course design. These findings, well aligned with self-regulated learning theory, highlight the association of learning contexts with the choice of learning tactics and strategies.

References

Baker, R. S. (2019). Challenges for the future of educational data mining: The Baker Learning Analytics Prizes. Journal of Educational Data Mining, 11(1), 1–17. https://dx.doi.org/10.5281/zenodo.3554746

Bernard, R. M., Borokhovski, E., Schmid, R. F., Tamim, R. M., & Abrami, P. C. (2014, 4). A meta-analysis of blended learning and technology use in higher education: from the general to the applied. Journal of Computing in Higher Education, 26(1), 87–122. https://dx.doi.org/10.1007/s12528-013-9077-3

Biggs. (1987). Student approaches to learning and studying. Retrieved from https://eric.ed.gov/?id=ED308201

Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64(1), 120928131529005. https://dx.doi.org/10.1146/annurev-psych-113011-143823

Broadbent, J. (2017). Comparing online and blended learner’s self-regulated learning strategies and academic performance. The Internet and Higher Education, 33, 24–32. https://dx.doi.org/10.1016/j.iheduc.2017.01.004

Broadbent, J., & Poon, W. L. (2015). Self-regulated learning strategies & academic achievement in online higher education learning environments: A systematic review. Internet and Higher Education, 27, 1–13. https://dx.doi.org/10.1016/j.iheduc.2015.04.007

Byrne, M., Flood, B., & Willis, P. (2010). The relationship between learning approaches and learning outcomes: A study of Irish accounting students. Accounting Education, 11, 27–42. https://dx.doi.org/10.1080/09639280210153254

Chonkar, S. P., Ha, T. C., Chu, S. S. H., Ng, A. X., Lim, M. L. S., Ee, T. X., . . . Tan, K. H. (2018). The predominant learning approaches of medical students. BMC Medical Education, 18(1), 1–8. https://dx.doi.org/10.1186/s12909-018-1122-5

Clarebout, G., Elen, J., Collazo, N. A. J., Lust, G., & Jiang, L. (2013, 1). Metacognition and the use of tools. In R. Azevedo & V. Aleven (Eds.), International handbook of metacognition and learning technologies (pp. 187–195). New York: Springer.

Dabbagh, N. (2007). The online learner: Characteristics and pedagogical implications. Contemporary Issues in Technology and Teacher Education, 7(3), 217–226. Retrieved from https://citejournal.org/volume-7/issue-3-07/general/the-online-learner-characteristics-and-pedagogical-implications/

Derry, S. J. (1989). Putting learning strategies to work. Educational Leadership, 47(5), 4–10.

DiFrancesca, D., Nietfeld, J. L., & Cao, L. (2016). A comparison of high and low achieving students on self-regulated learning variables. Learning and Individual Differences, 45, 228–236. https://dx.doi.org/10.1016/j.lindif.2015.11.010

Diseth, A., & Martinsen, (2003). Approaches to learning, cognitive style, and motives as predictors of academic achievement. Educational Psychology, 23(2), 195–207. https://dx.doi.org/10.1080/01443410303225

Dunlosky, J. (2013). Strengthening the student toolbox: Study strategies to boost learning. American Educator, 37(3), 12–21. Retrieved from http://www.aft.org/sites/default/files/periodicals/dunlosky.pdf

Dunlosky, J., Rawson, K. A., Marsh, E. J., Nathan, M. J., & Willingham, D. T. (2013, 1). Improving students’ learning with effective learning techniques: Promising directions from cognitive and educational psychology. Psychological Science in the Public Interest, 14(1), 4–58. https://dx.doi.org/10.1177/1529100612453266

Entwistle, N. J. (1991). Approaches to learning and perceptions of the learning environment: Introduction to the special issue. Higher Eduation, 22(3), 201–204. https://dx.doi.org/10.1007/BF00132287

Entwistle, N. J. (2007, 10). Research into student learning and university teaching. The British Psychological Society, 1–18. Retrieved from https://www.ingentaconnect.com/content/bpsoc/slaut/2007/00000001/00000001/art00001

Eriksson, T., Adawi, T., & St¨ohr, C. (2017, 4). “Time is the bottleneck”: A qualitative study exploring why learners drop out of MOOCs. Journal of Computing in Higher Education, 29(1), 133–146. https://dx.doi.org/10.1007/s12528-016-9127-8

Ferreira, D. R., & Gillblad, D. (2009). Discovering process models from unlabelled event logs. Business Process Management, 5701, 143–158. https://dx.doi.org/10.1007/978-3-642-03848-811

Fincham, O. E., Gaˇsevi´c, D. V., Jovanovi´c, J. M., & Pardo, A. (2018). From study tactics to learning strategies: An analytical method for extracting interpretable representations. IEEE Transactions on Learning Technologies, 1382(c), 1–14. https://dx.doi.org/10.1109/TLT.2018.2823317

Freeman, S., Eddy, S. L., McDonough, M., Smith, M. K., Okoroafor, N., Jordt, H., & Wenderoth, M. P. (2014). Active learning increases student performance in science, engineering, and mathematics. Proceedings of the National Academy of Sciences, 111(23), 8410–8415. https://dx.doi.org/10.1073/pnas.1319030111

Gabadinho, A., Ritschard, G., Studer, M., & Muller, N. S. (2008). Mining sequence data in R with the TraMineR package: A user’s guide. Department of Econometrics and Laboratory of Demography, University of Geneva, Switzerland, 1, 1–124. Retrieved from http://mephisto.unige.ch/traminer

Gaševi´c, D., Dawson, S., Rogers, T., & Gasevi´c, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. Internet and Higher Education, 28, 68–84. https://dx.doi.org/10.1016/j.iheduc.2015.10.002

Gaševi´c, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1). https://dx.doi.org/10.1007/s11528-014-0822-x

Gaševi´c, D., Jovanovi´c, J., Pardo, A., & Dawson, S. (2017). Detecting learning strategies with analytics: Links with self-reported measures and academic performance. Journal of Learning Analytics, 4(2), 113–128. https://dx.doi.org/10.18608/jla.2017.42.10

Gaševi´c, D., Kovanovi´c, V., & Joksimovi´c, S. (2017). Piecing the learning analytics puzzle: A consolidated model of a field of research and practice. Learning: Research and Practice, 3(1), 63–78. https://dx.doi.org/10.1080/23735082.2017.1286142

Gaševi´c, D., Mirriahi, N., Dawson, S., & Joksimovi´c, S. (2017). Effects of instructional conditions and experience on the adoption of a learning tool. Computers in Human Behavior, 67, 207–220. https://dx.doi.org/10.1016/j.chb.2016.10.026

Gatta, R., Lenkowicz, J., Vallati, M., & Stefanini, A. (2017). pMineR: Processes mining in medicine. Retrieved from https://cran.r-project.org/package=pMineR

Hadwin, A. F., Nesbit, J. C., Jamieson-Noel, D., Code, J., &Winne, P. H. (2007). Examining trace data to explore self-regulated learning. Metacognition and Learning, 2(2-3), 107–124. https://dx.doi.org/10.1007/s11409-007-9016-7

Joksimovi´c, S., Kovanovi´c, V., & Dawson, S. (2019, 1). The journey of learning analytics. HERDSA Review of Higher Education, 6, 37–63. Retrieved from www.herdsa.org.au/herdsa-review-higher-education-vol-6/37-63

Jovanovi´c, J., Gaševi´c, D., Dawson, S., Pardo, A., & Mirriahi, N. (2017). Learning analytics to unveil learning strategies in a flipped classroom. The Internet and Higher Education, 33, 74–85. https://dx.doi.org/10.1016/j.iheduc.2017.02.001

Kizilcec, R. F., P´erez-Sanagust´ın, M., & Maldonado, J. J. (2017). Self-regulated learning strategies predict learner behavior and goal attainment in massive open online courses. Computers & Education, 104, 18–33. https://dx.doi.org/http://dx.doi.org/10.1016/j.compedu.2016.10.001

Kovanovi´c, V., Gaˇsevi´c, D., Joksimovi´c, S., Hatala, M., & Olusola, A. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. Internet and Higher Education, 27, 74–89. https://dx.doi.org/10.1016/j.iheduc.2015.06.002

Lai, C. L., & Hwang, G. J. (2016, 9). A self-regulated flipped classroom approach to improving students’ learning performance in a mathematics course. Computers and Education, 100, 126–140. https://dx.doi.org/10.1016/j.compedu.2016.05.006

Lee, J.-s. (2014). The relationship between student engagement and academic performance: Is it a myth or reality? The Journal of Educational Research, 107(3), 177–185. https://dx.doi.org/10.1080/00220671.2013.807491

Lim, L. A., Gentili, S., Pardo, A., Kovanovi´c, V., Whitelock-Wainwright, A., Gaˇsevi´c, D., & Dawson, S. (2019). What changes, and for whom? A study of the impact of learning analytics-based process feedback in a large course. Learning and Instruction. https://dx.doi.org/10.1016/j.learninstruc.2019.04.003

Lust, G., Elen, J., & Clarebout, G. (2013a). Regulation of tool-use within a blended course: Student differences and performance effects. Computers and Education, 60(1), 385–395. https://dx.doi.org/10.1016/j.compedu.2012.09.001

Lust, G., Elen, J., & Clarebout, G. (2013b). Students’ tool-use within a web enhanced course: Explanatory mechanisms of students’ tool-use pattern. Computers in Human Behavior, 29, 2013–2021. https://dx.doi.org/10.1016/j.chb.2013.03.014

Maldonado-Mahauad, J., P´erez-Sanagust´ın, M., Kizilcec, R. F., Morales, N., & Munoz-Gama, J. (2018). Mining theory-based patterns from Big Data: Identifying self-regulated learning strategies in massive open online courses. Computers in Human Behavior, 80, 179–196. https://dx.doi.org/10.1016/j.chb.2017.11.011

Malmberg, J., J¨arvel¨a, S., & Kirschner, P. A. (2014). Elementary school students’ strategic learning: Does task-type matter? Metacognition and Learning, 9(2), 113–136. https://dx.doi.org/10.1007/s11409-013-9108-5

Malmberg, J., J¨arvenoja, H., & J¨arvel¨a, S. (2010). Tracing elementary school students’ study tactic use in gStudy by examining a strategic and self-regulated learning. Computers in Human Behavior, 26(5), 1034–1042. https://dx.doi.org/10.1016/j.chb.2010.03.004

Marton, F., & S¨alj¨o, R. (1976). On qualitative differences in learning: I—Outcome and process. British Journal of Educational Psychology, 46(1), 4–11. https://dx.doi.org/10.1111/j.2044-8279.1976.tb02980.x

Matcha, W., Ahmad Uzir, N., Gaˇsevi´c, D., & Pardo, A. (2019). A systematic review of empirical studies on learning analytics dashboards: A self-regulated learning perspective. IEEE Transactions on Learning Technologies, 1382(c), 1. https://dx.doi.org/10.1109/TLT.2019.2916802

Matcha, W., Gaˇsevi´c, D., Ahmad Uzir, N., Jovanovi´c, J., & Pardo, A. (2019). Analytics of learning strategies: Associations with academic performance and feedback. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (LAK ’19), 4–8 March 2019, Tempe, Arizona, USA (pp. 461–470). New York: ACM. https://dx.doi.org/10.1145/3303772.3303787

Matcha, W., Gaˇsevi´c, D., Ahmad Uzir, N., Jovanovi´c, J., Pardo, A., Maldonado-Mahauad, J., & P´erez-Sanagust´ın, M. (2019). Detection of learning strategies: A comparison of process, sequence and network analytic approaches. In European Conference on Technology Enhanced Learning (EC-TEL 2019), 16–19 September, Delft, Netherlands (pp. 525–540). Springer. https://dx.doi.org/10.1007/978-3-030-29736-739

Mattick, K., Dennis, I., & Bligh, J. (2004). Approaches to learning and studying in medical students: Validation of a revised inventory and its relation to student characteristics and performance. Medical Education, 38(5), 535–543. https://dx.doi.org/10.1111/j.1365-2929.2004.01836.x

McCabe, J. (2011, 4). Metacognitive awareness of learning strategies in undergraduates. Memory & Cognition, 39(3), 462–476. https://dx.doi.org/10.3758/s13421-010-0035-2

Molenaar, I. (2014). Advances in temporal analysis in learning and instruction. Frontline Learning Research, 6, 15–24. https://dx.doi.org/10.14786/flr.v2i4.118

Nandagopal, K., & Ericsson, K. A. (2012). An expert performance approach to the study of individual differences in self-regulated learning activities in upper-level college students. Learning and Individual Differences, 22(5), 597–609. https://dx.doi.org/10.1016/j.lindif.2011.11.018

Nugent, G., Guru, A., & Namuth-Covert, D. (2018). Students’ approaches to e-learning: Analyzing credit/noncredit and high/low performers. Interdisciplinary Journal of e-Skills and Lifelong Learning, 14, 143–158. https://dx.doi.org/10.28945/4133

O’Flaherty, J., & Phillips, C. (2015). The use of flipped classrooms in higher education: A scoping review. Internet and Higher Education, 25, 85–95. https://dx.doi.org/10.1016/j.iheduc.2015.02.002

Olney, T., Rienties, B., & Toetenel, L. (2018). Gathering, visualising and interpreting learning design analytics to inform classroom practice and curriculum design: A student-centred approach from the Open University. In From Data and Analytics to the Classroom: Translating Learning Analytics for Teachers. London: Routledge. Retrieved from https://www.taylorfrancis.com/books/e/9781351113038/chapters/10.4324/9781351113038-6

Pardo, A. (2018). A feedback model for data-rich learning experiences. Assessment & Evaluation in Higher Education, 43(3), 428–438. https://dx.doi.org/10.1080/02602938.2017.1356905

Pardo, A., Gaˇsevi´c, D., Jovanovi´c, J. M., Dawson, S., & Mirriahi, N. (2018). Exploring student interactions with preparation activities in a flipped classroom experience. IEEE Transactions on Learning Technologies. https://dx.doi.org/10.1109/TLT.2018.2858790

Pardo, A., & Mirriahi, N. (2017). Design, deployment and evaluation of a flipped learning first year engineering course. In C. Reidsema, L. Kavanagh, R. Hadgraft, & N. Smith (Eds.), The Flipped Classroom: Practice and Practices in Higher Education (pp. 177–191). Singapore: Springer. https://dx.doi.org/10.1007/978-981-10-3413-811

P´erez-Sanagust´ın, M., Hilliger, I., Alario-Hoyos, C., Kloos, C. D., & Rayyan, S. (2017, 4). H-MOOC framework: Reusing MOOCs for hybrid education. Journal of Computing in Higher Education, 29(1), 47–64. https://dx.doi.org/10.1007/s12528-017-9133-5

Pressley, M., Borkowski, J. G., & Schneider, W. (1987). Cognitive strategies: Good strategy users coordinate metacognition and knowledge. Annals of Child Development, 4(2), 89–129.

Proctor, B. E., Prevatt, F. F., Adams, K. S., Reaser, A., & Petscher, Y. (2006). Study skills profiles of normalachieving and academically-struggling college students. Journal of College Student Development, 47(1), 37–51. https://dx.doi.org/10.1353/csd.2006.0011

Rachal, C. K., Daigle, S., & Rachal, W. S. (2007). Learning problems reported by college students: Are they using learning strategies? Journal of Instructional Psychology, 34(4), 191–199. Retrieved from https://eric.ed.gov/?id=EJ790467

Rahman, A. A., Aris, B., Rosli, M. S., Mohamed, H., Abdullah, Z., & Zaid, N. M. (2015). Significance of preparedness in flipped classroom. Advanced Science Letters, 21(10), 3388–3390. https://dx.doi.org/10.1166/asl.2015.6514

Rodr´ıguez, M. F., Correa, J. H., P´erez-Sanagust´ın, M., Pertuze, J. A., & Alario-Hoyos, C. (2017). A MOOC-based flipped class: Lessons learned from the orchestration perspective. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 10254 LNCS, pp. 102–112). Springer Verlag. https://dx.doi.org/10.1007/978-3-319-59044-812

Tessmer, M., & Richey, R. C. (1997). The role of context in learning and instructional design. Educational Technology Research and Development, 45(2), 85–115. https://dx.doi.org/10.1007/BF02299526

Winne, P. H. (2006). How software technologies can improve research on learning and bolster school reform. Educational Psychologist, 41(1), 5–17. https://dx.doi.org/10.1207/s15326985ep4101

Winne, P. H. (2013). Learning strategies, study skills, and self-regulated learning in postsecondary education. Higher Education: Handbook of Theory and Research(28), 337–403. https://dx.doi.org/10.1007/978-90-481-8598-6

Winne, P. H. (2017). Learning analytics for self-regulated learning. In C. Lang, G. Siemens, A. Wise, & D. Gaševi´c (Eds.), Handbook of learning analytics (pp. 241–249). Society for Learning Analytics Research. https://dx.doi.org/10.18608/hla17.021

Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in Educational Theory and Practice (Vol. 93, pp. 277–304). Lawrence Erlbaum Associates.

Winne, P. H., Jamieson-Noel, D., & Muis, K. (2002). Methodological issues and advances in researching tactics, strategies, and self-regulated learning. In Advances in motivation and achievement: New directions in measures and methods (Vol. 12, pp. 121–155). Emerald Group.

Yip, M. C. W. (2007). Differences in learning and study strategies between high and low achieving university students: A hong kong study. Educational Psychology, 27(4), 597–606. https://dx.doi.org/10.1080/01443410701309126

Zeegers, P. (2001). Approaches to learning in science: A longitudinal study. British Journal of Educational Psychology, 71(1), 115–132. https://dx.doi.org/10.1348/000709901158424

Zhou, M., & Winne, P. H. (2012). Modeling academic achievement by self-reported versus traced goal orientation. Learning and Instruction, 22(6), 413–419. https://dx.doi.org/10.1016/j.learninstruc.2012.03.004

Zhu, Y., Au, W., & Yates, G. (2016). University students’ self-control and self-regulated learning in a blended course. Internet and Higher Education, 30, 54–62. https://dx.doi.org/10.1016/j.iheduc.2016.04.001

Zimmerman, B. J. (1986). Becoming a self-regulated learner: Which are the key subprocesses? Contemporary Educational Psychology, 11(4), 307–313. https://dx.doi.org/10.1016/0361-476X(86)90027-5

Downloads

Published

2020-09-19

How to Cite

Matcha, W., Gašević, D., Ahmad Uzir, N., Jovanović, J., Pardo, A., Lim, L., Maldonado-Mahauad, J., Gentili, S., Pérez-Sanagustín, M., & Tsai, Y.-S. (2020). Analytics of Learning Strategies: Role of Course Design and Delivery Modality. Journal of Learning Analytics, 7(2), 45-71. https://doi.org/10.18608/jla.2020.72.3

Issue

Section

LAK Extended Papers