Combining Checkpoint and Process Learning Analytics to Support Learning Design Decisions in Blended Learning Environments

Authors

DOI:

https://doi.org/10.18608/jla.2020.73.4

Keywords:

Learning analytics, Learning design, Blended learning, Teachers, Canvas, Social network analysis, Text network analysis

Abstract

Learning analytics (LA) constitutes a key opportunity to support learning design (LD) in blended learning environments. However, details as to how LA supports LD in practice and information on teacher experiences with LA are limited. This study explores the potential of LA to inform LD based on a one-semester undergraduate blended learning course at a Norwegian university. Our findings indicate that creating valuable connections between LA and LD requires a detailed analysis of student checkpoints (e.g., online logins) and process analytics (e.g., online content and interaction dynamics) to find meaningful learning behaviour patterns that can be forwarded to teachers in retrospect to support the redesign of courses. Moreover, the teachers in our study found the LA visualizations to be valuable for understanding student online learning processes, but they also requested the timely sharing of aggregated LA visualizations in a simple, easy-to-interpret format, yet detailed enough to be informative and actionable. We conclude the paper by arguing that the potential of LA to support LD is improved when multiple levels of LA are considered.

Author Biographies

Rogers Kaliisa, University of Oslo

Doctoral Research Fellow, The University of Oslo, Norway

Anders I. Mørch, University of Oslo

Professor, Department of Education, University of Oslo, Norway

References

Agostinho, S. (2011). The use of a visual learning design representation to support the design process of teaching in higher education. Australasian Journal of Educational Technology, 27(6). https://https://dx.doi.org/10.14742/ajet.923

Alhadad, S. S., & Thompson, K. (2017). Understanding the mediating role of teacher inquiry when connecting learning analytics with design for learning. Interaction Design and Architecture(s) Journal, 33, 54–74. Retrieved from https://eprints.qut.edu.au/202493/1/62149001.pdf

Andersen, R., & Mørch, A. I. (2016). Mutual development in mass collaboration: Identifying interaction patterns in customer-initiated software product development. Computers in Human Behavior, 65, 77–91. https://dx.doi.org/10.1016/j.chb.2016.08.005

Bakharia, A., Corrin, L., de Barba, P., Kennedy, G., Gašević, D., Mulder, R., Lockyer, L. (2016). A conceptual framework linking learning design with learning analytics. Proceedings of the 6th International Conference on Learning Analytics and Knowledge (LAK ʼ16), 25–29 April 2016, Edinburgh, UK (pp. 329–338). New York, NY: ACM. https://dx.doi.org/10.1145/2883851.2883944

Bennett, S., Agostinho, S., & Lockyer, L. (2015). Technology tools to support learning design: Implications derived from an investigation of university teachers’ design practices. Computers & Education, 81, 211–220. https://dx.doi.org/10.1016/j.compedu.2014.10.016

Benesty, J., Chen, J., Huang, Y., & Cohen, I. (2009). Pearson correlation coefficient. In I. Cohen, Y. Huang, J. Chen, & J. Benesty (Eds.), Noise reduction in speech processing (pp. 1–4). Berlin, Germany: Springer. https://dx.doi.org/10.1007/978-3-642-00296-0_5

Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-centred learning analytics. Journal of Learning Analytics, 6(2), 1–9. https://dx.doi.org/10.18608/jla.2019.62.1

Bonk, C. J., & Graham, C. R. (2012). The handbook of blended learning: Global perspectives, local designs. Hoboken, NJ: John Wiley & Sons. Retrieved from https://books.google.no/books?hl=en&lr=&id=2u2TxK06PwUC&oi=fnd&pg=PT14&dq=

Conole, G. (2012). Designing for learning in an open world (Vol. 4). Leicester, UK: Springer Science & Business Media. Retrieved from https://dl.uswr.ac.ir/bitstream/Hannan/130952/1/9781441985163.pdf https://dx.doi.org/10.1007/978-1-4419-8517-0

Dobozy, E., & Cameron, L. (2018). Special issue on learning design research: Mapping the terrain. Australasian Journal of Educational Technology, 34(2). https://dx.doi.org/10.14742/ajet.4390

Eradze, M., Rodriguez Triana, M. J., & Laanpere, M. (2017). Semantically annotated lesson observation data in learning analytics datasets: A reference model. Interaction Design and Architecture(s) Journal, 33, 75–91. Retrieved from http://www.mifav.uniroma2.it/inevent/events/idea2010/doc/33_4.pdf

Fritz, J. (2016). LMS course design as learning analytics variable. In J. Greer, M. Molinaro, X. Ochoa, & T. McKay (Eds.), Proceedings of the 1st Learning Analytics for Curriculum and Program Quality Improvement Workshop (PCLA 2016), 25 April 2016, Edinburgh, UK (pp. 15–19). Retrieved from https://www.semanticscholar.org/paper/LMS-Course-Design-As-Learning-Analytics-Variable-Fritz/be7e8fe65384513355467e56cc84b36829669482

Gašević, D., Dawson, S., Rogers, T., & Gasevic, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting academic success. The Internet and Higher Education, 28, 68–84. https://dx.doi.org/10.1016/j.iheduc.2015.10.002

Griffiths, D. (2017). The use of models in learning design and learning analytics. Interaction Design and Architecture(s) Journal, 33, 113–133. Retrieved from http://ubir.bolton.ac.uk/1330/1/Griffiths%20D%20IxD%26A%20article.pdf

Gunn, C., McDonald, J., Donald, C., Nichols, M., Milne, J., & Blumenstein, M. (2017). Building an evidence base for teaching and learning design using learning analytics. Wellington, Australia: Ako Aotearoa – The National Centre for Tertiary Teaching Excellence. Retrieved from https://ako.ac.nz/assets/Knowledge-centre/NPF-15-008

Haya, P. A., Daems, O., Malzahn, N., Castellanos, J., & Hoppe, H. U. (2015). Analysing content and patterns of interaction for improving the learning design of networked learning environments. British Journal of Educational Technology, 46(2), 300–316. https://dx.doi.org/10.1111/bjet.12264

Hernández‐Leo, D., Martinez‐Maldonado, R., Pardo, A., Muñoz‐Cristóbal, J. A., & Rodríguez‐Triana, M. J. (2019). Analytics for learning design: A layered framework and tools. British Journal of Educational Technology, 50(1), 139–152. https://dx.doi.org/10.1111/bjet.12645

Harel, D., & Koren, Y. (2001). A fast multi-scale method for drawing large graphs. In J. Marks (Ed.) Graph drawing. GD 2000. Notes in computer science (Vol. 1984). Berlin, Germany: Springer. https://dx.doi.org/10.1007/3-540-44541-2_18

Ifenthaler, D., Gibson, D., & Dobozy, E. (2018). Informing learning design through analytics: Applying network graph analysis. Australasian Journal of Educational Technology, 34(2), 117–132. https://dx.doi.org/10.14742/ajet.3767

Kaliisa, R., Mørch, A. I., & Kluge, A. (2019). Exploring social learning analytics to support teaching and learning decisions in online learning environments. In M. Scheffel, J. Broisin, V. Pammer-Schindler, A. Ioannou, & J. Schneider (Eds.), Transforming learning with meaningful technologies. Proceedings of the 14th European Conference on Technology Enhanced Learning (EC-TEL 2019), 16–19 September 2019, Delft, The Netherlands. Lecture Notes in Computer Science (Vol. 11722) (pp. 187–198). Cham, Switzerland: Springer. https://dx.doi.org/10.1007/978-3-030-29736-7_14

Kaliisa, R., Kluge, A., & Mørch, A. I. (in press a). Overcoming the challenges to learning analytics adoption at the practitioner level: A critical analysis of 18 learning analytics frameworks. Scandinavian Journal of Educational Research.

Kaliisa, R., Kluge, A., & Mørch, A. I.. (in press b). “My point of departure for analytics is extreme skepticism”: Implications derived from an investigation of university teachers’ learning analytics perspectives and design practices. Technology, Knowledge & Learning.

Knight, S., Shum, S. B., & Littleton, K. (2014). Epistemology, assessment, pedagogy: Where learning meets analytics in the middle space. Journal of Learning Analytics, 1(2), 23–47. https://dx.doi.org/10.18608/jla.2014.12.3

Law, N., Li, L., Herrera, L. F., Chan, A., & Pong, T.-C. (2017). A pattern language based learning design studio for an analytics informed inter-professional design community. Interaction Design and Architecture(s) Journal, 33, 92–112. Retrieved from http://www.mifav.uniroma2.it/inevent/events/idea2010/doc/33_5.pdf

Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459. https://dx.doi.org/10.1177/0002764213479367

Macfadyen, L. P., & Dawson, S. (2010). Mining LMS data to develop an “early warning system” for educators: A proof of concept. Computers & Education, 54(2), 588–599. https://dx.doi.org/10.1016/j.compedu.2009.09.008

Macfadyen, L. P., & Dawson, S. (2012). Numbers are not enough: Why e-learning analytics failed to inform an institutional strategic plan. Journal of Educational Technology & Society, 15(3), 149–163. Retrieved from https://www.jstor.org/stable/10.2307/jeductechsoci.15.3.149

Mangaroska, K., & Giannakos, M. N. (2018). Learning analytics for learning design: A systematic literature review of analytics-driven design to enhance learning. IEEE Transactions on Learning Technologies, 12(4), 516–534. https://dx.doi.org/10.1109/TLT.2018.2868673

Martinez-Maldonado, R., Kay, J., Yacef, K., & Schwendimann, B. (2012). An interactive teacher’s dashboard for monitoring groups in a multi-tabletop learning environment. In S. A. Cerri, W. J. Clancey, G. Papadourakis, & K. Panourgia (Eds.), Proceedings of the 11th International Conference on Intelligent Tutoring Systems (ITS 2012), 14–18 June 2012, Chania, Greece,. Lecture notes in computer science (Vol. 7315) (pp. 482–492). Berlin, Germany: Springer. https://dx.doi.org/10.1007/978-3-642-30950-2_62

McKenney, S., & Mor, Y. (2015). Supporting teachers in data‐informed educational design. British Journal of Educational Technology, 46(2), 265–279. https://dx.doi.org/10.1111/bjet.12262

Michos, K., Hernández‐Leo, D., & Albó, L. (2018). Teacher‐led inquiry in technology‐supported school communities. British Journal of Educational Technology, 49(6), 1077–1095. https://dx.doi.org/10.1111/bjet.12696

Mor, Y., Ferguson, R., & Wasson, B. (2015). Learning design, teacher inquiry into student learning and learning analytics: A call for action. British Journal of Educational Technology, 46(2), 221–229. https://dx.doi.org/10.1111/bjet.12273

Melero, J., Hernández‐Leo, D., Sun, J., Santos, P., & Blat, J. (2015). How was the activity? A visualization support for a case of location‐based learning design. British Journal of Educational Technology, 46(2), 317–329. https://dx.doi.org/10.1111/bjet.12238

Nguyen, Q., Huptych, M., & Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 141–150). New York, NY: ACM. https://dx.doi.org/10.1145/3170358.3170398

Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., & Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior, 76, 703–714. https://dx.doi.org/10.1016/j.chb.2017.03.028

Paranyushkin, D. (2019). InfraNodus: Generating insight using text network analysis. Proceedings of the 28th International Conference on World Wide Web (WWW ’19), 13–17 May 2019, San Francisco, CA, USA (pp. 3584–3589). New York, NY: ACM. https://dx.doi.org/10.1145/3308558.3314123

Persico, D., & Pozzi, F. (2015). Informing learning design with learning analytics to improve teacher inquiry. British Journal of Educational Technology, 46(2), 230–248. https://dx.doi.org/10.1111/bjet.12207

Rienties, B., Cross, S., & Zdrahal, Z. (2017). Implementing a learning analytics intervention and evaluation framework: What works? In D. Kei Daniel (Ed.) Big data and learning analytics in higher education (pp. 147–166). Cham, Switzerland: Springer. https://dx.doi.org/10.1007/978-3-319-06520-5_10

Rienties, B., & Toetenel, L. (2016). The impact of learning design on student behaviour, satisfaction and performance: A cross-institutional comparison across 151 modules. Computers in Human Behavior, 60, 333–341. https://dx.doi.org/10.1016/j.chb.2016.02.074

Rodríguez‐Triana, M. J., Martínez‐Monés, A., Asensio‐Pérez, J. I., & Dimitriadis, Y. (2015). Scripting and monitoring meet each other: Aligning learning analytics and learning design to support teachers in orchestrating CSCL situations. British Journal of Educational Technology, 46(2), 330–343. https://dx.doi.org/10.1111/bjet.12198

Samuelsen, J., Chen, W., & Wasson, B. (2019). Integrating multiple data sources for learning analytics: Review of literature. Research and Practice in Technology Enhanced Learning, 14(11), 1-20. https://dx.doi.org/10.1186/s41039-019-0105-4

Shibani, A., Knight, S., & Shum, S. B. (2019). Contextualizable learning analytics design: A generic model and writing analytics evaluations. Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2019, Tempe, AZ, USA (pp. 210–219). New York, NY: ACM. https://dx.doi.org/10.1145/3303772.3303785

Smith, M. A., Shneiderman, B., Milic-Frayling, N., Mendes Rodrigues, E., Barash, V., Dunne, C., & Gleave, E. (2009). Analyzing (social media) networks with NodeXL. Proceedings of the 4th International Conference on Communities and Technologies (C&T ’09), 25–27 June 2009, University Park, PA, USA (pp. 255–264). New York, NY: ACM. https://dx.doi.org/10.1145/1556460.1556497

Wise, A. F., & Jung, Y. (2019). Teaching with analytics: Towards a situated model of instructional decision-making. Journal of Learning Analytics, 6(2), 53–69. https://dx.doi.org/10.18608/jla.2019.62.4

Wise, A. F., & Shaffer, D. W. (2015). Why theory matters more than ever in the age of big data. Journal of Learning Analytics, 2(2), 5–13. https://dx.doi.org/10.18608/jla.2015.22.2

Downloads

Published

2020-12-17

How to Cite

Kaliisa, R., Kluge, A., & Mørch, A. I. (2020). Combining Checkpoint and Process Learning Analytics to Support Learning Design Decisions in Blended Learning Environments. Journal of Learning Analytics, 7(3), 33-47. https://doi.org/10.18608/jla.2020.73.4

Issue

Section

Special Section: Learning Design and Learning Analytics

Most read articles by the same author(s)