Narrowing the Feedback Gap

Examining Student Engagement with Personalized and Actionable Feedback Messages




feedback, learning analytics, higher education, feedback gap, data-driven approaches


One of the major factors affecting student learning is feedback. Although the importance of feedback has been recognized in educational institutions, dramatic changes - such as bigger class sizes and a more diverse student population - challenged the provision of effective feedback. In light of these changes, educators have increasingly been using new digital tools to provide student feedback, given the broader adoption and availability of these new technologies. However, despite these efforts, most educators have limited insight into the recipience of their feedback and wonder which students engage with feedback. This problem is referred to as the "feedback gap," which is the difference between the potential and actual use of feedback, preventing educators and instructional designers from understanding feedback recipience among students. In this study, a set of trackable call-to-action (CTA) links were embedded in feedback messages focused on learning processes and self-regulation of learning in one fully online marketing course and one blended bioscience course. These links helped us examine the association between feedback engagement and course success. We also conducted two focus groups with students from one of the courses to further examine student perceptions of feedback messages. Our results across both courses revealed that early engagement with feedback is positively associated with passing the course and that most students considered feedback messages helpful in their learning. Our study also found some interesting demographic differences between students regarding their engagement with the feedback messages. Such insight enables instructors to ask "why" questions, support students' learning, improve feedback processes, and narrow the gap between potential and actual use of feedback. The practical implications of our findings are further discussed.


Applegate, E. (2005). Strategic copywriting: How to create effective advertising. Lanham, MD, USA: Rowman & Littlefield.

Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: Using learning analytics to increase student success. In Proceedings of the Second International Conference on Learning Analytics and Knowledge (LAK ’12), 29 April–2 May 2012, Vancouver, Canada (pp. 267–270). New York: ACM.

Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64(1), 417–444.

Boud, D., & Molloy, E. (2013a). Feedback in higher and professional education: Understanding it and doing it well. New York, USA: Routledge.

Boud, D., & Molloy, E. (2013b). Rethinking models of feedback for learning: The challenge of design. Assessment & Evaluation in Higher Education, 38(6), 698–712.

Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017). Dashboard for actionable feedback on learning skills: Scalability and usefulness. In P. Zaphiris & A. Ioannou (Eds.), Learning and collaboration technologies. Technology in Education (pp. 229–241). Vancouver, BC, Canada: Springer International Publishing.

Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245–281.

Carless, D. (2020). Longitudinal perspectives on students’ experiences of feedback: A need for teacher–student partnerships. Higher Education Research & Development, 39(3), 1–14.

Carless, D., & Boud, D. (2018). The development of student feedback literacy: Enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315–1325.

Chong, S. W. (2019). College students’ perception of e-feedback: A grounded theory perspective. Assessment & Evaluation in Higher Education, 44(7), 1090–1105.

Chong, S. W. (2020). Reconsidering student feedback literacy from an ecological perspective. Assessment & Evaluation in Higher Education, 46(1), 1–13.

Corrin, L., & de Barba, P. (2014). Exploring students’ interpretation of feedback delivered through learning analytics dashboards. In Proceedings of the ASCILITE 2014 Conference, 23–26 November 2014, Dunedin, NZ (pp. 629–633). ASCILITE. Retrieved from

Corrin, L., & de Barba, P. (2015). How do students interpret feedback delivered via dashboards? In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (LAK 2015), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 430–431). New York: ACM.

Cugelman, B., Thelwall, M., & Dawes, P. (2011). Online interventions for social marketing health behavior change campaigns: A meta-analysis of psychological architectures and adherence factors. Journal of Medical Internet Research, 13(1), e17.

Dawson, P., Henderson, M., Ryan, T., Mahoney, P., Boud, D., Phillips, M., & Molloy, E. (2018). Technology and feedback design. In M. J. Spector, B. B. Lockee, & M. D. Childress (Eds.), Learning, Design, and Technology (pp. 1–45). Cham, Switzerland: Springer International Publishing.

Deighton, J., & Kornfeld, L. (2009). Interactivity’s unanticipated consequences for marketers and marketing. Journal of Interactive Marketing, 23(1), 4–10. (Anniversary Issue)

Ellis, C. (2013). Broadening the scope and increasing the usefulness of learning analytics: The case for assessment analytics. British Journal of Educational Technology, 44(4), 662–664.

Evans, C. (2013). Making sense of assessment feedback in higher education. Review of Educational Research, 83(1), 70–120.

Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1). New York: Springer.

Gannaway, D., Green, T., & Mertova, P. (2018). So how big is big? Investigating the impact of class size on ratings in student evaluation. Assessment & Evaluation in Higher Education, 43(2), 175–184.

Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71.

Gibbs, G., & Simpson, C. (2005). Conditions under which assessment supports students’ learning. Learning and Teaching in Higher Education, 1(1), 3–31. Retrieved from

Glass, R., & Callahan, S. (2014). The big data-driven business: How to use big data to win customers, beat competitors, and boost profits. Hoboken, NJ, USA: John Wiley & Sons.

Grann, J., & Bushway, D. (2014). Competency map: Visualizing student learning to promote student success. In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (LAK 2014), 24–28 March 2014, Indianapolis, IN, USA (pp. 168–172). New York: ACM.

Gunelius, S. (2018). Ultimate guide to email marketing for business. Irvine, CA, USA: Entrepreneur Press.

Hanna, R. C., Swain, S. D., & Smith, J. (2015). Email marketing in a digital world: The basics and beyond. New York: Business Expert Press.

Hartemo, M. (2016). Email marketing in the era of the empowered consumer. Journal of Research in Interactive Marketing, 10(3), 212–230.

Hattie,J. (1999). Influences on student learning. Inaugural Lecture at the University of Auckland, 2 August 1999, Auckland, NZ, 1–25. Retrieved from

Hattie, J., & Timperley, H.(2007).The power of feedback. Review of Educational Research, 77(1), 81–112.

Herodotou, C., Heiser, S., & Rienties, B. (2017). Implementing randomised control trials in open and distance learning: A feasibility study. Open Learning: The Journal of Open, Distance and e-Learning, 32(2), 147–162.

Howell, J. A., Roberts, L. D., & Mancini, V. O. (2018). Learning analytics messages: Impact of grade, sender, comparative information and message style on student affect and academic resilience. Computers in Human Behavior, 89, 8–15.

Iraj, H., Fudge, A., Faulkner, M., Pardo, A., & Kovanović, V. (2020). Understanding students’ engagement with personalised feedback messages. In Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK 2020), 23–27 March 2020, Frankfurt, Germany (pp. 438–447). New York: ACM.

Kizilcec, R., & Brooks, C. (2017). Diverse big data and randomized field experiments in massive open online courses. In C. Lang, G. Siemens, A. F. Wise, & D. Gašević (Eds.), The Handbook of Learning Analytics (pp. 211–222). Society for Learning Analytics Research.

Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical re-view, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284.

Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–89.

Krueger, A. B. (1999). Experimental estimates of education production functions. The Quarterly Journal of Economics, 114(2),497–532.

Lacey, A., & Luff, D. (2001). Qualitative data analysis. Nottingham: Trent Focus Sheffield.

Lim, L.-A., Gentili, S., Pardo, A., Kovanović, V., Whitelock-Wainwright, A., Gašević, D., & Dawson, S. (2019). What changes, and for whom? A study of the impact of learning analytics-based process feedback in a large course. Learning and Instruction, 72, 101202.

Liu, D. Y.-T., Bartimote-Aufflick, K., Pardo, A., & Bridgeman, A. J. (2017). Data-driven personalization of student learning support in higher education. In A. Peña-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends: A view of the current state of the art to enhance e-learning (pp. 143–169). Cham, Switzerland: Springer International Publishing.

Lockyer, L., & Dawson, S. (2011). Learning designs and learning analytics. In Proceedings of the First International Conference on Learning Analytics & Knowledge (LAK 2011), 27 February–1 March 2011, Banff, AB, Canada (pp.153–156).

Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 30–40.

McKay, T., Miller, K., & Tritz, J. (2012). What to do with actionable intelligence: E2coach as an intervention engine. In Proceedings of the Second International Conference on Learning Analytics & Knowledge (LAK 2012), 29 April–2 May2012, Vancouver, BC, Canada (pp. 88–91). New York: ACM.

Molloy, E., Boud, D., & Henderson, M. (2020). Developing a learning-centred framework for feedback literacy. Assessment &Evaluation in Higher Education, 45(4), 527–540.

Moodle. (2019).URL Resource—MoodleDocs. Retrieved from

Nguyen, Q., Huptych, M., & Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance. In Proceedings of the Eighth International Conference on Learning Analytics & Knowledge (LAK 2018), 5–8 March 2018, Sydney, Australia (pp. 141–150).

Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., & Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior, 76,703–714.

Nicol, D. (2010). From monologue to dialogue: Improving written feedback processes in mass higher education. Assessment & Evaluation in Higher Education, 35(5), 501–517.

Nicol, D., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218.

O’Neil, C. (2013). On being a data skeptic. Sebastopol, CA, USA: O’Reilly Media.

Pardo, A., Bartimote, K., Shum, S. B., Dawson, S., Gao, J., Gašević, D., . . . Vigentini, L. (2018). OnTask: Delivering data-informed, personalized learning support actions. Journal of Learning Analytics, 5(3), 235–249.

Pardo, A., Jovanovic, J., Dawson, S., Gašević, D., & Mirriahi, N. (2019). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology, 50(1), 128–138.

Parsons, A. L., & Lepkowska-White, E. (2010). Web site references in print advertising: An analysis of calls to action. Journal of Internet Commerce, 9 (3-4), 151–163.

Price, M., Handley, K., Millar, J., & O’Donovan, B. (2010). Feedback: All that effort, but what is the effect? Assessment & Evaluation in Higher Education, 35(3), 277–289.

R Core Team. (2019). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from

Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M.(2011). pROC: anopen-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12(1), 77.

Rodgers, J. L., & Nicewander, W. A. (1988). Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1), 59–66.

Rowe, A. D. (2017). Feelings about feedback: The role of emotions in assessment for learning. In D. Carless, S. M. Bridges, C. K. Y. Chan, & R. Glofcheski (Eds.), Scaling up assessment for learning in higher education (pp. 159–172). Singapore: Springer Singapore.

Rowe, A. D., & Wood, L. N. (2009). Student perceptions and preferences for feedback. Asian Social Science, 4(3), 78–88.

Ryan, T., Gašević, D., & Henderson, M. (2019). Identifying the impact of feedback over time and at scale: Opportunities for learning analytics. In M. Henderson, R. Ajjawi, D. Boud, & E. Molloy (Eds.), The impact of feedback in higher education: Improving assessment outcomes for learners (pp. 207–223). Cham, Switzerland: Springer International Publishing.

Sahni, N. S., Wheeler, S. C., & Chintagunta, P. (2018). Personalization in email marketing: The role of noninformative advertising content. Marketing Science, 37(2), 236–258.

Senko, C., Hulleman, C. S., & Harackiewicz, J. M. (2011). Achievement goal theory at the cross-roads: Old controversies, current challenges, and new directions. Educational Psychologist, 46(1), 26–47.

Sharp, B. (2017).Marketing: Theory, Evidence, Practice. Oxford, UK: Oxford University Press. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189.

Singh, G., Singh, H., & Shriwastav, S. (2019). Improving email marketing campaign success rate using personalization. In A. K. Laha (Ed.), Advances in Analytics and Applications (pp. 77–83). Singapore: Springer Singapore.

Sutton, P. (2012). Conceptualizing feedback literacy: Knowing, being, and acting. Innovations in Education and Teaching International, 49(1), 31–40.

Szymanski, D. M., & Hise, R. T. (2000). E-satisfaction: An initial examination. Journal of Retailing, 76(3), 309–322.

Teasley, S. D. (2017). Student facing dashboards: One size fits all? Technology, Knowledge and Learning, 22(3), 377–384.

Tempelaar, D., Rienties, B., Mittelmeier, J., & Nguyen, Q. (2018). Student profiling in a dispositional learning analytics application using formative assessment. Computers in Human Behavior, 78, 408–420.

Turner, G., & Gibbs, G. (2010). Are assessment environments gendered? An analysis of the learning responses of male and female students to different assessment environments. Assessment & Evaluation in Higher Education, 35(6), 687–698.

Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277–304). Mahwah, NJ, USA: Lawrence Erlbaum Associates Publishers. Retrieved from

Winstone, N. E. (2019). Facilitating students’ use of feedback: Capturing and tracking impact using digital tools. In M. Henderson, R. Ajjawi, D. Boud, & E. Molloy (Eds.), The impact of feedback in higher education: Improving assessment outcomes for learners (pp. 225–242). Cham, Switzerland: Springer International Publishing.

Winstone, N. E., Mathlin, G., & Nash, R. A. (2019). Building feedback literacy: Students’ perceptions of the developing engagement with feedback toolkit. Frontiers in Education, 4, 1–11.

Winstone, N. E., & Nash, R. (2016). The “Developing Engagement with Feedback Toolkit (DEFT)”: Integrating Assessment literacy into course design (Tech. Rep.). York, UK: Higher Education Academy. Retrieved from

Winstone, N. E., Nash, R. A., Parker, M., & Rowntree, J. (2017). Supporting learners’ agentic engagement with feedback: A systematic review and a taxonomy of recipience processes. Educational Psychologist, 52(1), 17–37.

Winstone, N. E., Nash, R. A., Rowntree, J., & Parker, M. (2017). ‘It’d be useful, but I wouldn’t use it’: Barriers to university students’ feedback seeking and recipience. Studies in Higher Education, 42(11), 2026–2041.

Wisniewski, B., Zierer, K., & Hattie, J. (2020). The power of feedback revisited: A meta-analysis of educational feedback research. Frontiers in Psychology, 10, 3087.

Wong, B. T. M., & Li, K. C. (2018). Learning analytics intervention: A review of case studies. In 2018 International Symposium on Educational Technology (ISET 2018), 31 July–2 August 2018, Osaka, Japan (pp. 178–182). New York: IEEE.

Zhang, X. A., Kumar, V., & Cosguner, K. (2017). Dynamically managing a profitable email marketing program. Journal of Marketing Research, 54(6), 851–866.




How to Cite

Iraj, H., Fudge, A., Khan, H., Faulkner, M., Pardo, A., & Kovanović, V. (2021). Narrowing the Feedback Gap: Examining Student Engagement with Personalized and Actionable Feedback Messages. Journal of Learning Analytics, 8(3), 101-116.



Extended Conference Papers

Most read articles by the same author(s)