Narrowing the Feedback Gap

Examining Student Engagement with Personalized and Actionable Feedback Messages

Authors

DOI:

https://doi.org/10.18608/jla.2021.7184

Keywords:

feedback, learning analytics, higher education, feedback gap, data-driven approaches

Abstract

One of the major factors affecting student learning is feedback. Although the importance of feedback has been recognized in educational institutions, dramatic changes - such as bigger class sizes and a more diverse student population - challenged the provision of effective feedback. In light of these changes, educators have increasingly been using new digital tools to provide student feedback, given the broader adoption and availability of these new technologies. However, despite these efforts, most educators have limited insight into the recipience of their feedback and wonder which students engage with feedback. This problem is referred to as the "feedback gap," which is the difference between the potential and actual use of feedback, preventing educators and instructional designers from understanding feedback recipience among students. In this study, a set of trackable call-to-action (CTA) links were embedded in feedback messages focused on learning processes and self-regulation of learning in one fully online marketing course and one blended bioscience course. These links helped us examine the association between feedback engagement and course success. We also conducted two focus groups with students from one of the courses to further examine student perceptions of feedback messages. Our results across both courses revealed that early engagement with feedback is positively associated with passing the course and that most students considered feedback messages helpful in their learning. Our study also found some interesting demographic differences between students regarding their engagement with the feedback messages. Such insight enables instructors to ask "why" questions, support students' learning, improve feedback processes, and narrow the gap between potential and actual use of feedback. The practical implications of our findings are further discussed.

References

Applegate, E. (2005). Strategic copywriting: How to create effective advertising. Lanham, MD, USA: Rowman & Littlefield.

Arnold, K. E., & Pistilli, M. D. (2012). Course signals at Purdue: Using learning analytics to increase student success. In Proceedings of the Second International Conference on Learning Analytics and Knowledge (LAK ’12), 29 April–2 May 2012, Vancouver, Canada (pp. 267–270). New York: ACM. https://doi.org/https://doi.org/10.1145/2330601.2330666

Bjork, R. A., Dunlosky, J., & Kornell, N. (2013). Self-regulated learning: Beliefs, techniques, and illusions. Annual Review of Psychology, 64(1), 417–444. https://doi.org/10.1146/annurev-psych-113011-143823

Boud, D., & Molloy, E. (2013a). Feedback in higher and professional education: Understanding it and doing it well. New York, USA: Routledge.

Boud, D., & Molloy, E. (2013b). Rethinking models of feedback for learning: The challenge of design. Assessment & Evaluation in Higher Education, 38(6), 698–712. https://doi.org/10.1080/02602938.2012.691462

Broos, T., Peeters, L., Verbert, K., Van Soom, C., Langie, G., & De Laet, T. (2017). Dashboard for actionable feedback on learning skills: Scalability and usefulness. In P. Zaphiris & A. Ioannou (Eds.), Learning and collaboration technologies. Technology in Education (pp. 229–241). Vancouver, BC, Canada: Springer International Publishing. https://doi.org/10.1007/978-3-319-58515-418

Butler, D. L., & Winne, P. H. (1995). Feedback and self-regulated learning: A theoretical synthesis. Review of Educational Research, 65(3), 245–281. https://doi.org/10.3102/00346543065003245

Carless, D. (2020). Longitudinal perspectives on students’ experiences of feedback: A need for teacher–student partnerships. Higher Education Research & Development, 39(3), 1–14. https://doi.org/10.1080/07294360.2019.1684455

Carless, D., & Boud, D. (2018). The development of student feedback literacy: Enabling uptake of feedback. Assessment & Evaluation in Higher Education, 43(8), 1315–1325. https://doi.org/10.1080/02602938.2018.1463354

Chong, S. W. (2019). College students’ perception of e-feedback: A grounded theory perspective. Assessment & Evaluation in Higher Education, 44(7), 1090–1105. https://doi.org/10.1080/02602938.2019.1572067

Chong, S. W. (2020). Reconsidering student feedback literacy from an ecological perspective. Assessment & Evaluation in Higher Education, 46(1), 1–13. https://doi.org/10.1080/02602938.2020.1730765

Corrin, L., & de Barba, P. (2014). Exploring students’ interpretation of feedback delivered through learning analytics dashboards. In Proceedings of the ASCILITE 2014 Conference, 23–26 November 2014, Dunedin, NZ (pp. 629–633). ASCILITE. Retrieved from https://ascilite.org/conferences/dunedin2014/files/concisepapers/223-Corrin.pdf

Corrin, L., & de Barba, P. (2015). How do students interpret feedback delivered via dashboards? In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (LAK 2015), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 430–431). New York: ACM. https://doi.org/10.1145/2723576.2723662

Cugelman, B., Thelwall, M., & Dawes, P. (2011). Online interventions for social marketing health behavior change campaigns: A meta-analysis of psychological architectures and adherence factors. Journal of Medical Internet Research, 13(1), e17. https://doi.org/10.2196/jmir.1367

Dawson, P., Henderson, M., Ryan, T., Mahoney, P., Boud, D., Phillips, M., & Molloy, E. (2018). Technology and feedback design. In M. J. Spector, B. B. Lockee, & M. D. Childress (Eds.), Learning, Design, and Technology (pp. 1–45). Cham, Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-319-17727-4124-1

Deighton, J., & Kornfeld, L. (2009). Interactivity’s unanticipated consequences for marketers and marketing. Journal of Interactive Marketing, 23(1), 4–10. (Anniversary Issue) https://doi.org/10.1016/j.intmar.2008.10.001

Ellis, C. (2013). Broadening the scope and increasing the usefulness of learning analytics: The case for assessment analytics. British Journal of Educational Technology, 44(4), 662–664. https://doi.org/10.1111/bjet.12028

Evans, C. (2013). Making sense of assessment feedback in higher education. Review of Educational Research, 83(1), 70–120. https://doi.org/10.3102/0034654312474350

Friedman, J., Hastie, T., & Tibshirani, R. (2001). The elements of statistical learning (Vol. 1). New York: Springer. https://doi.org/10.1007/978-0-387-84858-7

Gannaway, D., Green, T., & Mertova, P. (2018). So how big is big? Investigating the impact of class size on ratings in student evaluation. Assessment & Evaluation in Higher Education, 43(2), 175–184. https://doi.org/10.1080/02602938.2017.1317327

Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. https://doi.org/10.1007/s11528-014-0822-x

Gibbs, G., & Simpson, C. (2005). Conditions under which assessment supports students’ learning. Learning and Teaching in Higher Education, 1(1), 3–31. Retrieved from https://eprints.glos.ac.uk/3609/

Glass, R., & Callahan, S. (2014). The big data-driven business: How to use big data to win customers, beat competitors, and boost profits. Hoboken, NJ, USA: John Wiley & Sons.

Grann, J., & Bushway, D. (2014). Competency map: Visualizing student learning to promote student success. In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (LAK 2014), 24–28 March 2014, Indianapolis, IN, USA (pp. 168–172). New York: ACM. https://doi.org/10.1145/2567574.2567622

Gunelius, S. (2018). Ultimate guide to email marketing for business. Irvine, CA, USA: Entrepreneur Press.

Hanna, R. C., Swain, S. D., & Smith, J. (2015). Email marketing in a digital world: The basics and beyond. New York: Business Expert Press.

Hartemo, M. (2016). Email marketing in the era of the empowered consumer. Journal of Research in Interactive Marketing, 10(3), 212–230. https://doi.org/10.1108/JRIM-06-2015-0040

Hattie,J. (1999). Influences on student learning. Inaugural Lecture at the University of Auckland, 2 August 1999, Auckland, NZ, 1–25. Retrieved from https://cdn.auckland.ac.nz/assets/education/about/research/documents/influences-on-student-learning.pdf

Hattie, J., & Timperley, H.(2007).The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487

Herodotou, C., Heiser, S., & Rienties, B. (2017). Implementing randomised control trials in open and distance learning: A feasibility study. Open Learning: The Journal of Open, Distance and e-Learning, 32(2), 147–162. https://doi.org/10.1080/02680513.2017.1316188

Howell, J. A., Roberts, L. D., & Mancini, V. O. (2018). Learning analytics messages: Impact of grade, sender, comparative information and message style on student affect and academic resilience. Computers in Human Behavior, 89, 8–15. https://doi.org/10.1016/j.chb.2018.07.021

Iraj, H., Fudge, A., Faulkner, M., Pardo, A., & Kovanović, V. (2020). Understanding students’ engagement with personalised feedback messages. In Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK 2020), 23–27 March 2020, Frankfurt, Germany (pp. 438–447). New York: ACM. https://doi.org/10.1145/3375462.3375527

Kizilcec, R., & Brooks, C. (2017). Diverse big data and randomized field experiments in massive open online courses. In C. Lang, G. Siemens, A. F. Wise, & D. Gašević (Eds.), The Handbook of Learning Analytics (pp. 211–222). Society for Learning Analytics Research. https://doi.org/10.18608/hla17.018

Kluger, A. N., & DeNisi, A. (1996). The effects of feedback interventions on performance: A historical re-view, a meta-analysis, and a preliminary feedback intervention theory. Psychological Bulletin, 119(2), 254–284. https://doi.org/10.1037/0033-2909.119.2.254

Kovanović, V., Gašević, D., Joksimović, S., Hatala, M., & Adesope, O. (2015). Analytics of communities of inquiry: Effects of learning technology use on cognitive presence in asynchronous online discussions. The Internet and Higher Education, 27, 74–89. https://doi.org/10.1016/j.iheduc.2015.06.002

Krueger, A. B. (1999). Experimental estimates of education production functions. The Quarterly Journal of Economics, 114(2),497–532. https://doi.org/10.1162/003355399556052

Lacey, A., & Luff, D. (2001). Qualitative data analysis. Nottingham: Trent Focus Sheffield.

Lim, L.-A., Gentili, S., Pardo, A., Kovanović, V., Whitelock-Wainwright, A., Gašević, D., & Dawson, S. (2019). What changes, and for whom? A study of the impact of learning analytics-based process feedback in a large course. Learning and Instruction, 72, 101202. https://doi.org/10.1016/j.learninstruc.2019.04.003

Liu, D. Y.-T., Bartimote-Aufflick, K., Pardo, A., & Bridgeman, A. J. (2017). Data-driven personalization of student learning support in higher education. In A. Peña-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends: A view of the current state of the art to enhance e-learning (pp. 143–169). Cham, Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-319-52977-65

Lockyer, L., & Dawson, S. (2011). Learning designs and learning analytics. In Proceedings of the First International Conference on Learning Analytics & Knowledge (LAK 2011), 27 February–1 March 2011, Banff, AB, Canada (pp.153–156). https://doi.org/10.1145/2090116.2090140

Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 30–40. https://doi.org/10.17471/2499-4324/195

McKay, T., Miller, K., & Tritz, J. (2012). What to do with actionable intelligence: E2coach as an intervention engine. In Proceedings of the Second International Conference on Learning Analytics & Knowledge (LAK 2012), 29 April–2 May2012, Vancouver, BC, Canada (pp. 88–91). New York: ACM. https://doi.org/10.1145/2330601.2330627

Molloy, E., Boud, D., & Henderson, M. (2020). Developing a learning-centred framework for feedback literacy. Assessment &Evaluation in Higher Education, 45(4), 527–540. https://doi.org/10.1080/02602938.2019.1667955

Moodle. (2019).URL Resource—MoodleDocs. Retrieved from https://docs.moodle.org/37/en/URLresource

Nguyen, Q., Huptych, M., & Rienties, B. (2018). Linking students’ timing of engagement to learning design and academic performance. In Proceedings of the Eighth International Conference on Learning Analytics & Knowledge (LAK 2018), 5–8 March 2018, Sydney, Australia (pp. 141–150). https://doi.org/10.1145/3170358.3170398

Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, R., & Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behavior, 76,703–714. https://doi.org/10.1016/j.chb.2017.03.028

Nicol, D. (2010). From monologue to dialogue: Improving written feedback processes in mass higher education. Assessment & Evaluation in Higher Education, 35(5), 501–517. https://doi.org/10.1080/02602931003786559

Nicol, D., & Macfarlane-Dick, D. (2006). Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in Higher Education, 31(2), 199–218. https://doi.org/10.1080/03075070600572090

O’Neil, C. (2013). On being a data skeptic. Sebastopol, CA, USA: O’Reilly Media.

Pardo, A., Bartimote, K., Shum, S. B., Dawson, S., Gao, J., Gašević, D., . . . Vigentini, L. (2018). OnTask: Delivering data-informed, personalized learning support actions. Journal of Learning Analytics, 5(3), 235–249. https://doi.org/10.18608/jla.2018.53.15

Pardo, A., Jovanovic, J., Dawson, S., Gašević, D., & Mirriahi, N. (2019). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology, 50(1), 128–138. https://doi.org/10.1111/bjet.12592

Parsons, A. L., & Lepkowska-White, E. (2010). Web site references in print advertising: An analysis of calls to action. Journal of Internet Commerce, 9 (3-4), 151–163. https://doi.org/10.1080/15332861.2010.526487

Price, M., Handley, K., Millar, J., & O’Donovan, B. (2010). Feedback: All that effort, but what is the effect? Assessment & Evaluation in Higher Education, 35(3), 277–289. https://doi.org/10.1080/02602930903541007

R Core Team. (2019). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing. Retrieved from https://www.R-project.org/

Robin, X., Turck, N., Hainard, A., Tiberti, N., Lisacek, F., Sanchez, J.-C., & Müller, M.(2011). pROC: anopen-source package for R and S+ to analyze and compare ROC curves. BMC Bioinformatics, 12(1), 77. https://doi.org/10.1186/1471-2105-12-77

Rodgers, J. L., & Nicewander, W. A. (1988). Thirteen ways to look at the correlation coefficient. The American Statistician, 42(1), 59–66. https://doi.org/10.1080/00031305.1988.10475524

Rowe, A. D. (2017). Feelings about feedback: The role of emotions in assessment for learning. In D. Carless, S. M. Bridges, C. K. Y. Chan, & R. Glofcheski (Eds.), Scaling up assessment for learning in higher education (pp. 159–172). Singapore: Springer Singapore. https://doi.org/10.1007/978-981-10-3045-111

Rowe, A. D., & Wood, L. N. (2009). Student perceptions and preferences for feedback. Asian Social Science, 4(3), 78–88. https://doi.org/10.5539/ass.v4n3p78

Ryan, T., Gašević, D., & Henderson, M. (2019). Identifying the impact of feedback over time and at scale: Opportunities for learning analytics. In M. Henderson, R. Ajjawi, D. Boud, & E. Molloy (Eds.), The impact of feedback in higher education: Improving assessment outcomes for learners (pp. 207–223). Cham, Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-030-25112-312

Sahni, N. S., Wheeler, S. C., & Chintagunta, P. (2018). Personalization in email marketing: The role of noninformative advertising content. Marketing Science, 37(2), 236–258. https://doi.org/10.1287/mksc.2017.1066

Senko, C., Hulleman, C. S., & Harackiewicz, J. M. (2011). Achievement goal theory at the cross-roads: Old controversies, current challenges, and new directions. Educational Psychologist, 46(1), 26–47. https://doi.org/10.1080/00461520.2011.538646

Sharp, B. (2017).Marketing: Theory, Evidence, Practice. Oxford, UK: Oxford University Press. Shute, V. J. (2008). Focus on formative feedback. Review of Educational Research, 78(1), 153–189. https://doi.org/10.3102/0034654307313795

Singh, G., Singh, H., & Shriwastav, S. (2019). Improving email marketing campaign success rate using personalization. In A. K. Laha (Ed.), Advances in Analytics and Applications (pp. 77–83). Singapore: Springer Singapore. https://doi.org/10.1007/978-981-13-1208-38

Sutton, P. (2012). Conceptualizing feedback literacy: Knowing, being, and acting. Innovations in Education and Teaching International, 49(1), 31–40. https://doi.org/10.1080/14703297.2012.647781

Szymanski, D. M., & Hise, R. T. (2000). E-satisfaction: An initial examination. Journal of Retailing, 76(3), 309–322. https://doi.org/10.1016/S0022-4359(00)00035-X

Teasley, S. D. (2017). Student facing dashboards: One size fits all? Technology, Knowledge and Learning, 22(3), 377–384. https://doi.org/10.1007/s10758-017-9314-3

Tempelaar, D., Rienties, B., Mittelmeier, J., & Nguyen, Q. (2018). Student profiling in a dispositional learning analytics application using formative assessment. Computers in Human Behavior, 78, 408–420. https://doi.org/10.1016/j.chb.2017.08.010

Turner, G., & Gibbs, G. (2010). Are assessment environments gendered? An analysis of the learning responses of male and female students to different assessment environments. Assessment & Evaluation in Higher Education, 35(6), 687–698. https://doi.org/10.1080/02602930902977723

Winne, P. H., & Hadwin, A. F. (1998). Studying as self-regulated learning. In D. J. Hacker, J. Dunlosky, & A. C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 277–304). Mahwah, NJ, USA: Lawrence Erlbaum Associates Publishers. Retrieved from https://psycnet.apa.org/record/1998-07283-011

Winstone, N. E. (2019). Facilitating students’ use of feedback: Capturing and tracking impact using digital tools. In M. Henderson, R. Ajjawi, D. Boud, & E. Molloy (Eds.), The impact of feedback in higher education: Improving assessment outcomes for learners (pp. 225–242). Cham, Switzerland: Springer International Publishing. https://doi.org/10.1007/978-3-030-25112-313

Winstone, N. E., Mathlin, G., & Nash, R. A. (2019). Building feedback literacy: Students’ perceptions of the developing engagement with feedback toolkit. Frontiers in Education, 4, 1–11. https://doi.org/10.3389/feduc.2019.00039

Winstone, N. E., & Nash, R. (2016). The “Developing Engagement with Feedback Toolkit (DEFT)”: Integrating Assessment literacy into course design (Tech. Rep.). York, UK: Higher Education Academy. Retrieved from https://www.heacademy.ac.uk/system/files/resources/thedevelopingengagementwithfeedbacktoolkitdeft0.pdf

Winstone, N. E., Nash, R. A., Parker, M., & Rowntree, J. (2017). Supporting learners’ agentic engagement with feedback: A systematic review and a taxonomy of recipience processes. Educational Psychologist, 52(1), 17–37. https://doi.org/10.1080/00461520.2016.1207538

Winstone, N. E., Nash, R. A., Rowntree, J., & Parker, M. (2017). ‘It’d be useful, but I wouldn’t use it’: Barriers to university students’ feedback seeking and recipience. Studies in Higher Education, 42(11), 2026–2041. https://doi.org/10.1080/03075079.2015.1130032

Wisniewski, B., Zierer, K., & Hattie, J. (2020). The power of feedback revisited: A meta-analysis of educational feedback research. Frontiers in Psychology, 10, 3087. https://doi.org/10.3389/fpsyg.2019.03087

Wong, B. T. M., & Li, K. C. (2018). Learning analytics intervention: A review of case studies. In 2018 International Symposium on Educational Technology (ISET 2018), 31 July–2 August 2018, Osaka, Japan (pp. 178–182). New York: IEEE. https://doi.org/10.1109/ISET.2018.00047

Zhang, X. A., Kumar, V., & Cosguner, K. (2017). Dynamically managing a profitable email marketing program. Journal of Marketing Research, 54(6), 851–866. https://doi.org/10.1509/jmr.16.0210

Downloads

Published

2021-11-05

How to Cite

Iraj, H., Fudge, A., Khan, H., Faulkner, M., Pardo, A., & Kovanović, V. (2021). Narrowing the Feedback Gap: Examining Student Engagement with Personalized and Actionable Feedback Messages. Journal of Learning Analytics, 1-16. https://doi.org/10.18608/jla.2021.7184

Issue

Section

Extended Conference Papers

Most read articles by the same author(s)