Are We Living In LA (P)LA Land?
Reporting on the Practice of 30 STEM Tutors in their Use of a Learning Analytics Implementation at the Open University
Keywords:tutor practice, learning analytics implementation, social informatics, shadow practice
Most higher education institutions view their increasing use of learning analytics as having significant potential to improve student academic achievement, retention outcomes, and learning and teaching practice but the realization of this potential remains stubbornly elusive. While there is an abundance of published research on the creation of visualizations, dashboards, and predictive models, there has been little work done to explore the impact of learning analytics on the actual practice of teachers. Through the lens of social informatics (an approach that views the users of technologies as active social actors whose technological practices constitute a wider socio-technical system) this qualitative study reports on an investigation into the practice of 30 tutors in the STEM faculty at Europe’s largest distance learning organization, The Open University UK (OU). When asked to incorporate learning analytics (including predictive learning analytics) contained in the Early Alert Indicator (EAI) dashboard during the 2017–2018 academic year into their practice, we found that tutors interacted with this dashboard in certain unanticipated ways and developed three identifiable “shadow practices”.
Agnihotri, L., & Ott, A. (2014). Building a student at-risk model: An end-to-end perspective from user to data scientist. In J. Stamper et al. (Eds.), Proceedings of the 7th International Conference on Educational Data Mining (EDM2014), 4–7 July 2014, London, UK (pp. 209–212). International Educational Data Mining Society. Retrieved from http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.660.2885&rep=rep1&type=pdf
Arthars, N., Dollinger, M., Vigentini, L., Liu, Y.-T., Kondo, E., & King, D. (2019). Empowering teachers to personalise learning support. In D. Ifenthaler, J. Yau, & D. Mah (Eds), Utilising learning analytics to support study success. Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-319-64792-0_13
Bartimote, K., Pardo, A., & Reimann, P. (2019). The perspective realism brings to learning analytics in the classroom. In J. M. Lodge, J. C. Horvath & L. Corrin (Eds.), Learning analytics in the classroom: Translating learning analytics research for teachers. Abingdon-on-Thames, UK: Routledge.
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa
Calvert, C. (2014). Developing a model and applications for probabilities of student success: A case study of predictive analytics. Open Learning: The Journal of Open, Distance and e-Learning, 29(2), 160–173. http://doi.org/10.1080/02680513.2014.931805
Castaneda, L., & Selwyn, N. (2018). More than tool? Making sense of the ongoing digitizations of higher education. International Journal of Education Technology in Higher Education, 15(22), 1–10. https://doi.org/10.1186/s41239-018-0109-y
Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, 18(6), 683–695. https://doi.org/10.1080/13562517.2013.827653
Conole, G. (2012). Designing for learning in an open world. Dordrecht, Netherlands: Springer.
Creanor, L., & Walker, S. (2012). Learning technology in context: A case for the sociotechnical interaction framework as an analytical lens for networked learning research. In L. Dirckinck-Holmfeld, V. Hodgson, & D. McConnell (Eds.), Exploring the pedagogy and practice of networked learning (pp. 173–187). London, UK: Springer. https://doi.org/10.1007/978-1-4614-0496-5_10
Dalziel, J., Conole, G., Wills, S., Walker, S., Bennett, S., Dobozy, E., Cameron, L., Badilescu-Buga, E., & Bower, M. (2016). The Larnaca declaration on learning design. Journal of Interactive Media in Education, 1(7), 1–24. https://doi.org/10.5334/jime.407
Daniel, B. (2015). Big data and analytics in higher education: Opportunities and challenges. British Journal of Educational Technology, 46(5), 904–920. https://doi.org/10.1111/bjet.12230
Davis, F. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/249008
Edwards, P. (2018). We have been assimilated: Some principles for thinking about algorithmic systems. In U. Schultze, M. Aanestad, M. Mähring, C. Østerlund, & K. Riemer (Eds.), Living with monsters? Social implications of algorithmic phenomena, hybrid agency, and the performativity of technology. Proceedings of the Working Conference on Information Systems and Organizations (IS&O 2018), 11–12 December 2018, San Francisco, CA, USA (pp. 19–27). IFIP Advances in Information and Communication Technology, vol 543. Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-030-04091-8_3
Ferguson, R., Clow, D., Griffiths, D., & Brasher, A. (2019). Moving forward with learning analytics: Expert views. Journal of Learning Analytics, 6(3), 43–59. http://doi.org/10.18608/jla.2019.63.8
Foster, E. (2015). What have we learnt from implementing learning analytics at NTU? Retrieved from https://www.yumpu.com/en/document/view/54413096/what-have-we-learnt-from-implementing-learning-analytics-at-ntu
Gašević, D., Dawson, S., Rogers, T., & Gašević, D. (2016). Learning analytics should not promote one size fits all: The effects of instructional conditions in predicting learning success. Internet and Higher Education, 28, 68–84. https://doi.org/10.1016/j.iheduc.2015.10.002
Herodotou, C., Hlosta, M., Boroowa, A., Rienties, B., Zdrahal, Z., & Mangafa, C. (2019c). Empowering online teachers through predictive learning analytics. British Journal of Educational Technology, 50(6), 3064–3079. https://doi.org/10.1111/bjet.12853
Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., & Hlosta, M. (2019a). A large-scale implementation of PLA in higher education: The teacher’s role and perspective. Educational Technology Research and Development, 67, 1273–1306. https://doi.org/10.1007/s11423-019-09685-0
Herodotou, C., Rienties, B., Hlosta, M., Boroowa, A., & Mangafa, C. (2020). The scalable implementation of predictive learning analytics at a distance learning university: Insights from a longitudinal case study. The Internet and Higher Education, 45, 1–3. https://doi.org/10.1016/j.iheduc.2020.100725
Herodotou, C., Rienties, B., Verdin, B., & Boroowa, A. (2019b). Predictive Learning Analytics “at scale”: Towards guidelines to successful implementation in higher education based on the case of The Open University UK. Journal of Learning Analytics, 6(1), 85–95. https://doi.org/10.18608/jla.2019.61.5
Hidalgo, R. (2018). Analytics for action: Using data analytics to support students in improving their learning outcomes. In G. Ubachs & L. Konings (Eds.), The envisioning report for empowering universities (2nd ed.), (pp. 6–8). Maastricht: EADTU. Retrieved from https://tinyurl.com/envisioning-report
Hu, Y., Lo, C., & Shih, S. (2014). Developing early warning systems to predict students’ online learning performance. Computers in Human Behaviour, 36, 469–478. https://doi.org/10.1016/j.chb.2014.04.002
Jivet, I., Scheffel, M., Spect, M., & Drachsler, H. (2018). License to evaluate: Preparing learning analytics dashboards for educational practice. Proceedings of the 8th International Conference on Learning Analytics and Knowledge (LAK ’18), 5–9 March 2018, Sydney, NSW, Australia (pp. 31–40). New York: ACM. https://doi.org/10.1145/3170358.3170421
Klein, C., Lester, J., Rangwala, H., & Johri, A. (2019). Learning analytics tools in higher education: Adoption at the Intersection of institutional commitment and individual action. The Review of Higher Education, 42(2), 565–593. Baltimore, MD: John Hopkins University Press. https://doi.org/10.1353/rhe.2019.0007
Kling, R. (2007). What is social informatics and why does it matter? The Information Society, 23(4), 205–220. https://doi.org/10.1080/01972240701441556
Knight, S., & Buckingham Shum, S. (2017). Theory and learning analytics. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), The handbook of learning analytics, (pp. 17–22). Beaumont, AB: Society for Learning Analytics Research (SoLAR). https://doi.org/10.18608/hla17
Kop, R., Fournier, H., & Guillaume, D. (2017). A critical perspective on learning analytics and educational data mining. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), The handbook of learning analytics, (pp. 319–326). Beaumont, AB: Society for Learning Analytics Research (SoLAR). https://doi.org/10.18608/hla17
Kuzilek, J., Hlosta, M., Herrmannova, D., Zdrahal, Z., & Wolff, A. (2015). OU analyse: Analysing at-risk students at The Open University. Learning Analytics Review, LAK15-1, 1–16. Retrieved from https://oro.open.ac.uk/42529/1/__userdata_documents4_ctb44_Desktop_analysing-at-risk-students-at-open-university.pdf
Lamb, R., & Kling, R. (2003). Reconceptualising users as social actors in information systems research. MIS Quarterly, 27, 197–235. https://doi.org/10.2307/30036529
Lodge, J., Horvath, J., & Corrin, L. (2019). Introduction. In J. M. Lodge, J. C. Horvath & L. Corrin (Eds.), Learning analytics in the classroom: Translating learning analytics research for teachers. Abingdon-on-Thames, UK: Routledge.
Long, P., & Siemens, G. (2011). Penetrating the fog: Analytics in learning and education. Educause Review, September/October, 31–40. Retrieved from https://er.educause.edu/~/media/files/article-downloads/erm1151.pdf
Maguire, M., & Delahunt, B. (2017). Doing a thematic analysis: A practical, step-by-step guide for learning and teaching scholars. All Ireland Journal of Higher Education 9(3), 33501–33514. Retrieved from https://ojs.aishe.org/index.php/aishe-j/article/view/335
McCoy, C., & Rosenbaum, H. (2019). Uncovering unintended and shadow practices of users of decision support systems decision support in higher education institutions, Journal of the American Association for Information Science and Technology, 70(4), 370–384. https://doi.org/10.1002/asi.24131
Nguyen, Q., Rienties, B., Toetenel, L., Ferguson, F., & Whitelock, D. (2017). Examining the designs of computer-based assessment and its impact on student engagement, satisfaction, and pass rates. Computers in Human Behaviour, 76, 703–714. https://doi.org/10.1016/j.chb.2017.03.028
Olney, T., Rienties, B., & Toetenel, L. (2019). Gathering, visualising and interpreting learning design analytics to inform classroom practice and curriculum design: Learning analytics in the classroom. In J. M. Lodge, J. C. Horvath & L. Corrin (Eds.), Learning analytics in the classroom: Translating learning analytics research for teachers, (pp. 71–92). Abingdon-on-Thames, UK: Routledge.
Orlikowski, W., & Gash, D. (1994). Technological frames: Making sense of information technology in organizations. ACM Transactions on Information Systems, 12(2) 174–207. https://doi.org/10.1145/196734.196745
Piderit, S. (2000). Rethinking resistance and recognizing ambivalence: A multidimensional view of attitudes toward an organizational change. The Academy of Management Review, 25(4), 783–794. https://doi.org/10.2307/259206
Rienties, B., Boroowa, A., Cross, S., Kubiak, C., Mayles, K., & Murphy, S. (2016). Analytics4Action evaluation framework: A review of evidence-based learning analytics interventions at Open University UK. Journal of Interactive Media in Education, 1(2), 1–12. https://doi.org/10.5334/jime.394
Rienties, B., Herodotou, C., Olney, T., Schencks, M., & Boroowa, A. (2018). Making sense of learning analytics dashboards: A technology acceptance perspective of 95 teachers. The International Review of Research in Open and Distributed Learning, 19(5). https://doi.org/10.19173/irrodl.v19i5.3493
Sawyer, S., & Hartswood, M. (2014). Advancing social informatics. In P. Fichman & H. Rosenbaum (Eds.), Social informatics: Past, present and future (pp. 197–209). Newcastle upon Tyne, UK: Cambridge Scholars Publishing.
Schumacher, C., & Ifenthaler, D. (2018). Features students really expect from learning analytics. Computers in Human Behaviour, 78, 397–407. https://doi.org/10.1016/j.chb.2017.06.030
Schwendimann, B., Rodríguez-Triana, M. J., Vozniuk, A., Prieto, L. P., Shirvani Boroujeni, M., Holzer, A., Gillet, D., & Dillenbourg, P. (2017). Perceiving learning at a glance: A systematic literature review of learning dashboard research. IEEE Transactions on Learning Technologies, 10(1), 30–41. https://doi.org/10.1109/TLT.2016.2599522
Sclater, N., Peasgood, A., & Mullan, J. (2016, April 19). Learning analytics in higher education: A review of UK and international practice. Jisc. Retrieved from https://analytics.jiscinvolve.org/wp/2016/04/19/learning-analytics-in-higher-education-a-review-of-uk-and-international-practice/
Selwyn, N. (2019). What’s the problem with learning analytics? Journal of Learning Analytics, 6(3), 11–19. https://doi.org/10.18608/jla.2019.63.3
Shacklock, X. (2016). From bricks to clicks: The potential of data and analytics in higher education. Policy Connect. Higher Education Commission. Retrieved from https://www.policyconnect.org.uk/hec/research/report-bricks-clicks-potential-data-and-analytics-higher-education
SHEILA. (2018). Supporting higher education to integrate learning analytics framework. Retrieved from https://sheilaproject.eu/wp-content/uploads/2018/08/SHEILA-framework_Version-2.pdf
Simpson, O. (2006). Predicting student success in open and distance learning. Open Learning: The Journal of Open, Distance and e-Learning, 21(2), 125–138. https://doi.org/10.1080/02680510600713110
Slade, S. & Boroowa, A. (2014). Policy on ethical use of student data for learning analytics. Milton Keynes, UK: The Open University. Retrieved from https://help.open.ac.uk/documents/policies/ethical-use-of-student-data
Suchman, L. (1996). Supporting articulation work. In R. Kling (Ed.), Computerization and controversy: Value conflicts and social choices (2nd ed), (pp. 407–423). San Francisco, CA: Morgan Kaufmann.
Tanes, Z., Arnold, K., King, A., & Remnet, M. (2011). Using signals for appropriate feedback: Perceptions and practices. Computers & Education, 57(4), 2414–2422. https://doi.org/10.1016/j.compedu.2011.05.016
TEF. (2019). Teaching excellence and student outcomes framework. Retrieved from https://www.officeforstudents.org.uk/advice-and-guidance/teaching/what-is-the-tef/
Tsai, Y.-S., Moreno-Marcos, P., Jivet, I., Scheffel, M., Tammets, K., Kollom, K., & Gašević, D. (2018). The SHEILA framework: Informing institutional strategies and policy processes of learning analytics. Journal of Learning Analytics, 5(3), 5–20. https://doi.org/10.18608/jla.2018.53.2
van Harmelen, M., & Workman, D. (2012). Analytics for learning and teaching. Technical Report in CETIS Analytics Series, vol. 1, no. 3. Bolton, UK: CETIS.
Walker, S., & Creanor, L. (2009). The STIN in the tale: A sociotechnical interaction perspective on networked learning. Journal of Educational Technology and Society, 12(4), 305–316. Retrieved from https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.303.6882&rep=rep1&type=pdf
Walker, S., Olney, T., Wood, C., Clarke, A., & Dunworth, M. (2019). How do tutors use data to support their students? Open Learning: The Journal of Open, Distance and e-Learning, 34, 118–133. https://doi.org/10.1080/02680513.2018.1554476
White, S., & White, S. (2016). Learning designers in the “third space”: The socio-technical construction of MOOCs and their relationship to educator and learning designer roles in HE. Journal of Interactive Media in Education, 2016(1) 1–12. https://doi.org/10.5334/jime.429
Williamson, B. (2016). Digital education governance: Data visualization, predictive analytics, and real-time policy instruments. Journal of Education Policy, 31(2), 123–141. https://doi.org/10.1080/02680939.2015.1035758
Wise, A., & Vytasek, J. (2017). Learning analytics implementation design. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), The handbook of learning analytics, (pp. 151–160). Beaumont, AB: Society for Learning Analytics Research (SoLAR). https://doi.org/10.18608/hla17
How to Cite
Copyright (c) 2021 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) license that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).