More Than Figures on Your Laptop: (Dis)trustful Implementation of Learning Analytics

Authors

DOI:

https://doi.org/10.18608/jla.2021.7379

Keywords:

learning analytics, trust, institutional adoption, human-computer interaction, datafication

Abstract

The adoption of learning analytics (LA) in complex educational systems is woven into sociocultural and technical challenges that have induced distrust in data and difficulties in scaling LA. This paper presents a study that investigated areas of distrust and threats to trustworthy LA through a series of consultations with teaching staff and students at a large UK university. Surveys and focus groups were conducted to explore participant expectations of LA. The observed distrust is broadly attributed to three areas: the subjective nature of numbers, the fear of power diminution, and approaches to design and implementation of LA. The paper highlights areas to maintain existing trust with policy procedures and areas to cultivate trust by engaging with tensions arising from the social process of LA.

References

Arnold, K. E., Lonn, S., & Pistilli, M. D. (2014). An exercise in institutional reflection: The learning analytics readiness instrument (LARI). In Proceedings of the Fourth International Conference on Learning Analytics and Knowledge (LAK 2014), 24–28 March 2014, Indianapolis, IN, USA (pp. 163–167). New York: ACM. https://doi.org/10.1145/2567574.2567621

Baker, R., Xu, D., Park, J., Yu, R., Li, Q., Cung, B., . . . Smyth, P. (2020). The benefits and caveats of using clickstream data to understand student self-regulatory behaviors: Opening the black box of learning processes. International Journal of Educational Technology in Higher Education, 17, 1–24. https://doi.org/10.1186/s41239-020-00187-1

Bakharia, A., Corrin, L., De Barba, P., Kennedy, G., Gašević, D., Mulder, R., . . . Lockyer, L. (2016). A conceptual framework linking learning design with learning analytics. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 329–338). New York: ACM. https://doi.org/10.1145/2883851.2883944

Beer, D. (2018). The Data Gaze: Capitalism, Power and Perception. Sage.

Brown, M. (2020). Seeing students at scale: How faculty in large lecture courses act upon learning analytics dashboard data. Teaching in Higher Education, 25(4), 384–400. https://doi.org/10.1080/13562517.2019.1698540

Buckingham Shum, S., Ferguson, R., & Martinez-Maldonado, R. (2019). Human-centred learning analytics. Journal of Learning Analytics, 6(2), 1–9. https://doi.org/10.18608/jla.2019.62.1

Castañeda, L., & Selwyn, N. (2018). More than tools? Making sense of the ongoing digitizations of higher education. International Journal of Educational Technology in Higher Education, 15, 22. https://doi.org/10.1186/s41239-018-0109-y

Chen, B., & Zhu, H. (2019). Towards value-sensitive learning analytics design. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, AZ, USA (pp. 343–352). New York: ACM. https://doi.org/10.1145/3303772.3303798

Cohen, J. (1992). A power primer. Psychological Bulletin, 112(1), 155. https://doi.org/10.1037/0033-2909.112.1.155

Corrin, L., Kennedy, G., & Mulder, R. (2013). Enhancing learning analytics by understanding the needs of teachers. In Proceedings of the Australian Society for Computers in Learning in Tertiary Education 30th Annual Conference (ASCILITE 2013), 1–4 December 2013, Sydney, Australia (pp. 201–205). Retrieved from https://ascilite.org/conferences/sydney13/program/papers/Corrin.pdf

Davis, F. D., Bagozzi, R. P., &Warshaw, P. R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management science, 35(8), 982–1003. Retrieved from https://www.jstor.org/stable/2632151

Dawson, S., Joksimovic, S., Poquet, O., & Siemens, G. (2019). Increasing the impact of learning analytics. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, AZ, USA (pp. 446–455). New York: ACM. https://doi.org/10.1145/3303772.3303784

Dawson, S., Poquet, O., Colvin, C., Rogers, T., Pardo, A., & Gašević, D. (2018). Rethinking learning analytics adoption through complexity leadership theory. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 5–9 March 2018, Sydney, Australia (pp. 236–244). New York: ACM. https://doi.org/10.1145/3170358.3170375

Dollinger, M., Liu, D., Arthars, N., & Lodge, J. (2019). Working together in learning analytics towards the co-creation of value. Journal of Learning Analytics, 6(2), 10–26. https://doi.org/10.18608/jla.2019.62.2

Dollinger, M., & Lodge, J. M. (2018). Co-creation strategies for learning analytics. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 5–9 March 2018, Sydney, Australia (pp. 97–101). New York: ACM. https://doi.org/10.1145/3170358.3170372

Drachsler, H., & Greller, W. (2016). Privacy and analytics: It’s a DELICATE issue. a checklist for trusted learning analytics. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 89–98). New York: ACM. https://doi.org/10.1145/2883851.2883893

Ferguson, R. (2012). Learning analytics: Drivers, developments and challenges. International Journal of Technology Enhanced Learning, 4(5/6), 304–317. https://doi.org/10.1504/IJTEL.2012.051816

Ferguson, R., Clow, D., Griffiths, D., & Brasher, A. (2019). Moving forward with learning analytics: Expert views. Journal of Learning Analytics, 6(3), 43–59. https://doi.org/10.18608/jla.2019.63.8

Ferguson, R., Hoel, T., Scheffel, M., & Drachsler, H. (2016). Guest editorial: Ethics and privacy in learning analytics. Journal of Learning Analytics, 3(1), 5–15. https://doi.org/10.18608/jla.2016.31.2

Gašević, D., Dawson, S., & Siemens, G. (2015). Let’s not forget: Learning analytics are about learning. TechTrends, 59(1), 64–71. https://doi.org/10.1007/s11528-014-0822-x

Gašević, D., Kovanović, V., & Joksimović, S. (2017). Piecing the learning analytics puzzle: A consolidated model of a field of research and practice. Learning: Research and Practice, 3(1), 63–78. https://doi.org/10.1080/23735082.2017.1286142

Gašević, D., Tsai, Y.-S., Dawson, S., & Pardo, A. (2019). How do we start? An approach to learning analytics adoption in higher education. The International Journal of Information and Learning Technology, 36(4), 342–353. https://doi.org/10.1108/IJILT-02-2019-0024

Goodyear, P. (2015). Teaching as design. HERDSA Review of Higher Education, 2(2), 27–50.

Gray, G., Schalk, A., Rooney, P., & Lang, C. (2021). A stakeholder informed professional development framework to support engagement with learning analytics. In Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK 2021), 12–16 April 2021, online (pp. 237–247). New York: ACM. https://doi.org/10.1145/3448139.3448162

Grbich, C. (2012). Qualitative Data Analysis: An Introduction. Sage.

Hakami, E., & Hernández-Leo, D. (2020). How are learning analytics considering the societal values of fairness, accountability, transparency and human well-being? A literature review. In LASI-SPAIN 2020: Learning Analytics Summer Institute (Spain 2020: Learning Analytics. Time for Adoption?), 15–16 June 2020, Valladolid, Spain (pp. 121–141). Aachen, Germany: CEUR. Retrieved from http://ceur-ws.org/Vol-2671/paper12.pdf

Hattie, J., & Timperley, H. (2007). The power of feedback. Review of Educational Research, 77(1), 81–112. https://doi.org/10.3102/003465430298487

Herodotou, C., Rienties, B., Boroowa, A., Zdrahal, Z., & Hlosta, M. (2019). A large-scale implementation of predictive learning analytics in higher education: The teachers’ role and perspective. Educational Technology Research and Development, 67(5), 1273–1306. https://doi.org/10.1007/s11423-019-09685-0

Holstein, K., McLaren, B. M., & Aleven, V. (2019). Co-designing a real-time classroom orchestration tool to support teacher–AI complementarity. Journal of Learning Analytics, 6(2), 27–52. https://doi.org/10.18608/jla.2019.62.3

Howell, J. A., Roberts, L. D., Seaman, K., & Gibson, D. C. (2018). Are we on our way to becoming a “helicopter university”? Academics’ views on learning analytics. Technology, Knowledge and Learning, 23(1), 1–20. https://doi.org/10.1007/s10758-017-9329-9

Ifenthaler, D., & Schumacher, C. (2016). Student perceptions of privacy principles for learning analytics. Educational Technology Research and Development, 64(5), 923–938. https://doi.org/10.1007/s11423-016-9477-y

Jones, K. M., Asher, A., Goben, A., Perry, M. R., Salo, D., Briney, K. A., & Robertshaw, M. B. (2020). “We’re being tracked at all times”: Student perspectives of their privacy in relation to learning analytics in higher education. Journal of the Association for Information Science and Technology. https://doi.org/10.1002/asi.24358

Jones, K. M., Rubel, A., & LeClere, E. (2019). A matter of trust: Higher education institutions as information fiduciaries in an age of educational data mining and learning analytics. Journal of the Association for Information Science andTechnology, 71(10), 1227–1241. https://doi.org/10.1002/asi.24327

Jovanovic, J., Gasevic, D., Brooks, C., Devedzic, V., Hatala, M., Eap, T., & Richards, G. (2007). Using semantic web technologies to analyze learning content. IEEE Internet Computing, 11(5), 45–53. https://doi.ieeecomputersociety.org/10.1109/MIC.2007.116

Klein, C., Lester, J., Rangwala, H., & Johri, A. (2019). Learning analytics tools in higher education: Adoption at the intersection of institutional commitment and individual action. The Review of Higher Education, 42(2), 565–593. https://doi.org/10.1353/rhe.2019.0007

Knox, J. (2017). Data power in education: Exploring critical awareness with the “Learning Analytics Report Card”. Television & New Media, 18(8), 734–752. https://doi.org/10.1177/1527476417690029

Kwet, M., & Prinsloo, P. (2020). The “smart” classroom: A new frontier in the age of the smart university. Teaching in Higher Education, 1–17. https://doi.org/10.1080/13562517.2020.1734922

Lang, C., Macfadyen, L. P., Slade, S., Prinsloo, P., & Sclater, N. (2018). The complexities of developing a personal code of ethics for learning analytics practitioners: Implications for institutions and the field. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 5–9 March 2018, Sydney, Australia (pp. 436–440). New York: ACM. https://doi.org/10.1145/3170358.3170396

Leitner, P., Khalil, M., & Ebner, M. (2017). Learning analytics in higher education—A literature review. In A. Peña-Ayala (Ed.), Learning Analytics: Fundaments, Applications, and Trends (pp. 1–24). Springer. https://doi.org/10.1007/978-3-319-52977-6_1

Lodge, J. M., & Corrin, L. (2017). What data and analytics can and do say about effective learning. npj Science of Learning, 2, 5. https://doi.org/10.1038/s41539-017-0006-5

Lodge, J. M., & Lewis, M. J. (2012). Pigeon pecks and mouse clicks: Putting the learning back into learning analytics. In M. Brown, M. Hartnett, & T. Stewart (Eds.), Proceedings of the 29th Annual Conference of the Australian Society for Computers in Learning in Tertiary Education (ASCILITE 2012), 25–28 November 2012, Wellington, New Zealand (pp. 560–564). Australian Society for Computers in Learning in Tertiary Education. Retrieved from https://www.ascilite.org/conferences/Wellington12/2012/images/custom/lodge,_jason_-_pigeon_pecks.pdf

Long, P., Siemens, G., Conole, G., & Gašević, D. (2011). Proceedings of the International Conference on Learning Analytics and Knowledge. In Proceedings of the First International Conference on Learning Analytics and Knowledge (LAK 2011), 27 February–1 March 2011, Banff, AB, Canada. New York: ACM.

Macfadyen, L. P., Dawson, S., Pardo, A., & Gašević, D. (2014). Embracing big data in complex educational systems: The learning analytics imperative and the policy challenge. Research & Practice in Assessment, 9(Winter), 17–28.

Matcha, W., Gašević, D., Uzir, N. A., Jovanović, J., & Pardo, A. (2019). Analytics of learning strategies: Associations with academic performance and feedback. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, AZ, USA (pp. 461–470). New York: ACM. https://doi.org/10.1145/3303772.3303787

Matcha, W., Uzir, N. A., Gašević, D., & Pardo, A. (2020). A systematic review of empirical studies on learning analytics dashboards: A self-regulated learning perspective. IEEE Transactions on Learning Technologies, 13(2), 226–245. https://doi.org/10.1109/TLT.2019.2916802

Norris, C., & Lyon, D. (2003). Surveillance as Social Sorting: Privacy, Risk, and Digital Discrimination. Routledge.

Oster, M., Lonn, S., Pistilli, M. D., & Brown, M. G. (2016). The learning analytics readiness instrument. In Proceedings of the Sixth International Conference on Learning Analytics and Knowledge (LAK 2016), 25–29 April 2016, Edinburgh, UK (pp. 173–182). New York: ACM. https://doi.org/10.1145/2883851.2883925

Pardo, A., Jovanovic, J., Dawson, S., Gašević, D., & Mirriahi, N. (2019). Using learning analytics to scale the provision of personalised feedback. British Journal of Educational Technology, 50(1), 128–138. https://doi.org/10.1111/bjet.12592

Pardo, A., & Siemens, G. (2014). Ethical and privacy principles for learning analytics. British Journal of Educational Technology, 45(3), 438–450. https://doi.org/10.1111/bjet.12152

Prieto, L. P., Rodríguez-Triana, M. J., Martínez-Maldonado, R., Dimitriadis, Y., & Gašević, D. (2019). Orchestrating learning analytics (OrLA): Supporting inter-stakeholder communication about adoption of learning analytics at the classroom level. Australasian Journal of Educational Technology, 35(4). https://doi.org/10.14742/ajet.4314

Prieto-Alvarez, C. G., Martinez-Maldonado, R., & Anderson, T. (2018). Co-designing learning analytics tools with learners. In J. M. Lodge, J. C. Horvath, & L. Corrin (Eds.), Learning Analytics in the Classroom: Translating Learning Analytics Research for Teachers. Abingdon-on-Thames, UK: Routledge. https://doi.org/10.4324/9781351113038

Prinsloo, P., & Slade, S. (2015). Student privacy self-management: Implications for learning analytics. In Proceedings of the Fifth International Conference on Learning Analytics and Knowledge (LAK 2015), 16–20 March 2015, Poughkeepsie, NY, USA (pp. 83–92). New York: ACM. https://doi.org/10.1145/2723576.2723585

Prinsloo, P., & Slade, S. (2016). Student vulnerability, agency, and learning analytics: An exploration. Journal of Learning Analytics, 3(1), 159–182. https://doi.org/10.18608/jla.2016.31.10

Prinsloo, P., Slade, S., & Khalil, M. (2018). Stuck in the middle? Making sense of the impact of micro, meso and macro institutional, structural and organisational factors on implementing learning analytics. In A. Volungeviciene & A. Szucs (Eds.), Proceedings of the 27th Annual Conference of the European Distance and E-learning Network (EDEN 2018), 17–20 June 2018, Genoa, Italy (pp. 326–334). European Distance and E-learning Network. Retrieved from http://oro.open.ac.uk/55598/

Roberts, L. D., Howell, J. A., Seaman, K., & Gibson, D. C. (2016). Student attitudes toward learning analytics in higher education: “The fitbit version of the learning world”. Frontiers in Psychology, 7, 1959. https://doi.org/10.3389/fpsyg.2016.01959

Rubel, A., & Jones, K. M. (2016). Student privacy in learning analytics: An information ethics perspective. The Information Society, 32(2), 143–159. https://doi.org/10.1080/01972243.2016.1130502

Schumacher, C., & Ifenthaler, D. (2018). Features students really expect from learning analytics. Computers in Human Behavior, 78, 397–407. https://doi.org/10.1016/j.chb.2017.06.030

Sclater, N., & Bailey, P. (2015). Code of Practice for Learning Analytics. Jisc. Retrieved from https://www.jisc.ac.uk/guides/code-of-practice-for-learning-analytics

Selwyn, N. (2015). Data entry: Towards the critical study of digital data and education. Learning, Media and Technology, 40(1), 64–82. https://doi.org/10.1080/17439884.2014.921628

Selwyn, N. (2019). What’s the problem with learning analytics? Journal of Learning Analytics, 6(3), 11–19. https://doi.org/10.18608/jla.2019.63.3

Selwyn, N. (2020). Re-imagining “learning analytics”. . . A case for starting again? The Internet and Higher Education, 46, 100745. https://doi.org/10.1016/j.iheduc.2020.100745

Selwyn, N., & Gašević, D. (2020). The datafication of higher education: Discussing the promises and problems. Teaching in Higher Education, 25(4), 527–540. https://doi.org/10.1080/13562517.2019.1689388

Slade, S., & Prinsloo, P. (2013). Learning analytics: Ethical issues and dilemmas. American Behavioral Scientist, 57(10), 1510–1529. https://doi.org/10.1177/0002764213479366

Slade, S., Prinsloo, P., & Khalil, M. (2019). Learning analytics at the intersections of student trust, disclosure and benefit. In Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, AZ, USA (pp. 235–244). New York: ACM. https://doi.org/10.1145/3303772.3303796

The Open University. (2014). Policy on Ethical Use of Student Data for Learning Analytics. The Open University. Retrieved from https://help.open.ac.uk/documents/policies/ethical-use-of-student-data

Thompson, K., Alhadad, S. S., Buckingham Shum, S., Howard, S., Knight, S., Martinez-Maldonado, R., & Pardo, A. (2018). Connecting expert knowledge in the design of classroom learning experiences. In J. M. Lodge, J. C. Horvath, & L. Corrin (Eds.), From Data and Analytics to the Classroom: Translating Learning Analytics for Teachers. Taylor & Francis. Retrieved from http://hdl.handle.net/10453/127709

Tsai, Y.-S., & Gašević, D. (2017). Learning analytics in higher education—Challenges and policies: A review of eight learning analytics policies. In Proceedings of the Seventh International Conference on Learning Analytics and Knowledge (LAK 2017), 13–17 March 2017, Vancouver, BC, Canada (pp. 233–242). New York: ACM. https://doi.org/10.1145/3027385.3027400

Tsai, Y.-S., Moreno-Marcos, P. M., Jivet, I., Scheffel, M., Tammets, K., Kollom, K., & Gasevi ˇ c, D. (2018). The SHEILA framework: Informing institutional strategies and policy processes of learning analytics. Journal of Learning Analytics, 5(3), 5–20. https://doi.org/10.18608/jla.2018.53.2

Tsai, Y.-S., Perrotta, C., & Gašević, D. (2019). Empowering learners with personalised learning approaches? Agency, equity and transparency in the context of learning analytics. Assessment & Evaluation in Higher Education, 1–14. https://doi.org/10.1080/02602938.2019.1676396

Tsai, Y.-S., Poquet, O., Dawson, S., Pardo, A., & Gašević, D. (2019). Complexity leadership in learning analytics: Drivers, ´challenges, and opportunities. British Journal of Educational Technology, 50(6), 2839–2854. https://doi.org/10.1111/bjet.12846

Tsai, Y.-S., Rates, D., Moreno-Marcos, P. M., Munoz-Merino, P. J., Jivet, I., Scheffel, M., . . . Gašević, D. (2020). Learning analytics in European higher education–Trends and barriers. Computers & Education, 103933. https://doi.org/10.1016/j.compedu.2020.103933

Tsai, Y.-S., Whitelock-Wainwright, A., & Gašević, D. (2020). The privacy paradox and its implications for learning analytics. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, Frankfurt, Germany (pp. 230–239). New York: ACM. https://doi.org/10.1145/3375462.3375536

Uhl-Bien, M., & Arena, M. (2018). Leadership for organizational adaptability: A theoretical synthesis and integrative framework. The Leadership Quarterly, 29(1), 89–104. https://doi.org/10.1016/j.leaqua.2017.12.009

Viberg, O., Hatakka, M., Balter, O., & Mavroudi, A. (2018). The current landscape of learning analytics in higher education. Computers in Human Behavior, 89, 98–110. https://doi.org/10.1016/j.chb.2018.07.027

Vigentini, L., Liu, D. Y., Arthars, N., & Dollinger, M. (2020). Evaluating the scaling of a LA tool through the lens of the SHEILA framework: A comparison of two cases from tinkerers to institutional adoption. The Internet and Higher Education, 45, 100728. https://doi.org/10.1016/j.iheduc.2020.100728

Whitelock-Wainwright, A., Gašević, D., Tejeiro, R., Tsai, Y.-S., & Bennett, K. (2019). The student expectations of learning analytics questionnaire. Journal of Computer Assisted Learning, 35(5), 633–666. https://doi.org/10.1111/jcal.12366

Williamson, B. (2018). The hidden architecture of higher education: Building a big data infrastructure for the “smarter university”. International Journal of Educational Technology in Higher Education, 15(1), 12. https://doi.org/10.1186/s41239-018-0094-1

Williamson, B., Bayne, S., & Shay, S. (2020). The datafication of teaching in higher education: Critical issues and perspectives. Teaching in Higher Education, 25(4), 351–365. https://doi.org/10.1080/13562517.2020.1748811

Wintrup, J. (2017). Higher education’s panopticon? Learning analytics, ethics and student engagement. Higher Education Policy, 30(1), 87–103. https://doi.org/10.1057/s41307-016-0030-8

Zuboff, S. (2015). Big other: Surveillance capitalism and the prospects of an information civilization. Journal of Information Technology, 30(1), 75–89. https://doi.org/10.1057/jit.2015.5

Downloads

Published

2021-10-13

How to Cite

Tsai, Y.-S., Whitelock-Wainwright, A., & Gašević, D. (2021). More Than Figures on Your Laptop: (Dis)trustful Implementation of Learning Analytics. Journal of Learning Analytics, 8(3), 81-100. https://doi.org/10.18608/jla.2021.7379

Issue

Section

Extended Conference Papers