FoLA2 — A Method for Co-creating Learning Analytics–Supported Learning Design

Authors

  • Marcel Schmitz Zuyd University of Applied Sciences https://orcid.org/0000-0002-1816-6178
  • Maren Scheffel Ruhr University Bochum
  • Roger Bemelmans Zuyd University of Applied Sciences
  • Hendrik Drachsler DIPF Leibniz Institute for Research and Information in Education, Goethe University, Open University of the Netherlands https://orcid.org/0000-0001-8407-5314

DOI:

https://doi.org/10.18608/jla.2022.7643

Keywords:

learning design, learning analytics, learning activity , multidisciplinary cooperative method, learning enhancing technology, data and tool report

Abstract

Learning activities are at the core of every educational design effort. Designing learning activities is a process that benefits from reflecting on previous runs of those activities. One way to measure the behaviour and effects of design choices is to use learning analytics (LA). The challenge, however, lies in the unavailability of an easy-to-use, LA-supported learning design (LD) method. We established such a method—the Fellowship of Learning Activities and Analytics (FoLA2)—reinforced by a gameboard and cards, to provide structure and inspiration. The method enables several participants with different roles to interact with a set of card decks to collaboratively create an LA-supported LD. Using this method helps to design learning activities in a collaborative, practical way; it also raises awareness about the benefits of multidisciplinary co-design and connections between LA and LD. FoLA2 can be used to develop, capture, and systematize design elements and to systematically incorporate LA.

References

Alvarez, C. P., Martinez-Maldonado, R., & Buckingham Shum, S. (2020). LA-DECK: A card-based learning analytics co-design tool. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, online (pp. 63–72). ACM. https://doi.org/10.1145/3375462.3375476

Angelides, P. (2001). The development of an efficient technique for collecting and analyzing qualitative data: The analysis of critical incidents. International Journal of Qualitative Studies in Education, 14(3), 429–442. https://doi.org/10.1080/09518390110029058

Baldiris, S. M., Fabregat, R., & De la Hoz Manotas, A. (2010). A set of software tools to build an author assessment package on Moodle: Implementing the AEEA proposal. 2010 10th IEEE International Conference on Advanced Learning Technologies (ICALT 2010), 5–7 July 2010, Sousse, Tunisia (pp. 67–69). IEEE. https://doi.org/10.1109/ICALT.2010.26

Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human–Computer Interaction, 24(6), 574–594. https://doi.org/10.1080/10447310802205776

Beetham, H. (2007). An approach to learning activity design. In H. Beetham & R. Sharpe (Eds.), Rethinking pedagogy for a digital age: Designing and delivering e-learning (pp. 26–40). Taylor & Francis. https://www.taylorfrancis.com/chapters/edit/10.4324/9780203961681-12/approach-learning-activity-design-helen-beetham

Belbin, R. M. (2011). Management teams: Why they succeed or fail. Emerald Group Publishing Limited. https://doi.org/10.4324/9780080963594

Bellotti, F., Berta, R., & De Gloria, A. (2010). Designing effective serious games: Opportunities and challenges for research. International Journal of Emerging Technologies in Learning (iJET), 5(SI3), 22–35. https://doi.org/10.3991/ijet.v5s3.1500

Boschman, F., McKenney, S., & Voogt, J. (2015). Exploring teachers’ use of TPACK in design talk: The collaborative design of technology-rich early literacy activities. Computers & Education, 82, 250–262. https://doi.org/10.1016/j.compedu.2014.11.010

Brooke, J. (2013). SUS: A retrospective. Journal of Usability Studies, 8(2), 29–40. https://uxpajournal.org/sus-a-retrospective/ Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42. https://doi.org/10.3102/0013189X018001032

Celik, D., & Magoulas, G. D. (2016). A review, timeline, and categorization of learning design tools. In D. K. Chiu, I. Marenzi, U. Nanni, M. Spaniol, & M. Temperini (Eds.), Proceedings of the 15th International Conference on Web-Based Learning (ICWL 2016), 26–29 October 2016, Rome, Italy (pp. 3–13). Springer. https://doi.org/10.1007/978-3-319-47440-3_1

Colvin, C., Rogers, T., Wade, A., Dawson, S., Gasevic, D., Buckingham Shum, S., Nelson, K., Alexander, S., Lockyer, L., & Kennedy, G. (2016). Student Retention and Learning Analytics: A Snapshot of Australian Practices and a Framework for Advancement (tech. rep.). Australian Government Office for Learning and Teaching. https://www.research.ed.ac.uk/en/publications/student-retention-and-learning-analytics-a-snapshot-of-australian

Dalziel, J. (2003). Implementing learning design: The learning activity management system (LAMS) [Workshop presented at ASCILITE 2003, 7–10 December 2003, Adelaide, Australia].

Dawson, S., Joksimovic, S., Poquet, O., & Siemens, G. (2019). Increasing the impact of learning analytics. Proceedings of the Ninth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, AZ, USA (pp. 446–455). ACM. https://doi.org/10.1145/3303772.3303784

Deng, Y., Antle, A. N., & Neustaedter, C. (2014). Tango cards: A card-based design tool for informing the design of tangible learning games. Proceedings of the 2014 Conference on Designing Interactive Systems (DIS 2014), 21–25 June 2014, Vancouver, BC, Canada (pp. 695–704). https://doi.org/10.1145/2598510.2598601

Design School Kolding. (2017). DSKD method cards. https://www.designskolenkolding.dk/en/publications/sustainable-fur-cards

Dollinger, M., & Lodge, J. M. (2018). Co-creation strategies for learning analytics. Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 5–9 March 2018, Sydney, Australia (pp. 97–101). ACM. https://doi.org/10.1145/3170358.3170372

Drachsler, H., & Goldhammer, F. (2020). Learning analytics and eAssessment—Towards computational psychometrics by combining psychometrics with learning analytics. In D. Burgos (Ed.), Radical solutions and learning analytics (pp. 67–80). Springer. https://doi.org/10.1007/978-981-15-4526-9_5

Eno, B., & Schmidt, P. (1975). Oblique strategies. Opal. (Limited edition, boxed set of cards.) [rMAB].

Ferna ́ndez-Gallego, B., Lama, M., Vidal, J. C., & Mucientes, M. (2013). Learning analytics framework for educational virtual worlds. Procedia Computer Science, 25, 443–447. https://doi.org/10.1016/j.procs.2013.11.056

Florian-Gaviria, B., Glahn, C., & Gesa, R. F. (2013). A software suite for efficient use of the European qualifications framework in online and blended courses. IEEE Transactions on Learning Technologies, 6(3), 283–296. https://doi.org/10.1109/TLT.2013.18

Fournier, H., Kop, R., & Sitlia, H. (2011). The value of learning analytics to networked learning on a personal learning environment. Proceedings of the First International Conference on Learning Analytics and Knowledge (LAK 2011, 27 February–1 March 2011, Banff, AB, Canada (pp. 104–109). ACM. https://doi.org/10.1145/2090116.2090131

Gasevic, D., Tsai, Y.-S., Dawson, S., & Pardo, A. (2019). How do we start? An approach to learning analytics adoption in higher education. The International Journal of Information and Learning Technology, 6(4), 342–353. https://doi.org/10.1108/IJILT-02-2019-0024

Gasˇevic ́, D., Mirriahi, N., Dawson, S., & Joksimovic ́, S. (2017). Effects of instructional conditions and experience on the adoption of a learning tool. Computers in Human Behavior, 67, 207–220. https://doi.org/10.1016/j.chb.2016.10.026 Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning analytics. Journal of Educational Technology & Society, 15(3), 42–57. https://www.jstor.org/stable/jeductechsoci.15.3.42

Gruber, M. R. (2019). Designing for Great Teaching with Learning Design Cards. Universita ̈t-Zu ̈rich. https://dlf.uzh.ch/2019/04/01/designing-for-great-teaching-with-learning-design-cards

Haythornthwaite, C. (2017). An information policy perspective on learning analytics. Proceedings of the Seventh International Conference on Learning Analytics and Knowledge (LAK 2017), 13–17 March 2017, Vancouver, BC, Canada (pp. 253– 256). ACM. https://doi.org/10.1145/3027385.3027389

Hevner, A., & Chatterjee, S. (2010). Design science research in information systems. In A. Hevner & S. Chatterjee (Eds.), Design research in information systems (pp. 9–22). Springer. https://doi.org/10.1007/978-1-4419-5653-8_2

Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266. https://doi.org/10.1023/B:EDPR.0000034022.16470.f3

Joksimovic ́, S., Kovanovic ́, V., & Dawson, S. (2019). The journey of learning analytics. HERDSA Review of Higher Education, 6, 27–63. https://www.herdsa.org.au/herdsa-review-higher-education-vol-6/37-63

Katsamani, M., Retalis, S., & Boloudakis, M. (2012). Designing a Moodle course with the CADMOS learning design tool. Educational Media International, 49(4), 317–331. https://doi.org/10.1080/09523987.2012.745771

Kelly, N., Thompson, K., & Yeoman, P. (2015). Theory-led design of instruments and representations in learning analytics: Developing a novel tool for orchestration of online collaborative learning. Journal of Learning Analytics, 2(2), 14–43. https://doi.org/10.18608/jla.2015.22.3

Koehler, M., & Mishra, P. (2009). What is technological pedagogical content knowledge (TPACK)? Contemporary Issues in Technology and Teacher Education, 9(1), 60–70. https://doi.org/10.1177/002205741319300303

Koper, R. (2005). An introduction to learning design. In R. Koper & C. Tattersall (Eds.), Learning design (pp. 3–20). Springer. https://doi.org/10.1007/3-540-27360-3_1

Koper, R. (2006). Current research in learning design. Educational Technology & Society, 9(1), 13–22. https://www.jstor.org/ stable/10.2307/jeductechsoci.9.1.13

Koper, R., & Manderveld, J. (2004). Educational modelling language: Modelling reusable, interoperable, rich and personalised units of learning. British Journal of Educational Technology, 35(5), 537–551. https://doi.org/10.1111/j.0007- 1013.2004.00412.x

Koper, R., & Olivier, B. (2004). Representing the learning design of units of learning. Journal of Educational Technology & Society, 7(3), 97–111. https://www.learntechlib.org/p/75442/

Laurillard, D., Charlton, P., Craft, B., Dimakopoulos, D., Ljubojevic, D., Magoulas, G., Masterman, E., Pujadas, R., Whitley, E. A., & Whittlestone, K. (2013). A constructionist learning environment for teachers to model learning designs. Journal of Computer Assisted Learning, 29(1), 15–30. https://doi.org/10.1111/j.1365-2729.2011.00458.x

Law, N., & Liang, L. (2020). A multilevel framework and method for learning analytics integrated learning design. Journal of Learning Analytics, 7(3), 98–117. https://doi.org/10.18608/jla.2020.73.8

Leitner, P., Khalil, M., & Ebner, M. (2017). Learning analytics in higher education—a literature review. In A. Pen ̃a-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends (pp. 1–23). Springer. https://doi.org/10.1007/978-3-319- 52977-6_1

Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459. https://doi.org/10.1177/0002764213479367

Macfadyen, L. P., Lockyer, L., & Rienties, B. (2020). Learning design and learning analytics: Snapshot 2020. Journal of Learning Analytics, 7(3), 6–12. https://doi.org/10.18608/jla.2020.73.2

Mangaroska, K., & Giannakos, M. N. (2018). Learning analytics for learning design: A systematic literature review of analytics-driven design to enhance learning. IEEE Transactions on Learning Technologies, 12(4), 516–534. https://doi.org/10.1109/TLT.2018.2868673

Mangaroska, K., Sharma, K., Gasevic, D., & Giannakos, M. (2020). Multimodal learning analytics to inform learning design: Lessons learned from computing education. Journal of Learning Analytics, 7(3), 79–97. https://doi.org/10.18608/jla.2020.73.7

Mankins, J. C. (1995). Technology Readiness Levels (white paper). Advanced Concepts Office, Office of Space Access; Technology, NASA. http://www.artemisinnovation.com/images/TRL White Paper 2004-Edited.pdf

McKenney, S., & Mor, Y. (2015). Supporting teachers in data-informed educational design. British Journal of Educational Technology, 46(2), 265–279. https://doi.org/10.1111/bjet.12262

Means, B., & Neisler, J. (2020). Suddenly Online: A National Survey of Undergraduates during the COVID-19 Pandemic (tech. rep.). Digital Promise. https://www.everylearnereverywhere.org/resources/suddenly-online-national-undergraduate- survey/

Michos, K., Lang, C., Herna ́ndez-Leo, D., & Price-Dennis, D. (2020). Involving teachers in learning analytics design: Lessons learned from two case studies. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, online, 94–99. https://doi.org/10.1145/3375462.3375507

Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence-centered design. ETS Research Report Series, 2003(1), i–29. https://doi.org/10.1002/j.2333-8504.2003.tb01908.x

Moller, O. (n.d.). MethodKit. https://methodkit.com/research-method-cards/

Mor, Y., & Mogilevsky, O. (2013). The learning design studio: Collaborative design inquiry as teachers’ professional development. Research in Learning Technology, 21, 22054. https://doi.org/10.3402/rlt.v21i0.22054

Nguyen, Q., Rienties, B., & Whitelock, D. (2020). A mixed-method study of how instructors design for learning in online and distance education. Journal of Learning Analytics, 7(3), 64–78. https://doi.org/10.18608/jla.2020.73.6

Persico, D., & Pozzi, F. (2015). Informing learning design with learning analytics to improve teacher inquiry. British Journal of Educational Technology, 46(2), 230–248. https://doi.org/10.1111/bjet.12207

Peterson, C. (2003). Bringing ADDIE to life: Instructional design at its best. Journal of Educational Multimedia and Hypermedia, 12(3), 227–241. https://www.learntechlib.org/primary/p/2074/

Plattner, H. (2010). An Introduction to Design Thinking Process Guide. The Institute of Design at Stanford. https://web.stanford.edu/~mshanks/MichaelShanks/files/509554.pdf

Rienties, B., Nguyen, Q., Holmes, W., & Reedy, K. (2017). A review of ten years of implementation and research in aligning learning design with learning analytics at the Open University UK. Interaction Design and Architecture(s), 33, 134–154. https://doi.org/10.55612/s-5002-033-007

Rodr ́ıguez-Triana, M. J., Mart ́ınez-Mone ́s, A., Asensio-Pe ́rez, J. I., & Dimitriadis, Y. (2015). Scripting and monitoring meet each other: Aligning learning analytics and learning design to support teachers in orchestrating CSCL situations. British Journal of Educational Technology, 46(2), 330–343. https://doi.org/10.1111/bjet.12198

Roy, R., & Warren, J. P. (2019). Card-based design tools: A review and analysis of 155 card decks for designers and designing. Design Studies, 63, 125–154. https://doi.org/10.1016/j.destud.2019.04.002

Rubenstein-Montano, B., Liebowitz, J., Buchwalter, J., McCaw, D., Newman, B., Rebeck, K., & The Knowledge Management Methodology Team. (2001). A systems thinking framework for knowledge management. Decision Support Systems, 31(1), 5–16. https://doi.org/10.1016/S0167-9236(00)00116-0

Sarmiento, J. P., Campos, F., & Wise, A. (2020). Engaging students as co-designers of learning analytics. Companion Proceedings of the 10th International Learning Analytics and Knowledge Conference (LAK 2020), 23–27 March 2020, online (pp. 29–32). https://www.solaresearch.org/wp-content/uploads/2020/06/LAK20 Companion Proceedings.pdf

Scheffel, M., Drachsler, H., De Kraker, J., Kreijns, K., Slootmaker, A., & Specht, M. (2016). Widget, widget on the wall, am I performing well at all? IEEE Transactions on Learning Technologies, 10(1), 42–52. https://doi.org/10.1109/TLT.2016.2622268

Scheffel, M., Schmitz, M., van Hooijdonk, J., van Limbeek, E., Kockelkoren, C., Joppe, D., & Drachsler, H. (2021). The design cycle for education (DC4E). In A. Kienle, A. Harrer, J. M. Haake, & A. Lingnau (Eds.), DELFI 2021—Komplettband, 13–15 September 2021, online (pp. 229–240). Gesellschaft fu ̈r Informatik eV. https://dl.gi.de/handle/20.500.12116/37055

Schmitz, M., Scheffel, M., Bemelmans, R., & Drachsler, H. (2020). Fellowship of the Learning Activity—Learning Analytics 4 Learning Design Game. https://doi.org/10.25385/zuyd.9884279

Schmitz, M., Scheffel, M., van Limbeek, E., Bemelmans, R., & Drachsler, H. (2018a). “Make it personal!”—Gathering input from stakeholders for a learning analytics-supported learning design tool. In V. Pammer-Schindler, M. Pe ́rez- Sanagust ́ın, H. Drachsler, R. Elferink, & M. Scheffel (Eds.), European Conference on Technology Enhanced Learning (EC-TEL 2018), 3–5 September 2018, Leeds, UK (pp. 297–310). Springer. https://doi.org/10.1007/978-3-319-98572- 5_23

Schmitz, M., Scheffel, M., van Limbeek, E., van Halem, N., Cornelisz, I., van Klaveren, C., Bemelmans, R., & Drachsler, H. (2018b). Investigating the relationships between online activity, learning strategies and grades to create learning analytics-supported learning designs. In V. Pammer-Schindler, M. Pe ́rez-Sanagust ́ın, H. Drachsler, R. Elferink, & M. Scheffel (Eds.), European Conference on Technology Enhanced Learning (EC-TEL 2018), 3–5 September 2018, Leeds, UK (pp. 311–325). Springer. https://doi.org/10.1007/978-3-319-98572-5_24

Schmitz, M., Van Limbeek, E., Greller, W., Sloep, P., & Drachsler, H. (2017). Opportunities and challenges in using learning analytics in learning design. In E ́. Lavoue ́, H. Drachsler, K. Verbert, J. Broisin, & M. Pe ́rez-Sanagust ́ın (Eds.), European Conference on Technology Enhanced Learning (EC-TEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 209–223). Springer. https://doi.org/10.1007/978-3-319-66610-5_16

Shaffer, D. W. (2006). Epistemic frames for epistemic games. Computers & Education, 46(3), 223–234. https://doi.org/10.1016/j.compedu.2005.11.003

Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 30. https://er.educause.edu/articles/2011/9/penetrating-the-fog-analytics-in-learning-and-education

Slade, S., & Prinsloo, P. (2013). Learning analytics: Ethical issues and dilemmas. American Behavioral Scientist, 57(10), 1510–1529. https://doi.org/10.1177/0002764213479366

Sluijsmans, D., Dochy, F., & Moerkerke, G. (1998). Creating a learning environment by using self-, peer-and co-assessment. Learning Environments Research, 1(3), 293–319. https://doi.org/10.1023/A:1009932704458

Spiro, R. J., & Jheng, J. C. (2012). Cognitive flexibility and hypertext: Theory and technology for the nonlinear and multidi- mensional traversal of complex subject matter. In D. Nix & R. J. Spiro (Eds.), Cognition, education, and multimedia (pp. 177–220). Taylor & Francis. https://www.taylorfrancis.com/chapters/edit/10.4324/9780203052174-11/cognitive-flexibility-hypertext-theory-technology-nonlinear-multidimensional-traversal-complex-subject-matter-rand-spiro-jihn-changjehng

Sweller, J. (2011). Cognitive load theory. In J. P. Mestre & B. H. Ross (Eds.), Psychology of learning and motivation (pp. 37–76). Elsevier. https://doi.org/10.1016/B978-0-12-387691-1.00002-8

Tsai, Y.-S., & Gasevic, D. (2017). Learning analytics in higher education—challenges and policies: A review of eight learning analytics policies. Proceedings of the Seventh International Conference on Learning Analytics and Knowledge (LAK 2017), 13–17 March 2017, Vancouver, BC, Canada (pp. 233–242). ACM. https://doi.org/10.1145/3027385.3027400

Tsai, Y.-S., Moreno-Marcos, P. M., Jivet, I., Scheffel, M., Tammets, K., Kollom, K., & Gasˇevic ́, D. (2018). The SHEILA framework: Informing institutional strategies and policy processes of learning analytics. Journal of Learning Analytics, 5(3), 5–20. https://doi.org/10.18608/jla.2018.53.2

Van Merrienboer, J., & Kirschner, P. (2007). Ten steps to complex learning. Lawrence Erlbaum Associates.

Verpoorten, D., Poumay, M., & Leclercq, D. (2007). The eight learning events model: A pedagogic conceptual tool supporting diversification of learning methods. Interactive Learning Environments, 15(2), 151–160. https://doi.org/10.1080/10494820701343694

Vezzoli, Y., Mavrikis, M., & Vasalou, A. (2020). Inspiration cards workshops with primary teachers in the early co-design stages of learning analytics. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, online (pp. 73–82). ACM. https://doi.org/10.1145/3375462.3375537

Wenger, E., McDermott, R. A., & Snyder, W. (2002). Cultivating communities of practice: A guide to managing knowl- edge. Harvard Business Press. https://www.google.com.au/books/edition/Cultivating_Communities_of_Practice/m1xZuNq9RygC?hl=en&gbpv=0

Wise, A. F., Vytasek, J. M., Hausknecht, S., & Zhao, Y. (2016). Developing learning analytics design knowledge in the “middle space”: The Student Tuning Model and Align Design Framework for learning analytics use. Online Learning Journal, 20(2), 155–182. https://doi.org/10.24059/olj.v20i2.783

Downloads

Published

2022-08-31

How to Cite

Schmitz, M., Scheffel, M., Bemelmans, R., & Drachsler, H. (2022). FoLA2 — A Method for Co-creating Learning Analytics–Supported Learning Design. Journal of Learning Analytics, 9(2), 265-281. https://doi.org/10.18608/jla.2022.7643

Issue

Section

Data and Tool Reports