FoLA2 — A Method for Co-creating Learning Analytics–Supported Learning Design
DOI:
https://doi.org/10.18608/jla.2022.7643Keywords:
learning design, learning analytics, learning activity , multidisciplinary cooperative method, learning enhancing technology, data and tool reportAbstract
Learning activities are at the core of every educational design effort. Designing learning activities is a process that benefits from reflecting on previous runs of those activities. One way to measure the behaviour and effects of design choices is to use learning analytics (LA). The challenge, however, lies in the unavailability of an easy-to-use, LA-supported learning design (LD) method. We established such a method—the Fellowship of Learning Activities and Analytics (FoLA2)—reinforced by a gameboard and cards, to provide structure and inspiration. The method enables several participants with different roles to interact with a set of card decks to collaboratively create an LA-supported LD. Using this method helps to design learning activities in a collaborative, practical way; it also raises awareness about the benefits of multidisciplinary co-design and connections between LA and LD. FoLA2 can be used to develop, capture, and systematize design elements and to systematically incorporate LA.
References
Alvarez, C. P., Martinez-Maldonado, R., & Buckingham Shum, S. (2020). LA-DECK: A card-based learning analytics co-design tool. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, online (pp. 63–72). ACM. https://doi.org/10.1145/3375462.3375476
Angelides, P. (2001). The development of an efficient technique for collecting and analyzing qualitative data: The analysis of critical incidents. International Journal of Qualitative Studies in Education, 14(3), 429–442. https://doi.org/10.1080/09518390110029058
Baldiris, S. M., Fabregat, R., & De la Hoz Manotas, A. (2010). A set of software tools to build an author assessment package on Moodle: Implementing the AEEA proposal. 2010 10th IEEE International Conference on Advanced Learning Technologies (ICALT 2010), 5–7 July 2010, Sousse, Tunisia (pp. 67–69). IEEE. https://doi.org/10.1109/ICALT.2010.26
Bangor, A., Kortum, P. T., & Miller, J. T. (2008). An empirical evaluation of the system usability scale. International Journal of Human–Computer Interaction, 24(6), 574–594. https://doi.org/10.1080/10447310802205776
Beetham, H. (2007). An approach to learning activity design. In H. Beetham & R. Sharpe (Eds.), Rethinking pedagogy for a digital age: Designing and delivering e-learning (pp. 26–40). Taylor & Francis. https://www.taylorfrancis.com/chapters/edit/10.4324/9780203961681-12/approach-learning-activity-design-helen-beetham
Belbin, R. M. (2011). Management teams: Why they succeed or fail. Emerald Group Publishing Limited. https://doi.org/10.4324/9780080963594
Bellotti, F., Berta, R., & De Gloria, A. (2010). Designing effective serious games: Opportunities and challenges for research. International Journal of Emerging Technologies in Learning (iJET), 5(SI3), 22–35. https://doi.org/10.3991/ijet.v5s3.1500
Boschman, F., McKenney, S., & Voogt, J. (2015). Exploring teachers’ use of TPACK in design talk: The collaborative design of technology-rich early literacy activities. Computers & Education, 82, 250–262. https://doi.org/10.1016/j.compedu.2014.11.010
Brooke, J. (2013). SUS: A retrospective. Journal of Usability Studies, 8(2), 29–40. https://uxpajournal.org/sus-a-retrospective/ Brown, J. S., Collins, A., & Duguid, P. (1989). Situated cognition and the culture of learning. Educational Researcher, 18(1), 32–42. https://doi.org/10.3102/0013189X018001032
Celik, D., & Magoulas, G. D. (2016). A review, timeline, and categorization of learning design tools. In D. K. Chiu, I. Marenzi, U. Nanni, M. Spaniol, & M. Temperini (Eds.), Proceedings of the 15th International Conference on Web-Based Learning (ICWL 2016), 26–29 October 2016, Rome, Italy (pp. 3–13). Springer. https://doi.org/10.1007/978-3-319-47440-3_1
Colvin, C., Rogers, T., Wade, A., Dawson, S., Gasevic, D., Buckingham Shum, S., Nelson, K., Alexander, S., Lockyer, L., & Kennedy, G. (2016). Student Retention and Learning Analytics: A Snapshot of Australian Practices and a Framework for Advancement (tech. rep.). Australian Government Office for Learning and Teaching. https://www.research.ed.ac.uk/en/publications/student-retention-and-learning-analytics-a-snapshot-of-australian
Dalziel, J. (2003). Implementing learning design: The learning activity management system (LAMS) [Workshop presented at ASCILITE 2003, 7–10 December 2003, Adelaide, Australia].
Dawson, S., Joksimovic, S., Poquet, O., & Siemens, G. (2019). Increasing the impact of learning analytics. Proceedings of the Ninth International Conference on Learning Analytics and Knowledge (LAK 2019), 4–8 March 2019, Tempe, AZ, USA (pp. 446–455). ACM. https://doi.org/10.1145/3303772.3303784
Deng, Y., Antle, A. N., & Neustaedter, C. (2014). Tango cards: A card-based design tool for informing the design of tangible learning games. Proceedings of the 2014 Conference on Designing Interactive Systems (DIS 2014), 21–25 June 2014, Vancouver, BC, Canada (pp. 695–704). https://doi.org/10.1145/2598510.2598601
Design School Kolding. (2017). DSKD method cards. https://www.designskolenkolding.dk/en/publications/sustainable-fur-cards
Dollinger, M., & Lodge, J. M. (2018). Co-creation strategies for learning analytics. Proceedings of the Eighth International Conference on Learning Analytics and Knowledge (LAK 2018), 5–9 March 2018, Sydney, Australia (pp. 97–101). ACM. https://doi.org/10.1145/3170358.3170372
Drachsler, H., & Goldhammer, F. (2020). Learning analytics and eAssessment—Towards computational psychometrics by combining psychometrics with learning analytics. In D. Burgos (Ed.), Radical solutions and learning analytics (pp. 67–80). Springer. https://doi.org/10.1007/978-981-15-4526-9_5
Eno, B., & Schmidt, P. (1975). Oblique strategies. Opal. (Limited edition, boxed set of cards.) [rMAB].
Ferna ́ndez-Gallego, B., Lama, M., Vidal, J. C., & Mucientes, M. (2013). Learning analytics framework for educational virtual worlds. Procedia Computer Science, 25, 443–447. https://doi.org/10.1016/j.procs.2013.11.056
Florian-Gaviria, B., Glahn, C., & Gesa, R. F. (2013). A software suite for efficient use of the European qualifications framework in online and blended courses. IEEE Transactions on Learning Technologies, 6(3), 283–296. https://doi.org/10.1109/TLT.2013.18
Fournier, H., Kop, R., & Sitlia, H. (2011). The value of learning analytics to networked learning on a personal learning environment. Proceedings of the First International Conference on Learning Analytics and Knowledge (LAK 2011, 27 February–1 March 2011, Banff, AB, Canada (pp. 104–109). ACM. https://doi.org/10.1145/2090116.2090131
Gasevic, D., Tsai, Y.-S., Dawson, S., & Pardo, A. (2019). How do we start? An approach to learning analytics adoption in higher education. The International Journal of Information and Learning Technology, 6(4), 342–353. https://doi.org/10.1108/IJILT-02-2019-0024
Gasˇevic ́, D., Mirriahi, N., Dawson, S., & Joksimovic ́, S. (2017). Effects of instructional conditions and experience on the adoption of a learning tool. Computers in Human Behavior, 67, 207–220. https://doi.org/10.1016/j.chb.2016.10.026 Greller, W., & Drachsler, H. (2012). Translating learning into numbers: A generic framework for learning analytics. Journal of Educational Technology & Society, 15(3), 42–57. https://www.jstor.org/stable/jeductechsoci.15.3.42
Gruber, M. R. (2019). Designing for Great Teaching with Learning Design Cards. Universita ̈t-Zu ̈rich. https://dlf.uzh.ch/2019/04/01/designing-for-great-teaching-with-learning-design-cards
Haythornthwaite, C. (2017). An information policy perspective on learning analytics. Proceedings of the Seventh International Conference on Learning Analytics and Knowledge (LAK 2017), 13–17 March 2017, Vancouver, BC, Canada (pp. 253– 256). ACM. https://doi.org/10.1145/3027385.3027389
Hevner, A., & Chatterjee, S. (2010). Design science research in information systems. In A. Hevner & S. Chatterjee (Eds.), Design research in information systems (pp. 9–22). Springer. https://doi.org/10.1007/978-1-4419-5653-8_2
Hmelo-Silver, C. E. (2004). Problem-based learning: What and how do students learn? Educational Psychology Review, 16(3), 235–266. https://doi.org/10.1023/B:EDPR.0000034022.16470.f3
Joksimovic ́, S., Kovanovic ́, V., & Dawson, S. (2019). The journey of learning analytics. HERDSA Review of Higher Education, 6, 27–63. https://www.herdsa.org.au/herdsa-review-higher-education-vol-6/37-63
Katsamani, M., Retalis, S., & Boloudakis, M. (2012). Designing a Moodle course with the CADMOS learning design tool. Educational Media International, 49(4), 317–331. https://doi.org/10.1080/09523987.2012.745771
Kelly, N., Thompson, K., & Yeoman, P. (2015). Theory-led design of instruments and representations in learning analytics: Developing a novel tool for orchestration of online collaborative learning. Journal of Learning Analytics, 2(2), 14–43. https://doi.org/10.18608/jla.2015.22.3
Koehler, M., & Mishra, P. (2009). What is technological pedagogical content knowledge (TPACK)? Contemporary Issues in Technology and Teacher Education, 9(1), 60–70. https://doi.org/10.1177/002205741319300303
Koper, R. (2005). An introduction to learning design. In R. Koper & C. Tattersall (Eds.), Learning design (pp. 3–20). Springer. https://doi.org/10.1007/3-540-27360-3_1
Koper, R. (2006). Current research in learning design. Educational Technology & Society, 9(1), 13–22. https://www.jstor.org/ stable/10.2307/jeductechsoci.9.1.13
Koper, R., & Manderveld, J. (2004). Educational modelling language: Modelling reusable, interoperable, rich and personalised units of learning. British Journal of Educational Technology, 35(5), 537–551. https://doi.org/10.1111/j.0007- 1013.2004.00412.x
Koper, R., & Olivier, B. (2004). Representing the learning design of units of learning. Journal of Educational Technology & Society, 7(3), 97–111. https://www.learntechlib.org/p/75442/
Laurillard, D., Charlton, P., Craft, B., Dimakopoulos, D., Ljubojevic, D., Magoulas, G., Masterman, E., Pujadas, R., Whitley, E. A., & Whittlestone, K. (2013). A constructionist learning environment for teachers to model learning designs. Journal of Computer Assisted Learning, 29(1), 15–30. https://doi.org/10.1111/j.1365-2729.2011.00458.x
Law, N., & Liang, L. (2020). A multilevel framework and method for learning analytics integrated learning design. Journal of Learning Analytics, 7(3), 98–117. https://doi.org/10.18608/jla.2020.73.8
Leitner, P., Khalil, M., & Ebner, M. (2017). Learning analytics in higher education—a literature review. In A. Pen ̃a-Ayala (Ed.), Learning analytics: Fundaments, applications, and trends (pp. 1–23). Springer. https://doi.org/10.1007/978-3-319- 52977-6_1
Lockyer, L., Heathcote, E., & Dawson, S. (2013). Informing pedagogical action: Aligning learning analytics with learning design. American Behavioral Scientist, 57(10), 1439–1459. https://doi.org/10.1177/0002764213479367
Macfadyen, L. P., Lockyer, L., & Rienties, B. (2020). Learning design and learning analytics: Snapshot 2020. Journal of Learning Analytics, 7(3), 6–12. https://doi.org/10.18608/jla.2020.73.2
Mangaroska, K., & Giannakos, M. N. (2018). Learning analytics for learning design: A systematic literature review of analytics-driven design to enhance learning. IEEE Transactions on Learning Technologies, 12(4), 516–534. https://doi.org/10.1109/TLT.2018.2868673
Mangaroska, K., Sharma, K., Gasevic, D., & Giannakos, M. (2020). Multimodal learning analytics to inform learning design: Lessons learned from computing education. Journal of Learning Analytics, 7(3), 79–97. https://doi.org/10.18608/jla.2020.73.7
Mankins, J. C. (1995). Technology Readiness Levels (white paper). Advanced Concepts Office, Office of Space Access; Technology, NASA. http://www.artemisinnovation.com/images/TRL White Paper 2004-Edited.pdf
McKenney, S., & Mor, Y. (2015). Supporting teachers in data-informed educational design. British Journal of Educational Technology, 46(2), 265–279. https://doi.org/10.1111/bjet.12262
Means, B., & Neisler, J. (2020). Suddenly Online: A National Survey of Undergraduates during the COVID-19 Pandemic (tech. rep.). Digital Promise. https://www.everylearnereverywhere.org/resources/suddenly-online-national-undergraduate- survey/
Michos, K., Lang, C., Herna ́ndez-Leo, D., & Price-Dennis, D. (2020). Involving teachers in learning analytics design: Lessons learned from two case studies. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, online, 94–99. https://doi.org/10.1145/3375462.3375507
Mislevy, R. J., Almond, R. G., & Lukas, J. F. (2003). A brief introduction to evidence-centered design. ETS Research Report Series, 2003(1), i–29. https://doi.org/10.1002/j.2333-8504.2003.tb01908.x
Moller, O. (n.d.). MethodKit. https://methodkit.com/research-method-cards/
Mor, Y., & Mogilevsky, O. (2013). The learning design studio: Collaborative design inquiry as teachers’ professional development. Research in Learning Technology, 21, 22054. https://doi.org/10.3402/rlt.v21i0.22054
Nguyen, Q., Rienties, B., & Whitelock, D. (2020). A mixed-method study of how instructors design for learning in online and distance education. Journal of Learning Analytics, 7(3), 64–78. https://doi.org/10.18608/jla.2020.73.6
Persico, D., & Pozzi, F. (2015). Informing learning design with learning analytics to improve teacher inquiry. British Journal of Educational Technology, 46(2), 230–248. https://doi.org/10.1111/bjet.12207
Peterson, C. (2003). Bringing ADDIE to life: Instructional design at its best. Journal of Educational Multimedia and Hypermedia, 12(3), 227–241. https://www.learntechlib.org/primary/p/2074/
Plattner, H. (2010). An Introduction to Design Thinking Process Guide. The Institute of Design at Stanford. https://web.stanford.edu/~mshanks/MichaelShanks/files/509554.pdf
Rienties, B., Nguyen, Q., Holmes, W., & Reedy, K. (2017). A review of ten years of implementation and research in aligning learning design with learning analytics at the Open University UK. Interaction Design and Architecture(s), 33, 134–154. https://doi.org/10.55612/s-5002-033-007
Rodr ́ıguez-Triana, M. J., Mart ́ınez-Mone ́s, A., Asensio-Pe ́rez, J. I., & Dimitriadis, Y. (2015). Scripting and monitoring meet each other: Aligning learning analytics and learning design to support teachers in orchestrating CSCL situations. British Journal of Educational Technology, 46(2), 330–343. https://doi.org/10.1111/bjet.12198
Roy, R., & Warren, J. P. (2019). Card-based design tools: A review and analysis of 155 card decks for designers and designing. Design Studies, 63, 125–154. https://doi.org/10.1016/j.destud.2019.04.002
Rubenstein-Montano, B., Liebowitz, J., Buchwalter, J., McCaw, D., Newman, B., Rebeck, K., & The Knowledge Management Methodology Team. (2001). A systems thinking framework for knowledge management. Decision Support Systems, 31(1), 5–16. https://doi.org/10.1016/S0167-9236(00)00116-0
Sarmiento, J. P., Campos, F., & Wise, A. (2020). Engaging students as co-designers of learning analytics. Companion Proceedings of the 10th International Learning Analytics and Knowledge Conference (LAK 2020), 23–27 March 2020, online (pp. 29–32). https://www.solaresearch.org/wp-content/uploads/2020/06/LAK20 Companion Proceedings.pdf
Scheffel, M., Drachsler, H., De Kraker, J., Kreijns, K., Slootmaker, A., & Specht, M. (2016). Widget, widget on the wall, am I performing well at all? IEEE Transactions on Learning Technologies, 10(1), 42–52. https://doi.org/10.1109/TLT.2016.2622268
Scheffel, M., Schmitz, M., van Hooijdonk, J., van Limbeek, E., Kockelkoren, C., Joppe, D., & Drachsler, H. (2021). The design cycle for education (DC4E). In A. Kienle, A. Harrer, J. M. Haake, & A. Lingnau (Eds.), DELFI 2021—Komplettband, 13–15 September 2021, online (pp. 229–240). Gesellschaft fu ̈r Informatik eV. https://dl.gi.de/handle/20.500.12116/37055
Schmitz, M., Scheffel, M., Bemelmans, R., & Drachsler, H. (2020). Fellowship of the Learning Activity—Learning Analytics 4 Learning Design Game. https://doi.org/10.25385/zuyd.9884279
Schmitz, M., Scheffel, M., van Limbeek, E., Bemelmans, R., & Drachsler, H. (2018a). “Make it personal!”—Gathering input from stakeholders for a learning analytics-supported learning design tool. In V. Pammer-Schindler, M. Pe ́rez- Sanagust ́ın, H. Drachsler, R. Elferink, & M. Scheffel (Eds.), European Conference on Technology Enhanced Learning (EC-TEL 2018), 3–5 September 2018, Leeds, UK (pp. 297–310). Springer. https://doi.org/10.1007/978-3-319-98572- 5_23
Schmitz, M., Scheffel, M., van Limbeek, E., van Halem, N., Cornelisz, I., van Klaveren, C., Bemelmans, R., & Drachsler, H. (2018b). Investigating the relationships between online activity, learning strategies and grades to create learning analytics-supported learning designs. In V. Pammer-Schindler, M. Pe ́rez-Sanagust ́ın, H. Drachsler, R. Elferink, & M. Scheffel (Eds.), European Conference on Technology Enhanced Learning (EC-TEL 2018), 3–5 September 2018, Leeds, UK (pp. 311–325). Springer. https://doi.org/10.1007/978-3-319-98572-5_24
Schmitz, M., Van Limbeek, E., Greller, W., Sloep, P., & Drachsler, H. (2017). Opportunities and challenges in using learning analytics in learning design. In E ́. Lavoue ́, H. Drachsler, K. Verbert, J. Broisin, & M. Pe ́rez-Sanagust ́ın (Eds.), European Conference on Technology Enhanced Learning (EC-TEL 2017), 12–15 September 2017, Tallinn, Estonia (pp. 209–223). Springer. https://doi.org/10.1007/978-3-319-66610-5_16
Shaffer, D. W. (2006). Epistemic frames for epistemic games. Computers & Education, 46(3), 223–234. https://doi.org/10.1016/j.compedu.2005.11.003
Siemens, G., & Long, P. (2011). Penetrating the fog: Analytics in learning and education. EDUCAUSE Review, 46(5), 30. https://er.educause.edu/articles/2011/9/penetrating-the-fog-analytics-in-learning-and-education
Slade, S., & Prinsloo, P. (2013). Learning analytics: Ethical issues and dilemmas. American Behavioral Scientist, 57(10), 1510–1529. https://doi.org/10.1177/0002764213479366
Sluijsmans, D., Dochy, F., & Moerkerke, G. (1998). Creating a learning environment by using self-, peer-and co-assessment. Learning Environments Research, 1(3), 293–319. https://doi.org/10.1023/A:1009932704458
Spiro, R. J., & Jheng, J. C. (2012). Cognitive flexibility and hypertext: Theory and technology for the nonlinear and multidi- mensional traversal of complex subject matter. In D. Nix & R. J. Spiro (Eds.), Cognition, education, and multimedia (pp. 177–220). Taylor & Francis. https://www.taylorfrancis.com/chapters/edit/10.4324/9780203052174-11/cognitive-flexibility-hypertext-theory-technology-nonlinear-multidimensional-traversal-complex-subject-matter-rand-spiro-jihn-changjehng
Sweller, J. (2011). Cognitive load theory. In J. P. Mestre & B. H. Ross (Eds.), Psychology of learning and motivation (pp. 37–76). Elsevier. https://doi.org/10.1016/B978-0-12-387691-1.00002-8
Tsai, Y.-S., & Gasevic, D. (2017). Learning analytics in higher education—challenges and policies: A review of eight learning analytics policies. Proceedings of the Seventh International Conference on Learning Analytics and Knowledge (LAK 2017), 13–17 March 2017, Vancouver, BC, Canada (pp. 233–242). ACM. https://doi.org/10.1145/3027385.3027400
Tsai, Y.-S., Moreno-Marcos, P. M., Jivet, I., Scheffel, M., Tammets, K., Kollom, K., & Gasˇevic ́, D. (2018). The SHEILA framework: Informing institutional strategies and policy processes of learning analytics. Journal of Learning Analytics, 5(3), 5–20. https://doi.org/10.18608/jla.2018.53.2
Van Merrienboer, J., & Kirschner, P. (2007). Ten steps to complex learning. Lawrence Erlbaum Associates.
Verpoorten, D., Poumay, M., & Leclercq, D. (2007). The eight learning events model: A pedagogic conceptual tool supporting diversification of learning methods. Interactive Learning Environments, 15(2), 151–160. https://doi.org/10.1080/10494820701343694
Vezzoli, Y., Mavrikis, M., & Vasalou, A. (2020). Inspiration cards workshops with primary teachers in the early co-design stages of learning analytics. Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK 2020), 23–27 March 2020, online (pp. 73–82). ACM. https://doi.org/10.1145/3375462.3375537
Wenger, E., McDermott, R. A., & Snyder, W. (2002). Cultivating communities of practice: A guide to managing knowl- edge. Harvard Business Press. https://www.google.com.au/books/edition/Cultivating_Communities_of_Practice/m1xZuNq9RygC?hl=en&gbpv=0
Wise, A. F., Vytasek, J. M., Hausknecht, S., & Zhao, Y. (2016). Developing learning analytics design knowledge in the “middle space”: The Student Tuning Model and Align Design Framework for learning analytics use. Online Learning Journal, 20(2), 155–182. https://doi.org/10.24059/olj.v20i2.783
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2022 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons License, Attribution - NonCommercial-NoDerivs 3.0 Unported (CC BY-NC-ND 3.0) license that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
