A Method for Developing Process-Based Assessments for Computational Thinking Tasks
DOI:
https://doi.org/10.18608/jla.2024.8291Keywords:
process-based assessments, learning analytics, computational thinking, formative assessment, research paperAbstract
Computational thinking (CT) is a concept of growing importance to pre-university education. Yet, CT is often assessed through results, rather than by looking at the CT process itself. Process-based assessments, or assessments that model how a student completed a task, could instead investigate the process of CT as a formative assessment. In this work, we proposed an approach for developing process-based assessments using constructionist tasks specifically for CT assessment in K–12 contexts, with a focus on directly connecting programming artifacts to aspects of CT. We then illustrated such an assessment with 29 students who ranged in CT and programming experience. These students completed both a constructionist task and a traditional CT assessment. Data from the constructionist task was used to build a process-based assessment and results were compared between the two assessment methods. The process-based assessment produced groups of students who differed in their approach to the task with varying levels of success. However, there was no difference between groups of students in the scores on the traditional CT assessment. Process-based assessment from our approach may be useful as formative assessment to give process feedback, localized to the task given to students.
References
Almeida, F., & Xexéo, G. (2019). Word embeddings: A survey. arXiv:1901.09069. https://doi.org/10.48550/arXiv.1901.09069
Angeli, C. (2022). The effects of scaffolded programming scripts on pre-service teachers’ computational thinking: Developing algorithmic thinking through programming robots. International Journal of Child–Computer Interaction, 31, 100329. https://doi.org/10.1016/j.ijcci.2021.100329
Beheshti, E. (2017). Computational thinking (CT) in practice: How STEM professionals use CT in their work. Proceedings of the American Educational Research Association Annual Conference (AERA 2017), 27 April–1 May 2017, San Antonio, TX, USA. https://par.nsf.gov/biblio/10026245-computational-thinking-practice-how-stem-professionals-use-ct-work
Bennett, R. E. (2011). Formative assessment: A critical review. Assessment in Education, 18(1), 5–25. https://doi.org/10.1080/0969594X.2010.513678
Berland, M., Baker, R. S., & Blikstein, P. (2014). Educational data mining and learning analytics: Applications to constructionist research. Technology, Knowledge and Learning, 19(1), 205–220. https://doi.org/10.1007/s10758-014-9223-7
Berland, M., Martin, T., Benton, T., Smith, C. P., & Davis, D. (2013). Using learning analytics to understand the learning pathways of novice programmers. Journal of the Learning Sciences, 22(4), 564–599. https://doi.org/10.1080/10508406.2013.836655
Berndt, D. J., & Clifford, J. (1994). Using dynamic time warping to find patterns in time series. Proceedings of the 3rd International Conference on Knowledge Discovery and Data Mining (AAAIWS ’94), 31 July–1 August 1994, Seattle, WA, USA (pp. 359–370). AAAI Press. https://dl.acm.org/doi/10.5555/3000850.3000887
Blikstein, P. (2011). Using learning analytics to assess students’ behavior in open-ended programming tasks. Proceedings of the 1st International Conference on Learning Analytics and Knowledge (LAK ’11), 27 February–1 March 2011, Banff, AB, Canada (pp. 110–116). ACM Press. https://doi.org/10.1145/2090116.2090132
Blikstein, P., Worsley, M., Piech, C., Sahami, M., Cooper, S., & Koller, D. (2014). Programming pluralism: Using learning analytics to detect patterns in the learning of computer programming. Journal of the Learning Sciences, 23(4), 561–599. https://doi.org/10.1080/10508406.2014.954750
Bocconi, S., Chioccariello, A., Kampylis, P., Dagienė, V., Wastiau, P., Engelhardt, K., Earp, J., Horvath, M., Jasutė, E., Malagoli, C., Masiulionytė-Dagienė, V., & Stupurienė, G. (2022). Reviewing computational thinking in compulsory education: State of play and practices from computing education (A. Inamorato dos Santos, R. Cachia, N, Giannoutsou, & Y. Punie, Eds.). Publications Office of the European Union. https://doi.org/10.2760/126955
Bonner, S., Chen, P., Jones, K., & Milonovich, B. (2021). Formative assessment of computational thinking: Cognitive and metacognitive processes. Applied Measurement in Education, 34(1), 27–45. https://doi.org/10.1080/08957347.2020.1835912
Boom, K.-D., Bower, M., Siemon, J., & Arguel, A. (2022). Relationships between computational thinking and the quality of computer programs. Education and Information Technologies, 27(6), 8289–8310. https://doi.org/10.1007/s10639-022-10921-z
Brennan, K., & Resnick, M. (2012). New frameworks for studying and assessing the development of computational thinking. Proceedings of the American Educational Research Association Annual Conference (AERA 2012), 13–17 April 2012, Vancouver, BC, Canada.
Dong, Y., Catete, V., Jocius, R., Lytle, N., Barnes, T., Albert, J., Joshi, D., Robinson, R., & Andrews, A. (2019). PRADA: A practical model for integrating computational thinking in K–12 education. Proceedings of the 50th ACM Technical Symposium on Computer Science Education (SIGCSE ’19), 27 February–2 March 2019, Minneapolis, MN, USA (pp. 906–912). ACM Press. https://doi.org/10.1145/3287324.3287431
Fields, D. A., Lui, D., & Kafai, Y. B. (2019). Teaching computational thinking with electronic textiles: Modeling iterative practices and supporting personal projects in exploring computer science. In S.-C. Kong & H. Abelson (Eds.), Computational thinking education (pp. 279–294). Springer, Singapore. https://doi.org/10.1007/978-981-13-6528-7_16
Grover, S. (2021). Toward a framework for formative assessment of conceptual learning in K–12 computer science classrooms. Proceedings of the 52nd ACM Technical Symposium on Computer Science Education (SIGCSE ’21), 13–20 March 2021, Virtual, USA (pp. 31–37). ACM Press. https://doi.org/10.1145/3408877.3432460
Grover, S., & Pea, R. (2013). Computational thinking in K–12: A review of the state of the field. Educational Researcher, 42(1), 38–43. https://doi.org/10.3102/0013189X12463051
Grover, S., Pea, R., & Cooper, S. (2015). Designing for deeper learning in a blended computer science course for middle school students. Computer Science Education, 25(2), 199–237. https://doi.org/10.1080/08993408.2015.1033142
Hartmann, C., Rummel, N., & Bannert, M. (2022). Using HeuristicsMiner to analyze problem-solving processes: Exemplary use case of a productive-failure study. Journal of Learning Analytics, 9(2), 66–86. https://doi.org/10.18608/jla.2022.7363
Jiang, B., Zhao, W., Zhang, N., & Qiu, F. (2022). Programming trajectories analytics in block-based programming language learning. Interactive Learning Environments, 30(1), 113–126. https://doi.org/10.1080/10494820.2019.1643741
Kaufman, L., & Rousseeuw, P. J. (2009). Finding groups in data: An introduction to cluster analysis. Wiley.
Kite, V., & Park, S. (2023). Context matters: Secondary science teachers’ integration of process‐based, unplugged computational thinking into science curriculum. Journal of Research in Science Teaching, 61(1), 203–227. https://doi.org/10.1002/tea.21883
Lee, T. Y., Mauriello, M. L., Ahn, J., & Bederson, B. B. (2014). CTArcade: Computational thinking with games in school age children. International Journal of Child–Computer Interaction, 2(1), 26–33. https://doi.org/10.1016/j.ijcci.2014.06.003
Liu, Q., & Paquette, L. (2023). Using submission log data to investigate novice programmers’ employment of debugging strategies. Proceedings of the 13th International Conference on Learning Analytics and Knowledge (LAK ’23), 13–17 March 2023, Arlington, TX, USA (pp. 637–643). ACM Press. https://doi.org/10.1145/3576050.3576094
Lodi, M., & Martini, S. (2021). Computational thinking, between Papert and Wing. Science & Education, 30(4), 883–908. https://doi.org/10.1007/s11191-021-00202-5
Lye, S. Y., & Koh, J. H. L. (2014). Review on teaching and learning of computational thinking through programming: What is next for K–12? Computers in Human Behavior, 41, 51–61. https://doi.org/10.1016/j.chb.2014.09.012
Ma, J., Zhang, Y., Bin, H., Wang, K., Liu, J., & Gao, H. (2022). The development of students’ computational thinking practices in AI course using the game-based learning: A case study. 2022 International Symposium on Educational Technology (ISET), 19–22 July 2022, Hong Kong, China (pp. 273–277). https://doi.org/10.1109/ISET55194.2022.00065
Müller, M. (2007). Dynamic time warping. In Information retrieval for music and motion (pp. 69–84). Springer. https://doi.org/10.1007/978-3-540-74048-3_4
Neamtiu, I., Foster, J. S., & Hicks, M. (2005). Understanding source code evolution using abstract syntax tree matching. Proceedings of the 2005 International Workshop on Mining Software Repositories (MSR ’05), 17 May 2005, St. Louis, MO, USA (pp. 1–5). ACM Press. https://doi.org/10.1145/1083142.1083143
National Research Council. (2013). Next generation science standards: For states, by states. The National Academies Press. https://doi.org/10.17226/18290
OECD. (2020). PISA 2018 results: Effective policies, successful schools (Vol. V). OECD Publishing. https://doi.org/10.1787/ca768d40-en
Oliver, K. M., Houchins, J. K., Moore, R. L., & Wang, C. (2021). Informing makerspace outcomes through a linguistic analysis of written and video-recorded project assessments. International Journal of Science and Mathematics Education, 19(2), 333–354. https://doi.org/10.1007/s10763-020-10060-2
Papavlasopoulou, S., Giannakos, M. N., & Jaccheri, L. (2019). Exploring children’s learning experience in constructionism-based coding activities through design-based research. Computers in Human Behavior, 99, 415–427. https://doi.org/10.1016/j.chb.2019.01.008
Papert, S. (1980). Mindstorms: Children, computers, and powerful ideas. Basic Books.
Parham-Mocello, J., Nelson, A., & Erwig, M. (2022). Exploring the use of games and a domain-specific teaching language in CS0. Proceedings of the 27th ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE ’22), 8–13 July 2022, Dublin, Ireland (pp. 351–357). ACM Press. https://doi.org/10.1145/3502718.3524812
Park, H.-S., & Jun, C.-H. (2009). A simple and fast algorithm for K-medoids clustering. Expert Systems with Applications, 36(2, Part 2), 3336–3341. https://doi.org/10.1016/j.eswa.2008.01.039
Piech, C., Sahami, M., Huang, J., & Guibas, L. (2015). Autonomously generating hints by inferring problem solving policies. Proceedings of the 2nd ACM Conference on Learning @ Scale (L@S 2015), 14–18 March 2015, Vancouver, BC, Canada (pp. 195–204). ACM Press. https://doi.org/10.1145/2724660.2724668
Rabiner, L. R. (1989). A tutorial on hidden Markov models and selected applications in speech recognition. Proceedings of the IEEE, 77(2), 257–286. https://doi.org/10.1109/5.18626
Rich, P. J., Egan, G., & Ellsworth, J. (2019). A framework for decomposition in computational thinking. Proceedings of the 2019 ACM Conference on Innovation and Technology in Computer Science Education (ITiCSE ’19), 15–17 July 2015, Aberdeen, Scotland, UK (pp. 416–421). ACM Press. https://doi.org/10.1145/3304221.3319793
Roque, R., & Tamashiro, M. A. (2022). Making learning visible in constructionist learning contexts. Proceedings of the 21st Annual ACM Interaction Design and Children Conference (IDC ’22), 27–30 June 2022, Braga, Portugal (pp. 69–81). ACM Press. https://doi.org/10.1145/3501712.3534093
Rosé, C. P., McLaughlin, E. A., Liu, R., & Koedinger, K. R. (2019). Explanatory learner models: Why machine learning (alone) is not the answer. British Journal of Educational Technology, 50(6), 2943–2958. https://doi.org/10.1111/bjet.12858
Sands, P., Yadav, A., & Good, J. (2018). Computational thinking in K–12: In-service teacher perceptions of computational thinking. In M. S. Khine (Ed.), Computational thinking in the STEM disciplines: Foundations and research highlights (pp. 151–164). Springer, Cham. https://doi.org/10.1007/978-3-319-93566-9_8
Sedrakyan, G., De Weerdt, J., & Snoeck, M. (2016). Process-mining enabled feedback: “Tell me what I did wrong” vs. “tell me how to do it right.” Computers in Human Behavior, 57, 352–376. https://doi.org/10.1016/j.chb.2015.12.040
Silapachote, P., & Srisuphab, A. (2016). Teaching and learning computational thinking through solving problems in Artificial Intelligence: On designing introductory engineering and computing courses. 2016 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), 7–9 December 2016, Bangkok, Thailand (pp. 50–54). https://doi.org/10.1109/TALE.2016.7851769
Shute, V. J., Sun, C., & Asbell-Clarke, J. (2017). Demystifying computational thinking. Educational Research Review, 22, 142–158. https://doi.org/10.1016/j.edurev.2017.09.003
Tang, X., Yin, Y., Lin, Q., Hadad, R., & Zhai, X. (2020). Assessing computational thinking: A systematic review of empirical studies. Computers & Education, 148, 103798. https://doi.org/10.1016/j.compedu.2019.103798
Tikva, C., & Tambouris, E. (2021). Mapping computational thinking through programming in K–12 education: A conceptual model based on a systematic literature review. Computers & Education, 162, 104083. https://doi.org/10.1016/j.compedu.2020.104083
Van Petegem, C., Deconinck, L., Mourisse, D., Maertens, R., Strijbol, N., Dhoedt, B., De Wever, B., Dawyndt, P., & Mesuere, B. (2022). Pass/fail prediction in programming courses. Journal of Educational Computing Research, 61(1), 68–95. https://doi.org/10.1177/07356331221085595
Veerasamy, A. K., D’Souza, D., & Laakso, M.-J. (2016). Identifying novice student programming misconceptions and errors from summative assessments. Journal of Educational Technology Systems, 45(1), 50–73. https://doi.org/10.1177/0047239515627263
Weintrop, D., Wise Rutstein, D., Bienkowski, M., & McGee, S. (2021). Assessing computational thinking: An overview of the field. Computer Science Education, 31(2), 113–116. https://doi.org/10.1080/08993408.2021.1918380
Weiss, R., Du, S., Grobler, J., Cournapeau, D., Pedregpsa, F., Varoquaux, G., Mueller, A., Thirion, B., Nouri, D., Louppe, G., Vanderplas, J., Benediktsson, J., Buitinck, L., Korobov, M., McGibbon, R., Lattarini, S., Niculae, V., csytracy, Gramfort, A., … Rockhill, A. (2024). hmmlearn (Version 0.3.2) [Data set]. GitHub. https://github.com/hmmlearn/hmmlearn
Wing, J. M. (2006). Computational thinking. Communications of the ACM, 49(3), 33–35. https://doi.org/10.1145/1118178.1118215
Wong, G. K. W., & Jiang, S. (2018). Computational thinking education for children: Algorithmic thinking and debugging. 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), 4–7 December 2018, Wollongong, NSW, Australia (pp. 328–334). https://doi.org/10.1109/TALE.2018.8615232
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Journal of Learning Analytics
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
