Insights of Instructors and Advisors into an Early Prediction Model for Non-Thriving Students




early warning system, early identification system, non-thriving, prediction model, instructor perceptions, advisor perceptions, data-driven decision making, research paper


In this qualitative study (N=6), we explored insights of first-year students’ instructors and advisors into an early identification system aimed at detecting non-thriving students in the context of an all-campus first-year orientation course for undergraduates. Following the development of that prediction model in a bottom-up manner, using a plethora of available data, we focus on how its end-users could help us understand the underlying mechanisms that drive the identification of non-thriving students. As findings suggest, participants were appreciative overall of the prediction and its timing and came up with various behaviours that could explain non-thriving, mostly motivation and engagement. They suggested additional data that could predict non-thriving, including background information, academic engagement, and learning habits.


Agrusti, F., Bonavolontà, G., & Mezzini, M. (2019). University dropout prediction through educational data mining techniques: A systematic review. Journal of E-Learning and Knowledge Society, 15(3), 161–182.

Akçapınar, G., Altun, A., & Aşkar, P. (2019). Using learning analytics to develop early-warning system for at-risk students. International Journal of Educational Technology in Higher Education, 16(40), 1–20.

Alyahyan, E., & Düştegör, D. (2020). Predicting academic success in higher education: Literature review and best practices. International Journal of Educational Technology in Higher Education, 17(3), 1–21.

Balfanz, R., & Byrnes, V. (2019). Early warning indicators and intervention systems: State of the field. In J. A. Fredricks, A. L. Reschly & S. L. Christenson (Eds.), Handbook of student engagement interventions: Working with disengaged students (pp. 45–55). Elsevier Inc.

Bartolini, A. C., Running, C. L., Duan, X., & Alex Ambrose, G. (2020). Integrated closed-loop learning analytics scheme in a first-year engineering course. Paper presented at the 2020 ASEE Virtual Annual Conference.

Biswas, A. A., Majumder, A., Mia, M. J., Nowrin, I., & Ritu, N. A. (2019). Predicting the enrollment and dropout of students in the post-graduation degree using machine learning classifier. International Journal of Innovative Technology and Exploring Engineering, 8(11), 3083–3088.

Borrella, I., Caballero-Caballero, S., & Ponce-Cueto, E. (2019). Predict and intervene: Addressing the dropout problem in a MOOC-based program. In Proceedings of the 6th ACM Conference on Learning @ Scale (L@S), 24–25 June 2019, Chicago, IL, USA (Article 24, pp. 1–9). ACM Press.

Cates, J. T., & Schaefle, S. E. (2011). The relationship between a college preparation program and at-risk students’ college readiness. Journal of Latinos and Education, 10(4), 320–334.

Cebesoy, U. B., & Akinoglu, O. (2012). The effectiveness of elaboration and organizational strategies in science education on students’ academic achievement, attitude and concept learning. Energy Education Science and Technology Part B: Social and Educational Studies, (Special Issue), 212-218.

Clow, D. (2013). An overview of learning analytics. Teaching in Higher Education, 18(6), 683–695.

Cohen, A. (2017). Analysis of student activity in web-supported courses as a tool for predicting dropout. Educational Technology Research and Development, 65, 1285–1304.

Crompton, H. (2017). ISTE standards for educators: A guide for teachers and other professionals. International Society for Technology in Education.

Dames, S. (2019). The interplay of developmental factors that impact congruence and the ability to thrive among new graduate nurses: A qualitative study of the interplay as students transition to professional practice. Nurse Education in Practice, 36, 47–53.

Dang, S. C., & Koedinger, K. R. (2020). The ebb and flow of student engagement: Measuring motivation through temporal pattern of self-regulation. In A. N. Rafferty, J. Whitehill, C. Romero, & V. Cavalli-Sforza (Eds.), Proceedings of the 13th International Conference on Educational Data Mining (EDM2020), 10–13 July 2020, Online (pp. 61–68). International Educational Data Mining Society.

Doll, J. J. (2010, May). Teachers’ and administrators’ perceptions of the antecedents of school dropout among English language learners at selected Texas schools [Unpublished doctoral dissertation]. Texas A&M University.

Freeman, J. C., & All, A. (2017). Academic support programs utilized for nursing students at risk of academic failure: A review of the literature. Nursing Education Perspectives, 38(2), 69–74.

García-Poole, C., Byrne, S., & Rodrigo, M. J. (2019). Implementation factors that predict positive outcomes in a community-based intervention program for at-risk adolescents. Psychosocial Intervention, 28(2), 57–65.

Gray, C. C., & Perkins, D. (2019). Utilizing early engagement and machine learning to predict student outcomes. Computers and Education, 131, 22–32.

Hawken, L. S., Bundock, K., Kladis, K., O’Keeffe, B., & Barrett, C. A. (2014). Systematic review of the check-in, check-out intervention for students at risk for emotional and behavioral disorders. Education and Treatment of Children, 37(4), 635–658.

Henderson, M., Ryan, T., & Phillips, M. (2019). The challenges of feedback in higher education. Assessment and Evaluation in Higher Education, 44(8), 1237–1252.

Herodotou, C., Rienties, B., Verdin, B., & Boroowa, A. (2019). Predictive learning analytics ‘at scale’: Guidelines to successful implementation in higher education. Journal of Learning Analytics, 6(1), 85–95.

Holmes, H., Lara, A. E., & Brown, G. S. (2020). Social media use in college-age youth: A comprehensive review and a call to action. Current Psychopharmacology, 9(2), 128–143.

Holstein, K., McLaren, B. M., & Aleven, V. (2019). Co-designing a real-time classroom orchestration tool to support teacher–AI complementarity. Journal of Learning Analytics, 6(2), 27–52.

Hout, M. (2012). Social and economic returns to college education in the United States. Annual Review of Sociology, 38, 379–400.

Howard, E., Meehan, M., & Parnell, A. (2018). Contrasting prediction methods for early warning systems at undergraduate level. The Internet and Higher Education, 37, 66–75.

Hsieh, H. F., & Shannon, S. E. (2005). Three approaches to qualitative content analysis. Qualitative Health Research, 15(9), 1277–1288.

Hu, Y. H., Lo, C. L., & Shih, S. P. (2014). Developing early warning systems to predict students’ online learning performance. Computers in Human Behavior, 36, 469–478.

Hutt, S., Ocumpaugh, J., Andres, J. M. A. L., Bosch, N., Paquette, L., Biswas, G., & Baker, R. S. (2021). Investigating SMART models of self-regulation and their impact on learning. In S. I-Han, S. Shaghayegh, F. Bouchet & J.-J. Vie (Eds.), Proceedings of the 14th International Conference on Educational Data Mining (EDM2021), 29 June–2 July 2021, Online (pp. 580–587). International Educational Data Mining Society.

Ifenthaler, D., & Schumacher, C. (2016). Student perceptions of privacy principles for learning analytics. Educational Technology Research and Development, 64, 923–938.

Ifenthaler, D., & Yau, J. Y. K. (2020). Utilising learning analytics to support study success in higher education: A systematic review. Educational Technology Research and Development, 68, 1961–1990.

Iorfa, S., Ugwu, C., Ifeagwazi, C. M., & Chukwuorji, J. C. (2019). Substance use among youths: Roles of psychoticism, social alienation, thriving and religious commitment. African Journal of Drug and Alcohol Studies, 17(2), 133–146.

James, M. L. (2013). Understanding the role of athletics and resiliency in the persistence and success of African American males in a community college setting [Unpublished doctoral dissertation]. Northern Illinois University.

Jones, K. M. L. (2019). Learning analytics and higher education: A proposed model for establishing informed consent mechanisms to promote student privacy and autonomy. International Journal of Educational Technology in Higher Education, 16(24) 1–22.

Kappen, T. H., van Klei, W. A., van Wolfswinkel, L., Kalkman, C. J., Vergouwe, Y., & Moons, K. G. M. (2018). Evaluating the impact of prediction models: Lessons learned, challenges, and recommendations. Diagnostic and Prognostic Research, 2(11), 1–11.

Kehm, B. M., Larsen, M. R., & Sommersel, H. B. (2019). Student dropout from universities in Europe: A review of empirical literature. Hungarian Educational Research Journal, 9(2), 147–164.

Kennedy, C. B. (2017). An analysis of perceptions of dropout factors and interventions by middle school and high school teachers in a southeastern school district [Unpublished doctoral dissertation]. Liberty University.

Klerkx, J., Verbert, K., & Duval, E. (2017). Learning analytics dashboards. In C. Lang, G. Siemens, A. Wise, & D. Gašević (Eds.), Handbook of learning analytics (pp. 143–150). Society for Learning Analytics Research.

Knesting-Lund, K., Reese, D., & Boody, R. (2013). Teachers’ perceptions of high school dropout and their role in dropout prevention: An initial investigation. Journal of Studies in Education, 3(4), 57–71.

Kollom, K., Tammets, K., Scheffel, M., Tsai, Y. S., Jivet, I., Muñoz-Merino, P. J., Moreno-Marcos, P. M., Whitelock-Wainwright, A., Calleja, A. R., Gašević, D., Kloos, C. D., Drachsler, H., & Ley, T. (2021). A four-country cross-case analysis of academic staff expectations about learning analytics in higher education. The Internet and Higher Education, 49, 1–18.

Kommoju, N. (2019). The measure of our commitment is our commitment to measurement: A critical discourse analysis of the language of data in the public discourse of U.S. Secretaries of education [Unpublished master’s thesis]. Georgetown University.

Lammers, W. J., Gillaspy, J. A., & Hancock, F. (2017). Predicting academic success with early, middle, and late semester assessment of student–instructor rapport. Teaching of Psychology, 44(2), 145–149.

Lammers, W. J., Onwuegbuzie, A. J., & Slate, J. R. (2001). Academic success as a function of the gender, class, age, study habits, and employment of college students. Research in the Schools, 8(2), 71–81.

Larsen, A., Horvath, D., & Bridge, C. (2019). ‘Get ready’: Improving the transition experience of a diverse first year cohort through building student agency. Student Success, 11(2), 14–27.

Lawson, C., Beer, C., Rossi, D., Moore, T., & Fleming, J. (2016). Identification of ‘at risk’ students using learning analytics: The ethical dilemmas of intervention strategies in a higher education institution. Educational Technology Research and Development, 64, 957–968.

Lehmann, A. C. (2014). Using admission assessments to predict final grades in a college music program. Journal of Research in Music Education, 62(3), 245–258.

Lim, L.-A., Dawson, S., Gašević, D., Joksimović, S., Pardo, A., Fudge, A., & Gentili, S. (2020). Students’ perceptions of, and emotional responses to, personalised learning analytics-based feedback: An exploratory study of four courses. Assessment and Evaluation in Higher Education, 46(3), 339–359.

Liz-Domínguez, M., Caeiro-Rodríguez, M., Llamas-Nistal, M., & Mikic-Fonte, F. (2019). Predictors and early warning systems in higher education: A systematic literature review. In, M. Caeiro-Rodríguez, Á. Hernández-García & P. J. Muñoz-Merino (Eds.), Proceedings of the Learning Analytics Summer Institute Spain 2019 (LASI-SPAIN 2019), 27–28 June 2019, Vigo, Spain (pp. 84–99).

Lizzio, A., & Wilson, K. (2013). Early intervention to support the academic recovery of first-year students at risk of non-continuation. Innovations in Education and Teaching International, 50(2), 109–120.

Ma, J., Pender, M., & Welch, M. (2019). Education pays 2019: The benefits of higher education for individuals and society. CollegeBoard.

Macfarlane, B. (2015). Student performativity in higher education: Converting learning as a private space into a public performance. Higher Education Research and Development, 34(2), 338–350.

Manrique, R., Nunes, B. P., Marino, O., Casanova, M. A., & Nurmikko-Fuller, T. (2019). An analysis of student representation, representative features and classification algorithms to predict degree dropout. In Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2019, Tempe, AZ, USA (pp. 401–410). ACM Press.

Mastrodicasa, J., & Metellus, P. (2013). The impact of social media on college students. Journal of College and Character, 14(1), 21–30.

Masui, C., Broeckmans, J., Doumen, S., Groenen, A., & Molenberghs, G. (2014). Do diligent students perform better? Complex relations between student and course characteristics, study time, and academic performance in higher education. Studies in Higher Education, 39(4), 621–643.

McMahon, B. M., & Sembiante, S. F. (2020). Re‐envisioning the purpose of early warning systems: Shifting the mindset from student identification to meaningful prediction and intervention. Review of Education, 8(1), 266–301.

Michaeli, S., Kroparo, D., & Hershkovitz, A. (2020). View of Teachers’ use of education dashboards and professional growth. International Review of Research in Open and Distributed Learning, 21(4), 61–78.

Molenaar, I., Horvers, A., Dijkstra, R., & Baker, R. S. (2020). Personalized visualizations to promote young learners’ SRL: The learning path app. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 330–339). ACM Press.

Molenaar, I., & Knoop-van Campen, C. A. N. (2019). How teachers make dashboard information actionable. IEEE Transactions on Learning Technologies, 12(3), 347–355.

Na, K. S., & Tasir, Z. (2017). Identifying at-risk students in online learning by analysing learning behaviour: A systematic review. In Proceedings of the 2017 IEEE International Conference on Big Data Analysis (ICBDA 2017), 16–17 November, Kuching, Malaysia (pp. 118–123).

Nakayama, M., Mutsuura, K., & Yamamoto, H. (2017). The possibility of predicting learning performance using features of note taking activities and instructions in a blended learning environment. International Journal of Educational Technology in Higher Education, 14(6), 1–14.

Nik Nurul Hafzan, M. Y., Safaai, D., Asiah, M., Mohd Saberi, M., & Siti Syuhaida, S. (2019). Review on predictive modelling techniques for identifying students at risk in university environment. In L. M. Hee (Ed.), Proceedings of the Engineering Application of Artificial Intelligence Conference 2018 (EAAIC 2018), 3–5 December 2018, Kota Kinabalu, Malaysia (MATEC Web of Conferences 255, Article 03002, pp. 1–8). EDP Sciences.

Ocumpaugh, J., Baker, R. S., Pedro, M. O. C. Z. S., Hawn, M. A., Heffernan, C., Heffernan, N., & Slater, S. A. (2017). Guidance counselor reports of the ASSISTments college prediction model (ACPM). In Proceedings of the 7th International Conference on Learning Analytics and Knowledge (LAK ’17), 13–17 March 2017, Vancouver, BC, Canada (pp. 479–488). ACM Press.

Oh, P. S., & Oh, S. J. (2011). What teachers of science need to know about models: An overview. International Journal of Science Education, 33(8), 1109–1130.

O’Neill, L., Hartvigsen, J., Wallstedt, B., Korsholm, L., & Eika, B. (2011). Medical school dropout – Testing at admission versus selection by highest grades as predictors. Medical Education, 45(11), 1111–1120.

O’Shea, S., Lysaght, P., Roberts, J., & Harwood, V. (2016). Shifting the blame in higher education: Social inclusion and deficit discourses. Higher Education Research and Development, 35(2), 322–336.

Owen, J. P. (2009). Teachers’ views of what schools are lacking in dropout prevention [Unpublished master’s thesis]. California State University.

Owens, H., Christian, B., & Polivka, B. (2017). Sleep behaviors in traditional-age college students: A state of the science review with implications for practice. Journal of the American Association of Nurse Practitioners, 29(11), 695–703.

Paura, L., & Arhipova, I. (2014). Cause analysis of students’ dropout rate in higher education study program. Procedia: Social and Behavioral Sciences, 109, 1282–1286.

Pérez, L. X. (1998). Sorting, supporting, connecting, and transforming: Intervention strategies for students at risk. Community College Review, 26(1), 63–78.

Plak, S., Cornelisz, I., Meeter, M., & van Klaveren, C. (2022). Early warning systems for more effective student counseling in higher education: Evidence from a Dutch field experiment. Higher Education Quarterly, 76(1), 131-152.

Richman, E. L. (2013). The academic success of college students with attention deficit hyperactivity disorder and learning disabilities [Doctoral dissertation]. University of North Carolina.

Rovira, S., Puertas, E., & Igual, L. (2017). Data-driven system to predict academic grades and dropout. PLoS ONE, 12(2), 1–21.

Salehian Kia, F., Hatala, M., Baker, R. S., & Teasley, S. D. (2021). Measuring students’ self-regulatory phases in LMS with behavior and real-time self report. In Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK ’21), 12–16 April 2021, Irvine, CA, USA (pp. 259–268). ACM Press.

Schalk, C. N., W., Duan, X., Woodard, V. W., Young, K. M. H. & Ambrose, G. A. (2021). Visual and predictive analytics for early identification of non-thriving students in an introductory chemistry course (Poster presentation). Midwest Scholarship of Teaching & Learning (SoTL) Annual Conference. Virtual.

Scholes, V. (2016). The ethics of using learning analytics to categorize students on risk. Educational Technology Research and Development, 64, 939–955.

Seaver, W. B., & Quarton, R. J. (1976). Regression discontinuity analysis of dean’s list effects. Journal of Educational Psychology, 68(4), 459–465.

Selvig, D., Holaday, L. W., Purkiss, J., & Hortsch, M. (2015). Correlating students’ educational background, study habits, and resource usage with learning success in medical histology. Anatomical Sciences Education, 8(1), 1–11.

Seppälä, E. M., Bradley, C., Moeller, J., Harouni, L., Nandamudi, D., & Brackett, M. A. (2020). Promoting mental health and psychological thriving in university students: A randomized controlled trial of three well-being interventions. Frontiers in Psychiatry, 11(Article 590), 1–14.

Stiles Hanlon, A. (2018). Athletes to scholars: Campus ethos and non-cognitive factors that matter to student-athletes at California community colleges [Doctoral dissertation]. California State University.

Strauss, V. (2019, January 23). Why we should stop labeling students as ‘at risk’—and the best alternative. The Washington Post.

Suardiaz-Muro, M., Morante-Ruiz, M., Ortega-Moreno, M., Ruiz, M. A., Martín-Plasencia, P., & Vela-Bueno, A. (2020). Sueño y rendimiento académico en estudiantes universitarios: Revisión sistemática [Sleep and academic performance in university students: A systematic review]. Revista de Neurologia, 71(2), 43–53.

Sun, K., Mhaidli, A. H., Watel, S., Brooks, C. A., & Schaub, F. (2019). It’s my data! Tensions among stakeholders of a learning analytics dashboard. Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19), 4–9 May 2019, Glasgow, Scotland, UK (Paper No. 594, pp. 1–14). ACM Press.

Syed, M., Anggara, T., Lanski, A., Duan, X., Ambrose, G. A., & Chawla, N. V. (2019). Integrated closed-loop learning analytics scheme in a first year experience course. In Proceedings of the 9th International Conference on Learning Analytics and Knowledge (LAK ’19), 4–8 March 2019, Tempe, AZ, USA (Article 4, pp. 521–530). ACM Press.

Tamada, M. M., de Magalhães Netto, J. F., & de Lima, D. P. R. (2019). Predicting and reducing dropout in virtual learning using machine learning techniques: A systematic review. In Proceedings of the 2019 IEEE Frontiers in Education Conference (FIE), 16–19 October 2019, Covington, KY, USA (pp. 1–9). IEEE Computer Society.

Taylor, D. B., & Harrison, G. J. (2018). Supporting biomedical students struggling with second-choice-syndrome to thrive rather than just survive first year. Journal of College Student Retention: Research, Theory and Practice, 20(2), 176–196.

Torenbeek, M., Jansen, E., & Suhre, C. (2013). Predicting undergraduates’ academic achievement: The role of the curriculum, time investment and self-regulated learning. Studies in Higher Education, 38(9), 1393–1406.

Tsai, Y.-S., Rates, D., Moreno-Marcos, P. M., Muñoz-Merino, P. J., Jivet, I., Scheffel, M., Drachsler, H., Delgado Kloos, C., & Gašević, D. (2020). Learning analytics in European higher education: Trends and barriers. Computers and Education, 155, 103933.

Tsai, Y.-S., Moreno-Marcos, P. M., Jivet, I., Scheffel, M., Tammets, K., Kollom, K., & Gašević, D. (2018). The SHEILA framework: Informing institutional strategies and policy processes of learning analytics. Journal of Learning Analytics, 5(3), 5–20.

Tsai, Y.-S., Whitelock-Wainwright, A., & Gašević, D. (2021). More than figures on your laptop: (Dis)trustful implementation of learning analytics. Journal of Learning Analytics, 8(3), 81–100.

Tsiakmaki, M., Kostopoulos, G., Koutsonikos, C., Pierrakeas, S. & Ragos, O. (2018). Predicting university students’ grades based on previous academic achievements. In Proceedings of the 9th International Conference on Information, Intelligence, Systems and Applications (IISA), 23–25 July 2018, Zakynthos, Greece (pp. 1–6). IEEE Computer Society.

Valentine, J. C., Hirschy, A. S., Bremer, C. D., Novillo, W., Castellano, M., & Banister, A. (2011). Keeping at-risk students in school: A systematic review of college retention programs. Educational Evaluation and Policy Analysis, 33(2), 214–234.

van Leeuwen, A., Rummel, N., & van Gog, T. (2019). What information should CSCL teacher dashboards provide to help teachers interpret CSCL situations? International Journal of Computer-Supported Collaborative Learning, 14, 261–289.

Vezzoli, Y., Mavrikis, M., & Vasalou, A. (2020). Inspiration cards workshops with primary teachers in the early co-design stages of learning analytics. In Proceedings of the 10th International Conference on Learning Analytics and Knowledge (LAK ’20), 23–27 March 2020, Frankfurt, Germany (pp. 73–82). ACM Press.

Vuorikari, R., & Castaño Muñoz, J. (Eds.), Ferguson, R., Brasher, A., Clow, D., Cooper, A., Hillaire, G., Mittelmeier, J., Rienties, B., Ullmann, T., & Vuorikari, R. (2016). Research evidence on the use of learning analytics: Implications for education policy, EUR 28294. Publications Office of the European Union.

Waddington, R. J., Nam, S., Lonn, S., & Teasley, S. D. (2016). Improving early warning systems with categorized course resource usage. Journal of Learning Analytics, 3(3), 263–290.

Wagner, P., Schober, B., & Spiel, C. (2008). Time students spend working at home for school. Learning and Instruction, 18(4), 309–320.

Weinstein, C. E. (1982). Training students to use elaboration learning strategies. Contemporary Educational Psychology, 7(4), 301–311.

Whitman, M. (2020). “We called that a behavior”: The making of institutional data. Big Data and Society, 7(1).

Williams, L., Titus, K. J., & Pittman, J. M. (2021). How early is early enough: Correlating student performance with final grades. In Proceedings of the Conference on Computing Education Practice 2021 (CEP '21), 7 January 2021, Durham, UK (pp. 13–16). ACM Press.

Willis, J. E., Slade, S., & Prinsloo, P. (2016). Ethical oversight of student data in learning analytics: A typology derived from a cross-continental, cross-institutional perspective. Educational Technology Research and Development, 64, 881–901.

Woods-Weeks, G. (2017). Graduates of an early college high school: Perceptions of college readiness [Doctoral dissertation]. East Carolina University.

Wright, N. A. (2020). Perform better, or else: Academic probation, public praise, and students decision-making. Labour Economics, 62, 101773.

Yukselturk, E., Ozekes, S., & Türel, Y. K. (2018). Predicting dropout student: An application of data mining methods in an online education program. European Journal of Open, Distance and E-Learning, 17(1), 118–133.

Zabalou, S. J. (2021). Perceptions of high school teachers on factors impacting African American male dropout [Unpublished doctoral dissertation]. St. Thomas University.

Zhang, Y., Fei, Q., Quddus, M., & Davis, C. (2014). An examination of the impact of early intervention on learning outcomes of at-risk students. Research in Higher Education Journal, 26, pp. 1–12.




How to Cite

Hershkovitz, A., & Ambrose, A. (2022). Insights of Instructors and Advisors into an Early Prediction Model for Non-Thriving Students. Journal of Learning Analytics, 1-16.



Research Papers