Research Article
A Proposal for Holistic Assessment of Computational Thinking for Undergraduate: Content Validity

Debby Erce Sondakh, Kamisah Osman , Suhaila Zainudin

1184 1360

Article Metrics

Views

 

1184

Downloads

 

1360

Citations

Crossref

0

Scopus

7


Sondakh DE, Osman K, Zainudin S. A proposal for holistic assessment of computational thinking for undergraduate: content validity. European J Ed Res. 2020;9(1):33-50. doi: 10.12973/eu-jer.9.1.33
Sondakh, D. E., Osman, K., & Zainudin, S. (2020). A proposal for holistic assessment of computational thinking for undergraduate: content validity. European Journal of Educational Research, 9(1), 33-50. https://doi.org/10.12973/eu-jer.9.1.33
Sondakh Debby Erce, Kamisah Osman, and Suhaila Zainudin. "A Proposal for Holistic Assessment of Computational Thinking for Undergraduate: Content Validity," European Journal of Educational Research 9, no. 1 (2020): 33-50. https://doi.org/10.12973/eu-jer.9.1.33
Sondakh, DE Osman, K & Zainudin, 2020, 'A proposal for holistic assessment of computational thinking for undergraduate: content validity', European Journal of Educational Research, vol. 9, no. 1, pp. 33-50. Sondakh, Debby Erce et al. "A Proposal for Holistic Assessment of Computational Thinking for Undergraduate: Content Validity." European Journal of Educational Research, vol. 9, no. 1, 2020, pp. 33-50, https://doi.org/10.12973/eu-jer.9.1.33.

Abstract

Studies have acknowledged computational thinking (CT) as an efficient approach for problem-solving particularly required in digital workplaces. This research aims to identify indicators for a holistic CT assessment instrument for undergraduate students. A three-round fuzzy Delphi study has been conducted to gain comprehensive opinions and consensus from undergraduate lecturers of computer science disciplines and experts from the information technology industry. In round 1, the experts judged a set of predefined indicators describing CT skills and attitudes identified from the literature, while rounds 2 and 3 focused on variables selection. The consensus was achieved on holistic CT, and the indicators are teamwork, communication, spiritual intelligence, generalization, problem-solving, algorithmic thinking, evaluation, abstraction, decomposition, and debugging. Results demonstrate the importance of attitudes in the process of solving a problem and suggest higher education institutions to consider holistic CT in preparing qualified future graduates. Many CT studies focused only on the skills of CT. This study outlines the assessment indicators that consider both CT skills and attitudes, particularly at the undergraduate level.

Keywords: Computational thinking, assessment, fuzzy Delphi method, undergraduate education.


References

ACM and IEEE Computer Society. (2013). Computer science curricula 2013. New York, NY: ACM-IEEE. https://doi.org/10.1145/2534860

Aho, A. V. (2012). Computation and computational thinking. Computer Journal, 55(7), 833–835. https://doi.org/10.1093/comjnl/bxs074

Amram, Y., & Dryer, D. C. (2008, August). The integrated spiritual intelligence scale (ISIS): Development and preliminary validation. Paper presented at the 116th Annual Conference of the American Psychological Association. University of Padova, Boston, MA.

Angeli, C., Voogt, J., Fluck, A., Webb, M., Cox, M., Malyn-Smith, J., … Angeli Charoula; Voogt, J. F. A. (2016). A K-6 Computational thinking curriculum framework: Implications for teacher knowledge. Educational Technology & Society, 19(3), 47–57. https://doi.org/10.2307/jeductechsoci.19.3.47

Atmatzidou, S., & Demetriadis, S. (2014). How to Support students’ computational thinking skills in educational robotics activities. In Proceedings of 4th International Workshop Teaching Robotics, Teaching with Robotics & 5th International Conference Robotics in Education (pp. 43–50). Padova, Italy. :University of Padova

Barr, D., Harrison, J., & Conery, L. (2011). Computational thinking: A digital age skill for everyone. Learning and Leading with Technology, 38(6), 20–23.

Bell, T., & Roberts, J. (2016). Computational Thinking is More About Human than Computers. SET, 1(3), 3–7. https://doi.org/10.1145/1595496.1562941

Boateng, G. O., Neilands, T. B., Frongillo, E. A., Melgar-Quinonez, H. R., & Young, S. L. (2018). Best practices for developing and validating scales for health , social , and behavioral research : A primer. Frontiers in Public Health, 6(June), 1–18. https://doi.org/10.3389/fpubh.2018.00149

Bocconi, S., Chioccariello, A., Dettori, G., Ferrari, A., & Engelhardt, K. (2016). JRC science for policy report. developing computational thinking in compulsory education-implications for policy and practice. Retrieved from https://doi.org/10.2791/792158

Brennan, K., & Resnick, M. (2012, April). New frameworks for studying and assessing the development of computational thinking. Paper presented at Annual American Educational Research Association Meeting, Vancouver, BC, Canada.

Britton, E., Simper, N., Leger, A., & Stephenson, J. (2015). Assessing teamwork in undergraduate education: a measurement tool to evaluate individual teamwork skills. Assessment and Evaluation in Higher Education, 42(3), 378–397. https://doi.org/10.1080/02602938.2015.1116497

Bubno, K., & Takacs, V. L. (2017). The mathability of word problems as initial computer programming exercises. In Proceeding of The 8th IEEE International Conference on Cognitive Infocommunications 2017 (pp. 39–44). Debrecen, Hungary: IEEE. https://doi.org/10.1109/CogInfoCom.2017.8268213

Chang, C. K., Tsai, Y. T., & Chin, Y. L. (2017). A visualization tool to support analyzing and evaluating scratch projects. In T. Matsuo, N. Fukuta, M. Mori, K. Hashimoto, & S. Hirokawa (Eds.), Proceedings - 2017 6th IIAI International Congress on Advanced Applied Informatics, IIAI-AAI 2017, (pp. 498–502). Hamamatsu, Japan: CPS  https://doi.org/10.1109/IIAI-AAI.2017.83

Chang, P. L., Hsu, C. W., & Chang, P. C. (2011). Fuzzy Delphi method for evaluating hydrogen production technologies. International Journal of Hydrogen Energy, 36(21), 14172–14179. https://doi.org/10.1016/j.ijhydene.2011.05.045

Cheng, C., & Lin, Y. (2002). Evaluating the best main battle tank using fuzzy decision theory with linguistic criteria evaluation. European Journal of Operational Research, 142(1), 174–186.

Curzon, P., Dorling, M., Selby, C., & Woollard, J. (2014). Developing computational thinking in the classroom: A framework. Retrieved from http://eprints.soton.ac.uk/369594/10/DevelopingComputationalThinkingInTheClassroomaFramework.pdf

Damigos, D., & Anyfantis, F. (2011). The value of view through the eyes of real estate experts: A Fuzzy Delphi Approach. Landscape and Urban Planning, 101(2), 171–178. https://doi.org/10.1016/j.landurbplan.2011.02.009

de Lima, J. P. C., Carlos, L. M., Schardosim Simfo, J. P., Pereira, J., Mafra, P. M., & da Silva, J. B. (2016). Design and implementation of a remote lab for teaching programming and robotics. IFAC-PapersOnLine, 49(30), 86–91. https://doi.org/10.1016/j.ifacol.2016.11.133

Fronza, I., El Ioini, N., & Corral, L. (2016). Teaching software design engineering across the K-12 curriculum. In Proceedings of the 17th Annual Conference on Information Technology Education - SIGITE ’16 (pp. 97–101). New York, NY: ACM. https://doi.org/10.1145/2978192.2978220

Gouws, L., Bradshaw, K., & Wentworth, P. P. (2013). Computational thinking in educational activities an evaluation of the educational game Light-Bot. Paper presented at the 18th Annual Conference on Innovation and Technology in Computer Science Education. University of Kent, Canterbury, United Kingdom.

Gribble, C. (2014). Exploring “employability” in different cultural contexts. Deakin University Australia. Retrieved from https://www.srhe.ac.uk/downloads/gribble-cate.pdf

Grover, S., & Pea, R. (2013). Computational thinking in K-12: A review of the state of the field. Educational Researcher, 42(1), 38–43. https://doi.org/10.3102/0013189X12463051

Gunbatar, M. S., & Karalar, H. (2018). Gender differences in middle school students’ attitudes and self-efficacy perseptions towards mblock programming. European Journal of Educational Research, 7(4), 925–933. https://doi.org/10.12973/eu-jer.7.4.923

Hsu, Y., Lee, C., & Kreng, V. B. (2010). The application of Fuzzy Delphi Method and Fuzzy AHP in lubricant regenerative technology selection. Expert Systems With Applications, 37(1), 419–425. https://doi.org/10.1016/j.eswa.2009.05.068

Jonassen, D. (2000). Toward a design theory of problem solving. Educational Technology Research and Development, 48(4), 63–85. https://doi.org/10.1007/bf02300500

Kadkhoda, M., & Jahanic, H. (2012). Problem-solving capacities of spiritual intelligence for artificial intelligence. Procedia - Social and Behavioral Sciences, 32, 170–175. https://doi.org/10.1016/j.sbspro.2012.01.027

Kalelioğlu, F., Gülbahar, Y., & Kukul, V. (2016). A framework for computational thinking based on a systematic research review. Baltic J . Modern Computing, 4(3), 583–596.

Kazimoglu, C., Kiernan, M., Bacon, L., & MacKinnon, L. (2012). Learning programming at the computational thinking level via digital game-play. Procedia Computer Science, 9, 522–531. https://doi.org/10.1016/j.procs.2012.04.056

Kert, S. B. (2019). A proposal of in-service teacher training approach for computer science teachers. European Journal of Educational Research, 8(2), 477–489. https://doi.org/10.12973/eu-jer.8.2.477

Korkmaz, O., Cakir, R., & Ozden, M. Y. (2017). A Validity and Reliability Study of the Computational Thinking Scales (CTS). Computers in Human Behavior, 72, 558–569. https://doi.org/10.1016/j.chb.2017.01.005

L’Heureux, J., Boisvert, D., Cohen, R., & Sanghera, K. (2012). IT problem solving: an implementation of computational thinking in information technology. In R. Connolly (Ed.), Proceedings of the 13th annual conference on Information technology education - SIGITE ’12 (pp. 183–188). New York, NY: ACM. https://doi.org/10.1145/2380552.2380606

Lee, I., Martin, F., Denner, J., Coulter, B., Allan, W., Erickson, J., … Werner, L. (2011). Computational thinking for youth in practice. ACM Inroads, 2(1), 32–37.

Leopold, T. A., Ratcheva, V., & Zahidi, S. (2016). The Future of jobs employment, skills and workforce strategy for the fourth industrial revolution. World Economic Forum. https://doi.org/10.1177/1946756712473437

Li, T., & Wang, T. (2012). A unified approach to teach computational thinking for first year non–cs majors in an introductory course. IERI Procedia, 2, 498–503. https://doi.org/10.1016/j.ieri.2012.06.123

Microsoft. (n.d.). Education competencies: Dealing with ambiguity - Microsoft Education. Retrieved July 23, 2017, from https://www.microsoft.com/en-us/education/training-and-events/education-competencies/dealing_with_ambiguity.aspx

Ministry of Research Technology and Higher Education. (2015). Peraturan menteri riset, teknologi, dan pendidikan Tinggi Republik Indonesia No. 44 Tahun 2015 [Regulation of Ministry of Research, Technology and Higher Education No. 44 of 2015 of the Republic of Indonesia]. Retrieved from https://img.akademik.ugm.ac.id/unduh/2015/PERMENRISTEKDIKTI_Nomor_44_Tahun_2015_SNPT.pdf

Missiroli, M., Russo, D., & Ciancarini, P. (2017). Cooperative Thinking, or: Computational Thinking Meets Agile. In 30th IEEE Conference on Software Engineering Education and Training (pp. 187–191). Savannah, GA: IEEE. https://doi.org/10.1109/CSEET.2017.37

Mueller, J., Beckett, D., Hennessey, E., & Shodiev, H. (2017). Assessing computational thinking across the curriculum. In P. J. Rich & C. B. Hodges (Eds.), Emerging Research, Practice, and Policy on Computational Thinking (pp. 251–267). Cham, Switzerland:  Springer International Publishing. https://doi.org/10.1007/978-3-319-52691-1

Nasel, D. D. (2004). Spiritual orientation in relation to spiritual intelligence: A consideration of traditional Christianity and New Age/individualistic spirituality. University of South Australia.

Nikaido, B., & Ventura, J. (2016). Code puzzles - Robot Chronicle. In Proceeding IEEE Southeastcon 2016 (pp. 1-4). Norfolk, VA: IEEE. https://doi.org/10.1109/SECON.2016.7506772

Orchard, C. A., King, G. A., Khalili, H., & Bezzina, M. B. (2012). Assessment of Interprofessional Team Collaboration Scale (AITCS): Development and Testing of the Instrument. Journal of Continuing Education in the Health Professions, 32(1), 58–67. https://doi.org/10.1002/chp.21123.

Parmar, P. I. (2014). Construction and standardization of a spiritual intelligence scale for the pupils of higher secondary schools of Gujarat State. Gujarat University. Retrieved from https://shodhganga.inflibnet.ac.in/handle/10603/110890

Perkovic, L., Settle, A., Hwang, S., & Jones, J. (2010). A framework for computational thinking across the curriculum. In A. Clear & L. R. Dag (Eds.), Proceedings of the 15th Annual Conference on Innovation and Technology in Computer Science Education (pp. 123–127). New York, NY: ACM. https://doi.org/10.1145/1822090.1822126

Pulimood, S. M., Pearson, K., & Bates, D. C. (2016). A Study on the Impact of Multidisciplinary Collaboration on Computational Thinking. In C. Alphonche (Ed.), Proceedings of the 47th ACM Technical Symposium on Computing Science Education - SIGCSE ’16 (pp. 30–35). New York, NY: ACM. https://doi.org/10.1145/2839509.2844636

Qin, H. (2009). Teaching computational thinking through bioinformatics to biology students. ACM SIGCSE Bulletin, 41(1), 188–191. https://doi.org/10.1145/1539024.1508932

Rahayu, T., & Osman, K. (2018). Early study : Self-confidence on the computational thinking skills among science teacher candidates. Paper presented at the 5th International Conference on Islam and Higher Education, Universitas Negeri Padang, Indonesia.

Rahayu, T., & Osman, K. (2019). Knowledge level and self-confidence on the computational thinking skills among science teacher candidates. Al-Biruni Journal of Physics Education/Jurnal Ilmiah Pendidikan Fisika Al-Biruni, 8(April), 117–126. https://doi.org/10.24042/jipfalbiruni.v8i1.4450

Roman-Gonzalez, M., Perez-Gonzelez, J. C., & Jimenez-Fernandez, C. (2016). Which cognitive abilities underlie computational thinking? Criterion validity of the Computational Thinking Test. Computers in Human Behavior, 72, (pp. 1–14). https://doi.org/10.1016/j.chb.2016.08.047

Rowe, G., & Wright, G. (2001). Expert opinions in forecasting: The role of the Delphi technique. In J. S. Armstrong (Ed.), Principles of Forcasting (pp. 125–144). Boston, MA: Springer. https://doi.org/10.1016/j.accreview.2004.12.084

Ruthmann, A., Heines, J. M., Greher, G. R., Laidler, P., & Ii, C. S. (2010). Teaching computational thinking through musical live coding in Scratch. In G. Lewandowksi, S. Wolfman, T. J. Cortina, E. L. Walker, & D. R. Musicant (Eds.) SIGCSE’10 (pp. 3–7). New York, NY: ACM.

Saido, G. A. M., Siraj, S., DeWitt, D., & Al-Amedy, O. S. (2018). Development of an instructional model for higher order thinking in science among secondary school students: a fuzzy Delphi approach. International Journal of Science Education, 40(8), 847–466. https://doi.org/10.1080/09500693.2018.1452307

Saputra, W. S. (2014). Employers’ needs for employability skills of engineering graduates in Indonesia. In A. G. Abdullah, T. Aryanti, D. Kurnia, & S. Elvyanti (Eds.), The 3rd UPI International Conference on Technical and Vocational Education and Training (pp. 223–227). Bandung, Indonesia: Atlantis Press.

Schwab, K. (2016). The Fourth Industrial Revolution. Geneva, Switzerland: World Economic Forum.

Selby, C., & Woollard, J. (2013). Computational thinking : The developing definition. University of Southampton. Retrieved from https://eprints.soton.ac.uk/id/eprint/356481.

Senin, S., & Nasri, N. M. (2019). Teachers’ concern towards applying computational thinking skills in teaching and learning teachers ’ concern towards applying computational thinking skills in teaching and learning. International Journal of Academic Research in Business and Social Sciences, 9(1), 296–310. https://doi.org/10.6007/IJARBSS/v9-i1/5398

Shanmugam, L., Yassin, S. F., & Khalid, F. (2019). Enhancing students ’ motivation to learn computational thinking through mobile application development module ( M-CT ). International Journal of Engineering and Advanced Technology, 8(5), 1293–1303.

Shute, V. J., Sun, C., & Clarke, J. A. (2017). Demystifying computational thinking. Educational Research Review, 22, 142–158. https://doi.org/10.1016/j.edurev.2017.09.003

Sisk, D. A. (2002). Spiritual Intelligence: The tenth intelligence that integrates all other intelligences. Gifted Education International, 16(3), 208–213. https://doi.org/10.1177/026142940201600304

Sisk, D. A. (2016). Spiritual intelligence : Developing higher consciousness revisited. Gifted Education International, 32(3) , 1–15. https://doi.org/10.1177/0261429415602567

Snow, E., Rutstein, D., Bienkowski, M., & Xu, Y. (2017). Principled assessment of student learning in high school computer science. In J. Tenenberg (Ed.), Proceedings of the 2017 ACM Conference on International Computing Education Research - ICER ’17 (pp. 209–216). New York, NY: ACM. https://doi.org/10.1145/3105726.3106186

Strom, P. S., & Strom, R. D. (2011). Teamwork skills assessment for cooperative learning. Educational Research and Evaluation, 17(4), 233–251. https://doi.org/10.1080/13803611.2011.620345

Talib, N., Yasin, S. F. M., & Mohd, K. M. (2017). Teaching and learning computer programming using gamification and observation through action research. International Journal of Academic Research in Progressive Education and Development, 6(3), 1–11. https://doi.org/10.6007/IJARPED/v6-i3/3045

Walden, J., Doyle, M., Garns, R., & Hart, Z. (2013). An informatics perspective on computational thinking. In M. Goldwebber (Ed.), Proceedings of the 18th ACM Conference on Innovation and Technology in Computer Science Education. (pp. 4–9). New York, NY: ACM. https://doi.org/10.1145/2462476.2483797

Wing, J. M. (2006). Computational thinking. Communication of the ACM, 49(3), 33–35.

Wing, J. M. (2008). Computational thinking and thinking about computing. In 2008 IEEE International Symposium on Parallel and Distributed Processing (pp. 1-1.) Miami, FL: IEEE. https://doi.org/10.1109/IPDPS.2008.4536091

Wing, J. M. (2011). Research notebook: Computational thinking—What and why? The Link Magazine. Retrieved from http://www.cs.cmu.edu/link/research-notebook-computational-thinking-what-and-why

World Economic Forum (2018). The future of jobs report 2018. Retrieved from https://www.weforum.org/reports/the-future-of-jobs-report-2018

Yevseyeva, K., & Towhidnejad, M. (2012). Work in progress: Teaching computational thinking in middle and high school. In 2012 Frontiers in Education Conference Proceedings (pp. 1–2). Seattle, WA: IEEE. https://doi.org/10.1109/FIE.2012.6462487

Zakaria, E., Haron, Z., & Daud, M. (2004). The Reliability and construct validity of scores on the attitudes toward problem solving scale. Journal of Science and Mathematics Education in South East Asia, 27(2), 81–91.