logo logo European Journal of Educational Research

EU-JER is is a, peer reviewed, online academic research journal.

Subscribe to

Receive Email Alerts

for special events, calls for papers, and professional development opportunities.


Publisher (HQ)

Eurasian Society of Educational Research
Eurasian Society of Educational Research
7321 Parkway Drive South, Hanover, MD 21076, USA
Eurasian Society of Educational Research
7321 Parkway Drive South, Hanover, MD 21076, USA
education gaps mathematical competence mathematical literacy pisa assessment

Evaluating the Results of PISA Assessment: Are There Gaps Between the Teaching of Mathematical Literacy at Schools and in PISA Assessment?

Imam Kusmaryono , Widya Kusumaningsih

The problems in education in the countries of the Organization for Economic Cooperation and Development (OECD) vary from country to country. The diffe.


The problems in education in the countries of the Organization for Economic Cooperation and Development (OECD) vary from country to country. The differences between "upper class" and "lower class" countries in PISA assessment results have led to a research gap. The purpose of this study was to (a) test students' mathematical literacy skills on the Program for International Student Assessment (PISA) test and compare the results using the sum of means across OECD countries; (b) examine the relationship between students' mathematical competence, precision, and self-perception of mathematical literacy skills in the PISA test; and (c) analyze the gaps that exist between the implementation of mathematics instruction in school and the mathematical literacy as measured on the PISA test. This study was designed as a mixed method with an explanatory sequential design. The data collection methods included test procedures, questionnaires, and interviews. The result of this study showed that the overall mean score obtained was below the OECD average. In general, the respondents achieved only level 2 mathematics proficiency. A significant relationship was found between mathematical competence, precision, and self-perception in mathematical skills. On the other hand, there was a gap, namely the difference at the implementation level, where mathematical literacy measured by PISA differed from the measurement of mathematical learning achievement by teachers in school. The results showed that teaching that emphasizes only problem-solving procedures affects low mathematical competence and is not useful enough for students to deal with the PISA mathematics test.

Keywords: Education gaps, mathematical competence, mathematical literacy, PISA assessment.

cloud_download PDF
Article Metrics



Abramovich, S., Grinshpan, A. Z., & Milligan, D. L. (2019). Teaching mathematics through concept motivation and action learning. Education Research International, 2019, Article 3745406. https://doi.org/10.1155/2019/3745406

Acharya, B. R. (2017). Factors affecting difficulties in learning mathematics by mathematics learners. International Journal of Elementary Education, 6(2), 8–15. https://doi.org/10.11648/j.ijeedu.20170602.11

Adler, R. H. (2022). Trustworthiness in qualitative research. Journal of Human Lactation, 38(4), 598–602. https://doi.org/10.1177/08903344221116620

Ahyan, S., Turmudi, & Juandi, D. (2019). Mathematical literacy of ninth-grade students in solving PISA mathematics problems. International Journal of Innovation, Creativity and Change, 5(6), 483–495. https://bit.ly/3XXZk0m

Aksu, N., Aksu, G., & Saracaloğlu, S. (2022). Prediction of the factors affecting PISA mathematics literacy of students from different countries by using data mining methods. International Electronic Journal of Elementary Education, 14(5), 613–629. https://doi.org/10.26822/iejee.2022.267

Albaladejo, I. M. R., García, M. D. M., & Codina, A. (2015). Developing mathematical competencies in secondary students by introducing dynamic geometry systems in the classroom. Education and Science, 40(177), 43–58. https://doi.org/10.15390/EB.2015.2640

Almarashdi, H. S., & Jarrah, A. M. (2023). Assessing tenth-grade students’ mathematical literacy skills in solving PISA problems. Social Sciences, 12(1), Article 33. https://doi.org/10.3390/socsci12010033

Bem, D. J. (1972). Self-perception theory. Advances in Experimental Social Psychology, 6, 1–62. https://doi.org/10.1016/S0065-2601(08)60024-6

Bøyum, S. (2014). Fairness in education - a normative analysis of OECD policy documents. Journal of Education Policy, 29(6), 856–870. https://doi.org/10.1080/02680939.2014.899396

Castrol, F. G., Kellison, J. G., Boyd, S. J., & Kopak, A. (2010). A methodology for conducting integrative mixed methods research and data analyses. Journal Mix Methods Research, 4(4), 342–360. https://doi.org/10.1177/1558689810382916

Chamorro-Atalaya, O., Gamarra-Bustillos, C., Villanueva-Acosta, V., Samanamud-Loyola, O., Leva-Apaza, A., Tasayco-Jala, A., Torres-Quiroz, A., & Peralta-Eugenio, G. (2022). Self-perception on the acquisition of investigative competencies in the context of virtual learning during Covid-19. International Journal of Information and Education Technology, 12(12), 1417–1423. https://doi.org/10.18178/ijiet.2022.12.12.1766

Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods approaches. SAGE Publications.

Ehmke, T., van den Ham, A.-K., Sälzer, C., Heine, J., & Prenzel, M. (2020). Measuring mathematics competence in international and national large scale assessments: Linking PISA and the national educational panel study in Germany. Studies in Educational Evaluation, 65, Article 100847. https://doi.org/10.1016/j.stueduc.2020.100847

Gurkaynak, G., & Gulcu, A. (2012). The effect of mathematica on primary students’ perceptions of properties of three - dimensional geometric objects. Procedia - Social and Behavioral Sciences, 47, 507–514. https://doi.org/10.1016/j.sbspro.2012.06.689

Harju-Luukkainen, H., Vettenranta, J., Ouakrim-Soivio, N., & Bernelius, V. (2016). Differences between students’ PISA reading literacy scores and grading for mother tongue and literature at school: A geostatistical analysis of the Finnish PISA 2009 data. Education Inquiry, 7(4), Article 29413. https://doi.org/10.3402/edui.v7.29413

Hasibuan, S. A., Fauzi, K. M. A., & Mukhtar, M. (2020). Development of PISA mathematical problem model on the content of change and relationship to measure students mathematical problem-solving ability. International Electronic Journal of Mathematics Education, 15(2), Article em0570. https://doi.org/10.29333/iejme/6274

Hopfenbeck, T. N., Lenkeit, J., El Masri, Y., Cantrell, K., Ryan, J., & Baird, J.-A. (2018). Lessons Learned from PISA: A systematic review of peer-reviewed articles on the programme for international student assessment. Scandinavian Journal of Educational Research, 62(3), 333–353. https://doi.org/10.1080/00313831.2016.1258726

Hwang, J., & Ham, Y. (2021). Relationship between mathematical literacy and opportunity to learn with different types of mathematical tasks. Journal on Mathematics Education, 12(2), 199–222. https://doi.org/10.22342/JME.12.2.13625.199-222

Ioane, B. R. (2017). Cultural adaptation of research instruments – The case of materialism scales culturally adapted for use in China. Asian Business Research, 2(2), 7–22. https://doi.org/10.20849/abr.v2i2.148

Johnston-Wilder, S., Baker, J. K., McCracken, A., & Msimanga, A. (2020). A toolkit for teachers and learners, parents, carers and support staff: Improving mathematical safeguarding and building resilience to increase effectiveness of teaching and learning mathematics. Creative Education, 11(8), 1418–1441. https://doi.org/10.4236/ce.2020.118104

Kandeel, R. A. A. (2021). Learners’ mathematics proficiency levels on PISA 2018: A comparative study. International Journal of Instruction, 14(3), 393–416. https://doi.org/10.29333/iji.2021.14323a

Kaur, T., & Prendergast, M. (2022). Students’ perceptions of mathematics writing and its impact on their enjoyment and self-confidence. Teaching Mathematics and Its Applications, 41(1), 1–21. https://doi.org/10.1093/teamat/hrab008

Koyuncu, İ., & Fırat, T. (2020). Investigating reading literacy in PISA 2018 assessment. International Electronic Journal of Elementary Education, 13(2), 263–275. https://doi.org/10.26822/iejee.2021.189

Krawitz, J., Chang, Y.-P., Yang, K.-L., & Schukajlow, S. (2022). The role of reading comprehension in mathematical modelling: Improving the construction of a real-world model and interest in Germany and Taiwan. Educational Studies in Mathematics, 109, 337–359. https://doi.org/10.1007/s10649-021-10058-9

Kusmaryono, I., Ubaidah, N., & Basir, M. A. (2022). It doesn’t mean that students don’t have mathematics anxiety: A case study of mathematics learning with path analysis. European Journal of Educational Research, 11(3), 1683–1697. https://doi.org/10.12973/eu-jer.11.3.1683

Kusmaryono, I., Wijayanti, D., & Maharani, H. R. (2022). Number of response options, reliability, validity, and potential bias in the use of the Likert scale education and social science research: A literature review. International Journal of Educational Methodology, 8(4), 625–637. https://doi.org/10.12973/ijem.8.4.625

Lara-Porras, A. M., Rueda-García, M. D. M., & Molina-Muñoz, D. (2019). Identifying the factors influencing mathematical literacy in several spanish regions. South African Journal of Education, 39(2), Article 1630. https://doi.org/10.15700/saje.v39ns2a1630

Le Hebel, F., Montpied, P., Tiberghien, A., & Fontanieu, V. (2017). Sources of difficulty in assessment: Example of PISA science items. International Journal of Science Education, 39(4), 468–487. https://doi.org/10.1080/09500693.2017.1294784

Lester, J. N., Cho, Y., & Lochmiller, C. R. (2020). Learning to do qualitative data analysis: A starting point. Human Resource Development Review, 19(1), 94–106. https://doi.org/10.1177/1534484320903890

Mathew, J. S. (2017). Self-perception and academic achievement. Indian Journal of Science and Technology, 10(14), 1–6. https://doi.org/10.17485/ijst/2017/v10i14/107586

Melgar, Á. S., Romero, A. A., Orué, L. A. M., Ynjante, Ó. R. E., & Ocaña-Fernandez, Y. (2022). Learning experiences of mathematical situations in university students from the innovation project. International Journal of Health Sciences, 6(S7), 706–715. https://doi.org/10.53730/ijhs.v6ns7.11231

Michener, E. R. (1978). Understanding understanding mathematics. Cognitive Science, 2(4), 361–383. https://doi.org/10.1016/S0364-0213(78)80052-4

Miller, E. R. (2020). Analyzing the cognitive demand of enacted examples in precalculus: A comparative case study of graduate student instructors. Journal of Mathematics and Science: Collaborative Explorations, 16(1), 202–223. https://bit.ly/3KFZPrc

Möbus, V., Jackisch, C., Lück, H. J., du Bois, A., Thomssen, C., Kuhn, W., Nitz, U., Schneeweiss, A., Huober, J., Harbeck, N., von Minckwitz, G., Runnebaum, I. B., Hinke, A., Konecny, G. E., Untch, M., Kurbacher, C., Kahlert, S., Hauser, N., Marmé, F., … Morenz, B. (2018). Ten-year results of intense dose-dense chemotherapy show superior survival compared with a conventional schedule in high-risk primary breast cancer: Final results of AGO phase III iddEPC trial. Annals of Oncology, 29(1), 178–185. https://doi.org/10.1093/annonc/mdx690

Mouli, M. F., Zulkardi, Z., & Putri, R. I. I. (2023). Development of PISA-type questions and activities in a smartphone context. Mathematics Education Journal/ Jurnal Pendidikan Matematika, 17(1), 1–20. https://doi.org/10.22342/jpm.17.1.19386.1-20

Nowell, L. S., Norris, J. M., White, D. E., & Moules, N. J. (2017). Thematic analysis: Striving to meet the trustworthiness criteria. International Journal of Qualitative Methods, 16(1), 1–13. https://doi.org/10.1177/1609406917733847

Organization for Economic Co-operation and Development. (2009). PISA data analysis manual: SPSS (2nd ed.). OECD Publishing. https://doi.org/10.1787/9789264056275-en

Organization for Economic Co-operation and Development. (2014). 2014 report on the implementation of the OECD strategy on development. OECD Publishing. https://bit.ly/40L2xkM

Organization for Economic Co-operation and Development. (2017). PISA 2015 assessment and analytical framework: Science, reading, mathematic, financial literacy and collaborative problem solving (Revised ed.). OECD Publishing. https://bit.ly/3VU9f5f

Organization for Economic Co-operation and Development. (2018a). PISA for development assessment and analytical framework: Reading, mathematics and science. OECD Publishing. https://bit.ly/3FrAhvp

Organization for Economic Co-operation and Development. (2018b). PISA 2021 mathematics framework (draft). OECD Publishing. https://bit.ly/3uu8p3k

Perales, R. G., & Ruiz, A. P. (2021). Comparison between performance levels for mathematical competence: Results for the sex variable. Frontiers in Psychology, 12, Article 663202. https://doi.org/10.3389/fpsyg.2021.663202

Polya, G. (1962). Mathematical discovery: On understanding, learning, and teaching problem solving. John Wiley & Sons. https://bit.ly/3JtJ1BP

Prendergast, M., Johnson, P., Fitzmaurice, O., Liston, M., O’Keeffe, L., & O’Meara, N. (2014). Mathematical thinking: Challenging prospective teachers to do more than “talk the talk.” International Journal of Mathematical Education in Science and Technology, 45(5), 635–647. https://doi.org/10.1080/0020739X.2013.868538

Pulkkinen, J., & Rautopuro, J. (2022). The correspondence between PISA performance and school achievement in Finland. International Journal of Educational Research, 114, Article 102000. https://doi.org/10.1016/j.ijer.2022.102000

Rose, E., Hands, B., & Larkin, D. (2012). Reliability and validity of the self-perception profile for adolescents: An Australian sample. Australian Journal of Psychology, 64(2), 92–99. https://doi.org/10.1111/j.1742-9536.2011.00031.x

Scandura, J. M. (1966). Precision in research on mathematics learning: The emerging field of psycho-mathematics. Journal of Science in Teaching, 4(4), 253–274. https://doi.org/10.1002/tea.3660040413

Schleicher, A. (2019). PISA 2018: Insights and interpretation. OECD Publishing. https://bit.ly/3UyjCuw

Stacey, K. (2011). The PISA view of mathematical literacy in Indonesia. Journal on Mathematics Education, 2(2), 95–126. https://doi.org/10.22342/jme.2.2.746.95-126

Stahl, N. A. N., & King, J. R. (2020). Expanding approaches for research: Understanding and using trustworthiness in qualitative research. Journal of Developmental Education, 44(1), 26–28. https://bit.ly/3n7dZbK

Stringer, R. W., & Heath, N. (2008). Academic self-perception and its relationship to academic performance. Canadian Journal of Education, 31(2), 327–345.

Tekkumru-Kisa, M., Stein, M. K., & Schunn, C. (2015). A framework for analyzing cognitive demand and content-practices integration: Task analysis guide in science. Journal of Research in Science Teaching, 52(5), 659–685. https://doi.org/10.1002/tea.21208

Timler, A., McIntyre, F., Rose, E., & Hands, B. (2019). Exploring the influence of self-perceptions on the relationship between motor competence and identity in adolescents. PLoS ONE, 14(11), Article e0224653. https://doi.org/10.1371/journal.pone.0224653

Ulia, N., & Kusmaryono, I. (2021). Mathematical disposition of students’, teachers, and parents in distance learning: A survey. Premiere Educandum : Jurnal Pendidikan Dasar dan Pembelajaran, 11(1), 147–159. https://doi.org/10.25273/pe.v11i1.8869

Vostanis, A., Padden, C., Chiesa, M., Rizos, K., & Langdon, P. E. (2021). A precision teaching framework for improving mathematical skills of students with intellectual and developmental disabilities. Journal of Behavioral Education, 30, 513–533. https://doi.org/10.1007/s10864-020-09394-2

Wise, S. L., & DeMars, C. E. (2005). Low examinee effort in low-stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1–17. https://doi.org/10.1207/s15326977ea1001_1

Wisniewski, B., Röhl, S., & Fauth, B. (2022). The perception problem: A comparison of teachers’ self-perceptions and students’ perceptions of instructional quality. Learning Environments Research, 25, 775–802. https://doi.org/10.1007/s10984-021-09397-4

Yeong, M. L., Ismail, R., Ismail, N. H., & Hamzah, M. I. (2018). Interview protocol refinement: Fine-tuning qualitative research interview questions for multi-racial populations in Malaysia. The Qualitative Report, 23(11), 2700–2713. https://doi.org/10.46743/2160-3715/2018.3412

Zaim, M., Refnaldi, R., Zainil, Y., & Ramadhani, F. (2021). PISA reading literacy assessment and senior high school reading literacy assessments: How do they differ? International Journal of Research in Counseling and Education, 5(1), 72-78. https://doi.org/10.24036/00466za0002