logo logo European Journal of Educational Research

EU-JER is is a, peer reviewed, online academic research journal.

Subscribe to

Receive Email Alerts

for special events, calls for papers, and professional development opportunities.


Publisher (HQ)

Eurasian Society of Educational Research
Eurasian Society of Educational Research
Christiaan Huygensstraat 44, Zipcode:7533XB, Enschede, THE NETHERLANDS
Eurasian Society of Educational Research
Christiaan Huygensstraat 44, Zipcode:7533XB, Enschede, THE NETHERLANDS
analytical thinking learning management models meta analysis propensity score matching research synthesis

Quantifying Influence: Propensity Score Matching Unravels the True Effect Sizes of Learning Management Models on Students’ Analytical Thinking

Supansa Surin , Suntonrapot Damrongpanit

Analytical thinking is crucial for developing problem-solving, decision-making, and higher-order thinking skills. Many researchers have consistently d.


Analytical thinking is crucial for developing problem-solving, decision-making, and higher-order thinking skills. Many researchers have consistently developed learning management models to enhance students' analytical thinking, resulting in extensive knowledge but lacking clear systematic summaries. This study aims to: (a) explore the effect sizes and research characteristics influencing students' analytical thinking, and (b) compare the effect sizes of learning management models after adjusting for propensity score matching. In exploring 131 graduate research papers published between 2002 and 2021, the research utilized forms for recording research characteristics and questionnaires for assessing research quality for data collection. Effect sizes were calculated using Glass's method, while data analysis employed random effects, fixed effects, and regression meta-analysis methods. The findings indicate that (a) research on learning management models significantly impacts students' analytical thinking at a high level (d̅ = 1.428). Seven research characteristics, including year of publication, field of research, level, duration per plan, learning management process, measurement and evaluation, and research quality, statistically influence students' analytical thinking, and (b) after propensity score matching, learning through techniques such as KWL, KWL-plus, Six Thinking Hats, 4MAT, and Mind Mapping had the highest influence on students' analytical thinking. Recommendations for developing students' analytical thinking involve creating a learning management process that fosters understanding, systematic practical training, expanding thinking through collaborative exchanges, and assessments using learning materials and tests to stimulate increased analytical thinking.

Keywords: Analytical thinking, learning management models, meta-analysis, propensity score matching, research synthesis.

cloud_download PDF
Article Metrics



Adesope, O. O., Trevisan, D. A., & Sundararajan, N. (2017). Rethinking the use of tests: A meta-analysis of practice testing. Review of Educational Research, 87(3), 659-701. https://doi.org/10.3102/0034654316689306

Ahmed, I., Sutton, A. J., & Riley, R. D. (2012). Assessment of publication bias, selection bias, and unavailable data in meta-analyses using individual participant data: a database survey. BMJ, 344, Article d7762. https://doi.org/10.1136/bmj.d7762

Akinbadewa, B. O., & Sofowora, O. A. (2020). The effectiveness of multimedia instructional learning packages in enhancing secondary school students’ attitudes toward Biology. International Journal on Studies in Education, 2(2), 119-133. https://doi.org/10.46328/ijonse.19

Akkerman, S., Admiraal, W., Brekelmans, M., & Oost, H. (2008). Auditing quality of research in social sciences. Quality and Quantity, 42, 257-274. https://doi.org/10.1007/s11135-006-9044-4

Amer, A. (2005). Analytical thinking. Center for Advancement of Postgraduate Studies and Research in Engineering Sciences, Faculty of Engineering- Cairo University. https://bit.ly/3TuvErD

Ariyasinsomboon, W. (2001). การสังเคราะห์งานวิจัยในสาขาจิตวิทยาการศึกษา : การวิเคราะห์อภิมาน [Synthesis of research in the field of educational psychology: a meta-analysis] [Doctoral dissertation, Chulalongkorn University]. Chulalongkorn University Intellectual Repository. https://bit.ly/3W2zB8v

Austin, P. C. (2009). The relative ability of different propensity score methods to balance measured covariates between treated and untreated subjects in observational studies. Medical Decision Making, 29(6), 661-677. https://doi.org/10.1177/0272989X09341755

Badhiwala, J. H., Karmur, B. S., & Wilson, J. R. (2021). Propensity score matching: A powerful tool for analyzing observational nonrandomized data. Clinical Spine Surgery, 34(1), 22-24. https://doi.org/10.1097/BSD.0000000000001055

Bai, H. (2011). A comparison of propensity score matching methods for reducing selection bias. International Journal of Research and Method in Education, 34(1), 81-107. https://doi.org/10.1080/1743727X.2011.552338

Balta, N., & Sarac, H. (2016). The effect of 7e learning cycle on learning in science teaching: a meta-analysis study. European Journal of Educational Research, 5(2), 61-72. https://doi.org/10.12973/eu-jer.5.2.61

Benedetto, U., Head, S. J., Angelini, G. D., & Blackstone, E. H. (2018). Statistical primer: propensity score matching and its alternatives. European Journal of Cardio-Thoracic Surgery, 53(6), 1112-1117. https://doi.org/10.1093/ejcts/ezy167

Berkhout, S. W., Haaf, J. M., Gronau, Q. F., Heck, D. W., & Wagenmakers, E.-J. (2024). A tutorial on Bayesian model-averaged meta-analysis in JASP. Behavior Research Methods, 56, 1260-1282. https://doi.org/10.3758/s13428-023-02093-6

Binkley, M., Erstad, O., Herman, J., Raizen, S., Ripley, M., Miller-Ricci, M., & Rumble, M. (2012). Defining twenty-first century skills. In P. Griffin, B. McGaw, & E. Care (Eds.), Assessment and teaching of 21st century skills (pp. 17-66). Springer. https://doi.org/10.1007/978-94-007-2324-5_2

Bloom, B. S. (Ed.). (1956). Taxonomy of educational objectives: The Classification of Educational Goals. Longmans.

Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (Eds.). (2021). Introduction to meta-analysis (2nd ed.). John Wiley & Sons Ltd.

Burch, G. F., Giambatista, R., Batchelor, J. H., Burch, J. J., Hoover, J. D., & Heller, N. A. (2019). A meta‐analysis of the relationship between experiential learning and learning outcomes. Decision Sciences Journal of Innovative Education, 17(3), 239-273. https://doi.org/10.1111/dsji.12188

Card, N. A. (2012). Applied meta-analysis for social science research. The Guilford Press. https://bit.ly/3HWcMuP

Clark, K. R. (2018). Learning theories: constructivism. Radiologic Technology, 90(2), 180-182. https://bit.ly/3SafIsq

Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.). (2019). The handbook of research synthesis and meta-analysis (3rd ed.). Russell Sage Foundation. https://doi.org/10.7758/9781610448864

Cornell, D. (2024, January 3). 17 Analytical thinking examples. Helpful Professor. https://bit.ly/3S3lImV

Czodrowski, P. (2014). Count on kappa. Journal of Computer-Aided Molecular Design, 28, 1049-1055. https://doi.org/10.1007/s10822-014-9759-6

Donoghue, G. M., & Hattie, J. A. C. (2021). A meta-analysis of ten learning techniques. Frontiers in Education, 6, Article 581216. https://doi.org/10.3389/feduc.2021.581216

Dowdy, A., Tincani, M., & Schneider, W. J. (2020). Evaluation of publication bias in response interruption and redirection: A meta‐analysis. Journal of Applied Behavior Analysis, 53(4), 2151-2171. https://doi.org/10.1002/jaba.724

Egger, M., Smith, G. D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a simple, graphical test. BMJ, 315, 629-634. https://doi.org/10.1136/bmj.315.7109.629

Elder, L., & Paul, R. (2007). The thinker’s guide to analytic thinking. The Foundation for Critical Thinking. https://bit.ly/4bOFcE1

Esterhuizen, T. M., & Thabane, L. (2016). Con: Meta-analysis: some key limitations and potential solutions. Nephrology Dialysis Transplantation, 31(6), 882-885. https://doi.org/10.1093/ndt/gfw092

Ferguson, C. J., & Brannick, M. T. (2012). Publication bias in psychological science: Prevalence, methods for identifying and controlling, and implications for the use of meta-analyses. Psychological Methods, 17(1), 120-128. https://doi.org/10.1037/a0024445

Few, S. (2015, November). A course of study in analytical thinking. Perceptual Edge. https://bit.ly/3yz8SGT

Gillies, R. M. (2014). Cooperative learning: Developments in research. International Journal of Educational Psychology, 3(2), 125-140. http://dx.doi.org/10.4471/ijep.2014.08

Glass, G. V. (1976). Primary, secondary, and meta-analysis of research. Educational Researcher, 5(10), 3-8. https://doi.org/10.3102/0013189X005010003

González-Pérez, L. I., & Ramírez-Montoya, M. S. (2022). Components of education 4.0 in 21st century skills frameworks: systematic review. Sustainability, 14(3), Article 1493. https://doi.org/10.3390/su14031493

Harbord, R. M., Harris, R. J., & Sterne, J. A. (2009). Updated tests for small-study effects in meta-analyses. The Stata Journal, 9(2), 197-210. https://doi.org/10.1177/1536867X0900900202

Harris, H., & Horst, S. J. (2016). A brief guide to decisions at each step of the propensity score matching process. Practical Assessment, Research, and Evaluation, 21(4), 1-11. https://doi.org/10.7275/yq7r-4820

Harrison, J. S., Banks, G. C., Pollack, J. M., O’Boyle, E. H., & Short, J. (2017). Publication bias in strategic management research. Journal of Management, 43(2), 400-425. https://doi.org/10.1177/0149206314535438

Haukoos, J. S., & Lewis, R. J. (2015). The propensity score. JAMA, 314(15), 1637-1638. https://doi.org/10.1001/jama.2015.13480

Itsarangkul Na Ayutthaya, T., & Damrongpanit, S. (2022a). A meta-analysis of instructional management models affecting creative thinking development. European Journal of Educational Research, 11(4), 2069-2085. https://doi.org/10.12973/eu-jer.11.4.2069

Itsarangkul Na Ayutthaya, T., & Damrongpanit, S. (2022b). A meta-analysis of instructional management model for students' creative thinking development: An application of propensity score matching. European Journal of Educational Research, 11(4), 2429-2444. https://doi.org/10.12973/eu-jer.11.4.2429

Jacobs, G. M., & Renandya, W. A. (2019). Student centered cooperative learning: Linking concepts in education to promote student learning. Springer. https://doi.org/10.1007/978-981-13-7213-1

Kane, L. T., Fang, T., Galetta, M. S., Goyal, D. K. C., Nicholson, K. J., Kepler, C. K., Vaccaro, A. R., & Schroeder, G. D. (2020). Propensity score matching: a statistical method. Clinical Spine Surgery, 33(3), 120-122. https://doi.org/10.1097/BSD.0000000000000932

Kao, C.-Y. (2014). Exploring the relationships between analogical, analytical, and creative thinking. Thinking Skills and Creativity, 13, 80-88. https://doi.org/10.1016/j.tsc.2014.03.006

Kennedy, T. J., & Sundberg, C. W. (2020). 21st century skills. In B. Akpan, & T. J. Kennedy (Eds.), Science education in theory and practice (pp. 479-496). Springer. https://doi.org/10.1007/978-3-030-43620-9_32

Khairani, Z., Nasution, D., & Bukit, N. (2021). Analysis of science process skills using learning cycle 7e. Journal of Physics: Conference Series, 1811, Article 012085. https://doi.org/10.1088/1742-6596/1811/1/012085

Kmet, L. M., Cook, L. S., & Lee, R. C. (2004). Standard quality assessment criteria for evaluating primary research papers from a variety of fields. Alberta Heritage Foundation for Medical Research. https://bit.ly/3JuWIAU

Krathwohl, D. R. (2002). A revision of Bloom's taxonomy: An overview. Theory Into Practice, 41(4), 212-218. https://doi.org/10.1207/s15430421tip4104_2

Lambert, C. T., & Guillette, L. M. (2021). The impact of environmental and social factors on learning abilities: a meta‐analysis. Biological Reviews, 96(6), 2871-2889. https://doi.org/10.1111/brv.12783

Lampert, M., & Graziani, F. (2009). Instructional activities as a tool for teachers' and teacher educators' learning. The Elementary School Journal, 109(5), 491-509. https://doi.org/10.1086/596998

Lee, Y.-H. (2019). Strengths and limitations of meta-analysis. The Korean Journal of Medicine, 94(5), 391-395. https://doi.org/10.3904/kjm.2019.94.5.391

Lin, L., & Chu, H. (2018). Quantifying publication bias in meta‐analysis. Biometrics, 74(3), 785-794. https://doi.org/10.1111/biom.12817

Moallem, M., Hung, W., & Dabbagh, N. (Eds.). (2019). The Wiley handbook of problem-based learning. John Wiley & Sons. https://doi.org/10.1002/9781119173243

Morgan, C. J. (2018). Reducing bias using propensity score matching. Journal of Nuclear Cardiology, 25(2), 404-406. https://doi.org/10.1007/s12350-017-1012-y

Morrison, G. R., Ross, S. J., Morrison, J. R., & Kalman, H. K. (2019). Designing effective instruction. John Wiley & Sons.

Moust, J., Bouhuijs, P., & Schmidt, H. (2021). Introduction to problem-based learning (4th ed.). Routledge. https://doi.org/10.4324/9781003194187

Nakagawa, S., Lagisz, M., Jennions, M. D., Koricheva, J., Noble, D. W. A., Parker, T. H., Sánchez-Tójar, A., Yang, Y., & O'Dea, R. E. (2022). Methods for testing publication bias in ecological and evolutionary meta‐analyses. Methods in Ecology and Evolution, 13(1), 4-21. https://doi.org/10.1111/2041-210X.13724

Nicol, C. B., Gakuba, E., & Habinshuti, G. (2020). An overview of learning cycles in science inquiry-based instruction. African Journal of Educational Studies in Mathematics and Sciences, 16(2), 67-81. https://doi.org/10.4314/ajesms.v16i2.5

Niu, L., Behar-Horenstein, L. S., & Garvan, C. W. (2013). Do instructional interventions influence college students’ critical thinking skills? A meta-analysis. Educational Research Review, 9, 114-128. https://doi.org/10.1016/j.edurev.2012.12.002

Patton, M. Q. (1999). Enhancing the quality and credibility of qualitative analysis. Health Services Research, 34(5), 1189-1208. https://bit.ly/3RrxWpb

Ramadani, A. S., Supardi, Z. A. I., Tukiran, & Hariyono, E. (2021). Profile of analytical thinking skills through inquiry-based learning in science subjects. Studies in Learning and Teaching, 2(3), 45-60. https://doi.org/10.46627/silet.v2i3.83

Rannikmäe, M., Holbrook, J., & Soobard, R. (2020). Social constructivism-Jerome Bruner. In B. Akpan, & T. J. Kennedy (Eds.), Science education in theory and practice (pp. 259-275). Springer. https://doi.org/10.1007/978-3-030-43620-9_18

Rasheva-Yordanova, K., Iliev, E., & Nikolova, B. (2018). Analytical thinking as a key competence for overcoming the data science divide. In L.G. Chova, A. L. Martínez, & I. C. Torres (Eds.), EDULEARN18 Proceedings: 10th International Conference on Education and New Learning Technologies (pp. 7892-7898). IATED Academy. https://doi.org/10.21125/edulearn.2018.1833

Robbins, J. K. (2011). Problem solving, reasoning, and analytical thinking in a classroom environment. The Behavior Analyst Today, 12(1), 41-48. https://doi.org/10.1037/h0100710

Rodrangsee, B., Tuntiwongwanich, S., Pimdee, P., & Moto, S. (2022). Development of an online active learning model using the theory of multiple intelligence to encourage Thai undergraduate student analytical thinking skills. Journal of Higher Education Theory and Practice, 22(12), 63-75. https://doi.org/10.33423/jhetp.v22i12.5463

Rosenbaum, P. R., & Rubin, D. B. (2023). Propensity scores in the design of observational studies for causal effects. Biometrika, 110(1), 1-13. https://doi.org/10.1093/biomet/asac054

Rowland, C. A. (2014). The effect of testing versus restudy on retention: A meta-analytic review of the testing effect. Psychological Bulletin, 140(6), 1432–1463. https://doi.org/10.1037/a0037559

Rubin, D. B. (1997). Estimating causal effects from large data sets using propensity scores. Annals of Internal Medicine, 127(8), 757-763. https://doi.org/10.7326/0003-4819-127-8_Part_2-199710151-00064

Rücker, G., Schwarzer, G., Carpenter, J. R., & Schumacher, M. (2008). Undue reliance on /2 in assessing heterogeneity may mislead. BMC Medical Research Methodology, 8, Article 79. https://doi.org/10.1186/1471-2288-8-79

Sailer, M., & Homner, L. (2020). The gamification of learning: A meta-analysis. Educational Psychology Review, 32, 77-112. https://doi.org/10.1007/s10648-019-09498-w

Sartika, S. B. (2018). Teaching models to increase students’ analytical thinking skills. In T. W. Abadi, N. E. T. I. Santosa, M. T. Multazam, F. Megawati (Eds.), Proceedings of the 1st International Conference on Intellectuals' Global Responsibility (pp. 216-218). Atlantis Press. https://doi.org/10.2991/icigr-17.2018.52

Saylor, J. G., Alexander, W. M., & Lewis, A. J. (1981). Curriculum planning for better teaching and learning (4th ed.). Holt, Rinehart and Winston.

Sedgwick, P., & Marston, L. (2015). How to read a funnel plot in a meta-analysis. BMJ, 351, Article h4718. https://doi.org/10.1136/bmj.h4718

Sitthipon, A.-I. (2012). Development of teachers' learning management emphasizing on analytical thinking in Thailand. Procedia-Social and Behavioral Sciences, 46, 3339-3344. https://doi.org/10.1016/j.sbspro.2012.06.063

Slavin, R. E. (2014). Aprendizaje cooperativo y rendimiento académico:¿ por qué funciona el trabajo en grupo? [Cooperative learning and academic achievement: Why does groupwork work?]. Annals of Psychology/Anales De Psicología, 30(3), 785-791. https://doi.org/10.6018/analesps.30.3.201201

Smith, P. L., & Ragan, T. J. (2005). A framework for instructional strategy design. In P. L. Smith, & T. J. Ragan (Eds.), Instructional design (3rd ed., pp. 127-150). Wiley & Sons. https://bit.ly/3U79n28

Spaska, A. M., Savishchenko, V. M., Komar, O. A., Нritchenko, T. Y., & Maidanyk, O. V. (2021). Enhancing analytical thinking in tertiary students using debates. European Journal of Educational Research, 10(2), 879-889. https://doi.org/10.12973/eu-jer.10.2.879

Staffa, S. J., & Zurakowski, D. (2018). Five steps to successfully implement and evaluate propensity score matching in clinical research studies. Anesthesia and Analgesia, 127(4), 1066-1073. https://doi.org/10.1213/ANE.0000000000002787

StataCorp. (2023). Stata meta-analysis reference manual: Release 18. Stata Press. https://www.stata.com/manuals/meta.pdf

Stone, D. L., & Rosopa, P. J. (2017). The advantages and limitations of using meta-analysis in human resource management research. Human Resource Management Review, 27(1), 1-7. https://doi.org/10.1016/j.hrmr.2016.09.001

Suyatman, Saputro, S., Sunarno, W., & Sukarmin. (2021). The implementation of research-based learning model in the basic science concepts course in improving analytical thinking skills. European Journal of Educational Research, 10(3), 1051-1062. https://doi.org/10.12973/eu-jer.10.3.1051

Tan, O.-S. (2021). Problem-based learning innovation: Using problems to power learning in the 21st century. Gale Cengage Learning. https://bit.ly/3SYB9yw

Thaneerananon, T., Triampo, W., & Nokkaew, A. (2016). Development of a test to evaluate students' analytical thinking based on fact versus opinion differentiation. International Journal of Instruction, 9(2), 123-138. https://doi.org/10.12973/iji.2016.929a

Thoemmes, F. (2012). Propensity score matching in SPSS. arXiv. https://doi.org/10.48550/arXiv.1201.6385

Triantafyllou, S. A. (2022). Constructivist learning environments. In Proceedings of the 5th international conference on advanced research in teaching and education (pp. 1-6). Diamond Scientific Publishing. https://doi.org/10.33422/5th.icate.2022.04.10

Van Houwelingen, H. C., Arends, L. R., & Stijnen, T. (2002). Advanced methods in meta‐analysis: multivariate approach and meta‐regression. Statistics in Medicine, 21(4), 589-624. https://doi.org/10.1002/sim.1040

Varoglu, L., Yilmaz, A., & Sen, S. (2023). Effect of 5e learning cycle assisted with concept maps on conceptual understanding. Pedagogical Research, 8(3), Article em0161. https://doi.org/10.29333/pr/13167

Vevea, J. L., Coburn, K., & Sutton, A. (2019). Publication bias. In H. Cooper, L. V. Hedges, & J. C. Valentine (Eds.), The handbook of research synthesis and meta-analysis (3rd ed., pp. 383-429). Russell Sage Foundation. https://doi.org/10.7758/9781610448864.21

Wagner, T. (2008). The global achievement gap: Why even our best schools don't teach the new survival skills our children need, and what we can do about it. Basic Books.

World Economic Forum. (2023). Future jobs report 2023. https://bit.ly/3Hqg8Gn

Xu, E., Wang, W., & Wang, Q. (2023). The effectiveness of collaborative problem solving in promoting students’ critical thinking: A meta-analysis based on empirical literature. Humanities and Social Sciences Communications, 10, Article 16. https://doi.org/10.1057/s41599-023-01508-1