Turkish Preschool Teachers’ Opinions on Aggression: Uncertainty of Relational Aggression
Aggression in early childhood is considered to a common problem. Identification, reduction and prevention of aggression, especially in early childhood.
- Pub. date: April 15, 2020
- Pages: 471-487
- 764 Downloads
- 871 Views
- 3 Citations
Aggression in early childhood is considered to a common problem. Identification, reduction and prevention of aggression, especially in early childhood are also considered prevalent. Preschool teachers are known as one of the most important people in children’s lives as they are the first ones to meet the child after parents. Therefore, they hold an important role in identifying, reducing and preventing child aggression observed in early childhood. The purpose of the present study is to examine the knowledge and the awareness levels of preschool teachers about aggression and aggressive behaviours. The study is based on phenomenological approach, as one of the qualitative research designs. A semi-structured interview form, prepared for the study based on expert opinions, was applied to the participants. On the light of the findings from the preliminary findings, the related form was applied to a second group of teachers over the internet. Besides, an additional scale consisting of the items of two different instruments directed to determine the aggressive behaviours of preschool children were formed and applied to the second group of participants. The last scale, which was prepared depending on the findings obtained from the second group of teachers, was applied to a third group of participants because the first two participant groups could not recognize and define relational aggression. The relevant themes and sub-themes were created from the data obtained and the findings were discussed within the scope of the literature. According to the results of the study, it was seen that preschool teachers generally defined aggression as physical and verbal harm, and similarly, aggressive behaviours were categorized under physical aggression and verbal aggression types. It was determined that teachers generally recognized relational aggression when they saw the items related to it in the scale. However, they hardly emphasized it while they were defining aggression. So, it can be said that preschool teachers have limited knowledge and awareness of relational aggression as they recognize it only when they see it but can neither define nor name it. The results are discussed and some suggestions are recommended.
Keywords: Preschool teachers, preschool children, physical aggression, relational aggression, aggressive behaviors
References
Akcay, D., & Ozcebe, H. (2012). The effect of television on preschool children’s aggressive behavior [Televizyonun okul oncesi donemdeki cocuklarin saldirganlik davranisina etkisi]. The Journal of Child Health and Diseases/Cocuk Sagligi ve Hastaliklari Dergisi , 55(2), 82-87.
Allen, J. J., & Anderson, C. A. (2017). Aggression and violence: Definitions and distinctions. In P. Sturmey (Ed.), The Wiley handbook of violence and aggression (pp. 1-28). UK: John Wiley & Sons.
Alink, L. R., Mesman, J., Van-Zeijl, J., Stolk, M.N., Juffer, F., Koot, H. M., Bakermans-Kranenburg, M. J., & Van-Ijzendoorn, M. H. (2006). The early childhood aggression curve: development of physical aggression in 10- to 50-month-old children. Child Development, 77(4), 954-66.
Alisinanoglu, F., & Kesicioglu, O. S. (2010). Okul oncesi donem cocuklarinin davranis sorunlarinin cesitli degiskenler acisindan incelenmesi (Giresun Ili ornegi) [The research on the behavioral problems of preschoolers in terms of various variables]. Journal of Theoretical Educational Science /Kuramsal Egitimbilim, 3(1), 93-110.
Amca, D., & Kivanc Oztug, E. (2016). Examination of the social behavior of 4 age old preschool children according to teacher views. Journal of Computer and Education Research, 4(7), 82-102.
Amin, F. M., Behalik, S. G., & Soreety, W. H. E. (2011). Prevalence and factors associated with aggression among preschool age children at Baraem Bader Nursery School in Al-Asher 10th of Ramadan City, Egypt. Life Science Journal, 8(4), 929-938.
Archer, J., & Coyne, S. M. (2005). An integrated review of indirect, relational, and social aggression. Personality and Social Psychology Review, 9(3), 212-230.
Ari, M., & Yaban, E. H. (2016). Social Behavior in preschool children: the role of temperament and emotion regulation. Hacettepe University Journal of Education, 31(1), 125-141.
Bakircioglu, R. (2012). Egitim ve psikoloji sozlugu [Dictionary of education and psychology]. Ani Publishing.
Bandura, A., Ross, D., & Ross, S. A. (1961). Transmission of aggression through imitation of aggressive models. The Journal of Abnormal and Social Psychology, 63(3), 575-582.
Bauman, S., & Del Rio, A. (2006). Preservice teachers’ responses to bullying scenarios: comparing physical, verbal, and relational bullying. Journal of Educational Psychology, 98(1), 219-231.
Belden, A. C., Gaffrey, M. S., & Luby, J. L. (2012). Relational aggression in children with preschool onset (PO) psychiatric disorders. Journal of the American Academy of Child and Adolescent Psychiatry, 51(9), 1-20. https://doi.org/10.1016/j.jaac.2012.06.018
Bjorkqvist, K. (1997). Learning aggression from models: From a social learning toward a cognitive theory of modeling. In S. Feshbach, & J. Zagrdozka (eds.), Aggression (pp. 69-81). Springer. https://doi.org/10.1007/978-1-4615-5883-5_5
Bushman, B. J., & Huesmann, L. R. (2010). Aggression. In S. T. Fiske, D. T. Gilbert & G. Lindzey (Eds.), Handbook of social psychology (pp. 833-863). John Wiley and Sons.
Chang, L., Schwartz, D., Dodge, K. A., & McBride-Chang, C. (2003). Harsh parenting in relation to child emotion regulation and aggression. Journal of Family Psychology, 17(4), 598-606.
Campbell, S. B., Shaw, D. S., & Gilliom, M. (2000). Early externalizing behavior problems: toddlers and preschoolers at risk for later maladjustment. Developmental Psychopathology, 12(3), 467-88.
Carpenter, E. M., & Nangle, D. W. (2006). Caught between stages: Relational aggression emerging as a developmental advance in at-risk preschoolers. Journal of Research in Childhood Education, 21(2), 177-188.
Casper, D. M., & Card, N. A. (2017). Overt and relational victimization: A meta‐analytic review of their overlap and associations with social–psychological adjustment. Child Development, 88(2), 466-483.
Christakis, D. A., Garrison, M. M., Herrenkohl, T., Haggerty, K., Rivara, F. P., Zhou, C., & Liekweg, K. (2013). Modifying media content for preschool children: a randomized controlled trial. Pediatrics, 131(3), 431-438.
Cohen J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37-46.
Coplan, R. J., Bullock, A., Archbell, K., & Bosacki, S. (2015). Preschool teachers’ attitudes, beliefs, and emotional reactions to young children’s peer group behaviors. Early Childhood Research Quarterly, 30(1), 117–127. https://doi.org/10.1016/j.ecresq.2014.09.005
Cote, S. M., Vaillancourt, T., LeBlanc, J. C., Nagin, D. S., & Tremblay, R. E. (2006). The development of physical aggression from toddlerhood to pre-adolescence: A nationwide longitudinal study of Canadian children. Journal of Abnormal Child Psychology, 34(1), 71–85.
Coyne, S. M., & Ostrov, J. M., eds. (2018). The development of relational aggression. Oxford University Press.
Coyne, S. M., Robinson, S. L., & Nelson, D. A. (2010). Does reality backbite? Physical, verbal, and relational aggression in reality television programs. Journal of Broadcasting & Electronic Media, 54(2), 282-298.
Creswell, J. W. (2016). Research design: Qualitative, quantitative and mixed methods approaches (S. B. Demir, Trans.). Edifier Book.
Crick, N. R., Casas, J. F., & Mosher, M. (1997). Relational and overt aggression in preschool. Developmental Psychology, 33(4), 579-588.
Crick, N. R., Ostrov, J. M., Burr, J. E., Cullerton-Sen, C., Jansen-Yeh, E., & Ralston, P. (2006). A longitudinal study of relational and physical aggression in preschool. Applied Developmental Psychology, 27(3), 254-268. https://doi.org/10.1016/j.appdev.2006.02.006
Crick, N. R., & Rose, A. J. (2000). Toward a gender-balanced approach to the study of social-emotional development: A look at relational aggression. In P. H. Miller & S. E. Kofsky (Eds.), Toward a feminist developmental psychology (pp. 153-168). Taylor & Frances/Routledge.
Dailey, A. L. (2015). Relational aggression in school settings: The efficacy of the First Step to Success intervention (Unpublished doctoral dissertation). University of Louisville.
DeWall, C. N., Anderson, C. A., & Bushman, B. J. (2011). The general aggression model: Theoretical extensions to violence. Psychology of Violence, 1(3), 245-258.
Dias, D., & Ventura, T. (2017). The role of teacher`s attitudes towards the aggressive behavior of pupils. International Journal of Technology and Inclusive Education, 6(1), 1068-1076.
Dobbs, J., & Arnold, D. (2009). Relationship between preschool teachers’ reports of children’s behavior and their behavior toward those children. School Psychology Quarterly, 24(2), 95–105.
Ersan, C. (2015). The effect of fathers' language training program on the receptive language development of children. Education and Science, 40(180), 51-71.
Ersan, C. (2017). Okul oncesi donem cocuklarinin saldirganlik duzeylerinin duygu ifade etme ve duygu duzenleme acisindan incelenmesi [The study of the aggression levels of preschool children in terms of emotion expression and emotion regulation] (Unpublished doctoral dissertation). Pamukkale University.
Ersan, C. (2020). Physical aggression, relational aggression and anger in preschool children: The mediating role of emotion regulation. The Journal of General Psychology, 147(1), 18-42. Https://doi.org/10.1080/00221309.2019.1609897
Ettekal, I., & Ladd, G. W. (2017). Developmental continuity and change in physical, verbal, and relational aggression and peer victimization from childhood to adolescence. Developmental Psychology, 53(9), 1709-1721.
Girard, L. C., Pingault, J. B., Falissard, B., Boivin, M., Dionne, G., & Tremblay, R. E. (2014). Physical aggression and language ability from 17 to 72 months: cross-lagged effects in a population sample. PloS One, 9(11), e112185. https://doi.org/10.1371/journal.pone.0112185
Girard, L. C., Tremblay, R. E., Nagin, D., & Cote, S. M. (2019). Development of aggression subtypes from childhood to adolescence: a group-based multi-trajectory modelling perspective. Journal of Abnormal Child Psychology, 47(5), 825-838.
Goldstein, N. E., Arnold, D. H., Rosenberg, J. L., Stowe, R. M., & Ortiz, C. (2001). Contagion of aggression in day care classrooms as a function of peer and teacher responses. Journal of Educational Psychology, 93(4), 708-719.
Gonzalez-Pena, P., Egido, B. D., Carrasco, M. A., & Holgado-Tello, F. P. (2013). Aggressive behavior in children: The role of temperament and family socialization. The Spanish Journal of Psychology, 16(1), 1-12. https://doi.org/10.1017/sjp.2013.28
Gower, A. L., Lingras, K. A., Mathieson, L. C., Kawabata, Y., & Crick, N. R. (2014). The role of preschool relational and physical aggression in the transition to kindergarten: Links with social-psychological adjustment. Early Education and Development, 25(5), 619-640.
Gultekin-Akduman, G. (2012). Okul oncesi donemde akran zorbaliginin incelenmesi [An investigation of peer bullying at preschool period]. Journal of Society and Social Work /Toplum ve Sosyal Hizmet Dergisi, 23(1), 121-137.
Henninger, D. (2004). Relational versus physical aggression: An overview. Graduate Student Journal of Psychology, 6(1), 13-17.
Herrenkohl, T. I., Catalano, R. F., Hemphill, S. A., & Toumbourou, J. W. (2009). Longitudinal examination of physical and relational aggression as precursors to later problem behaviors in adolescents. Violence and Victims, 24(1), 3-19.
Huber, R., & Brennan, P. A. (2011). Aggression. In R. Huber, D. L. Bannasch & P. Brennan (Eds.), Advances in genetics (pp. 1-6). Elsevier.
Hurst, L. R. (2017). Relational aggression: what administrators know and how student needs are addressed (Unpublished doctoral thesis). The University of Toledo.
Ingram, G. P., & Bering, J. M. (2010). Children’s tattling: The reporting of everyday norm violations in preschool settings. Child Development, 81(3), 945-957.
Juliano, M., Werner, R. S., & Cassidy, K. W. (2006). Early correlates of preschool aggressive behavior according to type of aggression and measurement. Journal of Applied Developmental Psychology, 27(5), 395–410. https://doi.org/10.1016/j.appdev.2006.06.008
Karakus, A. (2008). Okul Oncesi Sosyal Davranis Olcegi Ogretmen Formu'nun guvenirlik ve gecerlik calismasi [Validity and reliablity study of Teacher’s Form of Preschool Social Behavior Scale] (Unpublished master’s thesis). Marmara University.
Kaynak, K. B., Kan, A., & Kurtulmus, Z. (2016). 36-72 aylik cocuklara yonelik “Saldirganlik Yonelim Olcegi” gelistirme calismasi [Development of “Aggression Tendency Scale” for 36-72 months-old children]. Turkish Studies, 11(3), 1457-1474.
Klein, H., & Shiffman, K. S. (2012). Verbal aggression in animated cartoons. International journal of child and adolescent health, 5(1), 7-12.
Kuppens, S., Laurent, L., Heyvaert, M., & Onghena, P. (2013). Associations between parental psychological control and relational aggression in children and adolescents: a multilevel and sequential meta-analysis. Developmental Psychology, 49(9), 1697-1712.
Landsford, J. E., Skinner, A. T., Sorbring, E., Di Giunta, L., Deater-Deckard, K., Dodge, K. A., Malone, P. S., Oburu, P., Pastorelli, C., Tapanya, S., Tirado, L. M. U., Zelli, A., Al-Hassan, S. M., Alampay, l. P., Baccihini, D., Bombi, A. S., Bornstein, M. H., & Chang, L. (2012). Boys’ and girls’ relational and physical aggression in nine countries. Aggressive Behavior, 38(4), 298–308. http://doi.org/10.1002/ab.21433
Lei, H., Chiu, M. M., Cui, Y., Zhou, W., & Li, S. (2018). Parenting style and aggression: A meta-analysis of Mainland Chinese children and youth. Children and Youth Services Review, 94(1), 446-455.
Lindley, K. A. (2014). “I won’t be your friend unless…” parent’s perceptions of relational aggression in preschoolers (Unpublished master’s thesis). Texas Tech University.
Llorca, A., Cristina Richaud, M., & Malonda, E. (2017). Parenting, peer relationships, academic self-efficacy, and academic achievement: direct and mediating effects. Frontiers in Psychology, 8, 2120. https://doi.org/10.3389/fpsyg.2017.02120
Masud, H., Ahmad, M. S., Cho, K. W., & Fakhr, Z. (2019). Parenting styles and aggression among young adolescents: a systematic review of literature. Community Mental Health Journal, 55(6), 1015-1030. https://doi.org/10.1007/s10597-019-00400-0
McGilloway, S., Hyland, L., Mhaille, G. N., Lodge, A., O’Neill, D., Kelly, P., Leckey, Y., Bywater, T., Comiskey, C., & Donnelly, M. (2011). Positive classrooms, positive children: A randomised controlled trial to investigate the effectiveness of the Incredible Years Teacher Classroom Management programme in an Irish Context (short-term outcomes) (Summary Report). The Incredible Years. http://www.incredibleyears.com/wp-content/uploads/Positive-classrooms-Archways-TCM-Report-2011.pdf
Morine, K. A., Crothers, L. M., Schreiber, J. B., Kolbert, J. B., Hughes, T. L., & Schmitt, A. J. (2011). Relational aggression in preschool students: An exploration of the variables of sex, age, and siblings. Child Development Research, 2011(1), 1-6. https://doi.org/10.1155/2011/931720
Morrison, G. M., Furlong, M. J., & Morrison, R. L. (1994) School violence to school safety: Reframing the issue for school psychologists. School Psychology Review, 23(2), 236-256.
Ogul, I. G., & Yurtal, F. (2014). Stories about aggression in pre-school children: teachers' and mothers' views and intervention strategies. International Journal of Academic Research Part B, 6(6), 63-68.
Oliver, R. M., Wehby, J. H., & Reschly, D. J. (2011). Teacher classroom management practices: Effects on disruptive or aggressive student behavior. Campbell Systematic Reviews, 7(1), 1-55.
Orpinas, P., & Home, A. M. (2004). A teacher-focused approach to prevent and reduce students' aggressive behavior: The GREAT Teacher Program. American Journal of Preventive Medicine, 26(1), 29–38. https://doi.org/10.1016/j.amepre.2003.09.016
Ostrov, J. M., Crick, N. R., & Stauffacher, K. (2006). Relational aggression in sibling and peer relationships during early childhood. Journal of Applied Developmental Psychology, 27(3), 241–253.
Ostrov, J. M., & Keating, C. F. (2004). Gender differences in preschool aggression during free play and structured interactions: An observational study. Social Development, 13(2), 255-277.
Ostrov, J. M., Woods, K. E., Jansen, E. A., Casas, J. F., & Crick, N. R. (2004). An observational study of delivered and received aggression and social psychological adjustment in preschool: “This white crayon doesn’t work.” Early Childhood Research Quarterly, 19(2), 355-371. https://doi.org/10.1016/j.ecresq.2004.04.009
Ozdemir, S., & Tepeli, K. (2015). Okul oncesi cocuklarda gorulen fiziksel ve iliskisel saldirganlikdavranislarinin incelenmesi [An investigation of physical and relational aggression behaviors during preschool period]. International Journal of Eurasia Social Sciences /Uluslararasi Avrasya Sosyal Bilimler Dergisi, 6(18), 229-245.
Ozelturkay, E. Y., & Bozyigit, S. (2017). Postmodern tuketim ekseninde cocuk dogum gunu kutlamalari: Kesifsel bir arastirma [Children's birthday celebrations on the axis of postmodern consumption: an exploratory research]. Journal of Academic Researches and Studies/ Akademik Arastirmalar ve Calismalar Dergisi, 9(17), 155-166.
Pathirana, B. D. (2016). Aggressive behaviors of Srilankan preschool children: A mixed method study. The International Journal of Indian Psychology, 3(4), 57- 66.
Pazarbasi, B. (2018). Content analysis of cartoons which broadcast on television and main theme is violence. Intermedia International Peer-Reviewed E-Journal of Communication Sciences, 5(8), 28-41.
Pingault, J.B., Cote, S. M., Lacourse, E., Galera, C., Vitaro, F., & Tremblay, R. E. (2013). Childhood hyperactivity, physical aggression and criminality: a 19-year prospective population based study. PLoS ONE, 8(5), e62594. https://doi.org/10.1371/journal.pone.0062594
Potegal, M., & Archer, J. (2004). Sex differences in childhood anger and aggression. Child and Adolescent Psychiatric Clinics, 13(3), 513-528.
Putallaz, M., Grimes, C. L., Foster, K. J., Kupersmidt, J. B., Coie, J. D., & Dearing, K. (2007). Overt and Relational Aggression and Victimization: Multiple Perspectives within the School Setting. Journal of school psychology, 45(5), 523-547. https://doi.org/10.1016/j.jsp.2007.05.003
Rich, E. C., Shepherd, E. J., & Nangle, D. W. (2008). Validation of the SSRS-T, preschool level as a measure of positive social behavior and conduct problems. Education and Treatment of Children, 31(2), 183-202.
Romi, S., Lewis, R. R., Roacheb, J., & Riley, P. (2011). The impact of teachers' aggressive management techniques on students' attitudes to schoolwork. The Journal of Educational Research, 104(4), 231-240. https://doi.org/10.1080/00220671003719004
Runions, K. C., & Shaw, T. (2013). Teacher–child relationship, child withdrawal and aggression in the development of peer victimization. Journal of Applied Developmental Psychology, 34(6), 319–327.
San Kuay, H., Tiffin, P. A., Boothroyd, L. G., Towl, G. J., & Centifanti, L. C. (2017). A new trait-based model of child-to-parent aggression. Adolescent Research Review, 2(3), 199-211.
Smith, R. L., Rose, A. J., & Schwartz‐Mette, R. A. (2010). Relational and overt aggression in childhood and adolescence: Clarifying mean‐level gender differences and associations with peer acceptance. Social Development, 19(2), 243-269.
Song, H., Kim, J., & Yoo, H. (2015). Measures of aggression in young Korean children: A review of 10 years of empirical research. Universal Journal of Psychology, 3(4), 97-105.
Suarez-Relinque, C., del Moral Arroyo, G., Leon-Moreno, C., & Jeronimo, J. E. C. (2019). Child-to-parent violence: which parenting style is more protective? A study with Spanish adolescents. International Journal of Environmental Research and Public Health, 16(8), 1320.
Sullivan, A. M., Johnson, B., Owens, L., & Conway, R. (2014). Punish them or engage them? Teachers’ views of unproductive student behaviours in the classroom. Australian Journal of Teacher Education, 39(6), 43-56.
Swit, C. S. (2019). Differential associations between relational and physical aggression: Why do teachers and parents perceive these behaviors differently? Early Child Development and Care. Advance online publication, 1-17. https://doi.org/10.1080/03004430.2019.1621304
Taylor, A. J. M., & Jose, M. (2014). Physical aggression and facial expression identification. Europe's Journal of Psychology, 10(4), 650–659. https://doi.org/10.5964/ejop.v10i4.816
Taylor, S. E., Peplau, L. A., & Sears, D. O. (2007). Social psychology (A. Donmez, Trans.). Imge Bookstore.
Taylor, G. G., & Smith, S. W. (2019). Teacher reports of verbal aggression in school settings among students with emotional and behavioral disorders. Journal of Emotional and Behavioral Disorders, 27(1), 52-64.
Trawick-Smith, J. (2014). Early childhood development: a multicultural perspective (B. Akman, Trans.). Nobel Publishing.
Tremblay, R. E. (2010). Developmental origins of disruptive behaviour problems: The “original sin” hypothesis, epigenetics and their consequences for prevention. Journal of Child Psychology and Psychiatry, 51(4), 341–367.
Tremblay, R. E. (2012). Aggression. Encyclopedia on early childhood development. The Encyclopedia on Early Childhood Development. http://www.child-encyclopedia.com/sites/default/files/dossiers-complets/en/aggression.pdf
Tremblay, R. E., Nagin, D. S., Seguin, J. R., Zoccolillo, M., Zelazo, P. D., Boivin, M., Perusse, d., & Japel, C. (2004). Physical aggression during early childhood: trajectories and predictors. Pediatrics. 114(1), 43-50.
Turkoglu, B. (2019). Preschool teachers' perspectives on aggressive behaviors in children: A Qualitative study. Journal of Education and Training Studies, 7(2), 169-183.
Tzoumakis, S., Lussier, P., & Corrado, R. R. (2014). The persistence of early childhood physical aggression: Examining maternal delinquency and offending, mental health, and cultural differences. Journal of Criminal Justice, 42(5), 408–420.
Uysal, H., & Dincer, C. (2013). An evaluation of physical and relational aggression during preschool period in relation to some variables. Education and Science, 38(169), 328-345.
Uz, R., & Bayraktar, M. (2019). Bullying toward teachers and classroom management skills. European Journal of Educational Research, 8(5), 647-657. https://doi.org/10.12973/eu-jer.8.2.647
Valles, N-L. (2012). The moderating role of child temperament in the relation between harsh and deficient parenting and child aggressive behaviors (Unpublished doctoral dissertation). University of Iowa, lowa, England.
Varburton, W.A., & Anderson, C. A. (2015). Social psychology of aggression. In J. Wright & J. Berry (Eds.), International encyclopedia of social and behavioral sciences (pp. 373-380). Elsevier.
Vasiliky, P. (2016). The relationship between teacher responsiveness and the engagement of children with aggressive behavior in preschool: A systematic literature review from 2000-2016 (Unpublished master’s thesis). Jonkoping University, Smaland, Sweden.
Werner, N. E., Senich, S., & Przepyszny, K. A. (2006). Mothers’ responses to preschoolers’ relational and physical aggression. Applied Developmental Psychology, 27(3), 193-208. https://doi.org/10.1016/j.appdev.2006.02.002
Weyns, T., Verschueren, K., Leflot, G., Onghena, P., Wouters, S., & Colpin, H. (2017). The role of teacher behavior in children's relational aggression development: A five-wave longitudinal study. Journal of School Psychology, 64(1), 17-27. https://doi.org/10.1016/j.jsp.2017.04.008
Works, D. M. (2015). Teachers' experiences concerning the rise in student aggression (Unpublished doctoral dissertation). Walden University, Minnesota, USA.
Yildirim, A., & Simsek, H. (2016). Sosyal bilimlerde nitel arastirma [Qualitative research in social sciences]. Seckin Publishing.
Zhang, Q., Tian, J., Cao, J., Zhang, D. J., & Rodkin, P. (2016). Exposure to weapon pictures and subsequent aggression during adolescence. Personality and Individual Differences, 90(1), 113-118. https://doi.org/10.1016/j.paid.2015.09.017