Peer Observation Tool Reliability Procedure in Higher Education (He-PoT)
DOI:
https://doi.org/10.47197/retos.v82.119089Keywords:
Reliability analysis, higher education, observation tools, observational methodology, peer observationAbstract
Introduction: Peer Observation of Teaching (POT) has gained increasing prominence in Higher Education institutions worldwide. However, there is a paucity of instruments that have undergone rigorous validation and reliability testing.
Objective: To analyze the reliability of the POT-Higher Education (HE-PoT) tool for its subsequent application in enhancing university teaching through a collaborative approach.
Methodology: A nomothetic, point, and multidimensional observational de-sign was employed to analyze 50 teaching sessions using the validated HE-PoT tool and Lince Plus software. The study involved three university instructors acting as observers and four faculty members from the Faculty of Sport Sciences being observed. Reliability was ensured through expert training and assessed using Cohen’s Kappa coefficient (intra- and inter-observer), alongside a generalizability Theory analysis of the observers, items, and sessions facets.
Results: The tool demonstrated excellent inter-observer reliability (mean=0.867) and high generalizability for the ites facet (e2=0.876). Findings highlight that the greatest variance (61.2%) resides in the item-session interaction, confirming observer stability despite the inherent heterogeneity of pedagogical behaviors.
Conclusions: The levels of reliability and generalizability indicate that the HE-PoT tool is a reliable instrument for peer observation among university faculty.
References
Álvarez Medina, J., Murillo Lorente, V., Casterad Seral, J., & Nuviala, A. (2022). Valoración del aprendi-zaje técnico del voleibol mediante la metodología observacional por pares en estudiantes universitarios. Cultura, Ciencia y Deporte, 17(51),31-39. http://dx.doi.org/10.12800/ccd.v17i51.1662
Amrein-Beardsley, A., & Popp, S. E. O. (2012). Peer observations among faculty in a college of educa-tion: Investigating the summative and formative uses of the Reformed Teaching Observation Protocol (RTOP). Educational Assessment, Evaluation and Accountability, 24(1), 5-24. https://doi.org/10.1007/s11092-011-9135-1
Anguera-Argilaga, M. T., Blanco Villaseñor, Á., Hernández Mendo, A., & Losada López, J. L. (2011). Dise-ños Observacionales: Ajuste y aplicación en psicología del deporte. Cuadernos de Psicología del Deporte, 11(2), 63–76. https://revistas.um.es/cpd/article/view/133241
Aránguiz, C., Rivera-Vargas, P. y Imbernón, F. (2021). A una década del Plan Bolonia: posibilidades y límites de su implementación en la Universidad de Barcelona. Revista de la educación superior, 50(200), 85-104. https://doi.org/10.36857/resu.2021.200.1891
Aznar Díaz, I., Hinojo-Lucena, F. J., Cáceres-Reche, M. P. y Romero-Rodríguez, J. M. (2020). Pedagogical Approaches in the Knowledge Society: The Flipped Classroom Method for the Development of Creativity and Dialogical Learning. International Journal of Emerging Technologies in Learning, 15(3), 4-14. http://dx.doi.org/10.3991/ijet.v15i03.11664
Bailey, K.M. (2006). Language Teacher Supervision: A Case-Based Approach. Cambridge University Press.
Bell, M. (2001). Supported reflective practice: A programme of peer observation and feedback for aca-demic teaching development. International Journal for Academic Development, 6(1), 29–39. https://doi.org/10.1080/13601440110033643
Brancato, V. C. (2003). Professional development in higher education. New Directions for Adult and Continuing Education, 98, 59-65. https://doi.org/10.1002/ace.100
Burk, K. S., Zhao, A. H., Glazer, D. I., Giess, C. S., Boland, G. W., & Khorasani, R. (2022). Assessment of a large-scale peer learning program’s value by manual review of case submissions. Journal of the American College of Radiology, 19(10), 1138-1150. https://doi.org/10.1016/j.jacr.2022.04.012
Byrne, J., Brown, H., & Challen, D. (2010). Peer development as an alternative to peer observation: A tool to enhance professional development. International Journal for Academic Development, 15(3), 215-228. https://doi.org/10.1080/1360144X.2010.497685
Carragher, J., & McGaughey, J. (2016). The effectiveness of peer mentoring in promoting a positive tran-sition to higher education for first-year undergraduate students: A mixed methods systematic review protocol. Systematic Reviews, 5(1), Article 68. https://doi.org/10.1186/s13643-016-0245-1
Chao, T., Saj, T., & Tessier, F. (2006). Establishing a quality review for online courses. Educause Quar-terly, 29(3), 32–39.
Chism, N. (2007). Peer review of teaching: A sourcebook. Anker.
Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological meas-urement, 20(1), 37-46. https://doi.org/10.1177/001316446002000104
Corcelles-Seuba M., Ortiz M., O’Leary M., & Duran D. (2025). A systematic review of reciprocal peer observation in higher education. Professional Development in Education, 1–18. https://doi.org/10.1080/19415257.2025.2480768
Corcelles-Seuba, M., Duran, D., Flores, M., Miquel, E., y Ribosa, J. (2023). Percepciones docentes sobre observación entre iguales: resistencias, agencia, procedimiento y objetivos de mejora. Estudios sobre Educación, 44, 35-58. https://doi.org/10.15581/004.44.002
Crabtree, J. L., Scott, P. J., & Kuo, F. (2016). Peer Observation and Evaluation Tool (POET): A Formative Peer Review Supporting Scholarly Teaching. The Open Journal of Occupational Therapy, 4(3). https://doi.org/10.15453/2168-6408.1273
Cronbach, L. J. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. Wiley.
De la Iglesia, B., Duma, L., Forteza, D., Llabrés, J., Moreno-Tallón, F. y Ribosa, J. (2025). El feedback entre docentes en un ciclo de observación entre iguales. REICE. Revista Iberoamericana sobre Cali-dad, Eficacia y Cambio en Educación, 23(3). https://doi.org/10.15366/reice2025.23.3.007
Domínguez-Montes, J.A., Cenizo-Benjumea, J.M., Gálvez-González, J. & Otero-Saborido, F. (2024). De-sign and validation of a university education peer observation tool. Revista de Gestão Social e Ambiental, 18(10), 1-19. https://doi.org/10.24857/rgsa.v18n10-301
Eddy, S., Converse, M., & Wenderoth, M. (2015). PORTAAL: A classroom observation tool assessing evi-dence-based teaching practices for active learning in large science, technology, engineering, and mathematics classes. CBE Life Sciences Education, 14(2), es2. https://doi.org/10.1187/cbe.14-06-0095
Escartí, A., Gutiérrez, M., Pascual, C., & Wright, P. (2013). Observación de las estrategias que emplean los profesores de educación física para enseñar responsabilidad personal y social. Revista de Psicologia del De¬porte, 22(1), 159-166.
Fletcher, J. A. (2018). Peer observation of teaching: A practical tool in higher education. The Journal of Faculty Development, 32(1), 51-64. https://doi.org/10.13140/RG.2.2.19455.82084
Flores, M., Miquel, E., & Duran, D. (2025). Teacher collaborative knowledge buildingin Reciprocal Peer Observation. European Journal of Psychology of Education, 40(1), 45. https://doi.org/10.1007/s10212-025-00945-7
Flores-Ferro, E., Maureira Cid, F., Castillo-Retamal, F., Matus-Castillo, C., Gamboa Jimenez, R., Moral Mo-reno, L., Muñoz Lara, M., Portugal Villar, J., & Saavedra Esteban, E. (2025). Construcción y vali-dación la Escala de Percepción de la Investigación Universitaria (EPIU) en estudiantes de Edu-cación Física de Chile. Retos, 73, 66-77. https://doi.org/10.47197/retos.v73.116575
Gabín, B., Camerino, O., Anguera, M.T. & Castañer, M. (2012). Lince: Multiplatform sport analysis soft-ware. ProcediaSocial and Behavioral Sciences, 46, 4692-4694. https://doi.org/10.1016/j.sbspro.2012.06.320
Gallardo-Fuentes, F., Carter Thuillier, B., Martínez-Angulo, C., Gallardo-Fuentes, J., & Peña Troncoso, S. (2025). Percepciones del alumnado en dos cohortes de la formación inicial en Educación Física: consistencia, compromiso y retos en la evaluación. Retos, 70, 39–50. https://doi.org/10.47197/retos.v70.113463
Garcia, I., James, R. W., Bischof, P., & Baroffio, A. (2017). Self-Observation and Peer Feedback as a Faculty Development Approach for Problem-Based Learning Tutors: A Program Evaluation. Teaching and Learning in Medicine, 29(3), 313-325. https://doi.org/10.1080/10401334.2017.1279056
Georgiou, H., Sharma, M., & Ling, A. (2018). Peer review of teaching: What features matter? A case study within STEM faculties. Innovations in Education and Teaching International, 55(2), 190–200. https://doi.org/10.1080/14703297.2017.1342557
Gosling, D. (2002). Models of Peer Review of Teaching. Generic Centre: Learning and Teaching Support Network, University of East London.
Hammersley-Fletcher, L., y Orsmond, P. (2005). Reflecting on reflective practices within peer observa-tion. Studies in Higher Education, 30(2), 213-224. https://doi.org/10.1080/03075070500043358
Hashim, N., & Ab Wahid, H. (2024). Entrepreneurial Mentoring Synergy: An In-Depth Analysis of Poly-technic and Community College (PolyCC) Lecturers as Mentors to Student Start-Up Entrepre-neurs. PaperASIA, 40(4b), 239–247. https://doi.org/10.59953/paperasia.v40i4b.206
Hernández-Mendo, A., Blanco-Villaseñor, Á., Pastrana, J. L., Morales- Sánchez, V., & Francisco Jesús Ra-mos-Pérez. (2016). SAGT: Aplica¬ción Informática para Análisis de Generalizabilidad. Revista Iberoame¬ricana de Psicologia del Ejercicio y el Deporte, 11(1), 77-89.
Huston, T., & Weaver, C.L. (2008). Peer coaching: Professional development for experienced faculty. Innovative Higher Education, 33(1), 5-20. https://doi.org/10.1007/s10755-007-9061-9
Ibarra-Sáiz, M. S., Rodríguez-Gómez, G., Boud, D., Rotsaert, T., Brown, S., Salinas Salazar, M. L., & Rodríguez Gómez, H. M. (2020). El futuro de la evaluación en la educación superior. RELIEVE - Revista Electrónica De Investigación Y Evaluación Educativa, 26(1). https://doi.org/10.7203/relieve.26.1.17323
Jia, Y., Spagnolo, A. B., Barrett, N., Murphy, A. A., Basto, P. M., Rothpletz-Puglia, P., & Luther, S. (2025). Validation of a peer observation and evaluation tool for online teaching in the US. Educational technology research and development, 73(1), 615-639. https://doi.org/10.1007/s11423-024-10428-z
Johnston, A. L., Baik, C., & Chester, A. (2022). Peer review of teaching in Australian higher education: a systematic review. Higher Education Research and Development, 41(2), 390-404. https://doi.org/10.1080/07294360.2020.1845124
Kojima, M., Kinomura, Y., & Kuzuhara, K. (2021). Development of observational indicators for evaluat-ing handstand posture in the mat exercise in physical education class: Validity and reliability. Journal of Physical Education and Sport, 21(3), 2087–2096. https://doi.org/10.7752/jpes.2021.s3266
Lachowsky NJ, Card KG, Cui Z, Sereda P, Roth EA, Hogg RS, Moore DM. (2019) Agreement between gay, bisexual and other men who have sex with men’s period prevalence and event-level recall of sexual behaviour: an observational respondent-driven sampling study. Sexual Health 16, 84–87. https://doi.org/10.1071/SH17223
Landis, J. R., y Koch, G. G. (1977). A one-way components of variance model for categorical data. Bio-metrics, 33, 671-679. https://doi.org/10.2307/2529465
Ley Orgánica 2/2023, de 22 de marzo, del Sistema Universitario. Boletín Oficial del Estado, núm. 70, de 23 de marzo de 2023, pp. 43267 a 43339. https://www.boe.es/eli/es/lo/2023/03/22/2/con
Little, B. B. (2009). Quality assurance for online nursing courses. Journal of Nursing Education, 48(7),381–387. https://doi.org/10.3928/01484834-20090615-05
Matarranz, M. (2021). El Espacio Europeo de Educación Superior y su sello de calidad. Revista española de educación comparada, 37, 153-1573. https://doi.org/10.5944/reec.37.2021.27728
McGahan, S. J., Jackson, C. M., & Premer, K. (2015). Online course quality assurance: Development of a quality checklist. InSight: A Journal of Scholarly Teaching, 10, 126–140. https://doi.org/10.46504/10201510mc
McNulty, C. A., Syeda, R. B., Brown, C. L., Bennett, C. V., Schofield, B., Allison, D. G., ... & Francis, N. (2020). Peer-education as a tool to educate on antibiotics, resistance and use in 16–18-year-olds: a fea-sibility study. Antibiotics, 9(4), 146. https://doi.org/10.3390/antibiotics9040146
Menges, R. J. (1987). Colleagues as catalysts for change in teaching. In Kurfiss, J. (Ed.), To improve the academy: Resources for faculty, in¬structional and organizational development, 6 (pp. 83–93). Stillwater, OK: New Forums Press.
OECD (2019). TALIS 2018 Results (Volume I): Teachers and school leaders as lifelong learners. OECD Publishing. https://www.oecd.org/education/talis-2018-re-sults-volume-i-1d0bc92a-en.htm
Otero-Saborido, F., Domínguez-Montes, J.A., Cenizo-Benjumea, J.M. & González-Calvo, G. (2024). Peer observation of teaching in higher education: Systematic review of observation tools. Educa-tional Process: International Journal, 13(1), 84-101. https://doi.org/10.22521/edupij.2024.131.6
Peel, D. (2005). Peer observation as a transformatory tool? Teaching in Higher Education, 10(4), 489-504. https://doi.org/10.1080/13562510500239125
Portell, M., Anguera, M. T., Chacón-Moscoso, S., & Sanduvete-Chaves, S. (2015). Guidelines for reporting evaluations based on observational methodology. Psicothema, 27(3), 283-289. https://doi.org/10.7334/ psicothema2014.276
Rabada-Rice, F., & Scott, R. S. (1986). A peer evaluation for measuring team teaching effectiveness. The Journal of Nursing Education, 25(6), 255-258. https://doi.org/10.3928/0148-4834-19860601-10
Richards, J. C., & Farrell, T. S. C. (2005). Professional development for language teachers: Strategies for teacher learning. Cambridge University Press.
Ridge, B. L., & Lavigne, A. L. (2020). Improving instructional practice through peer observation and feedback. Education Policy Analysis Archives, 28(61). https://doi.org/10.14507/EPAA.28.5023
Sachs, J., & Parsell, M. (2014). Peer review of learning and teaching in higher education: International perspectives. Springer.
Shortland, S. (2010). Feedback within peer observation: continuing professional development and un-expected consequences. Innovations in Education and Teaching International, 47(3), 295–304. https://doi.org/10.1080/14703297.2010.498181
Skinner, M. E., & Welch, F. C. (1996). Peer coaching for better teaching. College Teaching, 44, 153–156.
Smith, M. K., Jones, F. H., Gilbert, S. L., & Wieman, C. E. (2013). The classroom observation protocol for undergraduate STEM (COPUS): A new instrument to characterize university STEM classroom practices. CBE—Life Sciences Education, 12, 618–627. https://doi.org/10.1187/cbe.13-08-0154
Soto, A., Camerino, O., Iglesias, X., Anguera, M.T. y Castañer, M. (2019). LINCE PLUS: Software de inves-tigación para análisis de vídeo del comportamiento. Apunts. Educación Física y Deportes, 137, 149-153. https://dx.doi.org/10.5672/apunts.2014-0983.es.(2019/3).137.11
Soto-Fernández, A., Camerino, O., Iglesias, X., Anguera, M. T., & Castañer, M. (2022). LINCE PLUS soft-ware for systematic observational studies in sports and health. Behavior Research Methods, 54, 1263–1271. https://doi.org/10.3758/s13428-021-01642-1
Stone, A. A., & Shiffman, S. (2002). Capturing momentary, self-report data: A proposal for reporting guidelines. Annals of Behavioral Medici¬ne, 24(3), 236-243. https://doi.org/10.1207/S15324796ABM2403_09
Strahl A, Gerlich C, Alpers GW, Ehrmann K, Gehrke J, Müller-Garnn A, & Vogel H. (2018). Development and evaluation of a standardized peer-training in the context of peer review for quality assur-ance in work capacity evaluation. BMC Med Educ, 18(1), 135. https://doi.org/10.1186/s12909-018-1233-z
Sullivan, P. B., Buckle, A., Nicky, G., & Atkinson, S. H. (2012). Peer observation of teaching as a faculty development tool. BMC Medical Education, 12, 26. https://doi.org/10.1186/1472-6920-12-26
Thomas, S., Chie, Q. T., Abraham, M., Jalarajan Raj, S., & Beh, L. S. (2014). A qualitative review of litera-ture on peer review of teaching in higher education: An application of the SWOT framework. Review of Educational Research, 84(1), 112–159. https://doi.org/10.3102/0034654313499617
Torres, A. C., Lopes, A., Valente, J. M. S., & Mouraz, A. (2017). What catches the eye in class observation? Observers’ perspectives in a multidisciplinary peer observation of teaching program. Teaching in Higher Education, 22(7), 822-838. https://doi.org/10.1080/13562517.2017.1301907
Trujillo, J. M., DiVall, M. V., Barr, J., Gonyeau, M., Van Amburgh, J. A., Matthews, S. J., & Qualters, D. (2008). Development of a peer teaching-assessment program and a peer observation and eval-uation tool. American Journal of Pharmaceutical Education. https://doi.org/10.1016/S0002-9459(24)00511-4
Var, E., Cam, Y., Pektas, E., & Kuden, C. (2025). Does preoperative anxiety influence endodontic skill performance and students’ subjective perceptions in an observational simulation-based study? BMC Medical Education, 25(1), 1382. https://doi.org/10.1186/s12909-025-07957-3
Yiend, J., Weller, S., & Kinchin, I. (2014). Peer observation of teaching: The interaction between peer review and developmental models of practice. Journal of Further and Higher Education, 38(4), 465–484. https://doi.org/10.1080/0309877X.2012.726967
Downloads
Published
Issue
Section
License
Copyright (c) 2026 José Antonio Domínguez-Montes , Javier Gálvez-González , Roos Van Gasse, Fernando Manuel Otero-Saborido
This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and ensure the magazine the right to be the first publication of the work as licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgment of authorship of the work and the initial publication in this magazine.
- Authors can establish separate additional agreements for non-exclusive distribution of the version of the work published in the journal (eg, to an institutional repository or publish it in a book), with an acknowledgment of its initial publication in this journal.
- Is allowed and authors are encouraged to disseminate their work electronically (eg, in institutional repositories or on their own website) prior to and during the submission process, as it can lead to productive exchanges, as well as to a subpoena more Early and more of published work (See The Effect of Open Access) (in English).
This journal provides immediate open access to its content (BOAI, http://legacy.earlham.edu/~peters/fos/boaifaq.htm#openaccess) on the principle that making research freely available to the public supports a greater global exchange of knowledge. The authors may download the papers from the journal website, or will be provided with the PDF version of the article via e-mail.
