Use and acceptance of automatic evaluation: study of translation and reliability of an instrument to measure it
DOI:
https://doi.org/10.33010/ie_rie_rediech.v15i0.1923Keywords:
Influence of technology, college students, computer assisted testing, student attitudes, automationAbstract
The rapid development of Information and Communication Technologies (ICT) has opened new possibilities for teaching and assessment practices in higher education. This quantitative study, with a quasi-experimental design, examines the reliability of the Spanish version of a scale for evaluating the acceptance of Computer-Based Assessment (CBA) among university students. To conduct the research, descriptive analysis tools, such as Cronbach’s alpha coefficient, were used to evaluate the reliability of the Computer-Based Assessment Acceptance Model (CBAAM) in a private University in Mexico. The research was divided into two phases: 1) application of an assessment with automatically graded short-answer questions, and 2) analysis of the reliability of the final version of the CBAAM with 84 first-semester engineering students. The CBAAM showed a high and statistically significant internal consistency coefficient (Cronbach’s alpha = 0.9190). The internal consistency coefficients of the subscales ranged from 0.6838 to 0.9325, suggesting that the scale is reliable for use in this context. It is recommended to apply the CBAAM in other contexts for further confirmation.
References
Alexiou-Ray, J., y Bentley, C. C. (2015). Faculty professional development for quality online teaching. Online Journal of Distance Learning Administration, 18(4), 1-7. https://ojdla.com/archive/winter184/ray_bentley184.pdf
Al-Qdah, M., y Ababneh, I. (2017). Comparing online and paper exams: Performances and perceptions of Saudi students. International Journal of Information and Education Technology, 7(2), 106-109. https://doi.org/10.18178/ijiet.2017.7.2.850
Alruwais, N., Wills, G., y Wald, M. (2018). Advantages and challenges of using e-assessment. International Journal of Information and Education Technology, 8(1), 34-37. https://doi.org/10.18178/ijiet.2018.8.1.1008
Bralić, A., y Divjak, B. (2018). Integrating MOOCs in traditionally taught courses: Achieving learning outcomes with blended learning. International Journal of Educational Technology in Higher Education, 15(1). https://doi.org/10.1186/s41239-017-0085-7
Bukie, O. F. (2014). Understanding technologies for e-Assessment: A systematic review approach. Journal of Emerging Trends in Computing and Information Sciences, 5(12). http://www.cisjournal.org
Burrows, S., Gurevych, I., y Stein, B. (2015). The eras and trends of automatic short answer grading. International Journal of Artificial Intelligence in Education, 25, 60-117. https://doi.org/10.1007/s40593-014-0026-8
Casanova, M. A. (2020). Evaluación formativa e inclusiva en línea: ¿Cómo evaluar el aprendizaje ante los retos de la pandemia por el Covid 19? . Memorias del Quinto Congreso Internacional de Evaluación Socioformativa (Valora-2020). Centro Universitario CIFE, Cuernavaca, México. https://cife.edu.mx/recursos
Chau, K. Y., Law, K. M., y Tang, Y. M. (2021). Impact of self-directed learning and educational technology readiness on synchronous e-Learning. Journal of Organizational and End User Computing (JOEUC), 33(6), 1-20. https://doi.org/10.4018/JOEUC.20211101.oa26
Curran, P. J., West, S. G., y Finch, J. F. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis. Psychological Methods, 1(1), 16-29. https://doi.org/10.1037/1082-989X.1.1.16
Davis, F. (1989). Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quarterly, 13(3),319-340. https://doi.org/10.2307/249008
Davis, F. D. (2011). Foreword in technology acceptance in education: Research and issues. Sense Publishers.
Davis, F. D., Bagozzi, R. P., y Warshaw, P. R. (1989). User acceptance of computer technology: A comparison of two theoretical models. Management Science, 35(8), 982-1003. https://doi.org/10.1287/mnsc.35.8.982
Davis, M. M., Spohrer, J. C., y Maglio, P. P. (2011). Guest editorial: How technology is changing the design and delivery of services. Operations Management Research, 4, 1-5. https://doi.org/10.1007/s12063-011-0046-6
De la Oliva, D. (2020). La autoevaluación en las escuelas de educación primaria para la mejora de la calidad educativa y de la inclusión social. Memorias del Quinto Congreso Internacional de Evaluación Socioformativa (Valora-2020). https://cife.edu.mx/recursos/
Gañán, D., Caballé, S., Clarisó, R., Conesa, J., y Bañeres, D. (2017). ICT-FLAG: A web-based e-assessment platform featuring learning analytics and gamification. International Journal of Web Information Systems, 13(1), 25-54. https://doi.org/10.1108/IJWIS-12-2016-0074
Granic, A., y Marangunic, N. (2019). Technology acceptance model in educational context: A systematic literature review. British Journal of Educational Technology, 50(5), 2572-2593. https://doi.org/10.1111/bjet.12864
Hestenes, D., Wells, M., y Swackhamer, G. (1992). Force concept inventory. The Physics Teacher, 30(3), 141-158. https://doi.org/10.1119/1.2343497
Imtiaz, A., y Maarop, N. (2014). A review of technology acceptance studies in the field of education. Jurnal Teknologi, 69(2), 2180-3722. https://doi.org/doi:10.11113/jt.v69.3101
Jeong, H. (2011). An investigation of user perceptions and behavioral intentions towards the e-library. Library Collections, Acquisitions, and Technical Services, 35(2-3), 45-60. https://doi.org/10.1080/14649055.2011.10766298
JISC [Joint Information Systems Committee] (2007). Effective practice with e-assessment: An overview of technologies, policies and practice in further and higher education. https://people.cs.vt.edu/shaffer/cs6604/Papers/eAssessment.pdf
Joo, S., y Choi, N. (2015). Factors affecting undergraduates’ selection of online library resources in academic tasks usefulness, ease-of-use, resource quality, and individual differences. Library Hi Tech, 33(2), 272-291. https://doi.org/10.1108/LHT-01-2015-0008
Jordan, S. (2013). E-assessment: Past, present and future. New Directions, 9(1), 87-106. https://oro.open.ac.uk/38536/
Jordan, S., y Mitchell, T. (2009). e-Assessment for learning? The potential of short-answer free-text questions with tailored feedback. British Journal of Educational Technology, 40(2), 371-385. https://doi.org/10.1111/j.1467-8535.2008.00928.x
Kundu, A., y Bej, T. (2021). Experiencing e-assessment during COVID-19: an analysis of Indian students’ perception. Higher Education Evaluation and Development, 15(2), 114-134. https://doi.org/10.1108/heed-03-2021-0032
Llamas-Nistal, M., Fernández-Iglesias, M. J., González-Tato, J., y Mikic-Fonte, F. A. (2013). Blended e-assessment: Migrating classical exams to the digital world. Computers & Education, 62, 72-87. https://doi.org/10.1016/j.compedu.2012.10.021
Mac Callum, K., Jeffrey, L., y Kinshuk (2014). Comparing the role of ICT literacy and anxiety in the adoption of mobile learning. Computers in Human Behavior, 39, 8-19. https://doi.org/10.1016/j.chb.2014.05.024
Madrid, G., Angulo, J., García, L., y Olivares, C. (2016). Estado del conocimiento en la educación mediada por TIC en México. CIATA.
Marangunic, N., y Granic, A. (2015). Technology acceptance model: A literature review from 1986 to 2013. Universal Access in the Information Society, 14, 81-95. https://doi.org/10.1007/s10209-014-0348-1
Miller, A. (2020, abr. 7). Formative assessment in distance learning. Edutopia. https://www.edutopia.org/article/formative-assessment-distance-learning
Mo, D. Y., Tang, Y. M., Wu, E. Y., y Tang, V. (2022). Theoretical model of investigating determinants for a successful Electronic Assessment System (EAS) in higher education. Education and Information Technologies, 27(9), 12543-12566. https://doi.org/10.1007/s10639-022-11098-1
Mohammadyari, S., y Singh, H. (2015). Understanding the effect of e-learning on individual performance: The role of digital literacy. Computers & Education, 82, 11-25. https://doi.org/10.1016/j.compedu.2014.10.025
Muñiz, J., Elosua, P., y Hambleton, R. K. (2013). Directrices para la traducción y adaptación de los tests: segunda edición. Psicothema, 25(2), 151-157. https://doi.org/10.7334/psicothema2013.24
Ngai, E. W., Poon, J. K. L., y Chan, Y. H. (2007). Empirical examination of the adoption of WebCT using TAM. Computers & Education, 48(2), 250-267. https://doi.org/10.1016/j.compedu.2004.11.007
Nikou, S. A., y Economides, A. A. (2017). Mobile-based assessment: Investigating the factors that influence behavioral intention to use. Computers & Education, 109, 56-73. https://doi.org/10.1016/j.compedu.2017.02.005
Olivares, K., Angulo, A., Torres, G., y Madrid, G. (2016). Validación de un modelo de medida para la competencia digital en estudiantes universitarios. En M. E. Prieto y S. J. Pech (eds.), La tecnología como instrumento para potenciar el aprendizaje (pp. 72-78). CIATA-UCLM. https://doi.org/10.13140/RG.2.1.3802.3920
Scherer, R., Siddiq, F., y Tondeur, J. (2019). The technology acceptance model (TAM): A meta-analytic structural equation modeling approach to explaining teachers’ adoption of digital technology in education. Computers & Education, 128, 13-35. https://doi.org/10.1016/j.compedu.2018.09.009
Sousa, V. D., y Rojjanasrirat, W. (2011). Translation, adaptation and validation of instruments or scales for use in cross-cultural health care research: A clear and user-friendly guideline. Journal of Evaluation in Clinical Practice, 17(2), 268-274. https://doi.org/10.1111/j.1365-2753.2010.01434.x
Tang, Y. M., Au, K. M., Lau, H. C. W., Ho, G. T. S., y Wu, C. H. (2020). Evaluating the effectiveness of learning design with mixed reality (MR) in higher education. Virtual Reality, 24(4), 797-807. https://doi.org/10.1007/s10055-020-00427-9
Tang, Y. M., Chen, P. C., Law, K. M. Y., Wu, C. H., Lau, Y.-y., Guan, J., He, D., y Ho, G. T. S. (2021). Comparative analysis of Student’s live online learning readiness during the coronavirus (COVID-19) pandemic in the higher education sector. Computers & Education, 168, 104211. https://doi.org/10.1016/j.compedu.2021.104211
Tang, Y. M., y Yu, K. M. (2018). Development and evaluation of a mobile platform for teaching mathematics of CAD subjects. Computer-Aided Design and Applications, 15(2), 164-169. https://doi.org/10.1080/16864360.2017.1375665
Terzis, V., y Economides, A. A. (2011). The acceptance and use of computer based assessment. Computers & Education, 56(4), 1032-1044. https://doi.org/10.1016/j.compedu.2010.11.017
Terzis, V., Moridis, C. N., y Economides, A. A. (2012a). How student’s personality traits affect Computer Based Assessment Acceptance: Integrating BFI with CBAAM. Computers in Human Behavior, 28(5), 1985-1996. https://doi.org/10.1016/j.chb.2012.05.019
Terzis, V., Moridis, C. N., y Economides, A. A. (2012b). The effect of emotional feedback on behavioral intention to use computer based assessment. Computers & Education, 59(2), 710-721. https://doi.org/10.1016/j.compedu.2012.03.003
Terzis, V., Moridis, C. N., y Economides, A. A. (2013). Continuance acceptance of computer based assessment through the integration of user’s expectations and perceptions. Computers & Education, 62, 50-61. https://doi.org/10.1016/j.compedu.2012.10.018
Timmis, S., Broadfoot, P., Sutherland, R., y Oldfield, A. (2016). Rethinking assessment in a digital age: Opportunities, challenges and risks. British Educational Research Journal, 42(3), 454-476. https://doi.org/10.1002/berj.3215
Tirado-Lara, P. J., y Roque-Hernández, M. del P. (2022). Validación de la escala Uso y función de las TIC en contextos educativos para estudiantes de educación superior. Relatec Revista Latinoamericana de Tecnología Educativa, 21(1), 9-25. https://doi.org/10.17398/1695-288X.21.1.9
Venkatesh, V., Morris, M. G., Davis, G. B., y Davis, F. D. (2003). User acceptance of information technology: Toward a unified view. MIS Quarterly, 27(3), 425-478. https://doi.org/10.2307/30036540
Viramontes Anaya, E., y Viramontes Campos, O. (2021). Evaluación en educación a distancia por confinamiento. IE Revista de Investigación Educativa de la REDIECH, 12, e1216. https://doi.org/10.33010/ie_rie_rediech.v12i0.1216
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Julián Gerardo Torres Kauffman, Iván Olmos Pineda, Juan Manuel González Calleros
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
