Optimizing Manahij al-Tarjamah Learning Outcomes: A Goal-Free Evaluation of Contextual Systemic Collaboration
DOI:
https://doi.org/10.38073/lughawiyyat.v7i2.5680Keywords:
Manahij al-Tarjamah, goal-free evaluation, contextual learning, systemic collaboration, Arabic translation pedagogyAbstract
This study examines the consequences of contextual systemic collaboration in a Manahij al-Tarjamah course without using the programme's declared objectives as the initial frame of judgment. A qualitative goal-free evaluation was conducted to identify actual, unintended, positive, and constraining effects emerging from Arabic-Indonesian translation learning. The evidence corpus comprised lecturer and student interviews, classroom observations, reflective records, peer-feedback traces, and translation artefacts associated with contextual cases. ATLAS.ti was used to organise document groups, conduct open and process coding, compare code-document patterns, develop analytic memos, and construct a network of consequences. To reduce goal contamination, formal course targets were withheld during first-cycle coding and consulted only after the emergent effect map had stabilised. The analysis produced six interrelated findings. Contextual cases shifted students from method naming toward situational diagnosis; role-based collaboration made translation reasoning more visible; multi-source feedback deepened revision; digital tools broadened alternatives but also created verification dependency; reflective portfolios strengthened evaluative judgment and professional identity; and linguistic accuracy, participation equity, and fidelity-readability trade-offs remained uneven. An unintended benefit was the emergence of a shared metalanguage for defending translation decisions. Unintended burdens included documentation fatigue, time pressure, and the possibility that assigned roles became procedural rather than genuinely collaborative. The goal-free judgment is therefore conditional: contextual systemic collaboration optimised reasoning, feedback uptake, and revision more consistently than it optimised all features of final-text accuracy. The study recommends a genre-balanced case bank, rotating role audits, staged feedback, machine-translation literacy protocols, terminology calibration, and concise decision memos. It contributes a consequence-centred evaluation framework for translation-methodology courses in Arabic higher education.
Downloads
References
Bowker, Lynne. “Promoting Linguistic Diversity and Inclusion: Incorporating Machine Translation Literacy into Information Literacy Instruction for Undergraduate Students.” The International Journal of Information, Diversity, & Inclusion 5, no. 3 (2021): 127–51. doi:10.33137/ijidi.v5i3.36159.
Braun, V., and V. Clarke. “Using Thematic Analysis in Psychology.” Qualitative Research in Psychology 3, no. 2 (2006): 77–101. doi:10.1191/1478088706qp063oa.
Cahyanti, Aisyah Nur, and Abdul Malik. “Techniques and Quality Translation of Interrogative Sentence Dubbing: English-Indonesian and English-Arabic in The Cartoon Movie.” Arabiyat: Jurnal Pendidikan Bahasa Arab Dan Kebahasaaraban 10, no. 2 (2023): 256–68. doi:10.15408/a.v10i2.34197.
Carless, David, and David Boud. “The Development of Student Feedback Literacy: Enabling Uptake of Feedback.” Assessment & Evaluation in Higher Education 43, no. 8 (2018): 1315–25. doi:10.1080/02602938.2018.1463354.
Choiriyah, Nurul Aini, Muhammad Yunus Anis, Syed Nurulakla Syed Abdullah, and Nor Azuwan Yaakob. “Translation Quality of Arabic Directive Speech Acts in Pandemic’s Book: An Attempt to Delve into Covid-19.” Arabiyat: Jurnal Pendidikan Bahasa Arab Dan Kebahasaaraban 9, no. 1 (2022): 69–84. doi:10.15408/a.v9i1.25459.
Colina, Sonia. “Translation Quality Evaluation: Empirical Evidence for a Functionalist Approach.” The Translator 14, no. 1 (2008): 97–134. doi:10.1080/13556509.2008.10799251.
Crescenzi-Lanna, Lucrezia. “Literature Review of Learner-Centered Design in Digital Learning Environments.” British Journal of Educational Technology 53, no. 3 (2022): 716–35. doi:10.1111/bjet.13148.
Daems, Joke, Sonia Vandepitte, Robert J. Hartsuiker, and Lieve Macken. “Translation Methods and Experience: A Comparative Analysis of Human Translation and Post-Editing with Students and Professional Translators.” Meta 62, no. 2 (2017): 245–70. doi:10.7202/1041023ar.
Defrancq, Bart. “Interprofessional Training for Student Conference Interpreters and Students of Political Science through Joint Mock Conferences: An Assessment.” The Interpreter and Translator Trainer 16, no. 1 (2022). doi:10.1080/1750399X.2021.1919975.
Dewi, H. D. “Translation Competences for Indonesian Novice Translators.” Journal of Language and Literature 19, no. 1 (2019): 114–22. doi:10.24071/joll.v19i1.1817.
Doherty, Stephen. “The Impact of Translation Technologies on the Process and Product of Translation.” International Journal of Communication 10 (2016): 947–69.
Edidarmo, Toto, and Zainal Muttaqin. “Foreignization Strategy in the Quranic Translation by MoRA: Study of Surah Al-Baqarah.” Arabiyat: Jurnal Pendidikan Bahasa Arab Dan Kebahasaaraban 9, no. 1 (2022). doi:10.15408/a.v9i1.25207.
Ehrensberger-Dow, Maureen, Alice Delorme Benites, and Caroline Lehr. “A New Role for Translators and Trainers: MT Literacy Consultants.” The Interpreter and Translator Trainer 17, no. 3 (2023). doi:10.1080/1750399X.2023.2237328.
Fereday, Jennifer, and Eimear Muir-Cochrane. “Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development.” International Journal of Qualitative Methods 5, no. 1 (2006): 80–92. doi:10.1177/160940690600500107.
Friese, Susanne. Qualitative Data Analysis with ATLAS.Ti. London: SAGE Publications, 2019.
Godwin-Jones, Robert. “Partnering with AI: Intelligent Writing Assistance and Instructed Language Learning.” Language Learning & Technology 26, no. 2 (2022): 5–24. doi:10.10125/73474.
Hattie, J., and H. Timperley. “The Power of Feedback.” Review of Educational Research 77, no. 1 (2007): 81–112. doi:10.3102/003465430298487.
Herrington, Jan, and Ron Oliver. “An Instructional Design Framework for Authentic Learning Environments.” Educational Technology Research and Development 48, no. 3 (2000): 23–48. doi:10.1007/BF02319856.
Hmelo-Silver, Cindy E. “Problem-Based Learning: What and How Do Students Learn?” Educational Psychology Review 16, no. 3 (2004): 235–66. doi:10.1023/B:EDPR.0000034022.16470.f3.
Hurtado Albir, Amparo. “The Acquisition of Translation Competence: Competences, Tasks, and Assessment in Translator Training.” Meta 60, no. 2 (2015): 256–80. doi:10.7202/1032857ar.
Johnson, David W., and Roger T. Johnson. “An Educational Psychology Success Story: Social Interdependence Theory and Cooperative Learning.” Educational Researcher 38, no. 5 (2009): 365–79. doi:10.3102/0013189X09339057.
Jonsson, Anders, and Gunilla Svingby. “The Use of Scoring Rubrics: Reliability, Validity and Educational Consequences.” Educational Research Review 2, no. 2 (2007): 130–44. doi:10.1016/j.edurev.2007.05.002.
Kasneci, Enkelejda, Kathrin Sessler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, et al. “ChatGPT for Good? On Opportunities and Challenges of Large Language Models for Education.” Learning and Individual Differences 103 (2023): 102274. doi:10.1016/j.lindif.2023.102274.
Kelly, Dorothy. “Training the Trainers: Towards a Description of Translator Trainer Competence and Training Needs Analysis.” The Interpreter and Translator Trainer 2, no. 1 (2008): 99–125. doi:10.1080/1750399X.2008.10798769.
Kenny, Dorothy, and Stephen Doherty. “Statistical Machine Translation in the Translation Curriculum: Overcoming Obstacles and Empowering Translators.” The Interpreter and Translator Trainer 8, no. 2 (2014): 276–94. doi:10.1080/1750399X.2014.936112.
Kenny, John D., and Conor Cirkony. “Using Systems Perspectives to Develop Underlying Principles for Educational Reform.” Australian Journal of Teacher Education 47, no. 1 (2022): 80–99. doi:10.14221/ajte.2022v47n1.6.
Kiraly, Don. A Social Constructivist Approach to Translator Education: Empowerment from Theory to Practice. Manchester: St. Jerome Publishing, 2000.
Kohnke, Lucas, Benjamin Luke Moorhouse, and Di Zou. “ChatGPT for Language Teaching and Learning.” RELC Journal 54, no. 2 (2023): 537–50. doi:10.1177/00336882231162868.
Kuswanto, Welly. “Implementation of English Learning at Dalwa Islamic Boarding School in The 5.0 Era For Generation Z.” Jurnal Pendidikan Islam 13, no. 1 (2023): 93–109.
Lee, Sang-Min. “The Effectiveness of Machine Translation in Foreign Language Education: A Systematic Review and Meta-Analysis.” Computer Assisted Language Learning 36, nos. 1–2 (2023): 103–25. doi:10.1080/09588221.2021.1901744.
Massey, Gary, Elsa Huertas-Barros, and David Katan. “Meeting Evolution with Innovation: An Introduction to (Re-)Profiling T&I Education.” The Interpreter and Translator Trainer 17, no. 3 (2023). doi:10.1080/1750399X.2023.2237321.
Mellinger, Christopher D. “Translators and Machine Translation: Knowledge and Skills Gaps in Translator Pedagogy.” The Interpreter and Translator Trainer 11, no. 4 (2017): 280–93. doi:10.1080/1750399X.2017.1359760.
Molina, Lucía, and Amparo Hurtado Albir. “Translation Techniques Revisited: A Dynamic and Functionalist Approach.” Meta 47, no. 4 (2002): 498–512. doi:10.7202/008033ar.
Moorkens, Joss. “What to Expect from Neural Machine Translation: A Practical In-Class Translation Evaluation Exercise.” The Interpreter and Translator Trainer 12, no. 4 (2018): 375–87. doi:10.1080/1750399X.2018.1501639.
Neves, Josélia. “Project-Based Learning for the Development of Social Transformative Competence in Socially Engaged Translators.” The Interpreter and Translator Trainer 16, no. 4 (2022): 465–83. doi:10.1080/1750399X.2022.2084258.
Nicol, David. “The Power of Internal Feedback: Exploiting Natural Comparison Processes.” Assessment & Evaluation in Higher Education 46, no. 5 (2021): 756–78. doi:10.1080/02602938.2020.1823314.
Niño, Ana. “Machine Translation in Foreign Language Learning: Language Learners’ and Tutors’ Perceptions of Its Advantages and Disadvantages.” ReCALL 21, no. 2 (2009): 241–58. doi:10.1017/S0958344009000172.
Nowell, L. S., J. M. Norris, D. E. White, and N. J. Moules. “Thematic Analysis: Striving to Meet the Trustworthiness Criteria.” International Journal of Qualitative Methods 16, no. 1 (2017): 1–13. doi:10.1177/1609406917733847.
O’Brien, Sharon. “Pauses as Indicators of Cognitive Effort in Post-Editing Machine Translation Output.” Across Languages and Cultures 7, no. 1 (2006): 1–21. doi:10.1556/Acr.7.2006.1.1.
Öner Bulut, Senem, and Nilüfer Alimen. “Translator Education as a Collaborative Quest for Insights into the Re-Positioning of the Human Translator (Educator) in the Age of Machine Translation: The Results of a Learning Experiment.” The Interpreter and Translator Trainer 17, no. 3 (2023): 375–92. doi:10.1080/1750399X.2023.2237837.
PACTE Group. “First Results of PACTE Group’s Experimental Research on Translation Competence Acquisition: The Acquisition of Declarative Knowledge of Translation.” MonTI, no. Special Issue 1 (2014): 85–115. doi:10.6035/MonTI.2014.ne1.2.
Panadero, E., and A. Jonsson. “The Use of Scoring Rubrics for Formative Assessment Purposes Revisited: A Review.” Educational Research Review 9 (2013): 129–44. doi:10.1016/j.edurev.2013.01.002.
Patton, Michael Quinn. Qualitative Research & Evaluation Methods: Integrating Theory and Practice. 4th ed. SAGE Publications, 2015. https://us.sagepub.com/en-us/nam/qualitative-research-evaluation-methods/book232962.
Pym, Anthony. “Redefining Translation Competence in an Electronic Age: In Defence of a Minimalist Approach.” Meta 48, no. 4 (2003): 481–97. doi:10.7202/008533ar.
Risnawaty, Milisi Sembiring, Ihsan Fadilah, Henni Subagiarti, and Annim Hasibuan. “Macrostructure Analysis of Indonesian-Translation and Source Texts of the World Health Organization’s Covid-19 Pandemic Guidance.” Indonesian Journal of Applied Linguistics 12, no. 2 (2022): 533–44. doi:10.17509/ijal.v12i2.43858.
Rogat, Toni K., Cindy E. Hmelo-Silver, Brittany H. Cheng, Alyson Traynor, Temitope Adeoye, Andrea Gomoll, and Brian Downing. “A Multidimensional Framework of Groups’ Productive Disciplinary Engagement.” Frontline Learning Research 10, no. 2 (2022): 1–21. doi:10.14786/flr.v10i2.874.
Saleh, Asmalina, T. M. Phillips, Cindy E. Hmelo-Silver, Krista D. Glazewski, Bradford W. Mott, and James C. Lester. “A Learning Analytics Approach towards Understanding Collaborative Inquiry in a Problem-Based Learning Environment.” British Journal of Educational Technology 53, no. 5 (2022): 1321–42. doi:10.1111/bjet.13198.
Sarti, Gabriele, Arianna Bisazza, Ana Guerberof-Arenas, and Antonio Toral. “DivEMT: Neural Machine Translation Post-Editing Effort across Typologically Diverse Languages.” Transactions of the Association for Computational Linguistics 10 (2022): 1202–19. doi:10.1162/tacl_a_00518.
Scriven, Michael. “Pros and Cons about Goal-Free Evaluation.” Evaluation Comment 3, no. 4 (1972): 1–7.
Stake, Robert E. The Art of Case Study Research. Thousand Oaks, CA: SAGE Publications, 1995.
Stufflebeam, Daniel L. “The CIPP Model for Evaluation.” In Evaluation Models: Viewpoints on Educational and Human Services Evaluation, 279–317. Springer, 2000.
Stufflebeam, Daniel L., and Chris L. S. Coryn. Evaluation Theory, Models, and Applications. San Francisco: Jossey-Bass, 2014.
Terribile, Silvia. “Is Post-Editing Really Faster than Human Translation?” Translation Spaces, 2023. doi:10.1075/ts.22044.ter.
Tracy, Sarah J. “Qualitative Quality: Eight Big-Tent Criteria for Excellent Qualitative Research.” Qualitative Inquiry 16, no. 10 (2010): 837–51. doi:10.1177/1077800410383121.
Woods, Megan, Rani Macklin, and Gemma K. Lewis. “Researcher Reflexivity: Exploring the Impacts of CAQDAS Use.” International Journal of Social Research Methodology 19, no. 4 (2016): 385–403. doi:10.1080/13645579.2015.1023964.
Youker, Brandon W., and Allyssa Ingraham. “Goal-Free Evaluation: An Orientation for Foundations’ Evaluations.” The Foundation Review 6, no. 4 (2014): 79–92. doi:10.9707/1944-5660.1229.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Welly Kuswanto, Musleh Harry, Ade Agung Prayoga
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
