Tracking improvement from statistical to neural machine translation: An error-based evaluation of google translate
Main Article Content
Abstract
This study investigated the quality of Google Translate outputs after the system shifted from statistical machine translation (SMT) to neural machine translation (NMT). The examples originally analyzed by Al-Zebary (2012), when Google Translate operated under the SMT framework, were retranslated through Google Translate in 2023 and 2025. The resulting outputs were examined for both lexical and structural issues, and a comparative analysis was conducted across the three sets of translations to determine whether the problems previously identified continue to persist. The findings revealed that Google Translate has undergone substantial improvement under NMT, where many of the errors reported in earlier SMT output such as deletion, addition, misinterpretation of homographs, and literal translation of idioms have been significantly reduced. There was also a noticeable improvement from the 2023 NMT outputs to those produced in 2025, indicating continued refinement in Google Translate’s handling of English–Arabic translation. However, the study shows that unnatural phrasing continues to occur in linguistically complex contexts, especially those involving cross-linguistic structural mismatches. The results confirm that despite notable progress, machine translation still requires human intervention to ensure accuracy, naturalness, and contextual appropriateness in English–Arabic translation
Downloads
Article Details
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
This open-access article is distributed under a Creative Commons Attribution (CC-BY) 4.0 license.
You are free to: Share — copy and redistribute the material in any medium or format. Adapt — remix, transform, and build upon the material for any purpose, even commercially. The licensor cannot revoke these freedoms as long as you follow the license terms.
Under the following terms: Attribution — You must give appropriate credit, provide a link to the license, and indicate if changes were made. You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
No additional restrictions You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
How to Cite
Share
References
Ali, A. (2016). Exploring the Problems of Machine Translation from Arabic into English language Faced by Saudi University Student of Translation at the Faculty of Arts, Jazan University Saudi Arabia. IOSR Journal of Humanities and Social Science, 21(4), 55-66. https://www.iosrjournals.org/iosr-jhss/papers/Vol.%2021%20Issue4/Version-2/I2104025566.pdf
Alkhawaja, L., Ibrahim, H., Ghnaim, F., & Awwad, S. (2020). Neural machine translation: Fine-grained evaluation of Google translate output for English-to-Arabic translation. International Journal of English Linguistics, 10(4), 43. https://www.ccsenet.org/journal/index.php/ijel/article/view/0/42605
Almahasees, Z., Meqdadi, S., & Albudairi, Y. (2021). Evaluation of google translate in rendering English COVID-19 texts into Arabic. Journal of Language and Linguistic Studies, 17(4), 2065-2080. https://search.informit.org/doi/abs/10.3316/informit.228360028176817
ALPAC. (1966). Language And Machines Computers In Translation And Linguistics.
Aiken, M., & Ghosh, K. (2009). Automatic translation in multilingual business meetings. Industrial Management & Data Systems, 109(7), 916−925. https://doi.org/10.1108/02635570910982274
Al-Samawi, A. M. (2014). Language errors in machine translation of encyclopedic Texts from English into Arabic: the case of Google Translate. Arab World English Journal, 182-211.
Al-Zebary, Y. T. A. (2012). Lexical and Structural Ambiguity in Machine Translation (Doctoral dissertation, Sudan University of Science and Technology). http://repository.sustech.edu/handle/123456789/5975
Bonnie, D., Snover, M., & Madnani, N. (2010). Machine translation evaluation (Chapter 5). Retrieved December 31, 2022, from
Cheng, Y. (2019). Joint Training for Neural Machine Translation. Singapore: Springer Nature Singapore Ltd. https://doi.org/10.1007/978-981-32-9748-7
Condon, S. L., Parvaz, D., Aberdeen, J. S., Doran, C., Freeman, A., & Awad, M. (2010). Evaluation of machine translation errors in English and Iraqi Arabic (pp. 159–168). LREC, European Language Resources Association. https://doi.org/10.21236/ADA576234
Costa, Â., Ling, W., Luís, T., Correia, R., & Coheur, L. (2015b). A linguistically motivated taxonomy for Machine Translation error analysis. Machine Translation, 29(2), 127–161. https://doi.org/10.1007/s10590-015-9169-0
Ebrahim, S., Hegazy, D., Mostafa, M. G., & El-Beltagy, S. R. (2015). English-arabic statistical machine translation: State of the art. In Computational Linguistics and Intelligent Text Processing: 16th International Conference, CICLing 2015, Cairo, Egypt, April 14-20, 2015, Proceedings, Part I 16 (pp. 520-533). Springer International Publishing. https://link.springer.com/chapter/10.1007/978-3-319-18111-0_39
ElShiekh, A. A. (2012). Google Translate service: transfer of meaning, distortion or simply a new creation? An investigation into the translation process & problems at Google. English Language and Literature Studies, 2(1) 56-68. http://dx.doi.org/10.5539/ells.v2n1p56
Farrús, M., Farrús, F., Costa-Jussà, M. R., Mariño, J. B., Mariño, M., & Fonollosa, J. A. R. (2010). Linguistic-based Evaluation Criteria to identify Statistical Machine Translation Errors. http://translate.google.com
Flanagan, M. A. (1994). Error Classification for MT Evaluation. Amta, 65–72.
Frederking, R. E., Taylor, K. B., Elliott, D., Hartley, A., & Atwell, E. (2004). A Fluency Error Categorization Scheme to Guide Automated Machine Translation Evaluation. In LNAI (Vol. 3265). http://eprints.whiterose.ac.uk/82298/
Henisz-Dostert, B., Macdonald, R., & Zarechnak, M. (1979). Machine Translation - Bozena Henisz-Dostert, R. R. Macdonald, Michael Zarechnak - Google Books. https://books.google.mu/books?id=St4iXxXoIIAC&printsec=frontcover#v=onepage&q&f=false.
Hutchins, W., & Somers, H. (1992). An introduction to machine translation.
Khalaf, B. K., & Zinb, Z. M. (2020). Developing a Transcultural Tool for Teaching Cultures in Translation Studies. International Journal of Innovation, Creativity and Change, 11(8), 573-593.
Khalaf, B. K., Zin, Z. M., & Al-Abbas, L. S. (2022). Contemporary perspective on cognitive development: Reconceptualising situational context as embedded model. International Journal of Instruction, 15(1), 401-420.
Llitjós, A. F., Carbonell, J. G., & Lavie, A. (2005b). A Framework for Interactive and Automatic Refinement of Transfer-based Machine Translation (Vol. 87).
Google Translate. (2021). Google - About Google, Our Culture & Company News. https://about.google/?hl=en
Och, F. (2009). 51 Languages in Google Translate. Google Research Blog on August 31. Retrieved from https://ai.googleblog.com/2009/08/51-languages-in-google-translate.html
Okpor, M. D. (2014). Machine translation approaches: issues and challenges. International Journal of Computer Science Issues (IJCSI), 11(5), 159.
Sycz-Opoń, J., & Gałuskina, K. (2017). Machine translation in the hands of trainee translators–An empirical study. Studies in Logic, Grammar and Rhetoric, 49(1), 195-212. https://sciendo.com/article/10.1515/slgr-2017-0012
Selijan, S., Brkic, M. & Kucis, V. (2011). Evaluation of free online machine translations for Croatian-English and English-Croatian language pairs. Paper presented at 3rd International Conference of The future of Information Science (INFuture) in Zagreb, 9-11 November 2011. pp. 331-340.
Turovsky, B. (2017). Higher quality neural translations for a bunch more languages. The Keyword Google Blog. Google. Retrieved March 6, 2019, from https://www.blog.google/products/translate/higher-quality-neural-translations-bunch-more-languages/
Turovsky, B. (2018). Email interview: D’Monte, Leslie: Working to improve quality of translations. Google’s Barak Turovsky in EmTech India 2018—an emerging technology conference organized by Mint and MIT Technology Review. Retrieved January, 2023, from https://www.livemint.com/Companies/hrQAcRhDJm6acbKrwLKtlN/Working-to-improve-quality-of-translations-says-Googles-Ba.html
Vilar, D., Xu, J., Fernando D’haro, L., & Ney, H. (2007). Error Analysis of Statistical Machine Translation Output. http://www.tc-star.org
Yang, W. (2010). A Tentative Analysis of Errors in Language Learning and Use. Journal of Language Teaching and Research, 1(3), 266–268. https://doi.org/10.4304/jltr.1.3.266-268