Beyond the Literal: Machine Translation Performance and Strategies in Rendering Audiovisual Political Idioms

Abstract

This is a mixed-method research that integrates quantitative and qualitative research methodologies. It aims to identify and categorize the translation strategies employed by four selected MT systems in rendering audiovisual (AV) political idioms as categorized by Helleklev (2006). Moreover, it aims to evaluate the quality of the MT systems' translations of AV political idioms by comparing them to a human translation reference. By comparing the four MT systems, the study found that the system's most frequently used translation strategy is "using an explanatory everyday expression," which is used mainly by ChatGPT-4 (56.89%) compared to other systems. Notably, "an everyday expression is translated by an idiom" occurred only once in ChatGPT-4 translation (1.72%). The human translator exhibits a lower frequency of word-for-word translation than the four MT systems and a higher frequency of translating an idiom with an equivalent one. Correspondingly, the analysis reveals that Gemini employed strategies that align most closely with those of the human translator among all three other systems. Comparing the four systems to the human reference, the study also found that the BLEU scores vary significantly, with Matecat outperforming the other systems with a BLEU score 55.77. This indicates that its translation output is the closest to the human reference among all systems. Meanwhile, Microsoft Bing scored the lowest score of 24.39, suggesting it is unreliable in translating idioms. However, all systems have scored comparatively close to one another in length ratio. This demonstrates that, in comparison to the human reference, their translations are frequently of an appropriate length.