The Use of ChatGpt as an Arabic Translation Tool: A Study of BLEU Metrics Quality and Human Assessment
Pemanfaatan ChatGpt Sebagai Alat Terjemahan Bahasa Arab: Kajian Kualitas Metrik BLEU dan Penilaian Manusia
DOI:
https://doi.org/10.21070/ups.11566Keywords:
ChatGPT, Arabic translation, BLEUAbstract
This study aims to analyze the quality of ChatGPT translations from Arabic to Indonesian using a combined approach of automated evaluation based on the BLEU metric and human evaluation. The method used was descriptive quantitative analysis, involving 30 students as evaluators who assessed accuracy, acceptability and readability. The results show an average BLEU score of 0.4123, indicating moderate alignment with fluctuating variations. Human evaluation revealed that translation quality falls into the high category for accuracy and readability and the moderate category for acceptability. Furthermore the results indicate that BLEU scores and human evaluations do not always yield similar assessments because the two methods measure different aspects of translation quality. Therefore translation quality evaluation should be conducted in a complementary manner. These findings suggest that ChatGPT has the potential to serve as a translation aid but still requires human evaluation to achieve optimal results.
Downloads
References
R. Najwa, F. Azkiya, and T. Amelia, “Faktor-Faktor Kesulitan Mahasiswa : Pengaruh Gramatikal Bahasa Arab Terhadap Pembuatan Teks Bahasa Arab Factors of Student Difficulties : The Influence of Arabic Grammar on Arabic Text Creation,” pp. 11590–11597, 2025.
K. Peneliti, Bahasa arab di Indonesia. 1436. [Online]. Available: https://ketabpedia.com/تحميل/اللغة-العربية-في-إندونيسيا/
“Pelajaran Bahasa Arab untuk Non-Penutur Asli - Bagian Pertama.pdf.” [Online]. Available: https://archive.org/details/20191011_20191011_0820/دروس اللغة العربية لغير الناطقين بها - الجزء الأول/
L. Pertiwi, F. Zahara, J. K. Siagian, and H. A. Nasution, “Pengaruh Status Sosial terhadap Pilihan Dialek Bahasa Arab dalam Komunikasi Sehari-hari,” Al-Lughoh J. Bhs. Arab dan Sastra Arab, vol. 1, no. 1, pp. 7–15, 2025.
Ainur Rofiq Sofa and Ayun Febrianti, “Dialektologi Bahasa Arab: Analisis Perbedaan Linguistik Berdasarkan Kajian Pustaka,” Lencana J. Inov. Ilmu Pendidik., vol. 3, no. 2, pp. 76–87, 2025, doi: 10.55606/lencana.v3i2.5018.
A. N. Fradana, Buku Ajar Morfologi Bahasa. 2018. doi: 10.21070/2018/978-602-5914-31-7.
F. Rizka and F. M. Ammar, “Analisis Faktor Kesulitan Membaca Teks Berbahasa Arab Kelas VIII,” JIIP - J. Ilm. Ilmu Pendidik., vol. 7, no. 4, pp. 3660–3666, 2024, doi: 10.54371/jiip.v7i4.4295.
M. Z. Haq, “Arabic Idiomatic Translation Problems To Indonesian Problematika Penerjemahan Idiomatik Arab Ke Indonesia,” MUHIBBUL Arab. J. Pendidik. Bhs. Arab, vol. 2, no. 1, pp. 15–30, 2022.
M. I. Salim and L. M. A. KH. M. Abd. Rouf, Syarah Diwan Imam Asy-Syafi’i. Divapress, 2019. [Online]. Available: https://books.google.co.id/books?id=eJ27DwAAQBAJ
A. Marhamah, Imas Marliana, Halimah Ibrahim, and Sahkholid Nasution, “Analisis Kesalahan Linguistik Dalam Penerjemahan Teks Bahasa Arab Pada Google Translate,” AL-MUADDIB J. Kaji. Ilmu Kependidikan, vol. 7, no. 1, pp. 122–136, 2025, doi: 10.46773/muaddib.v7i1.1513.
S. Rifky et al., Artificial Intelligence : Teori dan Penerapan AI di Berbagai Bidang. PT. Sonpedia Publishing Indonesia, 2024. [Online]. Available: https://books.google.co.id/books?id=r8YMEQAAQBAJ
Ibrahim Ahmad Assegaf et al., “Pengembangan Chatbot Konsultasi Kesehatan Mental Kesehatan Mental Berbasis Open Ai Model Gpt-3.5 Turbo Menggunakan Media Whatsapp,” J. Inform. Teknol. dan Sains, vol. 6, no. 4, pp. 785–793, 2024, doi: 10.51401/jinteks.v6i4.4749.
L. Judijanto et al., Optimalisasi ChatGPT: Panduan dan Penerapan untuk Belajar, Mengajar, dan Membuat Konten Tanpa Batas. PT. Green Pustaka Indonesia, 2025. [Online]. Available: https://books.google.co.id/books?id=mhBlEQAAQBAJ
A. Ruhmadi and M. Z. Al Farisi, “Analisis Kesalahan Morfologi Penerjemahan Arab–Indonesia pada ChatGPT,” Aphorisme J. Arab. Lang. Lit. Educ., vol. 4, no. 1, pp. 55–75, 2023, doi: 10.37680/aphorisme.v4i1.3148.
S. N. Latifah and W. I. F. Djamilah, “Penggunaan chatgpt dalam penerjemahan bahasa arab,” Al Dirosah J. Pendidik. Agama Islam, vol. 01, no. 2, pp. 1–13, 2024, [Online]. Available: http://ojs.iai-darussalam.ac.id/index.php/pai
E. M. Z, “Minat Mahasiswa Menggunakan ChatGPT dalam Pembelajaran Bahasa,” vol. 5, no. 1, pp. 155–168, 2025.
A. S. M. Amadi and K. Hikmah, “Persepsi Mahasiswa Tentang Pemanfaatan Teknologi AI dalam Pembelajaran Bahasa Arab di Perguruan Tinggi Islam Indonesia,” J. Educ. Res., vol. 6, no. 2, pp. 291–301, 2025, doi: 10.37985/jer.v6i2.2343.
K. Papineni, S. Roukos, T. Ward, and W. J. Zhu, “BLEU: A method for automatic evaluation of machine translation,” Proc. Annu. Meet. Assoc. Comput. Linguist., vol. 2002-July, no. July, pp. 311–318, 2002.
M. Waruwu, S. N. Pu`at, P. R. Utami, E. Yanti, and M. Rusydiana, “Metode Penelitian Kuantitatif: Konsep, Jenis, Tahapan dan Kelebihan,” J. Ilm. Profesi Pendidik., vol. 10, no. 1, pp. 917–932, 2025, doi: 10.29303/jipp.v10i1.3057.
S. P. M. A. D. A. B. M. P. Dr. Ishak Bagea, Metode Penelitian Kuantitatif Kualitatif Campuran R & D. CV. AZKA PUSTAKA, 2025. [Online]. Available: https://books.google.co.id/books?id=KI5yEQAAQBAJ
M. T. Kholida, “Penilaian Kualitas Keterbacaan Terjemahan Google Translate pada Surah Al- Alaq,” vol. 1, pp. 98–104, 2025.
H. A. Karim, “Journal of Digital Business and Information Technology,” pp. 50–60, doi: 10.23971/jobit.v1i2.297.
I. Yadi, “Implentasi Arsitektur Transfomer untuk penerjemahan Bahasa Lampung - Indonesia,” 2024.
“Kama istusyhida sabiqa al-yaum Muḥammad Khalid ‘Abid bi-rasasin Isra’iliyyin fī ar-ra’s bi-mantiqati az-Zarqa’’ fī hayyi at-Tuffaḥ wafqan lil-masdari at-tibbi.,’” Al jazeera. [Online]. Available: https://www.aljazeera.net/news/2026/1/26/3-شهداء-بنيران-الاحتلال-في-مناطق-بقطاع
K. Papineni, S. Roukos, T. Ward, and W. Zhu, “B LEU : a Method for Automatic Evaluation of Machine Translation,” no. July, pp. 311–318, 2002.
M. Post, “A Call for Clarity in Reporting BLEU Scores ACL Anthology ID W18-6319 / revision 9 / 18 Sep 2025,” vol. 1, pp. 186–191, 2018.
A. Muam, C. D. Nugraha, and U. G. M. Press, Pengantar Penerjemahan. Gadjah Mada University Press, 2021. [Online]. Available: https://books.google.co.id/books?id=U3cWEAAAQBAJ
Z. S. S. Hariyanto, Translation: Bahasan Teori & Penuntun Praktis Menerjemahkan. Media Nusa Creative (MNC Publishing). [Online]. Available: https://books.google.co.id/books?id=LYZOEAAAQBAJ
Y. R. Uyuni, Menerjemahkan Makna Bukan Kata: Teori dan Evaluasi Penerjemahan Arab-Indonesia. Penerbit A-Empat, 2023. [Online]. Available: https://books.google.co.id/books?id=0kPAEAAAQBAJ
M. Julkarnain, E. Mardinata, and R. Susilowati, “Penerapan Transformer-Based Neural Machine Translation untuk Bahasa Bima,” pp. 793–802, 2025, doi: 10.33364/algoritma/v.22-2.2949.
Downloads
Additional Files
Posted
Categories
License
Copyright (c) 2026 UMSIDA Preprints Server
This work is licensed under a Creative Commons Attribution 4.0 International License.
