• Chung-Ang University

    AI Humanities Research Institute
    HK+ Artificial Intelligence Humanities

›Journals›Article & Issues

Article & Issues

eISSN: 2951-388X
Print ISSN: 2635-4691 / Online ISSN: 2951-388X
Title[JAIH Vol. 19] Comparative Analysis of LLM Responses to Prompts Related to Mainland China and Taiwan_Kim, Yu Joo2026-11-06 21:46
Writer Level 10
Attachment인공지능 인문학연구 19_01.김여주.pdf (1.06MB)

Abstract


This research compares and analyzes the responses of different Large Language Models (LLMs) to prompts related to common keywords between Mainland China and Taiwan, controlling for hyperparameters and languages. The study used four major LLMs: Qwen2-72B-Instruct (Mainland China), Llama-3-Taiwan-70B-Instruct (Taiwan), GPT-4o (United States), and Mistral-large-2411 (France). Questions on topics such as Taiwan (the island), history, politics, and cross-strait relations were presented in Simplified Chinese, Traditional Chinese, Korean, and English. Each question was repeated three times with different random seeds, resulting in a total of 576 responses. These responses were then quantified on a scale: Mainland China perspective (-1), neutral (0), and Taiwan perspective (+1). The results showed that even within the same model, responses to political issues such as cross-Strait relations varied depending on the model, language, and terminology used in the questions. In particular, Mainland China's Qwen2 consistently provided answers that strongly reflected the Mainland Chinese position, sometimes answering in Simplified Chinese even when questions were asked in Traditional Chinese. Language differences were also significant-when questions were asked in Simplified Chinese, most models produced responses aligned with Mainland China's perspective or maintained neutrality, whereas questions in Traditional Chinese elicited pro-Taiwan responses from all models except Qwen2. Qualitative differences between the models were also noted. GPT-4o and Mistral-large generally produced balanced responses that considered both sides and provided detailed information in English. Interestingly, while Korean responses showed limitations due to insufficient training data in some models, Qwen2 showed relatively neutral positions in certain Korean queries. This study confirms the political biases of LLMs and demonstrates how political and cultural contexts can be shaped by the language used. The findings suggest that models developed by state-led or state-based AI companies may reflect the political positions of their countries, highlighting the importance of considering language-specific biases when developing and using multilingual LLMs.

Chung-Ang University, Humanities Research Institute
#828, 310 Hall, 84 Heukseok-ro, Dongjak-gu, Seoul, 06974, Korea  TEL +82-2-881-7354  FAX +82-2-813-7353  E-mail : aihumanities@cau.ac.krCOPYRIGHT(C) 2017-2023 CAU HUMANITIES RESEARCH INSTITUTE ALL RIGHTS RESERVED