Quantitative Analysis of Political Party Understanding and the Impact of Political Bias through ChatGPT

Faculty of Data Science, Shiga University, Hikone, Japan

Abstract

In recent years, large language models (LLMs) such as ChatGPT have been utilized for acquiring political knowledge. However, there remain questions about their accuracy and fairness, as these models may harbour biases in understanding political parties. This study aims to quantify the understanding of Japanese political parties using the ChatGPT model and evaluate the model’s biases and their impacts. Specifically, we conducted experiments using pairs of questions and answers that reflect the stances of each party to investigate the extent to which the model demonstrates understanding toward specific parties. The experimental results revealed that ChatGPT-4 exhibits a significantly higher level of understanding towards the Liberal Democratic Party, while its understanding of newer parties like Reiwa Shinsengumi is lower. Additionally, the GPT model acting as a voter tends to have a positive bias towards certain parties and reflects progressive ideologies. It was also shown that the recognition of political parties influences the model’s understanding, with factors such as the number of seats, advertising expenses, and the frequency of party names in the dataset potentially playing crucial roles. Based on these findings, this study provides a foundation for enhancing the accuracy and fairness of party understanding using GPT models and proposes improvements for future research and practice.