Create accountLogin

Recent searches

No recent searches

Search options

Only available when logged in.
toad.social is one of the many independent Mastodon servers you can use to participate in the fediverse.
Mastodon server operated by David Troy, a tech pioneer and investigative journalist addressing threats to democracy. Thoughtful participation and discussion welcome.

Administered by:

Server stats:

303
active users

toad.social: AboutProfiles directoryPrivacy policy

Mastodon: AboutGet the appKeyboard shortcutsView source codev4.3.8

#measurementerror

0 posts0 participants0 posts today
Public
Dennis Alexis Valin Dittrich

Using Large Language Models for Qualitative Analysis can Introduce Serious #Bias
arxiv.org/abs/2309.17147
"…using LLMs to annotate text creates a risk of introducing biases that can lead to misleading inferences, in the sense that the errors that LLMs make in annotating are not random with respect to the characteristics of the subjects. Training simpler supervised models on high-quality human annotations with flexible coding leads to less #measurementError and bias than #LLM annotations"

arXiv.orgUsing Large Language Models for Qualitative Analysis can Introduce Serious BiasLarge Language Models (LLMs) are quickly becoming ubiquitous, but the implications for social science research are not yet well understood. This paper asks whether LLMs can help us analyse large-N qualitative data from open-ended interviews, with an application to transcripts of interviews with Rohingya refugees in Cox's Bazaar, Bangladesh. We find that a great deal of caution is needed in using LLMs to annotate text as there is a risk of introducing biases that can lead to misleading inferences. We here mean bias in the technical sense, that the errors that LLMs make in annotating interview transcripts are not random with respect to the characteristics of the interview subjects. Training simpler supervised models on high-quality human annotations with flexible coding leads to less measurement error and bias than LLM annotations. Therefore, given that some high quality annotations are necessary in order to asses whether an LLM introduces bias, we argue that it is probably preferable to train a bespoke model on these annotations than it is to use an LLM for annotation.
ExploreLive feeds
About