Portrait of Kai Kugler

Kai Kugler

Research Fellow

Kai Kugler is a computational and quantitative linguist whose research focuses on how neural language models encode and process linguistic units and structures. He is particularly interested in the relationship between linguistic representations in language models and the differences between human and synthetic texts. Working at the interfaces of lexicon, syntax, semantics, and pragmatics, he uses quantitative and computational methods to investigate how LLMs model linguistic phenomena and where their outputs diverge from human language use.

Publications

2026

  • Next Reply Prediction X (NRP-X) Dataset: Linguistic Discrepancies in Naively Generated Content

    Simon Münker, Nils Schwager, Kai Kugler, Michael Heseltine, Achim Rettinger

    LLMs4SSH @ LRECTWON
    Abstract

    The increasing use of Large Language Models (LLMs) as proxies for human participants in social science research presents a promising, yet methodologically risky, paradigm shift. While LLMs offer scalability and cost-efficiency, their “naive” application, where they are prompted to generate content without explicit behavioral constraints, introduces significant linguistic discrepancies that challenge the validity of research findings. This paper addresses these limitations by introducing a novel, history-conditioned reply prediction task on authentic X (formerly Twitter) data, to create a dataset designed to evaluate the linguistic output of LLMs against human-generated content. We analyze these discrepancies using stylistic and content-based metrics, providing a quantitative framework for researchers to assess the quality and authenticity of synthetic data. Our findings highlight the need for more sophisticated prompting techniques and specialized datasets to ensure that LLM-generated content accurately reflects the complex linguistic patterns of human communication, thereby improving the validity of computational social science studies.

2024

  • InvBERT: Reconstructing Text from Contextualized Word Embeddings by inverting the BERT pipeline

    Kai Kugler, Simon Münker, Johannes Höhmann, Achim Rettinger

    JCLS
    Abstract

    Digital Humanities and Computational Literary Studies apply automated methods to enable research on large corpora which are not feasible by manual inspection alone. However, due to copyright restrictions, the availability of relevant digitized literary works is limited. Derived Text Formats (DTFs) have been proposed as a solution. Here, textual materials are transformed in such a way that copyright-critical features are removed, but that the use of certain analytical methods remains possible. Contextualized word embeddings produced by transformer-encoders are promising candidates for DTFs because they allow for state-of-the-art performance on analytical tasks. However, in this paper we demonstrate that under certain conditions the reconstruction of the original text from token representations becomes feasible. Our attempts to invert BERT suggest that publishing the encoder together with the contextualized embeddings is unsafe, since it allows to generate data to train a decoder with a reconstruction accuracy sufficient to violate copyright laws.