Projectactive

Surprisal Program

A Python-based tool for calculating surprisal values

Leon Hammerla, Mercedes Martinez Bruera

2025

Abstract

We developed a Python-based tool for calculating surprisal values for textual data stored in CSV files. The program supports multiple language models available through Hugging Face and provides a unified workflow for working with both decoder-based and encoder-based model architectures.

Keywords

surprisalLLMsencoderdecoderHugging Facelinguisticsneglab

BibTeX

@project{INF:Surprisal:2025,
  name        = {Surprisal Program},
  subtitle    = {A Python-based tool for calculating surprisal values},
  author      = {Hammerla, Leon and Martinez Bruera, Mercedes},
  year        = {2025},
  abstract    = {We developed a Python-based tool for calculating surprisal
                 values for textual data stored in CSV files. The program
                 supports multiple language models available through Hugging
                 Face and provides a unified workflow for working with both
                 decoder-based and encoder-based model architectures.},
  url         = {https://github.com/LeonHammerla/surprisal_program},
  projectlink = {https://github.com/LeonHammerla/surprisal_program},
  keywords    = {surprisal, LLMs, encoder, decoder, Hugging Face, linguistics,
                 neglab},
  note        = {active}
}