Llamas don't understand fiction

Repositories containing the evaluation spreadsheets for the paper "Llamas Don’t Understand Fiction: Application and Evaluation of Large Language Models for Knowledge Extraction from Short Stories in English"

Overview

• human_evaluation: Spreadsheets containing death events manual annotation and comparison of LLM outputs with manual annotations (refer to paper).
• llms_output: Raw JSONL outputs generated by the LLMs. Deepseek output includes the reasoning process performed by the model into <\think> tags.
• scripts: Code to generate summaries via LLM and extract death event triplets from stories and summaries.

Repository Structure

llms-ke-fiction/
├── human_evaluation/
│   ├── llama/
│   │   ├── generic_summary_llama.csv
│   │   ├── specialised_summary_llama.csv
│   │   └── stories_llama.csv
│   └── deepseek/
│       ├── generic_summary_deepseek.csv
│       ├── specialised_summary_deepseek.csv
│       └── stories_deepseek.csv
├── llms_output/
│   ├── KE_GenericSummary_deepseek-r170b.jsonl
│   ├── KE_GenericSummary_llama3170b.jsonl
│   ├── KE_SpecialisedSummary_deepseek-r170b.jsonl
│   ├── KE_SpecialisedSummary_llama3170b.jsonl
│   ├── KE_stories_deepseek-r170b.jsonl
│   └── KE_stories_llama3170b.jsonl
└── scripts/
    ├── generic_summary_generation.py
    ├── specialised_summary_generation.py
    ├── KE_onSummaries.py
    └── KE_onStories.py