EvalPort import/export — portable eval datasets across DeepEval, Promptfoo, Inspect AI, and more

[adhabnr-ux]

Describe the feature or potential improvement

We maintain EvalPort (Apache 2.0), an open interchange format for LLM evaluation datasets — test cases, graders, eval suites, and result sets in a single JSON format, so evals aren't locked to one tool. It's already integrated with UK AISI's Inspect AI (in their official extensions list) and has a draft adapter for Microsoft AutoGen.

Langfuse's Datasets feature (items + expected_output, run against experiments) maps naturally onto EvalPort's TestCase/ResultSet shape. Native import/export would let Langfuse users pull in eval suites built for DeepEval, Promptfoo, Inspect AI, etc. without hand-converting them, and export Langfuse dataset run results in a format other tools can read.

Rough shape (using our SDK, pip install evalport-sdk / npm install evalport-sdk):

to_evalport(dataset_run) -> dict # Langfuse DatasetRunItem[] -> EvalPort ResultSet
from_evalport(suite) -> list # EvalPort TestCase[] -> Langfuse dataset items

Happy to open a draft PR ourselves, or just talk through the mapping here first if that's easier.

Additional information

Spec: https://github.com/adhabnr-ux/evalport/blob/main/spec/SPEC.md
Repo: https://github.com/adhabnr-ux/evalport
Reference integration (Inspect AI): UKGovernmentBEIS/inspect_ai#4797

[CodeCLS]

Hi @adhabnr-ux , this looks super cool!