feat: add live endpoint response collection capability to the adapter SDK

[csoceanu]

What problem does this solve?

Chatbot and RAG pipeline teams who want to evaluate their systems through
EvalHub currently need to write custom scripts outside of EvalHub to:

1. Take a set of test questions (e.g. a golden Q&A dataset created by sdg_hub)
2. Query their chatbot endpoint with each question
3. Collect and format the responses into the structure their evaluation framework expects
4. Only then submit an EvalHub job with the pre-formatted data

This manual glue code is rewritten from scratch by every team. There is no SDK-level capability to collect
responses from a live endpoint as part of an evaluation job.

Proposed solution

Add a data collection module to the SDK that any adapter can call during
the LOADING_DATA phase of run_benchmark_job() to query a chatbot endpoint
and collect responses.

The module would:

• Accept a pre-existing set of test questions as input (CSV/JSON/JSONL file)
• Query the chatbot endpoint for each question
• Collect responses with metadata
• Write an evaluation-ready dataset to an output directory

Existing adapters remain unchanged, this is opt-in for adapters that want to support live collection.

Alternatives considered

Build a standalone adapter that handles both collection and evaluation. Rejected because it would only serve one evaluation framework, whereas an SDK-level module benefits all adapters.

[csoceanu]

cc @ruivieira @julpayne

[julpayne]

One aspect that we need to look at is how this is exposed in the API. We already have a test_data_ref object per benchmark https://eval-hub.github.io/eval-hub/#tag/Evaluations/operation/post_evaluations_jobs that could be a possibility.

[ai-hustle-bro]

I took a quick look at the current SDK surface because the API exposure point in the maintainer comment seems like the key design choice here.

What I see:

• BenchmarkConfig.test_data_ref already exists on the client/API model.
• Current TestDataRef is S3-only.
• Adapter-side JobSpec does not appear to carry test_data_ref yet; adapters mostly receive model, parameters, callbacks, and metadata.
• JobPhase.LOADING_DATA is the natural hook for this.

That makes this feel like two separate decisions:

1. If this should be a first-class API/UI feature, add a live endpoint shape under test_data_ref and pass the validated object through to adapter JobSpec.
2. If the near-term goal is to unblock adapter authors, add an experimental SDK helper that adapters call from LOADING_DATA, with config temporarily passed through parameters["live_collection"].

I put a small first-pass packet here with a decision matrix, SSRF/trust-boundary notes, and a standalone OpenAI-compatible helper sketch: https://gist.github.com/ai-hustle-bro/f73b6f89c8c75bdc713a1e3cc5c806db

The helper is not meant as a drop-in PR; it is mainly to make the boundary concrete before the API shape is chosen.

[csoceanu]

@julpayne @ai-hustle-bro thanks! would it make sense to start with an SDK helper that adapters call from LOADING_DATA, with config passed through parameters? That way we can ship something usable without server changes, validate the approach, and then promote it to test_data_ref once it's proven. Or would you prefer we go straight to the API-level approach?

[ai-hustle-bro]

Yes, I would start with the SDK helper path, with two constraints:

1. Mark the parameters["live_collection"] shape as experimental.
2. Design the helper/config so it can move under test_data_ref.live_endpoint later without changing adapter code very much.

That gives you a small shippable path:

• one typed config model for live_collection,
• one helper entrypoint adapters call during JobPhase.LOADING_DATA,
• CSV/JSONL input readers,
• an OpenAI-compatible chat-completions collector as the first endpoint type,
• JSONL output plus a small manifest/errors summary,
• tests around retries, per-row errors, empty questions, redirect handling, and auth-env handling.

I would avoid making the first PR a server/API change unless hosted EvalHub needs to accept arbitrary end-user endpoint URLs immediately. The API-level route is the right long-term contract, but it forces the harder questions first: URL validation, secret references, hosted egress policy, whether API clients can submit this shape, and how test_data_ref is passed into adapter JobSpec.

My suggested split:

1. First PR: experimental SDK helper, config passed through parameters["live_collection"], one adapter/example proving the row schema.
2. Second PR, after the shape is proven: add test_data_ref.live_endpoint, pass validated test_data_ref into JobSpec, and support both config locations for one transition release.

So my answer is: yes, start with the SDK helper if the immediate goal is to unblock adapters and validate the behavior. Go straight to API-level only if the next user-facing feature must be submitted through the EvalHub API/UI from day one.

[julpayne]

@julpayne @ai-hustle-bro thanks! would it make sense to start with an SDK helper that adapters call from LOADING_DATA, with config passed through parameters? That way we can ship something usable without server changes, validate the approach, and then promote it to test_data_ref once it's proven. Or would you prefer we go straight to the API-level approach?

@csoceanu I would start with the helper, I think that the API we can look at later? @rui wdyt?