Skip to content

fix(completion): respect configured OpenAI prompt templates - #4529

Open
chyroc wants to merge 1 commit into
TabbyML:mainfrom
chyroc:fix/preserve-completion-prompt-template
Open

fix(completion): respect configured OpenAI prompt templates#4529
chyroc wants to merge 1 commit into
TabbyML:mainfrom
chyroc:fix/preserve-completion-prompt-template

Conversation

@chyroc

@chyroc chyroc commented Aug 13, 2026

Copy link
Copy Markdown

Summary

  • Respect a non-empty prompt_template for openai/completion and
    openai/legacy_completion.
  • Preserve the existing generic FIM fallback and prompt / suffix transport
    when no template is configured.
  • Add request-level coverage for the serialized /v1/completions payload.

Context

Related to #4523, for the OpenAI-compatible completion path only. This change
does not modify vllm/completion and does not claim to resolve the issue's
reported vLLM behavior.

PR #3341 intentionally gave the OpenAI aliases Tabby's generic
{prefix}<|FIM|>{suffix} default. This change keeps that default when
prompt_template is absent or empty, but lets an explicit model-specific
template override it. An explicitly configured template containing Tabby's
internal <|FIM|> marker also retains the existing split prompt / suffix
transport. deepseek/completion, mistral/completion, and
vllm/completion keep their current behavior.

Test plan

CARGO_NET_OFFLINE=true cargo test --locked -p http-api-bindings --lib
cargo fmt --all -- --check
CARGO_NET_OFFLINE=true cargo clippy --locked -p http-api-bindings \
  --all-targets --no-deps -- -D warnings
git diff --check

Results: 6 affected-crate tests passed, 0 failed, and 3 existing network
integration tests were ignored. The regression starts a loopback HTTP server
and checks the request path and complete JSON body for a configured Qwen-style
template, the default fallback, and an explicit internal FIM template.

The workspace-wide test command was also attempted, but this shallow checkout
does not contain the recursive llama.cpp submodule, so the unchanged
llama-cpp-server build script stopped before workspace tests ran.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant