workflow: refactor task logic for external providers

[hannahblair]

Description

the workflow was breaking when you tried to use models on providers like Together or Fireworks. i was getting 403s for Kimi-K3 which gave me an error "task not supported for provider"

the code was calling funcs like client.visual_question_answering(...) which only work on hf-inference

the fix

two routing changes in gradio/workflow.py:

1. vision tasks now go through chat_completion with a vision message

rerouted image-text-to-text, visual-question-answering, document-question-answering, and image-to-text away from the InferenceClient methods and onto chat_completion with an OpenAI-style vision content array:

{ "role": "user", "content": [
    { "type": "image_url", "image_url": { "url": "..." } },
    { "type": "text", "text": "..." }
]}

The specialised methods only work on hf-inference. chat_completion is the contract every provider implements, so provider="auto" routing stops 403ing when it lands on Together/Fireworks/etc.

2. text-generation tries chat_completion first, falls back to text_generation only if that fails

3. unknown task fallback goes to HF's raw inference API

If we don't recognise the pipeline_tag and no specialised endpoint matches, _raw_inference() POSTs directly to api-inference.huggingface.co. so any future task or unusual model works without waiting for InferenceClient to expose a method

4. local image paths are base64-encoded into data URIs before being sent to external provider

When a vision workflow runs locally, images sit at tmp paths which extneral providers can't access. if the resolved URL isn't http(s):// or already a data: URI, read the file, base64-encode it

[gradio-pr-bot]

🪼 branch checks and previews

•	Name	Status	URL
	Spaces	ready!	Spaces preview
	Website	ready!	Website preview
🦄	Changes	detected!	Details

---

Install Gradio from this PR

pip install https://huggingface.co/buckets/gradio/pypi-previews/resolve/345c25bf488cc9220142bb9d7322ffb2b5abaa95/gradio-6.20.0-py3-none-any.whl

Install Gradio Python Client from this PR

pip install "gradio-client @ git+https://github.com/gradio-app/gradio@345c25bf488cc9220142bb9d7322ffb2b5abaa95#subdirectory=client/python"

Import Gradio JS Client from this PR via CDN

import { Client } from "https://huggingface.co/buckets/gradio/npm-previews/resolve/345c25bf488cc9220142bb9d7322ffb2b5abaa95/browser.js";

[gradio-pr-bot]

🦄 change detected

This Pull Request includes changes to the following packages.

Package	Version
gradio	minor

---

• workflow: refactor task logic for external providers

---

Something isn't right?

• Maintainers can change the version label to modify the version bump.
• If the bot has failed to detect any changes, or if this pull request needs to update multiple packages to different versions or requires a more comprehensive changelog entry, maintainers can update the changelog file directly.

[Copilot]

Pull request overview

Refactors gradio.workflow model-task routing to avoid provider-specific InferenceClient.<task>() methods (which can 403 on non-hf-inference providers) by routing more tasks through chat_completion, and adds a generic fallback that uses Hugging Face’s raw inference HTTP API for unknown tasks.

Changes:

• Route vision “image → text” style tasks through chat_completion using a vision message payload.
• For text_generation, attempt chat_completion first and fall back to text_generation if needed.
• Replace the previous “unknown task” fallback with a dedicated _raw_inference() helper that POSTs to api-inference.huggingface.co.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
gradio/workflow.py	Updates task-to-endpoint routing, adds chat-based vision helpers, and introduces a raw inference fallback path for unknown tasks.
.changeset/dirty-pets-shine.md	Adds a changeset marking the workflow routing refactor as a minor Gradio release.

Comments suppressed due to low confidence (1)

gradio/workflow.py:913

• This refactor changes the vision/VQA code path to call chat_completion instead of the specialized InferenceClient methods. Existing unit tests currently assert client.visual_question_answering.assert_called_once_with(...) (see test/test_workflow.py:403-415), so CI will fail unless those tests are updated to assert chat_completion usage (and to cover the new vision message format).

    if endpoint in {
        "visual_question_answering",
        "document_question_answering",
        "image_to_text",
    }:

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hannahblair]

🚨 no need to review, this wheel is needed for a kimi demo but im working on a refactor for a cleaner task - endpoint implemenation