fix(url): route URL-keyed hosts to markitdown and upgrade the transcript API

[lufzle]

The bug

youtube-to-markdown returns the YouTube page footer instead of a transcript:

[About](https://www.youtube.com/about/)[Press](...)[Copyright](...)
© 2026 Google LLC

There are two independent causes. Fixing either one alone leaves the tool broken, which I think is why this has been reported more than once.

Cause 1: URL routing

All three URL tools download the page to a temp .html file and pass that path to markitdown. But YouTubeConverter.accepts() and BingSerpConverter.accepts() match on stream_info.url, which is unset for a local file:

# markitdown/converters/_youtube_converter.py
if not url.startswith("https://www.youtube.com/watch?"):
    return False

So accepts() returns False, the generic HTML converter runs, and you get page chrome. Same for Bing, which returns localized SERP boilerplate instead of parsed results.

Cause 2: markitdown's transcript pin

markitdown requires youtube-transcript-api~=1.0.0, and 1.0.x no longer parses YouTube's current transcript responses:

Attempt 1 failed: no element found: line 1, column 0
Attempt 2 failed: no element found: line 1, column 0
Attempt 3 failed: no element found: line 1, column 0

markitdown swallows that and emits the page without a transcript. Verified on a clean venv built by the current setup.sh: fixing the routing alone still yields no transcript. 1.2.4 parses correctly.

The fix

Routing (src/utils.ts, src/Markdownify.ts) — an allowlist of hosts whose markitdown converters key off the URL, checked after redirect resolution. Everything else keeps the existing download path.

Why an allowlist rather than passing every URL through (which is what #100 and #106 do): markitdown fetches with requests' default User-Agent, and some hosts reject it while accepting this server's fetch.

Host	markitdown fetches URL	current path (node fetches)
en.wikipedia.org/wiki/Markdown	403 Forbidden	200, 63 KB
stackoverflow.com/questions/tagged/python	403 Forbidden	200

Wikipedia is the sharp case: markitdown ships a WikipediaConverter, but it can never run on a live fetch, so a blanket passthrough trades working output for a hard 403. Confirmed by running both paths side by side.

safeFetch now returns the post-redirect URL alongside the response, so youtu.be and m.youtube.com links normalize into the www.youtube.com/watch?v= form the converter requires (youtu.be/ID → 303 → www.youtube.com/watch?v=ID&feature=youtu.be). Per-hop SSRF validation from #80 is unchanged; redirect and unused response bodies are now cancelled rather than left dangling.

Host matching is exact rather than suffix-based, so www.youtube.com.evil.test and evil-www.youtube.com do not match, and non-https is rejected.

Dependency (setup.sh, setup.bat) — upgrade youtube-transcript-api in a second step. It has to be two steps: installing both constraints at once fails with ResolutionImpossible against markitdown's ~=1.0.0. Afterwards pip check reports No broken requirements found.

Verification

bun test — 95/95 pass (79 existing + 16 new). bun run build and tsc --noEmit clean.

End to end through the MCP protocol:

Case	Result
youtube-to-markdown canonical URL	transcript, 11,972 chars
youtube-to-markdown youtu.be short link	transcript, 11,972 chars
bing-search-to-markdown	parsed results, was localized chrome
webpage-to-markdown Wikipedia	63 KB, unchanged (no 403)
webpage-to-markdown StackOverflow	unchanged (no 403)
webpage-to-markdown example.com	unchanged

New tests cover the allowlist boundaries (canonical/extra-params/no-video-id/non-watch paths, Bing search vs non-search, lookalike hosts, http, malformed URLs, Wikipedia staying off the list), that a YouTube URL reaches _markitdown as a URL, that a youtu.be redirect resolves first, that non-allowlisted URLs still go to a temp file, and that a redirect to a private IP is still rejected.

Relation to #100 and #106

Both propose the routing half via a blanket passthrough. That fixes YouTube but regresses Wikipedia and StackOverflow to 403, and neither addresses cause 2, so youtube-to-markdown still returns no transcript on a freshly built venv. Happy to fold this into either if you'd prefer one of those as the base. Likely also fixes #33, and #34's 1.2 MB maxBuffer blowup goes away since the watch page is no longer piped through stdout.

Notes

• pyproject.toml/uv.lock left alone deliberately: the lock is already stale (pins markitdown 0.1.5, no [all] extras, no transcript dep) and nothing in the repo invokes uv, so setup.sh is the real install path. Happy to add the override there too if you want the lock refreshed.
• YouTube geo-localizes title and description, so those fields may come back in the caller's region language while the transcript stays in the source language. Pre-existing markitdown behavior, not touched here.

🤖 Generated with Claude Code