fix: make FTS5 tokenizer configurable per collection

[mvanhorn]

What does this PR do?

FTSManager.createFtsTable() creates every SQLite FTS5 table with tokenize='porter unicode61'. Because unicode61 treats contiguous non-spacing text as a single token and EmDash emits prefix MATCH expressions, mid-sentence queries in Japanese, Chinese, Thai, Khmer, Lao, and Burmese silently miss indexed content. The tokenizer is not configurable, so there is no way out of it today.

This adds a backward-compatible optional tokenize field to the exported per-collection SearchConfig, constrained to the supported built-in configurations (porter unicode61, unicode61, trigram) so user input is never interpolated into FTS DDL unchecked. Omitting it retains the current English-stemming default, so existing collections are unaffected.

The selected tokenizer is threaded through enable, explicit rebuild, schema-triggered rebuild, repair, disable/re-enable persistence, and config parsing, so posting an updated enabled configuration drops, recreates, and repopulates the existing FTS table rather than leaving its tokenizer sticky. The option is exposed through the validated search-enable REST body, with docs on when to choose each tokenizer (Porter's English-only stemming, trigram's three-character minimum).

Closes #2245

Type of change

• Bug fix
• Feature (requires maintainer-approved Discussion)
• Refactor (no behavior change)
• Translation
• Documentation
• Performance improvement
• Tests
• Chore (dependencies, CI, tooling)

Checklist

• I have read CONTRIBUTING.md
• pnpm test passes (or targeted tests for my change)
• pnpm format has been run
• I have added/updated tests for my changes (if applicable)
• I have added a changeset (if this PR changes a published package)

Two checklist items are deliberately not ticked, with what I actually observed:

• pnpm typecheck — fails in my checkout with TS2307: Cannot find module '@emdash-cms/blocks' / '@emdash-cms/plugin-types', i.e. unbuilt workspace siblings rather than anything in this diff. Worth a maintainer confirming it is green in CI.
• pnpm lint — fails on demos/simple, fixtures/perf-site, infra/cache-demo, infra/do-solo-demo, and packages/registry-client with no-redundant-type-constituents. I verified those same failures reproduce on a pristine main checkout, and that oxlint --type-aware --deny-warnings is clean when run over only the files this PR changes.

AI-generated code disclosure

• This PR includes AI-generated code

AI was used for assistance.

Screenshots / test output

Targeted tests for this change:

$ pnpm --filter ./packages/core exec vitest run \
    tests/integration/search/tokenizer.test.ts tests/unit/api/search-schema.test.ts

 Test Files  2 passed (2)
      Tests  11 passed (11)

[changeset-bot]

🦋 Changeset detected

Latest commit: da8b64c

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages

Name	Type
emdash	Patch
@emdash-cms/cloudflare	Patch
@emdash-cms/sandbox-workerd	Patch
@emdash-cms/plugin-mcp-smoke	Patch
@emdash-cms/fixture-perf-site	Patch
@emdash-cms/perf-demo-site	Patch
@emdash-cms/cache-demo-site	Patch
@emdash-cms/do-demo-site	Patch
@emdash-cms/do-solo-demo-site	Patch
@emdash-cms/admin	Patch
@emdash-cms/auth	Patch
@emdash-cms/blocks	Patch
@emdash-cms/gutenberg-to-portable-text	Patch
@emdash-cms/x402	Patch
create-emdash	Patch
@emdash-cms/auth-atproto	Patch
@emdash-cms/plugin-embeds	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[pkg-pr-new]

Open in StackBlitz

@emdash-cms/admin

npm i https://pkg.pr.new/@emdash-cms/admin@2257

@emdash-cms/auth

npm i https://pkg.pr.new/@emdash-cms/auth@2257

@emdash-cms/auth-atproto

npm i https://pkg.pr.new/@emdash-cms/auth-atproto@2257

@emdash-cms/blocks

npm i https://pkg.pr.new/@emdash-cms/blocks@2257

@emdash-cms/cloudflare

npm i https://pkg.pr.new/@emdash-cms/cloudflare@2257

@emdash-cms/contentful-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/contentful-to-portable-text@2257

emdash

npm i https://pkg.pr.new/emdash@2257

create-emdash

npm i https://pkg.pr.new/create-emdash@2257

@emdash-cms/gutenberg-to-portable-text

npm i https://pkg.pr.new/@emdash-cms/gutenberg-to-portable-text@2257

@emdash-cms/plugin-cli

npm i https://pkg.pr.new/@emdash-cms/plugin-cli@2257

@emdash-cms/plugin-types

npm i https://pkg.pr.new/@emdash-cms/plugin-types@2257

@emdash-cms/registry-client

npm i https://pkg.pr.new/@emdash-cms/registry-client@2257

@emdash-cms/registry-lexicons

npm i https://pkg.pr.new/@emdash-cms/registry-lexicons@2257

@emdash-cms/registry-verification

npm i https://pkg.pr.new/@emdash-cms/registry-verification@2257

@emdash-cms/sandbox-workerd

npm i https://pkg.pr.new/@emdash-cms/sandbox-workerd@2257

@emdash-cms/x402

npm i https://pkg.pr.new/@emdash-cms/x402@2257

@emdash-cms/plugin-ai-moderation

npm i https://pkg.pr.new/@emdash-cms/plugin-ai-moderation@2257

@emdash-cms/plugin-atproto

npm i https://pkg.pr.new/@emdash-cms/plugin-atproto@2257

@emdash-cms/plugin-audit-log

npm i https://pkg.pr.new/@emdash-cms/plugin-audit-log@2257

@emdash-cms/plugin-color

npm i https://pkg.pr.new/@emdash-cms/plugin-color@2257

@emdash-cms/plugin-embeds

npm i https://pkg.pr.new/@emdash-cms/plugin-embeds@2257

@emdash-cms/plugin-field-kit

npm i https://pkg.pr.new/@emdash-cms/plugin-field-kit@2257

@emdash-cms/plugin-forms

npm i https://pkg.pr.new/@emdash-cms/plugin-forms@2257

@emdash-cms/plugin-webhook-notifier

npm i https://pkg.pr.new/@emdash-cms/plugin-webhook-notifier@2257

commit: da8b64c

[emdashbot]

This is the right fix for #2245. Hard-coding porter unicode61 silently breaks CJK/Thai search because those scripts have no word separators, so making the FTS5 tokenizer a per-collection SearchConfig option is additive, backward-compatible, and fits the existing schema-driven search state machine.

I checked the tokenize plumbing end-to-end:

• API schema (searchEnableBody) validates against the allowed built-in tokenizers before any DB work.
• Runtime validation (isSearchTokenizer / resolveSearchTokenizer) guards createFtsTable, rebuildIndex, and setSearchConfig, so unsupported values can’t reach DDL.
• The selected tokenizer is preserved through disable, re-enable, schema-triggered rebuilds, missing-index repair, and config parsing/serialization.
• Existing collections without a stored tokenize continue to default to porter unicode61.
• Tests cover the default tokenizer behavior, switching to trigram, preserving weights through a tokenizer-only change, persistence across disable/schema/repair cycles, and rejection of injection-style tokenizer values.
• Documentation in rest-api.mdx is updated to explain the three options.

The only issue I see is a small ordering nit: in createFtsTable, rebuildIndex, and enableSearch, the tokenizer is validated before the SQLite-only guard. On a non-SQLite database this can produce a misleading “unsupported tokenizer” error instead of the intended “SQLite only” no-op/error, and it does work that the dialect is about to discard. Moving the guard above the validation keeps the behavior consistent with the rest of the file.

Once that ordering is fixed this looks good to merge.

[emdashbot]

[suggestion] The tokenizer is validated before the SQLite-only early return. On a non-SQLite database this means an invalid tokenize value can throw an "Unsupported FTS5 tokenizer" error instead of the intended no-op (or the dedicated "only available with SQLite" error in enableSearch), and it performs validation work the dialect guard is about to discard.

Move the dialect check above the tokenize resolution so validation only runs when we’re actually going to build SQLite FTS DDL.

Suggested change

	const tokenizer = resolveSearchTokenizer(tokenize);
	if (!isSqlite(this.db)) return;
	): Promise<void> {
	if (!isSqlite(this.db)) return;
	const tokenizer = resolveSearchTokenizer(tokenize);

[github-actions]

Overlapping PRs

This PR modifies files that are also changed by other open PRs:

• feat(content): filter indexed custom fields #2213 (3 shared files)
• feat(core): configurable titleField and dateField per collection #1973 (4 shared files)

This may cause merge conflicts or duplicated work. A maintainer will coordinate.