feat(cache): normalize default-valued params in cache key + hit-rate metric (#892) - #901
Open
suantea wants to merge 5 commits into
Open
feat(cache): normalize default-valued params in cache key + hit-rate metric (#892)#901suantea wants to merge 5 commits into
suantea wants to merge 5 commits into
Conversation
The server build writes server/tsconfig.tsbuildinfo into the repo root; git status was showing it as untracked noise.
suantea
force-pushed
the
feat/cache-hit-rate-892
branch
from
August 17, 2026 08:43
7152876 to
70eff8f
Compare
suantea
force-pushed
the
feat/cache-hit-rate-892
branch
from
August 19, 2026 06:34
1f59aa6 to
1caf141
Compare
…metric (tashfeenahmed#892) - computeCacheKey: explicit-but-default-valued sampling params (top_p:1, presence_penalty:0, frequency_penalty:0, n:1) are dropped from the key so clients that always serialize full params hit the same entries as minimal ones; key version bumped 3->4 - CacheStats: add misses and hitRate (hits/(hits+misses)); miss counted on lookup with no valid entry (missing or TTL-expired) - settings dialog: response-cache stats block (hits/misses/hit-rate/quota saved) on the Advanced section, fetching /api/cache/stats - i18n: 7 new settings.cache* keys across all 60 locales Co-Authored-By: AtomCode (deepseek-v4-flash) <noreply@atomgit.com>
The cache now stores entries in the response_cache table (better-sqlite3) in addition to the in-memory LRU, so answers survive restarts — free-tier quota resets at UTC midnight, and the same tasks re-run after the reset hit again instead of burning quota anew. - getCachedResponse: on an in-memory miss, fall through to SQLite and re-warm the LRU; TTL expiry deletes from both views - storeCachedResponse: INSERT OR REPLACE into SQLite + enforce the entry cap there too (LRU by last_hit_at_ms/created_at_ms) - getCacheStats: aggregate from SQLite (source of truth), so stats match what a cold process would actually serve - clearCache: clears memory + SQLite - fail-safe via withDb() (mirrors ratelimit): DB uninitialized/query failure disables the cache in the proxy hot path instead of throwing Co-Authored-By: AtomCode (deepseek-v4-flash) <noreply@atomgit.com>
…ashfeenahmed#892 part 3) Streaming requests are now cacheable: the exact SSE chunk sequence written to the client on a complete success is stored, and an identical later streaming request is replayed from the cache without touching a provider. - cache.ts: schema gains stream_frames column (ALTER for pre-existing DBs); CachedResponse/StoreInput/CacheEntry carry streamFrames; read/write both in-memory and SQLite views - proxy.ts: cacheKey no longer requires non-streaming; a streamed hit replays frames + [DONE] with SSE headers; a streamed miss collects frames (preamble + chunks + usage) and stores them on success — truncated (finish 'length') turns are not cached, matching the non-stream path; a stream request hitting a non-stream entry falls through to the provider - proxy-cache.test.ts: replaced the old 'streaming bypasses' assertion with populate-then-replay; added cache cleanup to keep stats/clear assertions isolated Co-Authored-By: AtomCode (deepseek-v4-flash) <noreply@atomgit.com>
suantea
force-pushed
the
feat/cache-hit-rate-892
branch
from
August 20, 2026 02:53
1caf141 to
090cb59
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
中文说明
针对 issue #892（response cache 命中率提升）的 #1 参数规范化 + #4 命中率指标 两部分。
#1 参数规范化：
computeCacheKey现在把「显式传入但等于 API 默认值」的采样参数（top_p:1、presence_penalty:0、frequency_penalty:0、n:1）从缓存 key 中剔除——总是序列化完整参数的客户端（如 CLI/agent 默认带top_p:1）与最小化请求命中同一缓存条目，这是提高命中率的核心修复。key 版本 v3→v4；非默认值仍参与 key，不会串答案。#4 命中率指标：
CacheStats新增misses与hitRate（hits/(hits+misses)），miss 在查询无有效条目（不存在或 TTL 过期）时计数；GET /api/cache/stats自动带出新字段。前端设置对话框「高级」区新增 Response cache 命中率区块（命中/未命中/命中率/节省配额），i18n 新增 7 个settings.cache*键同步到全部 60 个 locale。测试：cache 套件 30 + proxy-cache 套件 10 全绿（含 2 个新测试：显式默认值命中同一 key / 非默认值仍区分）；client/server tsc 干净。
English
Part 1 (param normalization) + part 4 (hit-rate metric) of issue #892.
#1 Param normalization:
computeCacheKeynow drops explicit-but-default-valued sampling params (top_p:1,presence_penalty:0,frequency_penalty:0,n:1) from the cache key, so clients that always serialize full params (e.g. CLIs/agents that sendtop_p:1by default) hit the same entries as minimal requests — the core fix for the near-zero hit rate. Key version bumped 3→4; non-default values stay in the key, so answers can never collide.#4 Hit-rate metric:
CacheStatsgainsmissesandhitRate(hits/(hits+misses)); a miss is counted on lookup with no valid entry (missing or TTL-expired).GET /api/cache/statsautomatically includes the new fields. The settings dialog now shows a Response-cache stats block (hits/misses/hit-rate/quota saved) under Advanced, with 7 newsettings.cache*i18n keys synced across all 60 locales.Tests: cache suite 30 + proxy-cache suite 10 all green (incl. 2 new tests: explicit defaults share a key / non-defaults stay distinct); client & server tsc clean.
Closes #892