docs(webhooks): clarify rate limiting does not interrupt in-flight enrichments#1499
Open
george-dilthey wants to merge 1 commit into
Open
docs(webhooks): clarify rate limiting does not interrupt in-flight enrichments#1499george-dilthey wants to merge 1 commit into
george-dilthey wants to merge 1 commit into
Conversation
Contributor
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Added one sentence to the Throughput subsection of the Limits section in
docs/webhook-integration-guide.md, explicitly stating that the webhook rate limiter operates only at the point of entry — rows already accepted into the table and running through enrichment columns are not affected by throttling on new incoming requests.Why
Tanner Phillips (Angi, Enterprise) asked via the shared Slack channel: "if we sent MORE than 50k rows in the same batch, say in the same 30 second period — Would this crash? Bump a row before its completed? Actually work but be highly discouraged?"
The existing doc already covered: 10 RPS limit, burst of 20, 100KB payload max, 50k submission cap, auto-delete bypassing the cap, 429 errors returned when throttled. What was missing was an explicit answer to "bump a row before its completed?" — whether exceeding the inbound rate limit could interrupt enrichments already running on rows already in the table.
Human support agent Bo Morin answered: "It won't crash or bump in-flight rows… Clay will rate-limit the excess and return rate-limit errors instead of crashing or interrupting rows that are already enriching." This specific guarantee — that the rate limiter only affects ingestion and not the enrichment pipeline — was not documented.
The verifier sub-agent confirmed (PASS) via
clay-base/apps/api/v3/recordFlow/services/webhook.service.ts:436-499that the rate limit check is at ingestion time and the enrichment queue is a separate pipeline entirely unaffected by webhook throttling. All other limits cited (10 RPS, burst 20, 100KB, 50k cap, passthrough bypass) were also confirmed PASS against the same file.Source link(s)
clay-base/apps/api/v3/recordFlow/services/webhook.service.ts:436-499(rate limit before destination branch);clay-base/apps/api/v3/recordFlow/domain/webhook-source.ts:230-241(passthrough bypass)Uncertainties
None. All claims verified PASS by the verifier sub-agent against production code.
Related observations
docsbot/doc-1073-intercom-tag-0710-a1892d0d) also edits this file but in a completely different section — it changes the Submission limit paragraph (counter-decrement on row deletion) and adds a deletion FAQ. My change is in the Throughput section addressing an entirely different concern (in-flight enrichment safety). The two changes do not conflict. A reviewer merging both should verify line ordering remains coherent.useIsPassThroughTablesEnabled) but the backend enforcement of the cap bypass is unconditional — noted by the verifier but not a documentation concern since the feature is already available to Enterprise customers.numSourceRecordscan decrement on row deletion in theory for non-passthrough tables (relevant to PR docs(webhooks): correct submission limit deletion behavior + add FAQ #1492's topic), but this doesn't affect the Throughput section I edited.Closes DOC-1081