Conversation Pattern Forge

Conversation Pattern Forge is an autonomous OpenAI Build Week lab that maps work signals in audited conversation exports, lets a human inspect and approve a corpus, and prepares reviewed workflows and optional skill candidates for controlled promotion.

Implemented deterministic milestones

The Public Build Lane now implements a deterministic CLI-level Corpus Atlas slice over five synthetic conversations. It provides canonical serialization for DiscoveryRequest, ScanSpec, and ScanRun; an offline lexical index; guided signal extraction; inspectable candidate scoring; deterministic exploratory clustering; human drill-down state; separate corpus permissions; and a reproducible FrozenEvidenceSlice with candidate → cluster → approval lineage.

The historical bounded Core milestone consumes its checked-in FrozenEvidenceSlice as a regression replay. The current public reviewed-Core slice instead freezes normalized conversations selected in the active run and passes that FrozenEvidenceSlice directly to message-bounded Episode extraction and pattern-independent ComparableOccurrence normalization. Context-only approved conversations remain visible but do not fabricate episodes.

The third deterministic milestone compares eligible occurrences with versioned feature weights, lightweight TF-IDF, explicit structural agreements, a required shared guided gesture, and policy thresholds. It emits RecurringPattern only when at least two distinct conversations qualify and preserves every pairwise score and exclusion reason for inspection.

The fourth milestone creates versioned PatternAssessment proposals with Pydantic validation and offline synthetic rules. Every assessment exposes its rule checks, supporting evidence IDs, input hash, uncertainties, and synthetic provenance. A parallel corpus-wide public runner can now derive a proposal_only WorkflowCandidate when the exact assessment → pattern → occurrence lineage passes versioned deterministic rules. Step content comes from configured gesture templates; it is not freely discovered or human-approved.

The implementation deliberately stops before the later canonical ReviewDecision, approved workflows, skill eligibility/candidates, functional promotion, Action Preview, or live/replay providers. Separate read-only Streamlit projections expose synthetic results without approving or executing workflows.

System boundaries

The pre-existing forensic Parser remains the source of evidence. This lab does not import or modify its code. Obsidian, LLM Wiki, MiniVault, Morpheiai, WorkspaceIA, Threadfield, and _AIProjects are references or future destinations only; this repository never writes to them during the MVP.

Public Build Lane

The default path uses only synthetic or explicitly approved fixtures. It requires no private Parser pack, network connection, or OpenAI API key.

The question-driven public vertical slice reads the repository-owned Parser-shaped record fixture, compiles the human question into a deterministic ScanSpec, scans every conversation, and stops at uninspected drill-down states. Its output must be an explicit directory outside the repository:

PYTHONPATH=src .venv/bin/python -m conversation_pattern_forge.public_vertical_slice \
  --lane public-fixture \
  --input fixtures/synthetic/parser_shaped_public_pack \
  --question "Which conversations compare renderer options and select tools?" \
  --output /private/tmp/cpf-public-slice

The command emits only the request, scan specification/run, lexical index, conversation candidates, exploratory clusters, initial inspection_required drill-down states, and a run summary. It does not emit an approval, frozen slice, episode, occurrence, pattern, workflow, skill, provider result, or promotion artifact. The fixture implements the documented public Parser-shaped contract; it does not claim exact Parser V5 compatibility or real anonymization.

The same bounded runner also consumes the pinned public Parsing contract openai-obsidian-pack-v1.4 through a dedicated reader and normalization layer. The byte-identical contract-bundle V1.0.2 is pinned to Parsing commit 21605d162220d4658b23de30077d52cd79237ffb. It retains the conversation-scoped message-identity contract and adds the producer-owned conversation_id → relative_note_path locator relation; 00_Home.md and the locator remain navigation-only and never contribute to evidence or Atlas hashes. The checked-in upstream synthetic example is sufficient for a contract smoke test:

PYTHONPATH=src .venv/bin/python -m conversation_pattern_forge.public_vertical_slice \
  --lane public-fixture \
  --format openai-obsidian-pack-v1.4 \
  --input contracts/upstream/parser/openai-obsidian-pack-v1.4/synthetic-example/pack \
  --question "Find the synthetic alpha request" \
  --output /private/tmp/cpf-parser-v14-alpha

This path validates the complete 90_Evidence file set and reads records only from the pack manifest, conversations, messages, asset links, and the declared conversation-note locator. It never opens 20_Files or note contents. Source IDs remain reader-internal; artifacts use deterministic CPF corpus IDs, which are explicitly not privacy pseudonyms. The older parser-shaped-public/1.0.0 path remains a separate regression fixture.

Synthetic Obsidian review projection

The public review-loop contract turns one immutable synthetic Atlas result into a deterministic ReviewSession, append-only human inspection, classification, selection/rejection, and explicit reopen events, derived effective state, and an Obsidian-compatible Markdown projection. Events may carry an optional bounded human rationale. The display question or review label is a renderer-only input: the audit ledger retains the DiscoveryRequest identity, not the raw text or a standalone question fingerprint. The projection links to existing Parser-owned synthetic notes and does not copy their transcripts or write to a vault.

The pinned Parser V1.4 contract publishes the stable producer-owned conversation_id → relative_note_path relation under 40_Views. CPF consumes that relation internally and never reconstructs filenames or scans note frontmatter. Source conversation identities and the full relation remain absent from review/audit serialization. Renderer-reserved filename characters are percent-encoded only when the in-memory Markdown link is produced.

Candidate selection remains distinct from CorpusApproval. The synthetic bridge requires a separate approval with four deny-by-default permissions and emits an immutable ReviewApprovalReceipt binding the exact review session and ledger hash before creating a FrozenEvidenceSlice. A later reopen never mutates an existing approval lineage. It cannot create a WorkflowCandidate from a selected conversation; the canonical Core lineage through episodes, occurrences, patterns, and assessment remains mandatory. Pending Obsidian edits are not yet synchronized or recorded by CPF. See docs/OBSIDIAN_REVIEW_LOOP.md for the state machine and boundary details.

The bounded public preview command composes the pinned V1.4 reader, normalizer, question-driven Atlas, official note locator, empty review ledger, and Markdown renderer. It requires an absolute Parser pack path and a new external output directory, writes exactly one deterministic file, and stops before any review event or approval:

PYTHONPATH=src .venv/bin/python -m conversation_pattern_forge.review_preview \
  --input "$PWD/contracts/upstream/parser/openai-obsidian-pack-v1.4/synthetic-example/pack" \
  --output /private/tmp/cpf-synthetic-review-preview \
  --question "Find the synthetic alpha request" \
  --top-k 10

The output layout is closed to candidate-review-preview.md. It shows only the display question, opaque candidate IDs, ranks, scores, scoring reasons, evidence counts, initial inspection_required state, and controlled pack-relative Parser note references. It contains no transcript, source identity mapping, absolute path, provider result, or standalone question fingerprint. The command does not write to Obsidian; a future writer must choose the correct vault-relative path base and handle synchronization.

Public reviewed Core slice

The resumable public entrypoint closes the synthetic Atlas-to-Core junction without a downstream fixture replay. prepare performs one real Parser V1.4 read, normalization, question-driven scan, locator resolution, empty review session, and Markdown projection into a new external run directory:

PYTHONPATH=src .venv/bin/python \
  -m conversation_pattern_forge.public_reviewed_core_slice prepare \
  --lane public-fixture \
  --input /private/tmp/example-synthetic-parser-pack \
  --question "Which synthetic conversations draft motion prompts?" \
  --output /private/tmp/example-reviewed-core-run \
  --top-k 10

Human actions are recorded one at a time with explicit candidate IDs and timestamps. No action is inferred, and classification remains distinct from selection:

PYTHONPATH=src .venv/bin/python \
  -m conversation_pattern_forge.public_reviewed_core_slice record-event \
  --lane public-fixture \
  --run-dir /private/tmp/example-reviewed-core-run \
  --candidate-id candidate-fictitious \
  --action inspect \
  --recorded-at 2026-07-19T10:00:00Z

After an explicit inspect → classify → select sequence, continue requires a separate local-analysis approval. It revalidates the same synthetic input pack, creates the receipt, approval, and current frozen slice, and stops after Episodes and ComparableOccurrences:

PYTHONPATH=src .venv/bin/python \
  -m conversation_pattern_forge.public_reviewed_core_slice continue \
  --lane public-fixture \
  --input /private/tmp/example-synthetic-parser-pack \
  --run-dir /private/tmp/example-reviewed-core-run \
  --approved-for-local-analysis \
  --approved-at 2026-07-19T10:03:00Z \
  --segmentation-version 1.0.0

The closed run records scan artifacts, the session and hash-chained ledger, review receipt and approval, one or more frozen slices, episodes.jsonl, comparable_occurrences.jsonl, and a hash/lineage summary. It does not read the historical downstream fixtures, infer an Obsidian vault, or continue to Pattern, Assessment, Workflow, Skill, provider, or promotion output.

Corpus-wide pattern discovery

The question-free public runner analyzes every conversation in one approved synthetic CorpusPack. Atlas candidates and exploratory clusters are diagnostic only: ranking never filters the frozen corpus. The pipeline keeps all objects in memory through FrozenEvidenceSlice → Episode → ComparableOccurrence → RecurringPattern → PatternAssessment → proposed WorkflowCandidate, then atomically writes exactly 13 deterministic artifacts to a new external directory:

PYTHONPATH=src .venv/bin/python \
  -m conversation_pattern_forge.corpus_discovery_pipeline run \
  --lane public-fixture \
  --format parser-shaped-public/1.0.0 \
  --input "$PWD/fixtures/synthetic/parser_pattern_discovery_pack" \
  --output /private/tmp/cpf-corpus-discovery \
  --approved-for-local-analysis \
  --approved-at 2026-07-19T00:00:00Z \
  --segmentation-version 1.0.0

CorpusAnalysisApproval is distinct from candidate selection and authorizes only deterministic local analysis of all synthetic conversations. It keeps model, public-demo, and repository permissions false. Workflow output remains a configured proposal: no workflow review, approval, skill, provider, promotion, or external-system write occurs.

python3.12 -m venv .venv
.venv/bin/python -m pip install -e '.[dev]'
.venv/bin/python -m pip check
.venv/bin/python scripts/check_no_private_data.py
PYTHONPATH=src .venv/bin/python -m pytest -q tests/public
PYTHONPATH=src .venv/bin/python -m conversation_pattern_forge.milestone --check-fixtures
PYTHONPATH=src .venv/bin/python -m conversation_pattern_forge.core_milestone --check-fixtures
PYTHONPATH=src .venv/bin/python -m conversation_pattern_forge.pattern_milestone --check-fixtures
PYTHONPATH=src .venv/bin/python -m conversation_pattern_forge.assessment_milestone --check-fixtures

The milestone commands read only fixtures/synthetic and versioned local configuration, print deterministic hash summaries, and fail if checked-in public artifacts differ from a fresh render. They perform no network or provider call and require safe_demo_mode to remain enabled.

The project environment uses Python 3.12. Do not install the live extra during the offline milestones; it is reserved for a separately authorized provider preflight.

Supported platform for the Build Week demo: Python 3.11+ on macOS or Linux. Windows is not yet validated.

Provider modes

• openai_live: future optional live integration, requiring a separate credential decision and explicit model-processing approval.
• captured_openai_replay: future replay of an authentic, redacted, approved provider capture.
• synthetic_fixture: offline test data that does not claim OpenAI provenance.

No live or captured-replay provider is implemented or called by this milestone.

Private Validation Lane

PRIVATE RETRIEVAL VALIDATION HAS BEEN EXECUTED ONLY IN A SEPARATE, AUTHORIZED PRIVATE LANE. The private run and its source data remain outside this repository and are not part of the public demo, public CI, or this documentation surface. The public repository retains the fail-closed boundary and synthetic proof only; it does not reproduce, expose, or treat the private run as a public artifact.

The private scan command requires all three opt-ins before it examines either path: CPF_PRIVATE_VALIDATION=1, --lane private-validation, and --confirm-private-validation. Input and output must be explicit absolute paths outside the repository, must not be symlinks, and must not overlap. The example paths below are fictitious and suitable only after copying the public synthetic pack to an external directory:

CPF_PRIVATE_VALIDATION=1 PYTHONPATH=src .venv/bin/python \
  -m conversation_pattern_forge.private_validation_boundary scan \
  --lane private-validation \
  --confirm-private-validation \
  --input /private/tmp/example-parser-v14-pack \
  --output /private/tmp/example-private-ranking-run \
  --question "Find the synthetic alpha request" \
  --top-k 10

The scan writes a restricted external store with derived/, secrets/, and state/ zones. Derived candidates and evidence references contain only run-scoped HMAC-SHA-256 pseudonyms; the reversible source mapping and generated run secret remain in the restricted secrets/ zone. The aggregate scan console does not print the question, message text, source identities, mapping, or paths.

The review preview for that ranking is a separate projection of the verified run, not another scan. It preserves the published pseudonymous IDs, ordering, scores, reasons, and evidence references. The exact question is accepted only as display text after its run-scoped HMAC is compared with the committed run binding. All three opt-ins are required again before either the pack or run is accessed:

CPF_PRIVATE_VALIDATION=1 PYTHONPATH=src .venv/bin/python \
  -m conversation_pattern_forge.private_review_preview \
  --lane private-validation \
  --confirm-private-validation \
  --input /private/tmp/example-parser-v14-pack \
  --run-dir /private/tmp/example-private-ranking-run \
  --output /private/tmp/example-private-review-preview \
  --question "Find the synthetic alpha request"

The output is a new external 0700 directory containing only the 0600 file candidate-review-preview.md. The projection reads the restricted mapping only to bind ranked pseudonyms to the producer-owned note locator; it never copies that mapping, source identities, message payloads, absolute paths, or the run secret. It creates no review event and does not mark any candidate inspected, classified, selected, rejected, or approved.

A separately gated inspection command reveals only the synthetic evidence of one explicit candidate and does not record or infer that a human inspected it:

CPF_PRIVATE_VALIDATION=1 PYTHONPATH=src .venv/bin/python \
  -m conversation_pattern_forge.private_validation_boundary inspect \
  --lane private-validation \
  --confirm-private-validation \
  --input /private/tmp/example-parser-v14-pack \
  --run-dir /private/tmp/example-private-ranking-run \
  --candidate-id pcandidate-fictitious

The next boundary is also implemented and tested only on external copies of the public synthetic pack. It resumes the canonical ranking without rescanning, records explicit pseudonymous review events, and requires a distinct local-only approval before materializing selected evidence through FrozenEvidenceSlice → Episode → ComparableOccurrence:

CPF_PRIVATE_VALIDATION=1 PYTHONPATH=src .venv/bin/python \
  -m conversation_pattern_forge.private_reviewed_core prepare \
  --lane private-validation --confirm-private-validation \
  --ranking-run /private/tmp/example-private-ranking-run \
  --output /private/tmp/example-private-reviewed-core \
  --created-at 2026-07-19T12:00:00Z

record-event accepts only inspect, classify, select, reject, or reopen, and continue additionally requires the original pack, the same ranking run, --approved-for-local-analysis, an approval timestamp, and a segmentation version. The bridge does not invent an Atlas cluster: its ReviewedSelectionApproval binds the ordered selected pseudonyms directly to the canonical ranking. Only selected conversations are materialized, under 0700/0600 external storage; source identities and the reversible mapping remain solely in the ranking run's restricted secrets/ zone. The command stops after comparable_occurrences.jsonl and creates no Pattern, Assessment, Workflow, Skill, provider output, or Obsidian write.

No private pack should be supplied in the Public Build Lane. A real run requires a new, explicitly authorized Private Validation Lane session. Private opt-in tests remain excluded from public CI and must not run in the public /feedback session.

Do not place private data inside this repository, even temporarily.

Build Week provenance

Pre-existing work includes the Parser and the referenced knowledge-system experiments. New Build Week work begins after 13 July 2026 and lives in this repository. Codex with GPT-5.6 Sol at high reasoning was used to design, implement, test, document, and iterate on the Build Week functionality. The public demo application itself performs no provider call: its synthetic ranking, review state, patterns, assessments, and proposal-only workflow are produced by local deterministic code.

See docs/IMPLEMENTATION_HANDOFF.md for the implementation boundary, docs/SECURITY.md for data safety, docs/DEVPOST_SUBMISSION.md for the judge-facing project description, and docs/SUBMISSION_CHECKLIST.md for the remaining submission gates.

Public corpus-wide demo UI

The read-only Streamlit demo projects the existing synthetic corpus-wide result; it does not implement another pipeline. The fixture is not read until the single Run synthetic corpus discovery button is clicked. The complete result and immutable display view are retained in st.session_state, so ordinary reruns do not recalculate the corpus.

PYTHONPATH=src .venv/bin/streamlit run \
  src/conversation_pattern_forge/ui/streamlit_app.py \
  --server.headless true

The five read-only tabs expose corpus scope, exact pattern evidence, assessment, configured proposal-only workflow steps, and deterministic versions/hashes. There is no upload, question, approval control, provider, download, filesystem writer, Obsidian integration, or promotion action.

Public question-reviewed product demo

The default Streamlit mode presents a five-step human review journey: Question → Candidates → Review → Pattern → Workflow. That navigation is not the full execution pipeline. A separate read-only analysis status progressively shows the validated Parser pack, discovery request, scan specification, one canonical ranking, append-only review, explicit local evidence approval, reviewed-Core, frozen evidence, Episodes, ComparableOccurrences, recurring pattern, separate assessment, conditional workflow proposal, and final product projection. Only after Find candidate conversations is clicked, CPF copies the official vendored Parser V1.4 synthetic snapshot to an external temporary workspace and applies the closed public profile fixtures/synthetic/question_reviewed_product_demo_profile.v1.json. The vendored snapshot is never modified. The materialized pack remains contract valid and its two Parser note locators continue to resolve to notes whose visible synthetic messages match the evidence used by the scan.

The question is chosen from a closed public-synthetic selector and drives one real DiscoveryRequest, ScanSpec, ranking, and empty review session. Custom question text is rejected by the demo boundary before fixture or output access. Inspection, classification, selection, and local-analysis approval require separate explicit actions. The completed reviewed-Core run is then passed directly to recurrence detection, assessment, and configured proposal_only WorkflowCandidate derivation without a second scan or Core replay. Selecting only one conversation is an explicit non-result: no recurring Pattern or WorkflowCandidate is fabricated.

Before the run starts, the page truthfully describes the source as the official public synthetic Parser V1.4 demo pack that will be prepared after the user acts. After preparation, it reports the verified conversation count from the run itself and marks that source read-only. Human-readable candidate cards, review decisions, recurring-pattern summaries, assessments, and the proposed workflow are shown first. Candidate IDs, raw reasons, evidence/message references, hashes, versions, and canonical Parser-pack-relative note targets remain available in closed traceability panels. Each target is revalidated below the materialized pack at render time and shown as a truthful copyable reference; the Streamlit page does not claim a portable local-file opener. Visual back-navigation never rewrites the ledger, while reopen/removal remains an explicit append-only review event. The demo creates no workflow approval, export, provider call, skill, vault note, Parser mutation, or Obsidian write. The secondary Advanced diagnostics mode keeps the existing corpus-wide five-tab projection unchanged.