Skip to content

README.md

Latest commit

 

History

History
102 lines (79 loc) · 4.07 KB

README.md

File metadata and controls

102 lines (79 loc) · 4.07 KB
Raw

Knowledge Layer

This directory consolidates everything PFAGENT learns from, trains on, or loads at runtime. It is the merged successor of the legacy ANDES_Data/ and PowerFlow_Specific/ directories (refactored 2026-04-10).

Three sub-layers

knowledge/
├── raw/          ← unprocessed original sources
├── finetuning/   ← development-stage fine-tuning data + generators
└── rag/          ← deployment-stage runtime-loaded assets

The split mirrors the agent lifecycle:

Stage Sub-layer Touched by
Source raw/ one-off extraction / crawling
Development finetuning/ data generators, audits, smoke
Deployment rag/ runtime FAISS index + few-shot

raw/ — original unprocessed sources

Reference material in its original form. Nothing in this directory is loaded by the runtime.

Path Origin Contents
raw/manual_extraction/ ANDES_Data/Extract_From_Manual/ ANDES manual PDF + extracted API/example CSVs and the script that produced them
raw/github_issues/ ANDES_Data/GitHub_Issues_Crawler/ crawler + crawled andes_github_issues.csv
raw/sample_notebooks/ ANDES_Data/Sample_Code/ Jupyter notebooks demonstrating ANDES workflows

finetuning/ — development stage

Everything used to build the fine-tuning dataset. Scripts here run offline (development-time only) and are NOT loaded by the runtime.

finetuning/
├── README.md          legacy PowerFlow_Specific README
├── guide.txt          notes
├── data/              .jsonl / .json training datasets + audit reports
├── scripts/           generators, audit, smoke test, trainer helpers
└── generated/
    ├── current/       active experiment outputs
    └── old/           archived experiment outputs

Key scripts (under finetuning/scripts/):

Script Purpose
fine_tuning_dataset_utils.py shared dataset helpers + path constants
build_clean_finetune_dataset.py builds the cleaned .jsonl from raw + curated sources
audit_fine_tuning_data.py runs audits on a fine-tuning JSONL
generate_verified_finetune_examples.py strict-verified single-turn generator
generate_generalized_verified_finetune_examples.py multi-turn generator
separate_finetune_data.py train/validation split
smoke_test_finetune_examples.py CI smoke test (run on every PR)
finetune.py uploads data and creates an OpenAI fine-tune job
run_processes.py / run_processes.sh top-level orchestrator

Key datasets (under finetuning/data/):

File Purpose
fine_tuning_data.jsonl raw concatenated dataset
fine_tuning_data.cleaned.jsonl cleaned + audited version
train.cleaned.jsonl / validation.cleaned.jsonl train/val split
verified_training_examples.json strictly verified single-turn examples
generalized_verified_training_examples.json verified multi-turn conversations
curated_training_examples.json curated hard cases
*.audit.json dataset audit reports
examples.csv canonical task list

rag/ — deployment stage

The minimal artifact set the runtime actually loads. Everything here is consumed by the live Streamlit app via text-to-sim/src/.

Path Loaded by Purpose
rag/andes_manual.pdf src/andes_manual.py Indexed into FAISS for RAG retrieval
rag/code_examples/ src/codex_fixer.py Repo-aware fixer context + few-shot examples

Touch this directory only when you want to change the assets a deployed agent uses at request time. Do NOT mix in development-only data or raw sources here.

Data governance

The training/holdout boundary still applies:

  • knowledge/finetuning/ may be used for fine-tuning
  • verification/ is the holdout benchmark and must NOT be folded back into training data without an explicit governance decision

See docs/DATA_GOVERNANCE.md.