Skip to content

docs: add devnote for evaluation rewrite - #241

Open
memadi-nv wants to merge 3 commits into
mainfrom
memadi/docs/evaluation-anonymizer-rewrite-devnote
Open

docs: add devnote for evaluation rewrite#241
memadi-nv wants to merge 3 commits into
mainfrom
memadi/docs/evaluation-anonymizer-rewrite-devnote

Conversation

@memadi-nv

Copy link
Copy Markdown
Contributor

Summary

Adds a developer note explaining rewrite-mode evaluation, its strategy-specific scores, output columns, usage examples, and display_record() report. Includes a report screenshot.

Type of Change

  • Bug fix
  • New feature
  • Breaking change
  • Documentation update
  • Refactoring
  • CI, release, or contributor workflow update

Contributor Checklist

  • PR title follows Conventional Commits, for example fix: handle empty entity list
  • Related issue is linked, or a maintainer-owned no-issue reason is documented above
  • For non-trivial changes, a plan document is linked above, or the no-plan reason is documented above
  • Public API impact checked; skills/anonymizer/SKILL.md updated if needed
  • No real PII added to tests, docs, notebooks, fixtures, or artifacts
  • No API keys, service tokens, private keys, credentials, or real endpoint secrets added

Validation

  • Commands run:
  • Skipped checks or known failures:

Documentation and Artifacts

  • Docs updated, or not needed
  • If docs changed: make docs-build passes locally
  • If tutorial sources changed: notebooks regenerated with make convert-notebooks
  • If e2e, benchmark, or model-provider behavior changed: relevant validation is listed above

Signed-off-by: memadi <memadi@nvidia.com>
@memadi-nv
memadi-nv force-pushed the memadi/docs/evaluation-anonymizer-rewrite-devnote branch from a586797 to 898c419 Compare August 13, 2026 00:34
Signed-off-by: memadi <memadi@nvidia.com>
Signed-off-by: memadi <memadi@nvidia.com>
@memadi-nv
memadi-nv marked this pull request as ready for review August 13, 2026 23:52
@memadi-nv
memadi-nv requested review from a team as code owners August 13, 2026 23:52
@greptile-apps

greptile-apps Bot commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

Adds a Rewrite-mode evaluation developer note and registers it in the documentation navigation.

  • Explains the evaluate–repair loop, utility and leakage metrics, repair thresholds, and human-review behavior.
  • Documents optional post-hoc judges, output columns, persistence, and record display.
  • Adds a screenshot illustrating the Rewrite evaluation report.

Confidence Score: 4/5

The PR appears safe to merge after considering a non-blocking documentation hardening note for the pickle reload example.

The documented evaluation behavior matches the current implementation, navigation and links resolve, and the only accepted concern is the missing trusted-input warning around pickle deserialization.

Files Needing Attention: docs/devnotes/posts/evaluation-anonymizer-rewrite.md

Security Review

The persistence example should warn that pickle.load is safe only for trusted files, because loading a substituted result can execute arbitrary code. How this was verified: The example passes the saved file directly to pickle.load without any trust check or warning.

Important Files Changed

Filename Overview
docs/devnotes/posts/evaluation-anonymizer-rewrite.md The evaluation semantics and examples match the implementation, but the pickle reload example needs an explicit trusted-files warning.
mkdocs.yml Adds the new developer note to the existing Developer Notes navigation using the established structure.
docs/devnotes/posts/assets/evaluate-rewrite-display-record.png Adds the report screenshot referenced by the new article.

Reviews (1): Last reviewed commit: "nit" | Re-trigger Greptile

Comment on lines +71 to +72
saved_result = pickle.load(f)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 security Warn that pickle requires trust

If the saved result comes from or can be replaced through an untrusted location, pickle.load() can execute attacker-controlled code with the user's privileges. Add an explicit warning that this workflow must only load trusted files.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant