Skip to content

feat(agentkit/local): add MultiModalRead for images and PDFs#812

Open
JonXSnow wants to merge 4 commits into
mainfrom
feat/multimodal-read
Open

feat(agentkit/local): add MultiModalRead for images and PDFs#812
JonXSnow wants to merge 4 commits into
mainfrom
feat/multimodal-read

Conversation

@JonXSnow
Copy link
Copy Markdown
Contributor

@JonXSnow JonXSnow commented Apr 29, 2026

Add MultiModalRead to SandboxTool that returns structured multimodal content for image and PDF files, with fallback to Read for other types.

Images:

  • Read raw bytes via readAllBytes Python template (base64 transfer)
  • Detect MIME type via magic-number headers (PNG/JPEG/GIF/BMP/WebP/TIFF)
  • Enforce 10 MB size limit

PDFs:

  • Full read (no 'pages'): return raw PDF bytes, 20 MB limit
  • Paged read (with 'pages'): render page range to PNG via go-fitz at 150 DPI, 100 MB limit, max 20 pages per request
  • Validate 'pages' parameter syntax and ranges upfront
  • Clamp end page with warning when it exceeds total page count

Size enforcement:

  • Dual-layer check: Python side (os.path.getsize) rejects before read, Go side re-checks after decode as defense-in-depth
  • Use stdout sentinel marker (READALLBYTES_TOO_LARGE) instead of exit codes to distinguish size errors from other failures
  • Provide actionable error messages (e.g. suggest 'pages' param when full-read limit is exceeded)

Also:

  • Switch Write template to binary mode ('wb') for byte-level fidelity
  • Add table-driven tests for parsePagesParam
  • Bump go directive to 1.23; upgrade eino to v0.9.0-alpha.5

@github-actions
Copy link
Copy Markdown

github-actions Bot commented Apr 29, 2026

Need to create a new tag

The following modules have changes and may need version updates:

  • adk/backend/agentkit (Current: adk/backend/agentkit/v0.2.5)

⚠️ Please create and push new version tags for these modules after merging this PR.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 13, 2026

Need to create a new tag

The following modules have changes and may need version updates:

  • adk/backend/local (Current: adk/backend/local/v0.2.4)

  • adk/backend/agentkit (Current: adk/backend/agentkit/v0.2.5)

⚠️ Please create and push new version tags for these modules after merging this PR.

1 similar comment
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 13, 2026

Need to create a new tag

The following modules have changes and may need version updates:

  • adk/backend/local (Current: adk/backend/local/v0.2.4)

  • adk/backend/agentkit (Current: adk/backend/agentkit/v0.2.5)

⚠️ Please create and push new version tags for these modules after merging this PR.

@JonXSnow
feat(adk/backend): add MultiModalRead for images and PDFs
d99cf36 
Add MultiModalRead support to both agentkit and local backends, enabling
sandboxed reading of images and PDFs with configurable size and page
limits. PDF rendering honors context cancellation, and the size field is
widened to int64 to support larger files.
@JonXSnow JonXSnow force-pushed the feat/multimodal-read branch from d3b5ea9 to d99cf36 Compare May 13, 2026 07:49
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 13, 2026

Need to create a new tag

The following modules have changes and may need version updates:

  • adk/backend/local (Current: adk/backend/local/v0.2.4)

  • adk/backend/agentkit (Current: adk/backend/agentkit/v0.2.5)

⚠️ Please create and push new version tags for these modules after merging this PR.

@JonXSnow JonXSnow force-pushed the feat/multimodal-read branch from d99cf36 to 1bcfb71 Compare May 13, 2026 09:37
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 13, 2026

Need to create a new tag

The following modules have changes and may need version updates:

  • adk/backend/local (Current: adk/backend/local/v0.2.4)

  • adk/backend/agentkit (Current: adk/backend/agentkit/v0.2.5)

⚠️ Please create and push new version tags for these modules after merging this PR.

@JonXSnow JonXSnow force-pushed the feat/multimodal-read branch from 1bcfb71 to e53270b Compare May 13, 2026 10:04
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 13, 2026

Need to create a new tag

The following modules have changes and may need version updates:

  • adk/backend/local (Current: adk/backend/local/v0.2.4)

  • adk/backend/agentkit (Current: adk/backend/agentkit/v0.2.5)

⚠️ Please create and push new version tags for these modules after merging this PR.

@JonXSnow JonXSnow changed the title feat(agentkit): add MultiModalRead for images and PDFs feat(agentkit/local): add MultiModalRead for images and PDFs May 13, 2026
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 13, 2026

Need to create a new tag

The following modules have changes and may need version updates:

  • adk/backend/local (Current: adk/backend/local/v0.2.4)

  • adk/backend/agentkit (Current: adk/backend/agentkit/v0.2.5)

⚠️ Please create and push new version tags for these modules after merging this PR.

@JonXSnow
fix(adk/backend): use base64 encoding for file_path in readPythonCode…
b8e8189 
…Template

Align readPythonCodeTemplate with other templates (readAllBytes, glob,
execute) by base64-encoding the file_path parameter, preventing path
injection when paths contain special characters like single quotes.
@JonXSnow JonXSnow force-pushed the feat/multimodal-read branch from e4f933e to b8e8189 Compare May 13, 2026 11:36
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 13, 2026

Need to create a new tag

The following modules have changes and may need version updates:

  • adk/backend/local (Current: adk/backend/local/v0.2.4)

  • adk/backend/agentkit (Current: adk/backend/agentkit/v0.2.5)

⚠️ Please create and push new version tags for these modules after merging this PR.

@JonXSnow
fix(adk/backend): fix CI lint error and raise test coverage to 70%
e11dac6 
Replace "PNGs" with "images" in comments to avoid typo-checker false
positive. Add unit tests for isImageExt, isPDFExt, newImageContentPart,
and renderPDFPagesToImages to meet the 70% coverage threshold.
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 13, 2026

Need to create a new tag

The following modules have changes and may need version updates:

  • adk/backend/local (Current: adk/backend/local/v0.2.4)

  • adk/backend/agentkit (Current: adk/backend/agentkit/v0.2.5)

⚠️ Please create and push new version tags for these modules after merging this PR.

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 13, 2026

Need to create a new tag

The following modules have changes and may need version updates:

  • adk/backend/local (Current: adk/backend/local/v0.2.4)

  • adk/backend/agentkit (Current: adk/backend/agentkit/v0.2.5)

⚠️ Please create and push new version tags for these modules after merging this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@JonXSnow