Skip to content

feat: enable attaching images to agent chat#5290

Open
re-pixel wants to merge 9 commits into
superplanehq:mainfrom
re-pixel:feat/agent-chat-image-attachments
Open

feat: enable attaching images to agent chat#5290
re-pixel wants to merge 9 commits into
superplanehq:mainfrom
re-pixel:feat/agent-chat-image-attachments

Conversation

@re-pixel

@re-pixel re-pixel commented Jun 10, 2026

Copy link
Copy Markdown
Collaborator

Demo: https://drive.google.com/file/d/1bcmEU7CWOvt3aJRHPBG02LW93rtRkLmK/view?usp=sharing

Resolves #5136. Lets users paste or attach images (e.g. a screenshot of an error) in the canvas agent chat so the agent can act on them.

Backend

  • New AgentChatImage { media_type, data } proto type; images field added to the send-message request and stored message.
  • New images jsonb column on agent_session_messages (migration included).
  • Images forwarded to the Anthropic provider as image content blocks alongside the text.
  • gRPC handler validates media type (png/jpeg/gif/webp), base64, non-empty, ≤8 per message, and a combined ≤2.5 MB image payload per message (sized to keep the request under the gRPC limit); allows image-only messages.
  • Chat history does not embed base64: message responses carry image metadata only, and bytes are served out-of-band via a new cookie-authed endpoint (GET /api/v1/agents/chats/{chatId}/messages/{messageId}/images/{index}, scoped to the session's owner). This keeps history responses small enough to stay under the gRPC/HTTP response size limit.

Frontend

  • Paste images into the composer or pick them via a new attach button.
  • Removable thumbnail previews before sending; transcript images load lazily from the image endpoint via <img src> (URL built from chat/message/index), rather than inline data.
  • Paste only intercepts the clipboard when a supported, within-limit image is present, so unsupported/oversized pastes (and accompanying text) fall through normally.

@re-pixel re-pixel marked this pull request as draft June 10, 2026 15:13
Comment thread web_src/src/components/AgentSidebar/ChatComposer.tsx
Extend the SendAgentChatMessage API and the agent provider pipeline to
carry base64-encoded images alongside the text content, so a user can
send a screenshot for the agent to reason about.

- add repeated AgentChatImage to the request and stored message protos
- persist images on agent_session_messages (new jsonb column)
- forward images to the Anthropic provider as image content blocks
- validate media type, base64 encoding and size in the gRPC handler

Co-authored-by: Cursor <cursoragent@cursor.com>
Signed-off-by: re-pixel <relja.brdar@gmail.com>
@re-pixel re-pixel force-pushed the feat/agent-chat-image-attachments branch from 2d2a9c1 to 3a9a4e1 Compare June 10, 2026 15:21
Let users add screenshots to the agent chat so the agent can act on
them. Images can be pasted into the input or picked with a new attach
button, are previewed as removable thumbnails before sending, and render
as thumbnails in the conversation transcript.

Signed-off-by: re-pixel <relja.brdar@gmail.com>
@re-pixel re-pixel force-pushed the feat/agent-chat-image-attachments branch from 3a9a4e1 to 94a6ec2 Compare June 10, 2026 16:15
@re-pixel re-pixel marked this pull request as ready for review June 11, 2026 08:50
Signed-off-by: re-pixel <relja.brdar@gmail.com>
Comment thread pkg/grpc/actions/agents/send_agent_chat_message.go Outdated
Comment thread web_src/src/components/AgentSidebar/ChatComposer.tsx
re-pixel added 2 commits June 11, 2026 13:13
Signed-off-by: re-pixel <relja.brdar@gmail.com>
Signed-off-by: re-pixel <relja.brdar@gmail.com>

@cursor cursor Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 06892b9. Configure here.

Comment thread pkg/grpc/actions/agents/common.go
… responses under gRPC size limits

Signed-off-by: re-pixel <relja.brdar@gmail.com>
@re-pixel re-pixel force-pushed the feat/agent-chat-image-attachments branch from 908ab08 to ba7c1a6 Compare June 11, 2026 12:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support adding images to agent chat

1 participant