feat: enable attaching images to agent chat#5290
Open
re-pixel wants to merge 9 commits into
Open
Conversation
Extend the SendAgentChatMessage API and the agent provider pipeline to carry base64-encoded images alongside the text content, so a user can send a screenshot for the agent to reason about. - add repeated AgentChatImage to the request and stored message protos - persist images on agent_session_messages (new jsonb column) - forward images to the Anthropic provider as image content blocks - validate media type, base64 encoding and size in the gRPC handler Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: re-pixel <relja.brdar@gmail.com>
2d2a9c1 to
3a9a4e1
Compare
Let users add screenshots to the agent chat so the agent can act on them. Images can be pasted into the input or picked with a new attach button, are previewed as removable thumbnails before sending, and render as thumbnails in the conversation transcript. Signed-off-by: re-pixel <relja.brdar@gmail.com>
3a9a4e1 to
94a6ec2
Compare
Signed-off-by: re-pixel <relja.brdar@gmail.com>
Signed-off-by: re-pixel <relja.brdar@gmail.com>
Signed-off-by: re-pixel <relja.brdar@gmail.com>
Contributor
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 06892b9. Configure here.
… responses under gRPC size limits Signed-off-by: re-pixel <relja.brdar@gmail.com>
908ab08 to
ba7c1a6
Compare
# Conflicts: # db/structure.sql
# Conflicts: # pkg/agents/service_test.go
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Demo: https://drive.google.com/file/d/1bcmEU7CWOvt3aJRHPBG02LW93rtRkLmK/view?usp=sharing
Resolves #5136. Lets users paste or attach images (e.g. a screenshot of an error) in the canvas agent chat so the agent can act on them.
Backend
AgentChatImage { media_type, data }proto type;imagesfield added to the send-message request and stored message.imagesjsonb column onagent_session_messages(migration included).imagecontent blocks alongside the text.GET /api/v1/agents/chats/{chatId}/messages/{messageId}/images/{index}, scoped to the session's owner). This keeps history responses small enough to stay under the gRPC/HTTP response size limit.Frontend
<img src>(URL built from chat/message/index), rather than inline data.