Live browser preview of interpreted scenes (WASM)#255
Merged
Conversation
Compile the interpreted-scene runtime (interpreter + pixie + QuickJS) to
WebAssembly and add a "Live preview" button next to each scene's
activate/preview buttons that opens a modal and runs the scene entirely in
the browser.
Runtime (Nim):
- New wasm entry point src/wasm/wasm_main.nim (gated on -d:frameosWasm),
mirroring the ESP32 embedded runtime. Exports a small C ABI: init,
load_scenes, select_scene, render, event, scene_state/info, next_sleep.
- Reuse the existing frameosEmbedded seams: every host-only feature already
excluded on embedded (networking, child processes, threads, filesystem,
ImageMagick/exiftool) is now also excluded on wasm. apps_nim.py generator
gains the same gate.
- Must exit via emscripten_exit_with_live_runtime(): ARC/ORC otherwise frees
all module-level globals when main() returns, aborting the first call.
Build:
- tools/build_wasm.sh compiles QuickJS + the runtime with emscripten into
frontend/public/frameos-wasm/. GROWABLE_ARRAYBUFFERS=0 keeps memory
non-resizable so Chrome's TextDecoder accepts heap views.
- mprocs "wasm" pane builds it in dev; Dockerfile builds it (via emsdk)
before the frontend bundle. Bundle is gitignored.
Frontend:
- livePreviewLogic spawns a module Worker (preview-worker.js) that drives the
wasm module, paints frames to a canvas, and exposes scene event nodes as
clickable buttons plus live state/log readouts.
Secrets + external HTTP:
- frameos_wasm_init takes a settingsJson param so apps read real API keys
from frameConfig.settings. Frontend fetches them from a new project-authed
GET /frames/{id}/scene_preview_settings (get_frame_json settings).
- Data apps that fetch external URLs are routed through a same-origin,
project-authed, SSRF-guarded POST /frames/{id}/scene_preview_proxy so they
work despite CORS, like the device fetching server-side.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Scene state and runtime log now use a dark terminal surface (like the real Logs panel) instead of a light-gray box that looked washed out on the white modal in light mode. The terminal is theme-independent, so it reads well in both light and dark. - Match the real logs' text coloring: default lines light, scene events blue, error lines red; muted keys in the state box. - Auto-scroll the runtime log to the bottom as new lines arrive (ref+effect, same intent as Logs.tsx). - Rename the button and modal title from "Live preview" to "In-browser preview". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The proxy hardcoded a 30s timeout, so slow upstreams (openaiImage asks for 300s — DALL-E generation) failed with an empty-message ReadTimeout surfaced as "Proxy fetch failed:". Forward the runtime's requested timeoutMs through the envelope and use it (clamped to 5..300s, short connect timeout). Also include the exception type in the error detail so timeouts are diagnosable. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Docker "app-builder" stage runs build_wasm.sh -> makeapploaders.py, which
loads the Nim codegen from backend/app/codegen. That dir wasn't copied into
the stage, so both the Docker image and ESP32 firmware builds failed with
FileNotFoundError. Copy backend/app/codegen (self-contained, stdlib-only)
before the wasm build step.
- Update test_embedded_unavailable_apps_are_guarded_in_registry to match the
generator's new gate ("frameosEmbedded or frameosWasm" / "not available on
this build target").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "New chat" button created a fresh chat from the default context, which on the scene drawer resolved to the frame rather than the scene you were viewing — silently switching a scene chat into a frame chat. Clear resets the active conversation while keeping its context. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move Scene settings to the top of the utility toolbar (above the panel buttons) and move AI chat to the bottom. Swap Apps to a puzzle-piece icon (was a code bracket that clashed with Source) and Events to a bolt icon (was a generic bullet list). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a "Select multiple scenes" toggle under the scene-list "..." menu. When enabled, scene tiles show checkboxes and a Delete button appears to remove the ticked scenes. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Show each box's pixel dimensions in the cell (below "Drop scene" when empty, first in the label row when filled) and let you type exact width/height; maps back to the controlling split's ratios. - Add "Split vertically" / "Split horizontally" on a selected cell, plus a Remove button that collapses a panel back into its neighbors. - Drag one cell onto another to swap their scene + state. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Pass the user's entered field values (buildNextState) into openLivePreview so the in-browser WASM preview reflects form input, not stored defaults - Correct the preview modal blurb: the backend proxy is a browser CORS workaround; the device fetches external URLs directly - Shrink the "..." dropdown and "Open editor"/"Edit split" buttons on the frames home scene row Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… modal - Try-scene modal now has two submit targets: "Preview in browser" (WASM live preview, scenes passed explicitly since they're not installed) and "Preview on frame" (existing upload_scenes path); always opens so the user can pick a target - Template cards: "compiled" tag when no scene can run interpreted, less rounded thumbnails, favourite is a bare star at the card's top right - Live preview modal: near-full viewport height and wider (Modal gained panelClassName/bodyClassName overrides), runtime log styled like the real logs panel (timestamps, colors) with stick-to-bottom scrolling, scene state filterable by public/private (private hidden by default) - livePreviewLogic resolves scene metadata from explicitly-passed scenes so template previews get their name, events and field access info - Checkbox: label text is a span, not a nested <label>, so clicking the label toggles the box Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
openLivePreview stores the scene id under #livePreview= (frame scenes only — template previews can't be restored since their scenes aren't installed), closeLivePreview clears it, and ExpandedScene reopens the preview on mount when the hash matches its scene. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Canvas renders bare: no background box, padding, border or rounding
(loading overlay keeps a translucent scrim so the spinner stays legible)
- Runtime log parses the JSON lines the wasm runtime emits and renders
them like the real logs panel: event name highlighted, remaining keys
as key=value pairs; non-JSON lines fall back to the raw string
- "Edit" button next to the scene state checkboxes opens a second modal
(preview stays open underneath) with the scene's public fields
(showIf-aware); submitting dispatches setSceneState {state, render}
to the wasm worker, updating the running preview in place
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Configured gpio_buttons render as buttons in the preview controls row,
labeled by their label (pin number in the hover title). Clicking sends
the same "button" event the device's GPIO driver emits on a press
({pin, label, level: 0}). Scene-event entries duplicating a configured
GPIO button label are hidden.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New SceneActionsButton: a primary button that runs the selected action plus a chevron dropdown listing every option (activate / preview on frame / preview in browser) with descriptions. Picking an option runs it immediately and becomes the globally remembered default (persisted via kea-localstorage in sceneActionsLogic); until then each context preselects what the old standalone buttons did (preview-on-frame when the scene has changes, activate otherwise). Replaces the activate/preview/eye button clusters in ExpandedScene (scene control panel on the home page, scene workspace state panel) and the preview/activate buttons in the Diagram panel toolbar, which also gains the in-browser preview (modal now rendered from the toolbar too). The floating diagram toolbar variant is unchanged. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…vent The preview modal can be hosted from several components (scene card, diagram toolbar, template row) mounted at the same time; each rendered its own dialog, and with two identical stacked dialogs a click inside one counts as an outside-click for the other, closing the preview. A per-frame ownership registry now lets only one mounted host render the dialog, handing off if the owner unmounts. Also hide "button" scene-event entries from the controls row: configured GPIO buttons already cover them, and an unlabeled "button" entry sends an event no handler can distinguish. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…title Picking an option from the split-button dropdown no longer runs it — it just becomes the button's remembered action; click the button to run. Also "In-browser preview" → "Browser preview" in the modal title. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Clicking "Re-render" ran the scene but logged nothing, so it looked like nothing happened — especially when cached apps returned an identical image. frameos_wasm_render now logs the same render:scene and render:done (with ms) events runner.nim emits on the device. Also picks up the regenerated apps.nim asset from the build. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…build New wasmUnsupportedApps selector scans the previewed scene and every scene it references for apps excluded from the wasm bundle (chromium screenshot, rstp snapshot — child processes / external binaries) and the modal shows an amber notice naming them, so a failing node isn't a mystery. The list mirrors the exclusions in frameos/src/apps/apps.nim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
warn about localImage - "Preview on frame" button (after the state Edit button; in the controls row when the scene has no state) sends the scene to the frame via previewScene with the preview's current public state. Hidden for template previews, whose scenes aren't installed. - Clicking the canvas opens the current image as a PNG in a new tab (window opened synchronously so popup blockers allow it, blob URL filled in after encoding) - data/localImage added to the unsupported-apps notice: it's in the wasm build but reads the frame's local storage Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Always next to the re-render/event/GPIO buttons instead of moving between the state row and controls row. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…k updates Repository templates now carry a stable id (directory slug) and a version (explicit template.json version, or a content hash of the scenes). Installing a repo template stamps each scene with an origin (repository, template, template-side scene id, version). When the source template's version changes, the UI offers an update: an info badge on workspace scene tiles, an 'update' tag in the scene sidebar, an 'Update scene' entry in both scene dropdowns, and a banner in the scene info panel. Updating replaces the scene's content in place while keeping its id, name and start-on-boot flag, so links from other scenes and schedules keep working; multi-scene templates update as a group with cross-scene references remapped to the installed ids. The update lands in the frame form as an unsaved change, so the normal save/deploy flow applies. Also fixes a latent bug in duplicateScenes where non-scene config fields were dropped from source-node config.json during id remapping. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
applyTemplate now appends the template's scenes to the frame form (optionally opening the scene drawer) without persisting; applyTemplateAndSave and the persistOnInstall flag are gone. Blank scene creation (createBlankScene, formerly createBlankSceneAndSave) and AI-generated scenes follow the same rule. The user reviews and saves/deploys through the normal flow, like any other frame change. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Clicking a template's preview button no longer opens the state-fields form modal; it launches the in-browser WASM preview immediately with the scene's default public state. The browser preview modal's 'Preview on frame' button now also works for template previews: previewScene accepts an explicit scenes list for scenes not installed on the frame, which also routes template previews through the embedded USB path where available. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… always visible Template previews now carry their source template (and repository) into livePreviewLogic, so the browser preview modal can offer 'Add to frame' — or a disabled 'Added' when a scene with the template's name is already on the frame. 'Preview on frame' and the new button sit in the Scene state header row next to the Edit button, and that header row is now always shown even when the scene has no state. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hangelog Frame sharing: a new Home Assistant sync service (backend/app/ha) publishes every non-archived frame as an MQTT-discovered device — image entity with the latest render, status/scene/last-seen sensors — plus a hub sensor with the frame count and names. Frame events are forwarded to the HA event bus as `frameos_event` and to per-frame MQTT topics. Archiving or deleting a frame removes its device. The service runs as a singleton task in the arq worker, reacts to the existing broadcast_channel events, and reloads via a new `ha_sync` control channel. As an add-on, the HA connection and Mosquitto broker are auto-discovered through the Supervisor; standalone installs use the URL + token and new MQTT fields in Settings → Home Assistant, where a new "Share frames with Home Assistant" toggle and "Save & sync now" button live. get_frame_json now strips sync internals (MQTT credentials, flags) from the homeAssistant settings deployed to frames. Release workflow: release notes are generated once in a `release-notes` job; `update-addon-repo` prepends them to the add-on repo's CHANGELOG.md (Home Assistant's changelog tab was empty because that file never existed) and `github-release` reuses the same artifact. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nails
The "Update scene" feature made WorkspaceSceneDropDown and the home
dashboard scene tiles mount scenesLogic, which connects to controlLogic
and fetches /frames/{id}/states per frame on the frames home. Move
sceneUpdateVersions + updateSceneFromRepo into a new sceneUpdatesLogic
that only needs frameLogic and repositoriesModel; scenesLogic re-exposes
them for the frame workspace.
Repository templates now carry an id (their directory name), which made
Template.tsx request /templates/{id}/image — a saved-templates endpoint
that 404s for repo templates and failed the add-scene visual tests on
the frontend-error assertion. Resolve the repository image URL first and
only use templates/{id} when there is no repository.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Run interpreted scenes directly in the browser: FrameOS now ships a WebAssembly build of the scene runtime, and every scene, template, and scene-editor toolbar gets an In-browser preview that renders the scene client-side — rendering, code nodes (QuickJS), timers, and interactive events included. Around that headline feature, this PR also standardizes the scene action buttons, adds one-click scene updates from repositories, and lets FrameOS share frames with Home Assistant.
In-browser preview
livePreviewLogicspawns a module Web Worker (preview-worker.js) that drives the wasm module and paints frames to a canvas. The modal shows the scene's event nodes as clickable buttons, the frame's GPIO buttons, terminal-styled parsed logs (render:scene/render:done, like the device runner), and an editable live scene-state panel.localImage) get a warning; compiled-only templates are tagged, since only interpreted scenes can run in the preview.Runtime (Nim → WASM)
frameos/src/wasm/wasm_main.nim(gated on-d:frameosWasm), modeled on the ESP32 embedded runtime. Small C ABI:init,load_scenes,select_scene,render,event,scene_state/scene_info,next_sleep.frameosEmbeddedseams: every host-only feature already excluded on embedded (networking, child processes, threads, filesystem, ImageMagick/exiftool) is also excluded on wasm. Theapps_nim.pygenerator gains the same gate, so process-spawning apps (chromium screenshot, RTSP snapshot) are compiled out.emscripten_exit_with_live_runtime()— ARC/ORC otherwise emits destructors for all module-level globals whenmain()returns, freeing the scene/asset tables and aborting the first exported call.Build & shipping
frameos/tools/build_wasm.shcompiles QuickJS + the runtime with emscripten intofrontend/public/frameos-wasm/(bundle is gitignored).-sGROWABLE_ARRAYBUFFERS=0keeps memory non-resizable so Chrome'sTextDecoderaccepts heap views.wasmpane inmprocs.yamlbuilds it. Prod: theDockerfileinstalls emsdk and builds it before the frontend bundle (which copiespublic/intodist/).Secrets & external HTTP
frameos_wasm_inittakes asettingsJsonparam so apps read real API keys fromframeConfig.settings. The frontend fetches them from a new project-authedGET /frames/{id}/scene_preview_settings(returnsget_frame_json(...)["settings"]— the same per-app secret set the device receives).POST /frames/{id}/scene_preview_proxy(honoring the app's request timeout), so they work despite browser CORS — the backend fetches server-side and mirrors the upstream status/body, exactly like the device.Scene management UX
Scene updates from repositories
repository/template/sceneId+ a version that's an explicittemplate.jsonversion or a content hash of the scenes), so scene dropdowns and tiles offer a one-click Update scene that replaces the scene's content in place — multi-scene templates update as a group with ids preserved so links keep working.sceneUpdatesLogic, deliberately separate fromscenesLogic/controlLogicso the frames home doesn't fetch per-frame state just to render dropdowns.Home Assistant
backend/app/ha) publishes every non-archived frame as an MQTT-discovered HA device — image entity with the latest render, status/scene/last-seen sensors, plus a hub sensor — and forwards frame events to the HA event bus asframeos_event. Runs as a singleton arq worker task, reacting to existing broadcast events; add-on installs auto-discover the Supervisor connection and Mosquitto broker, standalone installs configure URL/token/MQTT in Settings → Home Assistant ("Share frames with Home Assistant" toggle + "Save & sync now").get_frame_jsonstrips the sync internals from settings deployed to frames.CHANGELOG.md(HA's changelog tab was empty because that file never existed); the GitHub release reuses the same artifact.Verification
1 → 111), renders a 1920×1080 frame (exercises memory growth + theTextDecoderfix), and renders adownloadImage → render/imagescene through the proxy (real photo, not an error frame).data/openaiText(missing-key error only when the key is absent). SSRF guard unit-checked (loopback/private blocked, public allowed). HA discovery/sync covered by backend tests.tsc+ production esbuild pass; the full frontend e2e + visual suite is green (including the frames-home "no/statesrequests" guard and the add-scene drawer, whose repository-template thumbnails now load).Note: device-only apps (chromium screenshot, camera snapshot) remain unavailable in the preview by design.
🤖 Generated with Claude Code