Skip to content

Live browser preview of interpreted scenes (WASM)#255

Merged
mariusandra merged 37 commits into
mainfrom
wasm
Jul 8, 2026
Merged

Live browser preview of interpreted scenes (WASM)#255
mariusandra merged 37 commits into
mainfrom
wasm

Conversation

@mariusandra

@mariusandra mariusandra commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Run interpreted scenes directly in the browser: FrameOS now ships a WebAssembly build of the scene runtime, and every scene, template, and scene-editor toolbar gets an In-browser preview that renders the scene client-side — rendering, code nodes (QuickJS), timers, and interactive events included. Around that headline feature, this PR also standardizes the scene action buttons, adds one-click scene updates from repositories, and lets FrameOS share frames with Home Assistant.

image image

In-browser preview

  • livePreviewLogic spawns a module Web Worker (preview-worker.js) that drives the wasm module and paints frames to a canvas. The modal shows the scene's event nodes as clickable buttons, the frame's GPIO buttons, terminal-styled parsed logs (render:scene / render:done, like the device runner), and an editable live scene-state panel.
  • The same modal works for templates: the template preview button opens the browser preview directly (scenes are passed explicitly since they aren't installed), with Add to frame and Preview on frame buttons next to it. Form field values are sent as the scene's initial state.
  • Open state persists in the URL hash across reloads. Scenes that use apps missing from the wasm build (or localImage) get a warning; compiled-only templates are tagged, since only interpreted scenes can run in the preview.

Runtime (Nim → WASM)

  • New entry point frameos/src/wasm/wasm_main.nim (gated on -d:frameosWasm), modeled on the ESP32 embedded runtime. Small C ABI: init, load_scenes, select_scene, render, event, scene_state/scene_info, next_sleep.
  • Reuses the existing frameosEmbedded seams: every host-only feature already excluded on embedded (networking, child processes, threads, filesystem, ImageMagick/exiftool) is also excluded on wasm. The apps_nim.py generator gains the same gate, so process-spawning apps (chromium screenshot, RTSP snapshot) are compiled out.
  • Gotcha: the module must end with emscripten_exit_with_live_runtime() — ARC/ORC otherwise emits destructors for all module-level globals when main() returns, freeing the scene/asset tables and aborting the first exported call.

Build & shipping

  • frameos/tools/build_wasm.sh compiles QuickJS + the runtime with emscripten into frontend/public/frameos-wasm/ (bundle is gitignored). -sGROWABLE_ARRAYBUFFERS=0 keeps memory non-resizable so Chrome's TextDecoder accepts heap views.
  • Dev: a wasm pane in mprocs.yaml builds it. Prod: the Dockerfile installs emsdk and builds it before the frontend bundle (which copies public/ into dist/).

Secrets & external HTTP

  • frameos_wasm_init takes a settingsJson param so apps read real API keys from frameConfig.settings. The frontend fetches them from a new project-authed GET /frames/{id}/scene_preview_settings (returns get_frame_json(...)["settings"] — the same per-app secret set the device receives).
  • Data apps that fetch external URLs are routed through a same-origin, project-authed, SSRF-guarded POST /frames/{id}/scene_preview_proxy (honoring the app's request timeout), so they work despite browser CORS — the backend fetches server-side and mirrors the upstream status/body, exactly like the device.

Scene management UX

  • Scene actions (activate / preview on frame / preview in browser) are standardized into one split-button with a dropdown; picking an option arms the button globally, clicking runs it.
  • Adding scenes to a frame no longer saves the frame automatically.
  • Homepage scene list gains multiselect + bulk delete; split scenes get precise box sizes with split/swap/remove panel controls.
  • Events panel: custom events can be dragged onto the scene as listen/dispatch nodes. Scene toolbar icons clarified; chat panel keeps only Clear.

Scene updates from repositories

  • Installed scenes track their repo origin (repository/template/sceneId + a version that's an explicit template.json version or a content hash of the scenes), so scene dropdowns and tiles offer a one-click Update scene that replaces the scene's content in place — multi-scene templates update as a group with ids preserved so links keep working.
  • The update UI lives in a lightweight sceneUpdatesLogic, deliberately separate from scenesLogic/controlLogic so the frames home doesn't fetch per-frame state just to render dropdowns.

Home Assistant

  • New sync service (backend/app/ha) publishes every non-archived frame as an MQTT-discovered HA device — image entity with the latest render, status/scene/last-seen sensors, plus a hub sensor — and forwards frame events to the HA event bus as frameos_event. Runs as a singleton arq worker task, reacting to existing broadcast events; add-on installs auto-discover the Supervisor connection and Mosquitto broker, standalone installs configure URL/token/MQTT in Settings → Home Assistant ("Share frames with Home Assistant" toggle + "Save & sync now"). get_frame_json strips the sync internals from settings deployed to frames.
  • Release workflow: release notes are generated once and prepended to the add-on repo's CHANGELOG.md (HA's changelog tab was empty because that file never existed); the GitHub release reuses the same artifact.

Verification

  • Native FrameOS and ESP32 embedded builds still compile clean (shared gates unchanged).
  • Headless Chromium end-to-end: renders the Counter sample (button events drive state 1 → 111), renders a 1920×1080 frame (exercises memory growth + the TextDecoder fix), and renders a downloadImage → render/image scene through the proxy (real photo, not an error frame).
  • Secret plumbing proven via data/openaiText (missing-key error only when the key is absent). SSRF guard unit-checked (loopback/private blocked, public allowed). HA discovery/sync covered by backend tests.
  • Frontend kea-typegen + tsc + production esbuild pass; the full frontend e2e + visual suite is green (including the frames-home "no /states requests" guard and the add-scene drawer, whose repository-template thumbnails now load).

Note: device-only apps (chromium screenshot, camera snapshot) remain unavailable in the preview by design.

🤖 Generated with Claude Code

mariusandra and others added 30 commits July 7, 2026 03:00
Compile the interpreted-scene runtime (interpreter + pixie + QuickJS) to
WebAssembly and add a "Live preview" button next to each scene's
activate/preview buttons that opens a modal and runs the scene entirely in
the browser.

Runtime (Nim):
- New wasm entry point src/wasm/wasm_main.nim (gated on -d:frameosWasm),
  mirroring the ESP32 embedded runtime. Exports a small C ABI: init,
  load_scenes, select_scene, render, event, scene_state/info, next_sleep.
- Reuse the existing frameosEmbedded seams: every host-only feature already
  excluded on embedded (networking, child processes, threads, filesystem,
  ImageMagick/exiftool) is now also excluded on wasm. apps_nim.py generator
  gains the same gate.
- Must exit via emscripten_exit_with_live_runtime(): ARC/ORC otherwise frees
  all module-level globals when main() returns, aborting the first call.

Build:
- tools/build_wasm.sh compiles QuickJS + the runtime with emscripten into
  frontend/public/frameos-wasm/. GROWABLE_ARRAYBUFFERS=0 keeps memory
  non-resizable so Chrome's TextDecoder accepts heap views.
- mprocs "wasm" pane builds it in dev; Dockerfile builds it (via emsdk)
  before the frontend bundle. Bundle is gitignored.

Frontend:
- livePreviewLogic spawns a module Worker (preview-worker.js) that drives the
  wasm module, paints frames to a canvas, and exposes scene event nodes as
  clickable buttons plus live state/log readouts.

Secrets + external HTTP:
- frameos_wasm_init takes a settingsJson param so apps read real API keys
  from frameConfig.settings. Frontend fetches them from a new project-authed
  GET /frames/{id}/scene_preview_settings (get_frame_json settings).
- Data apps that fetch external URLs are routed through a same-origin,
  project-authed, SSRF-guarded POST /frames/{id}/scene_preview_proxy so they
  work despite CORS, like the device fetching server-side.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Scene state and runtime log now use a dark terminal surface (like the real
  Logs panel) instead of a light-gray box that looked washed out on the white
  modal in light mode. The terminal is theme-independent, so it reads well in
  both light and dark.
- Match the real logs' text coloring: default lines light, scene events blue,
  error lines red; muted keys in the state box.
- Auto-scroll the runtime log to the bottom as new lines arrive (ref+effect,
  same intent as Logs.tsx).
- Rename the button and modal title from "Live preview" to "In-browser
  preview".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The proxy hardcoded a 30s timeout, so slow upstreams (openaiImage asks for
300s — DALL-E generation) failed with an empty-message ReadTimeout surfaced
as "Proxy fetch failed:". Forward the runtime's requested timeoutMs through
the envelope and use it (clamped to 5..300s, short connect timeout). Also
include the exception type in the error detail so timeouts are diagnosable.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Docker "app-builder" stage runs build_wasm.sh -> makeapploaders.py, which
  loads the Nim codegen from backend/app/codegen. That dir wasn't copied into
  the stage, so both the Docker image and ESP32 firmware builds failed with
  FileNotFoundError. Copy backend/app/codegen (self-contained, stdlib-only)
  before the wasm build step.
- Update test_embedded_unavailable_apps_are_guarded_in_registry to match the
  generator's new gate ("frameosEmbedded or frameosWasm" / "not available on
  this build target").

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The "New chat" button created a fresh chat from the default context,
which on the scene drawer resolved to the frame rather than the scene
you were viewing — silently switching a scene chat into a frame chat.
Clear resets the active conversation while keeping its context.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Move Scene settings to the top of the utility toolbar (above the panel
buttons) and move AI chat to the bottom. Swap Apps to a puzzle-piece
icon (was a code bracket that clashed with Source) and Events to a bolt
icon (was a generic bullet list).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a "Select multiple scenes" toggle under the scene-list "..." menu.
When enabled, scene tiles show checkboxes and a Delete button appears
to remove the ticked scenes.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Show each box's pixel dimensions in the cell (below "Drop scene" when
  empty, first in the label row when filled) and let you type exact
  width/height; maps back to the controlling split's ratios.
- Add "Split vertically" / "Split horizontally" on a selected cell,
  plus a Remove button that collapses a panel back into its neighbors.
- Drag one cell onto another to swap their scene + state.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- Pass the user's entered field values (buildNextState) into openLivePreview
  so the in-browser WASM preview reflects form input, not stored defaults
- Correct the preview modal blurb: the backend proxy is a browser CORS
  workaround; the device fetches external URLs directly
- Shrink the "..." dropdown and "Open editor"/"Edit split" buttons on the
  frames home scene row

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… modal

- Try-scene modal now has two submit targets: "Preview in browser" (WASM
  live preview, scenes passed explicitly since they're not installed) and
  "Preview on frame" (existing upload_scenes path); always opens so the
  user can pick a target
- Template cards: "compiled" tag when no scene can run interpreted, less
  rounded thumbnails, favourite is a bare star at the card's top right
- Live preview modal: near-full viewport height and wider (Modal gained
  panelClassName/bodyClassName overrides), runtime log styled like the
  real logs panel (timestamps, colors) with stick-to-bottom scrolling,
  scene state filterable by public/private (private hidden by default)
- livePreviewLogic resolves scene metadata from explicitly-passed scenes
  so template previews get their name, events and field access info
- Checkbox: label text is a span, not a nested <label>, so clicking the
  label toggles the box

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
openLivePreview stores the scene id under #livePreview= (frame scenes
only — template previews can't be restored since their scenes aren't
installed), closeLivePreview clears it, and ExpandedScene reopens the
preview on mount when the hash matches its scene.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Canvas renders bare: no background box, padding, border or rounding
  (loading overlay keeps a translucent scrim so the spinner stays legible)
- Runtime log parses the JSON lines the wasm runtime emits and renders
  them like the real logs panel: event name highlighted, remaining keys
  as key=value pairs; non-JSON lines fall back to the raw string
- "Edit" button next to the scene state checkboxes opens a second modal
  (preview stays open underneath) with the scene's public fields
  (showIf-aware); submitting dispatches setSceneState {state, render}
  to the wasm worker, updating the running preview in place

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Configured gpio_buttons render as buttons in the preview controls row,
labeled by their label (pin number in the hover title). Clicking sends
the same "button" event the device's GPIO driver emits on a press
({pin, label, level: 0}). Scene-event entries duplicating a configured
GPIO button label are hidden.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
New SceneActionsButton: a primary button that runs the selected action
plus a chevron dropdown listing every option (activate / preview on
frame / preview in browser) with descriptions. Picking an option runs it
immediately and becomes the globally remembered default (persisted via
kea-localstorage in sceneActionsLogic); until then each context
preselects what the old standalone buttons did (preview-on-frame when
the scene has changes, activate otherwise).

Replaces the activate/preview/eye button clusters in ExpandedScene
(scene control panel on the home page, scene workspace state panel) and
the preview/activate buttons in the Diagram panel toolbar, which also
gains the in-browser preview (modal now rendered from the toolbar too).
The floating diagram toolbar variant is unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…vent

The preview modal can be hosted from several components (scene card,
diagram toolbar, template row) mounted at the same time; each rendered
its own dialog, and with two identical stacked dialogs a click inside
one counts as an outside-click for the other, closing the preview. A
per-frame ownership registry now lets only one mounted host render the
dialog, handing off if the owner unmounts.

Also hide "button" scene-event entries from the controls row: configured
GPIO buttons already cover them, and an unlabeled "button" entry sends
an event no handler can distinguish.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…title

Picking an option from the split-button dropdown no longer runs it —
it just becomes the button's remembered action; click the button to run.
Also "In-browser preview" → "Browser preview" in the modal title.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Clicking "Re-render" ran the scene but logged nothing, so it looked like
nothing happened — especially when cached apps returned an identical
image. frameos_wasm_render now logs the same render:scene and
render:done (with ms) events runner.nim emits on the device.

Also picks up the regenerated apps.nim asset from the build.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…build

New wasmUnsupportedApps selector scans the previewed scene and every
scene it references for apps excluded from the wasm bundle (chromium
screenshot, rstp snapshot — child processes / external binaries) and the
modal shows an amber notice naming them, so a failing node isn't a
mystery. The list mirrors the exclusions in frameos/src/apps/apps.nim.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
warn about localImage

- "Preview on frame" button (after the state Edit button; in the
  controls row when the scene has no state) sends the scene to the
  frame via previewScene with the preview's current public state.
  Hidden for template previews, whose scenes aren't installed.
- Clicking the canvas opens the current image as a PNG in a new tab
  (window opened synchronously so popup blockers allow it, blob URL
  filled in after encoding)
- data/localImage added to the unsupported-apps notice: it's in the
  wasm build but reads the frame's local storage

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Always next to the re-render/event/GPIO buttons instead of moving
between the state row and controls row.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…k updates

Repository templates now carry a stable id (directory slug) and a version
(explicit template.json version, or a content hash of the scenes). Installing
a repo template stamps each scene with an origin (repository, template,
template-side scene id, version). When the source template's version changes,
the UI offers an update: an info badge on workspace scene tiles, an 'update'
tag in the scene sidebar, an 'Update scene' entry in both scene dropdowns,
and a banner in the scene info panel.

Updating replaces the scene's content in place while keeping its id, name
and start-on-boot flag, so links from other scenes and schedules keep
working; multi-scene templates update as a group with cross-scene references
remapped to the installed ids. The update lands in the frame form as an
unsaved change, so the normal save/deploy flow applies.

Also fixes a latent bug in duplicateScenes where non-scene config fields
were dropped from source-node config.json during id remapping.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
mariusandra and others added 7 commits July 8, 2026 02:29
applyTemplate now appends the template's scenes to the frame form (optionally
opening the scene drawer) without persisting; applyTemplateAndSave and the
persistOnInstall flag are gone. Blank scene creation (createBlankScene,
formerly createBlankSceneAndSave) and AI-generated scenes follow the same
rule. The user reviews and saves/deploys through the normal flow, like any
other frame change.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Clicking a template's preview button no longer opens the state-fields form
modal; it launches the in-browser WASM preview immediately with the scene's
default public state. The browser preview modal's 'Preview on frame' button
now also works for template previews: previewScene accepts an explicit scenes
list for scenes not installed on the frame, which also routes template
previews through the embedded USB path where available.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… always visible

Template previews now carry their source template (and repository) into
livePreviewLogic, so the browser preview modal can offer 'Add to frame' —
or a disabled 'Added' when a scene with the template's name is already on
the frame. 'Preview on frame' and the new button sit in the Scene state
header row next to the Edit button, and that header row is now always
shown even when the scene has no state.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hangelog

Frame sharing: a new Home Assistant sync service (backend/app/ha) publishes
every non-archived frame as an MQTT-discovered device — image entity with the
latest render, status/scene/last-seen sensors — plus a hub sensor with the
frame count and names. Frame events are forwarded to the HA event bus as
`frameos_event` and to per-frame MQTT topics. Archiving or deleting a frame
removes its device. The service runs as a singleton task in the arq worker,
reacts to the existing broadcast_channel events, and reloads via a new
`ha_sync` control channel. As an add-on, the HA connection and Mosquitto
broker are auto-discovered through the Supervisor; standalone installs use the
URL + token and new MQTT fields in Settings → Home Assistant, where a new
"Share frames with Home Assistant" toggle and "Save & sync now" button live.

get_frame_json now strips sync internals (MQTT credentials, flags) from the
homeAssistant settings deployed to frames.

Release workflow: release notes are generated once in a `release-notes` job;
`update-addon-repo` prepends them to the add-on repo's CHANGELOG.md (Home
Assistant's changelog tab was empty because that file never existed) and
`github-release` reuses the same artifact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…nails

The "Update scene" feature made WorkspaceSceneDropDown and the home
dashboard scene tiles mount scenesLogic, which connects to controlLogic
and fetches /frames/{id}/states per frame on the frames home. Move
sceneUpdateVersions + updateSceneFromRepo into a new sceneUpdatesLogic
that only needs frameLogic and repositoriesModel; scenesLogic re-exposes
them for the frame workspace.

Repository templates now carry an id (their directory name), which made
Template.tsx request /templates/{id}/image — a saved-templates endpoint
that 404s for repo templates and failed the add-scene visual tests on
the frontend-error assertion. Resolve the repository image URL first and
only use templates/{id} when there is no repository.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@mariusandra mariusandra merged commit e50adb9 into main Jul 8, 2026
@mariusandra mariusandra deleted the wasm branch July 8, 2026 09:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant