feat(vulkan): wolf:vulkan image (native Vulkan NV12 encode) + Fedora switch - #450
Draft
JBailes wants to merge 23 commits into
Draft
feat(vulkan): wolf:vulkan image (native Vulkan NV12 encode) + Fedora switch#450JBailes wants to merge 23 commits into
JBailes wants to merge 23 commits into
Conversation
JBailes
added a commit
that referenced
this pull request
Jun 27, 2026
Per review, keep this PR focused on the image smoke-test harness and the Wayland-socket-wait feature + tests. The wholesale Ubuntu->Fedora image switch (gpu-drivers/gstreamer/wolf Dockerfiles, embedded PulseAudio + supervisord, the Fedora devcontainer) is moved to the Wolf vulkan draft PR (#450), where it pairs with the native-Vulkan wolf:vulkan image. Reverts those files to stable here; the boost::json migration, the pulse ready-state guard and the sign.hpp/cstdint compile fix stay (distro-independent).
Add a VULKAN encoder type wired through the config/streaming path:
- VULKAN enum + encoder_type("vulkan") and a vulkanh264enc h264_encoders block
- producer_buffer_caps video/x-raw(memory:VulkanImage),format=NV12 for the
zero-copy path, and video_params_zero_copy="queue" so the encoder consumes the
producer's NV12 VulkanImage directly (no videoscale/convert/rate, which can't
negotiate the VulkanImage memory feature)
- waylanddisplaysrc vulkan=true when the buffer caps ask for VulkanImage
- WOLF_VULKAN_QUALITY / _REF_FRAMES / _B_FRAMES tunables (defaults tuned for
low-latency streaming: quality=0, num-ref-frames=1, b-frames=0)
Move gpu-drivers, gstreamer and wolf images from Ubuntu/apt to Fedora/dnf, building FROM ghcr.io/games-on-whales/base-app:fedora (the rolling GOW Fedora tag). Wolf runs its own PulseAudio server supervised by supervisord (the Fedora base ships a conflicting pipewire-pulseaudio shim, swapped out with --allowerasing), removing the external WolfPulseAudio sidecar and its startup race. wolf.cpp gains graceful SIGINT shutdown to match supervisord's stopsignal=INT/stopwaitsecs. Bundles the linked libicu and registers /usr/local/lib64 for the gst plugin's runtime libs. Moved out of #431 so that PR keeps only the smoke-test harness + Wayland resolution tests.
docker/wolf.vulkan.Dockerfile builds Wolf on top of ghcr.io/games-on-whales/gst-wayland-display:vulkan (patched GStreamer 1.28.4 vulkan-video + the plugin under /opt/gst), so the heavy gst-Vulkan + plugin build happens once in that image and here we only compile Wolf. A "Build Wolf (Vulkan)" step in docker-build.yml publishes ghcr.io/<owner>/wolf:vulkan.
JBailes
force-pushed
the
feat/vulkan-image
branch
from
June 27, 2026 06:35
d4d00b2 to
bbdac70
Compare
JBailes
added a commit
that referenced
this pull request
Jun 27, 2026
Add bin/test-image.sh — a 3-layer smoke test for the built wolf image: structural (docker inspect), cont-init smoke, and docker/tests/smoke.sh (wolf + fake-udev present and ldd-clean, the custom waylanddisplaysrc plugin loads, key env vars set), with bin/tests/lib.sh assertion helpers. Wired into docker-build.yml: PR builds load the image locally and run the harness when registry secrets are unavailable. Pared down to just the smoke harness — the Fedora switch moved to #450, and the embedded-PulseAudio (#422), wayland-socket-wait (#418), pulse fix (#420) and boost::json migration have since merged to stable independently.
… encoder When an AMD GPU is paired with a Vulkan Video encoder, enable RADV's low-latency encode mode (Mesa 26.1+) on Wolf's own process environment, where the encoder pipeline runs. User-set RADV_PERFTEST is left untouched.
RADV's Vulkan HEVC encoder corrupts the bottom/right partial coding-tree block whenever a stream dimension isn't a multiple of the 64px CTB (every standard height: 1080/720/2160) — a noise/green bar across that edge, while H.264 and the other HEVC encoders are unaffected. When the selected HEVC pipeline is vulkanh265enc, round the requested resolution up to whole CTBs so the encoder only ever sees full coding blocks; the client scales the slightly-larger frame back to its viewport. Scoped to the vulkanh265enc pipeline so other encoders keep their exact requested resolution.
WOLF_HDR=TRUE selects the P010 (10-bit) producer buffer format on the native Vulkan path; the producer pipeline then passes hdr=true to waylanddisplaysrc (BT.2020 PQ + mastering-display/content-light metadata), and vulkanh265enc emits a Main-10 HDR10 stream. Default stays 8-bit NV12 so SDR clients are never mis-signaled as HDR. Mirrors the existing vulkan=true producer prop.
Force a clean recompile so wolf:vulkan provably layers on the latest gst-wayland-display:vulkan (producer P010+HDR, encoder HDR SEI, vulkanh265enc patch) rather than a stale cached base from an earlier interleaved build.
…o] hdr) Replace the WOLF_HDR env toggle with a reflect-cpp config field GstVideoCfg.hdr (default false). [gstreamer.video] hdr = true selects the P010 10-bit producer format -> producer hdr=true (BT.2020/PQ) -> vulkanh265enc HDR10 Main-10. Config-driven so a managed/SmoothNAS deploy can enable it via the host-mounted config.toml instead of a container env var.
The [gstreamer.video] hdr=true toggle previously only switched the producer to P010, which broke HEVC entirely: the encoder pipeline's downstream capsfilter pins profile=main (8-bit), so vulkanh265enc rejected the 10-bit P010 input with 'No valid profile found' -> not-negotiated -> no video. And Wolf never advertised Main-10, so Moonlight warned 'host doesn't support HDR' and connected SDR/BT709 (mis-decoding a PQ stream). This wires the two missing handshake pieces, both gated on the hdr config: - serverinfo now ORs VIDEO_FORMAT_H265_MAIN10 into ServerCodecModeSupport (new Config::support_hdr) so Moonlight offers the HDR toggle and connects in BT2020/PQ mode. - the vulkan HEVC pipeline's profile=main capsfilter is rewritten to profile=main-10 so the encoder can negotiate the P010 Main-10 path.
… bar) The 64px CTB rounding that works around RADV's partial-block corruption was only applied to the encoder dimension in rtsp/commands.hpp (built fresh from the RTSP clientViewport). But the Vulkan producer renders the virtual compositor at StreamSession.display_mode, which is set earlier at /launch from the unrounded 'mode' header. So at a standard 1080p request the compositor filled 1080 rows while the encoder read full 1088 (17x64) CTBs -- the extra 8 rows were uninitialised P010 (~green), showing as a green bar along the bottom. It only looked clean when the client requested a literal 64-aligned height. Round display_mode at /launch for the Vulkan producer so the compositor matches the encoder, and extend the encoder-side rounding to vulkanh264enc too (CTB- immune, but it shares the now-rounded Vulkan compositor so its encode dimension must agree).
New [gstreamer.video] sdr_reference_white (default 203 nits) is exported as WOLF_SDR_REFERENCE_WHITE when hdr=true, feeding the producer shader's specialization constant. Tune SDR-content brightness in the HDR stream by editing config.toml + restarting wolf -- no rebuild.
The producer posts a 'wolf-hdr-state' application message (bool 'hdr') on HDR<->SDR content change; streaming.cpp catches it -> HDRModeEvent -> control.cpp sends the HDR_MODE (0x010e) control packet (ControlHdrModePacket + SS_HDR_METADATA, BT.2020/ 1000-nit/CLL 1000:400) to the client so its display flips HDR mode. Change-only. Foundation for Model B dynamic switching; the producer-side caps flip (P2.3) pairs with this.
…-copy) Adds a Vulkan Video HEVC encoder to the default hevc_encoders list, mirroring the existing vulkanh264enc entry. Unlike the nvcodec/va entries it omits the profile=main capsfilter, so the 10-bit P010 Main-10 HDR path negotiates (waylanddisplaysrc vulkan=true emits NV12/P010 memory:VulkanImage; zero-copy params are just a decoupling queue). rate-control=cqp; tunables from WOLF_VULKAN_QUALITY/_REF_FRAMES/_B_FRAMES. Listed first so the zero-copy HDR path is preferred where available; check_elements gates it out on Mesa/RADV (no HEVC vulkan encode yet) so Wolf falls through to nvcodec/va. Validated end-to-end on RTX 5080 (Blackwell): 'Using h265 encoder: vulkan'.
…ISABLE) The vulkanh26x encoder creates its own gst.vulkan.instance at session start; on mixed-GPU hosts that dlopen()s lavapipe whose libLLVM collides with the mesa-GLES libLLVM -> abort/ std::terminate (seen as a boost-mutex flood + SIGABRT on session start). The producer plugin's plugin_init already sets this, but the image ENV guarantees it for the encoder instance too, independent of plugin load order. Confirmed: Steam + FF7 Rebirth stream on RTX 5080 with it set.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
1. Native Vulkan NV12 H.264 encode (
feat(vulkan))A
VULKANencoder type wired through config/streaming: avulkanh264ench264_encodersblock,producer_buffer_caps = video/x-raw(memory:VulkanImage), format=NV12, andvideo_params_zero_copy = "queue"so the encoder consumes the producer's NV12VulkanImagedirectly (novideoscale/videoconvert/videorate, which can't negotiate thememory:VulkanImagefeature).waylanddisplaysrc vulkan=trueis set when the buffer caps ask for VulkanImage. Tunables:WOLF_VULKAN_QUALITY/_REF_FRAMES/_B_FRAMES(defaults tuned for low-latency:quality=0 num-ref-frames=1 b-frames=0).2.
wolf:vulkanimage (ci)docker/wolf.vulkan.DockerfileisFROM ghcr.io/games-on-whales/gst-wayland-display:vulkan— which already carries patched GStreamer 1.28.4 (vulkan-video + thevulkanh264encDPB-pool patch) and the plugin under/opt/gst. So the heavy gst-Vulkan + plugin build happens once in that image; here we only compile Wolf on top. ABuild Wolf (Vulkan)step indocker-build.ymlpublishesghcr.io/<owner>/wolf:vulkan.3. Fedora image switch (
feat(fedora))Moved here from #431 per request.
gpu-drivers,gstreamerandwolfimages build on Fedora (base-app:fedora) instead of Ubuntu/apt. Wolf runs its own PulseAudio under supervisord (the Fedora base ships a conflictingpipewire-pulseaudioshim, swapped with--allowerasing), with graceful SIGINT shutdown to matchstopsignal=INT.Relationship to other PRs
gst-wayland-display:vulkan, the base image this builds FROM.Status / testing
Hardware-validated end-to-end on an AMD RX 7900 XTX (RADV + RPM Fusion freeworld mesa):
waylanddisplaysrc → NV12 VulkanImage → vulkanh264enc → Moonlightstreams real content. Draft pending CI image builds and broader GPU validation.