Hextop

Proof-of-concept demonstrating that Windows' default shell (explorer.exe) can be completely replaced by a custom shell written in Java, with persistent OpenGL rendering injected behind desktop icons.

WIP

This project is Work In Progress, so it is to be only intended as a proof-of-concept.

What it does

Hextop hijacks the Windows desktop environment by:

1. Replacing the shell — manipulates the registry to register itself as the system shell instead of explorer.exe, with a failsafe mechanism that restores the original shell on abnormal exit.

2. Injecting a rendering surface — reverse-engineers the Windows desktop window hierarchy (specifically the WorkerW window) to insert a persistent OpenGL or Vulkan rendering context behind the desktop icons, allowing arbitrary graphics to be rendered on the desktop wallpaper layer.

3. Maintaining process hygiene — runs explorer.exe as a child process so the shell UI (taskbar, start menu, file explorer) still functions, while the custom shell controls the desktop canvas itself.

4. Handling WorkerW lifecycle — implements a custom event loop (ExplorerWatchdog) to survive Windows 11's aggressive WorkerW window recycling, which would otherwise kill the rendering surface.

Architecture

Windows Shell Integration (win/)

• Win32.java — FFM API bindings for Shell32.dll and User32.dll via Java's Project Panama Foreign Function & Memory API. Low-overhead native calls without JNI.
• WinExplorerSession.java — manages the explorer.exe child process and registry manipulation for shell persistence.
• WallpaperManagerWin11.java — locates and injects into the WorkerW window hierarchy. Windows 11-specific because the window structure differs from Windows 10.
• ExplorerWatchdog.java — monitors window hierarchy changes and re-injects the rendering surface when Windows destroys and recreates WorkerW.
• WinUtil.java — helper functions for window enumeration, message pumping, and event handling at the OS level.

Graphics Abstraction (gfx/, gl/, vk/)

The rendering layer is abstracted to support both OpenGL and Vulkan backends:

• RenderingContext — backend-agnostic interface for rendering.
• GLRenderingContext / VkRenderingContext — OpenGL 3.3+ and Vulkan implementations.
• Scene — scene graph with mesh registration, instancing support, and transform hierarchy.
• GLMesh / GLInstancedMesh — GPU buffer management for static and instanced geometry.

GLFW is used for window creation, but the actual rendering target is the hijacked WorkerW HWND, not a GLFW-managed window.

Platform Detection (platform/)

Runtime platform detection to ensure the shell replacement only activates on Windows. Other platforms fall back to a standard GLFW window without shell manipulation.

Technical notes

• Registry failsafe: On startup, the original shell path is stashed. On shutdown (normal or crash), a shutdown hook restores HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Winlogon\Shell to explorer.exe. This prevents bricking the system if Hextop crashes.

• FFM instead of JNI: Uses Java's Foreign Function & Memory API (Project Panama) for direct native calls. Much lower overhead than JNI and no need for separate native libraries.

• WorkerW injection: Windows creates a "WorkerW" window under the "Progman" (Program Manager) window to host the wallpaper. Hextop finds this window via EnumWindows, gets its HDC, and creates an OpenGL context on it.

• Windows 11 lifecycle quirks: Windows 11 aggressively destroys and recreates WorkerW windows during certain operations (desktop refresh, display settings changes). The watchdog thread detects this and re-injects the rendering surface.

Project structure

src/main/java/com/fractalino/hextop/
├── Hextop.java                    # Entry point
├── HextopSession.java             # Session management (platform-agnostic)
├── HextopWindowsSession.java      # Windows-specific shell hijacking
├── win/
│   ├── Win32.java                 # FFM bindings (Shell32, User32)
│   ├── WallpaperManagerWin11.java # WorkerW injection
│   ├── ExplorerWatchdog.java      # WorkerW lifecycle monitoring
│   ├── WinExplorerSession.java    # Explorer.exe child process + registry
│   └── WinUtil.java               # Window enumeration helpers
├── gfx/
│   ├── Scene.java                 # Scene graph
│   ├── Mesh.java / MeshRegister.java
│   └── RenderingContext.java      # Backend abstraction
├── gl/
│   ├── GLRenderingContext.java    # OpenGL backend
│   ├── GLMesh.java / GLInstancedMesh.java
│   └── GLShaderProgram.java
├── vk/
│   └── VkRenderingContext.java    # Vulkan backend (stub)
└── glfw/
    ├── DesktopWindow.java
    └── GLFWUtil.java              # GLFW initialization

License

GPLv3 https://www.gnu.org/licenses/gpl-3.0.en.html