Custom shader support for 3D scenes?

[phenolophthaleinum]

Description

I've read the documentation and checked the issues, and there is no mention of using custom shaders. Just to make sure, I want to ask the question and get the confirmation: Is there actually any way to get custom shaders to work in the current state of the package?
I do not have any meaningful code, since there is nothing I know I can write to make it. However, if nothing is possible right now, as a suggestion, I would imagine the following interface (with similarities from three.js implementation) would be reasonable for custom shading:

plane = scene.gltf(url='/static/plane.glb').scale(1)
plane.material(
    type='shader',
    vertex=read_shader('blendmodul_v.glsl'),
    fragment=read_shader('blendmodul_f.glsl'),
    uniforms={"baseTexture": scene.texture("/static/grass001aMM_color_512.png", coordinates="uv")}
)

[falkoschindler]

Thanks for the suggestion, @phenolophthaleinum!
As far as I know, custom shaders are currently not supported. At least there is no official API. (Maybe there's a workaround I'm not aware of.)
I converted your post into a feature request so that the community can discuss and vote on it.

Regarding your proposed API, I assume read_shader would basically return the file content of the GLSL file. Maybe it would be more efficient to pass a URL to material() to reduce the payload for the 3D object. The shader itself could be loaded separately and cached by the browser.

[phenolophthaleinum]

As a follow-up, for someone who might be looking for a similar problem in the future - it is possible to make a rough workaround which involves adding html with scripts of a three.js app using ui.add_body_html. I included three.js as a CDN, but it might be possible to add it from a local installation. It's possible to interact with that three.js app through ui.run_javascript, for example, I was able to open a custom model by feeding the path of the model but only relative path for statics works out of the box - /static/model.glb. I think it will be possible to use arbitrary paths by setting up a route for a function that will load the file, even through the file dialog in the native mode.
So basically the code outline would be the following:

ui.add_body_html('''
    <div id="threejs-container" style="width: 100%; height: 500px;"></div>
    <script src="https://cdnjs.cloudflare.com/ajax/libs/three.js/0.155.0/three.min.js"></script>
    <script src="https://cdn.jsdelivr.net/npm/three/examples/js/loaders/GLTFLoader.min.js"></script>
    <script>
          // any remaining three.js app code
        // load model
          function loadModel(url) {
                      if (currentModel) {
                          scene.remove(currentModel);
                      }
                      loader = new THREE.GLTFLoader();
                      loader.load(url, function (gltf) {
                          currentModel = gltf.scene;
                          scene.add(currentModel);
                      });
                  }
    </script>
    ''')
ui.button("Load Model", on_click=lambda: ui.run_javascript(f"loadModel('/static/plane.glb');"))

So generally it's an embedding of three.js app into nicegui app with some layer of communication. Not very nice, but I think it works...

[falkoschindler]

Good news: with #6085 (landing in the upcoming 3.16 release) there is now an official answer to this question. ui.scene has been refactored to a modular object system — you can subclass Object3D with your own JavaScript component and write arbitrary three.js code in it, including ShaderMaterial with custom vertex/fragment shaders and uniforms. That also makes the add_body_html workaround from this thread obsolete.

One caveat: the exact API proposed here — material(type='shader', vertex=..., fragment=..., uniforms=...) directly on a built-in object like a loaded GLTF — does not exist. To apply a custom shader to a model, you write a small custom object that loads the model and assigns the material itself. If a shader convenience on built-in objects is still desirable, that would be a good separate feature request.

Closing this as answered. ※