P5js 3D sketches - how to overlay a 2D image e.g. HUD or GUI

p5.js Coding Questions
1

This sketch shows a p5.js animated sketch using WebGL (based on a library example) and there are three 2D images overlaying the animation.

  • the inner frame
  • the avatar
  • the text

The images are drawn by shaders created using pure JavaScript (shader code courtesy of @scudly). The overlay shaders have no dependancy on p5.js and that is an essential part of any solution.

So you can see from the sketch that it seems to work. The problem is if I try to add lights or apply a texture to the balls it all goes to pot.

User interaction

Key    Action
 l     toggles lights on / off
 t     toggles texture / solid fill on the balls
 r     toggles restoring previous shader program 
                       (suggested by @davepagurek)

The sketch starts with lights off, solid fill and the previous shader program being restored. In this state turning the lights on or texturing the balls causes the overlays to disappear never to reappear even if we toggle back to the original state. (Refresh this page restart the sketch)

If the previous shader program is not restored then attempting to texture the balls cause the program to crash with the following warning

WebGL warning: uniform setter: UniformLocation is not from the current active Program.

It appears that success depends on the current state of the webgl2 drawing context which is constantly being changed by p5.js so a solution maybe more challenging than it seems.

The sketch code can be found in the p5.js web editor here. There are a lot of comments in the code which I hope helps you make sense of my code.

Any help or suggestions for things to try or research would be greatly appreciated.

2

The solution might be to completely by-pass p5’s shader and texture setting code and brute-force do it all yourself. Before settting any webgl values, get and save them to your own variable, then set your values, draw your overlays, and then set p5’s previous values back. E.g.:

// Get the buffer bound to attribute location 0 
const buffer = gl.getVertexAttrib(0, gl.VERTEX_ATTRIB_ARRAY_BUFFER_BINDING);

with similar queries for the current shader program and texture slots.

The queries will slow down rendering slightly, but since you’ll only do a few per frame, it shouldn’t be noticeable.

3

@scudly thanks very much for replying I have taken onboard what you said and my draw method looks like

function draw() {
    background(192);
    // p5.js code to draw balls removed for clarity

    // Save current webgl2 attributes 
    const curShaderProgram = gl.getParameter(gl.CURRENT_PROGRAM);
    const vertexBuffer = gl.getVertexAttrib(0, gl.VERTEX_ATTRIB_ARRAY_BUFFER_BINDING);
    const activeTexture = gl.getParameter(gl.ACTIVE_TEXTURE);

    // overlay the three images (HTMLCanvasElement's)
    for (let i = 0; i < hudCanvas.length; i++)
        drawHUD(gl, hudCanvas[i], hudShaderProgram[i]);

    // Restore webgl2 attributes
    if (curShaderProgram)
        gl.useProgram(curShaderProgram);
    gl.bindBuffer(gl.ARRAY_BUFFER, vertexBuffer);
    gl.activeTexture(activeTexture);
}

Unfortunately the overlays dissappeared when using lights or texture in animation code but on the positive side no warning or error messages appeared. :grin:

Is this what you expected me to do and have I missed any state that needed saving / restoring? :thinking:

4

Yes, that’s what I was hoping might fix it. You might also save/restore texture slot 0 specifically.

Another thing to try is instead of clearing the depth buffer, just gl.disable(gl.DEPTH_TEST); and re-enable after drawing the HUD. With the depth test disabled, pixels are put on the screen in order they are drawn and their z-depth isn’t written so the HUD would not interfere with other drawn geometry.

I can’t think of why the lighting might change things, but then I haven’t done anything with p5 2.x yet, so I don’t know its quirks. Maybe @davepagurek can think of some other state that you should save and restore.

5

I haven’t looked into the p5 version 2 code at all, but it’s possible that your desire to render the HUD fully independently of p5 might not work. OpenGL/Webgl is most efficient when you batch all of your drawing calls into buffers and send them to the GPU all at once. So, when you make a line() call, don’t actually draw anything but rather store the colors and position in a buffer. Only when draw() exits do you issue your GL commands to render the geometry. This means, if you draw your HUD in draw(), your GL commands happen right away, but the rest of the p5 geometry would only get drawn later and write all over your HUD.

I have no idea if p5 is doing any of that. If it is, you might need some sort of hook function that could get called after p5 does its rendering, but before it calls the page swap. Or you could bite the bullet and render your HUD into its own p5 framebuffer and use p5 clearDepth() and image() to render your HUD instead of using the shader.

6

Even if p5.js doesn’t do that now it might be implemented in some future version.

I will try the suggestions in your last post and let you know how I get on. Thanks.
:grinning: :+1: