Shaders: Drawing layers on top of each other

[hoend008]

Hey all,

I want to draw moving shapes on top of an image (a world map), for example ellipses representing airplanes moving over the map to their destination. I am also practicing using a shader to improve performance, because there will be thousands of airplanes moving at any given point.

My approach at this point, without a shader, is to use P3D and to draw 2 ‘layers’, and use the z value to make the ellipses appear on top of the world map image. Without shaders this works wonderfullly with translate, but when I am using a shader to draw an ellipse, it does not appear on the screen. I do get the world map to appear when I create a shape and use the image as texture, but the ellipses are nowhere to be seen…

My first question is: is this the correct approach? And if not, any pointer in the right direction would be very much appreciated.

Any comments also on the code will be greatly appreciated

PImage img;
PShader texShader;
int h;
int w;
float circleSize = 0.1;
float[] circlePosition = {0.5,0.2,0.1,0.4};

void setup(){
  size(2048,1024, P3D);
  
  h = height;
  w = width;
  
  texShader = loadShader("frag.glsl", "vert.glsl");
  
  img = loadImage("worldmap2048x1024-ChinaCentered.png");
}

void draw(){
  
  shader(texShader);
    
  textureMode(NORMAL); 
  beginShape();
  texture(img);
  vertex(0, 0, 0, 0);
  vertex(w, 0, 1, 0);
  vertex(w, h, 1, 1);
  vertex(0, h, 0, 1);
  endShape();

  texShader.set("circleSize", circleSize);
  texShader.set("circlePosition", circlePosition);

vertex shader

uniform mat4 transform;
uniform mat4 texMatrix;

attribute vec4 position;
attribute vec4 color;
attribute vec2 texCoord;

varying vec4 vertColor;
varying vec4 vertTexCoord;

uniform vec2 resolution;
varying vec2 uv;

void main() {
  gl_Position = transform * position;
  uv = (gl_Position.xy * resolution.xy) / resolution.y;

  vertColor = color;
  vertTexCoord = texMatrix * vec4(texCoord, 1.0, 1.0);
}

Fragment shader

#ifdef GL_ES
precision mediump float;
precision mediump int;
#endif

uniform sampler2D texture;
uniform float circleSize;
uniform float[4] circlePosition;

varying vec4 vertColor;
varying vec4 vertTexCoord;
varying vec2 uv;

void main() {
  vec4 col = vec4(0, 0, 0, 1);

  for(int i = 0; i < 4; i+=2){
    vec2 circlePos = vec2(circlePosition[i], circlePosition[i+1]);
    float d = length(uv-circlePos);
    float m = smoothstep(circleSize, .05, d);
    col += m;
  }

  //gl_FragColor = col;//texture2D(texture, vertTexCoord.st) * vertColor + col;
  gl_FragColor = texture2D(texture, vertTexCoord.st) * vertColor;
}

[scudly]

If you use P2D, that is with no depth buffering, later objects will be drawn on top of earlier objects, so you can just draw the world map first and then the planes and they will render on top of the map.

Your posted code cuts off in the middle of draw() so we can’t see how you’re drawing the planes.

You can barely see it here thanks to the low resolution, but in https://genart.social/deck/@scdollins/111783482446814936, I use a line segment fragment shader to connect the current position with the last:

String[] fragSrc = {"""
#version 330 core
in vec4 fColor;
in vec2 uv;
in vec2 p0;
in vec2 p1;
in float rad;
out vec4 outColor;

float sdSegment( in vec2 p, in vec2 a, in vec2 b ) {
  vec2 ba = b - a;
  vec2 pa = p - a;
  float h = clamp( dot(pa,ba) / dot(ba,ba), 0., 1. );
  return length( pa - h * ba );
}

void main() {
  if( sdSegment( uv, p0, p1 ) > rad ) discard;
  outColor = fColor;
}
"""};

[hoend008]

Dear Scudly,

You are absolute right, I shifted to P2D and got a step further now. A new problem came up, however, which is error C6020 : Constant register limit exceeded at ; more than 1024 registers needed to compile program"

In one approach I create the shapes in processing (with shape()) function, and then the shader renders these nicely. I can use the shader to manipulate the colors etc. But what I like to do is to get this effect you see here

In this sketch the color of a pixel is determined by its surroundings - namely the position of the particles. When I try to do this with 2000 airplanes (2000 particles), the shader can’t compile because it does not accept such very large arrays, in this case an array with position values for a few thousand objects.

In these situations people recommend using a uniform buffer object or texture buffer object to pass large amounts of data to the shader. In the processing documentation for the set function, it does not talk about these things. Is there a way to do this in Processing?

[scudly]

Yes, you can pass the data to the GPU using a texture image. Processing only supports 8-bit RGBA textures, so you have to encode the position (and orientation, if you want that as well) across those integer values using one or more pixels for each object. For instance, you could pass in the (x,y) coordinates as integers from 0 to 65535 in the AR and GB color channels of the texture. If you want a second (x,y) for the previous position so you can render a line, you’d pass that in a second pixel. The glsl function texelFetch() lets you read the values out as a vec4 which you then need to recombine to get your (x,y) pairs.

Keep in mind, though, that every pixel of your output image has to run the fragment shader calling texelFetch() once (or twice) for every object that you are drawing. In your code, that’s 2 million pixels each sampling 4000 texture pixels, so 8 billion texture reads per frame. You’d better be running this on a hefty GPU or it’s gonna be stuttery.

The more efficient alternative is to render the planes as you said, using shape() and then add the glow as a purely screen effect using a blur filter pass. It likely won’t look quite the same, but you can easily render hundreds of thousands of planes that way without melting your GPU. Try rendering the planes, blur them, then render the planes again over top to look more crisp.

[scudly]

Maybe you’d be happy with something like this:

int N = 256;

PShader blurShdr;

void setup() {
  size( 1000, 1000, P2D );
  background(0);
  blurShdr = new PShader( this, blurVertSrc, blurFragSrc );
  noStroke();
  colorMode( HSB, 1, 1, 1, 1 );
  frameRate(30);
}

PVector nextPos( float u, float t ) {
  t *= 0.5;
  float r = 2+sin(TAU*(21.17*u+2.32*t+0.1*sin(TAU*(23.421*u+5.23*t))));
  return new PVector( r*sin(TAU*(u+t)), r*cos(TAU*(u+t)));
}

void draw() {
  filter( blurShdr );
  
  translate( width/2, height/2 );
  float w = width/6.0;
  for( int i=0; i<N; i++ ) {
    float u = 1.0*i/N;
    strokeWeight( w*(0.03+0.02*sin(TAU*17.9*u)) );
    stroke( 87.618*u%1, 0.7, 1 );
    PVector p0 = nextPos( u, frameCount/500.0 );
    PVector p1 = nextPos( u, (frameCount+1)/500.0 );
    line( w*p0.x, w*p0.y, w*p1.x, w*p1.y );
  }
  noStroke();
  fill( 0, 0, 1 );
  circle( 0, 0, 50 );
}


String[] blurVertSrc = {"""
#version 330 core
uniform mat4 modelview;
uniform mat4 projection;
in vec4 position;
in vec4 texCoord;
out vec2 uv;
void main() {
  uv = texCoord.xy;
  uv.y = 1.0 - uv.y;
  gl_Position = projection * modelview * position;
}
"""};


String[] blurFragSrc = {"""
#version 330 core
uniform sampler2D ppixels;
in vec2 uv;
out vec4 fragColor;
float N = 6, R = 7.9;
void main() {
  vec2 inc = vec2(1.0)/textureSize( ppixels, 0 ).xy;
  vec3 col = vec3(0.);
  float sum = 0.;
  for( float j=-N; j<=N; j++ )
    for( float i=-N; i<=N; i++ ) {
      vec2 offs = vec2( i, j );
      float len = length( offs );
      if( len <= R ) {
        float w = 1.-pow(len/R, 1.0);
        col += texture( ppixels, uv+inc*offs ).rgb * w;
        sum += w;
      }
    }
  col /= sum;
  col *= 0.98;
  fragColor = vec4( col, 1.0 );
}
"""};

You can tweak the N and R variables in the shader as well as the 0.98 fade factor. Bigger N and R spread out the blur while slowing down the rendering.

[hoend008]

Oh my… that is beautiful, thank you so much!

I experimented with the textures - very clever to store coordinate information in an image as pixels, it works like a charm, although I have not yet tested it with large amounts of data. I trust your judgement that performance will suck, but for fun and as a learning experience I think I will give it a try as well.

But first let me study your code, the result it creates looks amazing!

[scudly]

Sorry, I didn’t think of this first, but rather than doing the blend in a single full-screen fragment shader, it would be way faster to render each plane along with its glow on individual polygons and combine them using blendMode( LIGHTEST ). This would be fast for hundreds of thousands of planes. This doesn’t work in 3D because you can’t depth sort through the transparent glow, but it works perfectly well in 2D such as you are doing.

The challenge, however, is how to blend the glow effect over your background image. The glow here would be fading to black, not to transparent. You could probably do it by rendering the planes a few different times using shaders to output a separate image and alpha-mask and use a single full-screen fragment shader to combine them with your background. There might be a simpler way, but I haven’t thought of it yet.

To get specific, how exactly do you want to render your planes? Are they just circles with a glow around them or are they multi-color bitmap images?

[scudly]

A method that would allow you to easily blend over the background image would be to draw them with a simple transparent glow around each individually and use the normal alpha blending to handle the mixing.

int N = 256;

float S3 = (float)Math.sqrt(3);
PShader glowShdr;

void setup() {
  size( 1000, 1000, P2D );
  background(0);
  glowShdr = new PShader( this, glowVertSrc, glowFragSrc );
  noStroke();
  colorMode( HSB, 1, 1, 1, 1 );
  frameRate(30);
}

PVector nextPos( float u, float t ) {
  t *= 0.5;
  float r = 2+sin(TAU*(21.17*u+2.32*t+0.1*sin(TAU*(23.421*u+5.23*t))));
  return new PVector( r*sin(TAU*(u+t)), r*cos(TAU*(u+t)));
}

void draw() {
  background( 0.4, 1., 0.3 );
  shader( glowShdr );
  translate( width/2, height/2 );
  float w = width/6.0;
  for( int i=0; i<N; i++ ) {
    float u = 1.0*i/N;
    float r = w*(0.09+0.06*sin(TAU*17.9*u));
    fill( 87.618*u%1, 0.5, 1 );
    PVector p0 = nextPos( u, frameCount/500.0 ).mult(w);
    //PVector p1 = nextPos( u, (frameCount+1)/500.0 );
    beginShape();
    vertex( p0.x, p0.y-2*S3*r, 0, 2 );
    vertex( p0.x+3*r, p0.y+S3*r,  3/S3, -1 );
    vertex( p0.x-3*r, p0.y+S3*r, -3/S3, -1 );
    endShape(CLOSE);
  }
  resetShader();
}


String[] glowVertSrc = {"""
#version 330 core
uniform mat4 modelview;
uniform mat4 projection;
in vec4 position;
in vec4 texCoord;
in vec4 color;
out vec2 uv;
out vec3 col;
void main() {
  uv = texCoord.xy;
  col = color.rgb;
  gl_Position = projection * modelview * position;
}
"""};


String[] glowFragSrc = {"""
#version 330 core
in vec2 uv;
in vec3 col;
out vec4 fragColor;
void main() {
  float d = length(uv);
  if( d > 1. ) discard;
  fragColor = vec4( col, (1.-d)/(1.+3.*d) );  // choose your own glow fade-away function
}
"""};

[hoend008]

Dear Scudly,

Again thank you for your help and suggestions, and the amazing code you provided!

To answer your question: the planes are supposed to be just circles, and the challenge was to make them look visually pleasing. At first I experimented with blur effects and gradients in Processing, but when rendering a few thousand airplanes it become way too slow. I really liked the glow effect in the earlier example, but as you pointed out, that too can get really slow.

Again I will have to study your code, you are way more advanced in shaders than I am, but it is a wonderful learning experience for me, so thank you again. In the last example, are you trying to limit the glow (fade) effect to the pixels near the location of the circle without affecting the remainder of the pixels? I was thinking that, if possible, that may be a good way to limit the number of calculations the shader has to make to create this kind of glow fade-away effect.

[scudly]

In my last example, the shader is being applied to a single equilateral triangle for each plane individually in which I discard any pixels outside of a circle in the middle of it. It then shades the circle with the given color, but fades off the opacity (increases the alpha) based on the distance from the center. If you comment out the discard line and replace fragColor = vec4( col, 1. );, you can see the raw triangles. Put the discard back in and you can see the full circles.

I like to play with functions in Desmos | Graphing Calculator to see the effects. https://www.desmos.com/calculator/olda1ezpoc will let you see the glow function I used. If you change the 3 to a bigger number, the glow will tighten up (drop off) more quickly. The 1-d on top was to make the glow drop to 0 at the edge of the circle.

[hoend008]

Dear Scudly,

Having studied your code that creates this beautiful tail/blur effect, I have a question about how to include a background image to this sketch. When I added an image

  pushMatrix();
  translate( width/2, height/2 );
  imageMode(CENTER);
  image(img, 0, 0);
  popMatrix();

the sketch either shows the image and the moving objects but without the tail, or it shows the objects with tail but the background image is no longer visible (background turns black). This depends on where in the code (before or after the filter( blurShdr ); ) you add the code to add the image.

I tried manipulating the opacity in the shader, but with no success (and I understand why I think). Do you know how to solve this issue?

[scudly]

Yes, the blur shader both blurs and fades to black which are both problems for dealing with a background image. My initial idea was to generate the planes and their trails in a separate PGraphics that you would then draw on top of your background image, but at the moment, I’m not sure how to replace the black of the planes layer with transparency.

At least the glow shader will work with a background image, though it’s not quite as pretty.

[hoend008]

Got the glow shader to work with the background, it looks very nice, the blendMode(ADD) effect also gives it an extra touch. Thank you very much, I would not got this far without your help.

Out of curiosity, do you have any resources you would recommend to get better at Processing and shaders?

[scudly]

If you use blendMode(ADD) and darken the background image a little, you can use the blur shader if you render the planes to a separate PGraphics and then just draw it on top.

int N = 256;

PImage bg;
PGraphics dots;
PShader blurShdr;

void setup() {
  size( 1000, 1000, P2D );
  colorMode( HSB, 1, 1, 1, 1 );
  background(0);

  bg = createImage( width, height, ARGB );
  bg.loadPixels();
  for( int j=0; j<height; j++ )
    for( int i=0; i<width; i++ ) {
      float h = (noise( i*0.01, j*0.01 ) - 0.5) * 1.2;
      bg.pixels[ j*width+i ] = h<0 ? color(0.66, 0.7, 0.6+h) : color(0.33, 0.7, 0.6-h); 
    }
  bg.updatePixels();

  dots = createGraphics( width, height, P2D );
  dots.beginDraw();
  dots.colorMode( HSB, 1, 1, 1, 1 );
  dots.noStroke();
  dots.background( 0 );
  dots.endDraw();
  
  blurShdr = new PShader( this, blurVertSrc, blurFragSrc );
  frameRate(30);
}

PVector nextPos( float u, float t ) {
  t *= 0.5;
  float r = 2+sin(TAU*(21.17*u+2.32*t+0.1*sin(TAU*(23.421*u+5.23*t))));
  return new PVector( r*sin(TAU*(u+t)), r*cos(TAU*(u+t)));
}

void draw() {
  image( bg, 0, 0 );
  
  dots.beginDraw();
  dots.filter( blurShdr );
  dots.translate( width/2, height/2 );
  float w = dots.width/6.0;
  for( int i=0; i<N; i++ ) {
    float u = 1.0*i/N;
    dots.strokeWeight( w*(0.03+0.02*sin(TAU*17.9*u)) );
    dots.stroke( 87.618*u%1, 0.7, 1 );
    PVector p0 = nextPos( u, frameCount/500.0 );
    PVector p1 = nextPos( u, (frameCount+1)/500.0 );
    dots.line( w*p0.x, w*p0.y, w*p1.x, w*p1.y );
  }
  dots.endDraw();

  blendMode(ADD);
  image(dots, 0, 0);
  blendMode(BLEND);
}


String[] blurVertSrc = {"""
#version 330 core
uniform mat4 modelview;
uniform mat4 projection;
in vec4 position;
in vec4 texCoord;
out vec2 uv;
void main() {
  uv = texCoord.xy;
  uv.y = 1.0 - uv.y;
  gl_Position = projection * modelview * position;
}
"""};


String[] blurFragSrc = {"""
#version 330 core
uniform sampler2D ppixels;
in vec2 uv;
out vec4 fragColor;
float N = 6, R = 7.9;
void main() {
  vec2 inc = vec2(1.0)/textureSize( ppixels, 0 ).xy;
  vec3 col = vec3(0.);
  float sum = 0.;
  for( float j=-N; j<=N; j++ )
    for( float i=-N; i<=N; i++ ) {
      vec2 offs = vec2( i, j );
      float len = length( offs );
      if( len <= R ) {
        float w = 1.-pow(len/R, 1.0);
        col += texture( ppixels, uv+inc*offs ).rgb * w;
        sum += w;
      }
    }
  col /= sum;
  col *= 0.98;
  fragColor = vec4( col, 1.0 );
}
"""};

As to learning, no, I don’t know of any particularly good single resource. I learned a lot about shaders and Processing from browsing shadertoy.com and also vertexshaderart.com, reading through https://iquilezles.org/, various sites and youtube series teaching OpenGL, browsing the Processing source on github, and years of reading people’s solutions here on the forum.

[hoend008]

Dear Scudly,

Thank you once more, a very clever and elegant solution! Also your willingness to help others, including people like me who you do not even know, is very much appreciated and inspiring as well!

Please feel free to contact me anytime in case you need help with something

Kind regards,

Wouter