GLSL Shaders using Processing Pi on a Pi 3 B+

[jshaw3]

Hey All,

It’s the first time I’m porting over a glsl shader from Processing OSX to run on Processing Pi running on a Raspberry Pi 3 B+. I have a very basic shader that dissolves between two videos playing. It runs totally fine on my mac but when it’s ported to Processing Pi and updated to use the Processing video library GLvideo instead it is throwing an error.

The shader was converted from a ShaderToy post so maybe there’s some compatibility issues? I looked around and wasn’t able to find anything specific that I think would cause this problem. So any references, pointers or help would be greatly appreciated.

The error that I am getting is below:

Cannot link shader program:
ERROR:LEX/PARSE-2 (fragment shader, line 27) Undefined identifier

I’ve added a more verbose error output, apologies that it’s a screenshot… Processing Pi wasn’t allowing me to copy the full error message.

I’ve pased the code sample below but I’ve also linked to a Dropbox folder w/ the source example that I’m working on. It’s: https://www.dropbox.com/sh/z1spn3bv1jkq233/AABsQGFLsM-cdeBeyrMi47KOa?dl=0

_
shaderDisolve_4.pde

import gohai.glvideo.*;

PShader secondShader;  

PGraphics pg;
PGraphics pg2;

GLMovie movie;
GLMovie movie2;

void setup() {
  size(600, 600, P2D);
  noSmooth();
  pg = createGraphics(600, 600, P2D);

  movie = new GLMovie(this, "Timelapse_HD_Sunrise_Sunset_France_1080p.mp4");
  movie.loop();

  movie2 = new GLMovie(this, "y2mate.com-10MinuteRealtimeBurningFireplaceinFullHD1080p_mFcSPt0sEuk_360p.mp4");
  movie2.loop();

  pg = createGraphics(width, height, P2D);
  pg2 = createGraphics(width, height, P2D);

  secondShader = loadShader("secondShader.glsl");
  secondShader.set("iResolution", float(width), float(height));
  secondShader.set("iTime", millis()/1000.);

}  

void movieEvent(GLMovie m) {
  m.read();
  redraw();
}

void draw() {
  
  pg.beginDraw();
    pg.image(movie, 0, 0, width, height);
  pg.endDraw();

  pg2.beginDraw();
    pg2.image(movie2, 0, 0, width, height);
  pg2.endDraw();

  secondShader.set("iTime", millis()/1000.);
  secondShader.set("iChannel0", pg);
  secondShader.set("iChannel1", pg2);
  
  shader(secondShader);
  rect(0, 0, width, height);
  
}

_
secondShader.glsl

#ifdef GL_ES
precision mediump float;
precision mediump int;
#endif

// Type of shader expected by Processing
#define PROCESSING_COLOR_SHADER

uniform float iTime;
uniform sampler2D iChannel0;
uniform sampler2D iChannel1;
uniform vec2 iResolution;

void mainImage( out vec4 fragColor, in vec2 fragCoord );

void main() {
    mainImage(gl_FragColor,gl_FragCoord.xy);
}

#define TEXTURED 1

void mainImage( out vec4 fragColor, in vec2 fragCoord )
{
    vec2 q = fragCoord.xy/iResolution.xy;
    
    // https://www.shadertoy.com/view/4s3XRf

    vec2 uv = fragCoord.xy / iResolution.xy;
    vec4 color0 = texture(iChannel0, uv);
    vec4 color1 = texture(iChannel1, uv);
    float duration = 10.0;
    
    float t = mod(float(iTime), duration) / duration;
    
    fragColor = mix(color0, color1, t);

}

On a more general note, is there a difference between .glsl written for a mac vs. Linux machine?

I appreciate the help,

Cheers,

[jeremydouglass]

Hi @jshaw3 – were you able to resolve this issue? I am not a GLSL expert, but the ShaderToy porting guide / example in this thread might be helpful:

Porting GLSL from Shadertoy to Processing

A few years ago, I wrote an example on how to port GLSL shader code from Shadertoy.com to Processing. Since then, Shadertoy has changed quite a lot (adding a lot of new uniforms and input methods) so an update was long overdue. You can find it here: https://github.com/SableRaf/Shadertoy2Processing There is still work to be done, but it should be a good starting point for anyone wanting to learn about shaders from the great material available at Shadertoy, and experiment with it in Processing. …

[jshaw3]

Hey @jeremydouglass, thanks for the additional references. I went through them, and I was still getting errors, so no not yet. I tried a few different things, including re-writing the shader to work outside of the “ShaderToy” nomenclature. I resized the videos to be smaller, updated the Pi’s GPU memory to 256mb etc. I ensured it still works on OSX, but when it’s run on a Raspberry Pi 3B+ the sketch is an empty white screen.

The only output in the console is:

Final caps: video/x-raw(memory:GLMemory), format=(string)RGBA, width=(int)640, height=(int)360, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)25/1, texture-target=(string)2D
Final caps: video/x-raw(memory:GLMemory), format=(string)RGBA, width=(int)640, height=(int)360, interlace-mode=(string)progressive, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)25/1, texture-target=(string)2D

Perhaps there’s no support to set PGraphics from Processing to a sampler2D texture in the shader on processing pi? It crossed my mind there’s something with GLVideo images when you set a texture3D. Also, maybe it’s the I’m mixing up something between how frag and color shaders work. At the moment I think I’m only using a Processing Color Shader.

shaderDisolveGLSL.pde

//import processing.video.*;
import gohai.glvideo.*;

PShader mixShader;  

PGraphics pg;
PGraphics pg2;

//Movie movie;
//Movie movie2;

GLMovie movie;
GLMovie movie2;

void setup() {
  size(640, 360, P2D);
  noSmooth();
  pg = createGraphics(640, 360, P2D);

  //movie = new Movie(this, "_sm/LabspaceDawnv1blur2.mp4");
  movie = new GLMovie(this, "_sm/LabspaceDawnv1blur2.mp4");
  movie.loop();

  //movie2 = new Movie(this, "_sm/LabspaceFireblur2.mp4");
  movie2 = new GLMovie(this, "_sm/LabspaceFireblur2.mp4");
  movie2.loop();

  pg = createGraphics(width, height, P2D);
  pg2 = createGraphics(width, height, P2D);

  mixShader = loadShader("fadeshader.glsl");
  mixShader.set("iResolution", float(width), float(height));
  mixShader.set("iTime", millis()/1000.);

  mixShader.set("iChannel0", pg);
  mixShader.set("iChannel1", pg2);

}  

//void movieEvent(Movie m) {
void movieEvent(GLMovie m) {
  m.read();
  redraw();
}

void draw() {
  
  pg.beginDraw();
    pg.image(movie, 0, 0, width, height);
  pg.endDraw();

  pg2.beginDraw();
    pg2.image(movie2, 0, 0, width, height);
  pg2.endDraw();
  
  shader(mixShader);
  rect(0, 0, width, height);
  
}

fadeshader.glsl

#ifdef GL_ES
precision mediump float;
precision mediump int;
#endif

// Type of shader expected by Processing
#define PROCESSING_COLOR_SHADER

uniform float iTime;
uniform sampler2D iChannel0;
uniform sampler2D iChannel1;
uniform vec2 iResolution;

void main() {
    
    vec2 uv = gl_FragCoord.xy / iResolution.xy;
    vec4 mixColor = vec4(0.0);
    vec4 color0 = vec4(uv.x,uv.y,0.0,1.0);
    vec4 color1 = vec4(uv.x,uv.y,0.0,1.0);

    color0 = texture2D(iChannel0, uv);
    color1 = texture2D(iChannel1, uv);

    float duration = 10.0;
    float t = mod(float(iTime), duration) / duration;
    
    mixColor = mix(color0, color1, t);
    gl_FragColor = mixColor;
}

I’ve updated a new version of the sample sketch with the smaller videos here if anyone was curious: https://www.dropbox.com/sh/fu2plxmqhf7shtp/AADxqmW9zf73EsdzworCb5ECa?dl=0

Any recommendations or areas of thought for me to look more into would be greatly appreciated!

[jshaw3]

For those who come across this, or run into a simular problem with shaders on the RPi. The project this was for was cancelled due to COVID-19, so this issue has taken a back burner for a bit. I’m going to take a look and try to figure out what was going on in the next couple of weeks. A solution will be posted here when solved.

[monkstone]

@jshaw3 I revised your code slightly:-

import processing.video.Movie;

PShader mixShader;

PGraphics pg;
PGraphics pg2;

Movie movie;
Movie movie2;

void settings() {
  size(640, 360, P2D);
  noSmooth();
}

void setup() {
  pg = createGraphics(640, 360, P2D);
  movie = new Movie(this, "LabspaceDawnv1blur2.mp4");
  movie.loop();
  movie2 = new Movie(this, "LabspaceFireblur2.mp4");
  movie2.loop();
  pg = createGraphics(width, height, P2D);
  pg2 = createGraphics(width, height, P2D);
  mixShader = loadShader("fadeshader.glsl");
  mixShader.set("iResolution", float(width), float(height));
  mixShader.set("iTime", millis()/1000.);

  mixShader.set("iChannel0", pg);
  mixShader.set("iChannel1", pg2);
}

void movieEvent(Movie m) {
  m.read();
}

void draw() {
  pg.beginDraw();
  pg.image(movie, 0, 0, width, height);
  pg.endDraw();
  pg2.beginDraw();
  pg2.image(movie2, 0, 0, width, height);
  pg2.endDraw();
  shader(mixShader);
  rect(0, 0, width, height);
}

Then I could run the sketch on RaspberryPI4 using ManjaroArmLinux (64 bit) and the latest iteration of processing-4.0 with the latest video library. I also got rid off the sub-folder in the data folder (ps also movie files should not have spaces in their names). Since buster I’ve not bothered trying to install processing on my RasberryPI3B+. I’ll have a go with my PiCrate gem on the RaspberryPI3B+, and report back.

[monkstone]

OK I’ve tried on PiCrate with processing4, works fine, the same sketch on my very pared down raspberryPI3B+, runs for quite a few frames and then crashes (I’ve tried with 64MB, 128MB and 264MB GPU memory and it makes no difference). My RaspberryPI4 has 4GB Ram with a default 64GB GPU Ram, suggesting perhaps RaspberryPI3B+ with RaspberryPI OS does not have enough muscle? PS contrary to stuff posted on processing RasbberryPI website you should use full KMS mode, and ignore original driver (at least since buster).

[monkstone]

This is where it gets interesting:-
Loaded the 64 bit Manjaro OS on RaspberryPI3B+ (now this is first time I tried this) using same microSD card. Sketch does runs for short time from Processing IDE and crashes as before. However when I run it as a PiCrate sketch it runs fine.

Note Manjaro ARM uses GStreamer 1.16.2, whereas Raspberry OS use GStreamer 1.14.x.