P2D & PGraphics Movie not displaying across render modes? + BONUS P2D performance issue

Processing Libraries
1

Hi friends,

Again, sad to come back to the forum with some puzzling behaviour. Maybe someone can point me towards something I’m doing wrong or perhaps a workaround?

I currently have a sketch that draws into an off-screen buffer (PGraphics) that I use to manipulate the on-screen image. The way I do the on-screen stuff uses vertex() calls so I’m forced to use the P2D renderer for this. Until now, it seemed fine keeping the off-screen PGraphics in the standard renderer, and everything was just dandy.

Yesterday however, I decided that rather than generating some noise in the off-screen buffer, I wanted to render a video into it. Easy enough I thought, yet when I draw the video into the buffer using pg.image() I get nothing. I’ve exhaustively debugged this - any static PImage draws fine from the P2D main program into the standard rendered PGraphics, but when I use a Movie, nothing shows up. Changing the PG to P2D solves the issue…

BUT

…introduces a MASSIVE performance hit (from 60 to 7fps) caused by function that samples the brightness (bit shifted red channel deriv) from the off-screen buffer to use for the on-screen content. Indeed it should be a bit of an expensive function, as it downloads the texture and iterates over all pixels, but like I said before, the exact sketch runs at 60fps when the buffer uses the standard renderer. Writing this I’m thinking, could it be that in P2D it does not check the downloaded flag on the texture, and re-downloads the texture with every loadPixels() call? Brb!

Below two snippets of code to illustrate:

P2D Video playback example:

import processing.video.*;

PGraphics pg;
Movie movie;


void setup()
{
  size(500, 500, P2D);

  movie = new Movie(this, "launch2.mp4");
  movie.loop();

  pg = createGraphics(width, height); // video does not display
  //pg = createGraphics(width, height, P2D); // video works fine, but causes performance bug in other code
}

//void movieEvent(Movie m) {
//  m.read();
//}

void draw()
{
  surface.setTitle("FPS: "+nf(frameRate, 1, 1));
  
  // update pg
  
  if (movie.available() == true) {
    movie.read();
  }
  
  pg.beginDraw();
  pg.image(movie, 0, 0, pg.width, pg.height);
  pg.endDraw();
  
  // draw
  
  image(pg, 0, 0);
}

Snippet that samples from the pg that seems to cause a performance hit in P2D, rather than in the standard renderer:

color sampleColour(PImage img, float normX, float normY)
{
  img.loadPixels();
  return img.pixels[int(normY * img.height) * img.width + int(normX * img.width)];
}

float sampleBrightness(PImage img, float normX, float normY)
{
  return (sampleColour(img, normX, normY) >> 16 & 0xFF);
}
2

Why is this surprising?! The texture download, and particularly the bit manipulation done to get the pixel data in the format expected by Processing, is really non-performant. So using the standard renderer has a lot less to do.

I’d look into shaders if you want this to perform well.

No idea why the standard renderer isn’t working, but the current video library is a little borked with OpenGL! (standard answer from me, try it in PraxisLIVE instead :wink: )

1 Like
3

Found the issue… whether my assumption of the ‘checking of a download’ flag is true or not, pulling the loadPixels() from that function, and manually just downloading the texture once solved the performance issue. :man_facepalming:

4

Filed them as two bugs:

5

Hi Neil, I’m not sure I understand what you’re saying here…? I am aware that manipulating the pixel array is expensive, and that shaders are the way to go when looking for performance. In some cases though you’d still need to download a texture to manipulate it cpu side.

The issue described is about why loadPixels() behaves differently on a PGraphics object created with a P2D than it does with the standard renderer, eg:

  pg = createGraphics(width, height); // works fine - seems to ignore subsequent loadPixels() calls
  pg = createGraphics(width, height, P2D); // massive performance hit with multiple loadPixels() calls