PGraphics - read pixels as they are drawn

Processing Coding Questions
1

with the PGraphics class, is there a way to read pixles as they get drawn?

for example

PGraphics c;

void settings() {
    size(100, 100, P3D);
}

void setup() {
    c = createGraphics(100,100, P3D);
    c.beginDraw();
}
void draw() {
    c.background(255);
    // read pixels
    c.background(100, 100, 100);
    // read pixels
}

i want to do this in order to, for example, eliminate the need to do beginDraw and endDraw every single time i want to update and display my pixels

as i am thinking that i could thus copy the live pixels from the PGraphics object into a PImage and then image that similar to how most renderers will directly access the graphics (stack?) buffer to update the screen and just grab what ever may be there even while a draw may be executing (single buffered)

as this is the only way i can think of that will speed up my rendering when

for example, rendering 2500 PGraphics objects, calling beginDraw(); /* ... */; endDraw(); makes my fps go to about 7 FPS

NOTE: loadPixels internally calls beginDraw and endDraw so this will add additional overhead and thus be ineffective - https://github.com/processing/processing/blob/78f7c30076de3f9a24536913249009be7e543a6b/core/src/processing/opengl/PGraphicsOpenGL.java#L5378

example:

PGraphics g1, g2;

void settings() {
  size(400, 400, P3D);
}

void setup() {
  background(0);
  g1 = createGraphics(500,500, P3D);
  g1.beginDraw();
  g2 = createGraphics(500,500, P3D);
  g2.beginDraw();
}

void draw() {
  g2.clip(0,0,50,50);
  g2.background(0, 128, 128);
  g2.clip(0,0,25,25);
  g2.background(0);
  g2.clip(35,35,65,65);
  g2.background(0, 228, 128);
  g2.clip(0,0,50,50);
  g2.flush();
  g1.image(g2.get(0,0,500,500), 0, 0);
  g1.flush();
  image(g1.get(0,0,500,500), 0, 0);
}
2

I do not know if I ‘get’ your question, but why don’t you get the whole screen. A 100 by 100 takes only 2 milliseconds. far faster then a draw loop. Only getting a screen 8 times bigger will start to take the 60 f/s down.

PImage pi;
int start_time;

void setup() {  
  size(100, 100, P3D);
}

void draw() {
  start_time = millis();
  for (int i = 0; i < 100; i++) {
    pi = get();
  }
  println(millis()-start_time);
}

I’ve been following your last topic about shapes, and I think these are valid things to try out. Even if you can’t use box(), you can construct them with vertices.

1 Like
3

get() does not seem to work for a PGraphics object

  g2.flush();
  g1.image(g2.get(0,0,500,500), 0, 0);
  g1.flush();
  image(g1.get(0,0,500,500), 0, 0);

see example in top post for full code

4

But isn’t PImage your end goal?

5

yes but is MUST be obtainable from a PGraphics object

6

Ok, I get it…

7

i tried doing

PGraphics g1, g2;

void settings() {
  size(400, 400, P3D);
}

void setup() {
  background(0);
  g1 = createGraphics(500,500, P3D);
  //g1.beginDraw();
  g2 = createGraphics(500,500, P3D);
  g2.beginDraw();
}

void draw() {
  g2.clip(0,0,50,50);
  g2.background(0, 128, 128);
  g2.clip(0,0,25,25);
  g2.background(0);
  g2.clip(35,35,65,65);
  g2.background(0, 228, 128);
  g2.clip(0,0,50,50);
  g2.flush();
  assert(g2.isGL());
  PGraphicsOpenGL g3 = (PGraphicsOpenGL) g2;
  FrameBuffer fb = g3.getFrameBuffer();
  fb.readPixels();
  java.nio.IntBuffer ib = fb.getPixelBuffer();
  int[] intArray = ib.array();
  PImage image = createImage(500, 500, ARGB);
  image.loadPixels();
  for (int i = 0; i < image.pixels.length; i++) {
    image.pixels[i] = intArray[i];
  }
  image.updatePixels();
  g1.beginDraw();
  g1.image(image, 0, 0);
  g1.endDraw();
  image(g1, 0, 0);
}

how ever i do not get anything displayed

8

What if instead of using the pixel array make your own custom int array and keep track of them?

9

cus i wont be able to use PGraphics calls on them, for example box(40);

since ill need to manually create the box myself using pixel data such as a library that draws shapes and stuff directly to a bitmap, which would severly limit the capabilities, such as PShape cannot be used as PShape probably does not know how to be read by the drawing library along with most other Processing specific classes

10

this DOES work HOWEVER i need to call endDraw() in order for g3.getTexture().get(image.pixels) to get any pixels from the texture, in which want to AVOID needing to call endDraw() so this IS NOT a solution, however i believe it will help

and

  FrameBuffer fb = g3.getFrameBuffer();
  fb.readPixels();
  fb.getPixels(image.pixels);

is empty both BEFORE and AFTER endDraw();

PGraphics g1, g2;

void settings() {
  size(400, 400, P3D);
}

void setup() {
  background(0);
  g1 = createGraphics(500,500, P3D);
  //g1.beginDraw();
  g2 = createGraphics(500,500, P3D);
  //g2.beginDraw();
}

void draw() {
  g2.beginDraw();
  g2.clip(0,0,50,50);
  g2.background(0, 128, 128);
  g2.clip(0,0,25,25);
  g2.background(0);
  g2.clip(35,35,65,65);
  g2.background(0, 228, 128);
  g2.clip(0,0,50,50);
  g2.endDraw();
  //g2.flush();
  assert(g2.isGL());
  PGraphicsOpenGL g3 = (PGraphicsOpenGL) g2;
  PImage image = createImage(500, 500, ARGB);
  image.loadPixels();
  g3.getTexture().get(image.pixels);
  image.updatePixels();
  println("image created");
  g1.beginDraw();
  g1.image(image, 0, 0);
  g1.endDraw();
  image(g1, 0, 0);
}
11

and

  //g2.endDraw();
  g2.flush();
  assert(g2.isGL());
  PGraphicsOpenGL g3 = (PGraphicsOpenGL) g2;
  PImage image = createImage(500, 500, ARGB);
  image.loadPixels();
  
  // frame buffer fails
  
  //FrameBuffer fb = g3.getFrameBuffer();
  //fb.readPixels();
  //fb.getPixels(image.pixels);
  
  // texture fails
  
  Texture t = g3.getTexture();
  t.updateTexels();
  t.get(image.pixels);
  image.updatePixels();
  println("image created");

the frame buffer is empty both before and after the endDraw();
however the texture is only empty before the endDraw();

unless it draws to a framebuffer i do not yet know about

12

To follow this I have to learn what assert even does.
I found this post where it’s said that when an error is encountered it stops executing the code right when the error happens, so I don’t yet understand the use of it in your code.
The example there really stops the code by summing too big numbers and prints the error to the console, but the second part doesn’t give a ‘wrong’ print.

void setup() {  
  // int i = sum(2147483647, 1);
  // println(i);
  int t = sum(0, 0);
  println(t);
}

int sum(int a, int b) {
  assert (Integer.MAX_VALUE - a >= b) : 
  "Value of " + a + " + " + b + " is too large to add.";
  final int result = a + b;
  assert (result - a == b) : 
  "Sum of " + a + " + " + b + " returned wrong sum " + result;
  return result;
}
13

assert(condition) is basically

if (condition == false) abort();

which usually just prints the error condition and then exits the program

however this behaviour is Programming Language implementation-dependent (eg not guarenteed to do the same thing in every programming language)

14

in my code i ensure that the graphics object is of type PGraphicsGL as that is what i am casting to and accessing methods from

15

in your example, it might be

void setup() {  
  // int i = sum(2147483647, 1);
  // println(i);
  int t = sum(0, 0);
  println(t);
}

int sum(int a, int b) {
  if (Integer.MAX_VALUE - a >= b) {
    String msg = "Value of " + a + " + " + b + " is too large to add.";
    PrintErrorAndPauseExecution(msg);
  }
  final int result = a + b;
  if (result - a == b) {
    String msg = "Sum of " + a + " + " + b + " returned wrong sum " + result;
    PrintErrorAndPauseExecution(msg);
  }
  return result;
}
16

anyway, im starting to think that this question is too high-level for you to help me, as it seems like you are inexperienced with OpenGL and the internals of Processing API (specifically PGraphics/PGraphicsGL)

17

Absolutely right! I’m following this to learn, and by answering many times maybe atract someone with tho knowledge to jump in.

18

I suggest studying the internals in question, @mgood7123.

In particular, look at endDraw, since this is what you are trying to avoid.

I am not an OpenGL expert, but my understanding is that OpenGL rendering of commands – such as calling box, or drawing a shape – combine geometry with texture/lighting/camera. It isn’t drawing a line of pixels, it is describing a 3D scene which later gets rendered in a render pipeline. So: before rendering, just geometry. After, pixels. At some point / periodically that geometry gets “flushed” down the pipeline.

There is a partial illustration of that pipeline in the Shaders tutorial.

This is one of the reasons that pixels may notappear when you display a PGraphics before the pipeline runs – they aren’t pixels yet, they are just geometry.

You may notice that endDraw() calls flush.

You seem to want to render many discrete areas (hundreds or thousands?) using an OpenGL pipeline in a high-performance way. You have started many closely related threads on this.

As I understand it, your key problem is that you want a big object-oriented class tree with lots of encapsulation – hundreds or thousands of PGraphics. That gives you hundreds or thousands of geometry queues with no pixels, and hundreds or thousands of pipelines to run.

I outlined an alternative in another of your threads – send all your geometry to a single shared PGraphics (or a small number), then call endDraw once. This has disadvantages, because the shared pipeline is shared – camera, texture, lighting – but it reduces your pipeline flush calls.

Otherwise you may need to rewrite PGraphics and/or create your own renderer. A question is whether what you want (that many separate pipeline flush calls per second) is possible with OpenGL in general – never mind Processing.

4 Likes
19

yea, i have already taken a bit of a look, as in PGraphics framebuffer can be rendered in `noLoop` mode but cannot be rendered in `loop` mode

however my main goal is obtaining the pixels of a PGraphics Object, like what

graphicsObject.endDraw();
image(graphicsObject);

does

geometry and other stuff such as lights are not particularly important, only the pixels that get drawn, now i CAN do this in noLoop mode (looping = false) however this seems broken, and i do not know why

note that i get

image
image540×569 10.2 KB

if i have

void drawBox() {
  // draw a box onto the PGraphics object
  g2.background(0);
  int c = wrap(frameCount, 255);
  g2.fill(0, c, c);
  if (frameCount == 1) g2.lights();
  g2.noStroke();
  //g2.translate(width/2, height/2);
  //g2.rotateX(frameCount/100.0);
  //g2.rotateY(frameCount/200.0);
  g2.box(40);
}

void draw() {
  if (!looping) looping = true;
  
  background(0);

  drawBox();
  
  // render the PGraphic's FBO, note this fails in loop mode
  g2.imageFrameBuffer();
  
  int oldColor = getGraphics().fillColor;
  fill(255);
  textSize(16);
  text("frameCount = " + frameCount + ", FPS: " + frameRate, 10, 20);
  fill(oldColor);
}

which indicates that all geometry and stuff gets reset somewhere in endDraw

note that


is used for g2.imageFrameBuffer();

in which if i do

  g2.beginDraw();
  drawBox();
  g2.endDraw();
  image(g2, 0, 0);

then lights gets reset in the next and subsequent draws

20

i think that the primary reason that endDraw() is slow when used more than once in a draw call, is that it internally called *swapBuffers(…) which is very expensive as, for example:

// C++ CODE
/**
 * buffer swapping is expensive if **vsync** is enabled
 *
 * if vsync is enabled, eglSwapBuffers always waits for
 * the screen to be refreshed once between swapbuffer calls
 */
EGLBoolean GLIS_SwapBuffers(class GLIS_CLASS & GLIS) {
    return eglSwapBuffers(GLIS.display, GLIS.surface);
}

when this is called, say 10 times per every draw, my FPS plummets to 12 FPS