P2D vs JAVA2D - strange differences in PGraphics methods and masking

alexolotl · June 5, 2019, 2:55am

I’ve created a test sketch to demonstrate and try to pinpoint some of the unusual behavior I’ve noticed when switching between P2D and JAVA2D renderers (specifically, what seem to be bugs with the P2D renderer)

Differences in PGraphics get() method:

in P2D, if you create a PGraphics pg, draw to it, then call imagecopy = pg.get(), the pg itself is no longer renderable
in JAVA2D, if you create a PGraphics pg, draw to it, then call imagecopy = pg.get(), the pg itself is still renderable

Masking differences:

in P2D, only PImage to PImage masking works properly. If a PGraphics is involved (with an image drawn to it), then it results in strange black and white / mirrored copy of one of the inputs
in JAVA2D, PImage to PImage masking works, as well as PGraphics cases

In both of these cases, it seems JAVA2D wins - unless I am doing something wrong. However, large images, framerate, and potential use of custom shaders will be a concern in the bigger project I’m gearing up towards - so I’m hoping that there is still a way for me to get PGraphics working well with P2D.

If anyone is interested in the code I was testing with:

input_image = None
mask_input_image = None
masked = None

# renderer = JAVA2D
renderer = P2D

# INSTRUCTIONS FOR TESTING

## I have different masking cases commented out in the setup(), try one at a time
## Then switch the renderer type above and try the cases again - gets very different results

# FINDINGS

## maybe this link is relevant - https://forum.processing.org/two/discussion/3658/issues-with-mask-on-pgraphics-objects
## in P2D, only PImage to PImage masking works properly. If a PGraphics is involved (with an image drawn to it), then it results in strange black and white / mirrored results
## in JAVA2D, PImage to PImage masking works, as well as PGraphics cases
## in P2D, if you create a PGraphics pg, draw to it, then call imagecopy = pg.get(), the pg itself is no longer renderable
## in JAVA2D, if you create a PGraphics pg, draw to it, then call imagecopy = pg.get(), the pg itself is still renderable 
## so JAVA2D seems the more reliable way to go, but we want to use shaders and openGL to boost performance - which requires P2D

def setup():
    global input_image
    global mask_input_image
    global masked
    
    size(400,400,renderer)
    
    # INITIALIZATION
    input_image = loadImage("some_image.jpg")
    input_image.resize(200,200)
    mask_input_image = loadImage("some_other_image.jpg")
    mask_input_image.resize(200, 200)
    input_copy = input_image.get()
    mask_input_copy = mask_input_image.get()
    # END INITIALIZATION
    
    
    ##### CASE 1 #####
    ##### PImage masking PImage #####
    ##### Works fine both with P2D and default renderer
    input_copy.mask(mask_input_copy)
    masked = input_copy.get()
    
    
    # ##### CASE 2 #####
    # ##### PGraphics masking PImage #####
    # ##### With P2D, doesnt work. With default, works
    # pg = createGraphics(200,200,renderer)
    # pg.beginDraw()
    # pg.background(255,255,255)
    # pg.image(mask_input_copy,0,0)
    # pg.endDraw()
    # input_copy.mask(pg)
    # masked = input_copy.get()
    
    
    # ##### CASE 3 #####
    # ##### PImage masking PGraphics #####
    # ##### Weird: with P2D, produces an upside down black and white image of the mask image. Works with defualt
    # pg = createGraphics(200,200,renderer)
    # pg.beginDraw()
    # pg.background(255,255,255)
    # pg.image(input_copy,0,0)
    # pg.endDraw()
    # pg.mask(mask_input_copy)
    # masked = pg.get()
    
    
    # ##### CASE 4 #####
    # ##### PGraphics masking PGraphics #####
    # ##### Weird: with P2D, produces a rightside up black and white image of the mask input. Works with default
    # pg = createGraphics(200,200,renderer)
    # pg.beginDraw()
    # pg.background(255,255,255)
    # pg.image(input_copy,0,0)
    # pg.endDraw()
    # pg2 = createGraphics(200,200,renderer)
    # pg2.beginDraw()
    # pg2.background(255,255,255)
    # pg2.image(mask_input_copy,0,0)
    # pg2.endDraw()
    # pg.mask(pg2)
    # masked = pg.get()
    
    
    # ##### CASE 5 #####
    # ##### A Test with pg.get() #####
    # ##### Weird: with P2D, once pg.get() is called, pg becomes null. This doesn't happen with default renderer
    # pg = createGraphics(200,200,renderer)
    # pg.beginDraw()
    # pg.background(255,255,255)
    # pg.image(input_copy,0,0)
    # pg.endDraw()
    # #####################
    # # # Option A - works to keep original pg
    # masked = pg
    # #####################
    # # # Option B - pg disappears once pg.get() is called
    # # some_var = pg.get()
    # # masked = pg
    # #####################

    
def draw():
    background(255)
    pushMatrix()
    image(input_image,0,0,200,200)
    text("input image",10,30)
    translate(200,0)
    image(mask_input_image,0,0,200,200)
    text("input mask",10,30)
    translate(-100,200)
    image(masked,0,0,200,200)
    popMatrix()

jeremydouglass · June 6, 2019, 3:16pm

These could be bugs worth reporting – and they might or might not be related. I wonder if they are in Python mode specifically, or if they are also in the underlying Java mode? However, the second possibly builds on the first – I’d suggest starting with the first one, work that out, and if you want to submit the second one now listing it as possibly related to the first.

The first one sounds like it might be specific to Python mode – but I have a question, why are you using “.get()” and not “.copy()” to make a copy? Is this an intentional difference in the python mode api?

aaronshenhao · June 6, 2019, 3:32pm

I haven’t looked at the code itself, but I tried CASE 2 it in Processing Java, which produced the same results. The get or copy method doesn’t seem to make a difference. P2D and P3D behaves the same. Only the renderer of the PGraphics canvas affects the outcome.

jeremydouglass · June 6, 2019, 4:22pm

Do you mean you get the same bug in processing Java? Or do you mean that in Java, case two is not buggy and JAVA2D and P2D results are equivalent?

aaronshenhao · June 6, 2019, 4:27pm

I’m getting the same result in Java. Not sure if it’s a bug, since the reference only lists valid arguments to the mask method for PImage as a mask array and or another PImage. I guess Processing is doing some behind the scenes pre-processing? I’ve tried implementing my own PImage class from the source, but I’m getting the error “The method mask(int[]) in the type PImageD is not applicable for the arguments (PGraphics)” currently. Trying to resolve it.

jeremydouglass · June 6, 2019, 4:34pm

Sorry, again, I don’t understand. I’m asking you to tell me WHAT is the same –

the processing.py results and the java results for your case 2, or
the Java results for P2D and the java results for JAVA2D – different in processing.py, but the same in Java

You have done a lot of investigation and so are working with multiple cases across two renderers (and now in two languages). Perhaps we can simplify by starting with establishing one single unexpected behavior in Java, describe that separately from everything else, and then build back up.

Edit: “you have done” – just realized this wasn’t the OP, sorry I was confused.

jeremydouglass · June 6, 2019, 4:36pm

Yes, but a PGraphics is a PImage.

http://processing.github.io/processing-javadocs/core/processing/core/PGraphics.html

aaronshenhao · June 6, 2019, 4:41pm

I am getting the same results as alexolotl for case 2 in Processing Java, the bottom image isn’t showing when using P2D or P3D in the PGraphics, but does when using Java2D.

Some comments from the source code that may be of relevance:

PGraphics subclasses PImage so that it can be drawn and manipulated in a similar fashion. As such, many methods are inherited from PGraphics, though many are unavailable: for instance, resize() is not likely to be implemented; the same goes for mask(), depending on the situation.

For image object in PGraphics (does mask get int array from here?):

Java AWT Image object associated with this renderer. For the 1.0 version
of P2D and P3D, this was associated with their MemoryImageSource.
For PGraphicsJava2D, it will be the offscreen drawing buffer.

jeremydouglass · June 6, 2019, 4:54pm

That’s really helpful about mask source comments – “depending on the situation” does sound mysterious.

Here is an example I’m able to run – it shows a PGraphics-on-PImage mask, or a PGraphics-on-PGraphics mask. It shows both outputs. You can change the size line to either JAVA2D or P2D. Both work correctly on my machine – so all four work correctly:


ok	JAVA2D	PGraphics-on-PImage
ok	JAVA2D	PGraphics-on-PGraphics
ok	P2D	PGraphics-on-PImage
ok	P2D	PGraphics-on-PGraphics

PImage photo;
PGraphics photo2;
PGraphics maskImage;
void setup() {
  size(1024, 512, JAVA2D);
  // size(1024, 512, P2D);
  photo = loadImage("https://forum.processing.org/processing-org.jpg");
  photo2 = createGraphics(512,512);
  photo2.beginDraw();
  photo2.image(photo,0,0);
  photo2.endDraw();
  // create mask
  maskImage = createGraphics(512,512);
  maskImage.beginDraw();
  maskImage.triangle(30, 480, 256, 30, 480, 480);
  maskImage.endDraw();
  // apply mask
  photo.mask(maskImage);
  photo2.mask(maskImage);
}
void draw() {
  // show masked image
  image(photo, 0, 0);
  image(photo2, 512, 0);
}

aaronshenhao · June 6, 2019, 4:59pm

Nice, that code simplifies it a lot! Changing the renderer of the main canvas does not affect it though, if you add the P2D renderer in createGraphics() for masked image and also in size(), it doesn’t work like in case 2:

size(1024, 512, P2D);
maskImage = createGraphics(512,512, P2D);

I created a test example where I’ve modified case 2 slightly. I reference the AWT Image associated with the PGraphics, and insert it into the built in constructor of PImage which takes in an AWT Image. I then pass the PImage into the mask method instead. If the renderer is JAVA2D, all goes well. However if the renderer is P2D/P3D, you get a null pointer exception. Not sure if this is what happens behind the scenes (doesn’t it use pixels array instead?), but the behaviour seems similar to what’s been happening, except that we don’t get an error message.

PGraphics pg = createGraphics(200, 200, JAVA2D);
pg.beginDraw();
//pg.background(255, 255, 255);
pg.image(mask_input_copy, 0, 0);
println(pg.image);
PImage test = new PImage(pg.image);
pg.endDraw();
input_copy.mask(test);
masked = input_copy.copy();

jeremydouglass · June 6, 2019, 5:19pm

Okay, I see what is at issue here in one bug – the renderer of the specific [EDIT] base image PGraphics, not the sketch renderer. Here is a test that produces a mis-oriented masking – but only when the base image is a PGraphics AND its renderer is P2D.

PImage photoPImage;
PGraphics photoPGJ2D;
PGraphics photoPGP2D;
PGraphics maskImage;
void setup() {
  size(1024, 512, P2D);
  photoPImage = loadImage("https://forum.processing.org/processing-org.jpg");
  photoPGJ2D = createGraphics(512,512);
  photoPGJ2D.beginDraw();
  photoPGJ2D.image(photoPImage,0,0);
  photoPGJ2D.endDraw();
  photoPGP2D = createGraphics(512,512, P2D);
  photoPGP2D.beginDraw();
  photoPGP2D.image(photoPImage,0,0);
  photoPGP2D.endDraw();
  // create mask
  maskImage = createGraphics(512,512);
  maskImage.beginDraw();
  maskImage.triangle(30, 480, 256, 30, 480, 480);
  maskImage.endDraw();
  // apply mask
  photoPImage.mask(maskImage);
  photoPGJ2D.mask(maskImage);
  photoPGP2D.mask(maskImage);
}
void draw() {
  // show masked image
  image(photoPImage, 0, 0);
  image(photoPGP2D, 256, 0);
  image(photoPGJ2D, 522, 0);
}

PGraphicsMaskFlippedBug

The same mask was applied to three images, but on the center image it is incorrect – it applies upside-down. That image is a PGraphics with a P2D renderer.

GoToLoop · June 6, 2019, 5:46pm

PImage::get() method has always been able to clone a PImage:

Actually, get() methods has always been associated w/ object cloning before they came up w/ the copy() idea.

Although I wonder why Processing devs didn’t simply choose clone() for it from the start:
Docs.Oracle.com/en/java/javase/11/docs/api/java.base/java/lang/Cloneable.html

aaronshenhao · June 6, 2019, 5:47pm

I tested the values of the pixels array (line 95 in PImage) for the PGraphics object with both renderers, in addition to the AWT image (line 545 in PGraphics) which we did before. If the renderer is Java2D, they are both filled with values. However, if the renderer is P2D/P3D, they both return null (array throws null error). Modified Case 2 example:

    PGraphics pg = createGraphics(200, 200, P2D);
    pg.beginDraw();
    pg.image(mask_input_copy, 0, 0);
    pg.endDraw();
    println(pg.image);
    println(pg.pixels.length);
    input_copy.mask(pg);
    masked = input_copy.copy();

Perhaps this could be the source of the problem? After all, the source code (line 992 in PImage) for the mask method (that takes in a PImage) is:

  public void mask(PImage img) {
    img.loadPixels();
    mask(img.pixels);
}

The mask() would be applied to img.pixels, which is null when P2D/P3D is used. When Java2D is used, they are populated. Perhaps this is what the comment in the source code meant about mask sometimes not working.

Edit (why no errors are thrown): img.loadPixels() in the mask() method above initializes the pixels array to 0 if it is null so no errors are thrown. (Not sure what the second if condition is, perhaps to account for resize?). This seems to be exactly what the code comments were referring to, and probably accounts for all the unexpected behaviours. Line 438 in PImage:

  public void loadPixels() {  // ignore
    if (pixels == null || pixels.length != pixelWidth*pixelHeight) {
      pixels = new int[pixelWidth*pixelHeight];
    }
    setLoaded();
}

neilcsmith · June 6, 2019, 8:38pm

Actually, they did but Cloneable is a pretty broken concept, and had specific issues, as well as Processing pre-dating covariant return types.

Now, speaking of return types, it is slightly odd really that copy() isn’t overridden to return the same type, as clone would do - that might be part of the issue here?!

GoToLoop · June 6, 2019, 8:43pm

PImage::get() has been meant to return an actual PImage since its inception.

There are many classes that inherit from PImage; like PGraphics, Movie, Capture, etc.

And each 1 of those subclasses deal w/ hardware, while a pure PImage doesn’t.

A PImage object is just pixels[] + width + height.

neilcsmith · June 6, 2019, 9:08pm

You’re the one who brought up clone() Having a copy() that actually returns a, you know, copy would be rather useful, and potentially less problematic.

GoToLoop · June 6, 2019, 9:27pm

Indeed… but in this case, I’ve just meant the word clone obviously makes more sense than get.

Changing get to copy is just another attempt for a more logical name.

Although I would prefer they’d go w/ clone rather than copy.

neilcsmith · June 6, 2019, 9:51pm

They already went with clone() - so use it if you want. But having a method behave differently than it’s defined to isn’t great, and the ways it’s defined is broken. So copy() makes sense. Now, adjusting the semantics of that might help with the issue in this thread. Certainly having a copy mechanism that is hardware / subclass aware is needed.

GoToLoop · June 6, 2019, 10:08pm

I didn’t know it. But it’s pretty useless given we need to use a try {} / catch () {} block for it, even though it’s merely a get() alias:

github.com

processing/processing/blob/processing-0269-3.5.3/core/src/processing/core/PImage.java#L521-L523


      
          public Object clone() throws CloneNotSupportedException {  // ignore
            return get();
          }

And both get() & copy() are just alias for return get(0, 0, pixelWidth, pixelHeight);:

github.com

processing/processing/blob/processing-0269-3.5.3/core/src/processing/core/PImage.java#L804-L813


      
          public PImage get() {
            // Formerly this used clone(), which caused memory problems.
            // http://code.google.com/p/processing/issues/detail?id=42
            return get(0, 0, pixelWidth, pixelHeight);
          }
          
          
          public PImage copy() {
            return get(0, 0, pixelWidth, pixelHeight);
          }

GoToLoop · June 7, 2019, 12:00am

Here’s my attempt of cloning a PGraphics, for both Java & Python modes:

Java Mode:

/**
 * Cloned PGraphics (v1.0)
 * GoToLoop (2019/Jun/06)
 *
 * https://Discourse.Processing.org/t/
 * p2d-vs-java2d-strange-differences-in-
 * pgraphics-methods-and-masking/11853/20
 */

static final String RENDER = P3D, URL =
  "https://" + "Forum.Processing.org/" + "processing-org.jpg";

PImage img;
PGraphics clonedPg;

void settings() {
  img = loadImage(URL);
  size(img.width, img.height, RENDER);

  smooth(8);
  noLoop();
}

void setup() {
  // Buggy for all renderers but JAVA2D under PDE 3.5.3:
  //final PGraphics pg = createGraphics(width, height, RENDER);

  final PGraphics pg = createGraphics(width, height);

  pg.beginDraw();
  pg.background(img);
  pg.endDraw();

  clonedPg = clonePGraphics(pg);
}

void draw() {
  background(clonedPg);
}

PGraphics clonePGraphics(final PGraphics pg) {
  final int w = pg.width, h = pg.height;
  final String render = getPGraphicsType(pg);
  final PGraphics cloned = createGraphics(w, h, render);

  cloned.beginDraw();
  cloned.loadPixels();
  cloned.endDraw();

  pg.loadPixels();
  arrayCopy(pg.pixels, cloned.pixels);
  cloned.updatePixels();

  return cloned;
}

static final String getPGraphicsType(final PGraphics pg) {
  if (pg.isGL()) return pg.is3D()? P3D : P2D;
  if (pg instanceof processing.javafx.PGraphicsFX2D) return FX2D;
  return JAVA2D;
}

Python Mode:

"""
 * Cloned PGraphics (v1.0)
 * GoToLoop (2019/Jun/06)
 *
 * https://Discourse.Processing.org/t/
 * p2d-vs-java2d-strange-differences-in-
 * pgraphics-methods-and-masking/11853/20
"""

RENDER = P3D
URL = "https://" + "Forum.Processing.org/" + "processing-org.jpg"

def settings():
    global img
    img = loadImage(URL)

    size(img.width, img.height, RENDER)
    smooth(8)
    noLoop()


def setup():
    # Buggy for all renderers but JAVA2D under PDE 3.5.3:
    # pg = createGraphics(width, height, RENDER)

    pg = createGraphics(width, height)

    with pg.beginDraw(): pg.background(img)

    global clonedPg
    clonedPg = clonePGraphics(pg)


def draw(): background(clonedPg)


def clonePGraphics(pg):
    w, h = pg.width, pg.height
    render, path = getPGraphicsType(pg), pg.path
    cloned = createGraphics(w, h, render, path)

    with cloned.beginDraw(): cloned.loadPixels()

    pg.loadPixels()
    arrayCopy(pg.pixels, cloned.pixels)
    cloned.updatePixels()

    return cloned


def getPGraphicsType(pg):
    if pg.isGL(): return pg.is3D() and P3D or P2D
    if hasattr(pg, 'snapshotImage'): return FX2D
    return JAVA2D

Indeed, when “cloning” any non-JAVA2D PGraphics, I get an empty 1 when using PDE 3.5.3!

However, when using older PDE versions 3.4 & 3.3.5 I still have laying around here, I could uncomment final PGraphics pg = createGraphics(width, height, RENDER); and I’ve still got a perfect cloned PGraphics!

Well, except for FX2D; which still today, can’t even be used for createGraphics() w/o crashing!

Topic		Replies	Views
PGraphics Empty with copy() Function Processing	15	1572	July 23, 2018
Strange Behavior Triggered by image() Function Coding Questions	19	1449	January 28, 2019
[SOLVED] Strange behavior using PGraphics as a recursive draw buffer Coding Questions	9	678	January 28, 2019
Errors of type "this.g" is null when accessing pure java sketch that extends PApplet as a BufferedImage, output only on 100x100 pixels, other PGraphics objects like size() cannot be used Coding Questions	10	2734	September 1, 2022
Weird issue with pgraphics Coding Questions	3	596	July 25, 2018

P2D vs JAVA2D - strange differences in PGraphics methods and masking

Java Mode:

Python Mode:

Related topics