Strange Behavior Triggered by image() Function

Hey there. First post, haven’t figured out if there’s anywhere else I could go for similar questions, thought I should ask here before creating an issue on Processing’s github issues board.

I’ve found some really strange behavior regarding a piece of code I’m writing, and I’ve isolated the issue here. This might be a java scope issue, a Reference/Value issue, or it could be an issue caused by the way Processing is intended to work.

Rather than explain what my issue is and how I’ve tried to solve it, and since I’m quite stuck, let me just start out with a demo.

PGraphics spriteImage;
MajorObject container;

class MajorObject{
  ArrayList<Object> contains;
  PGraphics localCanvas;
  MajorObject(){
    contains=new ArrayList<Object>();
    localCanvas=createGraphics(180,30);
  }
  void addObject(Object takingInObject){
    contains.add(takingInObject);
  }
  PImage displayObjects(){
    localCanvas.beginDraw();
    localCanvas.clear();
    for(int i=0; i<contains.size();i++){
      localCanvas.image(contains.get(i).getImage(),0,20*i);
    }
    localCanvas.endDraw();
    return localCanvas;
  }
  void printContainer(){
    image(displayObjects(),10,110);
  }
  PImage getOImage(int i){
    return contains.get(i).getImage();
  }
}

class Object{
  PImage storedImage;
  Object(PImage takingInImage){
    storedImage=takingInImage;
  }
  PImage getImage(){
    return storedImage;
  }
}

void setup(){
  frameRate(2);
  size(200,150);
  spriteImage=createGraphics(180,10);
  spriteImage.beginDraw();
  spriteImage.stroke(255);
  spriteImage.fill(0);
  spriteImage.rect(0,0,179,9);
  spriteImage.endDraw();
  container=new MajorObject();
  container.addObject(new Object(spriteImage));
  container.addObject(new Object(spriteImage));
}

void draw(){
  background(0);
  spriteImage.beginDraw();
  spriteImage.line(frameCount,0,frameCount,10);
  spriteImage.endDraw();
  image(spriteImage,10,10);                  //1     display from global object [line 60]
  image(container.displayObjects(),10,30);   //2, 3  display buffer from container [line 61]
  image(container.getOImage(0),10,70);       //4     get value directly from Object's child [line 62]
  image(container.getOImage(1),10,90);       //5     possibly referencing original global obj [line 63]
  container.printContainer();                //6, 7  run container's print of c's buffer [line 64]
}

In this demo, I have an image spriteImage in the global scope. I also have a class MajorObject that can contain objects of another type called Object, which are simply containers that store images.

I’d like to be able to have MajorObject be a kind of UI Frame that can hold several UI components. There are several ways to implement this but the route that I’m going down.

I’d like several UI Objects to be able to pull from the same image and synchronize all changes between them. The way I thought about doing this would be to pass a reference to every new UI Object with that Image as a PImage object. I might also want it to be a PGraphics object so it is editable. In my example, setup() creates these objects and puts them inside of MajorObject.

What’s happening in my code is I can (by line number)
60. print the original Image object
61. print a buffer stored in MajorObject that is composed of multiple Object’s images
62/63. print the image object inside of each Object container directly (acts like line 60 for some reason)
64. print the buffer than MajorObject receives internally after compositing the Object’s stored images

The main issue I’m having: running image(globally scoped or locally referenced image, int, int) separates what the global definition of the original PGraphics object from whatever references are made to the original object

For Example: Running line 60/62/63 first makes it so all future changes made to spriteImage in the global scope don’t apply to any of the references to spriteImage that may exist inside of any of the objects.
Running lines 61/64 first makes it so all future changes made to spriteImage in the global scope don’t apply to any of the references to spriteImage that may exist in the global scope.
Running any of these lines in setup instantly separates the references made and makes it so changes to spriteImage only affect the references made to global scope
Running none of these lines allows all of them to stay synchronized until image() is run somewhere. (try it out, wrap lines 60-63 in an if(key==‘a’) statement and after a few seconds of runtime press the ‘a’ key

Anyways what I expected to happen was for all of these to remain references and stay synchronized OR for every new Object with it’s run to Object’s constructor to act as a copy constructor and not pass the image object by reference but neither of those things happened.

I’d love to get help in understanding this behavior in any way possible. I can already think of an easy fix for the purpose of what I’m doing (have the constructor for an object act as a copy constructor in order to make it so nothing passes by reference) but I’d also love the ability to forcibly keep everything a reference in order to shrink memory consumption.

I think I’ve narrowed down the problem. If you do a println(storedImage.hashCode()) in getImage() method Object, then you’ll get a number which is (probably) based the memory address of the object. I did similar things in different parts of the program so as far as I could tell there’s no copying happening (which is to be expected) all the variables are references to the same spriteImage.

I then commented out some sections and came to the conclusion the only bug is in the displayObjects(). So I tried:

  PImage displayObjects(){
    //localCanvas.beginDraw();
    //localCanvas.clear();
    for(int i=0; i<contains.size();i++){
      //localCanvas.image(contains.get(i).getImage(0),0,20*i);
      image(contains.get(i).getImage(0),0,20*i);
    }
    //localCanvas.endDraw();
    return localCanvas;
  }

Which worked seeming to indicate that the problem is with the localCanvas PGraphics. Seems like the clear() or image()(when applied to PGraphics) method isn’t working properly and I can’t really figure out why. I spent awhile trying to figure it out, but this solution seemed to work even if it’s not the most efficient.

  PImage displayObjects(){
    localCanvas=createGraphics(180,30);
    localCanvas.beginDraw();
    for(int i=0; i<contains.size();i++){
      localCanvas.image(contains.get(i).getImage(0),0,20*i);
    }
    localCanvas.endDraw();
    return localCanvas;
  }

Might look into it more later and see if there’s something I’m missing. And you’re completely right that you would just copy the image in the constructor if you want each image to be independent.

Side note: I know this is an example and in your real program you would come up other names for classes but I would still avoid using Object as a name class in examples. Because Object the most general Class in Java and is used in inheritance. So when you use it in a method signature like void addObject(Object takingInObject){ Java will think you mean a java.lang.Object (and by inheritance every object) not specifically your Object. Which seemed to work out fine in this case but if you do this regularly you might get some weird implicit casting that might be hard to debug.

1 Like

println(storedImage.hashCode())
Genius! I even tried looking up how to detect if an object is a reference or not in java and couldn’t find it. this is a good solution.

I’m starting to come to terms with that too and I might have some more insight to that:

MajorObject container;

class MajorObject{
  ArrayList<Object> contains;
  PGraphics localCanvas;
  MajorObject(){
    contains=new ArrayList<Object>();
    localCanvas=createGraphics(180,30);
  }
  void addObject(Object takingInObject){
    contains.add(takingInObject);
  }
  PImage displayObjects(){
    localCanvas.beginDraw();
    localCanvas.clear();
    for(int i=0; i<contains.size();i++){
      image(contains.get(i).getImage(),10,30); //18 draws the internal reference to the image onto the screen
      localCanvas.image(contains.get(i).getImage(),0,20*i); //19 draws the image locally to the internal canvas
    }
    localCanvas.endDraw();
    return localCanvas;
  }
  void printContainer(){
    image(displayObjects(),10,50);
  }
}

class Object{
  PImage storedImage;
  Object(PImage takingInImage){
    storedImage=takingInImage;
  }
  PImage getImage(){
    PImage tempImage=storedImage;
    return tempImage;
  }
}

void setup(){
  frameRate(2);
  size(200,150);
  spriteImage=createGraphics(180,10);
  spriteImage.beginDraw();
  spriteImage.stroke(255);
  spriteImage.fill(0);
  spriteImage.rect(0,0,179,9);
  spriteImage.endDraw();
  container=new MajorObject();
  container.addObject(new Object(spriteImage));
  container.addObject(new Object(spriteImage));
}

void draw(){
  background(0);
  spriteImage.beginDraw();
  spriteImage.line(frameCount,0,frameCount,10);
  spriteImage.endDraw();
  image(spriteImage,10,10);                  //59    display from global object
  container.printContainer();                //60    draw global object's reference in buffer
}

updated code here

I no longer have 7 different examples of funky behavior, only 4 now. Seems like you can specifically toggle between the two types of behavior by commenting out lines 18 and 59.

I’m preparing to write an issue on Processing’s github page but gathering more information will make it seem like a much better idea to submit an issue. You’ve been helpful.

duly noted but you were right in assuming I was only using it for this example. :sweat_smile: My actual code looks a lot cleaner than any of this does too.

1 Like

Why are you using it at all and not using a list of PImage?

What do you mean? Everything is a reference unless it’s a primitive value.

I’d like to contain a PImage inside of a class so I can add functionality to it and add abstraction, like being able to consider an object something other than just a PImage. I thought this was clear as it’s a fundamental trait of object oriented programming to be able to encapsulate things. Also, the original piece of text you quoted was only in reference to a naming convention and had nothing to do with implementation. The subject of that sentence (what I was “using”) was the name Object and nothing more.

Once again I’m speaking on the level of abstraction. If I was to clarify what I was saying I would say “how to detect if an object is a reference to a particular object or not” but I was hoping that would have been made clear by the context.

Still, the best approach for assets is to load them all at once within setup(). :smirk:

After that, we can pass those already loaded assets as 1 of the arguments of our classes’ constructors. :bulb:

My main goal here is to dynamically generate assets during runtime, so things like character icons/UI objects can be edited and modified during real time (not like an animation) within the program I’m writing. I’d like it to be robust enough so that a group of objects that (such as the appearance of a generic “button” object that may have hundreds of instances) all share the same graphical design but can still change.

I do understand that static assets traditionally get loaded in during setup but I specifically have need for dynamically generated assets.

As a workaround in the meantime I’ve considered keeping an ArrayList of PGraphics objects in the global scope and storing only the array index location of each of them inside of each “Object”. The result is that even though every time we’re accessing the PGraphics object it’s changing, whenever it gets drawn to the PGraphics object stored inside of MajorObject (intended to be an offscreen buffer) the image that gets sent into the PGraphics object to be printed is stuck as when it was when it was first printed to the screen. Therefore this is a direct problem with PGraphics.image()'s behavior as figraham originally stated.

New replication steps:

  1. draw to “first” PGraphics object
  2. draw “first” PGraphics object to another “second” PGraphics object (either cast it to a PImage first or don’t, up to you)
  3. display “second” object and observe that it shows the same as the “first” after the “first” has been drawn to
  4. draw to “first” PGraphics object more
  5. display “first” PGraphics object (this step is crucial)
  6. draw “first” PGraphics object to “second” PGraphics object again
  7. display “second” object and observe how it hasn’t changed because either something went wrong with step 6 or it worked but the version of the “first” PGraphics object that the “second” received was incorrect and unchanged from the “first” PGraphics object’s last draw run.

Example Code:

PGraphics dataStream;
PGraphics offscreenBuffer;

size(280,280);
background(0);
dataStream = createGraphics(100, 100);
offscreenBuffer = createGraphics(100, 100);

//Part One
//draw unmodified offscreenBuffer to screen
text("unmodified", 0, 20);
text("offscreenBuffer", 0, 40);
image(offscreenBuffer, 0, 40);

//Part Two
//modify dataStream
dataStream.beginDraw();
dataStream.fill(255);
dataStream.rect(0,0,50,100);
dataStream.endDraw();

//draw dataStream to offscreenBuffer
offscreenBuffer.beginDraw();
offscreenBuffer.image(dataStream,0,0);
offscreenBuffer.endDraw();

//draw offscreenBuffer to screen
text("modified dataStream", 140, 20);
text("drawn to offscreenBuffer", 140, 40);
image(offscreenBuffer, 140, 40);

//Part Three
//modify dataStream again
dataStream.beginDraw();
dataStream.fill(255);
dataStream.rect(50,0,50,100);
dataStream.endDraw();

//draw dataStream to screen
text("modified dataStream", 0, 160);
text("drawing dataStream", 0, 180);
image(dataStream, 0, 180);

//draw dataStream to offscreenBuffer
offscreenBuffer.beginDraw();
offscreenBuffer.image(dataStream,0,0);
offscreenBuffer.endDraw();

//draw offscreenBuffer to screen again
text("drawn datastream", 140, 160);
text("to offscreenBuffer", 140, 180);
image(offscreenBuffer, 140, 180);

image
Here’s what the above code results in by the way.

Yes, it is, but as your class is not doing anything currently, and chaining up references (pointer to pointer) just complicates matters, it’s not helping you find where your bug is. Always simplify to the minimum code that reproduces the issue (MCVE) - you’ll probably find the issue, and if not it’ll help others.

Just use obj1 == obj2 to check for reference equality.

So, interestingly, adding a line to update the dataStream pixels before drawing it the second time fixes this - looks like you’ve found a bug on how offscreen PGraphics are marked modified after endDraw(), and by the look of it only when drawing to other offscreen surfaces. This only affects the default Java2D renderer. Using P2D in size() and createGraphics() works, although it flickers without using setup and draw (FX2D just throws a null pointer exception).

eg.

PGraphics dataStream;
PGraphics offscreenBuffer;

size(280,280);
background(0);
dataStream = createGraphics(100, 100);
offscreenBuffer = createGraphics(100, 100);

//Part One
//draw unmodified offscreenBuffer to screen
text("unmodified", 0, 20);
text("offscreenBuffer", 0, 40);
image(offscreenBuffer, 0, 40);

//Part Two
//modify dataStream
dataStream.beginDraw();
dataStream.fill(255);
dataStream.rect(0,0,50,100);
dataStream.endDraw();

//draw dataStream to offscreenBuffer
offscreenBuffer.beginDraw();
offscreenBuffer.image(dataStream,0,0);
offscreenBuffer.endDraw();

//draw offscreenBuffer to screen
text("modified dataStream", 140, 20);
text("drawn to offscreenBuffer", 140, 40);
image(offscreenBuffer, 140, 40);

//Part Three
//modify dataStream again
dataStream.beginDraw();
dataStream.fill(255);
dataStream.rect(50,0,50,100);
dataStream.endDraw();

//draw dataStream to screen
text("modified dataStream", 0, 160);
text("drawing dataStream", 0, 180);
image(dataStream, 0, 180);

dataStream.updatePixels(); // <---- ADD THIS!

//draw dataStream to offscreenBuffer
offscreenBuffer.beginDraw();
offscreenBuffer.image(dataStream,0,0);
offscreenBuffer.endDraw();

//draw offscreenBuffer to screen again
text("drawn datastream", 140, 160);
text("to offscreenBuffer", 140, 180);
image(offscreenBuffer, 140, 180);

1 Like

This is fascinating but also confusing.

I don’t have much if any experience with the P2D renderer as I mostly use Java2D and P3D but I’m switching over to it to do more testing and I’m finding weird results with all of the tests I’ve done so far.

demo 2 unchanged, java2d (unexpected result)
image

demo 2 with updatePixels(), java2d (expected result)
image

demo 2 with updatePixels(), same as above but with P2D (unexpected result)
image

demo 2 unchanged, P2D (unexpected result)
image

demo 1 unchanged, Java2D (unexpected result)
image

demo 1 with updatePixels(), Java2D (also unexpected result)
image

demo 1 with updatePixels(), P2D (unexpected result, somehow inverted)
image

using updatePixels() doesn’t seem to be an immediate fix nor does using P2D but we have valuable insight as well as one example of everything working as anticipated. Thank you very much Neil.

Can you share your P2D code without updatePixels()? Are you using it in the size and createGraphics lines? That works fine here, and this behaviour would completely break how PraxisLIVE works if it didn’t so I’ve done this a lot! :slight_smile:

1 Like

Actually, I didn’t even consider that. unchanged with P2D just has the main renderer in P2D mode, not either of the PGraphics objects.

My code is the same as I originally put up above, the only thing i’m changing in my tests is the presence of P2D mode as the main renderer and the presence of updatePixels on the original buffer.

PGraphics dataStream;
PGraphics offscreenBuffer;

size(280,280);
background(0);
dataStream = createGraphics(100, 100);
offscreenBuffer = createGraphics(100, 100);

//Part One
//draw unmodified offscreenBuffer to screen
text("unmodified", 0, 20);
text("offscreenBuffer", 0, 40);
image(offscreenBuffer, 0, 40);

//Part Two
//modify dataStream
dataStream.beginDraw();
dataStream.fill(255);
dataStream.rect(0,0,50,100);
dataStream.endDraw();

//draw dataStream to offscreenBuffer
offscreenBuffer.beginDraw();
offscreenBuffer.image(dataStream,0,0);
offscreenBuffer.endDraw();

//draw offscreenBuffer to screen
text("modified dataStream", 140, 20);
text("drawn to offscreenBuffer", 140, 40);
image(offscreenBuffer, 140, 40);

//Part Three
//modify dataStream again
dataStream.beginDraw();
dataStream.fill(255);
dataStream.rect(50,0,50,100);
dataStream.endDraw();

//draw dataStream to screen
text("modified dataStream", 0, 160);
text("drawing dataStream", 0, 180);
image(dataStream, 0, 180);

//draw dataStream to offscreenBuffer
offscreenBuffer.beginDraw();
offscreenBuffer.image(dataStream,0,0);
offscreenBuffer.endDraw();

//draw offscreenBuffer to screen again
text("drawn datastream", 140, 160);
text("to offscreenBuffer", 140, 180);
image(offscreenBuffer, 140, 180);

I just went through and tried putting P2D in different places and none of the results with only P2D mode for all buffers in use seem to really be correct now.

main renderer in p2d mode, both graphics buffers in p2d mode (unexpected results)
adding dataStream.updatePixels() changes nothing:
image

main renderer in p2d mode, first graphics buffer in p2d mode, offscreen buffer in java2d mode (unexpected results)
adding dataStream.updatePixels() changes nothing:
image

main renderer in p2d mode, first graphics buffer in java2d mode, offscreen buffer in p2d mode (expected results!)
adding dataStream.updatePixels() changes nothing:
image

main renderer in p2d mode, both graphics buffers in java2d mode (unexpected results)
adding dataStream.updatePixels() changes nothing:
image

main renderer in java2d mode, both graphics buffers in java2d mode (unexpected results)
image
dataStream.updatePixels() fixes the result and brings the expected output
image

so far
-keeping the main renderer in java2d mode and using updatePixels on the buffer I want to draw from just before drawing to the other buffer
and
-having the main renderer be in p2d mode, the graphics buffer I want to change be in java2d mode and the offscreen graphics buffer be in p2d mode

are the two options that work the best.

Following this pattern, I’ve implemented that process into my first demo, by making it so any offscreen buffers are P2D objects and any editable Graphics assets are java2d canvases
image
doing that gives the expected output

code:

ArrayList<PGraphics> spriteImage;
MajorObject container;

class MajorObject{
  ArrayList<Object> contains;
  PGraphics localCanvas;
  MajorObject(){
    contains=new ArrayList<Object>();
    localCanvas=createGraphics(180,30,P2D);
  }
  void addObject(Object takingInObject){
    contains.add(takingInObject);
  }
  PImage displayObjects(){
    localCanvas.beginDraw();
    localCanvas.clear();
    for(int i=0; i<contains.size();i++){
      PImage granadaImage=contains.get(i).getImage();
      localCanvas.image(granadaImage,0,20*i);
    }
    localCanvas.endDraw();
    return localCanvas;
  }
  void printContainer(){
    image(displayObjects(),10,110);
  }
  PImage getOImage(int i){
    return contains.get(i).getImage();
  }
}

class Object{
  int imageRef;
  Object(int iRef){
    imageRef=iRef;
  }
  PImage getImage(){
    return spriteImage.get(imageRef);
  }
}

void settings(){
  size(200,150,P2D);
}

void setup(){
  frameRate(2);
  spriteImage=new ArrayList<PGraphics>();
  spriteImage.add(createGraphics(180,10));
  spriteImage.get(0).beginDraw();
  spriteImage.get(0).stroke(255);
  spriteImage.get(0).fill(0);
  spriteImage.get(0).rect(0,0,179,9);
  spriteImage.get(0).endDraw();
  container=new MajorObject();
  container.addObject(new Object(0));
  container.addObject(new Object(0));
}

void draw(){
  background(0);
  spriteImage.get(0).beginDraw();
  spriteImage.get(0).line(frameCount,0,frameCount,10);
  spriteImage.get(0).endDraw();
  PImage canadaDryImage=spriteImage.get(0);
  image(canadaDryImage,10,10);                  //1     display from global object [line 60]
  image(container.displayObjects(),10,30);   //2, 3  display buffer from container [line 61]
  image(container.getOImage(0),10,70);       //4     get value directly from Object's child [line 62]
  image(container.getOImage(1),10,90);       //5     possibly referencing original global obj [line 63]
  container.printContainer();                //6, 7  run container's print of c's buffer [line 64]
}

These are results!

You mean this doesn’t work for you?? Correct results here -

PGraphics dataStream;
PGraphics offscreenBuffer;

size(280,280, P2D);
background(0);
dataStream = createGraphics(100, 100, P2D);
offscreenBuffer = createGraphics(100, 100, P2D);

//Part One
//draw unmodified offscreenBuffer to screen
text("unmodified", 0, 20);
text("offscreenBuffer", 0, 40);
image(offscreenBuffer, 0, 40);

//Part Two
//modify dataStream
dataStream.beginDraw();
dataStream.fill(255);
dataStream.rect(0,0,50,100);
dataStream.endDraw();

//draw dataStream to offscreenBuffer
offscreenBuffer.beginDraw();
offscreenBuffer.image(dataStream,0,0);
offscreenBuffer.endDraw();

//draw offscreenBuffer to screen
text("modified dataStream", 140, 20);
text("drawn to offscreenBuffer", 140, 40);
image(offscreenBuffer, 140, 40);

//Part Three
//modify dataStream again
dataStream.beginDraw();
dataStream.fill(255);
dataStream.rect(50,0,50,100);
dataStream.endDraw();

//draw dataStream to screen
text("modified dataStream", 0, 160);
text("drawing dataStream", 0, 180);
image(dataStream, 0, 180);

//draw dataStream to offscreenBuffer
offscreenBuffer.beginDraw();
offscreenBuffer.image(dataStream,0,0);
offscreenBuffer.endDraw();

//draw offscreenBuffer to screen again
text("drawn datastream", 140, 160);
text("to offscreenBuffer", 140, 180);
image(offscreenBuffer, 140, 180);

Yeah, like I said, when I ran everything in P2D mode, I got this as a result
image
which is incorrect. The bottom left image should have two rectangles in it, as dataStream has two rectangles drawn to it by that point in time. That’s the same code that I ran…

What did you get as a result? Does PraxisLIVE give different results in this situation?

Yes, the code I shared above works fine here, with two rectangles on both bottom images.

This is testing in Processing 3.4, not PraxisLIVE.

Alrighty, I just went back and tried 3.4 and you’re right it does work in 3.4. This problem just got a lot more complicated. What you did does appear to work correctly in Processing 3.4 but not in 3.5.2.

Thank you for opening the issue on github – it is here:

2 Likes