Using shapes as a video player

Processing Libraries
1

I’m trying to use shapes as part of a video player. For example, I code for an ellipse at a certain part in the window, and you only see the video through this ellipse. And then, take the pixels that are contained within those shapes and move them around the screen. I’d appreciate any help. Code I have so far:

import processing.video.*;

int videoScale = 8;
int cols, rows;

Movie movie;

void setup(){
  size(640, 480);
  cols = width/videoScale;
  rows = height/videoScale;

movie = new Movie(this, "lava bubbles volcano.mp4");

movie.loop();
}

void movieEvent(Movie movie){
  movie.read();
  }
  
void draw(){
  image(movie, 0, 0);
  tint(255);
}
2

I decided to use a PGraphics object as a buffer to draw the mask and then your video in. There, I make it fully black transparent using blendMode(REPLACE) and background(0,0);. After that, I draw a mask(i.e. the needed shape) with 2 circles and a colored rectangle as an example. After that, I use blendMode(MULTIPLY) and draw the video ontop.
Then I pass the buffer on the screen with needed coordinates.

import processing.video.*;

int videoScale = 8;
int cols, rows;

Movie movie;
//PImage movie;

PGraphics movieBuffer;

int posX, posY; //Example position of the buffer.

void setup(){
  size(640, 480);
  cols = width/videoScale;
  rows = height/videoScale;

movie = new Movie(this, "lava bubbles volcano.mp4");
//movie = loadImage(sketchPath("img.png"));
movieBuffer = createGraphics(640,480);
movie.loop();
}

void movieEvent(Movie movie){
  movie.read();
  }
  
void draw(){
  movieBuffer.beginDraw();
  movieBuffer.blendMode(REPLACE);
  movieBuffer.background(0,0);
  movieBuffer.blendMode(BLEND);
  //Define your mask here. Here's an example with shapes and colors:
  movieBuffer.noStroke();
  movieBuffer.fill(255);
  movieBuffer.ellipse(100,100,100,100);
  movieBuffer.ellipse(125+sin(millis()/500.0)*50,130,100,100); //A shape that constantly changes for fun
  movieBuffer.fill(255,127,0);
  movieBuffer.rect(125,130,150,20);
  //End of mask.
  movieBuffer.blendMode(MULTIPLY);
  movieBuffer.image(movie,0,0);
  movieBuffer.endDraw();
  
  if(mousePressed){ //Allows for dragging and dropping
    posX+=mouseX-pmouseX;
    posY+=mouseY-pmouseY;
  }
  background(0,127,255);
  image(movieBuffer, posX, posY);
}

It’s not the best, as blendMode(MULTIPLY); seems to be broken and ignores alpha channel of the mask, filling everything else with black. However, you can replace background(0,127,255); with background(0); to have the background behind the buffer also being black, keeping it all consistent.
EDIT: The issue above is fixable by converting the sketch to P2D and using a scary alien function as seen in reply #9 below.

Ignore the commented lines - I used them to test with an image because I don’t happen to have a video laying around to test it with. But it should work the same as it did for me.
Hopefully this bunch of information isn’t too hard to chew through! :smiley:

3

Wow, thank you for this! It’s going to take me a while to digest this, but I appreciate the help. :slight_smile:

4

You’re welcome!
I suggest using Processing reference for this!
https://processing.org/reference/PGraphics.html
https://processing.org/reference/blendMode_.html

1 Like
5

If I wanted to have multiple shapes on the screen, but control them independently, how would I go about doing that?

6

Same way you would control multiple normal shapes - whatever way you want to - except you draw them on movieBuffer instead.

I guess you could make keys 1 to 0 switch an int to according value, and then, based on what that int is set, make mouse move one figure or the other. You’d also need position variables for every shape, and have that code that moves it around for every shape…

In other words, the code that I gave you features this:

  if(mousePressed){ //Allows for dragging and dropping
    posX+=mouseX-pmouseX;
    posY+=mouseY-pmouseY;
  }

You could do instead:

  if(mousePressed){ //Allows for dragging and dropping
    if(figureSelected == 1){
      posX1+=mouseX-pmouseX;
      posY1+=mouseY-pmouseY;
    } else
    if(figureSelected == 2){
      posX2+=mouseX-pmouseX;
      posY2+=mouseY-pmouseY;
    } else ...
  }

And then, in movieBuffer mask, you could do instead:

  //Define your mask here. Here's an example with shapes and colors:
  movieBuffer.noStroke();
  movieBuffer.fill(255);
  movieBuffer.ellipse(posX1,posY1,100,100); //Shapes with different controllable positions...
  movieBuffer.ellipse(posX2,posY2,100,100); 
  movieBuffer.fill(255,127,0);
  movieBuffer.rect(125,130,150,20);
  //End of mask.

For changing the figureSelected thing, you could add to your code:

void keyPressed(){
  if(key == '1') figureSelected = 1;
  if(key == '2') figureSelected = 2;
  ...
}
7

I just wanna make sure I have it right. Here’s what I have:

import processing.video.*;

int videoScale = 8;
int cols, rows;
int figureSelected;

Movie movie;
//PImage movie;

PGraphics movieBuffer;

int posX1, posY1, posX2, posY2; //Example position of the buffer.

void setup(){
  size(640, 480);
  cols = width/videoScale;
  rows = height/videoScale;

movie = new Movie(this, "lava bubbles volcano.mp4");
//movie = loadImage(sketchPath("img.png"));
movieBuffer = createGraphics(640,480);
movie.loop();
}

void movieEvent(Movie movie){
  movie.read();
  }
  
  void draw(){
  movieBuffer.beginDraw();
  movieBuffer.blendMode(REPLACE);
  movieBuffer.background(0,0);
  movieBuffer.blendMode(BLEND);
  //Define your mask here. Here's an example with shapes and colors:
  movieBuffer.noStroke();
  movieBuffer.fill(255);
  movieBuffer.ellipse(posX1,posY1,100,100); //Shapes with different controllable positions...
  movieBuffer.ellipse(posX2,posY2,100,100); 
  movieBuffer.fill(255,127,0);
  movieBuffer.rect(125,130,150,20);
  //End of mask.
  movieBuffer.blendMode(MULTIPLY);
  movieBuffer.image(movie,0,0);
  movieBuffer.endDraw();
  
      if(mousePressed){ //Allows for dragging and dropping
    if(figureSelected == 1){
      posX1+=mouseX-pmouseX;
      posY1+=mouseY-pmouseY;
    } else
    if(figureSelected == 2){
      posX2+=mouseX-pmouseX;
      posY2+=mouseY-pmouseY;
    }
  }
  
  background(0,127,255);
  image(movieBuffer, posX1, posY1);

  }
  
  void keyPressed(){
  if(key == '1') figureSelected = 1;
  if(key == '2') figureSelected = 2;
  
}

I’m trying to figure out a way to see the ellipses but the video I’m using has a lot of black in it, and I think the ellipses are in the black part, making it impossible to see where they’re at.

And this part I’m a little confused with:

  background(0,127,255);
  image(movieBuffer, posX1, posY1);

How can I make all the shapes appear with this part, and be able to move them independently?

8

This thread might be useful but requires using P2D or P3D blendMode to reduce opacity? - #2 by neilcsmith

1 Like
9

That actually works! Thanks!

Although it needs to have the main sketch in P2D mode so that movieBuffer would be a P2D PGraphics too, so that it could be cast to PGraphicsOpenGL… Not a huge problem though.

Here’s my code above with that thing implemented:

import processing.video.*;

int videoScale = 8;
int cols, rows;

Movie movie;
//PImage movie;

PGraphics movieBuffer;

int posX, posY; //Example position of the buffer.

void setup(){
  size(640, 480, P2D); //NEW: Make this P2D.
  cols = width/videoScale;
  rows = height/videoScale;

movie = new Movie(this, "lava bubbles volcano.mp4");
//movie = loadImage(sketchPath("img.png"));
movieBuffer = createGraphics(640,480, P2D); //NEW: Make that P2D too
movie.loop();
}

void movieEvent(Movie movie){
  movie.read();
  }
  
void draw(){
  movieBuffer.beginDraw();
  movieBuffer.blendMode(REPLACE);
  movieBuffer.background(0,0);
  movieBuffer.blendMode(BLEND);
  //Define your mask here. Here's an example with shapes and colors:
  movieBuffer.noStroke();
  movieBuffer.fill(255);
  movieBuffer.ellipse(100,100,100,100);
  movieBuffer.ellipse(125+sin(millis()/500.0)*50,130,100,100); //A shape that constantly changes for fun
  movieBuffer.fill(255,127,0);
  movieBuffer.rect(125,130,150,20);
  //End of mask.
  movieBuffer.blendMode(MULTIPLY);
  ((PGraphicsOpenGL)movieBuffer).pgl.blendFunc(PGL.ZERO, PGL.SRC_COLOR);
  movieBuffer.image(movie,0,0);
  movieBuffer.endDraw();
  
  if(mousePressed){ //Allows for dragging and dropping
    posX+=mouseX-pmouseX;
    posY+=mouseY-pmouseY;
  }
  background(0,127,255);
  image(movieBuffer, posX, posY);
}
1 Like
10

Yes, far from not a problem, it’s a massive improvement. This way the multiply happens on the GPU rather than in Java code. Custom blend modes with the default renderer are slow! :slight_smile:

11

Also, sorry, I didn’t see your reply.
I guess by “all the shapes” you mean shapes that act as the mask through which one can see the video. That is not where you put them - look through the part of draw() with movieBuffer. repeated a lot of times - put your shapes with any code to control them in the marked out by the comments part with movieBuffer. before them - feel free to erase what I have already.

Also, for testing purposes, I suggest you temporarily replacing the video with something brighter - at least an image by uncommenting lines I left in the code. Although, with the fix introduced in the latest edit of my example code, you will have a bright background, letting you see the dark ellipses, so it doesn’t matter as much. :v

12

Oh, yeah, I noticed that. It’s so smooth now, thanks for informing!