Framerate in live video delay project : Ed Tannenbaum, Recollections

Adrien · August 31, 2018, 10:44am

Hi,

I try to create something like this : https://www.youtube.com/watch?v=YBKkVpRxUfs, using a kinect. At the moment I made some tests with something like this :

class User {

  PGraphics pgTotal; 
  PGraphics pgBodyBg; 
  PImage bodyImg; 
  DelayImg[] buffer;

  User() {
    pgTotal=createGraphics(512, 424);  //size of bodyImg
    buffer=new DelayImg[50];
  }

  void delayEffect() {
    fillBuffer(); 
    changeColor(); 
    display();
  }

  void fillBuffer() {
    PImage bi= bodyImg.copy(); 
    DelayImg di=new DelayImg(bi); 

    for (int i = buffer.length-1; i &gt; 0; i--) { //Shift all the image in the buffer
      buffer[i] = buffer[i-1];
    }
    buffer[0]=di; //buffer[0] is everytime the more recent frame 
  }

  void changeColor() {

    pgTotal.beginDraw(); 
    pgTotal.clear(); 
    pgTotal.pushStyle(); 

    for (int j=buffer.length-1; j&gt;=0; j--) { 
      if (buffer[j]!=null) buffer[j].changeColor(pgTotal);
    }
    pgTotal.popStyle();
    pgTotal.endDraw();
  }

  void display() {
    image(pgTotal, 0, 0);
  }
}



class DelayImg {

  PImage img; 
  PGraphics pgBg; 

  DelayImg(PImage _img) {
    img=_img; 
    img.filter(THRESHOLD); 
    pgBg=createGraphics(512, 424);
  }

  void changeColor(PGraphics pg) {

    pgBg.beginDraw(); 
    pgBg.background(random(255), random(255), random(255));  
    pgBg.mask(img); //Use this to be able to superposePGraphics with the shape of the body in color and transparency
    pgBg.endDraw();
    pg.image(pgBg, 0, 0);
  }
}

bodyImg is update each frame, coming from kinect, it’s a bi-color image. I filter it with THRESHOLD to be able to use is as a mask to change backgrounds color. I’m not using tint because I want to be able to have more complicate pattern in the futur.

I want to be able to change the color everytime, that why pgTotal is clear every frame and display fresh new images, but the way I do it is very slow (10 fps). Do you have any suggestion, different kind of algorithm, another approach to improve frameRate? I try to use some vertex and textures aswell, but no real improvement…

Thanks for your help!

A

Adrien · August 31, 2018, 11:20am

Perhaps the best approch will be to create one PImage (or PGraphics) with different “zones” like here :

and fill the different zones with different colors at each frame?
Do you have any idea how to do it? Or another approach?
Thanks!
A.

jb4x · August 31, 2018, 12:10pm

Hey,

I would advise you to time some part of your code to frame which part of your code is slowing everything down.

Also, are you using the P2D renderer? It might make it works a bit faster

Why do you need the pushStyle() and popStyle()? I thought it was used just for fill() stroke() noStroke() etc.

What you can do also is simply store all your black and white image in an array (like you do). Then your create a PImage with a random color for each one of them. You then use the mask() function to get the background color applied only where the person is on your alpha mask. And finally you combine them all at the end.

Hope it helps at least a bit!

Adrien · August 31, 2018, 12:38pm

Hi jb4x,

Thanks for taking time to answer me.

-The part of this code slowing everything down is the changeColor() method of the DelayImg class here

for (int j=buffer.length-1; j&gt;=0; j--) { 
      if (buffer[j]!=null) buffer[j].changeColor(pgTotal);
    }

-I’m already using P2D render
-You’re right, here I don’t need pushStyle and popStyle. It was here because I used to reduce alpha on old “layer”.
I don’t really understand the difference with what you sugest me :

What you can do also is simply store all your black and white image in an array (like you do). Then your create a PImage with a random color for each one of them. You then use the mask() function to get the background color applied only where the person is on your alpha mask. And finally you combine them all at the end.

and what I’m currently doing…Is it just replace the PGraphics I’m using with a PImage? (I’m using a PGraphics to be able to use the background method)

What I don’t understand is how Ed is managing and nearly unlimited nbr of frames/images. Do you thing is working with something like blobs, of time based paths…

Thanks!

A.

Adrien · August 31, 2018, 12:44pm

Perhaps he’s just working with the edge of the body shape

Something like blob or PShape, so another question could be :
Is there a way to automaticaly create a PShape from edge of an Image?

A.

jb4x · August 31, 2018, 12:53pm

There is none, I thought you were doing differently for some reasons

What you can try though is to not apply the mask at each frame but instead mask a fully white PGraphic when you create your object straight after the filter() call in DelayImg(). Then you would just have to tint it and display it and nothing more.

Adrien · August 31, 2018, 12:59pm

Thanks, but with your solution I wont be able to tint the different area with different color isn’t it?
A.

jb4x · August 31, 2018, 1:04pm

You would. I’ll try to be a bit more clear in my answer.

Every time you get a new pic from the Kinect:

You first apply a threshold
You create a complete white picture
You mask it with the kinect pic to get a white a transparent picture

The idea is to build an array of white and transparent picture representing all the position the person visited.

Then when you want to draw it you go through that array and for each image:

You tint it
You print it on top the the previous one (It should go from the oldest pic the the newest one then)

Adrien · August 31, 2018, 1:20pm

Thanks for your answer !

I just implement it :

PGraphics[] pgBuffer; 
PImage bodyImg; 
  User() {
    pgBuffer=new PGraphics[100];
  }


  void delayEffect() {
    createWhiteAndTransparentImage();
    colorAllPg(); 
  }

  void createWhiteAndTransparentImage() {
    bodyImg.filter(THRESHOLD); 
    PGraphics pgWhite=createGraphics(512, 424); 
    pgWhite.beginDraw(); 
    pgWhite.background(255); 
    pgWhite.mask(bodyImg); 
    for (int i = pgBuffer.length-1; i > 0; i--) {
      pgBuffer[i] = pgBuffer[i-1];
    }
    pgBuffer[0]=pgWhite;
  }

  void colorAllPg() {
    for (int i = pgBuffer.length-1; i > 0; i--) {
      if (pgBuffer[i]!=null) {
        PImage colorImg=(PImage)pgBuffer[i];
        tint(random(255), random(255), random(255)); 
        image(colorImg, 0, 0, width, height);
      }
    }
  }

And it’s definitly an improvement : I just go from 60fps to 30fps. I still “loose” 30fps, but it’s definitly better than before when I dropped at 10 fps…
I will continue to investigate, but thanks a lot already!
A.

jb4x · August 31, 2018, 1:30pm

for (int i = pgBuffer.length-1; i &gt; 0; i--) { 
  pgBuffer[i] = pgBuffer[i-1]; 
}

I also think that this loop is not really efficient. Instead of using a basic array, maybe use an ArrayList were the only thing you would have to do each frame is add the new image and get rid of the oldest one.

Adrien · August 31, 2018, 1:37pm

Juste made it :

class User {
  PImage bodyImg; 
  ArrayList<PGraphics> pgList; 
  User() {
    pgList = new ArrayList<PGraphics>();
  }

  void delayEffect() {
    createWhiteAndTransparentImage();
    colorAllPg();
  }

  void createWhiteAndTransparentImage() {
    bodyImg.filter(THRESHOLD); 
    PGraphics pgWhite=createGraphics(512, 424); 
    pgWhite.beginDraw(); 
    pgWhite.background(255); 
    pgWhite.mask(bodyImg); 
    pgWhite.endDraw();
    pgList.add(0, pgWhite);
    if (pgList.size()>100)pgList.remove(pgList.size()-1);
  }

  void colorAllPg() {

    for (int i = pgList.size()-1; i > 0; i--) {

      PImage colorImg=pgList.get(i);
      tint(random(255), random(255), random(255)); 
      image(colorImg, 0, 0, width, height); //This is the line that make me loose 30 fps : if I comment it, sketch running at 60fps. I tried to "print" all on a PGraphics and just display the PGraphics, but still the same fps lose...
    }
  }
}

Still around 30 fps…But thanks to take time to help me!
A.

jb4x · August 31, 2018, 1:47pm

I can see 2 more ways to increase framerate.

1. resize your PImage

When you create your white and transparent image, you can crop the image to get rid of all the transparent pixels on the side of the image. You would end up with a smaller image that you would draw in a specific (x, y) position to end up at the exact same spot as with the full image.

I should have an impact on the speed of the tint() function I think

2. Crop each image based on the one on top

Every time you get a new image, it will be set on top of the previous one. So what you can do is to delete the part of the previous image that is hidden by the new one. This way you would mask it a bit more and have even less pixels to deal with int the end, if combined with the first point.

Adrien · August 31, 2018, 1:54pm

Thanks I’ll try this!
A.

Adrien · August 31, 2018, 3:39pm

Try to make a first implementation of your first point ( crop PImage), but it actually make the frameRate drop dramatically (less than 10 fps).
Perhaps the idea is good but not my implementation…

class User {
  PImage bodyImg; 
  ArrayList<NewImg> imgList; 
  PGraphics pgTotal; 
  User() {
    imgList = new ArrayList<NewImg>();
    pgTotal=createGraphics(512, 424);
  }

  void delayEffect() {
    createWhiteAndTransparentImage();
    colorAllPg();
  }

  void createWhiteAndTransparentImage() {
    bodyImg.filter(THRESHOLD); 
    PGraphics pgWhite=createGraphics(512, 424); 
    pgWhite.beginDraw(); 
    pgWhite.background(255); 
    pgWhite.mask(bodyImg); 
    pgWhite.endDraw();
    NewImg newImg=new NewImg();
    newImg.trimImage(pgWhite); 
    imgList.add(0, newImg);
    if (imgList.size()>100)imgList.remove(imgList.size()-1);
  }

  void colorAllPg() {
    pgTotal.beginDraw(); 
    pgTotal.clear(); 
    for (int i = imgList.size()-1; i > 0; i--) {
      PImage colorImg=imgList.get(i).img;
      pgTotal.tint(random(255), random(255), random(255)); 
      float x=imgList.get(i).cornerPos.x; 
      float y=imgList.get(i).cornerPos.y; 
      pgTotal.image(colorImg, x, y);
    }
    pgTotal.endDraw(); 
    image(pgTotal, 0, 0, width, height);
  }
}

class NewImg {
  PVector cornerPos; 
  PImage img; 
  NewImg () {
  }


  void trimImage(PImage image) {
    image.loadPixels(); 
    int width = image.width;
    int height = image.height;
    int left = 0;
    int top = 0;
    int right = width - 1;
    int bottom = height - 1;
    int minRight = width - 1;
    int minBottom = height - 1;

  top:
    for (; top < bottom; top++) {
      for (int x = 0; x < width; x++) {
        if (alpha(image.get(x, top)) != 0) {
          minRight = x;
          minBottom = top;
          break top;
        }
      }
    }

  left:
    for (; left < minRight; left++) {
      for (int y = height - 1; y > top; y--) {
        if (alpha(image.get(left, y)) != 0) {
          minBottom = y;
          break left;
        }
      }
    }

  bottom:
    for (; bottom > minBottom; bottom--) {
      for (int x = width - 1; x >= left; x--) {
        if (alpha(image.get(x, bottom)) != 0) {
          minRight = x;
          break bottom;
        }
      }
    }

  right:
    for (; right > minRight; right--) {
      for (int y = bottom; y >= top; y--) {
        if (alpha(image.get(right, y)) != 0) {
          break right;
        }
      }
    }
    cornerPos=new PVector(left, top); 
    img= image.get(left, top, right - left + 1, bottom - top + 1);
  }
}

Adrien · August 31, 2018, 4:30pm

The thing I don’t understand is that it looks like his installation can “remember” and nearly infinite nbr of frame, that’s why I’m really wondering if my multiple PImage buffer is the right approach…

jb4x · August 31, 2018, 5:20pm

Are you sure he is doing it live?

Maybe he first recorded the movements and then used another sketch to post process it?

I’ll try to think about other way to do it.

Adrien · August 31, 2018, 5:59pm

Yeah it’s in real time.
but FYI, I think I find a nice approach : copy every new masked image on a PGraphics, but everytime you tint it with a different shade of gray (like a “tag” or a metadata actually). After you can check every pixels of the PGraphics, and depend of is shade of gray, look in a lookup table (it can be dynamic if you want color to move) for a new colors…
A.

jeremydouglass · September 1, 2018, 1:38am

While catching up on the old server I posted some discussion of this here – looks like it is the same as the direction this forum thread was going:

In summary: don’t use a stack of 50 images as the buffer. Instead, accumulate silhouettes on one buffer image – then display by transforming with a recoloring map. To create animated cycling effects, rotate entries in the color map.

This approach has far better performance.

For a recent discussion of mapping linear grayscale pixel brightness values to a color map, see lerpColor, and a full color-mapping example this old post:

kfrajer · September 1, 2018, 2:33am

Welcome back @jeremydouglass

Kf

jb4x · September 1, 2018, 10:02am

Hi Adrien,

I came back with a new idea but it is quite similar to @jeremydouglass idea. Let me explain it anyway.

The principle is to store a 1D array with a dimension of width*height. Each value of that array can then refer to the corresponding pixel of the pixels array.

Then when you load a new image you go through it and every time you find a white pixel, you update your 1D array with an integer. After each new image received, you increment that integer by one.

Finally, when you want to draw the colors, you can simply loop through that array and depending on the value your read, you know to which image it corresponds.

Hope it is clear enough, otherwise I can try to explain it a bit better.

Topic		Replies	Views
Video delay effect, adjustable delay time Libraries	4	185	October 29, 2023
How to create the code of the following video effect? Coding Questions homework	1	677	November 30, 2018
Kinect and processing 3 thesis project Libraries	3	1201	April 29, 2019
Kinect processing Libraries	1	150	May 26, 2024
Graphs and framerate? Coding Questions	1	221	April 5, 2022

Framerate in live video delay project : Ed Tannenbaum, Recollections

Related topics