Framerate in live video delay project : Ed Tannenbaum, Recollections

Hi,

I try to create something like this : https://www.youtube.com/watch?v=YBKkVpRxUfs, using a kinect. At the moment I made some tests with something like this :

class User {

  PGraphics pgTotal; 
  PGraphics pgBodyBg; 
  PImage bodyImg; 
  DelayImg[] buffer;

  User() {
    pgTotal=createGraphics(512, 424);  //size of bodyImg
    buffer=new DelayImg[50];
  }

  void delayEffect() {
    fillBuffer(); 
    changeColor(); 
    display();
  }

  void fillBuffer() {
    PImage bi= bodyImg.copy(); 
    DelayImg di=new DelayImg(bi); 

    for (int i = buffer.length-1; i > 0; i--) { //Shift all the image in the buffer
      buffer[i] = buffer[i-1];
    }
    buffer[0]=di; //buffer[0] is everytime the more recent frame 
  }

  void changeColor() {

    pgTotal.beginDraw(); 
    pgTotal.clear(); 
    pgTotal.pushStyle(); 

    for (int j=buffer.length-1; j>=0; j--) { 
      if (buffer[j]!=null) buffer[j].changeColor(pgTotal);
    }
    pgTotal.popStyle();
    pgTotal.endDraw();
  }

  void display() {
    image(pgTotal, 0, 0);
  }
}



class DelayImg {

  PImage img; 
  PGraphics pgBg; 

  DelayImg(PImage _img) {
    img=_img; 
    img.filter(THRESHOLD); 
    pgBg=createGraphics(512, 424);
  }

  void changeColor(PGraphics pg) {

    pgBg.beginDraw(); 
    pgBg.background(random(255), random(255), random(255));  
    pgBg.mask(img); //Use this to be able to superposePGraphics with the shape of the body in color and transparency
    pgBg.endDraw();
    pg.image(pgBg, 0, 0);
  }
}

bodyImg is update each frame, coming from kinect, it’s a bi-color image. I filter it with THRESHOLD to be able to use is as a mask to change backgrounds color. I’m not using tint because I want to be able to have more complicate pattern in the futur.

I want to be able to change the color everytime, that why pgTotal is clear every frame and display fresh new images, but the way I do it is very slow (10 fps). Do you have any suggestion, different kind of algorithm, another approach to improve frameRate? I try to use some vertex and textures aswell, but no real improvement…

Thanks for your help!

A

Perhaps the best approch will be to create one PImage (or PGraphics) with different “zones” like here :


and fill the different zones with different colors at each frame?
Do you have any idea how to do it? Or another approach?
Thanks!
A.

Hey,

I would advise you to time some part of your code to frame which part of your code is slowing everything down.

Also, are you using the P2D renderer? It might make it works a bit faster :thinking:

Why do you need the pushStyle() and popStyle()? I thought it was used just for fill() stroke() noStroke() etc.

What you can do also is simply store all your black and white image in an array (like you do). Then your create a PImage with a random color for each one of them. You then use the mask() function to get the background color applied only where the person is on your alpha mask. And finally you combine them all at the end.

Hope it helps at least a bit!

Hi jb4x,

Thanks for taking time to answer me.

-The part of this code slowing everything down is the changeColor() method of the DelayImg class here

for (int j=buffer.length-1; j>=0; j--) { 
      if (buffer[j]!=null) buffer[j].changeColor(pgTotal);
    }

-I’m already using P2D render
-You’re right, here I don’t need pushStyle and popStyle. It was here because I used to reduce alpha on old “layer”.
I don’t really understand the difference with what you sugest me :

What you can do also is simply store all your black and white image in an array (like you do). Then your create a PImage with a random color for each one of them. You then use the mask() function to get the background color applied only where the person is on your alpha mask. And finally you combine them all at the end.

and what I’m currently doing…Is it just replace the PGraphics I’m using with a PImage? (I’m using a PGraphics to be able to use the background method)

What I don’t understand is how Ed is managing and nearly unlimited nbr of frames/images. Do you thing is working with something like blobs, of time based paths…

Thanks!

A.

Perhaps he’s just working with the edge of the body shape


Something like blob or PShape, so another question could be :
Is there a way to automaticaly create a PShape from edge of an Image? :wink:
A.

There is none, I thought you were doing differently for some reasons :crazy_face:

What you can try though is to not apply the mask at each frame but instead mask a fully white PGraphic when you create your object straight after the filter() call in DelayImg(). Then you would just have to tint it and display it and nothing more.

Thanks, but with your solution I wont be able to tint the different area with different color isn’t it?
A.

You would. I’ll try to be a bit more clear in my answer.

Every time you get a new pic from the Kinect:

  1. You first apply a threshold
  2. You create a complete white picture
  3. You mask it with the kinect pic to get a white a transparent picture

The idea is to build an array of white and transparent picture representing all the position the person visited.

Then when you want to draw it you go through that array and for each image:

  1. You tint it
  2. You print it on top the the previous one (It should go from the oldest pic the the newest one then)
1 Like

Thanks for your answer !

I just implement it :

PGraphics[] pgBuffer; 
PImage bodyImg; 
  User() {
    pgBuffer=new PGraphics[100];
  }


  void delayEffect() {
    createWhiteAndTransparentImage();
    colorAllPg(); 
  }

  void createWhiteAndTransparentImage() {
    bodyImg.filter(THRESHOLD); 
    PGraphics pgWhite=createGraphics(512, 424); 
    pgWhite.beginDraw(); 
    pgWhite.background(255); 
    pgWhite.mask(bodyImg); 
    for (int i = pgBuffer.length-1; i > 0; i--) {
      pgBuffer[i] = pgBuffer[i-1];
    }
    pgBuffer[0]=pgWhite;
  }

  void colorAllPg() {
    for (int i = pgBuffer.length-1; i > 0; i--) {
      if (pgBuffer[i]!=null) {
        PImage colorImg=(PImage)pgBuffer[i];
        tint(random(255), random(255), random(255)); 
        image(colorImg, 0, 0, width, height);
      }
    }
  }

And it’s definitly an improvement : I just go from 60fps to 30fps. I still “loose” 30fps, but it’s definitly better than before when I dropped at 10 fps…
I will continue to investigate, but thanks a lot already! :smile:
A.

2 Likes
for (int i = pgBuffer.length-1; i > 0; i--) { 
  pgBuffer[i] = pgBuffer[i-1]; 
}

I also think that this loop is not really efficient. Instead of using a basic array, maybe use an ArrayList were the only thing you would have to do each frame is add the new image and get rid of the oldest one.

Juste made it :

class User {
  PImage bodyImg; 
  ArrayList<PGraphics> pgList; 
  User() {
    pgList = new ArrayList<PGraphics>();
  }

  void delayEffect() {
    createWhiteAndTransparentImage();
    colorAllPg();
  }

  void createWhiteAndTransparentImage() {
    bodyImg.filter(THRESHOLD); 
    PGraphics pgWhite=createGraphics(512, 424); 
    pgWhite.beginDraw(); 
    pgWhite.background(255); 
    pgWhite.mask(bodyImg); 
    pgWhite.endDraw();
    pgList.add(0, pgWhite);
    if (pgList.size()>100)pgList.remove(pgList.size()-1);
  }

  void colorAllPg() {

    for (int i = pgList.size()-1; i > 0; i--) {

      PImage colorImg=pgList.get(i);
      tint(random(255), random(255), random(255)); 
      image(colorImg, 0, 0, width, height); //This is the line that make me loose 30 fps : if I comment it, sketch running at 60fps. I tried to "print" all on a PGraphics and just display the PGraphics, but still the same fps lose...
    }
  }
}

Still around 30 fps…But thanks to take time to help me! :wink:
A.

I can see 2 more ways to increase framerate.

1. resize your PImage

When you create your white and transparent image, you can crop the image to get rid of all the transparent pixels on the side of the image. You would end up with a smaller image that you would draw in a specific (x, y) position to end up at the exact same spot as with the full image.

I should have an impact on the speed of the tint() function I think

2. Crop each image based on the one on top

Every time you get a new image, it will be set on top of the previous one. So what you can do is to delete the part of the previous image that is hidden by the new one. This way you would mask it a bit more and have even less pixels to deal with int the end, if combined with the first point.

Thanks I’ll try this!
A.

Try to make a first implementation of your first point ( crop PImage), but it actually make the frameRate drop dramatically (less than 10 fps).
Perhaps the idea is good but not my implementation…:wink:

class User {
  PImage bodyImg; 
  ArrayList<NewImg> imgList; 
  PGraphics pgTotal; 
  User() {
    imgList = new ArrayList<NewImg>();
    pgTotal=createGraphics(512, 424);
  }

  void delayEffect() {
    createWhiteAndTransparentImage();
    colorAllPg();
  }

  void createWhiteAndTransparentImage() {
    bodyImg.filter(THRESHOLD); 
    PGraphics pgWhite=createGraphics(512, 424); 
    pgWhite.beginDraw(); 
    pgWhite.background(255); 
    pgWhite.mask(bodyImg); 
    pgWhite.endDraw();
    NewImg newImg=new NewImg();
    newImg.trimImage(pgWhite); 
    imgList.add(0, newImg);
    if (imgList.size()>100)imgList.remove(imgList.size()-1);
  }

  void colorAllPg() {
    pgTotal.beginDraw(); 
    pgTotal.clear(); 
    for (int i = imgList.size()-1; i > 0; i--) {
      PImage colorImg=imgList.get(i).img;
      pgTotal.tint(random(255), random(255), random(255)); 
      float x=imgList.get(i).cornerPos.x; 
      float y=imgList.get(i).cornerPos.y; 
      pgTotal.image(colorImg, x, y);
    }
    pgTotal.endDraw(); 
    image(pgTotal, 0, 0, width, height);
  }
}

class NewImg {
  PVector cornerPos; 
  PImage img; 
  NewImg () {
  }


  void trimImage(PImage image) {
    image.loadPixels(); 
    int width = image.width;
    int height = image.height;
    int left = 0;
    int top = 0;
    int right = width - 1;
    int bottom = height - 1;
    int minRight = width - 1;
    int minBottom = height - 1;

  top:
    for (; top < bottom; top++) {
      for (int x = 0; x < width; x++) {
        if (alpha(image.get(x, top)) != 0) {
          minRight = x;
          minBottom = top;
          break top;
        }
      }
    }

  left:
    for (; left < minRight; left++) {
      for (int y = height - 1; y > top; y--) {
        if (alpha(image.get(left, y)) != 0) {
          minBottom = y;
          break left;
        }
      }
    }

  bottom:
    for (; bottom > minBottom; bottom--) {
      for (int x = width - 1; x >= left; x--) {
        if (alpha(image.get(x, bottom)) != 0) {
          minRight = x;
          break bottom;
        }
      }
    }

  right:
    for (; right > minRight; right--) {
      for (int y = bottom; y >= top; y--) {
        if (alpha(image.get(right, y)) != 0) {
          break right;
        }
      }
    }
    cornerPos=new PVector(left, top); 
    img= image.get(left, top, right - left + 1, bottom - top + 1);
  }
}

The thing I don’t understand is that it looks like his installation can “remember” and nearly infinite nbr of frame, that’s why I’m really wondering if my multiple PImage buffer is the right approach…

15

Are you sure he is doing it live?

Maybe he first recorded the movements and then used another sketch to post process it?

I’ll try to think about other way to do it.

Yeah it’s in real time.
but FYI, I think I find a nice approach : copy every new masked image on a PGraphics, but everytime you tint it with a different shade of gray (like a “tag” or a metadata actually). After you can check every pixels of the PGraphics, and depend of is shade of gray, look in a lookup table (it can be dynamic if you want color to move) for a new colors…
A.

While catching up on the old server I posted some discussion of this here – looks like it is the same as the direction this forum thread was going:

In summary: don’t use a stack of 50 images as the buffer. Instead, accumulate silhouettes on one buffer image – then display by transforming with a recoloring map. To create animated cycling effects, rotate entries in the color map.

This approach has far better performance.

For a recent discussion of mapping linear grayscale pixel brightness values to a color map, see lerpColor, and a full color-mapping example this old post:

2 Likes

Welcome back @jeremydouglass

Kf

Hi Adrien,

I came back with a new idea but it is quite similar to @jeremydouglass idea. Let me explain it anyway.

The principle is to store a 1D array with a dimension of width*height. Each value of that array can then refer to the corresponding pixel of the pixels array.

Then when you load a new image you go through it and every time you find a white pixel, you update your 1D array with an integer. After each new image received, you increment that integer by one.

Finally, when you want to draw the colors, you can simply loop through that array and depending on the value your read, you know to which image it corresponds.

Hope it is clear enough, otherwise I can try to explain it a bit better.