Framerate in live video delay project : Ed Tannenbaum, Recollections

I can see 2 more ways to increase framerate.

1. resize your PImage

When you create your white and transparent image, you can crop the image to get rid of all the transparent pixels on the side of the image. You would end up with a smaller image that you would draw in a specific (x, y) position to end up at the exact same spot as with the full image.

I should have an impact on the speed of the tint() function I think

2. Crop each image based on the one on top

Every time you get a new image, it will be set on top of the previous one. So what you can do is to delete the part of the previous image that is hidden by the new one. This way you would mask it a bit more and have even less pixels to deal with int the end, if combined with the first point.

Thanks I’ll try this!
A.

Try to make a first implementation of your first point ( crop PImage), but it actually make the frameRate drop dramatically (less than 10 fps).
Perhaps the idea is good but not my implementation…:wink:

class User {
  PImage bodyImg; 
  ArrayList<NewImg> imgList; 
  PGraphics pgTotal; 
  User() {
    imgList = new ArrayList<NewImg>();
    pgTotal=createGraphics(512, 424);
  }

  void delayEffect() {
    createWhiteAndTransparentImage();
    colorAllPg();
  }

  void createWhiteAndTransparentImage() {
    bodyImg.filter(THRESHOLD); 
    PGraphics pgWhite=createGraphics(512, 424); 
    pgWhite.beginDraw(); 
    pgWhite.background(255); 
    pgWhite.mask(bodyImg); 
    pgWhite.endDraw();
    NewImg newImg=new NewImg();
    newImg.trimImage(pgWhite); 
    imgList.add(0, newImg);
    if (imgList.size()>100)imgList.remove(imgList.size()-1);
  }

  void colorAllPg() {
    pgTotal.beginDraw(); 
    pgTotal.clear(); 
    for (int i = imgList.size()-1; i > 0; i--) {
      PImage colorImg=imgList.get(i).img;
      pgTotal.tint(random(255), random(255), random(255)); 
      float x=imgList.get(i).cornerPos.x; 
      float y=imgList.get(i).cornerPos.y; 
      pgTotal.image(colorImg, x, y);
    }
    pgTotal.endDraw(); 
    image(pgTotal, 0, 0, width, height);
  }
}

class NewImg {
  PVector cornerPos; 
  PImage img; 
  NewImg () {
  }


  void trimImage(PImage image) {
    image.loadPixels(); 
    int width = image.width;
    int height = image.height;
    int left = 0;
    int top = 0;
    int right = width - 1;
    int bottom = height - 1;
    int minRight = width - 1;
    int minBottom = height - 1;

  top:
    for (; top < bottom; top++) {
      for (int x = 0; x < width; x++) {
        if (alpha(image.get(x, top)) != 0) {
          minRight = x;
          minBottom = top;
          break top;
        }
      }
    }

  left:
    for (; left < minRight; left++) {
      for (int y = height - 1; y > top; y--) {
        if (alpha(image.get(left, y)) != 0) {
          minBottom = y;
          break left;
        }
      }
    }

  bottom:
    for (; bottom > minBottom; bottom--) {
      for (int x = width - 1; x >= left; x--) {
        if (alpha(image.get(x, bottom)) != 0) {
          minRight = x;
          break bottom;
        }
      }
    }

  right:
    for (; right > minRight; right--) {
      for (int y = bottom; y >= top; y--) {
        if (alpha(image.get(right, y)) != 0) {
          break right;
        }
      }
    }
    cornerPos=new PVector(left, top); 
    img= image.get(left, top, right - left + 1, bottom - top + 1);
  }
}

The thing I don’t understand is that it looks like his installation can “remember” and nearly infinite nbr of frame, that’s why I’m really wondering if my multiple PImage buffer is the right approach…

15

Are you sure he is doing it live?

Maybe he first recorded the movements and then used another sketch to post process it?

I’ll try to think about other way to do it.

Yeah it’s in real time.
but FYI, I think I find a nice approach : copy every new masked image on a PGraphics, but everytime you tint it with a different shade of gray (like a “tag” or a metadata actually). After you can check every pixels of the PGraphics, and depend of is shade of gray, look in a lookup table (it can be dynamic if you want color to move) for a new colors…
A.

While catching up on the old server I posted some discussion of this here – looks like it is the same as the direction this forum thread was going:

In summary: don’t use a stack of 50 images as the buffer. Instead, accumulate silhouettes on one buffer image – then display by transforming with a recoloring map. To create animated cycling effects, rotate entries in the color map.

This approach has far better performance.

For a recent discussion of mapping linear grayscale pixel brightness values to a color map, see lerpColor, and a full color-mapping example this old post:

2 Likes

Welcome back @jeremydouglass

Kf

Hi Adrien,

I came back with a new idea but it is quite similar to @jeremydouglass idea. Let me explain it anyway.

The principle is to store a 1D array with a dimension of width*height. Each value of that array can then refer to the corresponding pixel of the pixels array.

Then when you load a new image you go through it and every time you find a white pixel, you update your 1D array with an integer. After each new image received, you increment that integer by one.

Finally, when you want to draw the colors, you can simply loop through that array and depending on the value your read, you know to which image it corresponds.

Hope it is clear enough, otherwise I can try to explain it a bit better.

Thanks @jeremydouglass and @jb4x!
jb4x I still didn’t implement your idea, but I definitly will give it a try!
jeremydouglass your idea is working well, but I don’t know why this part make my sketch loose more than 30fps :

void colorGlobalPg() {
    pgBW.loadPixels(); 
    //println("new"); 
    for (int i = 0; i < pgBW.width; i++) {
      for (int j = 0; j < pgBW.height; j++) {
        color c = pgBW.pixels[j*pgBW.width+i];
        float b = brightness(c); 
        if (b!=0) {
          //pgColorTotal.pixels[j*pgBW.width+i]=color(255); // This run at 60 fps
          // pgColorTotal.pixels[j*pgBW.width+i]=lerpColors(constrain(b/255, 0, 1), colors); //when I use your custom lerpColors function from an other post fps go down at 20 fps
          pgColorTotal.pixels[j*pgBW.width+i]= colors[int(b%colors.length)]; //Same (fps at around 20 fps) when I grab color directly in a global color array
        }
      }
    }
  }

A.

Perhaps I should open a new topic, but I have a strange fps drop in some case :

  void colorAllPg() {
    pgBW.loadPixels(); 
    for (int y = 0; y < pgBW.height; y++) {
      for (int x = 0; x < pgBW.width; x++) {
        color c = pgBW.pixels[x+y*pgBW.width];
        float b = brightness(c); 
        if (b!=0) {
          int colors2[] = { #D8075B, #0AEDFF, #FAEE0A, #BCBDAC, #CFBE27, #F27435, #F02475, #3B2D38};
          //color cc=colors2[int(b%colors.length)];  //drop until 20fps
          // color cc=colors2[int(random(colors2.length))]; //drop until 30fps
          color cc=colors2[0]; //stay around 60fps
          pgColorTotal.pixels[x+y*pgBW.width]=cc;
        }
      }
    }
  }

Do you have any idea why there are such big difference just by accessing a local array? It’s looks like it’s even worst if it’s global. Any suggestion to fix it?

Thanks very much,

A.

Just my optimization attempt. Not tested! No idea if it’s any faster than yours… :woozy_face:

static final color[] PALETTE = { 
  #D8075B, #0AEDFF, #FAEE0A, #BCBDAC, 
  #CFBE27, #F27435, #F02475, #3B2D38
};

void colorAll(final PImage src, final PImage dst, final color[] palette) {
  src.loadPixels();
  
  final color[] p1 = src.pixels, p2 = dst.pixels;
  final int colors = palette.length;
  int idx = 0;
  
  for (final color c : p1) {
    final color b = (color) brightness(c);
    if (b > 0)  p2[idx] = palette[b % colors];
    ++idx;
  }
  
  dst.updatePixels();
}

Can it be the modulus operator or the random() function in addition with the int() function that causes the FPS drop and not accessing the array? :thinking:

You are calling them quite a lot in one frame.

Thanks @GoToLoop! it’s definitly an improvement (around 40fps), I have to investigate a bit more this static, final and all the structure of your algorythm :wink:

@jb4x I try to make a local variable with something like this :

 int colCounter=int(b%colors.length); 
 pgColorTotal.pixels[x+y*pgBW.width]= colors[colCounter];

But, no suprisly, it doesn’t change a lot…do you have something else in mind?
I guess I will keep @GoToLoop version, I’m around 50fps, that’s actually quite great :wink:

A.

You are still computing the int() operation and the modulus one for every pixel of the image.

Again, I really don’t know if this can be it, I’m not qualified enough to be sure about the performance of those functions. Someone more capable can surely tell us if it can be that.

Edit:

You can try something like this.
You create an array with 256 values let’s call it brightToCol.
Since you have 8 different colors you would fill it in like this:

brightToCol[0] = 0;
brightToCol[1] = 1;
...
brightToCol[6] = 6;
brightToCol[7] = 0;
brightToCol[8] = 1;
...

Then you can get your color with this line:

color cc=colors2[brightToCol[b]];

Is this better? Specially combined with @GoToLoop optimization.

Hi, thanks jb4x, I’ll perhaps need to try this, but actually my project went in a different direction, I’ll open a new topic :wink: But thanks for all your help with this!
A.

Hi All,

I just stumbled onto this thread. I’m the “Ed Tannenbaum” that designed the installations that you are trying to emulate with Processing. I’m not sure you are still attempting this, but I’ll give you a few hints…

I designed the first one in 1981. I built a 4 bit framebuffer with some digitizing logic that thresholded a video image and put it into the framebuffer only where it saw the silhouette. It was assigned a number based on time from 1-15. Zero was the background. The output of the framebuffer went to a hardware color lookup table where the numbers were assigned a color. The lookup table changed each frame during the vertical interval. It was controlled by an Apple II computer that was programmed in Forth.

Note that I used a single framebuffer. Memory was really expensive back then. With 4 bits I could display 16 colors at a time. It was the “colormap rotation” that created the animation.

Since then I’ve used the same method to create new versions in software. I’m not quite ready to release the code yet. I wouldn’t attempt it in Processing, or even openFrameworks as (from what I’ve been able to gleen from the docs) the rudimentrary interface to the necessary part of the graphics display isn’t there. It is built into the graphics cards, but isn’t available through OpenGL it seems. Perhaps I’m wrong. Please let me know.

I’d love to be able to write a shader that would create the effect.

ET

3 Likes

Hi Ed!

It’s a surprise, a pleasure and a honor to be able to speak to you! :wink:
First of all, my congratulations for all the different Recollections, they are truly inspiring (and inspired), innovative projects, thank you very much for taking the time to share these various advices.
I have actually achieved quite good results with the help anc the techniques proposed by @jb4x, @GoToLoop and @jeremydouglass (thanks again to them! ) but I finally didn’t continue in that direction : I still haven’t taken the time to upload the video of the final project, but here are some screenshots :

It still uses a Kinect, but this time users are immersed in a fluctuating world, with which they can interact and merge…Quite mystic :wink:

It’s an audiovisual work, and for the visuals I actually also wrote shaders, inspired by some I found on shadertoy.

I hope that one day I could experiment with your works in live
Thanks again for your work and advices,

All the best,

Adrien

4 Likes

Thank you so much for sharing this description, Ed – and for the inspirational art.

I’m not sure–setting shaders aside for the moment, by “rudimentary interface”, do you mean the pixels[] array? It is an int[] array of colors that you can write directly to with bit shifting, for example.