Help with pixels[] loadPixels(), etc

Processing Libraries
1

Hi all,

I’ve been building some code from scratch to compare the pixels of a reference image to the pixels of each new frame of video being captured by the webcam.

I was able to get it to work the way that I wanted (I still ned to tweak it a bit, but it’s coming along):

CODE A

void compareFrames_WORKS() {
  for (int h = 0; h < height; h++) { // a loop for the height of the image
    for (int w = 0; w < width; w++) { // a loop for the width of the image
      color A = reference.get(w, h); // Get color from reference image
      color B = get(w, h); // get color from current frame
      float diffR = abs(red(A) - red(B)); //abs() makes the number in the parameter positive
      float diffG = abs(green(A) - green(B)); 
      float diffB = abs(blue(A) - blue(B));
      if (diffR > thresh || diffG > thresh || diffB > thresh) {
        stroke(255); // fill white if above threshold
      } else {
        stroke(0); // fill black  if below threshold
      }
      point(w, h); // draw a new pixel
    }
  }
}

The problem is, it’s clearly not efficient because my computer’s fan kicks in within a few seconds of running the sketch. I looked around and found the FrameDifferencing example that comes with the Processing IDE.

The FrameDifferencing code does close to what I want, but it looks at the previous frame and compares it to the current frame, and I want to compare a reference frame (that is always the same) with the current frame.

My problem here is that I don’t fully understand how to use the pixel[] array properly, and I haven’t been able to figure it out with other forum posts and reference page examples. I’m confused about what’s being referenced, and when I need to use things like loadPixels(), and how dot syntax comes into play.

Using the FrameDifferencing file as reference, I have converted the CODE A to this:

CODE B

void compareFrames() {
  loadPixels();
  for (int i = 0; i < numPix; i++) { 
    color A = reference.pixels[1]; // Get color from reference image
    color B = videoCam.pixels[i]; // get color from current frame
    float diffR = abs(red(A) - red(B)); //abs() makes the number in the parameter positive
    float diffG = abs(green(A) - green(B)); 
    float diffB = abs(blue(A) - blue(B));
    if (diffR > thresh || diffG > thresh || diffB > thresh) {
      pixels[i] = color(255); // fill white if above threshold
    } else {
      pixels[i] = color(0); // fill black  if below threshold
    }
  }
  updatePixels();
}

But I’m missing something, because it’s not functioning as CODE A does.

Full (in progress) sketch below:



import processing.video.*;
Capture videoCam;


//-------------------------------//
//            CAPTURE            //
//-------------------------------//

void captureEvent(Capture videoCam) {
  videoCam.read();
}


//-----------------------------//
//            SETUP            //
//-----------------------------//

void setup() {  
  size(427, 240);
  videoCam = new Capture(this, 427, 240);
  videoCam.start();

  rectMode(CORNERS);
  stroke(255, 0, 0);
  noFill();
  instructionsSetup();
  numPix = width * height;
}


//----------------------------//
//            DRAW            //
//----------------------------//

void draw() {  
  if (state == instructions) {
    instructionsDraw();
  } else if (state == calibrate) {
    calibratePhase();
  } else if (state == run) {
    drawCamFrame();
    //background(255, 0, 0);
    compareFrames();
  }
}


//---------------------------------//
//            CALIBRATE            //
//---------------------------------//

PImage reference;
PImage now;

int instructions = 0;
int calibrate = 1;
int run = 2;

int state = instructions;
int calibrateStep = 0;
int maskStep = 0;

// first and second points for the rectancgle that outlines the "playing field" that the artist wants to work with.
// The program will ignore any visual information outside of this rectangle. 
float point0x;
float point0y;
float point1x;
float point1y;

PShape mask;


//-----------------------------------------//
//            INSTRUCTIONS PHASE           //
//-----------------------------------------//

// Instructions found in the draw loop.

void instructionsDraw() {
  textAlign(CENTER);
  textSize(16);
  text("See console for instructions.", width/2, height/2);
}

//-----------------------------------------//
//            INSTRUCTIONS PHASE           //
//-----------------------------------------//

// Instructions found in the draw loop.

void instructionsSetup() {
  println("----------------------");
  println("1) Click canvas to start.");
  println("2) Press 'C' to capture desired reference frame (e.g. the room with no one in it).");
  println("3) Indicate which portion of the image you would like to monitor for activity. Do so as follows:");
  println("   Click and drag to create a rectangle over the reference frame."); 
  println("   Make sure to drag from upper left to lower right, or the mask won't work.");
  println("   If satisfied, click 'S'. If you want to draw it again, click 'C'.");
  println("4) Once you click 'S' The program will automatically start monitoring the area specified.");
  println("----------------------");
}


//-----------------------------------------------//
//            CAPTURE REFERENCE FRAME            //
//-----------------------------------------------//

// Capture's reference image when C is pressed on the keyboard.

void saveTheFrame() {
  image(videoCam, 0, 0);
  if (keyPressed) {
    if (key == 'c' || key == 'C') {
      saveFrame("ref.jpg");
      reference = loadImage("ref.jpg");
      calibrateStep = 1;
    }
  }
}

//---------------------------------------//
//            REFERENCE AREA            //
//---------------------------------------//

// Creates a rectantle to outline the area of the video that you want to look for activity within.

void referenceArea() { 
  if (maskStep == 1) {
    image(reference, 0, 0);
    rect(point0x, point0y, mouseX, mouseY);
  }
  if (maskStep == 2) {
    image(reference, 0, 0);
    rect(point0x, point0y, point1x, point1y);
    println("Point 0: " + point0x +", " + point0y);
    println("Point 1: " + point1x +", " + point1y);
    maskStep = 3;
  }
  if (maskStep == 3) {
    if (keyPressed) {
      if (key == 'c' || key == 'C') {
        image(reference, 0, 0);
        maskStep = 0;
      }
    }
    if (key == 's' || key == 'S') {
      image(reference, 0, 0); // Draw the reference image.       
      generateMask();
      shape(mask); // Draw the mask on top of it.
      saveFrame("ref.jpg"); // Save the reference image with the mask superimposed.
      state = run;
    }
  }
}

//-----------------------------------//
//            RECT POINTS            //
//-----------------------------------//

// Saves the rectangle's coordinate points.

void rectPoints() { 
  if (maskStep == 1) {
    point1x = mouseX;
    point1y = mouseY;
    maskStep = 2;
  } else if (maskStep == 0) {
    point0x = mouseX;
    point0y = mouseY;
    maskStep = 1;
  }
}

//-------------------------------------//
//            MOUSE PRESSED            //
//-------------------------------------//

void mousePressed() {
  if (state == instructions) {
    state = calibrate;
  }
  if (calibrateStep == 1) {
    rectPoints(); // Saves the rectangle's coordinate points.
  }
}

//-------------------------------------------//
//            GENERATE BLACK MASK            //
//-------------------------------------------//

// Generate (but not yet draw (that's shape(mask)) a black mask to cover the unmonitoried area of the screen.

void generateMask() {
  // Make a shape
  mask = createShape();
  mask.beginShape();
  mask.noStroke();
  mask.fill(0);

  // Exterior part of shape
  mask.vertex(0, 0); // A
  mask.vertex(width, 0);  // B
  mask.vertex(width, height); // C
  mask.vertex(0, height);  // D
  mask.vertex(0, 0); // A

  // Interior part of shape
  mask.beginContour();
  mask.vertex(point0x, point1y); // D
  mask.vertex(point1x, point1y); // C
  mask.vertex(point1x, point0y); // B
  mask.vertex(point0x, point0y); // A
  mask.vertex(point0x, point1y); // D
  mask.endContour();

  // Finish off shape
  mask.endShape(CLOSE);
}

//---------------------------------------//
//            CALIBRATE PHASE            //
//---------------------------------------//

// Runs the Calibrate phase

void calibratePhase() {
  if (calibrateStep == 0) {
    saveTheFrame();
  } else if (calibrateStep == 1) {
    //perform calibrateStep 1 inside mousePressed() function (save the the mouseX and Y coordinates when mouse is clicked)
    referenceArea();
  }
}

//-----------------------------------//
//            MONITOR            //
//-----------------------------------//

int thresh = 40;
int numPix;
int[ ] pix = new int[ numPix ] ;


void drawCamFrame() {
  image(videoCam, 0, 0);
  shape(mask);
}


//---------------------------------------------------------------//
//            WANT TO WORK LIKE compareFrames_WORKS()            //
//---------------------------------------------------------------//

void compareFrames() {
  loadPixels();
  for (int i = 0; i < numPix; i++) { 
    color A = reference.pixels[1]; // Get color from reference image
    color B = videoCam.pixels[i]; // get color from current frame
    float diffR = abs(red(A) - red(B)); //abs() makes the number in the parameter positive
    float diffG = abs(green(A) - green(B)); 
    float diffB = abs(blue(A) - blue(B));
    if (diffR > thresh || diffG > thresh || diffB > thresh) {
      pixels[i] = color(255); // fill white if above threshold
    } else {
      pixels[i] = color(0); // fill black  if below threshold
    }
  }
  updatePixels();
}

//-----------------------------------------------------------------//
//            WORKS CLOSE TO THE WAY I WANT BUT IS SLOW            //
//-----------------------------------------------------------------//

void compareFrames_WORKS() {
  for (int h = 0; h < height; h++) { // a loop for the height of the image
    for (int w = 0; w < width; w++) { // a loop for the width of the image
      color A = reference.get(w, h); // Get color from reference image
      color B = get(w, h); // get color from current frame
      float diffR = abs(red(A) - red(B)); //abs() makes the number in the parameter positive
      float diffG = abs(green(A) - green(B)); 
      float diffB = abs(blue(A) - blue(B));
      if (diffR > thresh || diffG > thresh || diffB > thresh) {
        stroke(255); // fill white if above threshold
      } else {
        stroke(0); // fill black  if below threshold
      }
      point(w, h); // draw a new pixel
    }
  }
}

Thanks for your help!

2 Likes
2

The trick is to understand how to acces pixels in 2d and 1d.

Check out coding train vids they explain how to make use of pixel arrays very well. Also get may not be the fastest way of getting pixels.

And if the best performance or speed is a must shaders are the way to go.

3

This looks suspicious. Maybe i ?

1 Like
4

Thanks so much for directing me to this material! I’ll certainly check out the video, and also see if I can find out some information on “shaders” :slight_smile:

1 Like
5

Yes! That was it! Thanks so much Chrisir :slight_smile:

2 Likes
6

no big deal.

one of those facepalm moments… :wink:

1 Like
7

Hello @SheCurvesMobius,

This seems to match the working method:

color A = reference.pixels[i]; // Get color from reference image
color B = pixels[i]; // get color from current frame

I added these snippets to toggle between your two methods:

//global
boolean toggle = false;

 //mousePressed()
 if (mouseButton == CENTER) //mousePressed()
   {
   toggle = !toggle;
   }

 //draw()
  if (toggle)
    compareFrames_WORKS();
  else
    compareFrames();
  }

I also changed the white to yellow in one of the methods to see when I toggled.

:)