If you want to do image processing fast, shaders are usually the way to go. Fragment shaders are programs that operate on every pixel on the screen concurrently. This is very applicable for finding if pixels in an image are within a certain range. However, they are less applicable for finding the average position, but it is still possible to use shaders to speed up the process. I wrote a couple of shaders and modified your program to use them.
I’ll start with the shader to find pixels if they are within a threshold:
filename: colorDetect.glsl
#ifdef GL_ES
precision mediump float;
precision mediump int;
#endif
uniform sampler2D texture;
varying vec4 vertColor;
varying vec4 vertTexCoord;
uniform vec4 targetColor;
uniform float threshold; //between 0 and 1
void main() {
// get pixel color
vec4 texColor = texture2D(texture, vertTexCoord.st) * vertColor;
vec3 a = texColor.xyz;
vec3 b = targetColor.xyz;
// compute "distance" between colors rgb components
float dist = sqrt((b.x - a.x) * (b.x - a.x) + (b.y - a.y) * (b.y - a.y) + (b.z - a.z) * (b.z - a.z));
// colors are from 0-1, so the max distance between colors is sqrt(3)
if(dist < threshold * sqrt(3)){
// display inverse color where pixels are within the threshold
texColor = vec4(1) - texColor;
}
// force alpha to be 1 since inverting the color makes the alpha zero
gl_FragColor = vec4(texColor.xyz, 1);
}
While the syntax of shaders is different than Java, it is not difficult to learn. The key concepts to take away from this shader are vectors, the way colors are computed in shaders, and the difference between varying and uniform variables. In shaders, colors are stored with 4 component vectors with each component ranging from 0 to 1. The components are usually marked x, y, z, and w. You can get individual components by doing vector.x, or you could get more components at once and in any order by using something like vector.xyz or vector.zx. Vectors can have 2 to 4 components as a vec2, vec3, or vec4, with colors being a vec4.
Varying variables are different for every pixel the shader computes. Uniform variables are constants in the shader, but can be modified by the external program that calls the shader, in this case the processing sketch. A more comprehensive introduction to shaders can be found in this Processing Tutorial by Andres Colubri.
But, how would you use shaders to compute the average position of the pixels in the threshold? The way it was done in the original was by adding the screen positions of all detected pixels then divide by the number of detected pixels. While the sum and division parts of the calculation cannot be done in a shader, finding the positions of all detected pixels can. This is what this next shader does:
filename: colorPos.glsl
#ifdef GL_ES
precision mediump float;
precision mediump int;
#endif
uniform sampler2D texture;
varying vec4 vertColor;
varying vec4 vertTexCoord;
uniform vec4 targetColor;
uniform float threshold; //between 0 and 1
void main() {
vec4 texColor = texture2D(texture, vertTexCoord.st) * vertColor;
vec3 a = texColor.xyz;
vec3 b = targetColor.xyz;
float dist = sqrt((b.x - a.x) * (b.x - a.x) + (b.y - a.y) * (b.y - a.y) + (b.z - a.z) * (b.z - a.z));
bool cond = dist < threshold * sqrt(3);
// if color is within threshold, encode the pixel's position into red and green components
// and use blue component as a marker that the pixel was in range
// vertTexCoord is from 0 to 1, so after computing average, multiply by width and height to get screen position
gl_FragColor = cond ? vec4(vertTexCoord.x, vertTexCoord.y, 1, 1) : vec4(0, 0, 0, 1);
}
You may have noticed the varying variable vertTexCoord in these shaders. It represents the uv coordinates of the texture. Most simply, uv coordinates are a 2D vector that ranges from 0-1 in each component, where a uv of (0, 0) means you are at the top-left corner of the texture and (1, 1) the bottom right. To get the screen position, you multiply the uv by the screen width and height. For the purposes of these shaders, vertTexCoord is just the uv coordinate of the current pixel on the screen. Since shaders output color, I can output the uv coordinate by making it part of the output color. Since colors have 4 components, I can use one to say whether that pixel was within the threshold. Then, in the main program, I loop over every pixel from this other shader. For each pixel with a blue value > 0, use its red and green values to add to a total vector, then divide the number by the number of additions.
Processing sketch:
import processing.video.*;
PShader colorFinder, colorPosShader;
PGraphics overlay, posBuffer;
// Variable for capture device
Capture video;
// A variable for the color we are searching for.
color trackColor;
float threshold = 0.1;
void setup() {
//size(320, 240);
size(640, 480, P2D);
overlay = createGraphics(width, height, P2D);
posBuffer = createGraphics(width, height, P2D);
colorFinder = loadShader("colorDetect.glsl");
colorPosShader = loadShader("colorPos.glsl");
printArray(Capture.list());
video = new Capture(this, width, height);
video.start();
video.loadPixels();
// Start off tracking for red
trackColor = color(255, 0, 0);
}
void captureEvent(Capture video) {
// Read image from the camera
video.read();
}
void draw() {
colorFinder.set("threshold", threshold);
colorFinder.set("targetColor", red(trackColor) / 255.0, green(trackColor) / 255.0, blue(trackColor) / 255.0, 1.0);
colorPosShader.set("threshold", threshold);
colorPosShader.set("targetColor", red(trackColor) / 255.0, green(trackColor) / 255.0, blue(trackColor) / 255.0, 1.0);
overlay.beginDraw();
overlay.shader(colorFinder);
overlay.image(video, 0, 0);
overlay.endDraw();
posBuffer.beginDraw();
posBuffer.shader(colorPosShader);
posBuffer.image(video, 0, 0);
posBuffer.endDraw();
//compute average position by looking at pixels from position buffer
posBuffer.loadPixels();
PVector avg = new PVector(0, 0);
int count = 0;
for(int i = 0; i < posBuffer.pixels.length; i++){
// encoded so blue is > 0 if a pixel is within threshold
if(blue(posBuffer.pixels[i]) > 0){
count++;
// processing takes 0-1 (float) color values from shader to 0-255 (int) values for color
// to decode, we need to divide the color by 255 to get the original value
avg.add(red(posBuffer.pixels[i]) / 255.0, green(posBuffer.pixels[i]) / 255.0);
}
}
if(count > 0){
// we have the sum of positions, so divide by the number of additions
avg.div((float) count);
// convert 0-1 position to screen position
avg.x *= width;
avg.y *= height;
} else {
// appear offscreen
avg = new PVector(-100, -100);
}
image(overlay, 0, 0);
fill(trackColor);
stroke(0);
circle(avg.x, avg.y, 16);
fill(0, 50);
noStroke();
rect(0, 0, 150, 30);
fill(150);
text("Framerate: " + frameRate, 0, 11);
text("Threshold: " + threshold, 0, 22);
}
void mousePressed() {
// Save color where the mouse is clicked in trackColor variable
video.loadPixels();
int loc = mouseX + mouseY*video.width;
trackColor = video.pixels[loc];
}
void mouseWheel(MouseEvent e){
threshold -= e.getCount() * 0.01;
threshold = constrain(threshold, 0, 1);
}