Resizing Kinect depth image for use as 'reveal mask'

Processing Libraries
1

Hello all
I’ve been programming interactive apps using a Kinect v2 model 1520 using Processing 3.5.3. This app will be for a top-down projection that uses a persons blob to reveal an image below another. In effect, the blob will temporarily mask out the Top image to reveal the Base image. The current app uses a Kinect depth image as a PImage, which is used as an image mask. The below code works fine for my purpose but has one issue:

The Kinect depth image is 512x424, and to use this as a mask, the Top and Base images also need to be 512x424, or Processing gives an error. This is too low a resolution to be projected at 3m square.

Is there any way to resize the Kinect depth map to be 1280x720 or 1920x1080, so that the Top and Base images can be this resolution?

I have tried to resize the depth PImage using PImage.resize with no luck. I’ve also tried to put the depth data into an int array and use this as a mask but no luck with this either. The blob hasn’t got to be amazing resolution, as its being blurred anyway. But the Top and Base images need to be high resolution.

I would really appreciate some help in making the overall sketch run at this higher resolution. Thanks!

Heres the working code:

import org.openkinect.freenect.*;
import org.openkinect.freenect2.*;
import org.openkinect.processing.*;
import org.openkinect.tests.*;

Kinect2 kinect2;

PImage base;
PImage top;
PImage depthImg;

int minDepth =  100;
int maxDepth =  1500;

void setup() {
  fullScreen(P2D);
  kinect2 = new Kinect2(this);
  kinect2.initDepth();
  kinect2.initDevice();
  
  base = loadImage("base.jpg"); 
  top = loadImage("top.jpg");
  
  depthImg = new PImage(kinect2.depthWidth, kinect2.depthHeight);
}

void draw() {
  image(base, 0, 0, width, height);
  
  int[] rawDepth = kinect2.getRawDepth();
  for (int i=0; i < rawDepth.length; i++) {
    if (rawDepth[i] >= minDepth && rawDepth[i] <= maxDepth) {
      depthImg.pixels[i] = color(255);
    } else {
      depthImg.pixels[i] = color(0);
    }
  }

  depthImg.updatePixels();
  depthImg.filter(BLUR, 5);
  
  image(top, 0, 0, width, height);
  top.mask(depthImg);
}
1 Like
2

This can work just fine, but could also be slow if you are doing it every frame.

An alternative is to loop over the pixels of a full-size depthImg and set each pixel value based on the nearest corresponding rawDepth pixel value. Here is an example with map() to make it clear. No Kinect on hand, so the brightness of a 512x424 Processing logo is used instead.

// https://discourse.processing.org/t/resizing-kinect-depth-image-for-use-as-reveal-mask/14764

PGraphics dimg;
PImage kinect2;
int minDepth = 32;
int maxDepth = 192;

void setup() {
  size(1024, 720);
  // fake camera
  kinect2 = loadImage("https://upload.wikimedia.org/wikipedia/commons/thumb/2/2e/Processing_3_logo.png/480px-Processing_3_logo.png");
  kinect2.resize(512, 424);
  dimg = createGraphics(1024, 720, JAVA2D);
}

void draw() {

  int[] rawd = getRawDepth(kinect2); // fake camera load, actually: = kinect2.getrawd();

  dimg.beginDraw();
  dimg.loadPixels();
  for (int y=0; y < dimg.height; y++) {
    for (int x=0; x < dimg.width; x++) {
      int ty = int(map(y, 0, dimg.height, 0, kinect2.height));
      int tx = int(map(x, 0, dimg.width, 0, kinect2.width));
      float dval = rawd[ty*kinect2.width + tx];
      if (dval >= minDepth && dval <= maxDepth) {
        dimg.pixels[y*dimg.width + x] = color(255);
      } else {
        dimg.pixels[y*dimg.width + x] = color(0);
      }
    }
  }
  dimg.updatePixels();
  dimg.endDraw();
  
  image(dimg, 0, 0);
  
  // saveFrame("KinectScaling.png");
  // exit();
}

// fake kinect pixels load
int[] getRawDepth(PImage img) {
  int[] depth = new int[img.pixels.length];
  for (int i=0; i<img.pixels.length; i++) {
    depth[i] = int(brightness(img.pixels[i]));
  }
  return depth;
}

KinectScaling
KinectScaling1024×720 6.75 KB

The input is a 512x424 brightness array; the output is a 1024x720 image suitable for masking an image of the same size – although it can be any size.

The key idea here is that the target doesn’t match the source – you just loop over the target, assigning the value from the equivalent source pixel. For improved performance you can rewrite map() statements as multiplication with pre-computed ratios.

1 Like
3

Thank you very much for your help! This has greatly helped me with this job and allowed me to learn more about Processing. Cheers!