How shall I scale my Kinect interaction full screen?

Processing Libraries
1

Hello,
I have created an interactive “graphic” with Open Kinect and Processing where the Kinect detects your hand movements to reveal the “moon surface image” which is below the top image called “moon dirt”. Basically you have to move your hands to “clean the moon”.

I have read a bit that the Kinect depth sensor is only able to be at a fixed resolution, however I need it to be fullscreen. I am able to get the 2 images moonSurface and moonDirt to be full screen, however the interactive bit, where you “wipe off” the moonDirt image does not scale full screen.

Is there any quick method to force kinect to be full screen? Does this involve somehow upscalling everything, instead of using fullScreen();

It needs to be full screen because, the next step would be that it will be displayed by a projector onto a table, where the user would be doing a “wiping” action to clean off the orange moonDirt to reveal moonSurface.

Capture
Capture.PNG718×574 26.9 KB

As you can see, on the image above, the moon surface is revealed when kinect detects movement
Right now it is fixed at this resolution

import org.openkinect.freenect.*;
import org.openkinect.processing.*;
import processing.video.*;


Kinect kinect;

// Depth image
PImage depthImg;
PImage maskingImg;

PImage moonSurface;
PImage moonDirt;

// Which pixels do we care about?
int minDepth = 930;
int maxDepth = 940;

// What is the kinect's angle
float angle;

void setup() {
  size(640, 480);
  //fullScreen();
  kinect = new Kinect(this);
  kinect.initDepth();
  angle = kinect.getTilt();

  // Blank image
  depthImg = new PImage(kinect.width, kinect.height);

  // Masking image
  maskingImg = new PImage(kinect.width, kinect.height);

  // Load the movie files

  moonSurface = loadImage("MoonSurface.png");
  moonDirt = loadImage("MoonDirt.png");



  println("SETUP done");
}

void draw() {

  // Draw moon surface
  image(moonSurface, 0, 0, 640, 480);

  // Draw the moon dirt
  image(moonDirt, 0, 0);



  // Threshold the depth image
  int[] rawDepth = kinect.getRawDepth();
  for (int i=0; i < rawDepth.length; i++) {
    if (rawDepth[i] >= minDepth && rawDepth[i] <= maxDepth) {
      depthImg.pixels[i] = color(255);
      maskingImg.pixels[i] = color(255);
    } else {
      depthImg.pixels[i] = color(0);
    }
  }

  // Draw the thresholded image
  //depthImg.updatePixels();
  //maskingImg.updatePixels();

  //moonDirt.resize(640, 480);  //(640, 480);
  moonDirt.loadPixels();
  for (int i=0; i < rawDepth.length; i++) {
    if ( maskingImg.pixels[i] == color(255) ) {
      moonDirt.pixels[i] = color( 0, 0, 0, 0 );
    }
  }

  moonDirt.updatePixels();

  image(moonDirt, 0, 0, 640, 480);

  //image(depthImg, kinect.width, 0);
  //image(maskingImg, kinect.width, 0);


  //fill(0);
  text("TILT: " + angle, 10, 20);
  text("THRESHOLD: [" + minDepth + ", " + maxDepth + "]", 10, 36);
}

// Adjust the angle and the depth threshold min and max
void keyPressed() {
  if (key == CODED) {
    if (keyCode == UP) {
      angle++;
    } else if (keyCode == DOWN) {
      angle--;
    }
    angle = constrain(angle, -10, 10);
    kinect.setTilt(angle);
  } else if (key == 'a') {
    minDepth = constrain(minDepth+10, 0, maxDepth);
  } else if (key == 's') {
    minDepth = constrain(minDepth-10, 0, maxDepth);
  } else if (key == 'z') {
    maxDepth = constrain(maxDepth+10, minDepth, 2047);
  } else if (key =='x') {
    maxDepth = constrain(maxDepth-10, minDepth, 2047);
  }
}
1 Like
2

Were you able to resolve this issue? I’m not sure I understand, but one method would be to make MoonSurface.png and MoonDirt.png the same size as the Kinect image – then upscale the final output with image().

The depth camera only has 640x480 pixels of data. If you want to map that across interacting with a much larger massive surface, you need to decide how you want to do that – for example, do you want it to be pixel-y, or blurred, or random static…? You could also summarize your depth image to things that could be described geometrically – like finding a centerpoint for a circle – and then draw on the mask with that geometry.

3

changing

// Draw moon surface
image(moonSurface, 0, 0, width, height);

// Draw the moon dirt
image(moonDirt, 0, 0, width, height);

to width, height seems to have done the trick
and of course change setup canvas to fullScreen();

it stretches the depth masking effect full screen somehow.

2 Likes