Edit: SOLVED (more or less). I ended up doing my depth rendering and RGB video in a separate Processing sketch, and creating a Thread subclass to render the depth mask asynchronously. I’m using Spout to send the RGB image and depth mask to the pixelFlow sketch. I’m not positive if both of those steps are necessary, but it’s super stable and it feels more elegant. No more thread safety issues and depth mapping is no longer bottlenecking everything else. There is probably a better solution out there, but I’m getting decent video framerate (> 20fps) and 60fps particle simulation without any conflict.
Hello,
I am working on a program that combines Thomas Lengeling’s KinectV2 for Processing library with Thomas Diewald’s PixelFlow library. Basically I am using the kinect V2 IR camera to make a depth mask that is aligned with the color video image so that users can use their bodies to interact with particles in the PixelFlow world. (Inspired by Karl Sims’ Particle Mirror)
I am using depth thresholds and the included getMapDepthToColor() function to create an array of “pixels” with x and y values mapped to the color video image (the depth pixels move relative to the color image as a function of depth). With a depth threshold, I am rendering those values to a binary mask that I can use as an obstacle layer in PixelFlow.
Here is the relevant code:
public int[] sorted; //this is a little messy, but for the purposes of this example, it's just the length of the depth pixel array.
public int[] rawDepth;
public float[] mapDCT;
mapDCT = kinect.getMapDepthToColor(); // 434,176
rawDepth = kinect.getRawDepthData();
kinect.setLowThresholdPC(minD);
kinect.setHighThresholdPC(maxD);
dMask_.beginDraw();
dMask_.clear();
dMask_.stroke(0, 255);
dMask_.strokeWeight(1);
dMask_.rectMode(CENTER);
dMask_.fill(0, 255);
int resolution = 2;
for (int i = 0; i < sorted.length; i+= resolution) { //loop through depth "pixels"
int dctLookup = constrain(i*resolution, 0, mapDCT.length); //I'll be honest I just threw this in because I couldn't figure out an array out of bounds exception.
if (mapDCT[dctLookup] > 0 && mapDCT[dctLookup] < width &&
mapDCT[dctLookup+1] > 0 && mapDCT[dctLookup+1] < height &&
rawDepth[dctLookup/2] != 0) { //if depth value is greater than 0 (this has previously been limited by minD and maxD, so no further filtering is necessary)
float x = constrain(mapDCT[dctLookup]/bufferScale, 0, dMask_.width-1);
float y = constrain(mapDCT[dctLookup + 1]/bufferScale, 0, dMask_.height-1);
dMask_.point(x,y);
}
}
dMask_.endDraw();
My problem is that this is a bottleneck, and I can’t figure out why it’s so slow. I know that adding the constraints and if statements isn’t pretty, but the depth pixels move around so much, I don’t really see a way around it and I think the rendering is the bottleneck anyway. Updating the depth mask takes my framerate from mid-50s to like 15 fps. Is there a more efficient way to do this? It seems like the kind of thing a shader could handle almost instantaneously, but that still feels pretty far over my head.
I have tried running the depthmask update function in a separate thread, and that sort of works. The depth mask does not update every frame, but the video and particles advance at full speed, and it’s basically fine in terms of user experience. However, I ran into a lot of problems trying to render asynchronously in OpenGL. So right now I am running it asynchronously using the JAVA2D renderer, then copying the image into an OpenGL depth mask buffer every time the update is complete. But the depth mask image gets glitchy every few frames and occasionally the two threads conflict directly and the whole thing crashes. If I could eliminate the glitches/crashes, this method would work well enough for my purposes, even though it’s far from elegant.
Does anyone have any ideas? Thanks!
Console message on nonfatal glitch:
java.lang.NullPointerException
at sun.java2d.pipe.LoopPipe.fillSpans(LoopPipe.java:336)
at sun.java2d.pipe.LoopPipe.draw(LoopPipe.java:204)
at sun.java2d.SunGraphics2D.draw(SunGraphics2D.java:2497)
at processing.awt.PGraphicsJava2D.strokeShape(PGraphicsJava2D.java:1353)
at processing.awt.PGraphicsJava2D.line(PGraphicsJava2D.java:1201)
at processing.awt.PGraphicsJava2D.point(PGraphicsJava2D.java:1190)
at KinectWall_Core.dMaskUpdate(KinectWall_Core.java:463)
at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at processing.core.PApplet.method(PApplet.java:3860)
at processing.core.PApplet$2.run(PApplet.java:3898)
Console message on fatal crash (sorry this is an image. It’s a hard crash, so I can’t copy and past):
The “thread time elapsed…” parts are just the output of println functions and can be ignored.