Hello,
I’ve made ‘some’ progress, mostly in developing my understanding of what’s happening in the code.
So far, these are the changes:
I’ve modified @jeremydouglass’s code to
a) Add the webcam feed
b) I’ve changed the shape & colour of the objects so the relative changes are easy to visualize
I’ve tried to add detailed comments on the code explaining my understanding of what is happening at each section; Given my lack of coding experience, I often find most code baffling. I am hoping someone new like me reading this will walk away with a slightly better understanding of what the logic & flow is (assuming I am correct).
Needless to say, I am no where near completion, and I have a few questions I hope you guys can help me understand / resolve! 
// Modified from code by Jeremy Douglass @
// https://discourse.processing.org/t/request-for-guidance-making-3d-objects-appear-static-on-a-webcam-feed-using-arduino-an-imu/
//This sketch attempts to
// a) Create a virtual 3D space (I am thinking of it as an imaginary 3D sphere)
// b) Place 6 objects in the 3D sphere at fixed coordinates
// c) Overlay webcam feed on this
// d) Use mouse movement as proxy for IMU data
//
import processing.video.*; // Calls the processing video library
// Defines the function rad in ?? (degrees / radians),
//which is used to determine coordinates in the 3D sphere
//starting from lines 30 - 35 where the 3D objects will be "fixed"
float rad = (80);
Capture cam;
void setup() {
size(1280, 960, P3D); // size of the canvas; P3D invokes the 3D renderer,
//using openGL as opposed to the default Processing renderer, which is designed for 2D
fill(204);
// poll & list all webcams / cameras detected by processing as attached to the system
String[] cameras = Capture.list();
// code from l.no 27 through 38 checks if webcams / cameras are present & prints the list.
// Line 37 picks the camera resolution, etc. that is used in the code.
// Change this number based on your specific system, the resolution / FPS you want,
// etc from the "println" output
// Code from Processing's capture() refdocs https://processing.org/reference/libraries/video/Capture.html
if (cameras.length == 0) {
println("There are no cameras available for capture");
exit();
} else {
println("Available cameras:");
for (int i = 0; i <cameras.length; i++) {
println(cameras[i]);
}
cam = new Capture(this, cameras[135]);
cam.start();
}
}
void draw() { // draw loop
lights();
background(0);
// if webcam is detected, will read frames from cam.
if (cam.available() == true) {
cam.read();
}
set(0,0, cam);
// centers & displays webcam stream at on screen; note that this is starting from Processing 0,0;
// so if the video resolution is less than the size of the canvas defined in setup(),
// the video will be aligned to the top-left corner of the display
// set virtual camera in center of coordinate space (this is for the virtual 3D sphere)
camera(0, 0, 0, 0, 0, -(height/2.0) / tan(PI*30.0 / 180.0), 0, 1, 0);
// (first 3 values define position of eye; next 3 define position of the scene center;
// last 3 define the orientation of the axes(?));
// tan(PI*30.0/180.0) defines the Field of View (?)
noStroke();
// align based on IMU -- here faked with mouse
rotateY(map(mouseX, 0, width, -HALF_PI, HALF_PI));
//The Y/X axis rotation range is mapped to the movement of the mouse X/Y axis,
// where mouse values are scaled from 0 to the width of the screen;
// half pi defines the lower & upper bounds of movement
rotateX(map(mouseY, 0, height, -HALF_PI, HALF_PI));
// draw objects to their fixed coordinates in the virtual sphere --
// this is a set of six 3D objects with different colours, shapes & defined positions on the virtual sphere
objectA(rad, 0, 0, 25);
objectB(-rad, 0, 0, 25);
objectC(0, rad, 0, 25);
objectD(0, -rad, 0, 25);
objectE(0, 0, rad, 25);
objectF(0, 0, -rad, 25);
}
// This set of code (71 to 120) defines the objects & thier properties
void objectA(float x, float y, float z, float h){
pushMatrix();
translate(x, y, z);
box(h);
fill(245, 108, 40);
popMatrix();
}
void objectB(float x, float y, float z, float h){
pushMatrix();
translate(x, y, z);
box(h);
fill(241, 245, 40);
popMatrix();
}
void objectC(float x, float y, float z, float w){
pushMatrix();
translate(x, y, z);
box(w);
fill(40, 245, 51);
popMatrix();
}
void objectD(float x, float y, float z, float d){
pushMatrix();
translate(x, y, z);
sphere(d);
fill(40, 202, 245);
popMatrix();
}
void objectE(float x, float y, float z, float r){
pushMatrix();
translate(x, y, z);
sphere(r);
fill(76, 40, 245);
popMatrix();
}
void objectF(float x, float y, float z, float r){
pushMatrix();
translate(x, y, z);
sphere(r);
fill(189, 40, 245);
popMatrix();
}
As ever, I have a series of questions (that I’ve added in the comments as well, just so someone reading this understands I don’t know what I’m talking about).
a) Is my understanding of the “float rad = (80)” at the beginning of the sketch accurate, in that it is defining the placement of the 3D objects in space?
This code works in that it is placing 6 objects super-imposed on the webcam feed. Now, I am trying to figure out how to pull in the IMU data & map it to these objects. (i.e., replace the section that says “align based on IMU - here faked with mouse”).
A conceptual question: As of now, the 3D sphere is rotating based on the fake IMU input (i.e mouse movement). Am I correct that, when I plug in the IMU quaternion data, I will have to apply the inverse of the rotation (quaternion.inverse() ) to the 3D objects so that they appear static / fixed in space as the webcam is rotated?
Thanks again for your help ! I have a series of questions on processing the quaternions themelves, but I want to make sure I’m on the right track before pasting / sharing the modified / commented code.
)
