Help with projecting a 3D point to the 2D view

Processing Coding Questions
1

Hi all, can anyone offer insight into how the PGraphics3D “projection” and “modelview” (or combined “projmodelview” ) matrices are used to convert a 3d vector in the scene to a final 2d vector on the rasterized screen? I have been able to use the “modelview” matrix on its own to successfully project the point but only when in orthographic projection mode using “ortho()”. Heres an example with peasycam, the green X is drawn to the screen in 2d using peasycams hud drawing functionality and follows its 3d red circle counterpart this is exactly the behaviour I want although I cannot get it to work once a projection with perspective is added to the mix.

PeasyWithHUD
PeasyWithHUD601×640 5.56 KB 

import peasy.*;

PGraphics3D p3d;
PeasyCam camera;

PGraphics overlay;
PVector posScene;
PVector posScreen;

float markerSize = 12.;

void setup() {
  size(600, 600, P3D);
  
  camera = new PeasyCam(this, 500);
  
  p3d = (PGraphics3D)g;

  rectMode(CENTER);
  
  posScene = new PVector(0.,0.,0.);
  posScreen = new PVector(0.,0.,0.);
  
  //draw in order
  hint(DISABLE_DEPTH_TEST);
  
  ortho();
}

void draw() {
  background(100);
  
  posScene = new PVector(125,125,0);
  
  PMatrix trans = p3d.projmodelview.get();
  //trans.invert();
  
  trans.mult(posScene, posScreen);
  //print(posScreen.toString() + '\n');
  
  //draw rectangle 3d scene
  stroke(0);
  strokeWeight(1);
  rect(0,0,250,250);
  
  //draw red point in 3d scene
  stroke(255,0,0);
  strokeWeight(10);
  point(posScene.x, posScene.y, posScene.z);
  
  
  // draw green X in 2d Hud overlay
  camera.beginHUD();
  
  pushMatrix();
  translate(width/2,height/2);
  stroke(0,255,0);
  strokeWeight(2);
  
  PVector pos = new PVector(posScreen.x*width/2,-posScreen.y*height/2);
  
  // X
  line(pos.x-markerSize, pos.y-markerSize, pos.x+markerSize, pos.y+markerSize);
  line(pos.x-markerSize, pos.y+markerSize, pos.x+markerSize, pos.y-markerSize);
  
  camera.endHUD();
  popMatrix();
}

if “ortho()” is removed then the 2d vector outputs seem to become nonsensical. I believe I am applying the matrices correctly but the fact it breaks when introducing perspective implies that there is something wrong with the “projection” matrix, but not the “modelview” matrix. I believe there must be another transformation later in the pipeline (this would make sense as opengl draws in a range from -1 to +1 and processing draws in pixel coordinates), probably within opengl itself but I have been unable to track it down so far.

I will continue looking through the processing codebase but can anyone point me in the right direction or offer any insight? I have looked through peasycams code and don’t think it is having any ill effects on the transforms ultimately (other than manipulating a standard camera) so I think its more of a general issue/question about how processing interacts with opengl.

Ive found a few ultimately unanswered questions on this as a concept so it could be very helpful to figure out, cheers!

1 Like
2

I can’t answer that.

Let me just mention screenX() and screenY() and
modelX(), modelY(), modelZ()

3

I’ve had a bit of a play around with these functions already, they also seemed to offer incomplete transformations (maybe to do with peasycam), but Ill have another go. I would like to get to the root of how this all fits together so I can be more flexible with it in the future. It seems like its all accessible within the libraries, which is excellent. Just needs some unravelling.

1 Like
4

Hi @Flon,

if you just want to translate the 3D with transformation back into 2D …

Cheers
— mnse

import peasy.*;

PVector[] pts = {
  new PVector(-1, -1, 0),
  new PVector(1, -1, 0),
  new PVector(1, 1, 0),
  new PVector(-1, 1, 0)
};

PeasyCam camera;

void setup() {
  size(300, 300, P3D);
  camera = new PeasyCam(this, 500);  
}

void draw() {
  background(100);
  int m = millis();
  if ((m / 5000) % 2 == 0)
    ortho();    
  else
    perspective();
  
  rotateX(m/500.);
  rotateY(m/600.);
  rotateZ(m/700.);
  
  fill(128, 128, 255);
  noStroke();
  beginShape();
  scale(100, 100, 1);
  for (PVector p : pts)
    vertex(p.x, p.y, p.z);
  endShape(CLOSE);

  // convert the current x,y,z position back to x,y
  PVector[] tpts = new PVector[4];
  for (int i = 0; i < pts.length; i++) {
    tpts[i] = new PVector(
      screenX(pts[i].x, pts[i].y, pts[i].z),
      screenY(pts[i].x, pts[i].y, pts[i].z));
  }

  camera.beginHUD();
  fill(255,0,0);
  noStroke();    
  // draw cross at converted 3D to 2D location
  stroke(0, 255, 0);
  for (PVector p : tpts) {
    line(p.x-5, p.y-5, p.x+5, p.y+5);
    line(p.x+5, p.y-5, p.x-5, p.y+5);
  }
  camera.endHUD();
}

ezgif.com-crop

2 Likes
5

Very cool, looks like you’ve nailed it.

I found this resource also for a complete writeup on the transformations that processing is using
[OpenGL Transformation](https://OpenGL Transformation)
So I’ll work through that too to understand whats going on behind the scenes in opengl in general.

This is a great solution though and I will probably go with this method for processing sketches.

I’m not sure what I got wrong the first time around using the ScreenX and ScreenY functions. they look really straightforward now. I think I must have called them in the beginHud()/EndHud section from peasycam which applies an orthographic perspective for its duration. Therefore it gives the ScreenX, ScreenY for that instead of the one with perspective. Interesting little gotcha to know about there.

1 Like