Moving an object ( ellipse ball) between two keypoints (from posenet /ml5)?

Timo · September 6, 2021, 9:58am

Hi , I am trying exploring the ml5 library implementation of posenet. I am trying to implement a animation where a ball is rolling between the left ear and right ear in a straight line.
For this I have done the following

extracted the ear co -ordinates
Joined them with a line
Created an ellipse ball at the right ear which is the starting point and a constant speed of 1
I noticed that the ellipse that is created to move from left to right does not really move and seems to be fluctuating between at the ear keypoint.

I am not sure what is the reason and what could the best way to solve the problem and to me the logic seems to be correct and I am quite confused where I have gone wrong .

The sketch is below

let video;
let poseNet;
let pose;
let skeleton;
let endpoint 
// Ball exepriments moving 
var startpoint 
var xspeed =1;




function setup() {
  createCanvas(640, 480);
  video = createCapture(VIDEO);
  video.hide();
  poseNet = ml5.poseNet(video, modelLoaded);
  poseNet.on('pose', gotPoses);
}

function gotPoses(poses) {
  //console.log(poses); 
  if (poses.length > 0) {
    pose = poses[0].pose;


    skeleton = poses[0].skeleton;
  }
}


function modelLoaded() {
  console.log('poseNet ready');
}

function draw() {
  image(video, 0, 0);
  


  if (pose ) {
    
    let eyeR = pose.rightEye;
    let eyeL = pose.leftEye;
    let earL = pose.leftEar;
    let earR = pose.rightEar;
    
    let d = dist(eyeR.x, eyeR.y, eyeL.x, eyeL.y);
    // Draw the two points , start point and end point 
    stroke(0,100,10);
    ellipse(earR.x, earR.y,10,10)
    ellipse(earL.x,earL.y,10,10)
    //Draw a line between start and end point 
    stroke(255);
    line(earR.x, earR.y, earL.x,earL.y);
    // Ellipse / ball moves from left to right and right to left when it reaches endpoint
    // Output the ball does not move .
    startpoint = earR.x;
    startpoint = earR.x +xspeed ;
    /*
    To move it back and forth 
    if (startpoint == eaR.x) {
     xpeed *=-1 
    }
    if (startpoint >= earR.x){
    xspeed *= -1
    }
    
    */
    console.log(startpoint)
    ellipse(startpoint, 100, 20,20) // Y cor ordinate is just to test 
  
    for (let i = 0; i < pose.keypoints.length; i++) {
      let x = pose.keypoints[i].position.x;
      let y = pose.keypoints[i].position.y;
      //fill(0,255,0);
      //ellipse(x,y,16,16);
    }
    
    for (let i = 0; i < skeleton.length; i++) {
      let a = skeleton[i][0];
      let b = skeleton[i][1];
      strokeWeight(2);
      stroke(255);
      line(a.position.x, a.position.y,b.position.x,b.position.y);      
    }
  }

}

the index.html file

<!DOCTYPE html>
<html>

<head>
  <script src="https://cdn.jsdelivr.net/npm/p5@1.4.0/lib/p5.min.js"></script>
  <script src="https://cdn.jsdelivr.net/npm/p5@1.4.0/lib/addons/p5.sound.min.js"></script>
  <script src="https://unpkg.com/ml5@0.4.2/dist/ml5.min.js"></script>
  <meta charset="utf-8" />

</head>

<body>
  <script src="sketch.js"></script>
</body>

</html>

Any tips / suggestions would be highly appreciated .

josephh · September 6, 2021, 7:40pm

Hi @Timo,

I think that the reason the ball is not moving is that each time in the draw() loop, you are doing this:

startpoint = earR.x;
startpoint = earR.x + xspeed;

Two things are wrong here:

You are assigning a value to startpoint twice. Since the instructions are executed sequentially, startpoint is always going to be equal to earR.x + xspeed
Since startpoint = earR.x + xspeed every time, the startpoint is not moving, it’s the same value. You need to add something to it at each loop and store that result in order for the ball to move.

What you should do is to actually store the position of the ball (between 0 and 1) on the line between the two ear points rather than it’s real location in pixels. Since the points are always moving, just compute the location of the ball in pixels as being on a line (with a linear equation for example).

Hope it helps

Timo · September 7, 2021, 9:05am

Hello @josephh , thanks for pointing out the mistake. It just slipped out of my sight. From what i understand is that you suggest some kind of normalization ?

The idea i was trying to test was to have two points, start point ( right ear) and end point left ear ) irrespective of different person ( that is i wanted it to be dynamic depending on who the person is ) , So i dont know what is the best method to do so. I was thinking to dump the kepoints as soon as the ears are detected and then take the read the keypoints to move the ball. Perhaps that is not the right way to do so ?

My end goal is to make the eyes follow the ball ( in x direction) and have an indication (color change) if the x cordinate of eye is same as x co cordinate of the moving ball , we are following the ball in a correct way .

josephh · September 7, 2021, 12:22pm

Yes. Basically if you have two points, you can get the slope of the line going through the two points.
You can do that with vectors actually:

A = earL
B = earR

AB = earR - earL
ballPosition = A + B * location

(A, B, AB and ballPosition are vectors, you can use p5.Vector for that)
(location is the normalized position between 0 and 1)

you want a separate ball for each person (moving independently)? If so, you need to store the ball location for each person (depending on its index) but it might be difficult to tell which person is which.

For that, does the ML5 library provides functions to track eyes? I don’t think that with a simple webcam you should be able to do that, most of the time it involves special glasses that tracks the eye ball.

Timo · September 9, 2021, 9:10am

Hi @josephh ,
Thanks for the tips . And my mistake , I didnt see that coming
Perhaps i was wrong, in explaining how i was thinking about my sketch. I am just focussed on a single person. (not multi person webcam feed). So what I did was that, i used framecount property. So if the framecount==120 then i would draw a straight line that connects left ear and right ear at that moment.

I then compared the nose coordinates , and whenever the nose co ordinates was on the line then the color changed ( a very subtle change), I think the logic still is wrong

    
    if (noseX >= startx || noseY >= starty && noseX <= endx || noseY <= endy) {
      // Condition nose follows the line that is drawn 
      // No change in the color 
      // Prints out
      console.log("IN THE LINE");
      // Change color 
      stroke(0,255,255); // But the color does not change - need a check
      line(startx, starty, endx, endy); // This will draw the reference line to follow 
   

    }

However , what i wanted was to make an ellipse ball move from left to right on the line . The idea to follow the ellipse balls with nose co-ordinates and to see how many times it was wrong .

Is there a way to have a buffer , it seems like now the color changes when the nose cor ordinates are exactly crossing the line .

What do you think ?
Added the revised code snippet.

let video;
let poseNet;
let pose;
let skeleton;
let endpoint;
// Ball exepriments moving
var startpoint;
var speed = 1;
let test, startx, starty, endx, endy, x;
let correct = 0;
let wrong = 0;

function setup() {
  createCanvas(640, 480);
  video = createCapture(VIDEO);
  video.hide();
  poseNet = ml5.poseNet(video, modelLoaded);
  poseNet.on("pose", gotPoses);
  frameRate(30);
}

function gotPoses(poses) {
  //console.log(poses);
  if (poses.length > 0) {
    pose = poses[0].pose;

    skeleton = poses[0].skeleton;
  }
}

function modelLoaded() {
  console.log("poseNet ready");
}

function draw() {
  image(video, 0, 0);
  text("Face the webcam", 10, 30);
  fill(255, 200, 255);

  if (pose) {
    let eyeR = pose.rightEye;
    let eyeL = pose.leftEye;
    let earL = pose.leftEar;
    let earR = pose.rightEar;
    if (frameCount == 120) {
      console.log("Drawing the straight line");

      startx = earR.x;
      starty = earR.y;
      endx = earL.x;
      endy = earL.y;
      console.log(startx);
      console.log(endx);
     
    }

    line(startx, starty, endx, endy); // This will draw the reference line to follow 
    stroke(0,255,0);
    let noseX = pose.nose.x;
    let noseY = pose.nose.y;
    
    if (noseX >= startx || noseY >= starty && noseX <= endx || noseY <= endy) {
      // Condition nose follows the line that is drawn 
      // No change in the color 
      // Prints out
      console.log("IN THE LINE");
      // Change color 
      stroke(0,255,255); // But the color does not change - need a check
      line(startx, starty, endx, endy); // This will draw the reference line to follow 
   

    }
   

    for (let i = 0; i < pose.keypoints.length; i++) {
      let x = pose.keypoints[i].position.x;
      let y = pose.keypoints[i].position.y;
      //fill(0,255,0);
      //ellipse(x,y,16,16);
    }

    for (let i = 0; i < skeleton.length; i++) {
      let a = skeleton[i][0];
      let b = skeleton[i][1];
      strokeWeight(2);
      stroke(255);
      line(a.position.x, a.position.y, b.position.x, b.position.y);
    }
  }
      /*
     ellipse(startx,50,50,50);
     startx = startx + speed;
		if(startx >= endx || startx <= startx)
	{
		speed = -speed;
	}
   */
    // check the x axis co cordinates of nose and the line
}

Timo · September 24, 2021, 11:10am

I managed to find a temporary solution , thanks for the suggestions

Topic		Replies	Views
Calculating the displacement of keypoints in pixels Libraries	3	408	July 31, 2021
P5 Questions! How to change direction of object Coding Questions	1	1546	February 8, 2019
Eyeball moving effect Coding Questions homework	1	301	May 24, 2021
Spawn a shape or object whenever the initial object reaches a specific locaton Beginners	2	481	March 17, 2020
Animating an object from one position to other Coding Questions	5	1238	February 4, 2019

Moving an object ( ellipse ball) between two keypoints (from posenet /ml5)?

Related Topics