How to modify the program? Sound detection and face tracking

p5.js Libraries
#1

Hi,

I’m learning programming and I’m stuck in one place. According to the code Cassie wrote on the webinar (Creative Coding in p5.js - GitHub Satellite 2020 - YouTube), the sound comes when you click. I want the sound to go on all the time the program detects “open mouth”. How to do it?

And the second question - this code seems to be quite short, but even so, it happens quite often that the preview freezes completely. Only after refreshing the browser window (program restart) does it start to function properly. What it comes from? How can I improve the smooth running of the program?

Below I put the code to the program.

let osc, playing, freq, amp;
let capture;
let ctracker;
let positions;

function setup() {
  let cnv = createCanvas(400, 225);
  cnv.mousePressed(playOscillator);
  osc = new p5.Oscillator("sawtooth");
  capture = createCapture(VIDEO);
  capture.size(400, 225);
  capture.hide();

  ctracker = new clm.tracker();
  ctracker.init();
  ctracker.start(capture.elt);
}

function draw() {
  background(600);
  image(capture, 0, 0);

  positions = ctracker.getCurrentPosition();
  let mouthDist;
  if (positions) {
    positions.forEach((pos) => {
      fill(255, 0, 0);
      noStroke();
      circle(pos[0], pos[1], 3);
    });

    const mouthTop = createVector(positions[60][0], positions[60][1]);
    const mouthBottom = createVector(positions[57][0], positions[57][1]);

    mouthDist = mouthTop.sub(mouthBottom).mag();
  }
  freq = constrain(map(mouthDist, 4, 27, 50, 500), 100, 500);
  amp = constrain(map(mouthDist, 4, 27, 0, 0.5), 0, 1);

  if (playing) {
    osc.freq(freq, 0.1);
    osc.amp(amp, 0.1);
  }
}

function playOscillator() {
  osc.start();
  playing = true;
}

function mouseReleased() {
  osc.amp(0, 0.5);
  playing = false;
}
#2

Hi,

Welcome to the forum! :slight_smile:

Thanks for sharing this video I didn’t know Cassie!

In the video she’s computing the distance between two points of the mouse like so :

mouthDist = mouthTop.sub(mouthBottom).mag();

And this gives values between 3 and 17 when the mouse is open.

If you want to detect if the mouse is open, you need to check if the mouthDist value is greater than some value :

if (mouthDist > 3) {
  // Start playing the oscillator
}

So I would start the oscillator in the setup() function and then in the loop check if the mouse is open and call osc.amp() and the frequency according to that.

For the freezing issue it’s difficult to know why because it could be your computer, your browser (the number of tabs and processes running on your laptop), p5 web editor or the face tracking library initialization…

1 Like
#3

Hi Joseph,

Thank you for your response.

I don’t know if we got on well. My intention is to run the oscillator whether the mouse is open or not. I want the program to still respond to “mouthDist”, but without having to click the mouse. Forgive this request, but I’m not fluent in JS yet. Are you able to show your solution directly in the code?

As for freezing, I will probably have to test different configurations and diagnose the cause on that basis. Thank you very much :slight_smile:

1 Like
#4

Hi again,

Sorry I misspelled mouth with mouse, my bad! :hugs:

In the above code, this is how you play a sound with the oscillator :

// Start the oscillator
osc.start();

// Then set its frequency and amplitude
osc.freq(freq, 0.1);
osc.amp(amp, 0.1);

Then to know when the mouth is open or not, you have the mouthDist variable that gives you the distance between two points on the mouth.

The mouth is open when this value is greater than something like so :

if (mouthDist > minimumMouthDistance) {
  // Then do something
}

where minimumMouthDistance is the minimum threshold distance where the mouth is considered open (here 3 for example but it depends on the size of the face related to the camera).

For this check this pseudo code that resume the previous points :

const minimumMouthDistance = 3;

let osc;

function setup() {
  // Create your oscillator
  osc = new p5.Oscillator("sawtooth");

  // Start it
  osc.start();
}

function draw() {
  // Get the distance of the opening of the mouth
  mouthDist = mouthTop.sub(mouthBottom).mag();

  // Check if the 
  if (mouthDist > minimumMouthDistance) {
    // Play some sound
    osc.freq(freq, 0.1);
    osc.amp(amp, 0.1);
  } else {
    // Otherwise mute the oscillator
    osc.amp(0, 0.5);
  }
}
1 Like
#5

Thank you very much for the hints. I tested this solution, but unfortunately, it doesn’t work as it should.

I looked for information and found a clue. Starting an oscillator on a user gesture will enable audio in browsers that have a strict autoplay policy. That’s i why this code snippet is used:

function playOscillator () {
   osc.start ();
   playing = true;
}

But this is exactly what I want to avoid – my program will ultimately run offline on a local device and I don’t need it. So.

userStartAudio ();

Probably this is what will help solve the problem. The oscillator will start when it detects user movement.

But here comes the beginner’s problem again. I still don’t know how to implement this in the code :frowning:

1 Like
#6

No this is not related at all. Starting the oscillator based on a user action is a behaviour that you the programmer you are defining.

This function is just starting the oscillator and setting a boolean variable to true. But those were defined in the code by Cassie and are not mandatory to start playing the sound.

For example this code works just fine without having to use an additional variable :

let osc;

function setup() {
  createCanvas(400, 400);
  
  osc = new p5.Oscillator("sawtooth");

  osc.start();
}

function draw() {
  background(600);

  osc.freq(500);
  osc.amp(0.5);
}

And I could define the user activation as follows :

Start playing the sound when the mouse is on the second half part of the canvas on the X direction

Which translates to this :

let osc;

function setup() {
  createCanvas(400, 400);
  
  osc = new p5.Oscillator("sawtooth");

  osc.start();
}

function draw() {
  background(600);
  
  fill(0);
  rect(0, 0, width / 2, height);
  
  fill(255, 0, 0);
  textSize(40);
  
  if (mouseX > width / 2) {
    text("Playing!!", 50, 50);
    
    osc.freq(500);
    osc.amp(0.5);
  } else {
    text("Not playing", 50, 50);
    
    osc.amp(0);
  }
}

Here I defined the condition I needed to play the oscillator based on some user input but you can change it to whatever you want : in your case when the distance between two points on the mouth is greater than some value

Is it more clear? :wink: