# Speech Recognition

**URL:** <https://discourse.processing.org/t/speech-recognition/10438>\
**Category:** Libraries\
**Created:** [April 18, 2019, 8:55pm UTC](https://discourse.processing.org/t/speech-recognition/10438 "2019-04-18T20:55:23Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![dannfelixx](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/dannfelixx/32/4718_2.png) [@dannfelixx](https://discourse.processing.org/u/dannfelixx)\
**Post date:** [April 18, 2019, 8:55pm UTC](https://discourse.processing.org/t/speech-recognition/10438/1 "2019-04-18T20:55:23Z")

</div>

Hello!

I’m a student working on a small project for class and I’m having some troubles (I’m pretty new to p5.js). I’m using the p5.speech library to get speech recognition into my project; basically what I want to do is get the words that are being said to be displayed in the sketch.

In the sketch, it seems like the recognized words are being turned into strings (or a string), but I just cant figure out how to get them to actually draw. The words display in the console.log perfectly as you speak, which should mean that it would be simple (right?). I’ve also tried a bunch of things to find out how the variable that stores the words functions, but I can’t get anything to work. I’m pretty stumped. If anyone can figure out how to get the words displayed onto the sketch, or at least use them like any other string, that would be great.

Here’s a link to the project:  
[https://editor.p5js.org/felixd1@tcnj.edu/sketches/1E3Vv2zTA](https://editor.p5js.org/felixd1@tcnj.edu/sketches/1E3Vv2zTA)

---

<div class="post-metadata">

**Author:** ![josephh](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/josephh/32/210_2.png) [@josephh](https://discourse.processing.org/u/josephh)\
**Post date:** [April 18, 2019, 10:18pm UTC](https://discourse.processing.org/t/speech-recognition/10438/2 "2019-04-18T22:18:27Z")

</div>

Hi,

Welcome to the Processing community!  
When you don’t know how to do something, you often check the `p5.js` reference because almost everything is here :

- [https://p5js.org/reference/](https://p5js.org/reference/)

For drawing text on the canvas, here is what you need :

- [https://p5js.org/reference/#/p5/text](https://p5js.org/reference/#/p5/text)

---

<div class="post-metadata">

**Author:** ![josephh](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/josephh/32/210_2.png) [@josephh](https://discourse.processing.org/u/josephh)\
**Post date:** [April 18, 2019, 10:20pm UTC](https://discourse.processing.org/t/speech-recognition/10438/3 "2019-04-18T22:20:49Z")

</div>

Oh ok sorry didn’t see you were already using the `text()` function.

---

<div class="post-metadata">

**Author:** ![dannfelixx](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/dannfelixx/32/4718_2.png) [@dannfelixx](https://discourse.processing.org/u/dannfelixx)\
**Post date:** [April 19, 2019, 4:37am UTC](https://discourse.processing.org/t/speech-recognition/10438/4 "2019-04-19T04:37:43Z")

</div>

Yea I used the text() function to test a bunch of things, and try to display the text of course. Thanks though. My trouble lies in the variable that stores the words, since it really seems like it should work like any other string, but it just doesn’t.

---

<div class="post-metadata">

**Author:** ![janglee](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/janglee/32/4035_2.png) [@janglee](https://discourse.processing.org/u/janglee)\
**Post date:** [April 19, 2019, 5:00am UTC](https://discourse.processing.org/t/speech-recognition/10438/5 "2019-04-19T05:00:32Z")

</div>

@dannfelixx you have to write moving or changing stuff inside `draw` function. [read wiki](https://p5js.org/reference/#/p5/draw) to get more idea.

Put this lines in `draw` function to render words in sketch.

```auto
  if(String(mostrecentword) !== ""){
    text(String(mostrecentword), 20, 20);  
  }

```

---

<div class="post-metadata">

**Author:** ![dannfelixx](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/dannfelixx/32/4718_2.png) [@dannfelixx](https://discourse.processing.org/u/dannfelixx)\
**Post date:** [April 19, 2019, 3:06pm UTC](https://discourse.processing.org/t/speech-recognition/10438/6 "2019-04-19T15:06:24Z")

</div>

I tried a few different combinations of stuff in draw, and that line doesn’t seem to work ☹ that’s why its so confusing, the mostrecentword variable doesn’t act like a string even though it looks like it should

---

<div class="post-metadata">

**Author:** ![janglee](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/janglee/32/4035_2.png) [@janglee](https://discourse.processing.org/u/janglee)\
**Post date:** [April 20, 2019, 6:55am UTC](https://discourse.processing.org/t/speech-recognition/10438/7 "2019-04-20T06:55:14Z")

</div>

@dannfelixx when I run your sketch. I get `/api/session:1 Failed to load resource: the server responded with a status of 404 ()` error in console. can you confirm this ? you can open console by proessing `F12` or `ctrl+shift+I`.

---

<div class="post-metadata">

**Author:** ![kfrajer](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/kfrajer/32/196_2.png) [@kfrajer](https://discourse.processing.org/u/kfrajer)\
**Post date:** [April 20, 2019, 4:09pm UTC](https://discourse.processing.org/t/speech-recognition/10438/8 "2019-04-20T16:09:38Z")

</div>

Looking at your code, the first thing you need to be aware is when p5 is _available_, which is within the `setup()` function. I did a search an found [this](https://www.openprocessing.org/sketch/565564/)

Below is a working demo. If it shows an error, just start speaking and you will see it starts processing the audio. Tested in Chrome. It doesn’t seem to be supported in FF.

Kf

```auto
var myRec;
var mostrecentword = "";

var x, y;
var dx, dy;
var step;

function setup() {
  createCanvas(300,400);  
  
  
  myRec = new p5.SpeechRec('en-US'); // new P5.SpeechRec object
  myRec.continuous = true; // do continuous recognition
  myRec.interimResults = true; // allow partial recognition (faster, less accurate)
  
  myRec.onResult=parseResult;
  myRec.onError= showError;
  myRec.start(); 
  
  // graphics stuff:
  textFont('Georgia');
  textSize(12);
  textAlign(LEFT);
  fill(0, 0, 0, 255);
  x = width / 2;
  y = height / 2;
  dx = 0;
  dy = 0;
  
  step=10;

}

function draw() {
  background(200);

  text("Hello", 12, 30);
  
  if (mostrecentword != "") {
    x=x+dx;
    y=y+dy;
    text(String(mostrecentword),x ,y);
    dx=0;
    dy=0;
  }
  
}

function parseResult() {
  
  // recognition system will often append words into phrases.
  // so hack here is to only use the last word:
  mostrecentword = myRec.resultString.split(' ').pop();
  
  if (mostrecentword.indexOf("left") !== -1) {
    dx = -step;
    dy = 0;
  } else if (mostrecentword.indexOf("right") !== -1) {
    dx = step;
    dy = 0;
  } else if (mostrecentword.indexOf("up") !== -1) {
    dx = 0;
    dy = -step;
  } else if (mostrecentword.indexOf("down") !== -1) {
    dx = 0;
    dy = step;
  } else if (mostrecentword.indexOf("clear") !== -1) {
    //background(255);
  }
  console.log(myRec.resultString);
  console.log(myRec.resultString.split(' ').length);
}

function showError(){
	console.log('There is an error');
	text('There is an error', windowWidth/2, windowHeight/2);
}

```

---

<div class="post-metadata">

**Author:** ![GoToLoop](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/gotoloop/32/86_2.png) [@GoToLoop](https://discourse.processing.org/u/GoToLoop)\
**Post date:** [April 20, 2019, 4:33pm UTC](https://discourse.processing.org/t/speech-recognition/10438/9 "2019-04-20T16:33:59Z")

</div>

> [@kfrajer](#):
>
> , the first thing you need to be aware is when p5 is variable, which is within the **setup()** function.

- p5 is a variable everytime & everywhere!
- p5 is a global variable that stores the reference for the “p5js”’ constructor function (a.K.a. `class`).
- SpeechRec is a property of class p5, which the library “p5.speech” had appended to it.
- Like variable p5, property SpeechRec also refers to a `class` (constructor function).
- We can say that SpeechRec is a nested static `class` of `class` p5.
- BtW, **setup()** is a p5 callback function.

---

<div class="post-metadata">

**Author:** ![kfrajer](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/kfrajer/32/196_2.png) [@kfrajer](https://discourse.processing.org/u/kfrajer)\
**Post date:** [April 20, 2019, 5:42pm UTC](https://discourse.processing.org/t/speech-recognition/10438/10 "2019-04-20T17:42:21Z")

</div>

@GoToLoop Auto-corrector changed what I meant to say, not variable but available, so my mistake. I amended my post.

I did some more testing and doing `myRec = new p5.SpeechRec('en-US');` doesn’t seem to work in the global context. I do not have a certain answer as I am experiencing errors with the library from time to time. I am guessing it is because the library object is not properly disposed. From my limited experience with p5, should `speechRec()` be called in setup, right? Is this a requirement for “SpeechRec is a nested static `class` of `class` p5”? When is setup called during initialization? I assume the p5 constructor is called when the p5.js library is loaded but setup must be called in `window.onload()` or similar method?

Kf

---

<div class="post-metadata">

**Author:** ![GoToLoop](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/gotoloop/32/86_2.png) [@GoToLoop](https://discourse.processing.org/u/GoToLoop)\
**Post date:** [April 20, 2019, 8:21pm UTC](https://discourse.processing.org/t/speech-recognition/10438/11 "2019-04-20T20:21:28Z")

</div>

- Variable p5 is globally declared and initialized right after the library is loaded.
- An instance of p5 is created right after the p5js finds a **setup()** or **draw()** in the global context.
- Otherwise, p5js rechecks for their existence **onload()** event.
- Then **preload()** & **setup()** are called back once everything is properly initialized.

---

<div class="post-metadata">

**Author:** ![kfrajer](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/kfrajer/32/196_2.png) [@kfrajer](https://discourse.processing.org/u/kfrajer)\
**Post date:** [April 21, 2019, 1:09am UTC](https://discourse.processing.org/t/speech-recognition/10438/12 "2019-04-21T01:09:20Z")

</div>

Yes, I bit rusty with this life cycle @Gotoloop. So based on this, is `myRec` initialized in the global context legal?

Kf

---

<div class="post-metadata">

**Author:** ![GoToLoop](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/gotoloop/32/86_2.png) [@GoToLoop](https://discourse.processing.org/u/GoToLoop)\
**Post date:** [April 21, 2019, 1:34am UTC](https://discourse.processing.org/t/speech-recognition/10438/13 "2019-04-21T01:34:44Z")

</div>

As long as “p5.speech” is already loaded, I believe it should work globally. 🤔

---

<div class="post-metadata">

**Author:** ![dannfelixx](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/dannfelixx/32/4718_2.png) [@dannfelixx](https://discourse.processing.org/u/dannfelixx)\
**Post date:** [April 21, 2019, 1:53am UTC](https://discourse.processing.org/t/speech-recognition/10438/14 "2019-04-21T01:53:58Z")

</div>

Did a bit more testing, looks like the continuous speech recognition isn’t compatible with the draw function if you want to continuously draw the words. Logging the frame rate shows that it gets stuck on the first frame. I’m guessing that since it’s a continuous process, p5 technically can’t continue to run the code past it (at 60 times a second) like it normally would. In the example on the p5.speech site, it’s not drawing the words it recognizes as text - it only uses them to move the dot around.

---

<div class="post-metadata">

**Author:** ![kfrajer](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/kfrajer/32/196_2.png) [@kfrajer](https://discourse.processing.org/u/kfrajer)\
**Post date:** [April 21, 2019, 3:49pm UTC](https://discourse.processing.org/t/speech-recognition/10438/15 "2019-04-21T15:49:08Z")

</div>

> [@dannfelixx](#):
>
> continuous speech recognition isn’t compatible with the draw function if you want to continuously draw the words

Did you try my code above? Because I had a `draw()`. Also, what browser and OS are you using?

From my limited testing, it works continuously most of the time. If you are referring to the code you posted, it only pops the last recognized word. In my code, I was printing all the words in the stack. If you stop speaking, I think it stops recognizing. If you start speaking, it activates again. Where you watching the console log messages? I did notice the sketch would get stack when restarting. I am guessing the system was trying to load the rec object and it hangs there. I suggest you do a broader search in the js community to see if this is observed in other projects or probly suggestions how to resolve your current issue. But first, become familiar with the library documentation.

Kf

---

<div class="post-metadata">

**Author:** ![dannfelixx](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/dannfelixx/32/4718_2.png) [@dannfelixx](https://discourse.processing.org/u/dannfelixx)\
**Post date:** [April 21, 2019, 4:55pm UTC](https://discourse.processing.org/t/speech-recognition/10438/16 "2019-04-21T16:55:18Z")

</div>

I just got it to work on Chrome 🙂 I used your code and it works very well, now I can finally finish the rest of the project. Many thanks to you and everyone who helped!