# Posenet & P5js - classification of poses to trigger sounds

**URL:** <https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363>\
**Category:** Libraries\
**Created:** [May 28, 2020, 9:23am UTC](https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363 "2020-05-28T09:23:02Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![jhyde](https://avatars.discourse-cdn.com/v4/letter/j/fbc32d/32.png) [@jhyde](https://discourse.processing.org/u/jhyde)\
**Post date:** [May 28, 2020, 9:23am UTC](https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363/1 "2020-05-28T09:23:02Z")

</div>

Hi there! I’m a newbie to P5js and looking for some help. So what I am trying to do is use the outputs of a trained Posenet model to trigger sound files playing. I’m following Dan Shiffman’s ‘Posenet Classification’, tutorial and sketch, where he trains a new model to recognise 4 classified outputs of, Y, M, C, A. If the result of a pose is ‘Y’ I want to trigger a preloaded sample, then same for the other outputs. I’m just wondering where I would call the play sound function within this sketch. Could I call it in the ‘got results’ function of line 92, in the below sketch?

[https://editor.p5js.org/codingtrain/sketches/JoZl-QRPK](https://editor.p5js.org/codingtrain/sketches/JoZl-QRPK)

something like if poseLabel = Y, then play sound

any help, much appreciated!

---

<div class="post-metadata">

**Author:** ![AndreasRef](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/andreasref/32/133_2.png) [@AndreasRef](https://discourse.processing.org/u/AndreasRef)\
**Post date:** [May 28, 2020, 9:56am UTC](https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363/2 "2020-05-28T09:56:40Z")

</div>

Hi 🙂

For reference, I have this older example that does something similar, but for the more general featureExtractor (which looks at everything from the webcam stream, not just the body position): [https://editor.p5js.org/AndreasRef/sketches/ryLlIOJpX](https://editor.p5js.org/AndreasRef/sketches/ryLlIOJpX)

Perhaps give it a try and see if you can adapt the logic from that sketch onto the Posenet Classification one?

Good luck and have fun 😛

---

<div class="post-metadata">

**Author:** ![jhyde](https://avatars.discourse-cdn.com/v4/letter/j/fbc32d/32.png) [@jhyde](https://discourse.processing.org/u/jhyde)\
**Post date:** [May 28, 2020, 1:40pm UTC](https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363/3 "2020-05-28T13:40:44Z")

</div>

thanks I’ll give it a try!

---

<div class="post-metadata">

**Author:** ![jhyde](https://avatars.discourse-cdn.com/v4/letter/j/fbc32d/32.png) [@jhyde](https://discourse.processing.org/u/jhyde)\
**Post date:** [May 28, 2020, 2:49pm UTC](https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363/4 "2020-05-28T14:49:00Z")

</div>

@AndreasRef I tried to adapt what you’ve done in your sketch to my one, applying what I thought would work and similar logic, but alas no dice ☹ would you mind seeing if you can spot why the sounds aren’t playing? I tried to run this locally, with the wavs sitting in the porject folder.

[https://editor.p5js.org/jhyde/sketches/Tf\_Ra53hS](https://editor.p5js.org/jhyde/sketches/Tf_Ra53hS)

thanks 😄

---

<div class="post-metadata">

**Author:** ![Sven](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/sven/32/8293_2.png) [@Sven](https://discourse.processing.org/u/Sven)\
**Post date:** [May 28, 2020, 6:03pm UTC](https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363/5 "2020-05-28T18:03:15Z")

</div>

When running the sketch locally, is it directly from the file system (without a web server)? If so, you’ve run into this problem:

> [@Looking for help downloading from editor with interactive sketch](https://discourse.processing.org/t/looking-for-help-downloading-from-editor-with-interactive-sketch/20669/3):
>
> This has to do with how `loadImage` and `loadSound` are implemented. The functions use something called Fetch API and have that configured in such a way that downloading of files isn’t allowed locally. Believe it or not, it’s related to security.
> 
> There’s not much you can do about it really (except patching p5.js). Running a web server (locally or elsewhere) as you did is the way to go.
> 
> If you’re interested in the technical details, read up on [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) and [Fetch API](https://developer.mozilla.org/en-US/docs/Web/API/Fetch_API).

---

<div class="post-metadata">

**Author:** ![jhyde](https://avatars.discourse-cdn.com/v4/letter/j/fbc32d/32.png) [@jhyde](https://discourse.processing.org/u/jhyde)\
**Post date:** [May 28, 2020, 6:07pm UTC](https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363/6 "2020-05-28T18:07:06Z")

</div>

Thanks Sven. my process locally, is using Atom and a p5js package which creates a server and runs the sketch. The audio files are in a local path. I think that’s ok?

---

<div class="post-metadata">

**Author:** ![Sven](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/sven/32/8293_2.png) [@Sven](https://discourse.processing.org/u/Sven)\
**Post date:** [May 28, 2020, 6:37pm UTC](https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363/7 "2020-05-28T18:37:43Z")

</div>

Yeah, you should be okay then. Maybe it’s the dreaded the-user-must-interact-before-sound-can-be-played-feature causing trouble?

> [@Getting sounds to play in mobile?](https://discourse.processing.org/t/getting-sounds-to-play-in-mobile/20673/2):
>
> Some browsers, like Safari on iOS, won’t play audio unless the user interacts with the page first, like tapping a button. To successfully play audio in those browsers, you could add a “Press to play” button and start playing audio after the user tapped it.
> 
> From [Apple’s Safari HTML5 Audio and Video Guide](https://developer.apple.com/library/archive/documentation/AudioVideo/Conceptual/Using_HTML5_Audio_Video/PlayingandSynthesizingSounds/PlayingandSynthesizingSounds.html#//apple_ref/doc/uid/TP40009523-CH6-SW1):
> 
> > **Note:** On iOS, the Web Audio API requires sounds to be triggered from an explicit user action, such as a tap.

---

<div class="post-metadata">

**Author:** ![jhyde](https://avatars.discourse-cdn.com/v4/letter/j/fbc32d/32.png) [@jhyde](https://discourse.processing.org/u/jhyde)\
**Post date:** [May 28, 2020, 6:49pm UTC](https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363/8 "2020-05-28T18:49:10Z")

</div>

ah I see, this is very useful to know as I’m hoping to build a website at the end of the project. For now, it could well be that adding some interaction on the server site will help. I also have very little confidence that the script I adapted actually makes sense, so could be an issue in how its being called.

hopefully will get it working.

cheers

---

<div class="post-metadata">

**Author:** ![AndreasRef](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/andreasref/32/133_2.png) [@AndreasRef](https://discourse.processing.org/u/AndreasRef)\
**Post date:** [May 28, 2020, 9:37pm UTC](https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363/9 "2020-05-28T21:37:08Z")

</div>

It seems like your libraries are not in a subfolder called `libraries` in your linked sketch, which they should if you want to call them using `<script src="libraries/p5.js">`. But instead of fixing something that does not work, perhaps build on top of an example that works? Here is one (the first result on Google) where a sound is playing in the p5js editor: [https://editor.p5js.org/shawn/sketches/rJtCL40im](https://editor.p5js.org/shawn/sketches/rJtCL40im)

Perhaps adapt from that one?

---

<div class="post-metadata">

**Author:** ![jhyde](https://avatars.discourse-cdn.com/v4/letter/j/fbc32d/32.png) [@jhyde](https://discourse.processing.org/u/jhyde)\
**Post date:** [May 29, 2020, 6:52am UTC](https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363/10 "2020-05-29T06:52:37Z")

</div>

thanks, sorry that was my bad for not saying but the sketch I shared was just to show the script rather than how it is locally. I am not calling `<script src="libraries/p5.js">` . in the index.html locally, all the library files that are being called in index.html are in the folder directory, as are the sounds.

Thanks for the example, I’m just trying to see if it makes sense to call the play sound function, where I am in the screenshot? I understand the basics of playing sound files, just trying to get the triggering on result working 🙂

thanks again!

 ![Screenshot 2020-05-29 at 07.49.42](https://canada1.discourse-cdn.com/flex036/uploads/processingfoundation1/original/2X/0/04e051d45443e7de20f3ff396af809c7d2692199.png) ![Screenshot 2020-05-29 at 07.49.51](https://canada1.discourse-cdn.com/flex036/uploads/processingfoundation1/original/2X/1/1e355612f7e090275d4fd359a00a87fba1885aae.png)

---

<div class="post-metadata">

**Author:** ![AndreasRef](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/andreasref/32/133_2.png) [@AndreasRef](https://discourse.processing.org/u/AndreasRef)\
**Post date:** [May 29, 2020, 9:37am UTC](https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363/11 "2020-05-29T09:37:43Z")

</div>

Yeah, that looks about right from what I can see in the screenshots. Is it not working?

---

<div class="post-metadata">

**Author:** ![jhyde](https://avatars.discourse-cdn.com/v4/letter/j/fbc32d/32.png) [@jhyde](https://discourse.processing.org/u/jhyde)\
**Post date:** [May 29, 2020, 11:33am UTC](https://discourse.processing.org/t/posenet-p5js-classification-of-poses-to-trigger-sounds/21363/12 "2020-05-29T11:33:25Z")

</div>

thanks, I am going to try again as I realised about this need for user interaction on a page before audio can start!