Posenet & P5js - classification of poses to trigger sounds

Hi there! I’m a newbie to P5js and looking for some help. So what I am trying to do is use the outputs of a trained Posenet model to trigger sound files playing. I’m following Dan Shiffman’s ‘Posenet Classification’, tutorial and sketch, where he trains a new model to recognise 4 classified outputs of, Y, M, C, A. If the result of a pose is ‘Y’ I want to trigger a preloaded sample, then same for the other outputs. I’m just wondering where I would call the play sound function within this sketch. Could I call it in the ‘got results’ function of line 92, in the below sketch?


something like if poseLabel = Y, then play sound

any help, much appreciated!

Hi :slight_smile:

For reference, I have this older example that does something similar, but for the more general featureExtractor (which looks at everything from the webcam stream, not just the body position): https://editor.p5js.org/AndreasRef/sketches/ryLlIOJpX

Perhaps give it a try and see if you can adapt the logic from that sketch onto the Posenet Classification one?

Good luck and have fun :stuck_out_tongue:

thanks I’ll give it a try!

@AndreasRef I tried to adapt what you’ve done in your sketch to my one, applying what I thought would work and similar logic, but alas no dice :frowning: would you mind seeing if you can spot why the sounds aren’t playing? I tried to run this locally, with the wavs sitting in the porject folder.


thanks :smile:

When running the sketch locally, is it directly from the file system (without a web server)? If so, you’ve run into this problem:

Thanks Sven. my process locally, is using Atom and a p5js package which creates a server and runs the sketch. The audio files are in a local path. I think that’s ok?

Yeah, you should be okay then. Maybe it’s the dreaded the-user-must-interact-before-sound-can-be-played-feature causing trouble?

ah I see, this is very useful to know as I’m hoping to build a website at the end of the project. For now, it could well be that adding some interaction on the server site will help. I also have very little confidence that the script I adapted actually makes sense, so could be an issue in how its being called.

hopefully will get it working.


It seems like your libraries are not in a subfolder called libraries in your linked sketch, which they should if you want to call them using <script src="libraries/p5.js">. But instead of fixing something that does not work, perhaps build on top of an example that works? Here is one (the first result on Google) where a sound is playing in the p5js editor: https://editor.p5js.org/shawn/sketches/rJtCL40im

Perhaps adapt from that one?


thanks, sorry that was my bad for not saying but the sketch I shared was just to show the script rather than how it is locally. I am not calling <script src="libraries/p5.js"> . in the index.html locally, all the library files that are being called in index.html are in the folder directory, as are the sounds.

Thanks for the example, I’m just trying to see if it makes sense to call the play sound function, where I am in the screenshot? I understand the basics of playing sound files, just trying to get the triggering on result working :slight_smile:

thanks again!

Yeah, that looks about right from what I can see in the screenshots. Is it not working?

thanks, I am going to try again as I realised about this need for user interaction on a page before audio can start!