I am trying to create an audio visualiser using Processing sound library. And the visualiser runs okay while running the sketch but as soon as I use saveFrame, the frame rate drops considerably and the visuals are out of sync with the audio.
Problems:
- Audio is playing in real time but frames are exported at a slow rate. Hence the mismatch.
Possible Solutions and associated problems:
1. Use saveFrame in a different thread.
The problem here is draw function speed is too fast for the other thread to keep up.
2. Sample audio at fixed intervals and map the visuals to it.
Let’s say the audio is 5 seconds long with a sample rate of 44100khz. That amounts to 220500 frames in 5 seconds. Now if the video is supposed to be 30fps with a total of 150 frames for 5 seconds, I have to sample the audio at every 44100/30 = 1470 frames. Now, reading only the audio sample values at 150 frames instead of 220500 frames loses a great deal of information.
also, how do I run FFT on the 150 frame audio sample? and what are the values that have been read in the sample? Is that amplitude? Sorry, I have zero knowledge about audio engineering.
3. Save frames to a PImage array and export only after the program is finished
Suppose the audio is 1 min long and resolution of video is full HD, that would result in 1800 PImages in the array with a size of 1920x1080. Not very memory friendly, I’d say.
I have been stuck at this problem for very long. Please help me with an easy solution. I just want to make some audio reactive stuff, nothing too complex.