Movie Maker: Time gap between movie and its audio file

Processing Beginners
1

Hello all,

I am using Movie Maker with saveFrame(“frames/####.png”);
and inside Movie Maker also an Audio file.

I use the Audio file also in the Sketch.

I have a huge gap / offset between movie and Audio file.

I would like to have it so that the saved image occurs at the exact time (in the Audio) it was in during the running of the Sketch.

How can I assure this?

How can I map the images in the movie exactly to the audio?

So when the image in the Sketch changes after 1.7 secs I want it to be so in the movie as well so the Audio is mapped to the same images in the Sketch as in the Movie.

Thanks all.

Warm regards,

Chrisir

2

Hello @chrisir,

It has been a while since I did this so took it out for a spin!

MovieMaker uses FFmpeg under the hood and you can certainly customize.

It some effort to get Movie Maker working on my computer:

Movie Maker tool issues

Code to generate frames:

/**
 * MovieMaker Frame Generator for Testing
 * 
 * Author:  glv
 * Date:    2025-07-19
 * Version: 1 0 1
 * 
 * Description
 *
 * Generates frames for testing:
 * - target frameRate for MovieMaker
 * - background color change at marker (10s)
 *
 * Notes:
 * - frameCount for setup() is 0 and draw() starts at 1; adjust as necessary
 * - saves images from  000001.png to 000601.png
 * - sketch frames show text counting from 0 to 299 (yellow) and 300 to 600 (red) background
 * - tga are really fast!
 * - can increase frameRate for fast rendering! Not with audio. :)
 *
 */

int framesToSave;
int t0, t1;
int marker_10s;

void setup()
  {
  size(300, 300);         
  //surface.setVisible(false); // Does not speed things up!
  
  t0 = millis();
  
  int audioTimeSecs = 20;      // Must be in seconds 
  int desiredFrameRate = 30;   // For MovieMaker!
  
  frameRate(1000);             // Make this 1000 and let the rendering fly!  
  
  marker_10s = 10*desiredFrameRate;
  //println(marker_10s);
  
  framesToSave = audioTimeSecs*desiredFrameRate;
  println("It's show time!");
  println();
  println("audioTimeSecs:", audioTimeSecs);
  println("desiredFrameRate", desiredFrameRate);
  println("framesToSave", framesToSave );
  delay(1000);
  println();
  
  // Common to all text:
  textAlign(CENTER, CENTER);
  textSize(96);
  fill(0);
  }
  
void draw()
  {
  // I wanted this on top of draw for now...

  if(frameCount > framesToSave)  // Saves frames 1 to frameSave - 1
    {
    println("Last unsaved frameCount:", frameCount);
    t1 = millis();
    int dt = t1-t0;
    println("Total time:", nfs(dt/1000.0, 0, 2) + "s");
    println("Average frame:", nfs( (float) dt/framesToSave, 0, 2) + "ms");

    // Unsaved last frame:
    background(0, 255, 0);  
    text("Done!", width/2, height/2);

    //exit(); // Your choice!
    noLoop();  // See last frame
    return;    // Goes directly to end of draw from here!
    }  
    
// These frames will be saved:
  
  println("Saved:", frameCount);  // May slow things down!
    
  if(frameCount > marker_10s)
    background(255, 0, 0);
  else
    background(255, 255, 0);
    
  text(frameCount-1, width/2, height/2);  // Adjusted so starts at 0
  
  saveFrame(dataPath("") + "\\" + "######.tga"); // Hyperdrive! tga is fast!
  //saveFrame(dataPath("") + "\\" + "######.png"); // png is slow! Needs to compress.
  }

This (it is in there) was used to delay audio and worked:

https://superuser.com/questions/982342/in-ffmpeg-how-to-delay-only-the-audio-of-a-mp4-video-without-converting-the-au

There are other tools for doing this!
You will have to explore this… my toolbox needs an update.

Have fun!

:).

1 Like
3

Coming back to this old topic…

Thank you for this.

It’s not about when to stop the recording though.

It’s about how to have the visuals in sync with the music.

When in my visual a door slams, it must be met exactly the door sound in the Audio MP3.

It works precisely when I play the Sketch, but the recording is way off.

But we don’t achieve a sync between MP3-file and visual in the resulting movie. I tried hamoid library and your approach with the Tool Movie Maker (installing ffmpeg manually). Nothing works.

There are two params for ffmpeg, itsoffset and atempo= (for Drift issues). I also tried Muxing the visual and audio.

Not solved.

4

Elaborating on my previous post:

  • I used Movie Maker II with a current version of FFmpeg (see reference at the end) to create the movie (video) and add an audio file.
  • The frames came from the sketch I shared earlier:

    image
    image728×498 21 KB

  • The sketch displays frameCount and changes the background from yellow to red at the 10-second mark for a video that is 30 fps.
  • I used the following command line to offset the audio that was added by 10 seconds which then synced and started with frame 300:
    D:\Users\GLV\Documents\P4\saveFrame_sync_1_0_0\data>ffmpeg -i "test.mp4" -itsoffset 10 -i "test.mp4" -map 0:v -map 1:a -c copy "movie-audio-delayed.mp4
Command line output of above. < Click here to view! 
D:\Users\GLV\Documents\P4\saveFrame_sync_1_0_0\data>ffmpeg -i "test.mp4" -itsoffset 10 -i "test.mp4" -map 0:v -map 1:a -c copy "movie-audio-delayed.mp4
ffmpeg version 7.1.1-full_build-www.gyan.dev Copyright (c) 2000-2025 the FFmpeg developers
  built with gcc 14.2.0 (Rev1, Built by MSYS2 project)
  configuration: --enable-gpl --enable-version3 --enable-static --disable-w32threads --disable-autodetect --enable-fontconfig --enable-iconv --enable-gnutls --enable-lcms2 --enable-libxml2 --enable-gmp --enable-bzlib --enable-lzma --enable-libsnappy --enable-zlib --enable-librist --enable-libsrt --enable-libssh --enable-libzmq --enable-avisynth --enable-libbluray --enable-libcaca --enable-libdvdnav --enable-libdvdread --enable-sdl2 --enable-libaribb24 --enable-libaribcaption --enable-libdav1d --enable-libdavs2 --enable-libopenjpeg --enable-libquirc --enable-libuavs3d --enable-libxevd --enable-libzvbi --enable-libqrencode --enable-librav1e --enable-libsvtav1 --enable-libvvenc --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxavs2 --enable-libxeve --enable-libxvid --enable-libaom --enable-libjxl --enable-libvpx --enable-mediafoundation --enable-libass --enable-frei0r --enable-libfreetype --enable-libfribidi --enable-libharfbuzz --enable-liblensfun --enable-libvidstab --enable-libvmaf --enable-libzimg --enable-amf --enable-cuda-llvm --enable-cuvid --enable-dxva2 --enable-d3d11va --enable-d3d12va --enable-ffnvcodec --enable-libvpl --enable-nvdec --enable-nvenc --enable-vaapi --enable-libshaderc --enable-vulkan --enable-libplacebo --enable-opencl --enable-libcdio --enable-libgme --enable-libmodplug --enable-libopenmpt --enable-libopencore-amrwb --enable-libmp3lame --enable-libshine --enable-libtheora --enable-libtwolame --enable-libvo-amrwbenc --enable-libcodec2 --enable-libilbc --enable-libgsm --enable-liblc3 --enable-libopencore-amrnb --enable-libopus --enable-libspeex --enable-libvorbis --enable-ladspa --enable-libbs2b --enable-libflite --enable-libmysofa --enable-librubberband --enable-libsoxr --enable-chromaprint
  libavutil      59. 39.100 / 59. 39.100
  libavcodec     61. 19.101 / 61. 19.101
  libavformat    61.  7.100 / 61.  7.100
  libavdevice    61.  3.100 / 61.  3.100
  libavfilter    10.  4.100 / 10.  4.100
  libswscale      8.  3.100 /  8.  3.100
  libswresample   5.  3.100 /  5.  3.100
  libpostproc    58.  3.100 / 58.  3.100
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf61.7.100
  Duration: 00:00:20.00, start: 0.000000, bitrate: 112 kb/s
  Stream #0:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 300x300 [SAR 1:1 DAR 1:1], 78 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.19.101 libx264
  Stream #0:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 16000 Hz, mono, fltp, 75 kb/s (default)
      Metadata:
        handler_name    : SoundHandler
        vendor_id       : [0][0][0][0]
Input #1, mov,mp4,m4a,3gp,3g2,mj2, from 'test.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf61.7.100
  Duration: 00:00:20.00, start: 0.000000, bitrate: 112 kb/s
  Stream #1:0[0x1](und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 300x300 [SAR 1:1 DAR 1:1], 78 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.19.101 libx264
  Stream #1:1[0x2](und): Audio: aac (LC) (mp4a / 0x6134706D), 16000 Hz, mono, fltp, 75 kb/s (default)
      Metadata:
        handler_name    : SoundHandler
        vendor_id       : [0][0][0][0]
File 'movie-audio-delayed.mp4' already exists. Overwrite? [y/N] y
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
  Stream #1:1 -> #0:1 (copy)
Output #0, mp4, to 'movie-audio-delayed.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf61.7.100
  Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p(progressive), 300x300 [SAR 1:1 DAR 1:1], q=2-31, 78 kb/s, 30 fps, 30 tbr, 15360 tbn (default)
      Metadata:
        handler_name    : VideoHandler
        vendor_id       : [0][0][0][0]
        encoder         : Lavc61.19.101 libx264
  Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 16000 Hz, mono, fltp, 75 kb/s (default)
      Metadata:
        handler_name    : SoundHandler
        vendor_id       : [0][0][0][0]
Press [q] to stop, [?] for help
[out#0/mp4 @ 00000173c684eac0] video:192KiB audio:74KiB subtitle:0KiB other streams:0KiB global headers:0KiB muxing overhead: 3.740589%
frame=  600 fps=0.0 q=-1.0 Lsize=     275KiB time=00:00:17.93 bitrate= 125.7kbits/s speed=2e+03x
  • I watched the video and the audio started at frame 300 and frame background went from yellow to red.
  • I closed my eyes and played the video. I paused it when the audio started, and it showed frame 311, about 0.367 s off at 30 fps.
    This was within normal human response time from a bit of research.
    That was fun!

I made this request to ChatGPT to explore FFmpeg commands for audio and video sync:

Provide a list of what FFmpeg can do wrt to adding audio to video. < Click here to view!
ChatGPT said:

Here’s a structured list of what FFmpeg can do regarding adding or managing audio in videos. I’ve grouped it by functionality so it’s easier to see the possibilities:

1. Adding Audio to an Existing Video

  • Add a single audio track to a video without changing the video stream.
  • Add multiple audio tracks to a video (as separate selectable tracks in players).
  • Replace an existing audio track with a new one.
  • Append audio to the end of a video as a new track.
  • Offset audio relative to video (delay or advance).

2. Audio Timing & Synchronization

  • Apply -itsoffset to delay or advance an audio track.
  • Adjust audio start/end times to sync with specific video frames.
  • Trim audio to match video length (-ss and -t options).
  • Loop audio for shorter tracks to match longer videos.

3. Audio Processing / Mixing

  • Mix multiple audio tracks into a single track (amix filter).
  • Adjust volume per track or globally (volume filter).
  • Fade-in or fade-out audio (afade filter).
  • Pan or remix audio channels (pan filter).
  • Normalize or compress audio dynamically (loudnorm or acompressor).

4. Format Conversion

  • Convert audio to a different codec when adding (e.g., MP3 → AAC).
  • Change sample rate or channel layout (mono/stereo/surround).
  • Convert audio file type while adding (e.g., WAV → MP4-compatible AAC).

5. Advanced Effects

  • Add echo, reverb, or other effects (aecho, afir, etc.).
  • Apply audio filters for equalization or noise reduction.
  • Time-stretch audio without changing pitch (atempo).
  • Change audio pitch (asetrate + aresample).

6. Stream Mapping & Control

  • Map audio from one or more sources into specific tracks.
  • Exclude video or audio streams entirely when creating output.
  • Copy audio without re-encoding for lossless speed (-c:a copy).
  • Mix streams with precise control over start times and durations.

7. Testing & Debugging

  • Extract audio from a video (-vn) for editing or analysis.
  • Analyze audio streams for codec, sample rate, and channel layout.
  • Preview audio timing vs. video using short clips.

If you want, I can also make a concise “cheat sheet of FFmpeg commands for adding or syncing audio” — basically a ready-to-use reference for practically every scenario. This is super handy if you do frequent video/audio work.

Do you want me to make that cheat sheet?

I was inspired and already thinking of the possibilities!

This ChatGPT generated FFmpeg command-line instruction creates a movie (video) from the frames and audio file without using Movie Maker:

ffmpeg -framerate 30 -start_number 1 -i "D:\Users\GLV\Documents\P4\saveFrame_sync_1_0_0\data\%06d.png" -itsoffset 10 -i "D:\Users\GLV\Music\WAVs\bcch.wav" -map 0:v -map 1:a -c:v libx264 -pix_fmt yuv420p -c:a aac "movie-from-images.mp4"

I gave ChatGPT very clear instructions and it took a few passes with corrections along the way. It is learning and a useful tool for experienced users!

Future exploration:

  • Log the frameCount at key moments where audio should be added.
  • Use the logged data as markers for audio placement in the video.
  • Create an FFmpeg script to add the audio file(s).
  • Consider automating the process, which is achievable.
  • I am already developing a personal tool for this to support my creator projects.

I have used other tools (Camtasia for example) in the past to sync\add\modify audio but that goes beyond this topic.

If the tool you are using does not work or suit you find one that does or create your own.

I don’t have much more to add, other than… where there’s a will, there’s a way.
Seek and you may find a solution.
If you don’t find one, develop one.
Sometimes the best solutions come from within.
Enjoy the ride!

References:

:)

5

Even if it’s not solved yet for your needs, keep at it. Sometimes it is a compromise.

I’m returning to some creator projects, and this has inspired me to work with FFmpeg and Processing for custom solutions.

And I’m still working at it!

:)