Frame stepping with GStreamer StepEvent

Processing Libraries
1

Hey everyone.
I’m trying to use Processing for analyzing movement manually in videos. Unfortunately the video library doesn’t appear to have frame-stepping functionality built in, and the suggested example of calculating the frame time and seeking (or jumping) to it often skips frames, which doesn’t work for what I need.

I took some time to learn a little about the GStreamer backend and it seemed that I should be able to put the following in somewhere with a Listener and get it to framestep.

myMovie.playbin.sendEvent(new StepEvent(Format.BUFFERS, 1, 1, true, false));

However I find that in practice it comes up with a lot of unexpected (by me) behavior. In the following example I illustrate a couple of things.

import processing.core.*;
import java.util.EnumSet;
import org.freedesktop.gstreamer.event.SeekFlags;
import org.freedesktop.gstreamer.event.SeekType;
import processing.video.*;
import org.freedesktop.gstreamer.*;
import org.freedesktop.gstreamer.event.*;

Movie myMovie; 
float zoom;

void setup() {
  size(800,600);
  myMovie = new Movie(this, "Slow motion video of vertical jump with synchronized vertical force data.mp4");
  myMovie.play();
  myMovie.pause();
  myMovie.read();
  zoom = width*1.0 / myMovie.width;
  noLoop();
}

>void draw() {
    scale(zoom);
    myMovie.play();
    myMovie.pause();
    myMovie.read();
    background (128);  
    image(myMovie, 0, 0); 
}

void keyPressed() {
    if (key == CODED) {
      if (keyCode == LEFT) {
         myMovie.playbin.sendEvent(new StepEvent(Format.BUFFERS, 1, 1, true, false));
         redraw();
      } else if (keyCode == RIGHT) {
         myMovie.playbin.seek(1, Format.TIME, EnumSet.of(SeekFlags.FLUSH, SeekFlags.ACCURATE), SeekType.SET, myMovie.playbin.queryPosition(Format.TIME), SeekType.END, 0);
         myMovie.playbin.sendEvent(new StepEvent(Format.BUFFERS, 1, 1, true, false));
         redraw();
      }
    }
}
  • If I don´t play-pause before reading after sending the event, no change takes effect. Probably not surprising, but I´m wondering if there is a way around it.
  • In my example when I click the LEFT arrow, I send the step event without seeking beforehand. In this case it seems to step over about 5-6 frames. Probably something going on her I´m not seeing.
  • For the RIGHT arrow I added a seek event to the present time with rate 1. In this case I only step one frame, although sometimes it doesn’t load and I click twice and on the second press I jump forward two frames. What surprises me is that if I click RIGHT, it steps 1 frame, but then if I click LEFT afterwards it steps 6 frames.

My guess is there is a lot about the way the video library uses GStreamer that I am ignoring. I’d like to know if it is going to be possible to find a way to hack exact frame stepping in (and perhaps a hint at where to start), or if I am in way over my head. Any suggestions or pushes in the right direction will be much appreciated.

2

Hi! Welcome to the forum!

First of all please format the code:

1 Like
3

If you play-pause it’s never going to work as you want! Unfortunately no-one saw fit to finish implementing the preroll listener correctly to make this work :confounded: - processing-video/src/processing/video/Movie.java at main · processing/processing-video · GitHub

4

What would need to be added to this for it to work? Provided its not too complex we should be able to add the desired code and recompile the sound library.

Is it a case of adding some logic to handle framecount on pause and play?

5

It might be you could change the variables in the seek method.

private void seek(double rate, long start, long stop) {
    Gst.invokeLater(new Runnable() {
      public void run() {
        boolean res;
        if (stop == -1) {
          res = playbin.seek(rate, Format.TIME, EnumSet.of(SeekFlags.FLUSH, SeekFlags.ACCURATE), SeekType.SET, start, SeekType.NONE, stop);
        } else {
          res = playbin.seek(rate, Format.TIME, EnumSet.of(SeekFlags.FLUSH, SeekFlags.ACCURATE), SeekType.SET, start, SeekType.SET, stop);  
        }
        if (!res) {
          PGraphics.showWarning("Seek operation failed.");
        }
      }
    });    
  }

Format.time is likely the culprit here…

1 Like
6

Assuming you mean “video” library?! :smiley: It needs an implementation of the new preroll listener that actually does something, ie. the same or similar behaviour to the new sample listener. eg. if you look at the implementation in PraxisLIVE, where this actually works, the new preroll listener is identical - praxiscore/praxiscore-video-gstreamer/src/main/java/org/praxislive/video/gstreamer/components/PImageSink.java at master · praxis-live/praxiscore · GitHub

@ermaciu do you need to be using Processing and/or Processing Video for your task? Because the underlying GStreamer Java library sounds like a better bet.

1 Like
7

@micuat yes, good suggestion (and very easy I might add).

@neilcsmith makes sense that I can’t control what will happen if I do play-pause. I probably don’t need Processing to do what I want (mainly panning, zooming and registering clicks), but it did seem like the easiest option, until frame-stepping fell through. I might well turn to using the GStreamer library and a GUI toolkit. The SwingPlayer example probably gives me enough to send me in the right direction. Thanks for the guidance!

1 Like
8

@paulgoux I don’t really understand what you are suggesting I try. Something about trying different seek-formats? Sorry, I’m a bit green here. I’m happy to go in and play around with the library if I can figure out what you mean :slight_smile: Thanks!

9

Dont worry im pretty green myself. Its just ive finally learned to compile libraries with processing, and so was wondering how much work was actually needed for this. Is it a case of adding a few lines and recompiling the library or is it a lot more involved.