Video library to use new GStreamer bindings

These are posts discussing a potential upgrade to the Video library:

https://forum.processing.org/two/discussion/20383/video-1-0-1-read-a-frame-from-a-paused-video#latest

Help with text and video

https://github.com/processing/processing/wiki/Project-List#-video-priority

https://forum.processing.org/two/discussion/comment/120146/#Comment_120146

I decide to create this post to capture any future effort and to encourage future volunteers in case they are looking for a project to work on. I do not have enough experience with GStreamer and the requirements to bring the Video library up to speed to the latest GStreamer. @neilcsmith Do you mind commenting what are the skills required to work in this project? I understand from your previous post:

  1. There is Gstreamer 1.x
  2. The Java bindings for GStreamer 1.x which are based from your PraxisLIVE project
  3. The video library needs to be updated or be rewritten (I believe I linked the right repo from @gohai)

For anybody working in the project, do they need to become familiar with the GStreamer or could they work from the Java bindings?

Kf

Not quite @kfrajer - it’s actually really simple:

There is version 1.0.1 of the video library, released in 2015, which one would currently get when downloading the library through the Contribution Manger; this uses GStreamer version 0.10. Let’s call this the “current” version of the library. It works (unless it doesn’t).

Then, there are also more recent changes that were done to the main processing-video repository: these switch the library over to GStreamer version 1.x (using Neil’s bindings). Andres released a test version of the library containing these changes. Let’s call this the “next” version.

Now the issue with the “next” version, and that’s also why it was released as a “test” release only, was that it still has some features missing or not correctly working, compared to the “current” release. (The most complete list of these things is here.)

1 Like

@kfrajer the direct link to the GStreamer 1.x bindings is https://github.com/gstreamer-java/gst1-java-core They were updated for PraxisLIVE in 2015, but not based on it. They are a fork of the 0.10 bindings already used in the Video library, of which I was one of the maintainers. There were a number of failed attempts to write a new binding from scratch for GStreamer 1.x, including one that was a Processing GSoC project. I needed GStreamer 1.x support for a paid commission, hence the initial fork, but they are now used in a variety of free and commercial projects, and a lot of other people have now contributed. I’m still the only maintainer though - something I’m quite happy to share! :wink:

@gohai did much of the work required to update the Video library to work with the gst1-java-core bindings. Although, as he says, there’s a few things missing. One other thing missing seems to be direct texture upload when using the OpenGL renderers? This is in the original video library. It’s quite a performance hit if it’s not there.

In my opinion, now is the time to rethink the Video library rather than just update it. There are a fair few issues with the API as far as I’m concerned - threading issues which really hamper performance, the device listing and lack of ability to really use the power of the underlying libraries by passing in custom pipelines, lack of direct GPU support. I’m not saying they should all be fixed immediately, but they should be considered in breaking the API now and not trying to make it backwards compatible. I also find the idea that a video is a PImage rather than has a sequence of PImages (frames) odd conceptually and potentially makes things more difficult than they need to be.

Despite using and supporting Processing, PraxisLIVE has always had its own video library. I’ve recently updated the PraxisLIVE video components in order to allow them to be (live) recoded by the user. In doing this I initially tried to replicate the current Video library API, and gave up - there was no easy way to do that without making it perform far worse - therefore IMO the current API is part of the problem.

I’d be happy to look at backporting, adapting and improving the current PraxisLIVE code to work with vanilla Processing though - eg. https://github.com/praxis-live/praxis/blob/master/praxis.video.gstreamer/src/org/praxislive/video/gstreamer/components/GStreamerVideoPlayer.java Or we could work on something new that addresses the issues. If OTOH the plan is to keep updating the existing Video library as is, I can offer pointers from the bindings point of view, but am not going to put time into the library itself.

1 Like