Are there any experts on the “Video for Processing 4” library who might be able to explain possible causes of undesirable flickering and freezing behaviour? (Processing 4, Mac OS 13*). At present I can’t provide a clean example of the problem because I need to isolate it a bit better – it’s part of a massive program. I will post an update if I can get a clean example of the issue.
Basically, the video playback is jittering. It looks like it’s hopping back and forth between adjacent frames. It also freezes sometimes when looping.
Basic description of the context: I’m having to rotate the video and display it on a larger canvas for projection mapping. The larger canvas is then drawn onto another canvas with other video layers. The video is H.264 mp4. It’s possible my graphics card isn’t keeping up, but it seems to be running at a fine frame rate. I will post more detail.
- Long story – running on a MacBook M1 but with Rosetta because I’m using Syphon.
Update: OK I think I get it. When my sketch is running below the framerate that the video file is encoded in (because there’s too much going on on my laptop), I see the jitter (it looks like it is flicking back and forth between the current position and one or two frames back). This makes sense, but then I don’t perfectly understand what the video library is doing under the hood. Should the library not be better behaved and just skip ahead frames when it is behind? Note I’m using the update method that utilises the movieEvent() callback. I suppose the movie.read() function copies pixel data into a PImage, and when we’re running too slow then the copying still needs to happen, so we get backed up?
OK this question has become a private monologue on a public forum. You’re welcome.
Are you using OpenGL? The OpenGL integration of the Video library is utterly broken. It’s telling our library (GStreamer bindings) to release the frame back to the GStreamer buffer pool before it’s uploaded to the texture. That could cause the effect you’re seeing. It’s also a cause of segfaults when changing videos.
I’m curious if you can replicate the problem playing a video in PraxisLIVE as that should confirm whether it’s this issue or something else in the tech stack.
Oh hello Neil (hey by the way, I’m in London for the next few months, until Christmas).
Well the library uses OpenGL by default, via gstreamer. I haven’t changed any settings and I don’t know the underlying details.
There’s this exception that gets thrown (out of my hands - not in a thread I have access to), but I don’t think it’s related (and only seems to happen at the start)…
java.lang.NullPointerException: Cannot invoke “java.util.LinkedList.add(Object)” because “this.usedBuffers” is null
at processing.opengl.Texture.copyBufferFromSource(Texture.java:827)
at jdk.internal.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at processing.video.Movie$NewSampleListener.newSample(Unknown Source)
at org.freedesktop.gstreamer.elements.AppSink$2.callback(AppSink.java:232)
at jdk.internal.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:568)
at com.sun.jna.CallbackReference$DefaultCallbackProxy.invokeCallback(CallbackReference.java:585)
at com.sun.jna.CallbackReference$DefaultCallbackProxy.callback(CallbackReference.java:616)
So this is a big Processing sketch using controlP5, syphon and Video for Processing. Can I run that in Praxis / port it easily? If so I’d love to try.
But as I mentioned, I’m now quite sure that the glitching only happens when the framerate falls below the framerate of the video being played. I’d still imagine a good video library would handle that without jittering, but I don’t know what the underlying process is. I’ll own this one… my job to keep the framerate up.
As in, are you using P2D or P3D renderers?
Probably not easily.
The GStreamer bindings (gst1-java-core) used in Processing Video are a project of mine. PraxisLIVE uses them directly, and differently. The Processing Video AppSink code was adapted from PraxisLIVE’s but the way it passes the video to texture in Processing is different (and buggy!).
The idea of trying to replicate with video playback in PraxisLIVE (possibly with an artificially low framerate) would be to see whether the problem is in gst1-java-core or native code, or in Processing Video.
I’ll message you! 
I’m on top of the flickering but I’m also getting a memory explosion. Are you aware of any memory leak issues?
Not specifically with the GStreamer bindings or Processing Video. A few in Processing 3 I know about.
Your best bet is to start with running something like VisualVM against the project to see if anything particular looks suspicious.
One thing to be aware of is that GStreamer memory is off of the Java heap. That means that although the GStreamer bindings clear up native memory when the garbage collector runs, the size of the GStreamer memory usage doesn’t trigger the garbage collector. Hopefully that makes sense! You could try a test calling System.gc() every frame and see if that has any effect on your memory leak.
It’s possible to explicitly clear native memory in the bindings, but that’s not often necessary - video buffers are the big thing, but they’re already released explicitly in the Video library.
You could try a test calling System.gc() every frame and see if that has any effect on your memory leak.
Yeah that didn’t seem to change anything. And I assume if I instantiate a new Movie – myMov = new Movie(etc.) – then the old movie data is freed. I don’t see any explicit “dispose” method call. I’ll try VisualVM. Just like old times! Also yes I’m using P3D.