I’m working on a large video installation project, which is basically finished and ready. It loads in dozens of very small videos (500x500 movs), and loops them. The program is designed to run for up to an hour at a time at least and does so perfectly using the default renderer. The whole thing works completely perfectly using the default renderer and the standard video library with no errors.
However. Using the default renderer everything looks poor quality, and so in order to get that lovely antialiasing goodness I’ve switched to P2D.
In this mode, I get a huge number of errors relating to Gstreamer. They look like this:
Processing video library using GStreamer 1.16.2
Starting capture...
Nov 22, 2021 4:51:40 PM com.sun.jna.Native$1 uncaughtException
WARNING: JNA: Callback org.freedesktop.gstreamer.elements.AppSink$2@65e475de threw the following exception
java.lang.IllegalStateException: Native object has been disposed
at org.freedesktop.gstreamer.glib.NativeObject.getRawPointer(NativeObject.java:125)
at org.freedesktop.gstreamer.glib.Natives.getRawPointer(Natives.java:175)
at org.freedesktop.gstreamer.lowlevel.GTypeMapper$2.toNative(GTypeMapper.java:73)
at com.sun.jna.Function.convertArgument(Function.java:521)
at com.sun.jna.Function.invoke(Function.java:345)
at com.sun.jna.Library$Handler.invoke(Library.java:265)
at com.sun.proxy.$Proxy21.gst_buffer_unmap(Unknown Source)
at org.freedesktop.gstreamer.Buffer.unmap(Buffer.java:116)
at processing.video.Movie$NewSampleListener.newSample(Unknown Source)
at org.freedesktop.gstreamer.elements.AppSink$2.callback(AppSink.java:232)
at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.sun.jna.CallbackReference$DefaultCallbackProxy.invokeCallback(CallbackReference.java:520)
at com.sun.jna.CallbackReference$DefaultCallbackProxy.callback(CallbackReference.java:551)
Interestingly, the application does still work reasonably well for a while: the errors come in seemingly randomly, sometimes right away and sometimes well into the runtime of the program. Eventually the program crashes.
Is anyone familiar with this stuff? I’ve got no idea what the problem is, and I am tearing my hair out - this is a finished project that I need to install soon! Is it possible to update the internal GStreamer to a new version? I note Processing says it’s on 1.16.2 and I know there’s a new stable version available as a standalone thing. I couldn’t find much info on how it works within Processing, though. Or, is there anything I can do here to isolate the error and stabilise my project?
I’m building on Windows 10, if that helps. Thanks in advance!
On the off chance, I increased the available memory. The above error stops, but I get a new one, also very sporadic in the same manner. This one causes the whole app to instantly hang (where the original error appeared numerous times before killing the app):
java.lang.NullPointerException
at java.util.LinkedList.unlink(LinkedList.java:211)
at java.util.LinkedList.remove(LinkedList.java:526)
at processing.opengl.Texture.bufferUpdate(Texture.java:899)
at processing.opengl.PGraphicsOpenGL.getTexture(PGraphicsOpenGL.java:6396)
at processing.opengl.PGraphicsOpenGL$TexCache.getTexture(PGraphicsOpenGL.java:7439)
at processing.opengl.PGraphicsOpenGL.flushPolys(PGraphicsOpenGL.java:2374)
at processing.opengl.PGraphicsOpenGL.flush(PGraphicsOpenGL.java:2315)
at processing.opengl.PGraphicsOpenGL.endDraw(PGraphicsOpenGL.java:1480)
at processing.core.PApplet.handleDraw(PApplet.java:2500)
NullPointerException
at processing.opengl.PSurfaceJOGL$DrawListener.display(PSurfaceJOGL.java:866)
at jogamp.opengl.GLDrawableHelper.displayImpl(GLDrawableHelper.java:692)
at jogamp.opengl.GLDrawableHelper.display(GLDrawableHelper.java:674)
at jogamp.opengl.GLAutoDrawableBase$2.run(GLAutoDrawableBase.java:443)
at jogamp.opengl.GLDrawableHelper.invokeGLImpl(GLDrawableHelper.java:1293)
at jogamp.opengl.GLDrawableHelper.invokeGL(GLDrawableHelper.java:1147)
at com.jogamp.newt.opengl.GLWindow.display(GLWindow.java:759)
at com.jogamp.opengl.util.AWTAnimatorImpl.display(AWTAnimatorImpl.java:81)
at com.jogamp.opengl.util.AnimatorBase.display(AnimatorBase.java:452)
at com.jogamp.opengl.util.FPSAnimator$MainTask.run(FPSAnimator.java:178)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
This sort of smells like a memory issue to me, but I’m not sure how else to get around it. I’ve increased the available memory in Processing, and wildly reduced my videos in size. That being said, I did also notice the error occurs even if I strip back the complexity of the app, it still errors at roughly the same rate.
Sorry, you’re out of luck! This is a known issue with the video library - it’s not been properly updated to use our GStreamer 1.x bindings. The frame data that wraps the buffer from GStreamer is released back to GStreamer before it’s passed in to OpenGL. In simple cases this works as there’s a buffer pool, but if you do anything that clears the pool (like stopping / switching pipelines) you get exceptions, or even segfaults.
If you want something that works correctly with Processing OpenGL and GStreamer you could look at PraxisLIVE. Which also handles multiple videos in a better way (there’s no need to dispose of the pipelines). Or you’ll have to fix the Video library.
Hey Neil! This is so frustrating, it’s literally a finished and fully working project - we tested it throughout using the default renderer, but it looks grainy and bad on the screen it’s being designed for (which is about twenty feet wide, so we didn’t catch it until we tested). Obviously the P2D switch would solve this, and it allllllmost gets there sometimes, but isn’t stable enough to deploy.
I’ve definitely not got time to re-implement anything or port it. I wonder if there’s anything I can do to take the edge off the jaggies using the default renderer…
Yes, it’s a frustrating bug that’s been there a long time now. Surprised it’s that jaggy with default renderer though. Are you scaling the videos much?
Yeah, it’s a sort of generated image grid that grows in complexity over time, so videos are all sorts of sizes and change each day. It looks okay-ish on a screen, but curved edges and circles look a bit poor on the big installation screen. I’m toying with a different approach now, to see if I can be a bit more precise on pixels to use the limitations of it to my advantage perhaps. Tricky one.
My other slightly mad idea was to run the sketch using the default renderer at an insanely high resolution, and then shrink the output back down - I’m dumping raw frames to a directory and stitching them back together to create smooth 60fps video (can’t beat the classics, right), so perhaps I can do it that way.
Are you loading and looping lots of videos simultaneously at the start? Or… are you playing a single video on loop for bit then switching to another and so on? Because if the latter then there is a workaround I have used. The switch of the video media is the culprit – it never releases memory and after switching many times the thing grinds to a halt and freezes eventually. I got around by editing the videos into one long movie (and making a note of edit times) and writing some code to jump to different parts of the movie and loop it – not frame-accurate but worked and kept it alive indefinitely as no new memory was being allocated. I had to manually manage the loop looking out for start/end timings and jumping back to the loop start for each sequence – I put the edit timings into a CSV and read in at start. Hope this helps!
This isn’t the only problem with P2D / P3D, as described above. You can access the playbin field in the video class directly and change the media on it instead. You can also get more accurate seeking behaviour using it. Unfortunately, changing media on the playbin may still cause the crash due to the buffer pool releasing memory.
It’s always felt wrong to me that video support in Processing treats video as an image, as opposed to having a player that is a source of images.