GStreamer supports hardware decoding. It’s the bit after that that is currently the bottleneck, and possibly stopping hardware decoding being used. What’s actually required is OpenGL context sharing, and the ability to share that context between GStreamer and Processing. The GStreamer Java bindings don’t currently map that bit of GStreamer - it’s a high priority though.
So, PraxisLIVE uses GStreamer 1.x, which is possibly using GPU decoding but then needs to bring the pixel data into system memory to transfer to OpenGL / Processing libs. In PraxisLIVE this is done entirely without touching Java memory space, so it should be faster than current and beta Video library versions. But there is more work to do on all fronts! Once it’s in the GStreamer bindings then we can all start using it. Unfortunately, I’m not sure when I’m going to get a chance to look at it - happy to help anyone who wants to look at a PR for it!
Have you tried PraxisLIVE for comparison? Are you stuck with Intel GPU (for some reason seems slower at the above)?
EDIT - incidentally, having looked at your link - this should all function with the standard GStreamer PlayBin - should be no need to play around with their plugins.
@neilcsmith Thank you very much for the detailed answer.
Video playback does look a smoother in Praxis than in Processing,
but CPU usage is still high (since that also decodes the video).
I am not stuck with Intel GPU, can be nVidia, it’s just they can be more prevalent on small computers for installations (e.g. mac minis/Intel NUCs/etc.).
Regarding the plugins, I wasn’t sure if the GPU acceleration plugins ship with GStreamer precompiled.
I don’t know enough at this point about GStreamer, but if you could point me in the right direction I could have a stab at it.
Yes, it’s likely that even if the GPU decoding elements are available, they’re being dropped when there is a software based sink in use - just did some testing with GStreamer with full debug messaging and could see that happening on this machine. Still, at least in PraxisLIVE almost none of this should be happening on the same core as the renderer, so should be smoother, and GStreamer 1.x is better in that regard too.
Or Compute Stick! The fact that gst1-java-core exists at all is originally thanks to this project, and finding out that GStreamer 0.10 just couldn’t cut it on that hardware. So, yes, I get your use case and why. Unfortunately, discrete nVidia and AMD cards seem better at transferring pixel data - which is ironic in some respects!
@neilcsmith Thank you so much for the exhaustive information.
I’ll get started with JNA first. I remember fiddling a bit with JNI a while back, but JNA looks a bit more straight forward. @gohai’s GLVideo is a great starting point. It will take me a while to get my head around the OpenGL part, but should give it a go.
