How to hardware decode video in Processing?

Hi,

I’ve been using Processing for a while and mainly used the processing-video library to playback H.264 encoded videos.

I’ve noticed playback can be quite laggy for 1080p videos: it looks like decoding is done on the CPU.

I had a quick look at GStreamer Hardware Decoding
I’ll be honest, didn’t get very far trying to compile the good, the bad and the ugly plugins:

My question is a bit loaded to be honest, allow me me unpack:

  1. Is there a video playback library for Processing that can decode on the GPU ?
  2. If not, is there a Java video playback library with hardware support ?
  3. I’ve noticed @neilcsmith’s PraxisLive and gst-java bindings: do these work with GPU support ?

Any suggestions on smoothly playing back 1080p videos on the GPU, leaving the CPU free ?

Thank you,
George

1 Like

Hi!

You could look at GLVideo - https://github.com/gohai/processing-glvideo Might do what you need, but depends on your OS and actual requirements.

Yes, but it’s not that simple …

GStreamer supports hardware decoding. It’s the bit after that that is currently the bottleneck, and possibly stopping hardware decoding being used. What’s actually required is OpenGL context sharing, and the ability to share that context between GStreamer and Processing. The GStreamer Java bindings don’t currently map that bit of GStreamer - it’s a high priority though.

So, PraxisLIVE uses GStreamer 1.x, which is possibly using GPU decoding but then needs to bring the pixel data into system memory to transfer to OpenGL / Processing libs. In PraxisLIVE this is done entirely without touching Java memory space, so it should be faster than current and beta Video library versions. But there is more work to do on all fronts! Once it’s in the GStreamer bindings then we can all start using it. Unfortunately, I’m not sure when I’m going to get a chance to look at it - happy to help anyone who wants to look at a PR for it! :smile:

Have you tried PraxisLIVE for comparison? Are you stuck with Intel GPU (for some reason seems slower at the above)?

EDIT - incidentally, having looked at your link - this should all function with the standard GStreamer PlayBin - should be no need to play around with their plugins.

4 Likes

@neilcsmith Thank you very much for the detailed answer.

Video playback does look a smoother in Praxis than in Processing,
but CPU usage is still high (since that also decodes the video).

I am not stuck with Intel GPU, can be nVidia, it’s just they can be more prevalent on small computers for installations (e.g. mac minis/Intel NUCs/etc.).

Regarding the plugins, I wasn’t sure if the GPU acceleration plugins ship with GStreamer precompiled.

I don’t know enough at this point about GStreamer, but if you could point me in the right direction I could have a stab at it.

Yes, it’s likely that even if the GPU decoding elements are available, they’re being dropped when there is a software based sink in use - just did some testing with GStreamer with full debug messaging and could see that happening on this machine. Still, at least in PraxisLIVE almost none of this should be happening on the same core as the renderer, so should be smoother, and GStreamer 1.x is better in that regard too.

Or Compute Stick! :smile: The fact that gst1-java-core exists at all is originally thanks to this project, and finding out that GStreamer 0.10 just couldn’t cut it on that hardware. So, yes, I get your use case and why. Unfortunately, discrete nVidia and AMD cards seem better at transferring pixel data - which is ironic in some respects!

You will need to be comfortable with JNA and translating lowlevel C APIs. We don’t need to map everything at https://gstreamer.freedesktop.org/documentation/gl/index.html?gi-language=c but do need to cover all the bases for different OS that the GLVideo library handles in C at https://github.com/gohai/processing-glvideo/blob/master/src/native/impl.c

2 Likes

@neilcsmith Thank you so much for the exhaustive information.

I’ll get started with JNA first. I remember fiddling a bit with JNI a while back, but JNA looks a bit more straight forward. @gohai’s GLVideo is a great starting point. It will take me a while to get my head around the OpenGL part, but should give it a go.

1 Like