Screen capturer

Processing Coding Questions
1

Hello. I would like to discuss an issue related to a screen capturing program. The following code is capable of screen recording and overlaying the mouse cursor:

import java.awt.Robot;
import java.awt.image.BufferedImage;
import java.awt.Rectangle;
import java.awt.MouseInfo;
import java.awt.Point;
import java.awt.Toolkit;

Robot robot;
Point mouse;
Rectangle r;
BufferedImage img1;
PImage img2;

void setup() {

     size(1024, 768); // simulator resolution

     try { robot = new Robot(); }
     catch (Exception e) { println(e.getMessage()); }
     r = new Rectangle(Toolkit.getDefaultToolkit().getScreenSize());
     img1 = new BufferedImage(1024, 768, BufferedImage.TYPE_INT_RGB);

     stroke(255, 0, 0, 75);
     strokeWeight(6);
     fill(255, 255, 255, 100);
     frameRate(30);

}//setup

void draw() {

     background(0);

     img1 = robot.createScreenCapture(r);
     img2 = new PImage(img1);
     image(img2, 0, 0);

     mouse = MouseInfo.getPointerInfo().getLocation();
     ellipse(mouse.x, mouse.y, 50, 50);

     saveFrame("data/screen-######.tif");

}//draw

The program sends about 30 screenshots per second to the data folder. After recording, I use the Movie Maker tool to put all frames together as a video clip.
I have successfully tested the code by MS Flight Simulator X running in full screen mode. I also tried to get screenshots from Laminar Research X-Plane 9. In this case however, the program stored black frames only.
I presume the issue has something to do with the renderer. The MSFS X uses DirectX whilst the X-Plane uses OpenGL.
I would like to ask if it is possible to develop a workaround code in Processing which is capable of capturing screenshots in OpenGL rendering mode. Frankly, I don’t know where I should start from. Every advise is appreciated. Thank you in advance!

2

What operating system and computer hardware are you running this on?

You might find that using custom screen recording software is better suited to your purposes – screen recording is possible in Java / Processing, but if screen recording is your only goal then there are cheap and powerful options that will create the video clips for you directly in a number of formats – along with audio, if you wish.

3

Hello. I use Windows 10 x64 running on Core i7, 8 Gb RAM.
This is not a trivial project. I’d like to lay mouse and an eye-tracker gaze data over the captured video. This includes creating a heat map. I know, it sounds like a torture but any available software capable of doing that is VERY expensive. I thought Processing might offer a workaround.
Thanks for reply anyway.

4

possibly relevant old post mentions other screencap options in Java than Robot – although it is old:

Are you drawing mouse and eye-tracking indicators directly onto the frames before saving them, or are you saving the time/coordinates series as data and then altering the frames later?

5

Theoretically this is achievable with GStreamer, so would be good if the Video library supported the screen as a capture source. Unfortunately it makes it very difficult to change sources, and the relevant plugin might not be included and it’s also difficult to switch to a system installed GStreamer. To be honest, I’ve not tried screen capture with GStreamer on Windows as yet - works great on Linux though!