I’m making my second Steam game which uses Processing to handle all the window creation and graphics side of things (specifically the P2D renderer which allows me to use shaders for some fun effects).
Modern games are expected to offer a few different window modes for the game:
- Windowed mode
- Borderless Fullscreen / Windowed Fullscreen
- Exclusive Fullscreen
In this post I just want to document how I’ve achieved that with Processing and a few of the stumbling blocks I ran into, so that if anyone else is wanting to add these options to their own Processing sketches they don’t have to go through the insane amount of Googling and trail & error that I did! (There is also some information about OBS recording issues that is independent of Processing, so if you’re having trouble with that, this could also be helpful!)
If you just want to see the code, that’s at the bottom!
Fullscreen
By default, Processing only lets you make your sketch fullscreen in the setup() function, which makes it a little tricky to toggle between being windowed and fullscreen while the game is running.
Thankfully, dzaima on GitHub already had a snippet of code that gets the native GLWindow object that the PSurface uses behind the scenes for controlling the window with the P2D renderer:
public void fullscreen() {
// This will only work with the OpenGL backed renderers, ie P2D and P3D
GLWindow glw = (GLWindow) surface.getNative();
glw.setFullscreen(true);
}
The GLWindow object is part of the NEWT library, which in turn is the native windowing toolkit inside of JOGL, which is the OpenGL bindings used by the P2D and P3D renderers. This will come up again later…
The above code can’t just be called during the draw() loop though since Processing is mid-render, so you have to delay these calls until you’re outside of draw(). I believe you can do registerMethod("post") in setup(), which will get Processing to call a method you have to define called post(). This will get run after the draw loop has finished, so you can call the above fullscreen() method inside that. I created my own post-draw function caller, which I needed it for other parts of my game, and call it there.
Borderless Fullscreen
This gave me a great starting point, all I had to do was figure out how to do a borderless fullscreen mode, which was a bit more tricky to work out but ended up being pretty trivial.
Borderless fullscreen (also called windowed fullscreen) is a sort of pseudo fullscreen where you make the window take up the entire area of the monitor without taking full control of the monitor, like what happens with traditional fullscreen. The main advantage of this is that you can swap between windows (eg via alt-tabbing) without getting flickering or delay.
Initially I was messing around trying to find out which monitor the window was in, getting it’s resolution and trying to set the window to match the size, but in the end it took just two method calls:
public void borderlessFullscreen() {
GLWindow glw = (GLWindow) surface.getNative();
glw.setUndecorated(true);
glw.setMaximized(true, true);
}
An undecorated window is one without the title bar or the minimise/maximise/close button cluster, just the content of the window is displayed. So this code removes the title bar then makes the window the full size of the monitor, just what we want!
The issue…
This was all working really well, in my settings menu you could toggle between the three different window modes and it would change on the fly with no problems for the ordinary gamer.
What wasn’t working is trying to record the game in either of the two fullscreen modes (windowed mode was working fine!) with the OBS screen recorder, which is sort of the standard when it comes to gaming content creators and streamers.
The game would play just fine, but the recording would either be just a black screen, or it would stutter and freeze really badly. This is a major bummer, since a big part of getting your game seen is having content creators make videos about your game. If they can’t record it, your game isn’t going to do so well!
This is where I started to lose my sanity a bit. I was looking into possible causes and fixes for months with no luck. JOGL isn’t very widely used anymore so there are almost no discussions about it that I could find, and even fewer (or none…) on the topic of screen recording issues.
I went down rabbit holes, looking at whether other Java rendering libraries were having these problems and if so, what their fixes were, but it was all coming up blank, so I had to accept this was just something that I wouldn’t be able to solve - a tough pill to swallow!
The solution
Just two days ago I was poking around the Processing discord and saw that the WebGPU renderer was being developed, which piqued my interest. I was of the belief that the issue was inherent to the PSurfaceJOGL backend, but I needed the shader capability so using it was the only real option, other than a major refactor to use a different (ie not-Processing) graphics library for my game. If I could just swap out the backend of Processing, then I wouldn’t need to go through that refactor, but still get the upside it (hopefully) working with OBS!
Sadly the WebGPU backend isn’t officially supported yet and I don’t really want to use something so fresh when what I’m after is stability. It did, however, reignite my desire to get to the bottom of all this.
After digging into a tangentially realted forum post, I came across a reference to WinSpy++, a little utility that lets you inspect and modify the window style properties of a Windows window (that’s a lot of windows..!). I decided to take a look at my game with it, and came across something interesting.
When the window was put into either of the two fullscreen modes, it had the style WS_POPUP - it was being flagged as a popup window!
If I toggled that off using WinSpy++, OBS was able to pick up the window and record it just fine!!
Unfortunately I can’t just ship my game with WinSpy++ and get people to toggle off the WS_POPUP flag when they want to record, I’ve got to find out how to do that automatically.
As I mentioned before, NEWT is the windowing toolkit used by JOGL, and after digging into the code a bit, I found that when you set the window to be undecorated (which happens explicitly by me when going into borderless fullscreen, but also seems to happen behind the scenes when going into traditional fullscreen), it adds the WS_POPUP flag, but this is inside the native C code, and far too deep for me to reach from Processing land.
And thus begins the horrible hack that saves the day.
In order to toggle off the WS_POPUP flag, you have to use the Win32 api which is a native C library, so to do that from Java, I used the JNA library, which helpfully already has bindings to the Winuser.h functions required to pull this off:
if(Platform.isWindows()) {
// remove WS_POPUP flag from the window to allow it to work with OBS screen recording
WinDef.HWND hwnd = new WinDef.HWND(new Pointer(glw.getWindowHandle()));
int flags = User32.INSTANCE.GetWindowLong(hwnd, User32.GWL_STYLE);
flags |= User32.WS_OVERLAPPED;
flags ^= User32.WS_POPUP;
User32.INSTANCE.SetWindowLong(hwnd, User32.GWL_STYLE, flags);
}
By using this code after setting the window to either of the fullscreen modes, I can turn off the WS_POPUP flag and turn on the WS_OVERLAPPED flag (I’m not entirely sure if that second flag being turned on is necessary, but WinSpy++ did it automatically when I toggled off WS_POPUP, so I figured I should too!). It preserves every other flag that’s already set.
Conclusion
For some reason OBS just doesn’t react well to windows with the WS_POPUP flag. By turning it off OBS is now able to capture the two full screen modes of my game which is absolutely fantastic for me, but it’s not quite perfect.
When I first change the window mode (or boot the game) OBS freezes up again, but as soon as you leave the game then return focus, it starts working flawlessly. This is such a massive improvement on what it was before (completely unusable), but if you’ve got any ideas on how to fix this last little hiccup, I would LOVE to know!
Sorry this has been such a ramble, I just thought I share all the little details so that it can hopefully be helpful to people regardless of how far into this process they got before finding this post!
Below is the full code I use in my game 
The final code:
public void applyWindowMode() {
// addPostDrawEvent just calls the lambda function after drawing
// has finished on the current frame, if you don't want to set
// something like this up I believe you can use
// `registerMethod("post")` instead.
addPostDrawEvent(() -> {
// This just gets an Enum, this line is specific to my game
// Basically just gets which option we want to set it to
Settings.WindowMode mode = Settings.windowMode();
switch (mode) {
case WINDOWED -> windowed();
case BORDERLESS_FULLSCREEN -> borderlessFullscreen();
case FULLSCREEN -> fullscreen();
}
windowResized();
});
}
public void windowed() {
GLWindow glw = (GLWindow) surface.getNative();
glw.setMaximized(false, false);
glw.setUndecorated(false);
glw.setFullscreen(false);
int x = Settings.windowX();
int y = Settings.windowY();
if(x != -1 && y != -1) {
glw.setPosition(x, y);
}
windowResize(Settings.windowW(), Settings.windowH());
}
public void borderlessFullscreen() {
GLWindow glw = (GLWindow) surface.getNative();
glw.setUndecorated(false);
glw.setFullscreen(false);
glw.setPosition(Settings.windowX(), Settings.windowY());
windowResize(Settings.windowW(), Settings.windowH());
glw.setUndecorated(true);
glw.setMaximized(true, true);
if(Platform.isWindows()) {
// remove WS_POPUP flag from the window to allow it to work with OBS screen recording
WinDef.HWND hwnd = new WinDef.HWND(new Pointer(glw.getWindowHandle()));
int flags = User32.INSTANCE.GetWindowLong(hwnd, User32.GWL_STYLE);
flags |= User32.WS_OVERLAPPED;
flags ^= User32.WS_POPUP;
User32.INSTANCE.SetWindowLong(hwnd, User32.GWL_STYLE, flags);
}
}
public void fullscreen() {
GLWindow glw = (GLWindow) surface.getNative();
glw.setMaximized(false, false);
glw.setUndecorated(false);
glw.setFullscreen(false);
glw.setPosition(Settings.windowX(), Settings.windowY());
windowResize(Settings.windowW(), Settings.windowH());
glw.setFullscreen(true);
if(Platform.isWindows()) {
// remove WS_POPUP flag from the window to allow it to work with OBS screen recording
WinDef.HWND hwnd = new WinDef.HWND(new Pointer(glw.getWindowHandle()));
int flags = User32.INSTANCE.GetWindowLong(hwnd, User32.GWL_STYLE);
flags |= User32.WS_OVERLAPPED;
flags ^= User32.WS_POPUP;
User32.INSTANCE.SetWindowLong(hwnd, User32.GWL_STYLE, flags);
}
}
You might notice that there’s a bit more going on in these functions then when I was showing them above, basically to ensure repeatability, I reset the window to a know state each time before applying the desired mode. The order that things happen matters in some cases!