Giant LED Display Memory Leak?

Hi,

Using instructions online I built a giant LED video screen. It’s very cool, but I ran into a big problem: the screen runs for about ~8-12 minutes (depending on the video) then freezes. I can reset and re-run as many times as I want, to the same issue. My guess is that it’s a memory leak, but if anyone could take a look and see if they spot anything I’d be eternally grateful.

The wall uses a Teensy 3.2 microcontroller and the OctoWS2811 adapter. You must run an Arduino sketch first, then run a Processing program, which takes a part of your screen and “shares” it onto the LED wall. The code creator very graciously worked with me for a while before he had to move on. The sketch can be found on GitHub here, while the processing program can be found here.

There are a few errors shown in the Processing IDE:

  • Line 75: Type String[] of the last argument to method println(Object…) doesn’t match the varag parameter type. Cast Object[] to confirm the non-varags invocation, or pass individual arguments to type Object for a varags invocation.
  • Line 127: The value of the parameter layout is not used
  • Line 128: The value of the local variable “mask” is not used
  • Line 171: Null comparison always yields false: The variable line cannot be null at this location
  • Line 171: Dead code

I’m familiar with other languages, but a rote beginner with Processing so I’ll study anything someone suggests.

You can see the original instructions with tons of details and pictures here: https://makezine.com/projects/construct-giant-led-video-screen/ (I could only put two links in the first post due to forum rules).

The issue is at line 104 in the screenCapture.pde file:

img = new PImage(robot.createScreenCapture(new Rectangle(150, 150, width, height)));

This line creates a new PImage in every draw() iteration, which devours memory. This line needs to be rewritten to avoid the new keyword. There’s a small chance it could work by moving that line to the setup() method, but I suspect it doesn’t.

robot.createCreenCapture() returns a Buffered Image. The constructor of a PImage accepts a java.awt.BufferedImage to create a new instance of a PImage, but you can’t use a BufferedImage to alter an existing PImage object. Therefore, you might have to rewrite the sketch to use a BufferedImage (or a java.awt.Image) instead of a PImage. Or find a way to convert a BufferedImage into a PImage without having to create a new instance. I guess you could manually copy all the pixels over and be done with it.

The errors you listed, are they errors (red) or warnings (yellow)? If they are actual errors the code should not compile. If it does, there is a bug somewhere in the real time code checker and it should be reported.

Also, unrelated, how does this work when it works? I tried something similar (homebrewn with other leds) but the serial interface was not fast enough for my purposes. Can you really achieve near video framerate for 8 pins connected to 240 pixels? If so I need to investigate further. The Serial buffer of the Arduinos I used was too small to send a complete frame and splicing frames up to send them in bunches turned out to be disastrous. Maybe the Teensy doesn’t suffer from such limitations?

1 Like

Thanks! I’ll have to investigate. Rewriting this whole thing seems… way out of my depth, but hopefully I’m being pessimistic.

Regarding “errors” you’re correct, they’re all yellow warnings.

Regarding how it works, when it’s running, it runs almost flawlessly. I’ve seen it skip a frame or lag for a second, but it’s inconsistent and I think due to the coding issue since it’s very uncommon and I can’t reproduce it. The Teensy is much more capable than an Arduino, and the OctoWS2811 is… something else. I’d be lying if I said I really understood how the pin controls are coded, especially how they interact with the Octo library, hence why I’m pessimistic about rewriting the whole sketch. There’s another Processing sketch to run a movie file. Not ideal, but worth testing so that it’s at least operable.

Here is the Octo adapter: https://www.pjrc.com/store/octo28_adaptor.html
And the Octo library: https://www.pjrc.com/teensy/td_libs_OctoWS2811.html

Hmmmmm, they cost almost the same as a Raspberry Pi. I have some of those lying around and I want to see how well the GPIO pins hold out. People have warned against using GPIO on a Pi for led strips due to the strict timing constraints but we’ll see how that works. Still needs the 3.3V to 5V DC buffer though. Also, I’m using SK6812 LEDs instead of WS2812, which are similar but not quite (RGBW instead of RGB), so I’ll need to adapt the libraries I’m using anyway.

There’s no need to completely rewrite the sketch. Just adapt it like this:

// Modified by dan@marginallyclever.com 2015-07-03
// Mod CMB 2018-05-20: fixed memory leak issue

/*  OctoWS2811 movie2serial.pde - Transmit video data to 1 or more
 Teensy 3.0 boards running OctoWS2811 VideoDisplay.ino
 http://www.pjrc.com/teensy/td_libs_OctoWS2811.html
 Copyright (c) 2013 Paul Stoffregen, PJRC.COM, LLC
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
 in the Software without restriction, including without limitation the rights
 to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
 copies of the Software, and to permit persons to whom the Software is
 furnished to do so, subject to the following conditions:
 
 The above copyright notice and this permission notice shall be included in
 all copies or substantial portions of the Software.
 
 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
 THE SOFTWARE.
 */

// To configure this program, edit the following sections:
//
//  1: change myMovie to open a video file of your choice    ;-)
//
//  2: edit the serialConfigure() lines in setup() for your
//     serial device names (Mac, Linux) or COM ports (Windows)
//
//  3: if your LED strips have unusual color configuration,
//     edit colorWiring().  Nearly all strips have GRB wiring,
//     so normally you can leave this as-is.
//
//  4: if playing 50 or 60 Hz progressive video (or faster),
//     edit framerate in movieEvent().

import processing.video.*;
import processing.serial.*;
import java.awt.Rectangle;

// https://github.com/onformative/ScreenCapturer/issues/2
import java.awt.Robot;
import java.awt.Rectangle;
import java.awt.AWTException;
import java.awt.image.BufferedImage;

Robot robot;

final int SCREEN_WIDTH = 60;
final int SCREEN_HEIGHT = 36;

int numPorts=0;  // the number of serial ports in use
int maxPorts=24; // maximum number of serial ports

Serial[] ledSerial = new Serial[maxPorts];     // each port's actual Serial port
Rectangle[] ledArea = new Rectangle[maxPorts]; // the area of the movie each port gets, in % (0-100)
boolean[] ledLayout = new boolean[maxPorts];   // layout of rows, true = even is left->right
PImage[] ledImage = new PImage[maxPorts];      // image sent to each port
int errorCount=0;
float framerate=0;
int maxW, maxH;
PGraphics img;// = new PImage();
BufferedImage bImg;
//PImage cpy = new PImage();
byte[] ledData;


void setup() {
  size(640, 360);  // create the window
  surface.setLocation(20,20);
  maxW=maxH=0;
  String[] list = Serial.list();
  delay(20);
  println("Serial Ports List:");
  println(list);

  // either
  //serialConfigure("/dev/tty.usbmodem912641");  // change these to your port names
  // or let it try to choose automatically
  serialConfigure(list[list.length-1]);

  if (errorCount > 0) exit();

  int size=(maxW * maxH * 3);
  ledData =  new byte[size+3];
  ledData[size+0]=0;
  ledData[size+1]=0;
  ledData[size+2]=0;


  img=createGraphics(width, height);
  img.beginDraw();

  
  try {
    robot = new Robot();
  }
  catch (AWTException e) {
    println(e);
  }
}


// runs for each new frame of movie data
void movieEvent() {
  // read the movie's next frame
  bImg = robot.createScreenCapture(new Rectangle(150, 150, width, height));

  img.loadPixels();
  for(int i = 0; i < bImg.getWidth(); i++)
  {
    for(int j = 0; j < bImg.getHeight(); j++)
    {
      img.pixels[i+width*j]= bImg.getRGB(i, j);
      //println(i,j,binary(img.get(i,j)));
    }
  }
  img.updatePixels();


  //cpy.copy(img,0,0,img.width,img.height,0,0,img.width,img.height);

  for (int i=0; i < numPorts; i++) {    
    // copy a portion of the movie's image to the LED image
    int xoffset = percentage(img.width, ledArea[i].x);
    int yoffset = percentage(img.height, ledArea[i].y);
    int xwidth =  percentage(img.width, ledArea[i].width);
    int yheight = percentage(img.height, ledArea[i].height);
    ledImage[i].copy(img, xoffset, yoffset, xwidth, yheight, 
      0, 0, ledImage[i].width, ledImage[i].height);
    // convert the LED image to raw data
    image2data(ledImage[i], ledData, ledLayout[i]);
    // send the raw data to the LEDs  :-)
    ledSerial[i].write(ledData);
  }
}


// image2data converts an image to OctoWS2811's raw data format.
// The number of vertical pixels in the image must be a multiple
// of 8.  The data array must be the proper size for the image.
void image2data(PImage image, byte[] data, boolean layout) {
  int offset=0, y, mask, pixel;
  int size = image.height * image.width;
  int r, g, b;

  for (y = 0; y < size; y++) {
    pixel = image.pixels[y];
    r = ( pixel & 0xFF0000 ) >> 16; 
    g = ( pixel & 0x00FF00 ) >>  8; 
    b = ( pixel & 0x0000FF );

    if ( r==0 ) r=1;
    if ( g==0 ) g=1;
    if ( b==0 ) b=1;

    data[offset++] = (byte)(r);
    data[offset++] = (byte)(g);
    data[offset++] = (byte)(b);
  }
}


// ask a Teensy board for its LED configuration, and set up the info for it.
void serialConfigure(String portName) {
  if (numPorts >= maxPorts) {
    println("too many serial ports, please increase maxPorts");
    errorCount++;
    return;
  }
  try {
    ledSerial[numPorts] = new Serial(this, portName);
    if (ledSerial[numPorts] == null) throw new NullPointerException();
    ledSerial[numPorts].write('?');
  } 
  catch (Throwable e) {
    println("Serial port " + portName + " does not exist or is non-functional");
    errorCount++;
    return;
  }
  delay(250);

  String line;
  line = ""+SCREEN_WIDTH+","+SCREEN_HEIGHT+",0,0,0,0,0,100,100,0,0,0";
  //line = ledSerial[numPorts].readStringUntil('\n');

  if (line == null) {
    println("Serial port " + portName + " is not responding.");
    println("Is it really a Teensy 3.0 running VideoDisplay?");
    errorCount++;
    return;
  }
  //*/
  String param[] = line.split(",");
  if (param.length != 12) {
    println("Error: port " + portName + " did not respond to LED config query");
    errorCount++;
    return;
  }
  int w = Integer.parseInt(param[0]);
  int h = Integer.parseInt(param[1]);
  // only store the info and increase numPorts if Teensy responds properly
  ledImage[numPorts] = new PImage(w, h, RGB);
  ledArea[numPorts] = new Rectangle(Integer.parseInt(param[5]), Integer.parseInt(param[6]), 
    Integer.parseInt(param[7]), Integer.parseInt(param[8]));
  ledLayout[numPorts] = (Integer.parseInt(param[5]) == 0);
  numPorts++;
  if (maxW < w) maxW = w;
  if (maxH < h) maxH = h;
}


// draw runs every time the screen is redrawn - show the movie...
void draw() {
  movieEvent();

  // show the original video
  image(img, 0, 80, 640, 360-80);
  //image(img, 0,80,640,415);
  //image(ledImage[0], 0,80,640,415);

  // then try to show what was most recently sent to the LEDs
  // by displaying all the images for each port.
  for (int i=0; i < numPorts; i++) {
    // compute the intended size of the entire LED array
    int xsize = percentageInverse(ledImage[i].width, ledArea[i].width);
    int ysize = percentageInverse(ledImage[i].height, ledArea[i].height);
    // computer this image's position within it
    int xloc =  percentage(xsize, ledArea[i].x);
    int yloc =  percentage(ysize, ledArea[i].y);
    // show what should appear on the LEDs
    image(ledImage[i], 240 - xsize / 2 + xloc, 10 + yloc);
  }
}


// scale a number by a percentage, from 0 to 100
int percentage(int num, int percent) {
  double mult = percentageFloat(percent);
  double output = num * mult;
  return (int)output;
}


// scale a number by the inverse of a percentage, from 0 to 100
int percentageInverse(int num, int percent) {
  double div = percentageFloat(percent);
  double output = num / div;
  return (int)output;
}


// convert an integer from 0 to 100 to a float percentage
// from 0.0 to 1.0.  Special cases for 1/3, 1/6, 1/7, etc
// are handled automatically to fix integer rounding.
double percentageFloat(int percent) {
  if (percent == 33) return 1.0 / 3.0;
  if (percent == 17) return 1.0 / 6.0;
  if (percent == 14) return 1.0 / 7.0;
  if (percent == 13) return 1.0 / 8.0;
  if (percent == 11) return 1.0 / 9.0;
  if (percent ==  9) return 1.0 / 11.0;
  if (percent ==  8) return 1.0 / 12.0;
  return (double)percent / 100.0;
}

At line 111 and on, I retrieve the BufferedImage from the Robot instance and manually copy it over into the img pixel buffer. Both the BufferedImage (bImg) and the PImage img (that is now a PGraphics, a subclass of PImage, so I can draw to it) are retained in memory. It’s not super efficient but lots better than having the memory leak. There’s probably a way to retrieve the pixel array from the BufferedImage and do an efficient array copy but the documentation isn’t very clear.
On my system, the sketch is very slow though. One frame every five seconds. If I comment out serialConfigure() in the setup(), the frame capture is reasonably fast and images are displayed on screen in real time. Though, I plugged in a faulty Arduino to avoid a nullpointer error when it loads all serial objects. Poor thing fell from a truss and now only sends back garbage so that could be the cause of the slowness.

Hi,

Thanks so much. Unfortunately, tests ran for about 3 minutes, then stopped. I didn’t notice lag until just before it stopped. If I comment out setup()'s serialConfigure(), the code doesn’t run.

Regarding the Octo, if you’re curious while you do your Pi testing, I highly recommend checking out the Teensy forum. Doing a targeted Google search I see a few people working with SK6812 strips.

What happens when the code stops? Does Processing crash, no more output to the LEDs, something else? Is there a memory leak or anything else that points to problems?

If you comment out serialConfigure(), does the sketch not start up or is there no output to the LEDs?

Oh ok, well things just got weird. To double check your answers, I ran the Github code and it ran for 20 minutes on a file that normally runs 8 mins consistently. Dozens of tests consistent. This is the longest it’s run since it ran 2 hrs on one of the first tests I ran like 8 months ago, then never did it again.

Now I’m running your code and I got up to 22 minutes.

To answer your questions, for both the Github code and your code, the freeze is the same:

  • The LEDs freeze on a frame while still lit. I’ve left it running for 5-10 mins while frozen without it shutting down.
  • The video will continue to run without issue.
  • The Processing code still seems to be running; when I run the code a preview window opens that shows anything that will be shown on the LED and it still works. Hitting “Stop” on Processing closes that window without delay.
  • Sometimes the LED on the Teensy itself will shut down, but not always. The Teensy physical reset button always works, since my procedure is Stop Processing > Reset Button > Reupload Arduino code > Run Processing.

On your code, if I comment out serialConfigure(), the sketch does not run.

After that 22 min test of your code it went 7 mins > 29 mins > 7:30 mins > 4 mins > 6 mins.

This was all running the same video file that had consistently been putting out 8-9 mins with the Github file.

Sounds like there is a problem with the communication between the computer and the Teensy. Maybe the serial connection is overloaded and gets stuck? The memory leak might even be beneficial as it slows down the sketch, causing a slower connection speed… Just speculating here.

I can’t help you with this issue without access to the hardware. Try to find a way to debug the program, maybe using println() statements. See if everything is still running. Maybe you can rewrite the sketch so it reinitialises the serial connection every two minutes as a workaround?

Thanks for the leads, I’ll investigate. I also plan to test how long it can run using the pre-programmed demos, and the FastLED library. Hopefully some clarity will come from it.

Hey Brick, did you ever figure this out? I am working on a similar display and will let you know if I get the same issue.

Hey BuffaloFan32, unfortunately I never solved it. As much as I’d love for someone else to solve it for me, I hope you don’t run into similar issues. Would also love to see the final result.

Hi Brick, I got your code to work on my array of about 5,600 LEDs and an old laptop. It ran for about an hour with no freezing while I played a video on the screen that was being captured. The only problem I noticed is that the projection was very jerky. In other words the frame rate did not seem to be consistent and the resulting video would appear to skip sometimes. I do not have this problem when I run Paul’s video2serial script on the same display.

That’s amazing, I’m glad to hear it worked. Thanks for letting me know.

If you have time could you answer a two quick questions?

  • Did you use Paul’s basic instruction set to build it?
  • When you say “video2serial” do you mean Paul’s “movie2serial” that plays a video file from a specified path? Either way, this is both super interesting and incredibly frustrating to hear.

Yes, I used Paul’s instructions and I did mean movie2serial - sorry.

Hi, I’m about to make something similar but can’t seem to find the instructions from Paul.

Do you have a link?

Cheers

Phil