Pixels "stutter" when drawing (now with included data)

tribblesontoast · June 27, 2019, 1:20am

Edit: Data can be downloaded from here, forgot it again.

Hi,
Sorry for the complete rebuild, but what I wrote was garbage, i’ll try to simplify.

I’m visualising movie color data from three places on the screen, I have the data and drawing working.

The problem is that the data from the center of the screen is, for lack of a better term, stuttering, as shown below.

_M (stutter):
stutter

_L:
nostutter

_R:
nostutter_R

The program calculates how many pixels it needs from each line of the file in order to fit on the screen, gets them and moves on to the next line. This is always broken for the center of the screen, not always for the _L or _R data.

My thought is that it could be caused by the line wrapping across the edge of the PImage as if you hardcode the pxls value, the stuttering only happens at values above 10, but I could be wrong about the line wrapping as the _L and _R has minimal “stuttering”.

Is this something i’m doing wrong in the code?

Code example:

IntList colors;
PImage g;
float dispW, dispH;
int pxls = 0, numFrames = 0;

void setup() { 
  fullScreen();
  colorMode(255, 255, 255, RGB);
  colors = new IntList();
  g = createImage(width, height, RGB);

  String[] t = loadStrings("png_ColourDataHSB_R.txt");
  numFrames = t.length;

  init();

  for (int k = 0; k < t.length; k++) {
    String temp = t[k];
    String[] frames = split(temp, ";");

    for (int i = 1; i < pxls; i++) colors.append(int(frames[i])); // starts from 1 as first line is ;
  }

  for (int i = 0; i < colors.size(); i++) {
    g.loadPixels(); 
    g.pixels[i] = colors.get(i);
    g.updatePixels();
  }
  colors.clear();
}

void draw() {
  image(g, 0, 0);
}

void init() {
  dispW = width;
  dispH = height;

  pxls = int(ceil((dispW*dispH) / numFrames));
  //pxls = 19;
  println(pxls);
}

Or could there be a problem in the capture for the _M data itself? (not a complete working example)

int start = (i.pixels.length/2)-(pxls/2);
for (int x = start; x < start+pxls; x++) {
  // The functions hue(), sat(), and brightness() pull out the 3 color components from a pixel.
  float h = hue(i.pixels[x]);
  float s = saturation(i.pixels[x]);
  float b = brightness(i.pixels[x]);

jeremydouglass · July 3, 2019, 4:28am

Were you able to resolve this problem?

You’ve provided 24MB text files full of data, but the most likely thing is that your data was collected incorrectly – and you haven’t provided the complete code that collects your data.

First, you want to fill your screen based on x movie frames. So you are finding a tiny horizontal stripe of pixels that, if multiplied by the number of frames, adds up to just over the number of pixels on your screen.

pxls = int(ceil((dispW*dispH) / numFrames));

Looks like that is sometimes 19 pixels.

Okay, so where do you start collecting that stripe?

int start = (i.pixels.length/2)-(pxls/2);

Oops. You divided your pixels length by 2. That gives you the last pixel in exactly half the pixels. If your image has an even number of rows and you cut the image in half with scissors, it is the last pixels on the top image – a pixel on the right-hand edge of the image – not the middle of the screen. You then subtract half the run length. Now exactly half your stripe (e.g. 9 pixels) will begin on the right-hand edge, and the second half (9 pixels) will wrap around to the next row and be on the left-hand edge. This behavior is dependent on how many rows are in the image. Try it on tiny images to see what I mean:

pixels.length/2 is…

3x2: on the right edge of row 1
3x3: in the center of row 2
3x4: on the right edge of row 2
3x5: in the center of row 3

etc.

(Also, you should be dividing by 2.0 and then casting to int last.)

tribblesontoast · July 3, 2019, 7:34am

Hi Jeremy,
Thanks for looking this over.
No I didn’t resolve it, it continues to stump me.

Here’s the entire processing process (not compatible with the example above)

I thought it might be the wrapping issue you talked about, I’ll have to read it again as I didn’t get it the first time.

thanks

void processFolderSelected(File selection) { 
  if (selection == null) {
    setPrevState();
    state = START;
  } else {    
    fileName = selection.getAbsolutePath();
    fileName = cleanFileName(fileName);

    File folder = selection;
    File[] pics = folder.listFiles(PIC_FILTER);
    Arrays.sort(pics);

    String[] imgFilenames = new String[pics.length];
    averages = new String[pics.length];
    centerAverages = new String[pics.length];
    bottomAverages = new String[pics.length];
    output_L = createWriter("Movies/" + fileName + "/" + fileName + "_png_ColourDataHSB_L.txt");
    output_M = createWriter("Movies/" + fileName + "/" + fileName + "_png_ColourDataHSB_M.txt");
    output_R = createWriter("Movies/" + fileName + "/" + fileName + "_png_ColourDataHSB_R.txt");
    numFrames = pics.length;

    //checks to see if required files are present, if not, throws errors below
    if (imgFilenames.length == 0) {
      String error = "Wah Wah: No Images in this folder: " + fileName;
      String title = "No Images found";
      JOptionPane.showMessageDialog(null, error, title, JOptionPane.INFORMATION_MESSAGE);
      setPrevState();      
      state = START;
    } else {

      pxls = 20;
      picLength = pics.length;
      float pBar = 0;

      //imgs
      for (int i = 0; i < pics.length; i++) {
        imgFilenames[i] = pics[i].getPath();
        PImage p = loadImage(imgFilenames[i], "png");
        String l = processImg(p, 0);
        String m = processImg(p, 1);
        String r = processImg(p, 2);
        output_L.println(l);
        output_M.println(m);
        output_R.println(r);

        pBar = map(i, 0, picLength, 0, width-100);
        progressBar.beginDraw();
        progressBar.background(0);
        progressBar.fill(-1, 30);
        progressBar.rect(50, height/2-50, width-100, 50, 35);
        progressBar.fill(-1);
        progressBar.rect(50, height/2-50, pBar, 50, 35);
        progressBar.text(fileName, 70, height/2-55);
        progressBar.text(i + " / " + picLength, 70, height/2+15);
        progressBar.endDraw();
      }

      output_L.flush();
      output_L.close();

      output_M.flush();
      output_M.close();

      output_R.flush();
      output_R.close();

      //calculates the overall dominant colour to use in the worm segment
      int currentC = 0, maxC = 0, domC = 0;
      for (Map.Entry m : colorMap.entrySet()) {
        currentC = (int)m.getValue();
        if (currentC > maxC) {
          maxC = currentC; 
          domC = (color)m.getKey();
        }
      }
      String[] dom = new String[]{str(domC)};
      saveStrings("Movies/" + fileName + "/" + fileName + "_tileColor.txt", dom);
    }
  }
  progressBar.clear();
  colorMap.clear();
  setPrevState();
  state = START;
}

String processImg(PImage i, int screenPos) { 
  String colString = "";
  int threshold = 15;

  i.loadPixels();

  if (screenPos == 0) { //top left
    for (int x = 0; x < pxls; x++) { 
      // The functions hue(), sat(), and brightness() pull out the 3 color components from a pixel.
      float h = hue       (i.pixels[x]);
      float s = saturation(i.pixels[x]);
      float b = brightness(i.pixels[x]);
      colString += ";" + str(color(h, s, b));
    }
  } else if (screenPos == 1) { //center
    int start = (i.pixels.length/2)-(pxls/2);
    for (int x = start; x < start+pxls; x++) {
      // The functions hue(), sat(), and brightness() pull out the 3 color components from a pixel.
      float h = hue       (i.pixels[x]);
      float s = saturation(i.pixels[x]);
      float b = brightness(i.pixels[x]);

      if (b < threshold || s < threshold) /*do nothing*/; //tests if colour is black (brightness) or if white (saturation)
      else {
        color tempC = color(h, s, b); //get color

        if (colorMap.containsKey(tempC)) {
          int value = (colorMap.get(tempC)).intValue();
          colorMap.put(tempC, new Integer(value + 1));
        }// 
        else colorMap.put(tempC, 1);
      }

      colString += ";" + str(color(h, s, b));
    }
  } else if (screenPos == 2) { //bottom right
    int startPos = i.pixels.length-pxls;
    for (int x = startPos; x < i.pixels.length; x++) {
      // The functions hue(), sat(), and brightness() pull out the 3 color components from a pixel.
      float h = hue       (i.pixels[x]);
      float s = saturation(i.pixels[x]);
      float b = brightness(i.pixels[x]);

      colString += ";" + str(color(h, s, b));
    }
  }
  return colString;
}

Edit: Cleaned up some unnecessary text

GoToLoop · July 3, 2019, 12:54pm

processFolderSelected() is called back by another Thread.
And from within that same callback above, processImg() is invoked.
That processImg() is full of color-related PApplet methods, such as color(), hue(), saturation(), brightness().
That means it’s 99.9% chance of complete disaster, given only 1 Thread can use those color-related functions concurrently!
Either createGraphics() for your separate Thread, or use an existing 1 like your progressBar.
Just make sure such calls are confined to 1 Thread exclusively.

jeremydouglass · July 3, 2019, 6:41pm

Here are 40 pixels. pixels/2 = the 20th pixel, pixels[19]. The middle pixel is marked with an X.

*******************X********************

Those 40 pixels are a 10x4 image. Here is where pixels[19] is located on that image:

**********
*********X
**********
**********

That is the right side – not the center of the screen. Here is a stripe of pixels centered on that pixel:

**********
*******XXX
XX********
**********

That samples the right and left edges – not the center of the screen. If the screen is e.g. a simple gradient, that means half the stripe will be pixels of one color - half will be pixels of another color.

To recap: the “middle” pixel is not in the “center” of the image (when there are an even number of rows).

tribblesontoast · July 3, 2019, 9:21pm

So to account for both cases (odd and even), I’m guessing I should be using something like a modulus to check and then having appropriate position calculations.

if(img.height%2 == 0) {
    start = (i.pixels.length/2-(i.width/2))-(pxls/2);
} else {
    start = (i.pixels.length/2)-(pxls/2);
}

jeremydouglass · July 3, 2019, 10:12pm

Yes, this would work.

The alternative is to express the center of the screen as width/height, then convert this to an index using pixels[y*width+x] (as taken from the get() reference page).

So given:

int w=12;
int h=10;
int centerPixel = (h/2)*w + w/2;
println(centerPixel);

The output is:

66

That’s as close as you can get to the center: the low-center row, and the right-center pixel.

While for w=5, h=3, the output is 7 – so that is the center row and the center pixel. It works in both cases.

Here is a demo using a canvas. It marks the center pixel with a dot, no matter what size you make the canvas.

size(120,100);
int centerPixel = (height/2)*width + width/2;
println(centerPixel);
loadPixels();
pixels[centerPixel] = color(0);
updatePixels();

tribblesontoast · July 5, 2019, 6:26am

So because of this thread exclusivity, could it be adversely affecting the color data I am retrieving?

So do you mean draw the image in the progressBar PGraphics and get the pixels from there?
Apologies, I’m not entirely sure what you mean.

GoToLoop · July 5, 2019, 10:11am

I’ve meant the color-dependent methods like hue(), fill(), etc.
Only 1 Thread can invoke such methods at the same time.

tribblesontoast · July 5, 2019, 11:25am

I thought these were called as part of the thread that called processFolderSelected.

GoToLoop · July 5, 2019, 1:11pm

Yea, outside the “Animation” Thread.

Even a single background() in draw() is enough to screw color calculation in your other Thread!

tribblesontoast · July 6, 2019, 10:00am

So drawing the img into a pgraphics buffer like you said would avoid this disaster?

for each img
draw img to pgraphics
capture pixel data
avoid calamity

GoToLoop · July 6, 2019, 1:35pm

Again for the 4th time, my warning is about using color-related methods from the same PGraphics on more than 1 Thread at the same time.

You already have a PGraphics named progressBar that you mutate under the processFolderSelected() callback’s Thread.

The problem is that you also invoke brightness(), hue(), etc. there, which belong to the sketch’s PGraphics main canvas g, outside the “Animation” Thread!

jeremydouglass · July 6, 2019, 5:19pm

It could, but you should first implement the centering fix. Your L and R aren’t stuttering in the same way. This is because your centering math was wrong.

Fix that first. Test it. No stuttering?

THEN stop calling hue() etc. in a Thread.

jeremydouglass · July 6, 2019, 5:20pm

Or keep doing your own color math in the thread. This isn’t terribly difficult – you just need to stop using the shared caching variables that PGraphics uses to dramatically speed up computing the same pixel over and over again.

github.com

processing/processing/blob/master/core/src/processing/core/PGraphics.java#L8043-L8050


      
           * @see PGraphics#green(int)
           * @see PGraphics#blue(int)
           * @see PGraphics#alpha(int)
           * @see PGraphics#hue(int)
           * @see PGraphics#brightness(int)
           */
          public final float saturation(int rgb) {
            if (rgb != cacheHsbKey) {

Those “cache” variables can break if something outside the thread uses them – although right now, the only thing your animation thread is calling in draw is image, so I honestly don’t see the problem!

But you can do hue / saturation math just fine without them, if a bit slower for streaks of the same color. Here is one example:

import java.awt.Color;

void setup() {
  // generate random static
  loadPixels();
  for (int i=0; i<pixels.length; i++) {
    pixels[i] = color((int)random(256), (int)random(256), (int)random(256));
  }
  updatePixels();
}

void draw() {
  // launch a color analyzer in a thread
  if (frameCount==1) {
    thread("satThread");
  }
}

void satThread() {
  float[] hsbval = new float[3];
  int brightcount=0;
  int satcount=0;
  loadPixels();
  for (int i=0; i<pixels.length; i++) {
    // access color values using a local function,
    // bypassing the PGRaphics animation thread cache values
    hsbval = RGBtoHSB(pixels[i]);
    if (hsbval[0]*255>240) {
      brightcount++;
    }
    if (hsbval[1]*255>240) {
      satcount++;
    }
  }
  print("done: ", brightcount, satcount);
}

/* 
 * get HSB values in a thread
 * note the HSB results are scaled 0-1.0, not 0-255)
 */
public final float[] RGBtoHSB(int rgb) {
  final float[] HsbValue = new float[3];
  Color.RGBtoHSB((rgb >> 16) & 0xff, (rgb >> 8) & 0xff, 
    rgb & 0xff, HsbValue);
  return HsbValue;
}

I believe that if you have no cache variables, then there is no thread problem. Or you could even implement your own cache for that thread if you wanted to speed things up – but this is simpler.

Disclaimer: not an expert on using PGraphics with threads. I’m not sure what the implications are for how you interact with loadPixels from a thread, for example.

tribblesontoast · July 7, 2019, 11:06pm

I’ve fixed the center math and it looks better, there is still some stuttering but that is due to there simply being different colours at the start and end of the ribbon of collected colours. The weird pattern generation appears to have been thwarted.

Thankyou Jeremy for your assistance and patience and thankyou goToLoop for the heads up.

Topic		Replies	Views
Flickering background when updating img.pixels[] in another thread Coding Questions	2	394	November 19, 2018
Pixels buffer and framerate Coding Questions	11	381	November 9, 2018
Help on video effect Libraries	1	258	March 8, 2021
Help with pixels[] loadPixels(), etc Libraries	6	432	May 15, 2021
Avoid 'clunky' sketch/framerate Coding Questions	24	1344	June 19, 2019

Pixels "stutter" when drawing (now with included data)

Related Topics