Send video frame into OpenCV?

Processing Libraries
1

I’m trying to read the frame of a video and send it into OpenCV for blob detection, but am getting a strange error.

With this simple example:

import processing.video.*;

Movie mov;
OpenCV cv;

void setup() {
  size(640, 360);

  mov = new Movie(this, sketchPath("transit.mov"));   
  mov.play();
  mov.jump(0);

  cv = new OpenCV(this, mov.width, mov.height);
}


void draw() {
  if (mov.available()) {
    mov.read();
    cv.loadImage(mov);
    cv.threshold(150);
    image(cv.getOutput(), 0, 0);
  }
}

I get the error (three times, then it quits):

IndexOutOfBoundsException: Index: 3, Size: 0
IndexOutOfBoundsException: Index: 3, Size: 0

The video is from the Video library example, so I know it plays ok in Processing. Help!

2

Which line is the error coming from? Do you have a .mov file called “transit.mov”?

3

Sorry – it’s on the cv.loadImage(mov); line. Definitely have the video and it loads/plays fine. Crashes only when I try to pass the frame into OpenCV.

4

Ok, I got this to work but it’s a total hack:

import gab.opencv.*;
import processing.video.*;

Movie mov;
OpenCV cv;

void setup() {
  size(640, 360);

  mov = new Movie(this, sketchPath("transit.mov"));   
  mov.play();
  mov.jump(0);
  mov.loop();

  cv = new OpenCV(this, mov.width, mov.height);
}


void draw() {
  if (mov.available()) {
    mov.read();
    
    // display the frame onscreen
    image(mov, 0,0);

    // then pass it to OpenCV using get()
    cv = new OpenCV(this, get(0,0,width,height));
    
    cv.threshold(150);
    image(cv.getOutput(), 0, 0);
  }
}
5
  • For a better performance you can use a separate cloned PImage.
  • Then use arrayCopy() to transfer the library’s PImage to the cloned 1:
  • arrayCopy() / Reference / Processing.org
  • I believe it’s safe now to pass the cloned PImage to the OpenCV instance.
  • Take a look at this example sketch below using Capture w/ arrayCopy() and a separate PImage:
// https://Discourse.Processing.org/t/send-video-frame-into-opencv/18075/5
// GoToLoop 2020-Feb-26

import processing.video.Capture;

Capture cam;
PImage frame = new PImage();

static final int CAM = 1, DELAY = 5;

void setup() {
  size(640, 480);
  initFeed();
  println("Cam's size:", cam.width, 'x', cam.height);
}

void draw() {
  set(0, 0, frame);
  getSurface().setTitle( str(round(frameRate)) );
}

void captureEvent(final Capture c) {
  c.read();

  if (frame.width == c.width) {
    c.loadPixels();
    arrayCopy(c.pixels, frame.pixels);
    frame.updatePixels();
  } else frame = c.get();
}

void initFeed() {
  final String[] cams = Capture.list();
  printArray(cams);
  println("\nChosen Cam #" + CAM + ':', cams[CAM], ENTER);

  ( cam = new Capture(this, cams[CAM]) ).start();

  println("cam.width: " + cam.width);
  while (cam.width == 0)  delay(DELAY);
  println("cam.width: " + cam.width + ENTER);

  if (cam.width > 0)  getSurface().setSize(cam.width, cam.height);
}
6

Thanks! I’m actually not seeing any improvement in performance here, which is surprising. This sample video (from the Video library) is 640 x 360 but I’m still getting 60fps on both.

Would still be interested to know if this is an OpenCV or Video lib issue.

7
  • You need to consider that the video library is constantly mutating its pixels[] internal array.
  • Therefore passing its reference to another library can be problematic depending on what the other library does to it.
  • Passing instead a vanilla PImage clone to OpenCV seems like an excellent workaround IMO.
8

Totally – just weird because you can easily pass a Capture object from the same lib with no issues.

9

Is it true? In my posted example I’m passing the cloned PImage frame as image()'s argument in place of the original Capture cam.

The idea here is that you can use the same strategy when invoking OpenCV::loadImage().

10

No it isn’t.

This on the other hand is more interesting. The width and height of the Movie are probably going to be 0 when the cv = new OpenCV(this, mov.width, mov.height); line executes. Try outputting them. If so, maybe try -

 if (mov.available()) {
    mov.read();
    if (cv == null) {
        cv = new OpenCV(this, mov.width, mov.height);
    }