# Help with displaying video

**URL:** <https://discourse.processing.org/t/help-with-displaying-video/19006>\
**Category:** Processing for Android\
**Created:** [March 25, 2020, 3:59pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006 "2020-03-25T15:59:16Z")\
**Posts on this page:** 17\
**Page:** 2

<div class="post-metadata">

**Author:** ![akenaton](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@akenaton](https://discourse.processing.org/u/akenaton)\
**Post date:** [May 22, 2020, 9:37am UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/21 "2020-05-22T09:37:52Z")

</div>

@Mesalcode ===

- Have you tried what i have explained? (i have tested and it works)
- Can you put some code snippet wich shows what you are doing when it crashes?

---

<div class="post-metadata">

**Author:** ![noel](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/noel/32/213_2.png) [@noel](https://discourse.processing.org/u/noel)\
**Post date:** [May 22, 2020, 12:20pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/22 "2020-05-22T12:20:16Z")

</div>

> [@Mesalcode](#):
>
> My idea was to split the video into each frame, store the frames in an array and then iterate through the frames every time draw is called.

That’s far too memory consuming. When you use the Gif lib with a small file, you already have to extend processing’s memory use to 1G in the preferences. But I guess, maybe, you can get the frame of the “underlying” video, by capturing a frame at run time with the [MediaCodec class](https://developer.android.com/reference/android/media/MediaCodec.html) transferring it into a PGraphics, and display it in the sketch.

---

<div class="post-metadata">

**Author:** ![Mesalcode](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/mesalcode/32/9284_2.png) [@Mesalcode](https://discourse.processing.org/u/Mesalcode)\
**Post date:** [May 23, 2020, 1:04pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/23 "2020-05-23T13:04:44Z")

</div>

> Have you tried what i have explained? (i have tested and it works)

Sorry if I did not clarify the problem enough, but I only want to add a moving background to an existing game that fully consists of processing functions. So I would have to replace everything with native methods and basically would have wasted all my work just to add a background.

> Can you put some code snippet which shows what you are doing when it crashes?

This is a minimal reproducable example:

```auto
Animation animation1;

float xpos;
float ypos;
float drag = 30.0;

void setup() {
  fullScreen(P3D);
  background(255, 204, 0);
  frameRate(24);
  animation1 = new Animation("", 300);
  ypos = height * 0.25;
}

void draw() { 
  println(frameRate);
  float dx = mouseX - xpos;
  xpos = xpos + dx/drag;

  // Display the sprite at the position xpos, ypos
  if (mousePressed) {
    background(153, 153, 0);
    animation1.display(xpos-animation1.getWidth()/2, ypos);
  } 
  /*
  else {
    background(255, 204, 0);
    animation2.display(xpos-animation1.getWidth()/2, ypos);
  }*/
}

// Class for animating a sequence of GIFs

class Animation {
  PImage[] images;
  int imageCount;
  int frame;
  
  Animation(String imagePrefix, int count) {
    imageCount = count;
    images = new PImage[imageCount];

    for (int i = 0; i < imageCount; i++) {
      // Use nf() to number format 'i' into four digits
      String filename = imagePrefix + nf(i, 4) + ".jpg";
      images[i] = loadImage(filename);
    }
  }

  void display(float xpos, float ypos) {
    frame = (frame+1) % imageCount;
    image(images[frame], xpos, ypos);
  }
  
  int getWidth() {
    return images[0].width;
  }
}

```

---

<div class="post-metadata">

**Author:** ![Mesalcode](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/mesalcode/32/9284_2.png) [@Mesalcode](https://discourse.processing.org/u/Mesalcode)\
**Post date:** [May 23, 2020, 1:10pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/24 "2020-05-23T13:10:38Z")

</div>

That sounds really complicated because it is a mix of native android and processing, but that could work. I haven’t worked with the MediaCodec class yet and to be honest I don’t have the know how regarding codecs etc. Could you provide a code example that shows how to read a frame from a video file and how to get its pixels?

Edit: I found this code on StackOverflow but the author calls it very inefficient as it takes a fifth of a second to execute per bitmap:

```auto
public static void read(@NonNull final Context iC, @NonNull final String iPath)
{
    long time;

    int fileCount = 0;

    //Create a new Media Player
    MediaPlayer mp = MediaPlayer.create(iC, Uri.parse(iPath));
    time = mp.getDuration() * 1000;

    Log.e("TAG", String.format("TIME :: %s", time));

    MediaMetadataRetriever mRetriever = new MediaMetadataRetriever();
    mRetriever.setDataSource(iPath);

    long a = System.nanoTime();

    //frame rate 10.03/sec, 1/10.03 = in microseconds 99700
    for (int i = 99700 ; i <= time ; i = i + 99700)
    {
        Bitmap b = mRetriever.getFrameAtTime(i, MediaMetadataRetriever.OPTION_CLOSEST_SYNC);

        if (b == null)
        {
            Log.e("TAG", String.format("BITMAP STATE :: %s", "null"));
        }
        else
        {
            fileCount++;
        }

        long curTime = System.nanoTime();
        Log.e("TAG", String.format("EXECUTION TIME :: %s", curTime - a));
        a = curTime;
    }

    Log.e("TAG", String.format("COUNT :: %s", fileCount));
}

```

Edit 2: On codota I found **Bitmap.getPixels()** but I am unsure what it returns

---

<div class="post-metadata">

**Author:** ![noel](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/noel/32/213_2.png) [@noel](https://discourse.processing.org/u/noel)\
**Post date:** [May 23, 2020, 1:27pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/25 "2020-05-23T13:27:10Z")

</div>

> [@Mesalcode](#):
>
> the author calls it very inefficient:

Yes I saw that. He is talking about 5 frames per second. Still better than the one I found, 2f/s  
@akenaton has been my guru since I started with P4A, so I guess we will rely on him.  
Bitmap.getPixels() is the same as when we use ‘loadPixels()’

---

<div class="post-metadata">

**Author:** ![Mesalcode](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/mesalcode/32/9284_2.png) [@Mesalcode](https://discourse.processing.org/u/Mesalcode)\
**Post date:** [May 23, 2020, 1:35pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/26 "2020-05-23T13:35:16Z")

</div>

Okay. I hope @akenaton has an idea for performance improvements, if not I will have to live with the terrible performance of 5fps or less ☹ or replace the video animation with a object oriented processing animation that won’t look far as good as the current background.

---

<div class="post-metadata">

**Author:** ![noel](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/noel/32/213_2.png) [@noel](https://discourse.processing.org/u/noel)\
**Post date:** [May 23, 2020, 1:38pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/27 "2020-05-23T13:38:00Z")

</div>

Have you read [this one](https://stackoverflow.com/questions/56791589/how-to-get-frame-by-frame-from-mp4-mediacodec?rq=1)?

---

<div class="post-metadata">

**Author:** ![Mesalcode](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/mesalcode/32/9284_2.png) [@Mesalcode](https://discourse.processing.org/u/Mesalcode)\
**Post date:** [May 23, 2020, 1:39pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/28 "2020-05-23T13:39:32Z")

</div>

Yes, that is the thread I found.

---

<div class="post-metadata">

**Author:** ![Mesalcode](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/mesalcode/32/9284_2.png) [@Mesalcode](https://discourse.processing.org/u/Mesalcode)\
**Post date:** [May 23, 2020, 1:40pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/29 "2020-05-23T13:40:59Z")

</div>

On there it also refers to this site: [https://bigflake.com/mediacodec/#ExtractMpegFramesTest](https://bigflake.com/mediacodec/#ExtractMpegFramesTest)  
It is really user unfriendly but it states that it is able to perform with 30fps which would be perfectly fine, so I will have to try to understand this method.

---

<div class="post-metadata">

**Author:** ![Mesalcode](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/mesalcode/32/9284_2.png) [@Mesalcode](https://discourse.processing.org/u/Mesalcode)\
**Post date:** [May 23, 2020, 1:46pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/30 "2020-05-23T13:46:33Z")

</div>

This is an example I found on the site, which gets the first 10 frames of a video and exports it to png, which I would have to change to read the pixels:

[https://bigflake.com/mediacodec/ExtractMpegFramesTest.java.txt](https://bigflake.com/mediacodec/ExtractMpegFramesTest.java.txt)

Also its dependent on some other libraries which is also really subobtimal…

---

<div class="post-metadata">

**Author:** ![noel](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/noel/32/213_2.png) [@noel](https://discourse.processing.org/u/noel)\
**Post date:** [May 23, 2020, 1:51pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/31 "2020-05-23T13:51:45Z")

</div>

> [@Mesalcode](#):
>
> On there it also refers to this site: [https://bigflake.com/mediacodec/#ExtractMpegFramesTest](https://bigflake.com/mediacodec/#ExtractMpegFramesTest)  
> It is really user unfriendly but it states that it is able to perform with 30fps

That’s the one I’ve read yesterday. But you didn’t complete the sentence with “but the additional steps required to save it to disk as a PNG are expensive (about half a second).” That’s the 2 f/s.

---

<div class="post-metadata">

**Author:** ![Mesalcode](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/mesalcode/32/9284_2.png) [@Mesalcode](https://discourse.processing.org/u/Mesalcode)\
**Post date:** [May 23, 2020, 1:53pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/32 "2020-05-23T13:53:39Z")

</div>

But I’m not saving the images as a PNG so I’d would not lose the time needed for that or do I understand something incorrectly?

---

<div class="post-metadata">

**Author:** ![Mesalcode](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/mesalcode/32/9284_2.png) [@Mesalcode](https://discourse.processing.org/u/Mesalcode)\
**Post date:** [May 23, 2020, 2:00pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/33 "2020-05-23T14:00:08Z")

</div>

Maybe a other solution to this problem could be “compressing” the images and “decompressing” them when they are needed to minimize memory usage.  
This runs at a reasonable frameRate with just one image in memory:

```auto
class BackgroundAnimator{
  PImage lastImage;
  PImage image;
  PImage[] images = new PImage[1];
  BackgroundAnimator(){
    images[0] = loadImage("bg/0001.jpg");
    images[0].resize(width,height);
  }
  private PImage loadNext(){
    return images[0];
  }
  PImage getPImage(){
    if (frameCount==1)
      image = loadNext();
    else if (frameCount % 4 == 0){
      new Thread(new Runnable(){public void run(){
      lastImage = loadNext();
      }}).start();
    }
    Thread t = new Thread(new Runnable(){public void run(){
        while(lastImage==null);
        image = lastImage;
    }});
    t.start();
    return image;
    
  }
}

```

If somehow I could represent the images with less memory I should be able to display a video at a reasonable frameRate without the app crashing.

---

<div class="post-metadata">

**Author:** ![noel](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/noel/32/213_2.png) [@noel](https://discourse.processing.org/u/noel)\
**Post date:** [May 23, 2020, 3:06pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/34 "2020-05-23T15:06:10Z")

</div>

I think it’s not only compressing but the also very fact that the “next frame”, only contains the pixels that **changed** related to the previous frame.  
Just an idea; you could write a code to get the pixels array of every frame in P4PC with the video lib, save the ‘changed’ pixels of the array, in a byte file, and once done, get/set them at run time in android.  
Still would be a very big file though.

Edit: Converting video first in a one with fewer colors.

---

<div class="post-metadata">

**Author:** ![Mesalcode](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/mesalcode/32/9284_2.png) [@Mesalcode](https://discourse.processing.org/u/Mesalcode)\
**Post date:** [May 23, 2020, 3:18pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/35 "2020-05-23T15:18:21Z")

</div>

Good idea, will try this out later.

---

<div class="post-metadata">

**Author:** ![noel](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/noel/32/213_2.png) [@noel](https://discourse.processing.org/u/noel)\
**Post date:** [May 23, 2020, 3:19pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/36 "2020-05-23T15:19:21Z")

</div>

See the edit, we posted simultaneously.

---

<div class="post-metadata">

**Author:** ![Mesalcode](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/mesalcode/32/9284_2.png) [@Mesalcode](https://discourse.processing.org/u/Mesalcode)\
**Post date:** [May 23, 2020, 3:21pm UTC](https://discourse.processing.org/t/help-with-displaying-video/19006/37 "2020-05-23T15:21:30Z")

</div>

Ok will do this aswell. Thanks for the advise!

[Previous page](https://discourse.processing.org/t/help-with-displaying-video/19006.md?page=1)
