@Mesalcode ===
- Have you tried what i have explained? (i have tested and it works)
- Can you put some code snippet wich shows what you are doing when it crashes?
Thatâs far too memory consuming. When you use the Gif lib with a small file, you already have to extend processingâs memory use to 1G in the preferences. But I guess, maybe, you can get the frame of the âunderlyingâ video, by capturing a frame at run time with the MediaCodec class transferring it into a PGraphics, and display it in the sketch.
Have you tried what i have explained? (i have tested and it works)
Sorry if I did not clarify the problem enough, but I only want to add a moving background to an existing game that fully consists of processing functions. So I would have to replace everything with native methods and basically would have wasted all my work just to add a background.
Can you put some code snippet which shows what you are doing when it crashes?
This is a minimal reproducable example:
Animation animation1;
float xpos;
float ypos;
float drag = 30.0;
void setup() {
fullScreen(P3D);
background(255, 204, 0);
frameRate(24);
animation1 = new Animation("", 300);
ypos = height * 0.25;
}
void draw() {
println(frameRate);
float dx = mouseX - xpos;
xpos = xpos + dx/drag;
// Display the sprite at the position xpos, ypos
if (mousePressed) {
background(153, 153, 0);
animation1.display(xpos-animation1.getWidth()/2, ypos);
}
/*
else {
background(255, 204, 0);
animation2.display(xpos-animation1.getWidth()/2, ypos);
}*/
}
// Class for animating a sequence of GIFs
class Animation {
PImage[] images;
int imageCount;
int frame;
Animation(String imagePrefix, int count) {
imageCount = count;
images = new PImage[imageCount];
for (int i = 0; i < imageCount; i++) {
// Use nf() to number format 'i' into four digits
String filename = imagePrefix + nf(i, 4) + ".jpg";
images[i] = loadImage(filename);
}
}
void display(float xpos, float ypos) {
frame = (frame+1) % imageCount;
image(images[frame], xpos, ypos);
}
int getWidth() {
return images[0].width;
}
}
That sounds really complicated because it is a mix of native android and processing, but that could work. I havenât worked with the MediaCodec class yet and to be honest I donât have the know how regarding codecs etc. Could you provide a code example that shows how to read a frame from a video file and how to get its pixels?
Edit: I found this code on StackOverflow but the author calls it very inefficient as it takes a fifth of a second to execute per bitmap:
public static void read(@NonNull final Context iC, @NonNull final String iPath)
{
long time;
int fileCount = 0;
//Create a new Media Player
MediaPlayer mp = MediaPlayer.create(iC, Uri.parse(iPath));
time = mp.getDuration() * 1000;
Log.e("TAG", String.format("TIME :: %s", time));
MediaMetadataRetriever mRetriever = new MediaMetadataRetriever();
mRetriever.setDataSource(iPath);
long a = System.nanoTime();
//frame rate 10.03/sec, 1/10.03 = in microseconds 99700
for (int i = 99700 ; i <= time ; i = i + 99700)
{
Bitmap b = mRetriever.getFrameAtTime(i, MediaMetadataRetriever.OPTION_CLOSEST_SYNC);
if (b == null)
{
Log.e("TAG", String.format("BITMAP STATE :: %s", "null"));
}
else
{
fileCount++;
}
long curTime = System.nanoTime();
Log.e("TAG", String.format("EXECUTION TIME :: %s", curTime - a));
a = curTime;
}
Log.e("TAG", String.format("COUNT :: %s", fileCount));
}
Edit 2: On codota I found Bitmap.getPixels() but I am unsure what it returns
Yes I saw that. He is talking about 5 frames per second. Still better than the one I found, 2f/s
@akenaton has been my guru since I started with P4A, so I guess we will rely on him.
Bitmap.getPixels() is the same as when we use âloadPixels()â
Okay. I hope @akenaton has an idea for performance improvements, if not I will have to live with the terrible performance of 5fps or less 
or replace the video animation with a object oriented processing animation that wonât look far as good as the current background.
Have you read this one?
Yes, that is the thread I found.
On there it also refers to this site: https://bigflake.com/mediacodec/#ExtractMpegFramesTest
It is really user unfriendly but it states that it is able to perform with 30fps which would be perfectly fine, so I will have to try to understand this method.
This is an example I found on the site, which gets the first 10 frames of a video and exports it to png, which I would have to change to read the pixels:
https://bigflake.com/mediacodec/ExtractMpegFramesTest.java.txt
Also its dependent on some other libraries which is also really subobtimalâŚ
Thatâs the one Iâve read yesterday. But you didnât complete the sentence with âbut the additional steps required to save it to disk as a PNG are expensive (about half a second).â Thatâs the 2 f/s.
But Iâm not saving the images as a PNG so Iâd would not lose the time needed for that or do I understand something incorrectly?
Maybe a other solution to this problem could be âcompressingâ the images and âdecompressingâ them when they are needed to minimize memory usage.
This runs at a reasonable frameRate with just one image in memory:
class BackgroundAnimator{
PImage lastImage;
PImage image;
PImage[] images = new PImage[1];
BackgroundAnimator(){
images[0] = loadImage("bg/0001.jpg");
images[0].resize(width,height);
}
private PImage loadNext(){
return images[0];
}
PImage getPImage(){
if (frameCount==1)
image = loadNext();
else if (frameCount % 4 == 0){
new Thread(new Runnable(){public void run(){
lastImage = loadNext();
}}).start();
}
Thread t = new Thread(new Runnable(){public void run(){
while(lastImage==null);
image = lastImage;
}});
t.start();
return image;
}
}
If somehow I could represent the images with less memory I should be able to display a video at a reasonable frameRate without the app crashing.
I think itâs not only compressing but the also very fact that the ânext frameâ, only contains the pixels that changed related to the previous frame.
Just an idea; you could write a code to get the pixels array of every frame in P4PC with the video lib, save the âchangedâ pixels of the array, in a byte file, and once done, get/set them at run time in android.
Still would be a very big file though.
Edit: Converting video first in a one with fewer colors.
Good idea, will try this out later.
See the edit, we posted simultaneously.
Ok will do this aswell. Thanks for the advise!