Draw on top of video in Android Mode

Processing for Android
1

I have a relatively simple question, but I haven’t been able to crack this problem. I need to draw a sketch on top of a video that’s being played with MediaPlayer and SurfaceView. This sketch draws some GUI buttons, and interacts with the video, allowing the user to play/pause the video.

Whenever I try to draw something, that drawing ends up behind the video. What would be the right way to do this?

Here’s the code I’m using right now.

import android.app.Activity;
import android.content.Context;
import android.os.Bundle;
import android.media.session.MediaSession;
import android.media.session.MediaController;
import android.content.res.AssetFileDescriptor;
import android.media.MediaPlayer;
import android.view.ViewGroup;
import android.view.View;
import android.view.SurfaceView;
import android.view.SurfaceHolder;

Activity activity; // Current activity
Context context; // Context of the activity

MediaSession mSession; // TODO: Check if necessary
MediaController mController; // For controlling the player. Might be necessary for skipping videos
MediaPlayer mPlayer; // For playing video

AssetFileDescriptor video; // Variable storing the video
SurfaceView surface; // Surface where graphics are displayed
SurfaceHolder surfaceH; // Surface hook, used to display the video


// =============********** OVERRIDE ONCREATE **********===================
@Override
public void onCreate(Bundle savedInstanceState) {
  println("Create executed");
  // This runs when the android app is opened.
  super.onCreate(savedInstanceState);
  
  // Get the current Activity under which this sketch is running.
  activity = getActivity();
  context = activity.getApplicationContext();
  
  // ==================== OPENING THE VIDEO ================
  
  // Create a MediaPlayer. In charge of playing the video
  mPlayer = new MediaPlayer(); // Currently in the Idle state
  
  try {
    // Open the file and set it to the player
    video = context.getAssets().openFd("video.mp4");
    mPlayer.setDataSource(video);
    
    // Now it's safe to close
    video.close();
    // Currently in the Initialized State
  }
  catch (IllegalStateException e) {
    println("Could not open the file");
  }
  catch (IOException e) {
    // This handling is required. In case opening the file fails, run this.
    println("Could not set the Data Source");
  }
  
  // ================== SETTING THE DISPLAY =======================
  
  surface = new SurfaceView(context); // Create our display
  surface.setZOrderOnTop(false);
  surfaceH = surface.getHolder(); // get the hook
  
  surfaceH.addCallback(new SurfaceHolder.Callback() {
    @Override
    public void surfaceCreated(SurfaceHolder surfaceHolder) {
      mPlayer.setDisplay(surfaceHolder);
    }

    @Override
    public void surfaceChanged(SurfaceHolder surfaceHolder, int i, int i2, int i3) {
      mPlayer.setDisplay(surfaceHolder);
    }

    @Override
    public void surfaceDestroyed(SurfaceHolder surfaceHolder) {
    }
  });
  
  

  // Create a MediaSession. Used to interface with the MediaPlayer
  mSession = new MediaSession(context, "session_1");
  mController = mSession.getController();
}

// =============********** OVERRIDE ONSTART **********===================
@Override
public void onStart(){
  println("Start executed");
  try {
    mPlayer.prepare();
  }
  catch (IOException e) {
    e.printStackTrace();
  }
  
  activity.addContentView(surface, new ViewGroup.LayoutParams(width, height));
  
  mPlayer.start(); //Currently in the Started state
}

void setup() {
  background(0, 0);
  noStroke();

  // Draw something. This would be a GUI button.
  fill(204);
  triangle(18,18,18,360,81,360);
}
2

@aristizabal95 ===
have you tried this code??? - as for me it cannot run…

3

@akenaton now that you mention it, I tried it with an emulator and it worked perfectly, but running it on my cellphone crashed it. I guess I was depending on a high api to make it work. Just made some changes and it now runs on my phone, give it a try!

import android.app.Activity;
import android.content.Context;
import android.os.Bundle;
import android.media.session.MediaSession;
import android.media.session.MediaController;
import android.content.res.AssetFileDescriptor;
import android.media.MediaPlayer;
import android.view.ViewGroup;
import android.view.View;
import android.view.SurfaceView;
import android.view.SurfaceHolder;

Activity activity; // Current activity
Context context; // Context of the activity

MediaSession mSession; // TODO: Check if necessary
MediaController mController; // For controlling the player. Might be necessary for skipping videos
MediaPlayer mPlayer; // For playing video

AssetFileDescriptor video; // Variable storing the video
SurfaceView surface; // Surface where graphics are displayed
SurfaceHolder surfaceH; // Surface hook, used to display the video


// =============********** OVERRIDE ONCREATE **********===================
@Override
public void onCreate(Bundle savedInstanceState) {
  println("Create executed");
  // This runs when the android app is opened.
  super.onCreate(savedInstanceState);
  
  // Get the current Activity under which this sketch is running.
  activity = getActivity();
  context = activity.getApplicationContext();
  
  // ==================== OPENING THE VIDEO ================
  
  // Create a MediaPlayer. In charge of playing the video
  mPlayer = new MediaPlayer(); // Currently in the Idle state
  
  try {
    // Open the file and set it to the player
    video = context.getAssets().openFd("video.mp4");
    mPlayer.setDataSource(video.getFileDescriptor(), video.getStartOffset(), video.getLength());
    
    // Now it's safe to close
    video.close();
    // Currently in the Initialized State
  }
  catch (IllegalStateException e) {
    println("Could not open the file");
  }
  catch (IOException e) {
    // This handling is required. In case opening the file fails, run this.
    println("Could not set the Data Source");
  }
  
  // ================== SETTING THE DISPLAY =======================
  
  surface = new SurfaceView(context); // Create our display
  surface.setZOrderOnTop(true);
  surfaceH = surface.getHolder(); // get the hook
  
  surfaceH.addCallback(new SurfaceHolder.Callback() {
    @Override
    public void surfaceCreated(SurfaceHolder surfaceHolder) {
      mPlayer.setDisplay(surfaceHolder);
    }

    @Override
    public void surfaceChanged(SurfaceHolder surfaceHolder, int i, int i2, int i3) {
      mPlayer.setDisplay(surfaceHolder);
    }

    @Override
    public void surfaceDestroyed(SurfaceHolder surfaceHolder) {
    }
  });
  
  

  // Create a MediaSession. Used to interface with the MediaPlayer
  mSession = new MediaSession(context, "session_1");
  mController = mSession.getController();
}

// =============********** OVERRIDE ONSTART **********===================
@Override
public void onStart(){
  println("Start executed");
  try {
    println("preparing");
    mPlayer.prepare();
    println("prepared");
  }
  catch (IOException e) {
    e.printStackTrace();
  }
  
  activity.addContentView(surface, new ViewGroup.LayoutParams(width, height));
  
  mPlayer.start(); //Currently in the Started state
}

void setup() {
  background(0, 0);
  noStroke();

  fill(204);
  triangle(18,18,18,360,81,360);
}

Basically just changed the mPlayer.setDatasource() to one that is available from api 1 and changed surface.setZOrderOnTop(false); to true since the video wasn’t showing up after the change.

1 Like
4

@aristizabal95 ===

ok, you have done exactly what i thought to be corrected (though i cannot see why setDataSource will run with param like afd with any api);
now, as for the other question i think that using this code there is abolutely no way to add a button upon your video. Why? because the surfaceView you have created and added with addContentView is inside the FrameLayout created by P5 and is occupating the whole space. if you want to add a button you have to create another layout, relative or linear, then add your surface to it, then add the button, then add the linear (or relative layout) to the frameLayout) using addView. Another solution, more simple is to give to your surfaceView some size (width, height ) < display (for example the size of your video) and add your button out of the surfaceView.

1 Like
5

@akenaton sorry for the delay! I understand the issues. I initially wanted to use Processing to make a custom GUI without having to learn Android and Java programming. Seems like I’ll have to learn them anyway and adding processing is only making it harder. I decided to make everything on Android, and only use processing as a means to access a library I absolutely need (Spacebrew). Thanks for your input! It helped me get unstuck with my project!