# Native camera android in processing android

**URL:** <https://discourse.processing.org/t/native-camera-android-in-processing-android/26858>\
**Category:** Processing for Android\
**Created:** [January 7, 2021, 2:59pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858 "2021-01-07T14:59:31Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![dimpapadop](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dimpapadop](https://discourse.processing.org/u/dimpapadop)\
**Post date:** [January 7, 2021, 2:59pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/1 "2021-01-07T14:59:31Z")

</div>

hallo.  
I am trying to get the camera working in prcessing android.  
Using ketai, i get only 7fps no matter the size for example even in 160X120 capture i get 6-7fps  
How can i use the native android camera api within processing IDE so i get a pimage?

---

<div class="post-metadata">

**Author:** ![noel](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/noel/32/213_2.png) [@noel](https://discourse.processing.org/u/noel)\
**Post date:** [January 13, 2021, 3:44pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/2 "2021-01-13T15:44:45Z")

</div>

Do you want to launch the camera?

> <https://github.com/EmmanuelPil/Android-java-code-utilities-widgets-for-Processing-for-Android/blob/master/Launch%20camera>

---

<div class="post-metadata">

**Author:** ![dimpapadop](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dimpapadop](https://discourse.processing.org/u/dimpapadop)\
**Post date:** [January 13, 2021, 4:38pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/3 "2021-01-13T16:38:53Z")

</div>

thank you noel, as i see this is for launching the camera as you said!  
I need to have a realtime video preview inside my sketch (draw()) and get every frame to a Pimage.  
I am trying to achieve it but no luck yet! 🙂

---

<div class="post-metadata">

**Author:** ![akenaton](https://avatars.discourse-cdn.com/v4/letter/a/a88e4f/32.png) [@akenaton](https://discourse.processing.org/u/akenaton)\
**Post date:** [January 13, 2021, 6:13pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/4 "2021-01-13T18:13:19Z")

</div>

@dimpapadop ====i am trying for that ; till now i thaink a) taht the preview will be in a surface view with an overlay layout; that is possible to do (though somewhat not easy with P5); then i dont know wether the video format from Android (YUV) will not be a new problem for the fps; i have to see…

---

<div class="post-metadata">

**Author:** ![paulgoux](https://avatars.discourse-cdn.com/v4/letter/p/b9bd4f/32.png) [@paulgoux](https://discourse.processing.org/u/paulgoux)\
**Post date:** [January 13, 2021, 7:07pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/5 "2021-01-13T19:07:12Z")

</div>

not sure why you are getting poor framerate with ketai. This sketch might work for you. I run android 10 and it produces 60 + framerate.

```auto

int w = 1400,h = 580;

PShader edges;
KetaiCamera cam;
float counter;
boolean mdown;

void settings(){
  size(w,h,P2D);
};

void setup() {
  //edges = loadShader("edges.glsl");
  orientation(LANDSCAPE);
  imageMode(CENTER);
  
  int n = 2;
  cam = new KetaiCamera(this, 1200, (280)*n, 60);
  frameRate(60);
}

void draw() {
  background(0);
  logic();
  if (frameCount>100&&!cam.isStarted()){cam.start();}
  //edges.set("mult",1.0);
  //edges.set("type",counter);
  //shader(edges);
  image(cam, width/2, height/2);
  fill(0);
  rect(0,0,width,20);
  rect(width-20,0,20,height);
  rect(0,height-20,width,20);
  rect(0,0,20,height);
  fill(0);
  text(frameRate,10,10);
  text(counter,100,10);
}

void onCameraPreviewEvent(){
  cam.read();
}

// start/stop camera preview by tapping the screen
void mousePressed(){
  if (cam.isStarted()){
    cam.stop();
  }else cam.start();
};

void keyPressed() {
  //if (key == CODED) {
  // if (keyCode == MENU) {
  // if (cam.isFlashEnabled())
  // cam.disableFlash();
  // else
  // cam.enableFlash();
  // }
  //}
};

void logic(){
  if(mousePressed&&!mdown){
      mdown = true;
      counter++;
      
    }
    if(counter>6.0)counter = 0.0;
    if(!mousePressed)mdown = false;
    if(mousePressed)println(counter);
};

```

ignore the shader code as its not necessary for the sketch to run and I would then need to upload my shader files. I have left it in there in case you wish to experiment with your own.

Also note I place the ketai library jar, in a folder named code, within the sketch folder.

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/processingfoundation1/original/2X/0/07bab7d0cba20a1cb78a51a5055f194bc596ffe4.png)

---

<div class="post-metadata">

**Author:** ![dimpapadop](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dimpapadop](https://discourse.processing.org/u/dimpapadop)\
**Post date:** [January 14, 2021, 4:49am UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/6 "2021-01-14T04:49:55Z")

</div>

> [@paulgoux](#):
>
> ```auto
> int w = 1400,h = 580;
> 
> PShader edges;
> KetaiCamera cam;
> float counter;
> boolean mdown;
> 
> void settings(){
> size(w,h,P2D);
> };
> 
> void setup() {
> //edges = loadShader("edges.glsl");
> orientation(LANDSCAPE);
> imageMode(CENTER);
>   
> int n = 2;
> cam = new KetaiCamera(this, 1200, (280)*n, 60);
> frameRate(60);
> }
> 
> void draw() {
> background(0);
> logic();
> if (frameCount>100&&!cam.isStarted()){cam.start();}
> //edges.set("mult",1.0);
> //edges.set("type",counter);
> //shader(edges);
> image(cam, width/2, height/2);
> fill(0);
> rect(0,0,width,20);
> rect(width-20,0,20,height);
> rect(0,height-20,width,20);
> rect(0,0,20,height);
> fill(0);
> text(frameRate,10,10);
> text(counter,100,10);
> }
> 
> void onCameraPreviewEvent(){
> cam.read();
> }
> 
> // start/stop camera preview by tapping the screen
> void mousePressed(){
> if (cam.isStarted()){
> cam.stop();
> }else cam.start();
> };
> 
> void keyPressed() {
> //if (key == CODED) {
> // if (keyCode == MENU) {
> // if (cam.isFlashEnabled())
> // cam.disableFlash();
> // else
> // cam.enableFlash();
> // }
> //}
> };
> 
> void logic(){
> if(mousePressed&&!mdown){
> mdown = true;
> counter++;
>       
> }
> if(counter>6.0)counter = 0.0;
> if(!mousePressed)mdown = false;
> if(mousePressed)println(counter);
> };
> 
> ```

@paulgoux the sketch indeed can run at high framerate, the camera preview not.  
 ![image](https://canada1.discourse-cdn.com/flex036/uploads/processingfoundation1/original/2X/7/7f4a983f50436b54a6ffe3d6a7c7f5323ae15459.png)

---

<div class="post-metadata">

**Author:** ![dimpapadop](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dimpapadop](https://discourse.processing.org/u/dimpapadop)\
**Post date:** [January 14, 2021, 7:06am UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/8 "2021-01-14T07:06:19Z")

</div>

I need to track a color to be able to draw with that blob. So a 7fps is not a good framerate for drawing lines. You have to write painfully slow

---

<div class="post-metadata">

**Author:** ![paulgoux](https://avatars.discourse-cdn.com/v4/letter/p/b9bd4f/32.png) [@paulgoux](https://discourse.processing.org/u/paulgoux)\
**Post date:** [January 14, 2021, 7:33am UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/9 "2021-01-14T07:33:26Z")

</div>

Yes but why can you not make use of the other resolutions? You can still do image processing on them.

---

<div class="post-metadata">

**Author:** ![noel](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/noel/32/213_2.png) [@noel](https://discourse.processing.org/u/noel)\
**Post date:** [January 14, 2021, 10:59am UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/10 "2021-01-14T10:59:51Z")

</div>

> [@dimpapadop](#):
>
> I need to track a color to be able to draw with that blob.

Here’s a code that does this with relatively good results.

```auto
import ketai.camera.*;
KetaiCamera cam;

PImage prevFrame;
float threshold = 50;

void setup() {
  size(320, 240);
  cam = new KetaiCamera(this, 320, 240, 31);
  // cam.setCameraID(1); // If you want the front camara
  cam.start();
  prevFrame = createImage(cam.width, cam.height, RGB);
}

void onCameraPreviewEvent() {
  prevFrame.copy(cam, 0, 0, cam.width, cam.height, 0, 0, cam.width, cam.height);
  prevFrame.updatePixels();
  cam.read();
}

void draw() {
  background(255);

  // You don't need to display it to analyze it!
  image(cam, 0, 0);

  cam.loadPixels();
  prevFrame.loadPixels();

  // Start with a total of 0
  // These are the variables we'll need to find the average X and Y
  float sumX = 0;
  float sumY = 0;
  int motionCount = 0; 

  // Begin loop to walk through every pixel
  for (int x = 0; x < cam.width; x++ ) {
    for (int y = 0; y < cam.height; y++ ) {
      // What is the current color
      color current = cam.pixels[x+y*cam.width];

      // What is the previous color
      color previous = prevFrame.pixels[x+y*cam.width];

      // Step 4, compare colors (previous vs. current)
      float r1 = red(current); 
      float g1 = green(current);
      float b1 = blue(current);
      float r2 = red(previous); 
      float g2 = green(previous);
      float b2 = blue(previous);

      // Motion for an individual pixel is the difference between the previous color and current color.
      float diff = dist(r1, g1, b1, r2, g2, b2);

      // If it's a motion pixel add up the x's and the y's
      if (diff > threshold) {
        sumX += x;
        sumY += y;
        motionCount++;
      }
    }
  }

  // average location is the total location divided by the number of motion pixels.
  float avgX = sumX / motionCount; 
  float avgY = sumY / motionCount; 

  // Draw a circle based on average motion
  smooth();
  noStroke();
  fill(255);
  ellipse(avgX, avgY, 16, 16);
}

```

[![]( " - YouTube") ](https://www.youtube.com/watch?v=SseW0wd9V9w)

End here’s another one with a particle system.

```auto
import ketai.camera.*;
KetaiCamera cam;

PImage prevFrame;
float threshold = 50;
int videoWidth;
int videoHeight;
int reduction;
ParticleSystem system;

void setup() {
  size(320, 240);
  system = new ParticleSystem(30000);
  frameRate(30);
  reduction = 1;
  videoWidth = width/reduction;
  videoHeight = height/reduction;
  cam = new KetaiCamera(this, 320, 240, 20);
  cam.start();
  prevFrame = createImage(cam.width, cam.height, RGB);
}

void onCameraPreviewEvent() {
  prevFrame.copy(cam, 0, 0, cam.width, cam.height, 0, 0, cam.width, cam.height);
  prevFrame.updatePixels();
  cam.read();
}

void draw() {
  // image(cam, 0, 0);
  background(255,200);
  cam.loadPixels();
  prevFrame.loadPixels();
  for (int x=0; x< videoWidth; x+=3) {
    for (int y = 0; y < videoHeight; y+=4) {
      if (abs((cam.pixels[y* videoWidth +x] >> 16 & 0xFF)-(prevFrame.pixels[y* videoWidth +x] >> 16 & 0xFF)) < 75
        && abs((cam.pixels[y* videoWidth +x] >> 8 & 0xFF)-(prevFrame.pixels[y* videoWidth +x] >> 8 & 0xFF)) < 75
        && abs((cam.pixels[y* videoWidth +x] & 0xFF)-(prevFrame.pixels[y* videoWidth +x] & 0xFF)) < 75) {
      } else {
        for (int i=0; i<2; i++) {
          system.addParticle(( videoWidth - x - 1)*reduction, y*reduction, random(-100, 100), random(-320, 0), 1);
        }
      }
    }
  }      
  system.update(1/frameRate, cam);
  fill(0);
}

public class ParticleSystem {
    private ArrayList <Particle> particles;
    private int iterations;
    private int maxSize;

    ParticleSystem(int maxSize) {
        this.particles = new ArrayList<Particle>(maxSize);
        this.iterations = 1;
        this.maxSize = maxSize;
    }
    
    void update(float dt, KetaiCamera cam ) {
        for (int j = particles.size()-1; j >= 0; j--) {
            Particle particle = particles.get(j);
            particle.update(dt);
            if (particle.position.x > 0 && particle.position.x < width &&
                particle.position.y > 0 && particle.position.y < height) {
                particle.render(cam.pixels[floor(particle.position.y/reduction)*videoWidth+(videoWidth-floor(particle.position.x/reduction)-1)]);
            }
            if (particle.shouldDestroy()) {
                particles.remove(j);
            }
        }
    }
    
    void addParticle(float x, float y, float mass) {
        this.addParticle(x, y, 0, 0, mass);
    }
    
    void addParticle(float x, float y, float dx, float dy, float mass) {
        if (size() < maxSize) {
            particles.add(new Particle(x, y, dx, dy, mass));
        }        
    }
    
    int size() {
       return particles.size();
    } 
}

public class Particle {
    private PVector position;
    private PVector velocity;
    private PVector force;
    private float mass;
    private boolean fixed;
    private float diameter;
    private float time;
    private float maxTime;

    Particle(float x, float y, float mass) {
        this(x, y, 0.0, 0.0, mass);
    }
    
    Particle(float x, float y, float dx, float dy, float mass) {
        this.position = new PVector(x, y);
        this.velocity = new PVector(dx, dy);
        this.force = new PVector(0, 0);
        this.mass = mass;
        this.fixed = false;
        this.diameter = random(1, 10);
        this.time = 0;
        this.maxTime = random(3, 7);
    }
    
    void update(float dt) {
        time += dt;
        if (!fixed){
            //Gravity
            force.add(0.0, 200*mass, 0);
            //force.div(mass); take this out for simplicity, don't need it for free fall
            velocity.add(PVector.mult(force, dt));
            position.add(PVector.mult(velocity, dt));
        }
        //Zero forces
        force.mult(0);
    }
    
    void render(color c) {
        int transparency = floor((time/maxTime)*255);
        ellipseMode(CENTER);
        fill(blendColor(c, color(255, 255, 255, transparency), BLEND));
        noStroke();
        ellipse(position.x, position.y, diameter, diameter);
    }
    
    boolean shouldDestroy() {
        //destroy if off screen
        if (position.x > width || 
        position.x < 0 ||
        position.y > height) {
            return true;
        }
        
        //destroy if time is up
        if (time > maxTime) {
            return true;
        }
        
        return false;
    }
}

```

[![](https://img.youtube.com/vi/15Ham38eUBw/hqdefault.jpg "Particle Video") ](https://www.youtube.com/watch?v=15Ham38eUBw)

---

<div class="post-metadata">

**Author:** ![dimpapadop](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dimpapadop](https://discourse.processing.org/u/dimpapadop)\
**Post date:** [January 14, 2021, 12:08pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/11 "2021-01-14T12:08:21Z")

</div>

even in 160X120 the framerate stays the same paul!

---

<div class="post-metadata">

**Author:** ![dimpapadop](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dimpapadop](https://discourse.processing.org/u/dimpapadop)\
**Post date:** [January 14, 2021, 12:11pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/13 "2021-01-14T12:11:24Z")

</div>

noel Nice! and the particles are impressive! nice work!.  
I already implemented similarly the tracking part and works ok

[![](https://img.youtube.com/vi/zvCMQBaFHYw/maxresdefault.jpg "interactive air presenter picture file update") ](https://www.youtube.com/watch?v=zvCMQBaFHYw)

to be able to use the phone and not the webcam is the problem, with the low framerate

---

<div class="post-metadata">

**Author:** ![paulgoux](https://avatars.discourse-cdn.com/v4/letter/p/b9bd4f/32.png) [@paulgoux](https://discourse.processing.org/u/paulgoux)\
**Post date:** [January 14, 2021, 12:48pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/14 "2021-01-14T12:48:39Z")

</div>

I finally understand. Not sure why this is a thing. I have managed to load @noel sketch on android but as you say camera framerate is poor 15 currently. My bad I got confused because I assumed ketai worked like the processing video library which has significantly higher framerates.

![image](https://canada1.discourse-cdn.com/flex036/uploads/processingfoundation1/original/2X/7/73ea7510c6268920ec09ccc1b861a7f8cc27d147.png)

 ![Screenshot_20210114_124624_processing.test.sketch_210114a](https://canada1.discourse-cdn.com/flex036/uploads/processingfoundation1/original/2X/b/baead1a78ea33712780144a0d1c0b5e86a6d1f72.jpeg)

Also screen is currently flipped in the code.

---

<div class="post-metadata">

**Author:** ![noel](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/noel/32/213_2.png) [@noel](https://discourse.processing.org/u/noel)\
**Post date:** [January 14, 2021, 1:37pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/15 "2021-01-14T13:37:40Z")

</div>

> [@dimpapadop](#):
>
> to be able to use the phone and not the webcam is the problem, with the low framerate

I’ve looked info the Ketai câmara source code, and I doubt that this can improved. It takes an enormous amount of data processing, and a phone isn’t as fast as a computer. That’s why I user a small câmara frame that gives me at least 38 f/s . If you don’t need the actual câmara image, set it small sized and scale() your drawings

---

<div class="post-metadata">

**Author:** ![dimpapadop](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dimpapadop](https://discourse.processing.org/u/dimpapadop)\
**Post date:** [January 14, 2021, 1:40pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/16 "2021-01-14T13:40:05Z")

</div>

noel, strange! to me even with 160X120 i get the same 7fps

---

<div class="post-metadata">

**Author:** ![dimpapadop](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dimpapadop](https://discourse.processing.org/u/dimpapadop)\
**Post date:** [January 14, 2021, 1:40pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/17 "2021-01-14T13:40:55Z")

</div>

i only get 7fps even @ 160X120 but i have a simple xiaomi mi a3 how did you manage to get 15fps?

---

<div class="post-metadata">

**Author:** ![paulgoux](https://avatars.discourse-cdn.com/v4/letter/p/b9bd4f/32.png) [@paulgoux](https://discourse.processing.org/u/paulgoux)\
**Post date:** [January 14, 2021, 1:44pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/18 "2021-01-14T13:44:07Z")

</div>

Not sure if it’s my phone, huawei p20, I even changed the sketch so it’s almost the same size as my native resolution.

Heres the amended sketch i’m using, no image processing, just accessing the camera.

```auto

import processing.core.PImage;
import android.hardware.Camera;
import android.hardware.Camera.AutoFocusCallback;
import android.hardware.Camera.CameraInfo;
import android.hardware.Camera.Parameters;
import android.hardware.Camera.PictureCallback;
import android.hardware.Camera.PreviewCallback;
import android.hardware.Camera.Size;

Permission p;

import ketai.camera.*;
KetaiCamera cam;

PImage prevFrame;
float threshold = 50;
boolean started = false;

int w = 1400,h = 720;

void settings(){
  size(w,h);
};

void setup() {
  cam = new KetaiCamera(this, w-20, h-20, 120);
  //cam.setCameraID(1); // If you want the front camara
  //cam.start();
  prevFrame = createImage(cam.width, cam.height, RGB);
}

void onCameraPreviewEvent() {
  cam.read();
}

void draw() {
  background(255);
  if(!started)cam.start();
  image(cam, 0, 0);

  
};

```

---

<div class="post-metadata">

**Author:** ![dimpapadop](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dimpapadop](https://discourse.processing.org/u/dimpapadop)\
**Post date:** [January 14, 2021, 1:52pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/19 "2021-01-14T13:52:08Z")

</div>

i still take 7 fps 🙂  
 ![image](https://canada1.discourse-cdn.com/flex036/uploads/processingfoundation1/original/2X/c/c6369df84f416115d3d5759584522f1251a2c833.png)  
with the same sketch

---

<div class="post-metadata">

**Author:** ![dimpapadop](https://avatars.discourse-cdn.com/v4/letter/d/ba9def/32.png) [@dimpapadop](https://discourse.processing.org/u/dimpapadop)\
**Post date:** [January 14, 2021, 1:53pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/20 "2021-01-14T13:53:15Z")

</div>

i think for some reason it takes the smallest fps i see in your snapshot min is 15fps mine is 7fps minimum framerate of cam!  
 ![image](https://canada1.discourse-cdn.com/flex036/uploads/processingfoundation1/original/2X/a/a3732741b76b87594133a20cde8bb6e34a51d6ed.png)  
and yours  
 ![image](https://canada1.discourse-cdn.com/flex036/uploads/processingfoundation1/original/2X/2/25a8eba5ecaf88aab325f147262ab5c86bf1faec.png)  
strange

---

<div class="post-metadata">

**Author:** ![paulgoux](https://avatars.discourse-cdn.com/v4/letter/p/b9bd4f/32.png) [@paulgoux](https://discourse.processing.org/u/paulgoux)\
**Post date:** [January 14, 2021, 2:06pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/21 "2021-01-14T14:06:44Z")

</div>

That is odd, might be a hardware issue.

---

<div class="post-metadata">

**Author:** ![noel](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/noel/32/213_2.png) [@noel](https://discourse.processing.org/u/noel)\
**Post date:** [January 14, 2021, 2:13pm UTC](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858/22 "2021-01-14T14:13:04Z")

</div>

Mine is 15 equal to Paul’s, but that isn’t the actual current frameRate. I text() It on top of the img, and It gives me 30+

[Next page](https://discourse.processing.org/t/native-camera-android-in-processing-android/26858.md?page=2)