hallo.
I am trying to get the camera working in prcessing android.
Using ketai, i get only 7fps no matter the size for example even in 160X120 capture i get 6-7fps
How can i use the native android camera api within processing IDE so i get a pimage?
Do you want to launch the camera?
1 Like
thank you noel, as i see this is for launching the camera as you said!
I need to have a realtime video preview inside my sketch (draw()) and get every frame to a Pimage.
I am trying to achieve it but no luck yet! 
@dimpapadop ====i am trying for that ; till now i thaink a) taht the preview will be in a surface view with an overlay layout; that is possible to do (though somewhat not easy with P5); then i dont know wether the video format from Android (YUV) will not be a new problem for the fps; i have to see…
not sure why you are getting poor framerate with ketai. This sketch might work for you. I run android 10 and it produces 60 + framerate.
int w = 1400,h = 580;
PShader edges;
KetaiCamera cam;
float counter;
boolean mdown;
void settings(){
size(w,h,P2D);
};
void setup() {
//edges = loadShader("edges.glsl");
orientation(LANDSCAPE);
imageMode(CENTER);
int n = 2;
cam = new KetaiCamera(this, 1200, (280)*n, 60);
frameRate(60);
}
void draw() {
background(0);
logic();
if (frameCount>100&&!cam.isStarted()){cam.start();}
//edges.set("mult",1.0);
//edges.set("type",counter);
//shader(edges);
image(cam, width/2, height/2);
fill(0);
rect(0,0,width,20);
rect(width-20,0,20,height);
rect(0,height-20,width,20);
rect(0,0,20,height);
fill(0);
text(frameRate,10,10);
text(counter,100,10);
}
void onCameraPreviewEvent(){
cam.read();
}
// start/stop camera preview by tapping the screen
void mousePressed(){
if (cam.isStarted()){
cam.stop();
}else cam.start();
};
void keyPressed() {
//if (key == CODED) {
// if (keyCode == MENU) {
// if (cam.isFlashEnabled())
// cam.disableFlash();
// else
// cam.enableFlash();
// }
//}
};
void logic(){
if(mousePressed&&!mdown){
mdown = true;
counter++;
}
if(counter>6.0)counter = 0.0;
if(!mousePressed)mdown = false;
if(mousePressed)println(counter);
};
ignore the shader code as its not necessary for the sketch to run and I would then need to upload my shader files. I have left it in there in case you wish to experiment with your own.
Also note I place the ketai library jar, in a folder named code, within the sketch folder.
1 Like
@paulgoux the sketch indeed can run at high framerate, the camera preview not.
I need to track a color to be able to draw with that blob. So a 7fps is not a good framerate for drawing lines. You have to write painfully slow
Yes but why can you not make use of the other resolutions? You can still do image processing on them.
Here’s a code that does this with relatively good results.
import ketai.camera.*;
KetaiCamera cam;
PImage prevFrame;
float threshold = 50;
void setup() {
size(320, 240);
cam = new KetaiCamera(this, 320, 240, 31);
// cam.setCameraID(1); // If you want the front camara
cam.start();
prevFrame = createImage(cam.width, cam.height, RGB);
}
void onCameraPreviewEvent() {
prevFrame.copy(cam, 0, 0, cam.width, cam.height, 0, 0, cam.width, cam.height);
prevFrame.updatePixels();
cam.read();
}
void draw() {
background(255);
// You don't need to display it to analyze it!
image(cam, 0, 0);
cam.loadPixels();
prevFrame.loadPixels();
// Start with a total of 0
// These are the variables we'll need to find the average X and Y
float sumX = 0;
float sumY = 0;
int motionCount = 0;
// Begin loop to walk through every pixel
for (int x = 0; x < cam.width; x++ ) {
for (int y = 0; y < cam.height; y++ ) {
// What is the current color
color current = cam.pixels[x+y*cam.width];
// What is the previous color
color previous = prevFrame.pixels[x+y*cam.width];
// Step 4, compare colors (previous vs. current)
float r1 = red(current);
float g1 = green(current);
float b1 = blue(current);
float r2 = red(previous);
float g2 = green(previous);
float b2 = blue(previous);
// Motion for an individual pixel is the difference between the previous color and current color.
float diff = dist(r1, g1, b1, r2, g2, b2);
// If it's a motion pixel add up the x's and the y's
if (diff > threshold) {
sumX += x;
sumY += y;
motionCount++;
}
}
}
// average location is the total location divided by the number of motion pixels.
float avgX = sumX / motionCount;
float avgY = sumY / motionCount;
// Draw a circle based on average motion
smooth();
noStroke();
fill(255);
ellipse(avgX, avgY, 16, 16);
}
End here’s another one with a particle system.
import ketai.camera.*;
KetaiCamera cam;
PImage prevFrame;
float threshold = 50;
int videoWidth;
int videoHeight;
int reduction;
ParticleSystem system;
void setup() {
size(320, 240);
system = new ParticleSystem(30000);
frameRate(30);
reduction = 1;
videoWidth = width/reduction;
videoHeight = height/reduction;
cam = new KetaiCamera(this, 320, 240, 20);
cam.start();
prevFrame = createImage(cam.width, cam.height, RGB);
}
void onCameraPreviewEvent() {
prevFrame.copy(cam, 0, 0, cam.width, cam.height, 0, 0, cam.width, cam.height);
prevFrame.updatePixels();
cam.read();
}
void draw() {
// image(cam, 0, 0);
background(255,200);
cam.loadPixels();
prevFrame.loadPixels();
for (int x=0; x< videoWidth; x+=3) {
for (int y = 0; y < videoHeight; y+=4) {
if (abs((cam.pixels[y* videoWidth +x] >> 16 & 0xFF)-(prevFrame.pixels[y* videoWidth +x] >> 16 & 0xFF)) < 75
&& abs((cam.pixels[y* videoWidth +x] >> 8 & 0xFF)-(prevFrame.pixels[y* videoWidth +x] >> 8 & 0xFF)) < 75
&& abs((cam.pixels[y* videoWidth +x] & 0xFF)-(prevFrame.pixels[y* videoWidth +x] & 0xFF)) < 75) {
} else {
for (int i=0; i<2; i++) {
system.addParticle(( videoWidth - x - 1)*reduction, y*reduction, random(-100, 100), random(-320, 0), 1);
}
}
}
}
system.update(1/frameRate, cam);
fill(0);
}
public class ParticleSystem {
private ArrayList <Particle> particles;
private int iterations;
private int maxSize;
ParticleSystem(int maxSize) {
this.particles = new ArrayList<Particle>(maxSize);
this.iterations = 1;
this.maxSize = maxSize;
}
void update(float dt, KetaiCamera cam ) {
for (int j = particles.size()-1; j >= 0; j--) {
Particle particle = particles.get(j);
particle.update(dt);
if (particle.position.x > 0 && particle.position.x < width &&
particle.position.y > 0 && particle.position.y < height) {
particle.render(cam.pixels[floor(particle.position.y/reduction)*videoWidth+(videoWidth-floor(particle.position.x/reduction)-1)]);
}
if (particle.shouldDestroy()) {
particles.remove(j);
}
}
}
void addParticle(float x, float y, float mass) {
this.addParticle(x, y, 0, 0, mass);
}
void addParticle(float x, float y, float dx, float dy, float mass) {
if (size() < maxSize) {
particles.add(new Particle(x, y, dx, dy, mass));
}
}
int size() {
return particles.size();
}
}
public class Particle {
private PVector position;
private PVector velocity;
private PVector force;
private float mass;
private boolean fixed;
private float diameter;
private float time;
private float maxTime;
Particle(float x, float y, float mass) {
this(x, y, 0.0, 0.0, mass);
}
Particle(float x, float y, float dx, float dy, float mass) {
this.position = new PVector(x, y);
this.velocity = new PVector(dx, dy);
this.force = new PVector(0, 0);
this.mass = mass;
this.fixed = false;
this.diameter = random(1, 10);
this.time = 0;
this.maxTime = random(3, 7);
}
void update(float dt) {
time += dt;
if (!fixed){
//Gravity
force.add(0.0, 200*mass, 0);
//force.div(mass); take this out for simplicity, don't need it for free fall
velocity.add(PVector.mult(force, dt));
position.add(PVector.mult(velocity, dt));
}
//Zero forces
force.mult(0);
}
void render(color c) {
int transparency = floor((time/maxTime)*255);
ellipseMode(CENTER);
fill(blendColor(c, color(255, 255, 255, transparency), BLEND));
noStroke();
ellipse(position.x, position.y, diameter, diameter);
}
boolean shouldDestroy() {
//destroy if off screen
if (position.x > width ||
position.x < 0 ||
position.y > height) {
return true;
}
//destroy if time is up
if (time > maxTime) {
return true;
}
return false;
}
}
1 Like
even in 160X120 the framerate stays the same paul!
noel Nice! and the particles are impressive! nice work!.
I already implemented similarly the tracking part and works ok
to be able to use the phone and not the webcam is the problem, with the low framerate
I finally understand. Not sure why this is a thing. I have managed to load @noel sketch on android but as you say camera framerate is poor 15 currently. My bad I got confused because I assumed ketai worked like the processing video library which has significantly higher framerates.
Also screen is currently flipped in the code.
I’ve looked info the Ketai câmara source code, and I doubt that this can improved. It takes an enormous amount of data processing, and a phone isn’t as fast as a computer. That’s why I user a small câmara frame that gives me at least 38 f/s . If you don’t need the actual câmara image, set it small sized and scale() your drawings
1 Like
noel, strange! to me even with 160X120 i get the same 7fps
i only get 7fps even @ 160X120 but i have a simple xiaomi mi a3 how did you manage to get 15fps?
Not sure if it’s my phone, huawei p20, I even changed the sketch so it’s almost the same size as my native resolution.
Heres the amended sketch i’m using, no image processing, just accessing the camera.
import processing.core.PImage;
import android.hardware.Camera;
import android.hardware.Camera.AutoFocusCallback;
import android.hardware.Camera.CameraInfo;
import android.hardware.Camera.Parameters;
import android.hardware.Camera.PictureCallback;
import android.hardware.Camera.PreviewCallback;
import android.hardware.Camera.Size;
Permission p;
import ketai.camera.*;
KetaiCamera cam;
PImage prevFrame;
float threshold = 50;
boolean started = false;
int w = 1400,h = 720;
void settings(){
size(w,h);
};
void setup() {
cam = new KetaiCamera(this, w-20, h-20, 120);
//cam.setCameraID(1); // If you want the front camara
//cam.start();
prevFrame = createImage(cam.width, cam.height, RGB);
}
void onCameraPreviewEvent() {
cam.read();
}
void draw() {
background(255);
if(!started)cam.start();
image(cam, 0, 0);
};
i still take 7 fps
with the same sketch
i think for some reason it takes the smallest fps i see in your snapshot min is 15fps mine is 7fps minimum framerate of cam!
and yours
strange
That is odd, might be a hardware issue.
Mine is 15 equal to Paul’s, but that isn’t the actual current frameRate. I text() It on top of the img, and It gives me 30+
1 Like