I need to write a small, straightforward program that allows children to draw into a webcam picture. I don’t have much time to do it, so I want to ask the pros before sinking valuable hours into code that doesn’t work or is just coded in a way that takes too much computing resources. (Also my girlfriend is gone for a couple days and I want the program to work when she’s back.)
I’m working on an exhibition where children can work with light, projectors, and projection surfaces. One of the “stations” we’ve been using combines a simple webcam stream and paint 3D, plus a digital drawing pad. Kids use it; it’s okay.
It has worked OK-ish, but I’m not happy with it. Right now, the projector screen is split in half; one half is the webcam, and the other half is the paint program. There are many unnecessary elements, and kids often don’t get to “experiment.” I have repeatedly proposed writing a small piece of software to superimpose whatever is drawn onto the webcam picture. (But the suggestion has been turned down because it’s “too much work.”) I don’t need fancy colors or tools; I have bought a small macro pad that I’d use to switch colors so that I can theoretically skip any GUI and give the drawing pad and the macro pad, and the kids can draw right into the camera stream.
I’ve tested the webcam stream in Processing, which works without issues. I bet I can also get the macro pad to work efficiently when switching colors.
So my question:
What would be your general approach to writing this software?
Can I use layers?
Does something like this maybe already exist and I just haven’t found it?
import processing.video.*;
Capture cam;
void setup() {
size(640, 480);
String[] cameras = Capture.list();
if (cameras.length == 0) {
println("There are no cameras available for capture.");
exit();
} else {
println("Available cameras:");
for (int i = 0; i < cameras.length; i++) {
println(cameras[i]);
}
// The camera can be initialized directly using an
// element from the array returned by list():
cam = new Capture(this, cameras[0]);
cam.start();
}
}
void draw() {
if (cam.available() == true) {
cam.read();
}
image(cam, 0, 0);
// The following does the same, and is faster when just drawing the image
// without any additional resizing, transformations, or tint.
//set(0, 0, cam);
}
A simple example that will draw with the mouse on an offscreen buffer with a transparent background and superimpose it on the webcam image:
import processing.video.*;
Capture cam;
PGraphics pg;
boolean clear;
void setup()
{
size(640, 480);
pg = createGraphics(640, 480);
String[] cameras = Capture.list();
if (cameras.length == 0)
{
println("There are no cameras available for capture.");
exit();
}
else
{
println("Available cameras:");
for (int i = 0; i < cameras.length; i++)
{
println(cameras[i]);
}
// The camera can be initialized directly using an
// element from the array returned by list():
cam = new Capture(this, cameras[0]);
cam.start();
}
pg.beginDraw();
pg.clear(); // Transparent background
pg.endDraw();
}
void draw()
{
if (cam.available() == true)
{
cam.read();
}
pg.beginDraw();
pg.strokeWeight(10);
pg.stroke(mouseX*255.0/640, mouseY*255/480, 0);
pg.point(mouseX, mouseY);
pg.endDraw();
// The following does the same, and is faster when just drawing the image
// without any additional resizing, transformations, or tint.
set(0, 0, cam);
image(pg, 0, 0);
}
void mousePressed()
{
pg.clear();
}
Seriously, thanks this will speed up development immensely. Girlfriend is coming back in a few hours and I feel like I’ll have something to show until then.
One question:
This will probably just work in fullscreen and whatever resolution I’ll enter, right?
Anything I should look out for?
We’ve been using 1080p cameras to make use of the whole screen, but:
I found that getting the aspect ratio right is more important than the resolution itself
Better color rendering is better.
I’ve been looking at getting Sony IMX sensors in various packages, which tend to have the potential for better resolution, are all USB-OTG and UVC-compatible, and, first and foremost, come with nice C-mount packages. (Giving one more haptic feedback thing to play around with.)
Any thoughts on that? Either way I’ll report back.
For reference, these are the ones I looked at. I’ll be testing with an IMX323 as a module with pinhole lens because it’s there,
Sure! Here is an overview of the strengths and weaknesses of several Sony IMX sensors in a table format, including key specifications:
Sensor
Resolution
Low Light Performance
Max FPS (Full Res)
Dynamic Range
Strengths
Weaknesses
IMX291
2 MP (1920x1080)
Excellent (0.01 Lux)
60 fps
High
- Excellent low light performance - High sensitivity
- Lower resolution compared to newer sensors
IMX323
2 MP (1920x1080)
Very Good (0.01 Lux)
30 fps
High
- Good low light performance - Cost-effective
- Lower resolution - Moderate frame rate
IMX335
5 MP (2592x1944)
Good
30 fps
High
- Balanced performance - Good dynamic range
- Higher resolution can be taxing on older systems
IMX298
16 MP (4656x3496)
Moderate
30 fps
Moderate
- Very high resolution - Suitable for detailed imagery
- High data rate - Requires more processing power
IMX415
8 MP (3840x2160)
Good
60 fps
High
- High resolution with good low light performance - High frame rate
- Higher data rate and processing requirements
IMX577
12 MP (4056x3040)
Good
120 fps (1080p)
High
- High frame rate - Good resolution - Suitable for fast motion
- Higher cost - Requires substantial processing power
Also, another thing:
I’ve used this in Processing 4 but also tried getting p5js to tun on VScode. Works! Not the same way I had in mind, but it’s baby steps from here to where I want to get. (Next step: change the color by tapping a key. Should be simple enough… )
Am I correct that, theoretically, this would allow me to run the app on a website I can host (computing happening on the client side)?
That could be really useful!