I have an image that I want to be able to animate using a method similar to the Puppet Tool in After Effects, where points affect a mesh which has an image mapped on it.
If I want to do this am I going to have to reverse-engineer how puppet tool works myself, or is there a library that does this or something similar?
Hi stib!
You can do that in processing. What you want to do is create a mesh using QUAD_STRIPS, and assign (u,v) parameters to each vertex on the mesh. You can deform this mesh quite easily by playing with the vertex positions. Then, you can apply an imageShader that maps the image to the mesh by sampling at the (u,v) coordinates. Here’s a quick example I made from some stock image of an apple:
Processing Code:
PImage image;
PShader imageShader;
float a;
void setup() {
size(400, 400, P2D);
image = loadImage("apple.jpg");
imageShader = loadShader("imageShader.glsl");
imageShader.set("imageSampler", image);
}
void draw() {
background(0);
shader(imageShader);
drawMesh(15,15,a);
a += 0.1;
}
ArrayList<PVector[]> drawStrip(int x, int y, float a) {
ArrayList<PVector[]> stripList = new ArrayList<PVector[]>();
for (int j = 0; j < x+1; j++) {
PVector[] strip = new PVector[x+1];
for (int i = 0; i < y+1; i++) {
float param = 25.0*(i+j)/(x+1+y+1);
//Moving vectors
strip[i] = new PVector(width/y * i + 15*cos(a+param), height/(y-1) * j+ 15*sin(a+param), 0);
//Static vectors
//strip[i] = new PVector(width/y * i, height/(y-1) * j, 0);
}
stripList.add(strip);
}
return stripList;
}
void drawMesh(int x, int y, float a) {
ArrayList<PVector[]> strips = drawStrip(x, y, a);
for (int i = 0; i < strips.size()-2; i++) {
float v1 = 1.0*i/(strips.size()-2);
float v2 = 1.0*(i+1)/(strips.size()-2);
PVector[] list1=strips.get(i);
PVector[] list2=strips.get((i+1));
beginShape(QUAD_STRIP);
for (int j = 0; j < list1.length; j++) {
float u = 1.0*j/(list1.length-1);
PVector vec1 = list1[j];
PVector vec2 = list2[j];
vertex(vec1.x, vec1.y, u, v1);
vertex(vec2.x, vec2.y, u, v2);
}
endShape();
}
}
GLSL code
uniform sampler2D imageSampler;
varying vec4 vertTexCoord;
void main() {
vec2 st = vertTexCoord.st;
vec3 imageColor = texture2D(imageSampler, st).rgb;
gl_FragColor = vec4(imageColor, 1.0);
}
Feel free to play with this sketch and let me know how it works out. Load up your image and choose how you make your vertices and how you deform them.
EDIT: Just wanted to add that you can narrow your search by looking for mesh libraries that enable you to do the operations you want to do. Perhaps something that can do soft-select for smooth moves/rotations. As for mapping images, I think you can also load meshes mapped with images before hand, but I’m not sure. It’s worth looking into it. Also, the example you gave is essentially a background layer with another layer on top of the deformed mesh. Should be easy to recreate, good luck.
Nice! Just having a play with your code and the camera input inside PraxisLIVE. Might have to build an example out of this if you’re happy with me doing so?
Incidentally, you don’t need the custom shader for this as it’s built in already - note the two added lines after beginShape()
texture(in); // or image in your case!
textureMode(NORMAL);
By all means, go ahead.
Looks good! You can play with the “waviness” by changing the multiplier to something else (currently 25.0) for float param = 25.0*(i+j)/(x+1+y+1);
Thanks for reminding me about texture() and textureMode(NORMAL). I think I wanted to do something more with glsl shader but ended up scrapping it.
One thing you could do there for more accurate warping / mapping is the correction of q values as in processing-glvideo/examples/VideoMappingWithShader/VideoMappingWithShader.pde at master · gohai/processing-glvideo · GitHub
There is a library called FreeTransform available through the Contribution Manager which might be of interest to you. It doesn’t appear to have been updated for a couple years, but the Example still works. It doesn’t use meshes, but it probably has some code that could be gleaned and adapted for your purposes.
Thanks for that everyone. Looks like the image warping side of things is already doable, now I just have to work out how to map it so that it smoothly warps between control points.
Thanks. Yes, the illustration is slightly confusing. The mesh deformation is only happening on the layer with the triangular mesh on it.
I forgot to mention this the other day, but one more thing about assigning (u,v), especially if you’re loading meshes exported from other programs, is that you don’t have to go through the trouble of creating your own (u,v) texture coordinates like I did previously. They will be present in your obj file. PShape loads .obj, but I can’t seem to find a .getUV() method (perhaps others can chime in about this if they know). You can insread treat the .obj as a regular plain text file and write a simple parser for it. Sample code from wiki link:
# List of texture coordinates, in (u, v [,w]) coordinates, these will vary between 0 and 1, w is optional and defaults to 0.
vt 0.500 1 [0]
vt ...
...
Thanks for sharing this excellent example.
I have modified the shader code to accept alpha images, here it is:
imageShader.glsl
uniform sampler2D imageSampler;
varying vec4 vertTexCoord;
void main() {
vec2 st = vertTexCoord.st;
vec4 imageColor = texture2D(imageSampler, st).rgba;
gl_FragColor = vec4(imageColor);
}
In the sketch load a png image:
warpAlpha.pde
PImage image;
PShader imageShader;
float a;
void setup() {
size(400, 400, P2D);
image = loadImage("apple.png");
imageShader = loadShader("imageShader.glsl");
imageShader.set("imageSampler", image);
}
void draw() {
background(0,255,0);
shader(imageShader);
drawMesh(15,15,a);
a += 0.1;
}
ArrayList<PVector[]> drawStrip(int x, int y, float a) {
ArrayList<PVector[]> stripList = new ArrayList<PVector[]>();
for (int j = 0; j < x+1; j++) {
PVector[] strip = new PVector[x+1];
for (int i = 0; i < y+1; i++) {
float param = 25.0*(i+j)/(x+1+y+1);
//Moving vectors
strip[i] = new PVector(width/y * i + 15*cos(a+param), height/(y-1) * j+ 15*sin(a+param), 0);
//Static vectors
//strip[i] = new PVector(width/y * i, height/(y-1) * j, 0);
}
stripList.add(strip);
}
return stripList;
}
void drawMesh(int x, int y, float a) {
ArrayList<PVector[]> strips = drawStrip(x, y, a);
for (int i = 0; i < strips.size()-2; i++) {
float v1 = 1.0*i/(strips.size()-2);
float v2 = 1.0*(i+1)/(strips.size()-2);
PVector[] list1=strips.get(i);
PVector[] list2=strips.get((i+1));
beginShape(QUAD_STRIP);
for (int j = 0; j < list1.length; j++) {
float u = 1.0*j/(list1.length-1);
PVector vec1 = list1[j];
PVector vec2 = list2[j];
vertex(vec1.x, vec1.y, u, v1);
vertex(vec2.x, vec2.y, u, v2);
}
endShape();
}
}