Hello forum members,
I’m new to using shaders in Processing and need some help please. There is something I don’t understand about my code and I’m hoping one of you experts will be able to give me some guidance.
In summary: I’m experimenting with storing particle positions in a texture, then retrieving the information to plot the particle. Everything seems to work as expected so long as I use low resolution positions which will fit into the 8-bit channel of the texture, i.e. if I save the positions as vec4(x, y, 0., 1.). However, I need more resolution but when I try to pack the channels like this: vec4(LSB-x, LSB-y, MSB-x, MSB-y) I run into problems.
I’ve searched this forum and the internet for clues and there are a few leads, but nothing that directly answers my questions. These links provided helpful context:
To add some more details about my particular code:
-
I use 2 shaders: ‘initShader’ (frag only) initializes the positions of N particles and writes these positions into a texture. It is my intention to use random positions, but while I’m debugging I’m currently setting all positions to the middle of the sketch window i.e. (x,y) = (0.5, 0.5).
-
‘plotShader’ (vertex+fragment) to plot the particle on the sketch window: ‘plot.vert’ takes the texture and extracts the position information, then ‘plot.frag’ draws a green dot at the correct position.
-
The texture ‘tex’ is a PGraphic with size (N x 1) where N is the number of particles. It is my understanding that each of the N entries can hold four 8-bit channels i.e. vec4(R, G, B, A)
-
There is also a PShape ‘shp’ containing N vertices. I draw this shape to ensure the ‘plot.vert’ shader is executed the correct number (N) times - once for each particle. The vertex coordinates (0,0), (1,0), (2,0), … (N-1,0) are simply used as an index into the texture which contains the actual particle positions.
Here is the full code:
“Particle_packing.pde”
//------------------------------------------------------------------------------------------
PShader initShader, plotShader;
PGraphics tex;
PShape shp;
ArrayList<PVector> vectors = new ArrayList<PVector>();
int N = 100; // number of particles
//------------------------------------------------------------------------------------------
void setup() {
size(400, 400, P3D);
strokeWeight(2); stroke(255); smooth(2);
// define shaders
initShader = loadShader("init.frag"); // randomly initializes the particle positions
plotShader = loadShader("plot.frag", "plot.vert"); // plots the particle positions
// initialize the texture - it holds the actual particle positions
tex = createGraphics(N, 1, P2D); // N particles
tex.beginDraw();
//To aid with debugging, 'initShader' currently assigns all particles positions to (0.5, 0.5)
tex.shader(initShader);
tex.rect(0,0,tex.width, tex.height); // write info to the texture
tex.endDraw();
// Create a shape containing N vertices, where N is the number of particles we want
// We use this shape to ensure the plot.vertex shader is executed the correct number(N) times - once for
// each particle. The vertex coordinate is only used as an index into the texture which
// contains the actual particle positions
for (int i = 0; i<N; i++) { vectors.add(new PVector(i,0)); }
shp = createShape();
shp.beginShape(POINTS); // We want individual points to be plotted, not quads
for (PVector v: vectors) { shp.vertex(v.x, v.y); } // Encode index into shp vertex position (1,0),...(N-1,0)
shp.endShape();
}
//------------------------------------------------------------------------------------------
void draw(){
background(0);
// Use 'plotShader' instead of the default shader for future drawing
shader(plotShader);
// pass the texture containing the particle positions
plotShader.set("tex", tex);
// pass the size of sketch window as a uniform
plotShader.set("sketchSize",float(width), float(height));
// Use 'plotShader' to draw 'shp': It should pull particle positions from the texture and plot these
shape(shp, 0, 0);
}
//------------------------------------------------------------------------------------------
“init.frag”
//----------------------------------------------------------------------------------------
void main() {
//To aid with debugging, temporarily assign all particles positions to (0.5, 0.5)
gl_FragColor = vec4( .5, .5, 0., 1.); //debug override
}
//----------------------------------------------------------------------------------------
“plot.vert”
//----------------------------------------------------------------------------------------
// These uniforms and attributes are automatically generated by Processing
uniform mat4 projection;
uniform mat4 modelview;
attribute vec4 position; // the vertex data of 'shp' - i.e. (0,0), ...(N-1,0)
attribute vec2 offset;
// These are manually passed
uniform sampler2D data; // texture containing position data
uniform vec2 sketchSize;
//----------------------------------------------------------------------------------------
void main() {
// This shader is executed once for each vertex in the shp object, i.e. numParticles times
// The 'position' attribute for each vertex in shp has been set to (1,1),...(N-1,N-1)
// We use 'position.x' as an index into the data buffer which contains the actual particle positions
// 'data' is an Nx1 buffer, to extract the data correctly use the col index of 0
vec4 extractedPos = texelFetch(data, ivec2(position.x,0), 0); // e.g. (.5, .5, 0., 1.)
// Now plot to the screen... these additional transformation steps are necessary when using P3D
extractedPos.xy *= sketchSize; // convert from normalized to absolute position - only multiply .xy, not .z
vec4 pos = modelview * extractedPos;
vec4 clip = projection * pos;
gl_Position = clip + projection * vec4(offset, 0, 0);
}
//----------------------------------------------------------------------------------------
“plot.frag”
//----------------------------------------------------------------------------------------
void main() {
gl_FragColor = vec4(0., 1., 0., 1.); // plot a green dot at the particle position
}
//----------------------------------------------------------------------------------------
This code works as expected with all the particles being plotted at the same position - so the sketch window shows a single green dot in the middle. If you change (.5, .5, 0., 1.) in the initShader you can move this dot to other locations to convince yourself that this is working as intended. Also, if you modify ‘initShader’ to save each particle at a different random position you get N green dots, as expected.
But as I mentioned above, I need a higher resolution for my particle positions otherwise I’m limited to a 256x256 sketch window. Many coders have described how to pack position data into 4-bytes (2 for x, 2 for y) in order to get a resolution of 65536 instead. They typically pack the data like this: vec4(LSB-x, LSB-y, MSB-x, MSB-y), so the packed data will still fit within a normal texture. This is the code I used in the shaders:
//--------------------------------------------------------------------
vec4 pack(vec2 pos) { // pos is normalized 0-1
return vec4( fract(pos*255.), floor(pos*255.)/255. ); // (LSB-x, LSB-y, MSB-x, MSB-y)
}
//--------------------------------------------------------------------
vec2 unpack(vec4 data){
return vec2(data.x/255. + data.z, data.y/255. + data.w ); // returns normalized pos 0-1
}
//--------------------------------------------------------------------
In the second version of my code I include this pack/unpack feature: In the ‘initShader’ I pack the position into 4-bytes (as above) then write it to the texture. In the ‘plot.vert’ shader I read back this 4-byte packed-data from the texture, then unpack to position data (float(x,y)) before plotting.
Here’s the new code: Please note that only ‘init.frag’ and ‘plot.vert’ differ from the ‘no-packing version’ - all other code is the same as above:
“init.frag”
//--------------------------------------------------------------------
vec4 pack(vec2 pos) { // pos is normalized (0-1)
return vec4( fract(pos*255.), floor(pos*255.)/255. ); // (LSB-x, LSB-y, MSB-x, MSB-y )
}
//----------------------------------------------------------------------------------------
void main() {
vec2 pos = vec2(.5, .5);
gl_FragColor = pack(pos);
}
//----------------------------------------------------------------------------------------
“plot.vert”
//----------------------------------------------------------------------------------------
uniform mat4 projection;
uniform mat4 modelview;
attribute vec4 position; // the vertex data of 'shp' - i.e. (0,0),(1,0),...(N-1,0)
attribute vec2 offset;
uniform sampler2D tex; // texture containing position data
uniform vec2 sketchSize;
//--------------------------------------------------------------------
vec2 unpack(vec4 data){
return vec2(data.x/255. + data.z, data.y/255. + data.w ); // returns (0, 1)
}
//----------------------------------------------------------------------------------------
void main() {
vec4 packedData = texelFetch(tex, ivec2(position.x,0), 0); // (LSB-x, LSB-y, MSB-x, MSB-y )
vec4 unpackedData = vec4(unpack(packedData), 0., 1.);
unpackedData.xy *= sketchSize; // convert from normalized to absolute position
vec4 pos = modelview * unpackedData;
vec4 clip = projection * pos;
gl_Position = clip + projection * vec4(offset, 0, 0);
}
//----------------------------------------------------------------------------------------
However, this version of the code doesn’t run correctly - instead of a single dot in the middle of the screen I now get 2 dots, and neither is in the right place.
What have I tried?
5. If I calculate my ‘pack’ and ‘unpack’ functions using a pencil and paper they seem fine - even for small numbers < 1/255, e.g. unpack(pack(.001)) = .001. So I believe the functions are correct, i.e. unpack is working as the inverse of pack, and my resolution is increased beyond 8-bits.
-
In the ‘initShader’ I have tried to pack, then immediately unpack, the position data before writing it to the texture. The vertex shader reads back this already unpacked-data and plots it correctly.
-
I can do a similar ‘unpack(pack(…))’ thing in the vertex shader, again it plots correctly.
These 3 tests suggest that the packing and unpacking is being handled correctly, but there is something about writing packed-data into the texture that I don’t understand.
Also, why do I get 2 dots? There should only be one since all positions are initialized the same.
- What I have found is that if I change the number of particles this can vary: if N=1 there is only one dot (but in the wrong place); if N>=2 there are two dots (both wrong place).
To summarize the problem: My code for storing packed-data in a texture is not working properly. The output positions are wrong and I get 2 different outputs when I should only get one. Similar code with non-packed-data is OK. The tests I performed suggest that there is something about the way I am writing packed-data into the texture that I don’t properly understand.
Sorry this is such a long post. I know that there are other ways of doing this but I’d really like to understand why this particular approach doesn’t work. I’m slowly learning shaders, but at the moment I’m stumped and would really appreciate any insights. Thanks!
