Ideas and suggestions needed for Improving p5.js WebGL mode in GSOC 2019

[sanketsingh24]

EDIT: I recently completed my proposal. Please Review @kjhollen @stalgiag.
Here is the link to my proposal

Hey! My name is Sanket Singh, and I intend to work on improving and enhancing the WebGL mode in this year’s Google Summer Of Code. I have been contributing to p5.js since last December, so I know basic things about p5 and its workflow. At the moment, these are some of the functionalities, along with a little briefing, which can be implemented (clicking in my mind atm) and might benefit the community-

• spotLight() - A spotlight is a type of directional light that is not coming from infinitely far away. That is, it has a source position. The closest real-world example would be of a torch’s light, in the dark.
• emissiveMaterial() - What if, the material itself glows? Well, it is effectively an ambientMaterial() which does not rely on lighting up to a certain extent.
• specularMaterial() && specularLight() - If your sketch contains a red rubber ball, and a white light shines on it, diffuse red light is reflected in all directions. If that ball were to land in a puddle, the water on the surface of the ball would shine, or reflect a specular light. Specular refers to light which bounces off a surface in a preferred direction (rather than bouncing in all directions like a diffuse light).
• fallOff() for lights - What if you are standing with a lamp in your hands? Things which are further away from you would appear darker, or the light would ‘fall off’.
• removeLights() - Want to reset all the lights defined earlier, or just want no lighting effect in some part of your sketch?
• a function, by which the users can crop or resize the image in the sketch itself, would prove helpful, along with a sister function, which would return the original image. Removed this because there are many open source tools which are better.
• frustum() - An alternative to perspective camera with more freedom to user to select the projection.

I need reviews and suggestions on the above, along with maybe new ideas which can be implemented?

Also, I posted an issue on GitHub for the same. Thanks for your time!

Edit: Currently, all the implementation are very brief. I would include the actual code changes in th proposal itself.

[sanketsingh24]

spotLight(v1, v2, v3, x, y, z, nx, ny, nz, angle, concentration)-
We can understand this by taking the example of the torch. It is essentially an extension of pointlight() while also considering the opening of the torch, the direction of the torch, and torch’s strength.

Arguments -

1. The torch’s light has a color, depicted by the three RGB values v1,v2,v3 or we can pass p5.color object.
2. The position of the torch, whereas x,y,z being the coordinates of the position, a p5.vector can also be used.
3. The direction of the light emitted by the torch, with nx, ny, nz being the coefficient for a normalized vector describing the direction of the light, and p5.vector can also be used here.
4. The width of the opening of the torch, which basically resembles a cone, here is measured by the solid angle angle. It will be a number.
5. The strength of the torch, used here as concentration. The higher value of concentration will let you concentrate the light in the middle more.

Also, I am thinking of making the angle and concentration arguments as optional, and setting them to some default value when not supplied by the user.

Advantages and Motivation-

• The spotlight is very useful in creating a lot of lighting effects, and can be heavily used by the community when introduced.
• Although, it is not a fundamental lighting method used in WebGL, It is still very different from the other existing lighting methods present in p5.

Implementation-
Currently, I don’t think any previous methods defined in p5.src needs to be changed, but the light shader and phong shader will have to be changed to incorporate the solid angle and concentration.

[sanketsingh24]

emissiveMaterial(v1, v2, v3)-
Here, the material doesn’t emit the light, it just glows. So the neighboring objects don’t react to its light, but if there is no light, you can still see the emissivematerial(). It can be a lot like ambient material. You can also check this demonstration to understand more-

The code for this will be -

void draw(){
 
  background(0);
 
  ambientLight(128, 128, 128);
 
  push();
    emissiveMaterial(255,0,0);
    sphere(width * 0.2);

  pop();
 
  push();
    ambientMaterial(255,0,0);
    sphere(width * 0.2);

  pop();
}

So, here we can see if the strength of light is lesser than its emissive strength, emissive behavior will be shown, whereas if strength of light is more, both strengths add up (This is doubtful to me, as they should add up or light’s strength just shadow over the emissive strength, I need some advice here ) .

Here, v1, v2, v3 will be the color, can be also passed on as p5.color object.

Edit: Processing does add up the strengths to depict the material.

Motivation-
It will allow users to create a new kind of material, which has a property commonly seen in real life (like our Sun) and replicate it in their sketch. Although this won’t make the material emit the light itself, they can easily use other methods (like pointLight() in the case of Sun).

Implementation-
This won’t change any previous method, and would add a new object in p5 object. Also, the light shaders need to be modified.

[sanketsingh24]

fallOff(constant, linear, quadratic)

In a general sense, it can be used to limit the spread of diffusion from the lights, or vice versa. The falloff factor can also control the intensity of light.

Arguments-

1. Constant
2. Linear
3. Quadratic

These arguments would be used to calculate the falloff as-

falloff = 1 / (CONSTANT + d * LINEAR + (d*d) * QUADRATIC)

where d is the distance from light position to vertex position.

[sanketsingh24]

removeLights() or resetLights()

At the moment, there is no method in p5.js which resets all the lighting effect, and this method, would remove all the previously defined lights from the sketch, so that the user doesn’t have to do extra work.

Arguments-
No arguments.

[stalgiag]

Hi sanketsingh24 I think all of these suggestions are great! I think that the number and quality of lighting options in p5 WebGL mode is a little lacking. You are taking a good approach in looking to Processing in this regard. I think the specifics of the implementations can be worked out with your mentor if your project is chosen for GSOC. I think the important things to figure out when writing the proposal are :

• What is a feasible amount of work to accomplish in the time alloted to a GSOC project?
• Is there anything about the current implementation for lights that should be changed in order to make these new features work well?
• Is there an order to how these features would need to implemented? For example, I believe many of these other features would be heavily reliant on falloff.

Working on lighting in WebGL is a fairly technical project to propose. When writing this kind of proposal it is useful to indicate that you have a strong high-level understanding of how your additions will fit into and impact the existing code base. It is also always helpful to offer an idea for how these additions will enhance the power of p5 WebGL as a creative tool and intro to 3D graphics. I think you have done some of this here in your description of use cases. Anyways – that is my two cents. When writing remember to focus on the high-level aspects of how your change will fit into the existing codebase, what the major challenges will be, and how your additions serve the mission of the library. Good luck!

[sanketsingh24]

Thanks @stalgiag, I thought processing was a good place to start looking for ideas, and so I started there. And you are listed as mentor for this project . Although this list is far from over, I am just trying to design and formulate what each functionality is exactly going to do, and at last is required or not, so I can create a specific list which will help the community.

@stalgiag

It is also always helpful to offer an idea for how these additions will enhance the power of p5 WebGL as a creative tool and intro to 3D graphics.

I think I should include these right now, will do it on the priority. Soon I will start writing the proposal, and would ask for your review for sure. Thanks for the help, especially the proposal part, I was feeling a bit nervous as I have never done it before.

@stalgiag

• What is a feasible amount of work to accomplish in the time alloted to a GSOC project?
• Is there anything about the current implementation for lights that should be changed in order to make these new features work well?
• Is there an order to how these features would need to implemented? For example, I believe many of these other features would be heavily reliant on falloff.

Working on lighting in WebGL is a fairly technical project to propose. When writing this kind of proposal it is useful to indicate that you have a strong high-level understanding of how your additions will fit into and impact the existing code base

Will surely include these things in my proposal. Thanks again!

[sanketsingh24]

specularLight(v1, v2, v3)-

Currently, p5 only has support for specularMaterial(), and so, this method will also add the support for specular highlighting by light and not just by its material property. The specular component can be entirely different from the ambient component. Mainly, It will be useful in those scenarios when there are multiple lights and will give more control to the user. I will explain this in detail soon.

Arguments -

1. The color of the light

[sanketsingh24]

Edit: No longer implementing this, see below

• start/stopCamera() -

Using time, we can start and stop effects of current p5.camera methods like tilt(), setPosition() move(), pan(), lookAt(). See these cases -

1. startCamera( t ) / startInterval( t ) - starts the effects after t seconds. If none of the above methods are called before, the default camera() is used. After t seconds, the previous methods are stopped and subsequent methods are called.

   The usage can be like this.-

   cam1 = createCamera();
   cam1.setPosition(30, 0, 50);
   cam1.lookAt(0, 0, 0);
   cam1.startcamera(5);
   cam1.move(5,0,0);
   

   Here, after 5 seconds of rendering the sketch, the camera will start to move.

2. stopCamera( t ) / stopInterval( t ) - ends the effects after t seconds. If none of the above methods are called after, the default camera() is used. After t seconds, the previous methods are stopped and subsequent methods are called.

   cam1 = createCamera();
   cam1.setPosition(30, 0, 50);
   cam1.lookAt(0, 0, 0);
   cam1.startcamera(5);
   cam1.move(5,0,0);
   cam1.stopcamera(10);
   cam1.lookAt(10,0,0);
   

   Here, after 5 seconds of rendering the sketch, the camera will start to move. And then after 5 more seconds, it will stop, and will be oriented to look at (10,0,0),

At the moment, I cannot decide a name for this function, as start/stopCamera() is a kind of misnomer, and a little misdirecting, but as is start/stopInterval(). I will update as I come up with new ideas.

[sanketsingh24]

Hello all, I recently completed my First Draft of the proposal on the above Idea. Please correct me if there are any language errors, as my first language is not English . Also, you can comment on the lines about anything, and all comments are welcome ! Thanks for your time! @stalgiag @lmccart @kjhollen

EDIT: Will update the proposal soon w.r.t. this.

EDIT 2: I updated my proposal and added frustum

EDIT 3: I completed my proposal
Here is the link to my proposal.

[jeremydouglass]

@sanketsingh24

Here, after 5 seconds of rendering the sketch, the camera will start to move. And then after 5 more seconds, it will stop, and will be oriented to look at (10,0,0),

startCamera and stopCamera don’t make sense to me for time-based camera orientation events or path animation. p5.js animation fundamentally occurs in the draw loop – you can’t delay on one line and then execute the following line five seconds later, it doesn’t work that way. You have to be able to execute all the lines in draw every frame that renders to the screen. Maybe I’m not understanding what your examples above are proposing.

If you want to do path-based interpellation over time, then it would normally look like frameCount or millis() driving p5.Vector.lerp(), something like this:

EDIT: replaced pseudocode with actual p5.js sketch

let cam1;
let camStart, camLook, camEnd, camEndLook;

function setup() {
  createCanvas(400, 400, WEBGL);
  normalMaterial();
  cam1 = createCamera();
  camStart = createVector(30, 0, 50);
  camStartLook = createVector(0, 0, 0);
  camEnd = createVector(0, 50, 25);
  camEndLook = createVector(20, 0, 0);
  cam1.setPosition(camStart.x, camStart.y, camStart.z);
  cam1.lookAt(camStartLook.x, camStartLook.y, camStartLook.z);
}
function draw() {
  background(214);
  if (frameCount > 0 && frameCount <= 300) { // update cam for time period
    let amt = frameCount/300;
    let camPos = p5.Vector.lerp(camStart, camEnd, amt);
    cam1.setPosition(camPos.x, camPos.y, camPos.z);
    let camLook = p5.Vector.lerp(camStartLook, camEndLook, amt);
    cam1.lookAt(camLook.x, camLook.y, camLook.z);
  }
  box(10);
  translate(20, 0, 0);
  box(10);
}

Then if you wanted to rig tilt, pan, or anything else, you would do it in the same way.

This also gives you the freedom to decide whether you want to implement your lerp based on frame-independent input (millis) or frame-dependent input (frameCount), as these kinds of motion can give different results on different devices (WebGL - Animation).

An idea: If you wanted to create an advanced feature for managing timed camera movement, then one approach would be a CameraAnimation object that stores a series of frame-stamped camera matrices then can interpellate them and return a camera when passed a frameCount. You would program it with the standard camera methods (setPosition, lookAt) that took an additional time argument, and these would be used to generate its timestamped camera matrix stack.

Still, unless this is already built in to WebGL a camera animation class seems like quite an advanced feature – possibly its own separate library.

[sanketsingh24]

Hey @jeremydouglass, Thanks for the reply! And Sorry I could not answer earlier,

@jeremydouglass

startCamera and stopCamera don’t make sense to me for time-based camera orientation events or path animation. p5.js animation fundamentally occurs in the draw loop – you can’t delay on one line and then execute the following line five seconds later, it doesn’t work that way. You have to be able to execute all the lines in draw every frame that renders to the screen.

Ok, I firmly agree with you on this point. I think that’s right.

@jeremydouglass

If you want to do path-based interpellation over time, then it would normally look like frameCount or millis() driving p5.Vector.lerp(), something like this:

I got the idea upon seeing this example, and I thought about the problems which can arise, some of which are seen in the link you gave →

@jeremydouglass

This also gives you the freedom to decide whether you want to implement your lerp based on frame-independent input (millis) or frame-dependent input (frameCount), as these kinds of motion can give different results on different devices (WebGL - Animation).

I think using millis is probably a better idea, as it doesn’t depends on fps drops. Its implementation is the same as I would have implemented.

@jeremydouglass

An idea: If you wanted to create an advanced feature for managing timed camera movement, then one approach would be a CameraAnimation object that stores a series of frame-stamped camera matrices then can interpellate them and return a camera when passed a frameCount. You would program it with the standard camera methods (setPosition, lookAt) that took an additional time argument, and these would be used to generate its timestamped camera matrix stack.

I think this is a bigger Idea, and I won’t be able to complete my project in time If I would proceed to implement them .

I think what I proposed can already be done, So there is no point implementing it. I asked before the same question, but nobody answered, so I thought it is not possible. millis() can also be used for the camera related method, And while proposing this, It just didn’t come to my mind, I am extremely sorry for that.

[jeremydouglass]

@sanketsingh24

I think this is a bigger Idea, and I won’t be able to complete my project in time If I would proceed to implement them

That is fine! It is good to propose something that can actually be completed.

Yes, I think you could probably just drop the start/stop idea – as you can see, it is something that someone can already do in p5 with if + lerp.

[sanketsingh24]

frustum(left, right, bottom, top, zNear, zFar)-

A frustum is simply a pyramid, with its tip cut off. In this kind of projection, the viewer’s eye is placed at the tip of the pyramid, the other six planes act as the clipping planes. So, anything inside the clipping planes visible, and anything outside is not visible.

It is similar to glFrustum, but it replaces the current projection matrix with the new frustum projection matrix, whereas in glFrustum, which is a function of OpenGL, it multiplies the new frustum projection matrix with the current projection matrix.

Arguments -

1. left - The coordinate for the left-vertical clipping plane.

2. right - The coordinate for the right-vertical clipping plane.

3. bottom - The coordinate for the bottom-horizontal clipping plane.

4. top - The coordinate for the bottom-horizontal clipping plane.

5. zNear - The distances to the near-depth clipping plane. Must be positive.

6. zFar- The distances to the far-depth clipping planes. Must be positive.