# P5 for Spatial Computing / Mixed Reality / p5xr

**URL:** <https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043>\
**Category:** Libraries\
**Created:** [March 7, 2024, 3:50pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043 "2024-03-07T15:50:18Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [March 7, 2024, 3:50pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/1 "2024-03-07T15:50:18Z")

</div>

I’m exploring p5 and p5xr to do `Spatial Computing` or `Mixed Reality`.

Do you know any people who are working with p5 in 3D? Most examples use game engines like Unity or maybe Zach Lieberman’s work with ARKit and OpenFrameworks.

I have only found the [p5.xr](https://p5xr.org/#/) project by @stalgiag and related forum posts. I made some modifications to make it work through Oculus Quest 3 using passthrough, and I’m working on further examples and sketches.

I’m interested in making a better interface for sketching in 3D by having the headset as the camera and the environment light as a given; I think it could remove a lot of friction from using 3D, as Processing did back in the day with its simplified API.

---

<div class="post-metadata">

**Author:** ![stalgiag](https://avatars.discourse-cdn.com/v4/letter/s/f1d935/32.png) [@stalgiag](https://discourse.processing.org/u/stalgiag)\
**Post date:** [March 9, 2024, 8:16pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/2 "2024-03-09T20:16:53Z")

</div>

Hi! This looks great!

Unfortunately, progress has more or less stalled on development for p5.xr. I originally started the project because I wanted an easy way to introduce students to XR. I was teaching in environments without access to VR headsets and my hope was that students would be able to easily view their existing p5 sketches in simple cardboard viewers with their phones. At the time, it seemed like Apple was going to adopt the WebXR spec in Safari and in the interim, the Mozilla WebXR app was a fine workaround. Unfortunately, after years of indicating otherwise, Apple still hasn’t adopted the WebXR spec and Mozilla abandoned the development of the WebXR Viewer for iOS. For these reasons, I haven’t continued development. That said, now that Quest headsets have a pretty broad audience, I think that p5 could be used to make some great XR art. There have also been some great updates to the WebGL implementation in p5.js that make p5/p5.xr an even better candidate.

If you or anyone else who is doing active XR development with a headset is interested in becoming a steward of this project, please do let me know.

---

<div class="post-metadata">

**Author:** ![sableraph](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/sableraph/32/251_2.png) [@sableraph](https://discourse.processing.org/u/sableraph)\
**Post date:** [March 12, 2024, 11:07am UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/3 "2024-03-12T11:07:55Z")

</div>

Hi @TiborUdvari

I love this! 😍

I have been working on a tutorial about the `camera()` in p5.js and playing with [Zig Sim Pro](https://1-10.github.io/zigsim/getting-started.html) to map ARKit camera position to the p5.js camera (see the gif below). This is still very much a WIP but I’d be very curious to hear more about your research in AR with the Quest 3!

![output](https://canada1.discourse-cdn.com/flex036/uploads/processingfoundation1/original/3X/a/d/ad9efed6d391f8a110d590d37a1070d17f1d03cd.gif)

Maybe @davepagurek can tag other ppl doing cool work in 3D with p5.js?

---

<div class="post-metadata">

**Author:** ![davepagurek](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/davepagurek/32/17535_2.png) [@davepagurek](https://discourse.processing.org/u/davepagurek)\
**Post date:** [March 12, 2024, 1:09pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/4 "2024-03-12T13:09:24Z")

</div>

Hey, great work so far!

For lighting based on the scene: p5 recently added support for [(static) image-based lighting](https://p5js.org/reference/#/p5/imageLight) based on a single equirectangular image, so if you’re always in the same scene, you could potentially grab an image from the headset’s camera and then use that for the lighting going forward? One of the things I’d like to work on at some point is a more dynamic approach where you could sample the current frame of a video rather than using a single image though. I’m happy to help consult if anyone else is interested in taking that on!

---

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [March 13, 2024, 7:10pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/5 "2024-03-13T19:10:51Z")

</div>

Thank you for initiating this project!

I’m very interested in contributing more to the project. Look out for extra pull requests and activity on GitHub. I don’t know what it entails to be a project steward, but I would like to help.

I listed my [motivations](https://www.tiborudvari.com/blog/lowering-floor-spatial-computing-case-p5-mixed-reality) (draft) for why this is interesting, as it’s hard to explain quickly.

I think the main hiccup at the moment is having a way to quickly edit parameters. Ideally, I want to come up with a way to do this directly in `immersive` mode. Even though editing in the headset speeds things up, going in and out of the sketch adds extra friction, but this is probably something that is not the highest priority to address right now.

I will keep adding updates here occasionally; getting feedback on these sorts of things is important.

---

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [March 13, 2024, 7:16pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/6 "2024-03-13T19:16:25Z")

</div>

I quickly glanced at the [WebXR](https://github.com/immersive-web/lighting-estimation/blob/main/lighting-estimation-explainer.md) documentation about this, it seems the main way they return lighting data is through something called spherical harmonics, although there is also a cube map. I will have to check what is actually implemented and I might get back to you.

I’m in the process of better learning WebGL for this project, I see there is an implementation of this in [ThreeJS](https://threejs.org/docs/#examples/en/webxr/XREstimatedLight) already. I will look deeper into how it works and I will get back on that.

---

<div class="post-metadata">

**Author:** ![sableraph](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/sableraph/32/251_2.png) [@sableraph](https://discourse.processing.org/u/sableraph)\
**Post date:** [March 13, 2024, 9:03pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/7 "2024-03-13T21:03:39Z")

</div>

Super interesting, thank for sharing!

Talking about on-device editing, did you ever try [Rec Room’s “circuits”](https://www.youtube.com/watch?v=zkiyy9wwJ4c&list=PLvl3UKBiG_Dhbk1JjMb-5-Xol2uT6aelz)? I found it to be one of the best examples of VR-based programming. A similar system based on p5 to create immersive environments would be incredible.

 ![image](https://canada1.discourse-cdn.com/flex036/uploads/processingfoundation1/original/3X/a/b/ab02c921b44ed7ace44fe89fe947ded7829ac199.jpeg)

---

<div class="post-metadata">

**Author:** ![davepagurek](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/davepagurek/32/17535_2.png) [@davepagurek](https://discourse.processing.org/u/davepagurek)\
**Post date:** [March 14, 2024, 1:18pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/8 "2024-03-14T13:18:43Z")

</div>

It would be great to be able to swap out p5’s static image based lighting with one that can use the dynamic spherical harmonics data from WebXR!

Possibly one route to this is through a proposed API to be able to swap out small chunks of p5’s materials with your own code: [Make the WebGL material system more easy to make libraries for · Issue #6144 · processing/p5.js · GitHub](https://github.com/processing/p5.js/issues/6144#issuecomment-1962677182) With that, you could in theory replace the function that uses the static equirectangular lookup with something to evaluate the spherical harmonics. That’s all still in progress though, but hopefully would unblock plugins like p5.xr from adding support for the things you’ve mentioned.

---

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [March 26, 2024, 3:22pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/9 "2024-03-26T15:22:40Z")

</div>

I managed to get live coding working with p5.live by Ted Davis / @ffd8 . Sometimes it still crashes out into normal mode, but it is very workable for the most part.

---

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [April 20, 2024, 12:14pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/10 "2024-04-20T12:14:55Z")

</div>

Here are some updates, I am currently working on hand tracking:

---

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [April 20, 2024, 12:15pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/11 "2024-04-20T12:15:44Z")

</div>



---

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [April 20, 2024, 12:16pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/12 "2024-04-20T12:16:30Z")

</div>



---

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [April 20, 2024, 12:18pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/13 "2024-04-20T12:18:55Z")

</div>



---

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [April 20, 2024, 12:22pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/14 "2024-04-20T12:22:48Z")

</div>

It’s an interesting challenge to come up with a p5 like interface that is friendly for beginners. The WebXR spec is very complete, but most of the features are probably not that useful for quick interactions, sketching.

For example I came up with the idea of having a main finger, the index behave somewhat like mouseX, mouseY for quick interactions.

Of course it also needs to take into account handedness, ie. left handed people or right handed people.

For example the sphere on the finger boils down to:

```js
function preload() {
  createARCanvas();
}

function setup() {
  describe("A sphere on your right index finger");
  mainHandMode(RIGHT);
}

function draw() {
  normalMaterial();
  push();
  translate(finger.x, finger.y, finger.z);
  sphere(0.01);
  pop();
}

```

---

<div class="post-metadata">

**Author:** ![sableraph](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/sableraph/32/251_2.png) [@sableraph](https://discourse.processing.org/u/sableraph)\
**Post date:** [April 24, 2024, 12:05pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/15 "2024-04-24T12:05:16Z")

</div>

This is great! Is there a beta version we can try out?

---

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [April 29, 2024, 12:49pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/16 "2024-04-29T12:49:49Z")

</div>

I’m putting my experiments here: [https://www.tiborudvari.com/sketchbook](https://www.tiborudvari.com/sketchbook)

I’ve been working on an upload script to the web editor.  
I should get it working in the next couple of days, there will be links where you can try out the code as well.

---

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [May 14, 2024, 3:10pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/17 "2024-05-14T15:10:37Z")

</div>

The sketchbook is updated with links to the editor as well to be able to change code.

---

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [May 14, 2024, 3:11pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/18 "2024-05-14T15:11:13Z")

</div>

Here are some other examples

With live coding the positions

---

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [May 14, 2024, 3:11pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/19 "2024-05-14T15:11:50Z")

</div>

Working with finger interactions

---

<div class="post-metadata">

**Author:** ![TiborUdvari](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/tiborudvari/32/18018_2.png) [@TiborUdvari](https://discourse.processing.org/u/TiborUdvari)\
**Post date:** [May 14, 2024, 3:12pm UTC](https://discourse.processing.org/t/p5-for-spatial-computing-mixed-reality-p5xr/44043/20 "2024-05-14T15:12:16Z")

</div>

Indexing