Processing2Hologram — a Processing 4 library for Looking Glass holographic displays

Processing2Hologram — a Processing 4 library for Looking Glass holographic displays

Hi everyone!

I’d like to share a small open-source project I’ve been working on: Processing2Hologram, a Processing 4 library that lets you render regular P3D scenes directly to Looking Glass holographic displays.

GitHub: GitHub - ShinghoiXu/Processing2Hologram: A Library that Turns PGraphics into Quilt Images for Looking Glass Hologram · GitHub

The main idea is to keep the workflow as close to normal Processing as possible. You draw your scene using familiar PGraphics, shapes, lights, materials, textures, and transforms, and the library takes care of generating the calibrated multiview cameras, assembling the quilt, and sending it to Looking Glass Bridge.

A minimal sketch looks roughly like this:

import processing2hologram.*;

LookingGlass hologram;
float angle;

void setup() {
  size(900, 700, P3D);
  hologram = new LookingGlass(this);
}

void draw() {
  angle += 0.012;
  hologram.render(this::drawScene);

  image(hologram.preview(), 0, 0, width, height);
}

void drawScene(PGraphics pg) {
  pg.background(15, 18, 28);
  pg.lights();

  pg.pushMatrix();
  pg.translate(pg.width * 0.5, pg.height * 0.5);
  pg.rotateY(angle);
  pg.box(150);
  pg.popMatrix();
}

drawScene() is rendered once for every view in the quilt, while your animation/simulation state is updated only once per Processing frame.

The library currently supports:

  • standard Processing P3D rendering

  • calibrated multiview rendering from a single center camera

  • automatic quilt layout and view-cone configuration from the connected Looking Glass

  • direct presentation through Looking Glass Bridge

  • Windows x64 and macOS

  • center-view preview without any Looking Glass hardware

  • full quilt preview and quilt PNG export

  • adjustable holographic depth and Bridge-side zoom

One thing I particularly wanted was for the library to remain useful without owning or having the display connected. If Bridge or the hardware isn’t available, the same sketch simply runs in preview mode, so you can still develop the scene and inspect the complete quilt.

I’ve also included several examples beyond the basic cube:

Flocking — 128 interactive 3D boids
AuroraRibbons — procedural ribbons, particles, and transparent geometry
KineticBloom — a mechanical flower demonstrating lights, materials, and nested transforms
SolarSystem — an eight-planet system with an Earth–Moon pair, asteroid belt, and Saturn rings
WaveGarden — an interactive triangle-strip terrain
DepthPlayground — an interactive demo for understanding holographic depth controls

The first hardware target I’ve validated extensively is Looking Glass Portrait, but the implementation does not hard-code Portrait’s optical configuration. When a device is connected, the library asks Bridge for its recommended quilt dimensions, grid, view cone, and device information.

The project is still quite young, so I’d really appreciate feedback from Processing users — especially anyone experimenting with Looking Glass, multiview rendering, creative coding, or unusual display systems.

If you have a Looking Glass and are willing to test it on different models/hardware configurations, that would also be extremely helpful.

You can download the ready-to-use universal package from the latest GitHub Release:

Just place the extracted Processing2Hologram folder inside your Processing Sketchbook’s libraries directory and restart Processing.

Hope this is useful to someone here — and I’d love to see what people make with it!

Chengkai Xu

When downloading the .zip file on the link that you provided the name of the .jar file is incorrect:

It should be just Processing2Hologram.jar and not as shown above.

The demo source code won’t run if used as is.

Hello @ChengkaiXu,

Cool beans!

Thank you for sharing!

I will most certainly be getting a Looking Glass holographic display in the future now!

The library works as is with Windows 11 and Processing 4.5.6

:)

I did not have issues installing or using this library with Windows 11 as is.

There are install instructions here:

Is this a MacOS related issue?

:)

There is a difference in the way MacOS and Windows handles .zip files. When I download a .zip file on a mac I get an unzipped folder containing a bunch of files including a .jar file like I showed above, i.e, the .jar fileName is a url to a folder system that was not created. When you download a .zip file on Windows you have to browse the location that you want to extract it to. If you choose the Processing/Document/libraries folder it automatically creates the folder system with all the files where they are supposed to be and named correctly; that doesn’t happen on a mac and you have to rename the .jar file and manually create the file system. I can confirm that the demo runs as advertised on a Windows 11 system.

Here is an screenshot of the contents of a downloaded Processing2Hologram automatically unzipped folder on a mac:

Hello @ChengkaiXu ,

I opened an issue on your GitHub page related to the zip archives and compatibility with other OSes.

Thanks for opening the issue and for investigating this!

I tested the current release ZIP again on my own Mac. In my case, simply double-clicking the ZIP in Finder and extracting it with the built-in Archive Utility creates the expected folder structure correctly — including Processing2Hologram/library/Processing2Hologram.jar — so unfortunately I haven’t been able to reproduce the behavior shown in @svan’s screenshot.

The GitHub issue does point out something important, though: the ZIP entries appear to use backslashes (\) rather than forward slashes (/) internally. According to the ZIP specification, forward slashes should be used for compatibility with Unix-like systems. So even though macOS Archive Utility handles the archive correctly on my machine, this is still something I should fix in the packaging process.

I’ve created a Temporary test ZIP test with the archive paths generated in the other way. You may access that with the link. If you’re on macOS, it would be very helpful if you could try extracting this version and let me know whether the folder structure is now created correctly.

In particular, after extraction it should look like:

Processing2Hologram/library/Processing2Hologram.jar

rather than files with names such as:

Processing2Hologram\library\Processing2Hologram.jar

At the moment, I’m not sure whether the original extraction issue is specific to a particular macOS version/environment or whether some Macs handle these archives differently. Testing this temporary ZIP on a few different Macs should help confirm whether the path separator is indeed the cause.

Thanks @glv and @svan for testing and reporting this!

On my macOS system that one works as expected; a folder structure is created which could be drag and dropped into the Processing/Document/libraries folder. With the original link all you have to do is click on it once. Double clicking does nothing but download it; I don’t have the opportunity to use an Archiver (MacOS M4 with Tahoe 26.5 operating system).

Thanks for testing this on your M4 Mac — that confirms it!

The issue was caused by the original ZIP using Windows-style backslashes (\) inside its archive entry paths. Some extractors tolerate this and reconstruct the directory structure correctly, which is why I couldn’t reproduce it on my own Mac, while other macOS environments can end up treating the backslashes as literal filename characters.

I’ve now fixed the packaging process and closed the GitHub issue. The latest v0.2.0 ZIP uses forward slashes (/) for all archive paths, so downloading the latest release should work correctly on macOS as well.

For anyone interested, I found a couple of very similar cases while investigating this:

So this turned out to be a useful little cross-platform packaging edge case. Thanks @svan and @glv for helping track it down!

And of course, if you find Processing2Hologram useful or interesting, a :star: on the GitHub repository would be very much appreciated. :slight_smile:

hi,
testing it with a lookinglass portrait and it s amazing !!!
i didn’t expect 48 point of view will render at full frame rate, it s very smooth (and beautiful example too)
thanks a lot for sharing this :face_blowing_a_kiss:

hi, i tried to get interaction calculated from the mouse position inside the hologram and not the main screen preview.
it works with those modifications:

import processing2hologram.*;

  /* added */
import java.awt.MouseInfo;
import java.awt.Point;
import java.awt.GraphicsDevice;
import java.awt.GraphicsEnvironment;
PVector LookingGlassOrigin, LookingGlassMouse;
  /* end*/

final int BOID_COUNT = 128;

final float CAMERA_FOV = 42;
final float CAMERA_NEAR_CLIP = 1;
final float CAMERA_FAR_CLIP = 4000;

final float DEPTH_PLANE_DISTANCE = 300;
final float DEPTH_REPULSION_ZONE = 60;
final float DEPTH_ATTRACTION = 0.0011;
final float DEPTH_REPULSION = 0.005;
final float DEPTH_DAMPING = 0.0006;
final float BOID_VISUAL_RADIUS = 24;
final float SPAWN_OUTSIDE_PADDING = 18;

final float DESIRED_SEPARATION = 32;
final float NEIGHBOR_DISTANCE = 76;
final float POINTER_RADIUS = 300;

LookingGlass hologram;
Boid[] flock = new Boid[BOID_COUNT];

int lastFrameMillis;
int lastPointerMillis = -10000;
float simulationTime;
boolean paused;

PVector cameraEye = new PVector(0, -35, 875);
PVector cameraTarget = new PVector(0, 0, 0);
PVector cameraForward = new PVector();
PVector cameraRight = new PVector();
PVector cameraUp = new PVector();
PVector pointerWorld = new PVector();
float cameraFocusDistance;

void setup() {
  // One Portrait quilt view is 420 x 560. fitWindowToPreview() adapts this
  // window if a connected Looking Glass reports a different aspect ratio.
  size(420, 560, P3D);
  surface.setTitle("Processing2Hologram - Interactive Flocking");

  hologram = new LookingGlass(this);
  hologram.camera()
    .fov(CAMERA_FOV)
    .clip(CAMERA_NEAR_CLIP, CAMERA_FAR_CLIP)
    .depthScale(1.05)
    .lookAt(
    cameraEye.x, cameraEye.y, cameraEye.z,
    cameraTarget.x, cameraTarget.y, cameraTarget.z
    );

  updateCameraBasis();
  resetFlock();
  lastFrameMillis = millis();
/* added */
  LookingGlassOrigin=getDisplayXY (  );  
  LookingGlassMouse=new PVector(0, 0);
/* end */
}

void draw() {
  if (frameCount == 1) fitWindowToPreview();

  int now = millis();
  float simulationSteps = constrain((now - lastFrameMillis) / (1000.0 / 60.0), 0.1, 2.4);
  lastFrameMillis = now;

  boolean pointerActive = true;/*now - lastPointerMillis < 1100;
   if (mousePressed) {
   pointerActive = true;
   lastPointerMillis = now;
   }*/

  /* added */
  Point screenPt = MouseInfo.getPointerInfo().getLocation();

  LookingGlassMouse.x=(screenPt.x-LookingGlassOrigin.x)/3.5;
  LookingGlassMouse.y=(screenPt.y-LookingGlassOrigin.y)/3.5;
  /* end*/

  updatePointerWorld();

  if (!paused) {
    simulationTime += simulationSteps / 60.0;
    updateFlock(simulationSteps, pointerActive);
  }

  // The simulation above advances once. drawFlock() only renders the immutable
  // snapshot, because LookingGlass calls it once for every quilt view.
  hologram.render(this::drawFlock);

  background(5, 9, 16);
  image(hologram.preview(), 0, 0, width, height);
  drawOverlay(pointerActive);
}

  /* added */
PVector getDisplayXY (  )
{
  PVector origin=new PVector();
  final GraphicsDevice device = GraphicsEnvironment.getLocalGraphicsEnvironment ().getDefaultScreenDevice ();
  final GraphicsDevice[] devices = GraphicsEnvironment.getLocalGraphicsEnvironment ().getScreenDevices ();
  for ( final GraphicsDevice d : devices )
  {
    if ( d != device )

    {
      System.out.println ( "Secondary screen bounds: " + d.getDefaultConfiguration ().getBounds () );
      origin.x=(int)d.getDefaultConfiguration().getBounds().getX();
      origin.y=(int)d.getDefaultConfiguration().getBounds().getY();
    }
  }
  return origin;
}
void updateCameraBasis() {
  cameraForward.set(cameraTarget);
  cameraForward.sub(cameraEye);
  cameraFocusDistance = cameraForward.mag();
  cameraForward.normalize();

  cameraRight.set(cameraForward.cross(new PVector(0, 1, 0)));
  if (cameraRight.magSq() < 0.0001) cameraRight.set(1, 0, 0);
  cameraRight.normalize();

  cameraUp.set(cameraRight.cross(cameraForward));
  cameraUp.normalize();
}

// Convert the mouse position into a point on the fixed camera's focus plane.
void updatePointerWorld() {
  float nx = map(constrain(LookingGlassMouse.x, 0, width), 0, max(1, width), -1, 1);
  float ny = map(constrain(LookingGlassMouse.y, 0, height), 0, max(1, height), -1, 1);
  float halfHeight = tan(radians(CAMERA_FOV * 0.5)) * cameraFocusDistance;
  float halfWidth = halfHeight * hologram.quiltSettings().viewAspect();

  pointerWorld.set(cameraTarget);
  pointerWorld.add(PVector.mult(cameraRight, nx * halfWidth));
  pointerWorld.add(PVector.mult(cameraUp, ny * halfHeight));
}

void updateFlock(float step, boolean pointerActive) {
  float separationSq = DESIRED_SEPARATION * DESIRED_SEPARATION;
  float neighborSq = NEIGHBOR_DISTANCE * NEIGHBOR_DISTANCE;
  float pointerRadiusSq = POINTER_RADIUS * POINTER_RADIUS;

  for (int i = 0; i < flock.length; i++) {
    Boid boid = flock[i];

    float sepX = 0;
    float sepY = 0;
    float sepZ = 0;
    float alignX = 0;
    float alignY = 0;
    float alignZ = 0;
    float centerX = 0;
    float centerY = 0;
    float centerZ = 0;
    int separationCount = 0;
    int neighborCount = 0;

    for (int j = 0; j < flock.length; j++) {
      if (i == j) continue;
      Boid other = flock[j];

      float dx = boid.position.x - other.position.x;
      float dy = boid.position.y - other.position.y;
      float dz = boid.position.z - other.position.z;
      float distanceSq = dx * dx + dy * dy + dz * dz;
      if (distanceSq < 0.0001) continue;

      if (distanceSq < separationSq) {
        sepX += dx / distanceSq;
        sepY += dy / distanceSq;
        sepZ += dz / distanceSq;
        separationCount++;
      }

      if (distanceSq < neighborSq) {
        alignX += other.velocity.x;
        alignY += other.velocity.y;
        alignZ += other.velocity.z;
        centerX += other.position.x;
        centerY += other.position.y;
        centerZ += other.position.z;
        neighborCount++;
      }
    }

    boid.acceleration.set(0, 0, 0);

    if (separationCount > 0) {
      addSteering(
        boid,
        sepX / separationCount,
        sepY / separationCount,
        sepZ / separationCount,
        1.55
        );
    }

    if (neighborCount > 0) {
      addSteering(
        boid,
        alignX / neighborCount,
        alignY / neighborCount,
        alignZ / neighborCount,
        1.0
        );
      addSteering(
        boid,
        centerX / neighborCount - boid.position.x,
        centerY / neighborCount - boid.position.y,
        centerZ / neighborCount - boid.position.z,
        0.92
        );
    }

    // A different phase for every boid prevents the flock from becoming rigid.
    boid.acceleration.x += sin(simulationTime * 0.63 + boid.phase * 1.7) * 0.0030;
    boid.acceleration.y += cos(simulationTime * 0.57 + boid.phase * 1.25) * 0.0030;
    boid.acceleration.z += sin(simulationTime * 0.49 + boid.phase * 2.1) * 0.0030;

    if (!boid.hasEnteredView) {
      addIngressSteering(boid);
    }

    addDepthForces(boid);

    if (pointerActive) {
      float dx = pointerWorld.x - boid.position.x;
      float dy = pointerWorld.y - boid.position.y;
      float dz = pointerWorld.z - boid.position.z;
      float distanceSq = dx * dx + dy * dy + dz * dz;

      if (distanceSq > 0.0001 && distanceSq < pointerRadiusSq) {
        float distance = sqrt(distanceSq);
        float falloff = 1.0 - distance / POINTER_RADIUS;
        // Moving the pointer attracts. Holding the mouse pushes the flock away.
        float strength = mousePressed ? -0.135 : 0.078;
        strength *= 0.3 + 0.7 * falloff;
        boid.acceleration.x += dx / distance * strength;
        boid.acceleration.y += dy / distance * strength;
        boid.acceleration.z += dz / distance * strength;
      }
    }

    boid.velocity.x += boid.acceleration.x * step;
    boid.velocity.y += boid.acceleration.y * step;
    boid.velocity.z += boid.acceleration.z * step;
    limit(boid.velocity, boid.maxSpeed);

    boid.position.x += boid.velocity.x * step;
    boid.position.y += boid.velocity.y * step;
    boid.position.z += boid.velocity.z * step;

    boolean fullyOutside = isCompletelyOutsideView(boid);
    if (!boid.hasEnteredView) {
      // A recycled boid is allowed to travel in from its off-screen spawn.
      if (!fullyOutside) boid.hasEnteredView = true;
    } else if (fullyOutside) {
      // Destroy exactly one departed boid and replace it just outside a random
      // edge, aimed roughly at the focal center so it enters without popping.
      flock[i] = new Boid(true);
    }
  }
}

void addSteering(Boid boid, float desiredX, float desiredY, float desiredZ, float weight) {
  float desiredLength = sqrt(
    desiredX * desiredX + desiredY * desiredY + desiredZ * desiredZ
    );
  if (desiredLength < 0.0001) return;

  float desiredScale = boid.maxSpeed / desiredLength;
  float steerX = desiredX * desiredScale - boid.velocity.x;
  float steerY = desiredY * desiredScale - boid.velocity.y;
  float steerZ = desiredZ * desiredScale - boid.velocity.z;
  float steerLength = sqrt(steerX * steerX + steerY * steerY + steerZ * steerZ);

  if (steerLength > boid.maxForce) {
    float forceScale = boid.maxForce / steerLength;
    steerX *= forceScale;
    steerY *= forceScale;
    steerZ *= forceScale;
  }

  boid.acceleration.x += steerX * weight;
  boid.acceleration.y += steerY * weight;
  boid.acceleration.z += steerZ * weight;
}

// Guide recycled boids into a slowly wandering region around screen center.
// Keeping this correction in the camera's XY plane avoids another depth pull.
void addIngressSteering(Boid boid) {
  float fromEyeX = boid.position.x - cameraEye.x;
  float fromEyeY = boid.position.y - cameraEye.y;
  float fromEyeZ = boid.position.z - cameraEye.z;
  float viewX = fromEyeX * cameraRight.x
    + fromEyeY * cameraRight.y
    + fromEyeZ * cameraRight.z;
  float viewY = fromEyeX * cameraUp.x
    + fromEyeY * cameraUp.y
    + fromEyeZ * cameraUp.z;
  float wanderX = boid.ingressOffsetX
    + sin(simulationTime * 0.71 + boid.phase * 1.3) * 42;
  float wanderY = boid.ingressOffsetY
    + cos(simulationTime * 0.63 + boid.phase * 1.7) * 52;
  float desiredViewX = wanderX - viewX;
  float desiredViewY = wanderY - viewY;
  float desiredLength = sqrt(
    desiredViewX * desiredViewX + desiredViewY * desiredViewY
    );
  if (desiredLength < 0.0001) return;

  float desiredScale = boid.maxSpeed / desiredLength;
  float velocityViewX = boid.velocity.x * cameraRight.x
    + boid.velocity.y * cameraRight.y
    + boid.velocity.z * cameraRight.z;
  float velocityViewY = boid.velocity.x * cameraUp.x
    + boid.velocity.y * cameraUp.y
    + boid.velocity.z * cameraUp.z;
  float steerX = desiredViewX * desiredScale - velocityViewX;
  float steerY = desiredViewY * desiredScale - velocityViewY;
  float steerLength = sqrt(steerX * steerX + steerY * steerY);
  if (steerLength > boid.maxForce) {
    float forceScale = boid.maxForce / steerLength;
    steerX *= forceScale;
    steerY *= forceScale;
  }

  float weight = 0.42;
  boid.acceleration.x += (cameraRight.x * steerX + cameraUp.x * steerY) * weight;
  boid.acceleration.y += (cameraRight.y * steerX + cameraUp.y * steerY) * weight;
  boid.acceleration.z += (cameraRight.z * steerX + cameraUp.z * steerY) * weight;
}

// A very weak spring and damping gently favor the focal plane. Two soft walls
// add a small quadratic repulsion near the front and back depth limits.
void addDepthForces(Boid boid) {
  float fromFocusX = boid.position.x - cameraTarget.x;
  float fromFocusY = boid.position.y - cameraTarget.y;
  float fromFocusZ = boid.position.z - cameraTarget.z;
  float depthOffset = fromFocusX * cameraForward.x
    + fromFocusY * cameraForward.y
    + fromFocusZ * cameraForward.z;
  float depthVelocity = boid.velocity.x * cameraForward.x
    + boid.velocity.y * cameraForward.y
    + boid.velocity.z * cameraForward.z;

  float depthForce = -depthOffset / DEPTH_PLANE_DISTANCE * DEPTH_ATTRACTION;
  depthForce -= depthVelocity * DEPTH_DAMPING;

  float repulsionStart = DEPTH_PLANE_DISTANCE - DEPTH_REPULSION_ZONE;
  float distanceFromCenter = abs(depthOffset);
  if (distanceFromCenter > repulsionStart) {
    float penetration = constrain(
      (distanceFromCenter - repulsionStart) / DEPTH_REPULSION_ZONE,
      0,
      2
      );
    float directionToCenter = depthOffset < 0 ? 1 : -1;
    depthForce += directionToCenter * penetration * penetration * DEPTH_REPULSION;
  }

  boid.acceleration.x += cameraForward.x * depthForce;
  boid.acceleration.y += cameraForward.y * depthForce;
  boid.acceleration.z += cameraForward.z * depthForce;
}

// Use the fixed center camera's actual view frustum. The visual-radius margin
// means recycling only happens after the entire wireframe has left the view.
boolean isCompletelyOutsideView(Boid boid) {
  float fromEyeX = boid.position.x - cameraEye.x;
  float fromEyeY = boid.position.y - cameraEye.y;
  float fromEyeZ = boid.position.z - cameraEye.z;
  float cameraDepth = fromEyeX * cameraForward.x
    + fromEyeY * cameraForward.y
    + fromEyeZ * cameraForward.z;

  if (cameraDepth + BOID_VISUAL_RADIUS < CAMERA_NEAR_CLIP) return true;
  if (cameraDepth - BOID_VISUAL_RADIUS > CAMERA_FAR_CLIP) return true;
  // A boid whose center crossed the eye plane can still have a visible tip.
  if (cameraDepth <= 0) return false;

  float viewX = fromEyeX * cameraRight.x
    + fromEyeY * cameraRight.y
    + fromEyeZ * cameraRight.z;
  float viewY = fromEyeX * cameraUp.x
    + fromEyeY * cameraUp.y
    + fromEyeZ * cameraUp.z;
  float halfHeight = tan(radians(CAMERA_FOV * 0.5)) * cameraDepth;
  float halfWidth = halfHeight * hologram.quiltSettings().viewAspect();

  return abs(viewX) > halfWidth + BOID_VISUAL_RADIUS
    || abs(viewY) > halfHeight + BOID_VISUAL_RADIUS;
}

void spawnBoidInView(Boid boid) {
  float depthOffset = random(
    -DEPTH_PLANE_DISTANCE * 0.72,
    DEPTH_PLANE_DISTANCE * 0.72
    );
  float cameraDepth = cameraFocusDistance + depthOffset;
  float halfHeight = tan(radians(CAMERA_FOV * 0.5)) * cameraDepth;
  float halfWidth = halfHeight * hologram.quiltSettings().viewAspect();
  float spawnX = random(
    -max(1, halfWidth - BOID_VISUAL_RADIUS) * 0.88,
    max(1, halfWidth - BOID_VISUAL_RADIUS) * 0.88
    );
  float spawnY = random(
    -max(1, halfHeight - BOID_VISUAL_RADIUS) * 0.88,
    max(1, halfHeight - BOID_VISUAL_RADIUS) * 0.88
    );

  setBoidViewPosition(boid, cameraDepth, spawnX, spawnY);
  giveBoidRandomVelocity(boid);
  boid.hasEnteredView = true;
}

void spawnBoidOutsideView(Boid boid) {
  float depthOffset = random(
    -DEPTH_PLANE_DISTANCE * 0.72,
    DEPTH_PLANE_DISTANCE * 0.72
    );
  float cameraDepth = cameraFocusDistance + depthOffset;
  float halfHeight = tan(radians(CAMERA_FOV * 0.5)) * cameraDepth;
  float halfWidth = halfHeight * hologram.quiltSettings().viewAspect();
  float outsideOffset = BOID_VISUAL_RADIUS + SPAWN_OUTSIDE_PADDING;
  float spawnX;
  float spawnY;

  if (random(1) < 0.5) {
    spawnX = random(1) < 0.5
      ? -halfWidth - outsideOffset
      : halfWidth + outsideOffset;
    spawnY = random(-halfHeight * 0.82, halfHeight * 0.82);
  } else {
    spawnX = random(-halfWidth * 0.82, halfWidth * 0.82);
    spawnY = random(1) < 0.5
      ? -halfHeight - outsideOffset
      : halfHeight + outsideOffset;
  }

  setBoidViewPosition(boid, cameraDepth, spawnX, spawnY);
  boid.ingressOffsetX = random(-halfWidth * 0.22, halfWidth * 0.22);
  boid.ingressOffsetY = random(-halfHeight * 0.22, halfHeight * 0.22);
  aimBoidAtFocalCenter(boid);
  boid.hasEnteredView = false;
}

void setBoidViewPosition(Boid boid, float cameraDepth, float viewX, float viewY) {
  boid.position.set(cameraEye);
  boid.position.x += cameraForward.x * cameraDepth
    + cameraRight.x * viewX + cameraUp.x * viewY;
  boid.position.y += cameraForward.y * cameraDepth
    + cameraRight.y * viewX + cameraUp.y * viewY;
  boid.position.z += cameraForward.z * cameraDepth
    + cameraRight.z * viewX + cameraUp.z * viewY;
}

void aimBoidAtFocalCenter(Boid boid) {
  float aimX = cameraTarget.x - boid.position.x;
  float aimY = cameraTarget.y - boid.position.y;
  float aimZ = cameraTarget.z - boid.position.z;
  float aimLength = sqrt(aimX * aimX + aimY * aimY + aimZ * aimZ);
  if (aimLength < 0.0001) {
    aimX = cameraForward.x;
    aimY = cameraForward.y;
    aimZ = cameraForward.z;
    aimLength = 1;
  }

  aimX /= aimLength;
  aimY /= aimLength;
  aimZ /= aimLength;
  float sidewaysJitter = random(-0.32, 0.32);
  float verticalJitter = random(-0.32, 0.32);
  float depthJitter = random(-0.12, 0.12);
  aimX += cameraRight.x * sidewaysJitter
    + cameraUp.x * verticalJitter + cameraForward.x * depthJitter;
  aimY += cameraRight.y * sidewaysJitter
    + cameraUp.y * verticalJitter + cameraForward.y * depthJitter;
  aimZ += cameraRight.z * sidewaysJitter
    + cameraUp.z * verticalJitter + cameraForward.z * depthJitter;
  float jitteredLength = sqrt(aimX * aimX + aimY * aimY + aimZ * aimZ);
  float speed = random(1.0, boid.maxSpeed);
  boid.velocity.set(
    aimX / jitteredLength * speed,
    aimY / jitteredLength * speed,
    aimZ / jitteredLength * speed
    );
  boid.acceleration.set(0, 0, 0);
}

void giveBoidRandomVelocity(Boid boid) {
  float velocityX;
  float velocityY;
  float velocityZ;
  float lengthSq;
  do {
    velocityX = random(-1, 1);
    velocityY = random(-1, 1);
    velocityZ = random(-1, 1);
    lengthSq = velocityX * velocityX + velocityY * velocityY + velocityZ * velocityZ;
  } while (lengthSq < 0.001);

  float speed = random(1.0, boid.maxSpeed) / sqrt(lengthSq);
  boid.velocity.set(velocityX * speed, velocityY * speed, velocityZ * speed);
  boid.acceleration.set(0, 0, 0);
}

void limit(PVector vector, float maximum) {
  float lengthSq = vector.magSq();
  if (lengthSq > maximum * maximum) {
    vector.mult(maximum / sqrt(lengthSq));
  }
}

void drawFlock(PGraphics pg) {
  pg.background(9, 15, 27);
  pg.colorMode(HSB, 360, 100, 100, 100);
  pg.noFill();
  pg.blendMode(ADD);
  pg.hint(DISABLE_DEPTH_MASK);

  for (Boid boid : flock) {
    pg.pushMatrix();
    pg.translate(boid.position.x, boid.position.y, boid.position.z);
    orientAlongVelocity(pg, boid.velocity);
    pg.rotateZ(boid.phase + simulationTime * 0.17);

    float pulse = 0.5 + 0.5 * sin(simulationTime * 1.8 + boid.phase);
    float scale = 0.38 + boid.baseScale * (0.22 + pulse * 0.78);
    pg.scale(scale);

    pg.stroke(boid.hue, 90, 100, 82);
    pg.strokeWeight(1.25 / max(0.55, scale));
    drawWireCone(pg);
    pg.popMatrix();
  }

  pg.hint(ENABLE_DEPTH_MASK);
  pg.blendMode(BLEND);
  pg.colorMode(RGB, 255);
}

void orientAlongVelocity(PGraphics pg, PVector velocity) {
  float speed = velocity.mag();
  if (speed < 0.0001) return;

  float dx = velocity.x / speed;
  float dy = velocity.y / speed;
  float dz = constrain(velocity.z / speed, -1, 1);
  float angle = acos(dz);

  // Cross product of local +Z and the desired direction.
  float axisX = -dy;
  float axisY = dx;
  float axisLength = sqrt(axisX * axisX + axisY * axisY);

  if (axisLength > 0.0001) {
    pg.rotate(angle, axisX / axisLength, axisY / axisLength, 0);
  } else if (dz < 0) {
    pg.rotateX(PI);
  }
}

void drawWireCone(PGraphics pg) {
  float radius = 4.68;
  float baseZ = -7.02;
  float tipZ = 10.92;

  pg.beginShape(LINES);

  // Diamond-shaped base.
  pg.vertex(radius, 0, baseZ);
  pg.vertex(0, radius, baseZ);
  pg.vertex(0, radius, baseZ);
  pg.vertex(-radius, 0, baseZ);
  pg.vertex(-radius, 0, baseZ);
  pg.vertex(0, -radius, baseZ);
  pg.vertex(0, -radius, baseZ);
  pg.vertex(radius, 0, baseZ);

  // Four edges converge at the forward tip.
  pg.vertex(radius, 0, baseZ);
  pg.vertex(0, 0, tipZ);
  pg.vertex(0, radius, baseZ);
  pg.vertex(0, 0, tipZ);
  pg.vertex(-radius, 0, baseZ);
  pg.vertex(0, 0, tipZ);
  pg.vertex(0, -radius, baseZ);
  pg.vertex(0, 0, tipZ);

  pg.endShape();
}

void drawOverlay(boolean pointerActive) {
  hint(DISABLE_DEPTH_TEST);
  camera();
  noLights();

  noStroke();
  fill(4, 8, 16, 196);
  rect(12, 12, width - 24, 72, 10);

  fill(255);
  textAlign(LEFT, TOP);
  textSize(14);
  text("INTERACTIVE FLOCKING "+LookingGlassMouse, 24, 23);

  fill(185, 205, 225);
  textSize(11);
  String interaction = mousePressed
    ? "Repelling flock - release to attract"
    : pointerActive
    ? "Attracting flock - hold mouse to repel"
    : "Move mouse to attract - hold to repel";
  text(interaction, 24, 46);
  text("SPACE pause/resume   R reset", 24, 63);

  fill(4, 8, 16, 174);
  rect(12, height - 36, width - 24, 24, 8);
  fill(150, 174, 198);
  text(
    hologram.isConnected() ? "Looking Glass connected" : "Portrait quilt preview (Bridge offline)",
    24,
    height - 30
    );

  hint(ENABLE_DEPTH_TEST);
}

void resetFlock() {
  for (int i = 0; i < flock.length; i++) {
    flock[i] = new Boid(false);
  }
}

void fitWindowToPreview() {
  PGraphics preview = hologram.preview();
  float scale = min(
    1,
    min(displayWidth * 0.8 / preview.width, displayHeight * 0.8 / preview.height)
    );
  int newHeight = max(128, round(preview.height * scale));
  int newWidth = max(128, round(newHeight * preview.width / (float) preview.height));
  if (newWidth != width || newHeight != height) {
    surface.setSize(newWidth, newHeight);
  }
}

void mouseMoved() {
  lastPointerMillis = millis();
}

void mouseDragged() {
  lastPointerMillis = millis();
}

void mousePressed() {
  lastPointerMillis = millis();
}

void keyPressed() {
  if (key == ' ') {
    paused = !paused;
  } else if (key == 'r' || key == 'R') {
    resetFlock();
  }
}

class Boid {
  float hue = random(360);
  float baseScale = random(0.78, 1.72);
  float phase = random(TWO_PI);
  float maxSpeed = random(2.15, 3.05);
  float maxForce = random(0.034, 0.058);

  PVector position = new PVector();
  PVector velocity = new PVector();
  PVector acceleration = new PVector();
  boolean hasEnteredView;
  float ingressOffsetX;
  float ingressOffsetY;

  Boid(boolean startOutsideView) {
    if (startOutsideView) {
      spawnBoidOutsideView(this);
    } else {
      spawnBoidInView(this);
    }
  }
}

i m sure there is a better way to implement that from inside the library, knowing the number of views and hologram screen size but was not skilled enough to get those data from quilt. do you think it s possible?

Thanks for sharing this — I think your workaround is a useful experiment, and it also reveals an important limitation that I had not considered carefully enough before.

The Looking Glass is reported by the OS as a normal secondary display, so in principle we can get its desktop position and dimensions and normalize the global mouse position to that physical panel. However, the image shown on the panel is not a normal 2D image: it is the final interlaced light-field output used together with the lenticular optics to present multiple views. If you’re curious about what this actually looks like, you can capture the Looking Glass output with software such as OBS and see the rendered image directly.

Because of that, a physical mouse position on the Looking Glass does not map back uniquely to a point in the quilt, or to a single point in the 3D scene.

So I think there are really two separate problems here:

  1. Finding the pointer position on the physical Looking Glass panel.
    This should be possible to make more robust. Bridge can identify the actual Looking Glass display, so the library could eventually expose its desktop bounds instead of relying on “the non-primary monitor” and a hard-coded scale such as /3.5.

  2. Deciding what that 2D pointer position means inside the holographic scene.
    Without viewer / eye tracking, there is no single physically correct 3D ray, because different viewing angles see different views of the light field.

For the Flocking example, I think the most practical convention would be to interpret the normalized panel position using the center holographic view and project it onto the camera’s focus plane.

This is actually quite close to what the current Flocking example already does with Processing’s mouseX / mouseY. Your /3.5 workaround seems to approximate this mapping, which is probably why it works reasonably well on your setup, although the scale will vary across Looking Glass models and display configurations.

A more general API could potentially expose something like a normalized panel coordinate, a center-view interaction ray, or a focus-plane pointer helper. I would want the API name and documentation to make it clear that this is a defined interaction convention rather than exact inverse mapping of the holographic image.

There is also a separate input issue: Processing’s mousePressed state belongs to the Processing sketch window, so direct clicking on the Looking Glass output may need additional handling as well.

I think this is a very reasonable direction for the library, and your example helped make the problem much clearer. I’m quite busy at the moment, so I may not be able to implement this in the near term, but I’ll keep it in mind for a future update. If you or anyone else has time to improve this, please feel free to submit a pull request — contributions would be very welcome.

Thanks again for experimenting with the library and for documenting your approach.