I wanted a sphere to rotate in a way that the point closest to the viewer always rotates up when the cursor is moved upwards and always left and right when the cursor is moved left and right.
I had problems because after the transformation matrix caused the rotation axis to no longer correspond to the screen axis when using e.g. rotateY();

In this version of the sketch I managed to make it work along one axis, but I feel like I need some more complicated trigonometry to subtract out the rotation along both axes.

This does not achieve the effect I tried to describe.
The three ellipses are supposed to form a sphere.
Now, on this sphere, the imagine the point that is closest to the viewer, which is always in the center of the canvas.
This point should always rotate in the direction the mouse is moving in.

I already implemented this for one axis, but cant make it work in both x and y direction.
Would appreciate it a lot if someone took another look at this, because i feel like this is doable but you need some trigonometry formula i canâ€™t quite figure out.

The â€śred dotâ€ť might help to clarify what I mean.
The point on the sphere that moves in the way the mouse does, is not supposed to be a fixed point on the sphere. Rather, in every rotated form of the sphere, the point i want to move is the point of the sphere in the exact center of the sketch.
This is my problem, because, after a rotation of e.g. 90 degrees around the y axis, moving the mouse upwards doesnâ€™t move the point in the middle upwards, but instead the red point which was fixed in the beginning.
To accomplish what I what, I think, you have to calculate the y axis, which represents the y axis of my screen and not the y axis of the currently rotated coordinate system. Hence why I started messing around with trigonometry as above. Though there an upward movement behaves as I want with a rotation along the y axis already applied, it does not the other way around.

Yes, referring to your second example.
In it, after a 90 degree rotation around y, the red dot travels up the side of the sphere when you move the mouse upward.
What I want is, with the same setup, that the red dot stays in its place and the point of the sphere in the center of the sketch, moves upward. This is no longer achievable by rotateX(), because a rotation matrix has already been applied.
I should note that I actually switched to p5js because it allows me to rotate around a specified axis, instead of just x, y and z.

Thank you for your effort, but that only gets me to the point of where I quit in p5.
I tried my best to understand some of the maths in the medium.com article, but I still canâ€™t figure out how to calculate the â€śscreen axisâ€ť that is needed to turn the rotated sphere as intended.

actually i think there is view problem,
while for a full 3D view the mouse interpretation like

mouseX, 0, width, -PI ,PI

is optimal, it looks bad.

a much more natural view i feel with

mouseX, 0, width, -PI/2, PI/2

anyhow the whole concept with this mouse requires the thinking
that NULL is when the mouse points at the CENTER of the canvas,
( a other approach would be to use delta valuesâ€¦)

to repair the processing y axis points down,
can use a (-1)*â€¦

as we play here 3D view, what is that without a ZOOM
use mouseWheel.

add i try to separate the

view code

axis code

user circles code

float x=0,y=0,zmag = 1;
float Y = -1; // to invert processing y axis
void setup() {
size(500, 500, P3D);
noSmooth();
noFill();
//ortho();
strokeWeight(2);
println("use: mouse X Y, wheel for zoom");
}
void draw() {
background(200, 200, 0);
my_Camera();
my_Drawing();
}
void my_Drawing() {
axis();
spheres();
}
void axis() {
stroke(200, 0, 0);
line(0, 0, 0, height/4, 0, 0);
sphereAt(height/4, 0, 0, 6);
stroke(0, 200, 0);
line(0, 0, 0, 0, Y*height/4, 0);
sphereAt(0, Y*height/4, 0, 6);
stroke(0, 0, 200);
line(0, 0, 0, 0,0,height/4);
sphereAt(0,0,height/4, 6);
}
void spheres() {
push();
stroke(0, 200, 0);
ellipse(0, 0, height/2, height/2);
rotateX(HALF_PI);
stroke(200, 0, 0);
ellipse(0, 0, height/2, height/2);
rotateY(HALF_PI);
stroke(0, 0, 200);
ellipse(0, 0, height/2, height/2);
pop();
}
void my_Camera() { // limit from +- PI to +- PI/2 like more natural view??
x = map(mouseX, 0, width, -PI/2, PI/2);
y = Y*map(mouseY, 0, height, -PI/2, PI/2); // invert processing Y
translate(width/2, height/2);
scale(zmag);
rotateY(x);
rotateX(y);
// here the view is axis correct if mouse is middle/center of canvas!!!
// right hand rule: thumb X RIGHT (RED), point finger Y UP(GREEN), middle finger Z FRONT (BLUE)( points to you )
}
void sphereAt(float x, float y, float z, float diam ) {
push();
translate(x, y, z);
sphere(diam);
pop();
}
void mouseWheel(MouseEvent event) {
float e = event.getCount(); //println(e);
zmag += e*0.01;
}

I very much appreciate all these attempts to help me, but none of these examples behave the way I tried to describe.

The rotation commands to achieve this cannot be as simple as rotateY(x) and rotateX(y), because the second rotation is executed in an already rotated system, and therefore do not rotate around the axes â€śof my monitorâ€ť that never change.

In your example, compare the behaviour of

rotateY(x);
rotateX(y);

to

rotateY(x);
rotate(y, cos(x), 0, sin(x));

my problem is achieving the effect rotate(y, cos(x), 0, sin(x)) has, but for both axes.

What is to be seen in your gif is the behaviour of my original post, although with the axes flipped.
Thank you for adding grid and lines, they might help describe my problem:

This is exactly what I meant:

This is what I am trying to get rid of:

As you can see, the sphere rotates around the yellow x-axis.
What it should rotate around is the not yet drawn stationary â€śx-axisâ€ť that lies on the grid.
Just as the y-rotation in your sketch always happens around a not drawn y-axis on the grid.

This is not an assignment and I am not trying to bait you into doing it for me
This is a random sketch idea I had (further problems here lol P5js: 3D Typography Problems/Bugs?) where something I thought would be trivial apparently turned out to be kind of tricky and Iâ€™m almost certain thereâ€™s some clever math involved in solving this.