Not sure what you mean about applyMatrix not using a “conventional” coordinate system?
There is an example in the applyMatrix reference. The Y-axis is vertical here. But that will change depending on how the space (coordinate system) and / or the camera view is rotated.
As for calculations I guess it depends on what you want to do. But Processing have a few built-in matrix functions as you can see in the references page (translate, rotate, scale, shear…).
The applyMatrix matrix is called an augmented matrix, since it has one more dimension to include translation (affline translation). Btw I’m not too familiar with the subject myself, though I have briefly touched it before.
Here’s another page on the subject.
Altered the above example slightly to also show Z-axis translation via applyMatrix:
size(200, 200, P3D);
noFill();
translate(100,100, 0);
rotateY(PI/6);
stroke(0);
box(50);
// Set rotation angles
float ct = cos(PI/9.0);
float st = sin(PI/9.0);
// Matrix for rotation around the Y axis
applyMatrix( ct, 0.0, st, 0.0,
0.0, 1.0, 0.0, 0.0,
-st, 0.0, ct, 0.0,
0.0, 0.0, 0.0, 1.0);
stroke(255);
box(75);
//translate(0,0,25); // same as below
applyMatrix( 1.0, 0.0, 0.0, 0.0,
0.0, 1.0, 0.0, 0.0,
0.0, 0.0, 1.0, 25.0,
0.0, 0.0, 0.0, 1.0);
stroke(255, 100, 50);
box(75);