If you look over the reference to the API, you will see a mix of things – there are a few trig terms, a number of mathematical operators, and a LOT of concepts from geometry.
If you had to pick one basic foundation area of math to explore – and you don’t have to pick just one! – it would probably be geometry, and basic linear algebra (vectors).
That said, creative coding can use any math. There are Processing forum posts and tutorials about:
- context free grammars
- machine learning
- cellular automata
- signal processing
- computer vision
- collision detection
- wave collapse
- fluid or softbody dynamics
These can use many different kinds of math – for example, the PeasyCam library implements a rotating camera uses quaternions.
A common thread for many examples is either perception or, more commonly, simulation. Some of the common forms of simulation – like collision detection / raycasting – get back to geometry and linear algebra in their implementation. Keep in mind that algorithms to approximate / compute equations are often quite different from mathematical equation solving and proofs – you might want to look at the actual code of parts of Processing and related libraries / examples / tutorials that you are interested in, in addition to reading about concepts.
Hope that was helpful.