It sounds like you are missing requirements that need to be installed for the Java API to work – but if that isn’t it, can you execute Matlab scripts from the command line? If so, perhaps you could use exec().
Another possibility is, if you need math processing and Processing rendering combined but don’t need MatLab or Java code specifically, you could use the Processing.R mode, which gives you the R language and the Processing Java API.