Multiple independent audio channels - best approach?

I’m making a 2D game thing, and I want to be able to have as much control of sound as possible. For example, I want to have about 50 sound channels, with full control over each one of them - being able to modify every single playing sound at any point in time(i.e. change panning and pitch), and even switch sound files for every channel.

Right now, the approach for this that I settled on is using Minim classes - store a dynamic array of MultiChannelBuffer instances where every instance is a single sample, and then load them at any point into any one instance of Sampler from a static array of 50 Sampler instances.
So far it works, and with attributed instances of “Pan” and “TickRate” objects together with the “Sampler” allow for control over any one of them, while “X=(X+1)%50” on every mousePressed() event allows to choose the oldest used one at any point.

However, obviously, creating 50 instances of “Sampler” objects is a little bit demanding - simply them existing (and not doing any audio) causes noticeable CPU load - the audio is really choppy on my machine until I set it down to 10 - which, while sufficient, doesn’t seem like the best way of dealing with the problem.

I’m thinking: Is there a way to make “Sampler” stop hogging CPU when not used, without turning it into null and recreating it when needed(which takes noticeable amount of time)? And, if not, what could be a better approach for this whole thing?

Should I try to deal with Java’s built-in audio library? (i.e. javax.sound.sampled)
I think I can handle that, but I’m wondering if there’s any easier solution to this that I’m not aware of.

I don’t think that the Processing Sound library will suit it, because, from what I heard, it can only load mono files.

I could just load up a single instance of Sampler for every single sound file, but that would be inefficient as some sound files would potentially be played a lot of times at once, while others would be used once every half an hour - and hand-picking best voice count for every sound file doesn’t sound like a good resolution for me. And then again, I’m not sure on how I would control pitch and panning of each voice independently inside a single Sampler.

So far, my testing code is this:
And if you don’t happen to have a spare test.wav file, here’s one: