Massive latency when doing FFT with Minim on live audio from mixer

I wanted to share a program here, but I’m having trouble making it behave well.

The idea is to visualize (FFT) whatever music you are playing on your system, so you don’t need to load a track into Processing, but assume you are already playing music from soundcloud, mixcloud, youtube, etc.

It works already, but there is a ridiculous latency, and it seems to change from time to time. Sometimes it’s 2 seconds, sometimes it’s 20 seconds, and it also seems to drift. I was wondering if it was something about 44100 vs 48000 hz, or two buffer sizes that do not match…

The latency & drift are not present when working with an audio file.

Any ideas?

import ddf.minim.*;
import ddf.minim.analysis.*;
import javax.sound.sampled.*;

Minim minim;
AudioInput audio;
FFT fft;

// This affects latency, and in theory lower = less latency
int bufferSize = 512; // try 128 256 512 1024

void setup() {
  size(800, 100, P2D);
  // here you configure your audio device. Run it once and look in the console
  // to see available names. In my system one of the lines looks like
  // [2] default [default], version 4.18.0-12-generic
  // so I grab the part between [num] and the comma.
  startAudioInput("default [default]");
  noStroke();
}

void draw() {
  background(0);
  if (fft != null) {
    fft.forward(audio.mix);
    int bands = fft.avgSize();
    for (int i = 0; i < bands; i++) {
      float val = fft.getAvg(i);
      float x = map(i, 0, bands, 0, width);
      rect( x, height, 2, -val);
    }
  }
}

private void startAudioInput(String deviceName) {
  minim = new Minim(this);

  Mixer.Info[] mixerInfo = AudioSystem.getMixerInfo();

  for (Mixer.Info info : mixerInfo) {
    if (info.getName().equals(deviceName)) {
      minim.setInputMixer(AudioSystem.getMixer(info));
      audio = minim.getLineIn(Minim.STEREO, bufferSize);

      fft = new FFT(audio.bufferSize(), audio.sampleRate());
      fft.logAverages(22, 3);
      return;
    }
  }
  println(deviceName, "not found! Use one of these:");
  printArray(mixerInfo);
}

I also opened an issue on GitHub.

Forgot to mention: Ubuntu 18.10

Is Minim still using an output in this case? Have you tried setting both input and output mixer? One cause of this might be mismatched devices, but I don’t know Minim very well.

@hamoid – were you ever able to resolve this weird latency issue?

The GitHub issue above has very useful things to try, but unfortunately I didn’t have time yet.

@hamoid===
OSX Sierra
i tried your code changing somewhat your default (which seems to be Linux specific) to built-input-microphone: not any latency or nothing that i can notice when i put some music and see the graphics result… As for the bufferSize() 512 is probably a good value as minim mutiply it X4: 2048. As for the ouput which could be the culprit (according to @nellcsmith) i am quite sure that in this case not any output is used… Have you tried with other OS??? Could this issue be Linux spec.???

Thanks for checking @akenaton!

I don’t think I would have latency if I used the microphone. The issue is when I try to use the internal sound being played by the computer. As I mentioned above, what I wanted to do is to use whatever music I’m listening to as an input to drive my current experimentation :slight_smile: I will try the suggestions by the author from the GitHub link and report back.

@hamoid===

ok, i understand ; can you put here the list you get with info.getName() (not only the default)??? As for me all solutions provided are working except one which fires an error from Minim : “cannot secure the audio input”…

sorry: i have found that i get also a “Default Audio Device” which works the same…without latency

I assume you’re using PulseAudio loopback? Is this just for you? Because Beads, JACK and PulseAudio sink might be a better performing option.

Will test loopback on my setup later. Am very confused what Minim author is doing to get bad latency on Linux, as it’s the better performing of the JavaSound implementations!

Mind you, are you using Processing’s included JDK or OpenJDK? Latter has additional devices.

@neilcsmith===

yes i am using PulseAudio (yet i have used Jack) + P5 jdk; never tried OpenJDK: has it many advantages??

In my case, yes, just for me, to have a randomized always-available data stream while prototyping. I can try Beads too, I’ve used it before. I’m on OpenJDK.

It’s pretty much identical (even more so since Java 11). However, many Linux distributions include a direct PulseAudio backend as the default JavaSound device. In other words, default [default] is not the default! :grin:

Well Beads has a JACK backend (which I wrote, although their integration is old) and a potentially better JavaSound integration (not fond of Minim’s approach). Also, on OpenJDK try the PulseAudio backend if available.

I’m having the same problem.
Elementary 5.0, Processing 3.5.3.
I noticed that it starts with pretty much no lag, then the lag builds up getting worse and worse. I’ll look into my audiodevices and report back.

Update: I managed to get rid of the lag using the audioStream as proposed in the gibGub topic link

Update 2: the lag has returned somehow :thinking: