How to improve frequency detection? with mini library, or with a an other library?

Hello,

I work on a machine whose movement is actualized with certain frequencies of a music.
You can see an exemple here:
I started with the beat detect example from the minim library. But the detection is not very precise, I would like to have a program where we see the frequencies of the music, then another program, where we can detect a set of 10 frequencies for example between 50 and 150 hz, 150 and 250 hz …, 14,000 Hz to 16000 Hz. Is it possible?

I put the exemple of minim library

/**

  * with successive buffers of audio. You can do this inside of <code>draw</code>, but you are likely to miss some 
  * audio buffers if you do this. The sketch implements an <code>AudioListener</code> called <code>BeatListener</code> 
  * so that it can call <code>detect</code> on every buffer of audio processed by the system without repeating a buffer 
  * or missing one.
  * <p>
  * This sketch plays an entire song so it may be a little slow to load.
  * <p>
  * For more information about Minim and additional features, 
  * visit http://code.compartmental.net/minim/
  */

import ddf.minim.*;
import ddf.minim.analysis.*;

Minim minim;
AudioPlayer song;
BeatDetect beat;
BeatListener bl;

float kickSize, snareSize, hatSize;

class BeatListener implements AudioListener
{
  private BeatDetect beat;
  private AudioPlayer source;
  
  BeatListener(BeatDetect beat, AudioPlayer source)
  {
    this.source = source;
    this.source.addListener(this);
    this.beat = beat;
  }
  
  void samples(float[] samps)
  {
    beat.detect(source.mix);
  }
  
  void samples(float[] sampsL, float[] sampsR)
  {
    beat.detect(source.mix);
  }
}

void setup()
{
  size(512, 200, P3D);
  
  minim = new Minim(this);
  
  song = minim.loadFile("marcus_kellis_theme.mp3", 1024);
  song.play();
  // a beat detection object that is FREQ_ENERGY mode that 
  // expects buffers the length of song's buffer size
  // and samples captured at songs's sample rate
  beat = new BeatDetect(song.bufferSize(), song.sampleRate());
  // set the sensitivity to 300 milliseconds
  // After a beat has been detected, the algorithm will wait for 300 milliseconds 
  // before allowing another beat to be reported. You can use this to dampen the 
  // algorithm if it is giving too many false-positives. The default value is 10, 
  // which is essentially no damping. If you try to set the sensitivity to a negative value, 
  // an error will be reported and it will be set to 10 instead. 
  // note that what sensitivity you choose will depend a lot on what kind of audio 
  // you are analyzing. in this example, we use the same BeatDetect object for 
  // detecting kick, snare, and hat, but that this sensitivity is not especially great
  // for detecting snare reliably (though it's also possible that the range of frequencies
  // used by the isSnare method are not appropriate for the song).
  beat.setSensitivity(300);  
  kickSize = snareSize = hatSize = 16;
  // make a new beat listener, so that we won't miss any buffers for the analysis
  bl = new BeatListener(beat, song);  
  textFont(createFont("Helvetica", 16));
  textAlign(CENTER);
}

void draw()
{
  background(0);
  
  // draw a green rectangle for every detect band
  // that had an onset this frame
  float rectW = width / beat.detectSize();
  for(int i = 0; i < beat.detectSize(); ++i)
  {
    // test one frequency band for an onset
    if ( beat.isOnset(i) )
    {
      fill(0,200,0);
      rect( i*rectW, 0, rectW, height);
    }
  }
  
  // draw an orange rectangle over the bands in 
  // the range we are querying
  int lowBand = 5;
  int highBand = 15;
  // at least this many bands must have an onset 
  // for isRange to return true
  int numberOfOnsetsThreshold = 4;
  if ( beat.isRange(lowBand, highBand, numberOfOnsetsThreshold) )
  {
    fill(232,179,2,200);
    rect(rectW*lowBand, 0, (highBand-lowBand)*rectW, height);
  }
  
  if ( beat.isKick() ) kickSize = 32;
  if ( beat.isSnare() ) snareSize = 32;
  if ( beat.isHat() ) hatSize = 32;
  
  fill(255);
    
  textSize(kickSize);
  text("KICK", width/4, height/2);
  
  textSize(snareSize);
  text("SNARE", width/2, height/2);
  
  textSize(hatSize);
  text("HAT", 3*width/4, height/2);
  
  kickSize = constrain(kickSize * 0.95, 16, 32);
  snareSize = constrain(snareSize * 0.95, 16, 32);
  hatSize = constrain(hatSize * 0.95, 16, 32);
}


the problem with beat detection is that as the document says, it only works for, e.g., dance music that has repetitive percussive sounds. I’m a bit confused because the material you work doesn’t contain such sound if you are referring to the video as the sound material.

and for FFT, it’s not easy to find exact frequency because instruments usually contain several frequencies even if you play a single note. The “pure” tone only happens when you play a sine wave. So when you play 440 Hz, you will also see peaks at 880 Hz, for example.

There are a few ways to improve - the “easiest” way is perhaps to find the peak in the frequency domain, and assume that maximum value as a dominant frequency. If the input/recording is clean it should work. Another way is to use machine learning to detect the pitch although I don’t know if minim has such a feature. Though it is in javascript, ml5.js has an example. Lastly, if you have access to the original content before mixing - or even a midi data of the song, you should use this information instead of running FFT.

Hi micuat,

you said

Blockquote
the “easiest” way is perhaps to find the peak in the frequency domain, and assume that maximum value as a dominant frequency

There is an example called sound spectrum in minim library. In their example, it is techno music
If I made a test, or a combination of tests to recognize several peaks in several frequency domain, could I manage to detect the bass instrument of the music in their exemple? And better, different note of the bass, you can find in their exemple ?

I can’t use use ml5.js, because I don’t know how to transform this program in the Processing language.
This program is made to recognize the key of the music or each different note? Because in order to have the key scale of music (A, B, G#…), you need to recognize each note, and it should be very useful to me, to control events with each different pitch or note. If, ml5.js it is possible to detect each different note, I will post an other request to transform the program from python to Processing langage.
I also found a website to find the key,[https://getsongkey.com/tools/key-finder(https://getsongkey.com/tools/key-finder)
I put a techno music track and their program say my track is in G min and I have also this representation.

I think their program is powerful to find different note or events from different instrument, sound, sample (in techno music). Their representation is very precise. Do you think it would the perfect program to me to detect different events?

Maybe their program is the same as ml5.js ? Maybe their program is in java and easy to transform in Processing program.

I can’t find a midi data file for my music that I want to illustrate with movement, but it’s a great idea.
If you have for example a midi and mp3 file of Satie, Debussy or Mozart, and the music in mp3, please send me the link.
I found this website, [MIDI/ZIP large collection! Classical ZIP files] (MIDI/ZIP large collection! Classical ZIP files) but it is not free.

Thank a lot mister for yours brillant ideas and the next ones.

I think the problem comes down to - if you want to do it real time or offline in terms of music. If the music is somehow improvised (you don’t know what music will be played, or the music itself is synthesized on the fly), then you have to stick to techniques we’ve been discussing. But if it is “offline”, meaning that you know what song is played at precise timing, then things are so much easier. For example, the representation of key finder you shared looks very promising. I don’t know if it gives you metadata like json file, but even if not, you can take that image and make a simple processing sketch to extract exact timings for each note or chord. Also, you can add some markers/cues by hand if you want.

Porting ml5js would not be an easy task because it relies on tensorflow.js that is specifically made for browsers. But if you decide to use ml5js or other kind of library that is not available for Processing/Java, like in the example above, you can extract information, save it as json or something and read it by Processing. Of course, again, this only works if you know the song already.