Music vizualiser and player with ftt

Hello there !
So I have been given a code and the goal is to create a music player .
I must add features such as play, pause, stop and volume buttons and also a basic bar with the music timer that fills in color, like in VLC.

For now I would like to focus on the music vizualiser part, I have this idea that the music could paint a picture. I need to use FTT (music spectrum) :
-the ftt spectrum must take all of my screen (like in VLC)
-the bands need to leave traces after their passage, with a randomly generated color
If you have any other suggestions I am open, as soon as we stick to my idea of painting something out with a music !

import g4p_controls.*; // G4P library for GUI button, etc
import processing.sound.*; // sound library for audio processing

PFont f; // font for showing text on the canvas
String audioFileName1;
SoundFile audio1;
FFT fft;
int bands = 64;
float[] spectrum = new float[bands];
boolean playstop = false; // keep track of play/pause status
boolean loading = false; // keep track of if we're loading a file

void setup()
{
  f = createFont("SansSerif-24.vlw", 24); // load the font
  textFont(f);
  fft = new FFT(this, bands); // prepare the fft
  size(800, 600); // canvas size
  frameRate(30); // canvas refresh rate
  background(0); // black background/clear scren
  createGUI(); // initialise the GUI
}

void draw()
{
  background(0); // clear screen
  if(audio1 != null) // check if we have a sound file loadd
  {
    loading = false; // if so, we're not loading anymore
    fft.analyze(spectrum); // analyse the file
      for(int i = 0; i < bands; i++) // show the analysis
      {
      // The result of the FFT is normalized.
      // draw a rectangle for each frequency band.
        noStroke();
        fill(spectrum[i]*255, 255-spectrum[i]*255,0 ); // pick a color depending on energy in the band
        ellipse(i*width/bands, height - spectrum[i]*height, 
        width/bands, spectrum[i]*height); // draw a rectangle with height proportional to the spectral energy
    }
  }
  else
  {
    if(loading == true) // we're loading
    {
      loadingScreen(); // show a message
    }
  }
}

void fileSelectedA(File selection) 
{
  if (selection == null) 
  {
    println("Window was closed or cancelled.");
  } 
  else 
  {
    audioFileName1 = selection.getAbsolutePath();    
    println(audioFileName1);
    loading = true;
    audio1 = new SoundFile(this, audioFileName1);
    fft.input(audio1);
    playstop = true;
    audio1.play();
  }
}

void loadingScreen()
{
    textAlign(CENTER);
    fill(#FFFFFF);
    text("loading...", width/2, height/2);
}

void keyPressed() {
  if (key == ' ') // check if user hits space bar, play/pause.
  {
    if(playstop == true)
    {
      playstop = false;
      audio1.pause();
    }
    else
    {
      playstop = true;
      audio1.play();
    }
  }
}

here is the GUI file with the button so you can upload a song

/* =========================================================
 * ====                   WARNING                        ===
 * =========================================================
 * The code in this tab has been generated from the GUI form
 * designer and care should be taken when editing this file.
 * Only add/edit code inside the event handlers i.e. only
 * use lines between the matching comment tags. e.g.

 void myBtnEvents(GButton button) { //_CODE_:button1:12356:
     // It is safe to enter your event code here  
 } //_CODE_:button1:12356:
 
 * Do not rename this tab!
 * =========================================================
 */

public void button1_click1(GButton source, GEvent event) { //_CODE_:selectfile1:208344:
  //println("selectfile1 - GButton >> GEvent." + event + " @ " + millis());
  selectInput("Select primary audio file:", "fileSelectedA");  
} //_CODE_:selectfile1:208344:



// Create all the GUI controls. 
// autogenerated do not edit
public void createGUI(){
  G4P.messagesEnabled(false);
  G4P.setGlobalColorScheme(GCScheme.BLUE_SCHEME);
  G4P.setMouseOverEnabled(false);
  surface.setTitle("Sketch Window");
  selectfile1 = new GButton(this, 702, 12, 80, 30);
  selectfile1.setText("Select file");
  selectfile1.addEventHandler(this, "button1_click1");
}

// Variable declarations 
// autogenerated do not edit
GButton selectfile1; 

Hi @Ysaelya
The problem that I have with the sound library, is that it does not play mp3 files (aif works) on windows 7. It gives an NullPointerException.

Hi @noel,
That’s weird with windows 10 I have no error with this code.I don’t know how to help you fix this error :frowning:

Evening,
I would really appreciate if some of you could help me on this forum, I am really stuck there and this little work that I have is important :frowning:

This is an attempt to create a history that fades over time. The HXDEPTH (history depth) is a variable that allows you to setup the depth of the history. A value of three only stores the last 3 FFT calculations, for example. I have tried a value of 50 with no issues.

The code does not aim for efficiency but more for demonstration. I hope this paves a way so you adapt it for your goal.

A couple of things:

  • It is great you provided your code. You will get more responses if you post a picture of what you code does so far so there is no need to run your code to figure out what the current output looks like.
  • Consider renaming your variable playStop as it is not clear what the state is. What about isPlaying?
  • I have include a return() statement in draw so to preserve the image when the audio is paused.
  • I have change the height of the ellipses when drawn to a fixed value
  • Finally, I have included the fourth value in fill() to manage ellipses’s transparency based on history depth.

Image below with a history depth of 100. I hope this helps.

Kf

import g4p_controls.*; // G4P library for GUI button, etc
import processing.sound.*; // sound library for audio processing

PFont f; // font for showing text on the canvas
String audioFileName1;
SoundFile audio1;
FFT fft;
int bands = 64;
float[] spectrum = new float[bands];
boolean playstop = false; // keep track of play/pause status
boolean loading = false; // keep track of if we're loading a file


final int HXDEPTH=50; //History depth
ArrayList<float[]> history;
float[] deepCopy;

void setup()
{
  f = createFont("SansSerif-24.vlw", 24); // load the font
  textFont(f);
  fft = new FFT(this, bands); // prepare the fft
  size(800, 600); // canvas size
  frameRate(30); // canvas refresh rate
  background(0); // black background/clear scren
  createGUI(); // initialise the GUI

  history = new ArrayList<float[]>();  
}

void draw()
{
  
  //Next is to preserve image if audio is paused
  if(!playstop) return;
  
  background(20); // clear screen
  if (audio1 != null) // check if we have a sound file loadd
  {
    loading = false; // if so, we're not loading anymore
    fft.analyze(spectrum); // analyse the file

    //Remove only if max history size was reached
    if (history.size() >= HXDEPTH) {
      history.remove(0);
    }    
    
    deepCopy = new float[bands];
    arrayCopy(spectrum,deepCopy);
    //deepCopy[13]=(frameCount%(HXDEPTH*0.9))/(HXDEPTH*1.0);
    history.add(deepCopy);

    for (int j=0; j<history.size(); j++) {
      float[] slice = history.get(j);
      
      
      for (int i = 0; i < bands; i++) // show the analysis
      {

        // The result of the FFT is normalized.
        // draw a rectangle for each frequency band.
        noStroke();
        //fill(slice[i]*255, 255-slice[i]*255, 0);
        fill(slice[i]*255, 255-slice[i]*255, 0 ,j*255.0/HXDEPTH); // pick a color depending on energy in the band
        ellipse(i*width/bands, height - slice[i]*height, 
          width/bands, 5); // draw a rectangle with height proportional to the spectral energy
      }
    }
  } else
  {
    if (loading == true) // we're loading
    {
      loadingScreen(); // show a message
    }
  }
}

void fileSelectedA(File selection) 
{
  if (selection == null) 
  {
    println("Window was closed or cancelled.");
  } else 
  {
    audioFileName1 = selection.getAbsolutePath();    
    println(audioFileName1);
    loading = true;
    audio1 = new SoundFile(this, audioFileName1);
    fft.input(audio1);
    playstop = true;
    audio1.play();
  }
}

void loadingScreen()
{
  textAlign(CENTER);
  fill(#FFFFFF);
  text("loading...", width/2, height/2);
}

void keyPressed() {
  if (key == ' ') // check if user hits space bar, play/pause.
  {
    if (playstop == true)
    {
      playstop = false;
      audio1.pause();
    } else
    {
      playstop = true;
      audio1.play();
    }
  }
}



/* =========================================================
 * ====                   WARNING                        ===
 * =========================================================
 * The code in this tab has been generated from the GUI form
 * designer and care should be taken when editing this file.
 * Only add/edit code inside the event handlers i.e. only
 * use lines between the matching comment tags. e.g.
 
 void myBtnEvents(GButton button) { //_CODE_:button1:12356:
 // It is safe to enter your event code here  
 } //_CODE_:button1:12356:
 
 * Do not rename this tab!
 * =========================================================
 */

public void button1_click1(GButton source, GEvent event) { //_CODE_:selectfile1:208344:
  //println("selectfile1 - GButton >> GEvent." + event + " @ " + millis());
  selectInput("Select primary audio file:", "fileSelectedA");
} //_CODE_:selectfile1:208344:



// Create all the GUI controls. 
// autogenerated do not edit
public void createGUI() {
  G4P.messagesEnabled(false);
  G4P.setGlobalColorScheme(GCScheme.BLUE_SCHEME);
  G4P.setMouseOverEnabled(false);
  surface.setTitle("Sketch Window");
  selectfile1 = new GButton(this, 702, 12, 80, 30);
  selectfile1.setText("Select file");
  selectfile1.addEventHandler(this, "button1_click1");
}

// Variable declarations 
// autogenerated do not edit
GButton selectfile1; 
1 Like

Hey @kfrajer,
Thank you for those clear explanations
I have a few points I would like to discuss :slight_smile:
-Why is the spectrum acting only on the left, not on the whole screen ? (what lines on the code does that ?)
-The screen draws ellipse because of one function line 42, but I would like my spectrum to be with normal bands (rectangles). How to change that ?

The idea is to make this music player a bit artistic, so I need the bands to act like paint, here is what I’m looking for :
Random color generated for each band,
The bands don’t disapear like they normally would, they stay (until the end of the song)
I also need to know how to select the width of the bands
This way it would actually draw/paint something that could be beautiful, that could look like a city with buildings.

open File > Examples > Sound > Analysis > FFTSpectrum. That will help eliminate a lot of code that you wont need, especially at higher “bands” values (loose deepCopy, HXDEPTH, and history and handle it mathematically with a smoothing value, instead of making lots of calculations to get the same result), then start to multiply “bands” by 2 as there is a correlation between “bands” and audio resolution. Feel free to message me directly about this topic or try to message @neilcsmith as he is far more knowledgeable than me about sound in processing and made praxis live which might also have some better options for you, but the scope of your question goes beyond a simple forum response. You are on the right path though, so keep it up

I recently made a post showing how to visualise fft bands, also you could check out the examples in processing.

Hey @PHuzzeyMusic,
Because I am a new user on this forum, I cannot send personnal messages. :frowning:
Also, I am a beginner in processing (and in coding in general) so I gotta keep it very simple, each things that I have asked are supposed to be 1 line of code more or less (said my teacher…)
Because of Covid my University is closed, I am back in my country and I can’t speak directly with my class and teacher, we are left alone in this. This is why I need this community (you guys) to help me :slight_smile:

Okay, sorry I also don’t really use discourse that much so I didn’t realize it was like that. I can send you a modified version of your sketch. I tried to leave some comments all over to help explain what is happening.

import g4p_controls.*; // G4P library for GUI button, etc
import processing.sound.*; // sound library for audio processing

PFont f; // font for showing text on the canvas
String audioFileName1;
SoundFile audio1;
FFT fft;
int bands = 1024;
float[] spectrum = new float[bands];
boolean playstop = false; // keep track of play/pause status
boolean loading = false; // keep track of if we're loading a file

float smoothingFactor = 0.15;
int padding = 10;
int canvasClearAmount = 100;
ArrayList<Float>outputValues;

void setup()
{
  noStroke();
  f = createFont("SansSerif-24.vlw", 24); // load the font
  textFont(f);
  fft = new FFT(this, bands); // prepare the fft
  size(800, 600); // canvas size
  //frameRate(30); // canvas refresh rate
  background(0); // black background/clear scren
  createGUI(); // initialise the GUI

  outputValues = new ArrayList();
  
}

void draw()
{
  wipeScreen();
  if (audio1 != null) // check if we have a sound file loadd
  {
    loading = false; // if so, we're not loading anymore
    updateAudioSignals();
    outputValues = getOutput(padding);
    
    //draw all the output values
    for (int i = 0; i < outputValues.size(); i++) {
      avRect(norm(i, 0, outputValues.size()), outputValues.get(i), outputValues.size());
    }

  } else
  {
    if (loading == true) // we're loading
    {
      loadingScreen(); // show a message
    }
  }
}

void fileSelectedA(File selection) 
{
  if (selection == null) 
  {
    println("Window was closed or cancelled.");
  } else 
  {
    audioFileName1 = selection.getAbsolutePath();    
    println(audioFileName1);
    loading = true;
    audio1 = new SoundFile(this, audioFileName1);
    println(audio1.sampleRate());
    fft.input(audio1);
    playstop = true;
    audio1.play();
  }
}

void loadingScreen()
{
  textAlign(CENTER);
  fill(#FFFFFF);
  text("loading...", width/2, height/2);
}

void keyPressed() {
  if (key == ' ') // check if user hits space bar, play/pause.
  {
    if (playstop == true)
    {
      playstop = false;
      audio1.pause();
    } else
    {
      playstop = true;
      audio1.play();
    }
  }
}

//use the mouseY from below method to clear the screen faster or slower
void wipeScreen() {
  pushStyle();
  fill(20, canvasClearAmount);
  rect(0, 0, width, height);
  popStyle();
}

//use the mouseX to adjust the smoothing value
//use the mouseY to adjust how quickly the screen wipes clean
void mouseDragged() {
  //adjust this to see how the smoothing factor changes the updateAudioSignals method
  smoothingFactor = map(mouseX, 0, width, 0.01, 1);
  canvasClearAmount = (int)map(mouseY, 0, height, 0, 255);
}

//method to combine previous signal with current signal, lower smoothing is less dramatic
void updateAudioSignals() {
  fft.analyze(); // analyse the file
    for (int i = 0; i < spectrum.length; i++) {
      spectrum[i] += (fft.spectrum[i] - spectrum[i]) * smoothingFactor; 
    }
}

//prevents from having to manually adjust your values everytime, also
//looking at every band by itself isn't always the best approach since frequencies 
//increase exponentially, instead make them into groups
float calculateAudioIntoGroups(int start, int stop, float scale) {
  float res = 0.0f;

  for (int j = start; j < stop; j++) {
    res += spectrum[j];
  }

  return (res * scale);

 }

ArrayList<Float> getOutput(float padding) {
  //clear previous results
  outputValues.clear();

  int spreadValue = 5; //how many steps to skip by, adjust to see smaller or larger sections grouped together
  int hiCut = (int)(bands * 0.66); //when to stop looking at the signal, above 66% is no longer that useful (musically)

  for (int i = 0; i < hiCut; i += spreadValue) {
    //add some padding to later values since they will not be as strong as bassier values
    outputValues.add(calculateAudioIntoGroups(i, i + spreadValue, 1.2f + map(i, 0, hiCut, 1, padding)));
    if (i > 140) { //adjust this to make the spread start to increase later
      spreadValue++;
    }
  }

  return outputValues;
  
}

// draw the rectangle with the audio values, this is where you could start to change colors, etc
void avRect(float position, float audioVal, int totalSize) {
  pushStyle();
  fill(0, audioVal * 255, position * 255); // change to all sorts of different things
  float rWidth = (float) width / totalSize;
  rect(width * position, height, rWidth, -audioVal * (height / 2));
  popStyle();
}



/* =========================================================
 * ====                   WARNING                        ===
 * =========================================================
 * The code in this tab has been generated from the GUI form
 * designer and care should be taken when editing this file.
 * Only add/edit code inside the event handlers i.e. only
 * use lines between the matching comment tags. e.g.
 
 void myBtnEvents(GButton button) { //_CODE_:button1:12356:
 // It is safe to enter your event code here  
 } //_CODE_:button1:12356:
 
 * Do not rename this tab!
 * =========================================================
 */

public void button1_click1(GButton source, GEvent event) { //_CODE_:selectfile1:208344:
  //println("selectfile1 - GButton >> GEvent." + event + " @ " + millis());
  selectInput("Select primary audio file:", "fileSelectedA");
} //_CODE_:selectfile1:208344:



// Create all the GUI controls. 
// autogenerated do not edit
public void createGUI() {
  G4P.messagesEnabled(false);
  G4P.setGlobalColorScheme(GCScheme.BLUE_SCHEME);
  G4P.setMouseOverEnabled(false);
  surface.setTitle("Sketch Window");
  selectfile1 = new GButton(this, 702, 12, 80, 30);
  selectfile1.setText("Select file");
  selectfile1.addEventHandler(this, "button1_click1");
}

// Variable declarations 
// autogenerated do not edit
GButton selectfile1; 

Here is an example of what you might get, with some variation you could always make it cooler. Hope this was more helpful.

1 Like

Hey @PHuzzeyMusic
Thank you for that, I have read through the lines and here are some questions =
by clicking/dragging the mouse to the right or to the left the spectrum is going faster or slower, but I would rather have only 1 feature, when you click it would stop erasing the rectangles and keep drawing, this time rectangles would have a stroke so it could draw my city (look like buildings). As you click again, this would stop(clear screen) and return to normal activity.
This would be funny I think , like a little game
How could we do that ?

Also from line 153 to 160, you told me that I can change the color and stuff. ive tried to change the numbers but it seems that only opacity is changing (transparency) . The gradient is from green on the left to blue on the right, how can I change from pink to orange for example ?

cheers !

Hi again,
By the way if someone knows how to use the GUI builder to create play, pause and stop buttons that would be nice
I have also seen people creating a bar with the timing of the music but I can’t find the final code to this .