Triangled terrain audio visualizer

Processing Libraries
1

Inspired by Coding Challenge: 3D Terrain Generation with Perlin Noise I would like to create an audio visualizer that display several successive audio frames as a terrain. I managed to create a basic working sketch but I can’t go further on achieving a proper display of the waveform. Yellow is the current stored wave frame and the terrain is the history of previously stored wave frames.

image
image597×599 23.9 KB

I assume the problem here is that the terrain does not have enough resolution for displaying waveforms. Can someone point me some solutions or alternatives visualizations? Any other idea on how to display the “history” of the stored audio frames in 3D?

Here is my code:

import processing.sound.*;

SoundFile sample;
Waveform waveform;

int scl = 20;
int w = 1200;
int h = 900;
int cols = w / scl;
int rows = h / scl;
float time = 0;
float [][] terrain;
int audioCols, audioRows;


float waveHist[][] = new float[cols][rows];

public void setup() {
  size(600, 600, P3D);
  terrain = new float[cols][rows];

  sample = new SoundFile(this, "beat.aiff");
  sample.loop();

  waveform = new Waveform(this, cols);
  waveform.input(sample);

  for (int y=0; y<rows; y++) {
    for (int x=0; x<cols; x++) {
      waveHist[x][y] = 0;
    }
  }
}


public void draw() {
  waveform.analyze(); //read the audio data

  // fill the current row
  int counter = frameCount % rows; // stores waveforms
  for (int i = 0; i < cols; i++) {
    waveHist[i][counter] = waveform.data[i];
  }


  time -= 0.1; 
  float yoff = time;
  for (int y=0; y<rows; y++) {
    float xoff=0;
    for (int x=0; x<cols; x++) {
      //terrain[x][y] = map(noise(xoff, yoff), 0, 1, -100, 100);
      terrain[x][y] = map(waveHist[x][y],-1,1,-200,200);
      xoff += 0.1;
    }
    yoff += 0.1;
  }


  // Set background color, noFill and stroke style
  background(0);
  stroke(255, 255, 0);
  strokeWeight(2);
  noFill();

  //------------------------------------  2d waveform
  beginShape();
  for (int i = 0; i < cols; i++) {
    // Draw current data of the waveform
    // Each sample in the data array is between -1 and +1
    vertex(
      map(i, 0, cols, 0, width),
      map(waveform.data[i], -1, 1, 0, height) -200
      );
  }
  endShape();

  stroke(255);
  //------------------------------ terrain
  translate(width/2, height/2);
  rotateX(PI/3);
  //rotateZ(-PI/3);
  translate(-w/2, -h/2);
  for (int y=0; y<rows-1; y++) {
    beginShape(TRIANGLE_STRIP);
    for (int x=0; x<cols; x++) {
      vertex(x*scl, y*scl, terrain[x][y]);
      vertex(x*scl, (y+1)*scl, terrain[x][y+1]);
    }
    endShape();
  }
}

The refresh rate is also pretty slow compared with the youtube video tutorial, so I assume I am doing something wrong and messed up…

Thanks a lot!!!

2

Hello @funnyrectangles ,

Consider scrolling the data to give the impression of the terrain moving

A simple example to get you thinking about it:

ArrayList<Float> data = new ArrayList<Float>();
float x, y;

void setup()
  {
  size(600, 300);
  background(200);
  }
  
void draw()
  {
  //background(255);
  x = frameCount%600;
  if (x == 0)
    background(200);
  
  float a = (x)*(TAU/300);    
  y = 20*sin(a) + 5*sin(7.5*a);
  data.add(y);
  if (data.size() > width)
    data.remove(0);
  
  strokeWeight(2);
  stroke(255, 0, 0);
  point(x, y+50);
  
  noStroke();
  rect(0, 200, width, 100); // Like background for bottom 1/3
  
  stroke(0, 255, 0);
   
  for(int xi = 0; xi < data.size(); xi++)
    {
    point(xi, data.get(xi)+150);  
    }

  stroke(0);    
  // The other direction  
  for(int xi = 0; xi < data.size(); xi++)
    {
    point(width-xi, data.get(xi)+250);  
    }        
  }

:)

3

Hi @funnyrectangles,

my first naïve approach as a starting point, based on your code, could look like this …

Hope that helps.

Cheers
— mnse

import processing.sound.*;
import java.util.stream.*;

SoundFile sample;
Waveform waveform;

int scl       = 5;
int w         = 1200;
int h         = 900;
int cols      = w / scl;
int rows      = h / scl;
float scaling = 150.;
ArrayList<ArrayList<Float>> terrain;

public void setup() {
  size(600, 600, P3D);
  sample = new SoundFile(this, "test.mp3");
  sample.loop();

  waveform = new Waveform(this, cols);
  waveform.input(sample);

  terrain=new ArrayList<>();
  for (int y=0; y<rows; y++) {
    terrain.add(Stream.generate(() -> 0.0f).limit(cols).collect(Collectors.toCollection(ArrayList::new)));
  }

  colorMode(HSB, scaling, 1, 1); // coloring based on absolute amplitude
  noStroke();  
}


public void draw() {
  background(0);

  waveform.analyze();
  
  //back to front
  terrain.add(0,IntStream.range(0, waveform.data.length).mapToObj(i -> Float.valueOf(waveform.data[i]*scaling)).collect(Collectors.toCollection(ArrayList::new)));
  terrain.remove(terrain.size()-1);

  //front to back
  //terrain.add(IntStream.range(0, waveform.data.length).mapToObj(i -> Float.valueOf(waveform.data[i])).collect(Collectors.toCollection(ArrayList::new)));
  //terrain.remove(0);

  translate(width/2, height/2);
  rotateX(PI/3);
  translate(-w/2, -h/2);
  // triangle strip surface 
  beginShape(TRIANGLE_STRIP);
  for (int y = 0; y < rows-1; y++) {
    if (y % 2 == 0) { // even rows, right to left
      for (int x = cols - 1; x >= 0; x--) {
        fill(abs(terrain.get(y+1).get(x)), 1, 1); // abs(amplitude) [0-scaling]
        vertex(x * scl, (y + 1) * scl, terrain.get(y+1).get(x));
        fill(abs(terrain.get(y).get(x)), 1, 1);
        vertex(x * scl, y * scl, terrain.get(y).get(x));
      }
    } else { // odd rows left to right 
      for (int x = 0; x < cols; x++) {
        fill(abs(terrain.get(y).get(x)), 1, 1);
        vertex(x * scl, y * scl, terrain.get(y).get(x));
        fill(abs(terrain.get(y+1).get(x)), 1, 1);
        vertex(x * scl, (y + 1) * scl, terrain.get(y+1).get(x));
      }
    }
  }
  endShape();
}

demo

1 Like
4

Consider plotting the frequency domain and use FFT.

Slow melody (Light of the Seven by Ramin Djawadi):

image
image1002×839 106 KB

Fast beat (beat.aiff):

image
image1002×839 95.6 KB

I used the lower frequencies.

Reference:
Spectrogram - Wikipedia

:)