Most performant and efficient way of saving data to a text file

Processing Coding Questions
1

Hi all!

I have written this small piece of code to save data to a textfile:

int time_prev;
int time_now;
int dt;

StringBuilder save2Doc_StringBuilder = new StringBuilder();
String[] save2Doc_String = new String[10000];
int counter= 1;

import processing.javafx.*;

void setup() {
  size(1820,660,FX2D); 
  frameRate(100);
  surface.setLocation(50, 50);
  background(255);
}

void draw(){
  time_prev = time_now;
  time_now = millis();
  dt = time_now - time_prev;
  save_and_count();
  saveData();
}

void save_and_count(){
  save2Doc_StringBuilder.append(dt);
  save2Doc_String[counter] = save2Doc_StringBuilder.toString();
  save2Doc_StringBuilder.setLength(0);
  counter++;
}

void saveData(){
  saveStrings("DATA.txt", save2Doc_String);
}

Is there any other way of doing this that is more performing and more efficient? I.e., for the code to run faster.

Thanks in advance!

2

Hi @MrImskiy,

even if I think it isn’t the most elegant implementation, my question would be, why do you think it is slow?
I ran your code on my box and it runs with a frameRate of ~100…

Cheers
— mnse

1 Like
3

Hello @MrImskiy,

This may be useful to determine the time to execute a block of your code:

void setup() 
  {
  size(300, 300, FX2D);
  }

void draw()
  {
  int t1 = millis();
  delay(5);           //block of code to check dt (delta time)
  int t2 = millis();
  int dt = t2 - t1; // delta t (difference)
  println(dt);
  }

Feel free to adapt to your code and share the results.

:)

1 Like
4

Thanks for the reply @mnse! I need to integrate accelerometer values twice to get position, so my iteration time has to be very small.

5

Thanks for the useful advice @glv !

6

Hi @MrImskiy,

ok! but still not understand why you put the array into a file. I’ve you’re in the process you can use a Map for storing your values in memory ?!

Cheers
— mnse

7

btw. @glv,

I enhanced your measure example a bit, even if there are already many out of the box, but as a kind of a show-case and just for the fun doing… :slight_smile:

Cheers
— mnse

//-------------------------------------------------------------------------------------------------
// Main sketch
//-------------------------------------------------------------------------------------------------
PerformanceMeasure perf;

void setup() {
  size(700, 150);
  perf= new PerformanceMeasure();
  textAlign(LEFT, CENTER);
  textSize(24);
}

void draw() {
  background(0);
  fill(255, 128, 0); 
  // Old-School variant
  // int which = floor(random(5));
  // perf.start("Test"+which);
  // delay(floor(random(1000)));
  // perf.stop("Test"+which);

  // More modern way 
  perf.measure(String.format("Task%s",floor(random(5))), () -> {
    delay(floor(random(1000)));
  });   

  text(perf.consoleAndGetLongestRunner(), 20, height/2);
  
  if (millis() > 60000) {
    perf.show();
    exit();
  }  
}
//-------------------------------------------------------------------------------------------------
// PerformanceMeasure stuff
//-------------------------------------------------------------------------------------------------
import java.util.Map;
import java.util.HashMap;
import java.util.AbstractMap;
import java.time.Duration;
import java.time.temporal.ChronoUnit;
import java.util.stream.Collectors;
import java.util.stream.Stream;

@FunctionalInterface
interface Task {
  void execute();
}

class PerformanceMeasureEntry {
  public String  name;
  public int     amountCalls;
  public int     initialStart;
  public int     currentStart;
  public int     lastDuration;
  public int     overallDuration;
  public boolean isActive;

  public PerformanceMeasureEntry(String label) {
    name=label;
    amountCalls=0;
    initialStart=-1;
  }

  public void start() {
    amountCalls++;
    isActive=true;
    int now = millis();
    if (initialStart < 0)
      initialStart = now;
    currentStart=now;
  }

  public void stop() {
    int now = millis();
    lastDuration = now - currentStart;
    overallDuration = now - initialStart;
    isActive=false;
  }
  @Override
    public String toString() {
    return String.format("name=%s => calls=[%d,%s], last=[%s], overall=[%s]", name, amountCalls, (isActive==true?"active":"inactice"), formatDuration(lastDuration), formatDuration(overallDuration));
  }

  public  String formatDuration(int ms) {
    Duration duration = Duration.of(ms, ChronoUnit.MILLIS);
    return Stream.of(
      new AbstractMap.SimpleEntry<>("d", duration.toDaysPart()),
      new AbstractMap.SimpleEntry<>("h", duration.toHoursPart()),
      new AbstractMap.SimpleEntry<>("m", duration.toMinutesPart()),
      new AbstractMap.SimpleEntry<>("s", duration.toSecondsPart()),
      new AbstractMap.SimpleEntry<>("ms", duration.toMillisPart())
      ).filter(entry -> entry.getValue().intValue() != 0)
      .map(entry -> entry.getValue() + entry.getKey())
      .collect(Collectors.joining(" "));
  }
}

class PerformanceMeasure {
  private Map<String, PerformanceMeasureEntry> measuremap;

  public PerformanceMeasure() {
    measuremap = new HashMap<>();
  }

  public void start(String label) {
    measuremap.computeIfAbsent(label, a -> {
      return new PerformanceMeasureEntry(a);
    }).start();
  }
  
  public void stop(String label) {
    measuremap.values().stream().filter(a -> a.name.equals(label)).forEachOrdered(a -> a.stop());
  }
  
  public void measure(String label, Task task) {
    start(label);
    task.execute();
    stop(label);
  }

  public String consoleAndGetLongestRunner() {
    return measuremap.values().stream().sorted((a, b) -> {
      return Integer.compare(b.overallDuration, a.overallDuration);
    }
    ).peek(a -> {
      println(String.format("[processes=%d] %s", measuremap.size(), a));
    }).map(Object::toString).findFirst().orElse("");
  }

  void show() {
    println("--- Ranking ---");
    measuremap.values().stream().sorted((a, b) -> {
      return Integer.compare(a.overallDuration, b.overallDuration);
    }).forEach(PApplet::println);
  }
}

Currently writes to stdout, but could easily changed to use a file (short dump)…

[processes=1] name=Task2 => calls=[1,inactice], last=[543ms], overall=[543ms]
[processes=1] name=Task2 => calls=[2,inactice], last=[731ms], overall=[1s 288ms]
[processes=2] name=Task2 => calls=[2,inactice], last=[731ms], overall=[1s 288ms]
[processes=3] name=Task2 => calls=[2,inactice], last=[731ms], overall=[1s 288ms]
[processes=3] name=Task2 => calls=[2,inactice], last=[731ms], overall=[1s 288ms]
[processes=4] name=Task2 => calls=[2,inactice], last=[731ms], overall=[1s 288ms]
[processes=5] name=Task2 => calls=[2,inactice], last=[731ms], overall=[1s 288ms]
[processes=5] name=Task2 => calls=[2,inactice], last=[731ms], overall=[1s 288ms]
[processes=5] name=Task2 => calls=[3,inactice], last=[556ms], overall=[4s 258ms]
[processes=5] name=Task2 => calls=[3,inactice], last=[556ms], overall=[4s 258ms]
[processes=5] name=Task2 => calls=[3,inactice], last=[556ms], overall=[4s 258ms]
[processes=5] name=Task2 => calls=[3,inactice], last=[556ms], overall=[4s 258ms]
[processes=5] name=Task2 => calls=[3,inactice], last=[556ms], overall=[4s 258ms]
[processes=5] name=Task2 => calls=[3,inactice], last=[556ms], overall=[4s 258ms]
[processes=5] name=Task3 => calls=[3,inactice], last=[979ms], overall=[5s 489ms]
[processes=5] name=Task3 => calls=[3,inactice], last=[979ms], overall=[5s 489ms]
[processes=5] name=Task0 => calls=[6,inactice], last=[741ms], overall=[5s 903ms]
[processes=5] name=Task3 => calls=[4,inactice], last=[157ms], overall=[6s 946ms]
[processes=5] name=Task3 => calls=[5,inactice], last=[346ms], overall=[7s 293ms]
[processes=5] name=Task1 => calls=[3,inactice], last=[613ms], overall=[8s 271ms]
[processes=5] name=Task1 => calls=[3,inactice], last=[613ms], overall=[8s 271ms]
--- Ranking ---
name=Task2 => calls=[3,inactice], last=[556ms], overall=[4s 258ms]
name=Task0 => calls=[6,inactice], last=[741ms], overall=[5s 903ms]
name=Task4 => calls=[4,inactice], last=[414ms], overall=[7s 273ms]
name=Task3 => calls=[5,inactice], last=[346ms], overall=[7s 293ms]
name=Task1 => calls=[3,inactice], last=[613ms], overall=[8s 271ms]
8

It’s part of my master’s project. Does saving my data into a text file slow down the code? If yes, what other more efficient way is there to store data? If there is no better way, then I may remove this text file function.

9

Hi @MrImskiy,

This should be quite fast … even if this is using a non-blocking buffering to write the output, it also depends a little bit on you I/O device to get the maximum out.

Hope that helps you …

Cheers
— mnse

import java.util.logging.ConsoleHandler;
import java.util.logging.FileHandler;
import java.util.logging.Handler;
import java.util.logging.Formatter;
import java.util.logging.Level;
import java.util.logging.LogRecord;
import java.util.logging.Logger;

final Logger logger = Logger.getLogger("MyLogger");
FileHandler handler;

void setup() {
  size(500, 500);
  initLogger();
}

void draw() {
  log("Frame: " + frameCount);
  surface.setTitle(""+frameRate);
}

void exit() {
  if (handler != null)
    handler.close();
  super.exit();
}

// ------------------------------------------
void log(String str) {
  logger.log(Level.INFO, String.format("%s\n", str));
}

void initLogger() {
  try {
    handler = new FileHandler(sketchPath("my.log"), true); // true if it should append false if it should reset
    handler.setFormatter(new Formatter() {
      @Override
      public String format(LogRecord record) {
        return record.getMessage();
      }
    }
    );
    logger.addHandler(handler);
    logger.setUseParentHandlers(false);
  }
  catch(Exception e) {
    println("Can't create file: "+ e.getMessage());
    exit();
  }
}
10

Hi @mnse,

I just tested out your code, and it works as good as the StringBuilder variant I have. They both work best on 10ms. Thanks anyways!

I am wondering how fast processing IDE can get… It seems like 10ms is the limit, even for small codes. I hope I am wrong :stuck_out_tongue: Can anyone prove me wrong please? :slight_smile:

And what other way is to check for the iteration time instead of logging or saving to a file? Using println() in the console slows down the code to 60fps. So, if you don’t log or save the iteration times, there’s is no way you can get to know it…

11

Processing’s draw() callback is likely locked to the refresh rate of your display … depending on what device or graphics drivers you are using. If you are not trying to draw anything, maybe you could move all of your code into setup() and use your own loop. That would use the full cpu speed without Processing’s graphics calls slowing you down. The catch might be in the way events are handled – it might be that they are only dealt with in between draw() calls which means you still need to use draw().

You have your frameRate set to 100 which will force 10 ms between frames. Did you try using 1000 or higher?

Another speed up would be to save your timing numbers in a simple float or int array during runtime and only convert to strings and write the numbers out at the end when you quit the program. That way the processing and writing of the data won’t interfere with the timings.

12

Thanks for your reply @scudly!

I do need the draw() because I have to plot the signals in realtime.

I have already tried 1000fps, and when I take an average of all the logged iteration times, and divide the standard deviation by it, I get 34%, which is too high. When I do the same for 100fps, I get a ratio of approximately 3%, which is good.

Ok, I will try to do this :slight_smile:

1 Like