Problem with splitTokens?!

Hello,
Here is a short code below to count words and more, but I face a strange problem:
when I run it some words of the original string don’t appear on the screen!? How come?
Did I forgot something using splitTokens ?! Thanks alot for your help in advance.
All the best,
L

IntDict counts;
String [] tokens;

void setup() { 
  size(1000, 1000);
  background(0);
  counts = new IntDict();
  String [] lines = {"If only you were paying a bit attention to me sometimes.", 
    "I am not just a care giver I also need soemone to hug me tenderly."}; 
  String alltext = join(lines, " ");
  tokens = splitTokens(alltext, "\n\";.?!'():\n ");
    
  for (int i =0; i<tokens.length; i++) {
    String word = tokens[i].toLowerCase();

    if (counts.hasKey(word)) {   
      counts.increment(word);
    } else {
      counts.set(word, 1);
    }
    println(word);
  }
 
    String []keys = counts.keyArray();
    for (int i=0; i<keys.length; i++) {
    
      textSize(20);
      float x = 50;
      float y= 50+15*i;
      text(keys[i], x, y);
    }
}

void draw() {
}
1 Like

which words are they?

2 Likes

I think it’s just because you eliminated the duplicates using the dictionary so the repeated words won’t show up on the window :slight_smile:

3 Likes

Hi @Chrisir,
Thanks for answering so fast! Well mostly short words miss in the second sentence : I, a, me the repeated ones.
But the strings all appear in the println list so the problem may be with the use of dictionnary.
like @micuat suggest. So or I use an Hashmap or InDict functions for counting words or I can screen all of them as in the original text?! Thanks a lot for your help as usual.

Thank you very much @micuat for your answer. yes, I noticed that these are the repeated words, so I can’t screen the whole words of my original text when using IntDict since it increments and show up the repeated words?!
Maybe I can use it only to specify the textSize of the most repeatde text and use another array to show the whole text?! Not sure I am able to do this! Thanks a lot for your help.
L

1 Like

this gives out tokens and the word count from counts



IntDict counts;
String [] tokens;

void setup() { 
  size(1000, 1000);
  background(0);
  counts = new IntDict();
  String [] lines = {"If only you were paying a bit attention to me sometimes.", 
    "I am not just a care giver I also need soemone to hug me tenderly."}; 
  String alltext = join(lines, " ");
  tokens = splitTokens(alltext, "\n\";.?!'():\n ");

  for (int i =0; i<tokens.length; i++) {
    String word = tokens[i].toLowerCase();

    if (counts.hasKey(word)) {   
      counts.increment(word);
    } else {
      counts.set(word, 1);
    }
    println(word);
  }

  /*
  String []keys = counts.keyArray();
   for (int i=0; i<keys.length; i++) {
   
   textSize(20);
   float x = 50;
   float y= 50+15*i;
   text(keys[i], x, y);
   }*/

  // String []keys = counts.keyArray();
  for (int i=0; i<tokens.length; i++) {
    textSize(20);
    float x = 50;
    float y= 50+15*i;
    text(tokens[i] 
      + " "
      + counts.get(tokens[i].toLowerCase()), 
      x, y);
  }
}

void draw() {
}

2 Likes

so you want to show the original text but the size depends on the frequency? it may also help if you can draw a sketch or something…

basically keys only contain the unique words (that are in the dictionary) so if you iterate on keys you will not see the words that are repeated again. If you want to show the original text, perhaps you want to iterate on tokens instead. And then, for each token, you can look up the dictionary to see how many times it appeared (not that you cannot combine this with the first loop! you need to first make a dictionary as you did, and then do another loop on the same content, this time to render on the screen)

1 Like

Thank you very much Chrisir!! And then I can use the counts.get(tokens[i]).toLowerCase() value for the textSize :wink: Great! Thank you so much!

2 Likes

Thank you very much @micuat I wrote the two loops and then use counts.get(tokens[i]).toLowerCase() value for the textSize. Still a problem with textLeading…
But it may be fine. Thank you so much! :slightly_smiling_ :blush:

2 Likes