# Problem with splitTokens?!

**URL:** https://discourse.processing.org/t/problem-with-splittokens/28612
**Category:** Coding Questions
**Created:** [March 17, 2021, 2:55pm UTC](https://discourse.processing.org/t/problem-with-splittokens/28612 "2021-03-17T14:55:48Z")
**Posts on this page:** 9
**Page:** 1

<div class="post-metadata">

### Author: ![lolonulu](https://avatars.discourse-cdn.com/v4/letter/l/ad7895/32.png) [@lolonulu](https://discourse.processing.org/u/lolonulu)
#### Post date: [March 17, 2021, 2:55pm UTC](https://discourse.processing.org/t/problem-with-splittokens/28612/1 "2021-03-17T14:55:48Z")

</div>

Hello,  
Here is a short code below to count words and more, but I face a strange problem:  
when I run it some words of the original string don’t appear on the screen!? How come?  
Did I forgot something using splitTokens ?! Thanks alot for your help in advance.  
All the best,  
L

```auto
IntDict counts;
String [] tokens;

void setup() { 
  size(1000, 1000);
  background(0);
  counts = new IntDict();
  String [] lines = {"If only you were paying a bit attention to me sometimes.", 
    "I am not just a care giver I also need soemone to hug me tenderly."}; 
  String alltext = join(lines, " ");
  tokens = splitTokens(alltext, "\n\";.?!'():\n ");
    
  for (int i =0; i<tokens.length; i++) {
    String word = tokens[i].toLowerCase();

    if (counts.hasKey(word)) {   
      counts.increment(word);
    } else {
      counts.set(word, 1);
    }
    println(word);
  }
 
    String []keys = counts.keyArray();
    for (int i=0; i<keys.length; i++) {
    
      textSize(20);
      float x = 50;
      float y= 50+15*i;
      text(keys[i], x, y);
    }
}

void draw() {
}

```

---

<div class="post-metadata">

### Author: ![Chrisir](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/chrisir/32/45_2.png) [@Chrisir](https://discourse.processing.org/u/Chrisir)
#### Post date: [March 17, 2021, 8:32pm UTC](https://discourse.processing.org/t/problem-with-splittokens/28612/2 "2021-03-17T20:32:24Z")

</div>

> [@lolonulu](#):
>
> some words of the original string don’t appear on the screen

which words are they?

---

<div class="post-metadata">

### Author: ![micuat](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/micuat/32/19407_2.png) [@micuat](https://discourse.processing.org/u/micuat)
#### Post date: [March 17, 2021, 8:41pm UTC](https://discourse.processing.org/t/problem-with-splittokens/28612/3 "2021-03-17T20:41:05Z")

</div>

I think it’s just because you eliminated the duplicates using the dictionary so the repeated words won’t show up on the window 🙂

---

<div class="post-metadata">

### Author: ![lolonulu](https://avatars.discourse-cdn.com/v4/letter/l/ad7895/32.png) [@lolonulu](https://discourse.processing.org/u/lolonulu)
#### Post date: [March 17, 2021, 9:00pm UTC](https://discourse.processing.org/t/problem-with-splittokens/28612/4 "2021-03-17T21:00:08Z")

</div>

Hi @Chrisir,  
Thanks for answering so fast! Well mostly short words miss in the second sentence : I, a, me the repeated ones.  
But the strings all appear in the println list so the problem may be with the use of dictionnary.  
like @micuat suggest. So or I use an Hashmap or InDict functions for counting words or I can screen all of them as in the original text?! Thanks a lot for your help as usual.

---

<div class="post-metadata">

### Author: ![lolonulu](https://avatars.discourse-cdn.com/v4/letter/l/ad7895/32.png) [@lolonulu](https://discourse.processing.org/u/lolonulu)
#### Post date: [March 17, 2021, 9:09pm UTC](https://discourse.processing.org/t/problem-with-splittokens/28612/5 "2021-03-17T21:09:35Z")

</div>

Thank you very much @micuat for your answer. yes, I noticed that these are the repeated words, so I can’t screen the whole words of my original text when using IntDict since it increments and show up the repeated words?!  
Maybe I can use it only to specify the textSize of the most repeatde text and use another array to show the whole text?! Not sure I am able to do this! Thanks a lot for your help.  
L

---

<div class="post-metadata">

### Author: ![Chrisir](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/chrisir/32/45_2.png) [@Chrisir](https://discourse.processing.org/u/Chrisir)
#### Post date: [March 17, 2021, 9:16pm UTC](https://discourse.processing.org/t/problem-with-splittokens/28612/6 "2021-03-17T21:16:43Z")

</div>

this gives out tokens and the word count from counts

```auto

IntDict counts;
String [] tokens;

void setup() { 
  size(1000, 1000);
  background(0);
  counts = new IntDict();
  String [] lines = {"If only you were paying a bit attention to me sometimes.", 
    "I am not just a care giver I also need soemone to hug me tenderly."}; 
  String alltext = join(lines, " ");
  tokens = splitTokens(alltext, "\n\";.?!'():\n ");

  for (int i =0; i<tokens.length; i++) {
    String word = tokens[i].toLowerCase();

    if (counts.hasKey(word)) {   
      counts.increment(word);
    } else {
      counts.set(word, 1);
    }
    println(word);
  }

  /*
  String []keys = counts.keyArray();
   for (int i=0; i<keys.length; i++) {
   
   textSize(20);
   float x = 50;
   float y= 50+15*i;
   text(keys[i], x, y);
   }*/

  // String []keys = counts.keyArray();
  for (int i=0; i<tokens.length; i++) {
    textSize(20);
    float x = 50;
    float y= 50+15*i;
    text(tokens[i] 
      + " "
      + counts.get(tokens[i].toLowerCase()), 
      x, y);
  }
}

void draw() {
}

```

---

<div class="post-metadata">

### Author: ![micuat](https://yyz2.discourse-cdn.com/flex036/user_avatar/discourse.processing.org/micuat/32/19407_2.png) [@micuat](https://discourse.processing.org/u/micuat)
#### Post date: [March 17, 2021, 9:17pm UTC](https://discourse.processing.org/t/problem-with-splittokens/28612/7 "2021-03-17T21:17:11Z")

</div>

so you want to show the original text but the size depends on the frequency? it may also help if you can draw a sketch or something…

basically `keys` only contain the unique words (that are in the dictionary) so if you iterate on `keys` you will not see the words that are repeated again. If you want to show the original text, perhaps you want to iterate on `tokens` instead. And then, for each token, you can look up the dictionary to see how many times it appeared (not that you cannot combine this with the first loop! you need to first make a dictionary as you did, and then do another loop on the same content, this time to render on the screen)

---

<div class="post-metadata">

### Author: ![lolonulu](https://avatars.discourse-cdn.com/v4/letter/l/ad7895/32.png) [@lolonulu](https://discourse.processing.org/u/lolonulu)
#### Post date: [March 17, 2021, 10:33pm UTC](https://discourse.processing.org/t/problem-with-splittokens/28612/8 "2021-03-17T22:33:40Z")

</div>

Thank you very much Chrisir!! And then I can use the counts.get(tokens[i]).toLowerCase() value for the textSize 😉 Great! Thank you so much!

---

<div class="post-metadata">

### Author: ![lolonulu](https://avatars.discourse-cdn.com/v4/letter/l/ad7895/32.png) [@lolonulu](https://discourse.processing.org/u/lolonulu)
#### Post date: [March 17, 2021, 10:38pm UTC](https://discourse.processing.org/t/problem-with-splittokens/28612/9 "2021-03-17T22:38:05Z")

</div>

Thank you very much @micuat I wrote the two loops and then use counts.get(tokens[i]).toLowerCase() value for the textSize. Still a problem with textLeading…  
But it may be fine. Thank you so much! :slightly\_smiling\_ 😊