Basic Breaking Up of Serial Data

Processing
1

Hello, I am working on a project where I am grabbing CAN Bus data from an ECU. I am using an Arduino and a CAN Bus shield, to which I am grabbing the 2 pieces of data I need from the CAN Bus and then sending it over serial to Processing.

The two pieces of data I am grabbing are an CAN Address and the CAN Message. These are sent together, with my format (which I can easily change) in [Address]|[Message]#

i.e.

169|19 44 39 28 53 50#

122|63 10 28 16 87 56#

188|126 92 45 44 50 58#

The two symbols are to break the address and message and provide myself location points so that I could (hopefully), look at the serial data between those delimiters and easily grab the address or the message. Those symbols can be placed wherever in the string relative to the messages or the addresses, so changing the delimiters is a really easy fix that I assume I will have to do first.

My end goal is to be able to have a list of addresses, and next to each address is their most recent CAN Bus message. My challenge is that I don’t know how to read a segment of the Serial data stream and save that segment to a variable.

Hopeful end result is that I could have a string of numbers (and symbols)
1234#5678910|1234#5678910

And then be able to grab either 1234 or 5678910 and save that to a variable.

I’m not very experienced with Processing, but I have been on the official webpage, reading all about Serial and the related functions, but I just can’t piece together how I would achieve this.

EDIT: I have found out the answer to the question that I was asking, so I am editing the original post to update it with what I learned.

  1. Reading off serial can happen in a few ways, but you need to be sure of what type your data being transmitted is, otherwise you will run into error messages.
  2. String and String[] are not the same thing. The brackets (as far a I have determined) are for an array form of the datatype. Often times, it is as easy to fix as just adding brackets to the end of the datatype designation. I’ve run into this error often in working with strings from serial.
  3. When you can, don’t use strings. bytes or character arrays are way easier to work with, and I discovered that a little too late.
  4. split() was indeed the answer to getting my serial data divided up into an array. I used split() twice, designating the split tokens as two different characters. It was super simple to get the data broken down. With this, there was some additional things. I nested the split() function within an integer array, so that I had actually usable numbers that I could perform math on, not just strings or character arrays.
  5. I created a dictionary eventually in order to store data and recall it. It made the process really easy, but there was a definite issue that I still run in to. I am processing serial data into an array, and storing portions of the array to either the key or the value in my library. I need to be sure to sterilize my input heavily, and I still run into issues where I attempt to store data to a portion of an array that doesn’t yet exist, thus getting some array range issues.

The eventual code I developed (portions of it):

import processing.serial.*;

String lines;
String list;
StringDict canBus;
Serial myPort;

void setup(){
 size(1080,800);
 
 printArray(Serial.list());
 myPort = new Serial(this, Serial.list()[4], 9600);
 myPort.clear();
 
 canBus = new StringDict();
 
 String lines = myPort.readStringUntil('\n');
 lines = null;
}

void draw(){

 while (myPort.available() > 0){
   String lines = myPort.readStringUntil('\n');
   String[] list = split(lines, ':');
     if (lines != null){
       canBus.set(list[0], list[1]);
     }
  }
  background(0);
  //Full listing of CAN Bus Addresses and their messages 
  for (int i = 0; i < canBus.keyArray().length; i++){
    String addresses = canBus.keyArray()[i];
    String fullMessage = canBus.get(addresses);
    int[] message = int(split(canBus.get(addresses), ','));
    fill(0);
    //This is an example of how I used the above code; got addresses, the full messages, and then a specific part
    text("Bus " + addresses + " " + fullMessage + " Temperature: " + message[2] * 14.27, 100, 25+18*i);
}
2
3

My first question is, what do you have so far related to software? Do you ahve your stream of data from the CAN bus? Is that being accepted by your arduino? Can you grabbed in Processing via the serial mechanisms?

Not sure if you can process CAN speeds using your arduino. I mean, arduino (UNO)'s analog input should be able to process up to 10KHz and probly you could push it to 100KHz (need to confirm… some hack I read a while ago). Not sure if you can transfer that much data via serial.

Now, from what I read from your post, it seems you already implemnted all the hardware-software integration and if your question is based on this:

My end goal is to be able to have a list of addresses, and next to each
address is their most recent CAN Bus message. My challenge is that I
don’t know how to read a segment of the Serial data stream and save that
segment to a variable.

Then this can be accomplish using split and probably a hashMap or a dictionary object. To watch only for the most recent value associated to an address, I will build my own class managing this functionality. If you provide some data I could put something together this weekend.

Kf

4

I am using Sparkfun’s CAN Bus shield. I do get data in Processing, and I am (fairly) certain that I am not losing anything along the way, so yes, I have the hardware all figured out.

What data exactly are you looking for to create that class? Would raw CAN Bus data work to put something together, because I have a butt tonne of it. I love the idea of assistance, and I would give you anything required. I am really novice to more high level programming theory, and I mainly sit in the implementation of hardware space, more so than the fine detail.

Other than that, I will look into using split and whatever a hashMap is.

5

Great, the hardest part is done :stuck_out_tongue:

You might no need to build a class after all. If I am not mistaken, you could use a dictionary, like the FloatDict which allow you to load key, value pairs. The key could be the address and the value could be the value associated to this address. Dictionaries are always sorted and, if I am not mistaken, they do not allow repeated key entries. One approach, after you split an address and value into a key-value pair, you could try checking if the key exist in the dictionary. If it doesn’t exist, add the key and its value. If it does exist, then you need to update the value. This way, your dictionary object will always contain the most recent value associated to your address.

I could provide some code, but I believe there is some code already out there. Give it a try and if you get stuck, provide your code.

https://forum.processing.org/two/discussion/24377/is-it-possible-to-get-the-key-from-a-value
Like dictionary - Processing 2.x and 3.x Forum
Alternative to FloatDict - Processing 2.x and 3.x Forum

Kf

6

Holy shindigs. It’s a little early and I haven’t dug into the topic just yet, but it sounds like you nailed exactly what I am trying to do. ESPECIALLY if it can build the dictionary with new keys on its own to then update its value to the most recent. Lordy, today is a good Friday

7

So it looks like the process I want to go through is this.

  1. From the Arduino, I want to send a package with the CAN Address and the Messages, with a common delimiter, ending with a line break.
  2. In Processing, I want to read the string from serial until a line break.
  3. I then want to use the split() function to break the string into parts.
  4. The first item of the list would be the Address, the rest would be the Message.
  5. From there, I want to build a dictionary of the Address with the Message. The data I am receiving is in hexadecimal, and received as a string from serial… so would I instead opt for a StringDict()?

My only issue, now, is that CAN Bus messages are long, with multiple bytes of info, of which only a portion of it is the value that I need to translate to decimal and display.

An example of a CAN Bus message is:
0x3C6 00 80 FF 47 00 00

Say I am only interested in the 47 for one measurement, not the rest of the string at the time. Could I save the rest as an array? Say I use a different delimiter between the message and the address, and I record a string of “00,80,FF,47,00,00”, can I convert that into an array and work from there? If I could store the array to the address, I then could really easily grab the necessary value to convert to decimal and then display. Also, I would need the entire message and can’t just slice off parts that are unnecessary. That specific CAN Bus address holds messages for 3 different components, and other addresses have info I need to read on different bytes.

EDIT: Discovered that it is pretty simple to convert a string to an array.
EDIT 2: .get() will definitely be the way I get the info I need, I think. as long as I can use .get() with a variable and the variable value will be the key. I feel that I am being quite naive about this.

8

This is part of your design. Save one the byte that you want or save all the bytes. There are two thoughts:

  1. If your arduino, the unit tapping into the CAN bus, does some processing to get certain value, like 47 above, your arduino acts as a middle one. This is a good strategy (pending point 2) as you extract exactly what you need. However, when it comes to design and flexibility, you could avoid this all together. Instead, you process all the data in your PC directly on your program. For instance, if you want to extract data from other devices, you could do that in your PC and no need to modify your arduino. The job of the arduino could be just one: it is the bridge, it feeds your PC with data. The disadvantage: The data that you start managing can grow quite fast instead oh having small chunks of exactly the data you want. Most of the data you capture is discarded.

  2. Something you need to consider is the speed of your desktop, the arduino and the data throughput from your CAN device. Taking single data tokens or full data stream depends on what would work at the end. If your data arrives at a rate of 5khz, then either design above should work. If we are dealing with 200KHz then probly one is better. I like to think arduino’s have limited processing power. I am saying that without having any experience on them… I am looking at them from the price tag. I will be happy to be wrong. On the other hand, a desktop has lots of computer power compare to a single chip, so I can see you processing the data there easily.

Some other questions to consider. The data of interest, what is its rate? You want to use your unit and process data in real time or post-proc is ok? Are you looking into creating a persistence model aka. saving data for further processing later on?

Another thing you can do is to use a serial capture software, like realterm, to capture your data. You could capture few seconds of data and use this data to build your data processing code.

Kf

9

I will have the arduino essentially just send data without any real filtering. My plan from the get-go was just to use the arduino as a pass-through.

So far, I actually have been making a lot of headway with the dictionary route. I don’t have access right now to a live data stream, so I just had a text file of messages that I recorded previously to which I parsed strings from. You telling me that there is a program I could use to record and play back serial events?? Would it act as thought it was a serial data stream on playback? With a port and everything?

The end result for the data is essentially a digital instrument/gauge panel. We have height sensors, steering sensors, voltage, temperature, rpm, pressure, etc. The data is sent over the CAN Bus. With the arduino, I can read it, send it over serial to Processing. In Processing, I will have to have a list of addresses and their most recent messages. From there, I expect some mess in having to grab the specific byte (based on what I am monitoring) from a specific message (determined by the address) to then convert to decimal and apply some math. From there, I will have a numerical display or even GUI objects in controlp5, interfasia, etc to display that value in a more decorative manner. The display should update pretty quickly, but if there is a 1-2 second delay, it’s still miles ahead of how we make adjustments now. This was all a pretty long-form way of saying I will take take data over serial from an Arduino and use Processing to create an instrument panel that is as close to real-time as we could ask for.

EDIT: I wanted to add that I currently just imitate the CAN Bus data stream by using random numbers generated from an arduino, since It’s super annoying to try and debug only every 6 minutes with a 1 minute window.

10

You can record serial events, not play them. You could play them back in Processing, exactly the same way as you are working with previously saved data as string.

If you have the arduino-CAN setup, I would suggest you record data from that setup and then you feed it in Processing. Notice you do not need to introduce random char separators in your data. Instead, you should be able to process the data directly from the stream, assuming the CAN data has a defined format of headers, field associated to those headers and even their own custom delimiters like end of line or start of package. These last ones should be described in the docs of your CAN bus protocol.

If you are introducing characters,as your initial post, there is nothing wrong, you are just adding extra work that you might not need.

Kf

11

The data sent over has some specific delimiters that I inserted at the arduino point, which was to help in breaking down the strings over serial. I need a bit more time to play around with Strings and changing datatypes, as well as messing around more with getting the split data in different places, I will probably run into a few issues with refreshing and how to properly handle the incoming data.

For a naive example, I most likely want to do the breaking down of the serial strings and storing it to a library in a serialEvent() loop. Since i found that if I have it done in setup, I will never display anything, and if I do it draw, I will have some funky refreshing.