Hi, I’m working up a serial communications app with a Teensy/Arduino and a processing sketch. It’s going well, and I want to implement some checksums to safeguard data transmissions. I’ve found several libraries on the Arduino side, but so far nothing in processing.
Are there any good libraries for CRC calculations that would work? I’d want CRC32 ideally, and other variations would be nice.
Thanks for the help,
Mike
Hello @Thundercat,
Arduino side:
This is the one I found quickly in Arduino 2.2.1 Library Manager and installed and used example from there:
Processing side:
Using the Arduino CRC32 example (modified) to send and Processing example (modified) to receive:
References:
:)
Hi that was very helpful!
I’ve been trying to get my head around this for a long time, so your post was great!
I will test it out and see if I can make this work.
Thank you again! I really appreciate it!!
Mike
By the way, any chance you can post your modified code so I can understand what you did a little better?
Thanks again!
Mike
Arduino side (minimal example):
Processing side (minimal example and simulation):
Hope that helps.
The serial communication between Arduino and Processing is another topic and discussion.
I only assisted with the CRC processing on Arduino and Processing.
The Arduino example provided with the library is also a minimal example that runs once in setup().
:)
I so appreciate that. Thank you for being so generous!
I already have serial com working between a Teensy unit and a Processing sketch, but it’s not reliable. Sometimes it works, sometimes it doesn’t, and I think it’s because messages are getting corrupted or something.
However, I was using strings, not char arrays, for the data transfer. So now trying to implement CRC32 I’m having a lot of issues. And I know some people think strings are from the devil lol.
I think I need to change over to using char arrays, and perhaps using enums. This is all new to me, but strings isn’t cutting it.
Thanks again!
Mike
Hello,
I have not had any issues with sending and receiving strings.
Any issues I did have early on I have now sorted out.
I use other options if I need to optimize for throughout.
Consider starting another topic on this with a minimal example to illustrate what you are doing (without the CRCs).
:)
Hi glv,
I am able to get your example working perfectly! Many thanks for this, as you saved me a lot of hours of frustration!
However, when I am trying to use it with my actual application, it is not working correctly.
I have discovered the issue has to do with converting the string in Arduino code to a char.
If I use this code in Arduino, it works fine:
void SSStr(String str) {
char msg[] = "Fader Presidential"; ////HARD CODING THE STRING TO A CHAR ARRAY
uint32_t crc32_res = crc32.calc((uint8_t *)msg, strlen(msg));
Serial.write(HEADER);
Serial.print(',');
Serial.print(str);
Serial.print(',');
Serial.print(crc32_res);
Serial.write(LF);
delay(ComDelay);
}
In this case, with the hard-coded character array, it is sending a correct crc and it matches what processing generates.
However, I need to be able to send various type of messages, so I don’t want to hard code the char array. I want to use the str parameter that is passed into the function. But when I try to convert a string to a char array, it is not calculating correctly for some reason.
Here is the code that generates the wrong CRC from Arduino:
void SSStr(String str) {
int str_len = str.substring(3).length() + 1;
char str_array[str_len];
str.toCharArray(str_array, str_len);
uint32_t crc32_res = crc32.calc((uint8_t *)str_array, strlen(str_array));
Serial.write(HEADER);
Serial.print(',');
Serial.print(str);
Serial.print(',');
Serial.print(crc32_res);
Serial.write(LF);
delay(ComDelay);
}
What comes across is the string value with the wrong CRC calculated.
Also, here is how I am calling it:
SSStr("ID,Fader Presidential");
Any idea what I’m doing wrong here? It’s obviously the conversion of the string to the char array that I’m getting wrong.
Thanks for any insights. I hope this is a minimal and clear enough example as you stated.
Mike
Well I solved it by separating the strings into two separate inputs to the function:
void SSStr2(String strMsgType, String strMsgBody) {
char str_array[strMsgBody.length() + 1];
strMsgBody.toCharArray(str_array, strMsgBody.length() + 1);
uint32_t crc32_res = crc32.calc((uint8_t *)str_array, strlen(str_array));
Serial.write(HEADER);
Serial.print(',');
Serial.print(strMsgType);
Serial.print(',');
Serial.print(strMsgBody);
Serial.print(',');
Serial.print(crc32_res);
Serial.write(LF);
delay(ComDelay);
}
Calling it like this:
SSStr2("ID", "Fader Presidential");
Thank you again for your support.
Mike
I thought I would share an even more robust example now that I got some things under control. It is a “minimal example” - with two sets of data - as I spent a couple days frustrated that the crc’s were not matching, only to discover you have to reset the CRC in between data calls.
I’m using this for Arduino/Processing communication, so the reset is very important.
Thank you again to @glv for the fantastic support!!!
import java.util.zip.CRC32;
CRC32 crc = new CRC32();
String response = "";
String message = "@,Fader Presidential,840248536";
String [] incomingData = split(message.trim(), ',');
String crcString;
if(incomingData[0].charAt(0) == '@') {
if (incomingData.length > 3) {
for (int i = 1; i < incomingData.length - 1; i++) {//remove the crc sent - that's why -2 and not -1
response += incomingData[i];
if (i != incomingData.length - 2) response += ",";
}
}
else response = incomingData[1];
}//if
crcString = incomingData[incomingData.length - 1];
println("message: " + message);
println();
println("response: '" + response + "'");
printArray(incomingData);
println();
println("Processing CRC generated: ");
crc.update(response.getBytes());
String enc0 = Long.toHexString(crc.getValue());
long enc1 = crc.getValue();
println(enc0, enc1);
if(crcString.equals(Long.toString(enc1))) println("CRC Match!");
if(Long.parseLong(crcString) == enc1) println("CRC Match!");
crc.reset(); //IMPORTANT!!! Or the crc's will never match!
response = "";
message = "@,ReqGranted,2435164217";
String []incomingData2 = split(message.trim(), ',');
if(incomingData2[0].charAt(0) == '@') {
if (incomingData2.length > 3) {
for (int i = 1; i < incomingData2.length - 1; i++) {//remove the crc sent - that's why -2 and not -1
response += incomingData2[i];
if (i != incomingData2.length - 2) response += ",";
}
}
else response = incomingData2[1];
}//if
crcString = incomingData2[incomingData2.length - 1];
println("message: " + message);
println();
println("response: '" + response + "'");
printArray(incomingData2);
println();
println("Processing CRC generated: ");
crc.update(response.getBytes());
enc0 = Long.toHexString(crc.getValue());
enc1 = crc.getValue();
println(enc0, enc1);
if(crcString.equals(Long.toString(enc1))) println("CRC Match!");
if(Long.parseLong(crcString) == enc1) println("CRC Match!");
What about MD5, SHA256, etc?
Why specifically CRC32?
Nobody’s going to hack a USB connection.
Why do you need to safeguard data transmission?
Nothing to do with security. Only transmission integrity. Without it I’m sometimes getting corrupted transmissions. I need to know when data wasn’t transmitted correctly, and crc32 seems the best balance of speed and function.
Thx,
Mike
I see.
That is a good point.
Checking the hash is a forgotten step for many.
Files are intact so much of the time most people just skip it.
However, .dmg files are verified onmacOS when opened by default.
Fun fact: macOS .DMG also use CRC32!
What are you trying to accomplish?
What Arduino are you using?
What type of cable are you using?
USB should not corrupt data in the first place.
Also, if the original data became corrupt, isn’t it therefore possible for the hash to become corrupt?
CRC32 was not really designed for this.
You could always send the data twice and compare the chunks.
If they don’t match, request another until the 2 most recent data chunks match
Why are you using Processing?
I reccommmend you code a Java class and access it from your Processing app.
There are so many Java libraries to choose from.
I appreciate what you are saying.
Learning full blown Java is more than what I’m willing to do at the moment - I’ve taken some online Java classes through Udemy but Processing seems much more accessible. I don’t want to spend months learning Java - I need to get this app done, and I’m pretty sure I can do it with Processing. And I’ve been able to do a lot with Processing so far.
Also, yes, the hash itself could become corrupt - it’s endless right? And the hash for the hash, etc. But when you say it “shouldn’t happen” that data gets corrupted over USB you are contradicting what DOES happen sometimes - enough that I need to verify the data is correct before calling it “good.” About 40% of the time the large dataset I send from Arduino to Processing doesn’t work correctly, and that’s with a good cable at 115200 baud.
And sending data twice with large datasets would impact performance; much better to just check the checksums and if they match then I don’t have to resend the data. If they don’t match, then I can resend.
Does this not sound reasonable?
Thank you for your thoughts.
Mike
Just a reminder: Processing’s IDE (PDE) recognizes “.java” files besides “.pde”.
Also, we can drag “.jar” files into the PDE and a “code/” subfolder will be created.
Some sketches that mix “.pde” & “.java” files together:
Does the connection have to be over USB?
What about FireWire, thunderbolt, ethernet, old-school serial, etc?
USB is bad.
115200 baud could be part of the problem.
9600 baud is standard.
Above 9600 baud, integrity might decrease.
The device I’m using with this (a Teensy 4) is a USB device. So using thunderbolt etc is not an option.
I did manage to get it all working though.
Thanks for the advice 