I’m working on a program that displays data from an oscilloscope over a serial connection and have run into a problem converting from scientific notation. If I receive a number, for example, 5.72E-1, I then convert it using the pow function as follows:
println(5.72 * pow(10, -1));
The result I get, instead of being 0.572, is 0.57199997. Is this expected behavior? The answer certainly is close, but this seems like something a modern computer would be able to get exactly right. Am I making a mistake in how I process the data?
1 Like
You are doing nothing wrong. This is a quirk of computers - even modern ones.
Basically, float numbers lose some precision because they are trading off exact values for the ability to store so many different numbers. Internally, all numbers are stored in binary, and there are only so many bits reserved for a float to use. One bit is for the sign (+), some other bits are for the exponent (-2). And the rest are for what is called the mantissa - this is the 0.5712.
But you can’t always store “0.5712” in the few bits the mantissa has. So you get rounding errors.
… There are probably a bunch of better explanation about this online.
1 Like
Ok that makes sense. Is there a simple way to round the answer to an appropriate value? The amount I would want to round to depends on the length of the original value.
Most of the time, you want to round when you want to present your data, which means it gets converted to string, as in the case of the value used in println(). Usually in Processing I use nfs(). If you want to use scientific notation, I recommend you check standard ways this is done in Java.
As an example, I could use nfs(1.0/3.0 ,0 ,2) to print to decimal points.
Kf
1 Like
Number formatting in Processing are a set of similar-but-not-the-same nf functions. I sometimes find myself reviewing them as a group.
Number formatting: