Reason why accelerationX, accelerationY, accelerationZ and and rotationX, rotationY, rotationZ values are not recorded sometimes?

[togo]

Hello,

I have a question about accelerationX, accelerationY, accelerationZ and rotationX, rotationY, rotationZ. For a class project, I made a sketch on p5.js that records both the acceleration and the rotation of the mobile device and connected this sketch with firebase so that the recorded values will be stored there. I asked a bunch of my friends to run this sketch and I was observing how the values will be stored on the firebase database.

After seeing a few recorded data, I’ve observed that, while the users are all using iPhone, there are times when accelerationX, Y, Z, rotationX, Y, Z are recorded and not recorded.

Can you think of a reason why this is happening? According to the useragent data, they are both coming from an iPhone. The only reason I can think of is that there might be a setting on iPhone that prohibits from sharing these kind of data to others?

[An example entry on firebase when it’s not recording any values]

for this particular entry, the useragent was this:
“Mozilla/5.0 (iPhone; CPU iPhone OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/73.0.3683.68 Mobile/15E148 Safari/605.1”

[An example entry when it’s recording values properly]

for this particular entry, theuser agent was this:
“Mozilla/5.0 (iPhone; CPU iPhone OS 12_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.1 Mobile/15E148 Safari/604.1”

[kll]

does the browser ask you if you want to share anything?
( like for location usually )

if not, esp. in the case where it worked, then
-a- you are hacking peoples phones!
-b- this browser and iPhone are not save to use!

please, i am sure that you had no bad intentions,
but it should not be possible unless the user clicks
“yes” on share with this site!

[togo]

Hello kll,

Thank you for your reply.
No, the browser is not asking explicitly anything about sharing since I’m not taking the location data. It’s only the acceleration / rotation data of the phone.

Does the acceleration / rotation data from the mobile device need explicit approval in order to be accessed from p5.js?

For instance, in this example code below,
https://p5js.org/examples/mobile-acceleration-ball-bounce.html
I’m assuming that if you access with a mobile device, all most all smart phones will be able to run this. Is this not the case?

I’m sorry it’s an open ended question, but I guess I’m trying to figure out a specific situation when acceleration / rotation data cannot be accessed through p5.js.

[kll]

possibly you misunderstood my comment,
i not speak for p5.js, not even about p5.js

if:
getting phones internal data
via the browser is possible
without the user’s approval

you found a LEAK.

@togo

Does the acceleration / rotation data from the mobile device need explicit approval in order to be accessed

sorry, i don’t know,
but it SHOULD.

---

? i am the only one who thinks that is a very serious matter?

there is a big difference,
if i enable the autorotation, so the apps ( also the browser )
can render landscape mode, still that is all local.
but sending to ( taking from ) server is a totally different question?

---

above you linked an example what states

Open this page on a mobile device to display the sketch.

i did check on by PC and old android tablet and nothing happened.
( i mean no auto run or canvas )
run it in the editor and look from 2 devices and it run,
but nothing moved.
anyhow: no questions asked about using phones resources.

but that test might not be valid as the phone possibly is older than you.
yes,
testing with an android 8.0 it run and worked ( moved ) without asking for permission,
still, is that data used locally or really got send to the server
as you do in your database application.
insofar your comparison might not be valid.

[kfrajer]

@togo

there are times when accelerationX, Y, Z, rotationX, Y, Z are recorded and not recorded.

You provided two user agents. Just to confirm, in the first example that you reported zero readings, this happens from time to time, right? If you check that data, what percentages of those readings are zero and what percentage are a non-zero value?

What is the rate you are sampling your data? 60fps? What is your rate saving into firebase?

One possible cause is that you are reading the data when no new data is available. For instance, using arduino units, if you read faster than your incoming data, you will get null in some of those readings. So before one process and saves the data, one need to verify data was available. if this were also happening in your case using acc/rot data, there are two approaches: you only save good data, discarding all those non-valid entries. Another option is to saved the latest available value. My suggestion would be to save only available data.

You are using the code that you provided in the p5js sketch, right? You can also read details about browser compatibilities here: DeviceOrientationEvent - Web APIs | MDN as it is a good reference and, if this is the way they are implementing it in the P5.js code, then I would expect you will not get any data from iphone users as Opera for safari (I am assuming this is what iphone users use) is not compatible.

I ran the p5js sketch on my Android phone and it works without asking for any permissions. You need to shake it hard for few seconds to get some motion (maybe that multiplier factor could be increased a bit). I also notice that when a change to landscape orientation, the example does not swap the width and the height, so there is a bug there in the actual example. The example should detect if the phone is in portrait and landscape and then define the width and height values. This is minor and not related to your original post.

Kf

[togo]

Hello kll,

Thank you for raising your concern. I believe you are talking about security. I actually did a little bit of search and apparently there is a toggle that limits the access of acceleration / orientation data from the mobile device.

Apple to Limit Accelerometer and Gyroscope Access in Safari on iOS 12.2 for...

So to your point, these data are not entirely taken without explicit consent, however, as you touched upon, it’s not a permission that is asked every time you run a javascript code taking the acceleration / rotation data.

[togo]

Hello krajer,

Thank you for taking the time to respond to my question. Your input is highly helpful.

Just to confirm, in the first example that you reported zero readings, this happens from time to time, right?

No, actually this is happening throughout the entry. Either the entry is taking the acceleration / rotation data perfectly fine or not. On my firebase database, I have 8 entries. Among those 8 entries, I confirmed 2 have the acc / rot data correctly. The rest of 6 are recording 0s all the way through within that entry.

The useragents for the ones recorded correctly:
Entry 1:
"Mozilla/5.0 (iPhone; CPU iPhone OS 12_1_2 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1

Entry 2:
“Mozilla/5.0 (Linux; Android 8.1.0; Redmi Note 5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.90 Mobile Safari/537.36”

What is the rate you are sampling your data? 60fps? What is your rate saving into firebase?

I’m using setInterval() method to periodically collect the acc / rot data and right now, I’m recording these data with 50ms frequency.

One possible cause is that you are reading the data when no new data is available.

Yes…what it’s not making sense to me is that, while executing the same code, the acc / rot are sometimes being recorded and sometimes not.

You are using the code that you provided in the p5js sketch, right?

Yes, that is correct.

Another weird thing I noticed is the following:
If I run the following sample codes from my browser, they are working fine.

What Web Can Do Today: Device Position

examples | p5.js

But with this code, for example, the values remain 0.

coursescript.com - coursescript Resources and Information.

I’ve also tried running the following simple code in my own environment and tried accessing through a mobile browser. While the code does run on my mobile browser, the accelerationX, Y, Z are constantly zero.

function setup() {
  createCanvas(windowWidth, windowHeight);
}

function draw() {
  background(255, 255, 255, 255);
  text(accelerationX, 10, 15);
  text(accelerationY, 10, 30);
  text(accelerationZ, 10, 45);
}

[kll]

@togo

So to your point, these data are not entirely taken without explicit consent, however, as you touched upon, it’s not a permission that is asked every time you run a javascript code taking the acceleration / rotation data.

ok, what i understand on first glance,
that was a apple security leak found 2017, and fixed 02/2019
so there is now a setting where you allow that access by apps.

ha. but that is still not the same as taking that info and sending it back to server

but i am very happy that i got you on that path to check about that
and understand that coding is a responsibility
and the users must be made aware what is happening when they agree ( like on a intro page )
to run your p5.js sketch in their phone / browser.

[mnmly]

to add a note to @kll’s answer;

p5.js’s acceleration{X,Y,Z} is updated per devicemotion event → https://github.com/processing/p5.js/blob/master/src/events/acceleration.js#L636

But there’re the two blockers for this devicemotion event:

1. Security over protocol: http vs https → It has to be served from https.
2. Security on iOS level: As of iOS 12.2, Safari has the additional security over motion data: security note from apple: About the security content of iOS 12.2

So… in order to make sure your data is tracked properly, you have to make sure the users’ Safari granting the permission to the motion data (Settings → Safari → Motion & Orientation Access)

the annoying part is that the safari setting is off by default

[mnmly]

and the discussion over the permission is happening at w3c/deviceorientation where it seems the change is already pushed to WebKit → https://bugs.webkit.org/show_bug.cgi?id=195329. We should be able to ask the permission through JavaScript API without asking people to go to Settings on toggle the permission there once the new iOS shipped…

[togo]

Hello kll,

Since I didn’t explain previous, to clarify a little, I am briefing every individual about my research intentions and explaining what kind of data is being collected by going through this p5.js sketch. So it’s definitely not my intention to secretly collect data and the participants are fully aware of the process and have the right to stop going through the p5.js sketch if they feel not doing so.

But thank you very much for bringing up the point. “Coding is a responsibility” is something I will keep in my mind, not just for this occasion, but as a general practice.

[togo]

Hello mnmly,

Thank you for taking the time to respond with a thorough explanation.
I tried with several phones with using https connection and the safari motion toggle on, and it seems like the data is going through correctly. Given that the default setting is off for most of the people, I will reconsider the way how I will conduct my experiment. Thank you.