Sensing What I am doing with my Computer


Sensor Project

This project is about playing with sensors. Since then I am interested in if the graph of my right hand's acceleration tells if I am playing games (specifically Heroes of the Storm since it is my most played game right now ), typing a document, or watching videos on YouTube.

Materials Needed

  • tape

  • scissor (optional)

  • phone (with acceleration sensor app installed)

  • laptop/PC with mouse/trackpad

Procedure

Download any acceleration sensor apps onto your smartphone. For example, I used phyphox here for this project (link to phyphox website: https://phyphox.org/). Open the app and go to acceleration (both accelerations with G and without G works fine here). I used acceleration without G here for my project. 

Then I taped my phone onto the back of my hand. I taped it a bit lower so that it does not interfere with my hand motions quite that much.

Tap the little triangle at the top of the interface at the orange bar to start the timer to test if the acceleration sensor is working. Delete the graph and get prepared for the  experiment after the acceleration sensor seems working successfully.

Then I started the experiment by having the sensor taped to my right wrist while I am using my PC to play my games, watch videos on YouTube, and type my homework. Here are my results.

From my experience with my interaction between my keyboard and my mouse, I can see why there is much difference between the graphs of my right hand's acceleration. While I am gaming (especially the game Heroes of the Storm), I have to constantly right click on my mouse to direct my avatar to walk around on the map. This explains why there is constant direction change in the linear acceleration graph since my middle finger is constantly pressing down the mouse button and then come back up while I am moving my mouse rapidly. Some parts of the graph there is a sudden spike and I interpret it as I lifted my right hand to type on my keyboard to communicate with my teammates, or lifted and moved my mouse a bit since it is getting to the edge of my desk. There are also times where there is less movement than the other parts of the graph, and that's probably when my avatar is dead and I am waiting for my avatar to respawn. 

When I am watching videos on Youtube, I usually click on the recommendations and watch whatever is there, and change the video if I am not interested in the recommendation. Since then there is no constant change in the linear accelerations of x, y, and z. The big spikes are probably when I am moving my mouse to select or to scroll down the website, and where the linear acceleration is close to 0 is when I clicked on a video and watched that video for a certain period of time. 

The last graph corresponds to when I am typing out my homework on the computer. This graph has the largest range of linear acceleration compared to the other two graphs since this action requires my right hand to get onto the keyboard and press down keys in order to record information on the computer. Compared to just moving the mouse around, this requires much more linear acceleration in the z-direction. While the other two graphs had a range of about -2 to 2 m/s2 linear acceleration in the z-direction, typing had a range that's about -5 to 5 m/s2. I guess just based on the linear acceleration in the z-direction, we can tell if my right hand is using the keyboard or the mouse. 

Conclusions

+

Further Exploration

  • Based on the linear acceleration graph, it is possible to tell that what I am doing with my computer just by looking at the linear acceleration of my right hand. 

  • If there is not much linear acceleration (about -2 to 2 m/s2) in the z-direction for a long period of time, that probably means that my right hand is using the mouse; if there is much linear acceleration in the z-direction (-5 to 5 m/s2), that probably means that my right hand is using the keyboard. 

  • If we want to push the experiment further, we can explore different games that I play, and see if the graphs can tell which game I am specifically playing (because some of the games do not require that much of clicking on the mouse, and some of them require the right hand to stay on the keyboard, etc). 

  • The same can be applied to figuring out what type of homework I am doing with my computer. Because I am a design major, sometimes I need to use software like photoshop to do my homework. But as for photoshop, my right hand is usually on the mouse to help to drag things and manipulate things around. I wonder if this linear acceleration graph will be different than these linear acceleration graphs for gaming and watching YouTube videos. 

©2020 by Yufei Wang.