A One-Handed Interaction Technique for Numeric Input
Summary
The project Single-Handedly develops and tests a novel one-handed interaction technique that enables the user to ‘fingerspell’ numbers to a computer mouse to input numbers into a computer. I designed a way to intuitively encode numbers into finger gestures executable while holding a mouse, built two electronic prototypes and user tested the interaction technique as a full-fledged input method to find its capability.
Introduction
Motivation
This is where hands typically rest while using CAD software.
The right hand is used to hold the mouse while the left hand rests at the bottom left corner of the keyboard with fingers on the most used navigational hotkeys - Ctrl/Cmd, Shift and Alt/Option. These keys are used for controlling the ‘type of camera movement’ in a way; dragging the mouse while holding down Shift produces a different type of camera movement than dragging the mouse without, for example.
#1 Most Frequent Action
Object Exploration
Using Shift, Ctrl/Cmd or Alt/Option keys in conjunction with the mouse to move the 3D model around in the virtual space
#2 Most Frequent Action
Numeric Input
Modifying dimentions not by dragging the mouse, but by directly inputting its value using the number row or numpad
Problem Identification
Task Analysis
This is what hands and eyes do -
1. Resting Position
The left hand is at the bottom left and the right is on the mouse. The user is able to do object exploration but not input numbers.
2. Lift Right Hand
User clicks on a numeric text field to input a dimension. The right hand in this case, is the dominant hand and will be used to input numbers. As a result, the right hand has to lift up and leave the mouse to reach for the number row or numpad.
3. Search for Keys and Press
As the location of the mouse is always changing, there is no muscle memory as to how far the number keys are from the right hand. The user has to take their eyes off the screen to find them and press them.
4. Reset and repeat
Finally, the user locates the mouse using their peripheral vision, grasp it and look back at the screen. Hands return to the resting state. Repeat drill each time a number has to be input.
Problem Statement
How do we eliminate the need for the hand to reach over?
In repetitive workflows like CAD, short, redundant and seemingly trivial sets of movements can add up to cause siginificant fatigue and/or time wastage. Optimisation is all about removing even a few steps required to do a task. If we can eliminate the time required to find the number keys, reach over and press them, confirm the input, locate the mouse and reach back, we may be able to help optimise the CAD workflow.
Solution
Intended Solution
Steps eliminated. Time saved.
If we can eliminate the need to reach over to the keyboard to input numbers, we can reduce the time required to do the task. The intended solution makes the mouse capable of taking numerical input using gestures doable while holding the mouse in it’s resting position and eliminates the need to either reach over or look away from the screen.
Microgestures
Microgestures refer to very small movements that can be used to trigger an action in the virtual world.
Ideal for augmenting a set of interactions onto the core human-mouse interaction, they also reduce the effort and range of motion required to input data, so that users aren’t required to leave the mouse and can keep the wrist rested.
(October 2022) Update: Meta showed off a wrist-wearable prototype that interprets motor neuron signals to identify microgestures at their October 2022 Keynote. This device can easily implement the set of interactions detailed in this project.
Fingerspelling
A language that goes beyond language.
To make learning the microgestures easier, I wanted to encode them in the most intuitive way possible. Gesturing numbers is a very way natural communication, so I decided to build my encoding scheme around fingerspelling.
User Testing
It’s simple to learn. Mostly.
Fingerspelling Getures Demonstration
Algorithm
Designing algorithms to eliminate false input
Fingerspelling gestures are virtually independent of finger position in space, as they are governed only by a) whether a particular finger was lifted or not and b) which fingers were lifted. It has to be clearly understood what counts as a gesture and what does not. To prevent accidental keypresses, I came up with an algorithm that the system uses as a safetynet or filter to make sure the user intends to do the gesture.
Prototyping
Since the intended design is a set of gestures, the implementation can take various forms.
Mouse Rig
Built to augment fingerspelling gestures onto a generic mouse. In order to have the fingers rest comfortably on the touch sensors, the surface area had to be increased and supports had to be added for the ulnar side.
Image enlarged. Not to scale.
Touchpad Rig
Built to simulate the experience of a multitouch trackpad. Even though not the entire surface is touch sensitive, the sensors were positioned in a way to roughly accomodate the average hand in its relaxed pose.
Image enlarged. Not to scale.
User Testing
Fingerspelling training session using the custom built trainer app
Onboarding
A trainer app was built to help users memorise gestures.
The gestures were demonstrated to each user.This was followed by a trainer program, which displayed a phrase along with an illustrated graphic of what fingers the user is supposed to lift, and a text field below to enter the phrase. They were allowed to use the trainer program until the user said they were comfortable to do the test without the graphic.
Question #1
How fast can it be?
The first task is to evaluate the learning curve of the Fingerspelling gestures as a standard input method in terms of metrics like CPM (Characters per Minute) and ER (Error Rate). This will tell us how feasible it is as an input method, if at all.
Fingerspelling
The first task required users to only input numbers that appeared on screen and press Enter.
The typing test program required users to input the number being displayed on screen into a text field, and press the Enter key on the rig to go to the next number. There were a total of 30 numbers in a session, and a total of 10 such sessions were conducted over 1 week.
Question #2
Can it be fast enough?
In order to be actually useful, Fingerspelling has to be at least as fast as the two most popular traditional numeric input devices - the numpad (grid of numbers on a full sized keyboard) and numrow (row of numbers above a QWERTY layout).
In the blind test, users’ view of the mouse and keyboard was obsecured using a flat board.
The test was done after participants had done 6 or more sessions of the first task and had memorised the gestures.
Users were shown a number with a text field and were required to highlight the text field by hovering the mouse pointer over it and clicking the left mouse button, and then enter the numeric phrase on-screen using the given mode of input. To submit a phrase, the participants were required to press Enter, after which the number and text field would move to a random new location on-screen.
Results
Here are the results.
Learning Curve
Mean typing speed in terms of characters per minute and error rate in terms of percentage of errors out of total input.
Sighted and Blind Comparison
Speed
Input speed in characters per minute on each device. The higher, the better.
Sighted and Blind Comparison
Error Rate
Percent of errors out of total inputs on each device. The lower, the better.
Difficulty Rating
Mean ratings of each gesture with 1 being the easiest and 5 being the hardest. The lower, the better.
...which means...
The prototypes show a promising learning curve and are faster than traditional input methods*
* faster in the intended usage scenario - using input devices while keeping eyes on screen / not looking at the input device
Fingerspelling 🙈 = 🐵
In real world applications, while switching between mouse and keyboard might be quicker for numeric entry, it requires taking one’s eyes off the screen to locate the keys. My tests prove that this problem is solved by the Fingerspelling Mouse Rig since there is almost no difference between sighted and blind usage.
Learning Curve 📈
The CPM and ER scores of participants steadily increased as they built up muscle memory of the gestures. It is almost a linear increase in typing speed, and it can be concluded that 10 sessions is far too few to predict where the learning curve will plateau.
What users said -
“Memorisation is fairly easy”
Users memorised the gestures fairly quickly. Some users could even type their phone numbers into a blank text field before using the trainer application.
“Numbers above 5 are confusing and counterintuitive”
Users made errors when consecutively executing gestures that required the same number of fingers to be lifted (like 64, 73 and 82). Numbers below 5 were recieved as simple.
“I know the gestures but my fingers need better muscle memory to execute them”
Over three users said it was cognitively harder for them to lift their fingers for a gesture rather than press them.
“I have trouble doing 6/7/8”
Users had trouble memorising 6, 7, 8 and 9. They said they required a lot of practice to execute and even then, it was cognitively taxing.
“Having different kind of buttons for Enter and Backspace definitely helps”
Most users said Enter and Backspace were too close, but agreed it was necessary to have them both near the thumb.
“I can’t locate the sensors without looking after I’ve lifted all my fingers”
Few users initially lifted all fingers after each gesture, and few continued to lift all fingers for pressing Enter or Backspace.
Learnings
Solving false clicks
A major problem with the mouse rig is that users would often click mouse buttons while doing the fingerspelling gestures. This can be resolved by either using firmer switches in a mouse or having a dedicated area for the index and ring fingers to rest on, and another area for right and left click.
Using a bigger 'hitbox' on mice
Even the users who preferred the mouse rig over the touchpad rig would find themselves looking over and around the mouse trying to find the sensor for the little finger and carefully placing it on the sensor. This can very easily be solved using bigger, prominent touch areas.
Positioning Enter and Backspace keys
The best probable finger to execute Enter and Backspace commands is the thumb owing to its dexterity. In the initial stages, users required backspace more often so having it closer to the thumb than Enter helped, but then users found it better to have Enter closer to the thumb as the primary action and Backspace as the secondary action. IT is evident, however, that there needs to be some tactile differentiation between the two actions.
Comparing prototype rigs
Some users strongly preferred the mouse rig over the touchpad rig because it offered better support to their palm and fingers. Other participants felt the touchpad rig was more comfortable because they could clearly tell the position of their fingers and that it was more relaxed. Overall, the performance on both the rigs was almost identical but some people preferred mouse ergonomics while others preferred touchpad ergonomics.
How to improve the Encoding
It is observed that after explaining the logic behind the encoding scheme of gestures, they are overall simple to memorise, but it cannot be said that numbers above 5 are intuitive.
The original encoding logic was to find the opposite number in the decimal system to say, 6; 10 minus 6 is 4 so the user lifts 4 fingers, for 7, 10 - 7 is 3 so the user lifts 3 fingers and so on. The gestures for 6, 7 and 8 are the most cognitively and physically taxing since one first has to think about the reverse/opposite number and then lift multiple fingers to execute them.
Instead, as 4 users suggested, it was more intuitive to reverse the encoding above 5 so that the higher the number, the more fingers one has to lift. This also makes sense using another argument that the gesture for 7, for example, is lifting the thumb and index finger, ie. lifting two fingers, which is more intuitive since it semantically means 5 + 2 = 7.
Conclusion
Wrapping Up...
In real world applications, while switching between mouse and keyboard might be quicker for numeric entry, it requires taking one’s eyes off the screen to locate the keys. This problem is solved by the Fingerspelling Mouse Rig since there is almost no difference between the sighted and blind tests. While the ergonomics need to be optimised and the encoding scheme made intuitive, fingerspelling seems like a viable single-handed interaction method.
