CI small grants holder Josh Hosking reports on his project exploring the grammars of gesture in HCI
Josh will be presenting this project at the Small Grants Showcase on Wednesday 18th November 4-6pm. Tickets here: https://www.eventbrite.co.uk/e/127873686639
In this project I set out to develop a grammar for writing and detecting gestures, on low powered devices. The aim was to allow users to create their own gestures and gesture devices, to be used in a wide range of applications. The main focus of areas were in the creative fields and accessibility solutions.
Throughout the project a number of devices were built, exploring a range of gestures and applications. The main three devices were an air mouse for Computer Aided Design (CAD) which can be used to create and manipulate digital objects, a head mouse for users who cannot use a hand operated mouse, which is cheaper and more functional than existing devices and a one handed video game controller which is simpler than existing options and allows for a range of expressive inputs.
The project opened up a range of interesting avenues but there were a few key knowledge outcomes as well as the devices built:
One – Simple data structures allow for simple gesture construction
Using a standard but simple structure for recognising gestures from various data streams allowed for gestures to be developed and tested, quickly and easily. The data is split into two categories, core data and modifier data. The core data is whatever best represents the main action of the gesture. In a waving gesture this would be the orientation of the device. Modifier data is any other data which forms part of the gesture, this could be a pressed button. Inversely, in a gesture defined as a button press, when the device is tilted sideways the button press is the core data and the orientation is modifier data.
Two – Data in hand is worth two in the bush
One of the key changes in my approach to defining gestures was focusing less on the data which I expected to best define the gesture and more on the data that could be reliably captured. Initially there was an attempt to detect movement vectors by looking at acceleration over time. This proved to be difficult slow and unreliable. However, it became apparent that the change in orientation of the sensor is a very good analogue for the direction of travel, this is largely due to the pivoting of the human elbow. This technique of finding a reliable analogue for difficult to capture data is often useful.
Three – Stillness is an active gesture
In human to human communication the distinction between gestures and incidental hand movement is made in nuances of the context of those movements. In data sets from sensors these nuances are far less readily available and so in order for a device to robustly recognise gestures it must have it’s own cues for what is and is not a gesture. One very reliable way of doing this is to define a state where the device is considered still. This stillness gesture then cues the device to look for a gesture in the preceding data.
Four – A few gestures can build a lot of controllers
A large portion of the project was spent developing a range of reliably detectable gestures. However, once these gestures could be consistently recognised, the process of developing controllers for a range of applications became far more rapid. The first controller was the gestural air mouse used for CAD. The gestures from this, alongside others, were then used to build a low cost head mouse and a simple one handed video game controller; suitable for a wide range of games. With an easy to use library of gestures it would be exciting to see how users would develop controllers for a variety of uses.
The ability too easily create gestural controllers is exciting on a number of levels. Controllers like the gestural CAD mouse are entirely new and allow for novel experiences and could open up new accessibility opportunities. Most features of the head mouse and the one handed video game controller already exist but this system allows them to be made with custom features at far less cost. The head mouse contains less than £40 of parts, comparatively a commercial units cost upwards of £500 retail.
Five – New interactions create new experiences
Developing new controllers opens up a range of exciting possibilities for interacting with existing systems. Outside of touchscreens, most computer interactions are based on keyboard and mouse inputs; which have remained fairly constant for 50 years. Gestural CAD inputs allow for the manipulation of digital objects in a manner far more similar to physical manipulation, opening up new avenues for digital crafts.
One of the most unexpected experiences was during a brief experiment using the CAD controller to play the game Tetris. Gestural control entirely changed the experience of manipulating the shapes. Using the gesture controller to physically fling the shapes downwards into place creates a visceral experience from a fairly simple game. Similarly, on the one handed video game controller some buttons on a traditional controller are replaced with gestural movement. As well as allowing for one handed operation this makes it possible to replace button presses with actions similar to those being acted out in game.
The next stage of the project is to set up a front end interface, so that people can use either existing gestures or use the grammar used to create their own. This would allow for people to start to make their own gesture controllers and inform another iteration of the gestures. I also believe there is scope for further research in CAD interactions and how they influence the way people use CAD software and the objects they create with it.
By reimagining how we interact with computers and electronic devices we can entirely change the experiences taking up an increasing portion of our lives, making them more accessible, more tangible and more joyful.