Challenges of Situational Impairments during Interaction with Mobile Devices

Challenges of Situational Impairments during Interaction with Mobile Devices

Introduction

The number of personal mobile devices is increasing. The greatest advantage of the mobile device is its mobility. Therefore, theoretically the owners of mobile devices should be able to use it under any circumstances everywhere regardless of the surrounding environment. However, it is not the case to be. Various contextual factors such as ambient temperature [18], ambient noise [22], mobile state of the user [5], encumbrance [16], ambient light [5] and divided attention [7] are known to adversely affect interaction with mobile devices. All of the aforementioned contextual factors are known as situational impairments. The term situationally-induced impairments (situational impairments) was first introduced by [20] to establish the relationship between the user, the nature of tasks that the user is engaged in, surrounding environment where the user is performing tasks, and the design of technology that the user is using to complete the task. The ubiquitous nature of mobile devices obligates them to be able to accommodate to different situational impairments and reduce the difficulties that people might experience during mobile interaction.

The importance of investigating the effect of situational impairments on mobile interaction has been highlighted on multiple occasions in the literature [1] [22] [14] [5]. For instance, previous work has argued that environmental and contextual changes might affect mobile device users in a similar way cognitive and physical impairments affect users with disabilities [1]. This connection was showcased in study by Yesilada et al. [23] that showed that a situationally impaired user without physical impairments performed a similar number of errors on a mobile device to a user with physical impairments. As a result, designs for one user group could benefit another group. As an example, technology designed for motor-impaired users may be used in reducing the effect of situational impairments on interaction with mobile devices [5]. Similarly, improvements to mobile interaction to address situational impairments can improve the user experience of people with disabilities when interacting with the devices. Situational impairments can potentially exacerbate usability issues of people with permanent impairments when interacting with mobile devices [14]. For example, a visually impaired person navigating an unfamiliar route while interacting with their mobile device would need to divide attention between swiping the cane and accessing the mobile device. Therefore, it is necessary to consider situational impairments when designing technology to assist permanently impaired users. An important implication of research regarding situational impairments during mobile interaction is that it affects users of all abilities, and therefore mobile interfaces that adapt to contextual changes can benefit all users [7].

Impact of Situational Impairments during Mobile Interaction

Ambient Temperature

A number of studies have investigated the effect of cold ambient temperature on interaction with mobile devices [6] [18]. In the study by Goncalves et al. [6] the researchers investigated the effect of cold temperature on smartphone input performance. Participants were asked to complete tapping tasks using the software developed by MacKenzie with the temperature thermistors attached [11] in a controlled laboratory experiment held in a cold room (-10 C) and in a warm room (+20 C) consecutively. The results showed that lower finger temperatures were associated with lower throughput and higher error rate when interacting with the mobile device in two-handed operation mode. Further, the researchers showed that adding finger temperature as a parameter into a Fitts’ law formula increases the law’s predictive power of determining movement time.

Sarsenbayeva et al. [18] investigated the effect of cold ambient temperature not only on fine-motor movements, but also on vigilance. The researchers followed a experimental design similar to the one reported in [6]; however, they used two custom developed applications to measure fine-motor performance (i.e., touch offset, time taken to tap a target) and vigilance (i.e., time taken to find a memorised icon). Interestingly, the results showed that while fine-motor performance was adversely affected by cold temperatures (larger offset and longer time taken to tap a target when compared to warm temperatures), cognitive performance was not affected. However, the researchers argued that vigilance would likely get affected under longer exposure to cold temperatures as  reported in [4]

In [7] researchers demonstrate that situational impairments worsen the performance during mobile interaction for visually and motor impaired users. Researchers determine such situational impairments as crowded spaces, lighting and weather, walking, and interruptions.  Further, researchers identify guidelines for more accessible and empowering mobile device designs [7]. They suggest using the device’s built-in sensors to identify user’s activity and location to adapt user interface for improving accessibility of the device.

Mobile State of the User

Several studies aimed to identify the effects of the mobile state of the user on mobile interaction. For example, walking has been found to have adverse effect on performance during mobile interaction. In a study by Lin et al. [10] participants performed standard target selection tasks while walking on treadmill under slow and fast walking conditions. Results of the study did not indicate a significant change in performance in the transition from being seated to being mobile. However, in a follow up study, researchers found that task completion time, error rate and workload measures (e.g., mental and physical demands) differed significantly while performing the tapping tasks under conditions of being seated, walking on treadmill, and free-style walking with obstacles [9] . Researchers establish that obstacles course condition was more challenging for the users compared to the seated condition when completing tapping tasks on mobile devices. Bastian and Rukzio [19]  investigated the negative effect of walking on target selection tasks performance and reading comprehension. They suggested that the negative effect of walking on target selection tasks can be compensated by increasing the target sizes. However, larger text size did not result in a better reading performance. Similarly, Mizobuchi et al. [12] investigated if the size of the keyboard buttons affected the text input performance while walking. The results showed that 2.5 mm is a minimum and 3.0 mm is the preferred key width for text input on soft keyboard for text input speed, error rate and subjective ease of text input not to be affected while walking.

WalkType is an adaptive text-entry system for smartphone devices designed specifically to overcome situational impairments caused by motion using the device accelerometer [5]. WalkType improves text entry accuracy by considering multiple features of accelerometer data such as displacement, acceleration and movement inference. The system also incorporates tap location and the finger travel distance during taps to eliminate imprecision of the input. WalkType significantly improved typing speed by 12.9% and reduced uncorrected error rate by 45.2% compared to the control condition while users were walking. In another example, researchers evaluated an adaptive walking user interface which would enlarge soft buttons when user movement is detected [8]. Their findings suggest that adaptive walking interfaces are feasible, but they should have taken into account individual preferences and characteristics of the user as well as target specific tasks affected the greatest while walking. Another strategy explored in previous work entails the use of audio feedback to improve touch screen interaction while walking [3]. The results of the study showed that adding a sound to the buttons significantly improved the usability of standard (16×16 px) and small (4×4 px) buttons as well as significantly reduced the user workload. One of the ways scientists suggest to reduce situational impairments effect caused by walking is using bezel gestures. For instance, in [2] researchers found that bezel gestures were not affected by environmental factors such as walking and distraction in terms of speed or accuracy during mobile touch-screen interaction.

Walking has also been found to cause deterioration in text legibility [13] as well as reading comprehension and cognitive performance, measured by a word searching task [1]. Moreover, researchers in [21] argue that reading while walking (on-the-go) is limited as it requires the user to divide their attention to handle two tasks in parallel: comprehending the text and navigation through the environment. In the study by Vadas et al. [21] participants were asked to complete reading comprehension tasks in four conditions: audio-walking and audio-sitting using a speech-synthesis audio display, and visual-walking and visual-sitting using a handheld visual display. The researchers found that audio interface improved participants’ navigation within the environment. The participants also rated the audio interface as less demanding compared to a visual display when completing the reading tasks. Not only walking, but ambient light has also been shown to have negative effect on reading efficiency and response selection speed as well as on time taken to find a word [1].

Encumbrance

It is common that users interact with their mobile devices while being encumbered with different objects (e.g., handbags, umbrellas, shopping bags and boxes). Therefore, encumbrance is another situational impairment that can affect mobile interaction. Previous work has highlighted how encumbrance can cause decreases in accuracy while performing target acquisition tasks on mobile devices  [15]. They also found that the encumbrance affected the user’s preferred walking speed making the participants walk slower than their typical rate. In a follow-up study, Ng et al. [16] investigated the effect of encumbrance on interaction with mobile device in one- and two-handed interaction modes. They evaluated three postures of holding and interacting with the mobile device: two-handed using index finger, one-handed using thumb, and two-handed using both thumbs. The results of the experiment showed that encumbrance caused a significant deterioration in performance completing tapping tasks for both one- and two-handed interaction modes when carrying a bag in each hand during the interaction with mobile device. Moreover, in [17] the researchers measured performance in completing target acquisition tasks on a mobile phone while encumbered in two conditions: walking on a treadmill and walking on the ground. The results of the experiment did not show the difference in performance of target acquisition tasks, although the researchers found that ground walking method provided a better representation of user’s preferred walking speed compared to treadmill.

References

[1] BARNARD, L., YI, J.S., JACKO, J.A., and SEARS, A., 2007. Capturing the effects of context on human performance in mobile computing systems. Personal Ubiquitous Comput. 11, 2, 81-96. DOI= http://dx.doi.org/10.1007/s00779-006-0063-x.

[2] BRAGDON, A., NELSON, E., LI, Y., and HINCKLEY, K., 2011. Experimental analysis of touch-screen gesture designs in mobile environments. In Proceedings of the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada2011), ACM, 1979000, 403-412. DOI= http://dx.doi.org/10.1145/1978942.1979000.

[3] BREWSTER, S., 2002. Overcoming the Lack of Screen Space on Mobile Computers. Personal Ubiquitous Comput. 6, 3, 188-205. DOI= http://dx.doi.org/10.1007/s007790200019.

[4] FLOURIS, A.D., WESTWOOD, D.A., and CHEUNG, S.S., 2007. Thermal Balance Effects on Vigilance During 2-Hour Exposures To -20°C. Aviation, Space, and Environmental Medicine 78, 7, 673-679.

[5] GOEL, M., FINDLATER, L., and WOBBROCK, J., 2012. WalkType: using accelerometer data to accomodate situational impairments in mobile touch screen text entry. In Proceedings of the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (2012), 2687-2696. DOI= http://dx.doi.org/10.1145/2207676.2208662.

[6] GONCALVES, J., SARSENBAYEVA, Z., VAN BERKEL, N., HOSIO, S., RISSANEN, S., RINTAMAKI, H., and KOSTAKOS, V., 2017. Tapping Task Performance on Smartphones in Cold Temperature. Interacting with Computers 29, 3, 355-367. DOI= http://dx.doi.org/10.1093/iwc/iww029.

[7] KANE, S.K., JAYANT, C., WOBBROCK, J.O., and LADNER, R.E., 2009. Freedom to roam: a study of mobile device adoption and accessibility for people with visual and motor disabilities. In Proceedings of the Proceedings of the 11th international ACM SIGACCESS conference on Computers and accessibility (2009), 115-122. DOI= http://dx.doi.org/10.1145/1639642.1639663.

[8] KANE, S.K., WOBBROCK, J.O., and SMITH, I.E., 2008. Getting off the treadmill: evaluating walking user interfaces for mobile devices in public spaces. In Proceedings of the Proceedings of the 10th international conference on Human computer interaction with mobile devices and services (2008), 109-118. DOI= http://dx.doi.org/10.1145/1409240.1409253.

[9] LIN, M., GOLDMAN, R., PRICE, K.J., SEARS, A., and JACKO, J., 2007. How do people tap when walking? An empirical investigation of nomadic data entry. International journal of human-computer studies 65, 9 (2007/09/01/), 759-769. DOI= http://dx.doi.org/http://dx.doi.org/10.1016/j.ijhcs.2007.04.001.

[10] LIN, M., PRICE, K.J., GOLDMAN, R., SEARS, A., and JACKO, J., 2005. Tapping on the move-Fitts’ law under mobile conditions. In Proc. IRMA, 132-135.

[11] MACKENZIE, I.S., 2015. Fitts Throughput and the Remarkable Case of Touch-Based Target Selection. In Proceedings of the HCI International (2015), Springer, 238-249. DOI= http://dx.doi.org/10.1007/978-3-319-20916-6_23.

[12] MIZOBUCHI, S., CHIGNELL, M., and NEWTON, D., 2005. Mobile text entry: relationship between walking speed and text input task difficulty. In Proceedings of the Proceedings of the 7th international conference on Human computer interaction with mobile devices & services (Salzburg, Austria2005), ACM, 1085798, 122-128. DOI= http://dx.doi.org/10.1145/1085777.1085798.

[13] MUSTONEN, T., OLKKONEN, M., and HAKKINEN, J., 2004. Examining mobile phone text legibility while walking. In Proceedings of the CHI ’04 Extended Abstracts on Human Factors in Computing Systems (Vienna, Austria2004), ACM, 986034, 1243-1246. DOI= http://dx.doi.org/10.1145/985921.986034.

[14] NAFTALI, M. and FINDLATER, L., 2014. Accessibility in context: understanding the truly mobile experience of smartphone users with motor impairments. In Proceedings of the Proceedings of the 16th international ACM SIGACCESS conference on Computers & accessibility (2014), 209-216. DOI= http://dx.doi.org/10.1145/2661334.2661372.

[15] NG, A., BREWSTER, S.A., and WILLIAMSON, J., 2013. The Impact of Encumbrance on Mobile Interactions. In Human-Computer Interaction – INTERACT 2013: 14th IFIP TC 13 International Conference, Cape Town, South Africa, September 2-6, 2013, Proceedings, Part III, P. KOTZÉ, G. MARSDEN, G. LINDGAARD, J. WESSON and M. WINCKLER Eds. Springer Berlin Heidelberg, Berlin, Heidelberg, 92-109. DOI= http://dx.doi.org/10.1007/978-3-642-40477-1_6.

[16] NG, A., BREWSTER, S.A., and WILLIAMSON, J.H., 2014. Investigating the effects of encumbrance on one- and two- handed interactions with mobile devices. In Proceedings of the Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada2014), ACM, 2557312, 1981-1990. DOI= http://dx.doi.org/10.1145/2556288.2557312.

[17] NG, A., WILLIAMSON, J.H., and BREWSTER, S.A., 2014. Comparing evaluation methods for encumbrance and walking on interaction with touchscreen mobile devices. In Proceedings of the Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services (Toronto, ON, Canada2014), ACM, 2628382, 23-32. DOI= http://dx.doi.org/10.1145/2628363.2628382.

[18] SARSENBAYEVA, Z., GONCALVES, J., GARCÍA, J., KLAKEGG, S., RISSANEN, S., RINTAMÄKI, H., HANNU, J., and KOSTAKOS, V., 2016. Situational Impairments to Mobile Interaction in Cold Environments. In Proceedings of the International Joint Conference on Pervasive and Ubiquitous Computing (2016), 85-96. DOI= http://dx.doi.org/10.1145/2971648.2971734.

[19] SCHILDBACH, B. and RUKZIO, E., 2010. Investigating selection and reading performance on a mobile phone while walking. In Proceedings of the Proceedings of the 12th international conference on Human computer interaction with mobile devices and services (Lisbon, Portugal2010), ACM, 1851619, 93-102. DOI= http://dx.doi.org/10.1145/1851600.1851619.

[20] SEARS, A., LIN, M., JACKO, J., and XIAO, Y., 2003. When computers fade: Pervasive computing and situationally-induced impairments and disabilities. In HCI International, 1298-1302.

[21] VADAS, K., PATEL, N., LYONS, K., STARNER, T., and JACKO, J., 2006. Reading on-the-go: a comparison of audio and hand-held displays. In Proceedings of the Proceedings of the 8th conference on Human-computer interaction with mobile devices and services (Helsinki, Finland2006), ACM, 1152262, 219-226. DOI= http://dx.doi.org/10.1145/1152215.1152262.

[22] WOBBROCK, J.O., 2006. The future of mobile device research in HCI. In Proceedings of the CHI 2006 workshop proceedings: what is the next generation of human-computer interaction (2006), 131-134.

[23] YESILADA, Y., HARPER, S., CHEN, T., and TREWIN, S., 2010. Small-device users situationally impaired by input. Computers in Human Behavior 26, 3 (2010/05/01/), 427-435. DOI= http://dx.doi.org/http://dx.doi.org/10.1016/j.chb.2009.12.001.