Sensory Perception & Interaction, Research Group

This work investigates a novel interaction paradigm of implicit, low-attention user control, accomplished by monitoring a user’s physiological state. Additionally, we provide feedback to the user regarding system changes in response to this implicit control through touch. We explored the implicit interaction concept, termed the ‘Haptic-Affect Loop’ (HALO), in the context of two use cases. In the first, we developed a HALO interaction to bookmark then resume listening to an audio stream when interrupted. A user’s galvanic skin response is monitored for orienting responses (ORs) to external interruptions; our prototype automatically bookmarks the media, allowing the user to resume listening from the point he/she is interrupted. In a controlled environment, we found an OR detection accuracy of 84%. We further investigated the usefulness of two forms of haptic feedback for bookmarking: notification of bookmark placement and display of bookmarks during navigation. Results show that haptic notification is able to provide a significant performance benefit in terms of navigation speed when paired with visual-spatial indications of where bookmarks have been placed over conditions with no notification; also, performance was no worse with the haptic display of bookmarks than the visual display. Participants tended to prefer haptic notification at interruption time, and both haptic and visual display of bookmarks at resumption. We used a second use case, music-listening, as a framework for another HALO implementation to estimate user preference for music. In a pilot experiment, we collected physiological and music rating data while participants listened to music. Combining this with structural feature data extracted from the music, we obtained state-space models which were used by a Kalman filter to estimate user rating of music selections. Results currently show poor performance in terms of ability to estimate rating of music both known and unknown to users possibly due to a false assumption of system linearity. The outcome of this effort was a first pass implementation of HALO which provided insight into its strengths and weaknesses. This work enables further exploration in how HALO can benefit interactions in other contexts by providing validation of the technological feasibility, utility and behavior of the Haptic-Affect Loop.