2014 MeasuringUserEngagement

(Lalmas et al., 2014) ⇒ Mounia Lalmas, Heather O'Brien, and Elad Yom-Tov. (2014). “Measuring User Engagement.” Morgan & Claypool Publishers. ISBN:1627052615, 9781627052610 doi:10.2200/S00605ED1V01Y201410ICR038

Subject Headings: User Engagement Measurement.

Notes

Related to (Lalmas et al., 2013) WWW-2013 Tutorial
- http://www.dcs.gla.ac.uk/~mounia/Papers/WWWTutorial2013UserEngagement.pdf
- http://www2013.w3c.br/program/measuring-user-engagement/

Cited By

Quotes

Author Keywords

user engagement, user experience, methods, measurement, online activity, eye tracking, loyalty, mouse tracking, popularity, physiological measurement, self-report measurement, web analytics, intra-session, inter-session, reliability and validity, multi-tasking, dwell time, mixed methods

Abstract

User engagement refers to the quality of the user experience that emphasizes the positive aspects of interacting with an online application and, in particular, the desire to use that application longer and repeatedly. User engagement is a key concept in the design of online applications (whether for desktop, tablet or mobile), motivated by the observation that successful applications are not just used, but are engaged with. Users invest time, attention, and emotion in their use of technology, and seek to satisfy pragmatic and hedonic needs. Measurement is critical for evaluating whether online applications are able to successfully engage users, and may inform the design of and use of applications. User engagement is a multifaceted, complex phenomenon; this gives rise to a number of potential measurement approaches. Common ways to evaluate user engagement include using self-report measures, e.g., questionnaires; observational methods, e.g. facial expression analysis, speech analysis; neuro-physiological signal processing methods, e.g., respiratory and cardiovascular accelerations and decelerations, muscle spasms; and web analytics, e.g., number of site visits, click depth. These methods represent various trade-offs in terms of the setting (laboratory versus [[in the wild]]), object of measurement (user behaviour, affect or cognition) and scale of data collected. For instance, small-scale user studies are deep and rich, but limited in terms of generalizability, whereas large-scale web analytic studies are powerful but negate users' motivation and context. The focus of this book is how user engagement is currently being measured and various considerations for its measurement. Our goal is to leave readers with an appreciation of the various ways in which to measure user engagement, and their associated strengths and weaknesses. We emphasize the multifaceted nature of user engagement and the unique contextual constraints that come to bear upon attempts to measure engagement in different settings, and across different user groups and web domains. At the same time, this book advocates for the development of good measures and good measurement practices that will advance the study of user engagement and improve our understanding of this construct, which has become so vital in our wired world.

Table of Contents: Preface / Acknowledgments / Introduction and Scope / Approaches Based on Self-Report Methods / Approaches Based on Physiological Measurements / Approaches Based on Web Analytics / Beyond Desktop, Single Site, and Single Task / Enhancing the Rigor of User Engagement Methods and Measures / Conclusions and Future Research Directions / Bibliography / Authors' Biographies / Index

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv
1 Introduction and Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
1.1 Definitions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 User Engagement vs. User Experience . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Characteristics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3.1 Focused Attention . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3.2 Positive Affect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.3.3 Aesthetics Appeal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.4 Endurability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.5 Novelty . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.3.6 Richness and Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.7 Reputation, Trust, and Expectation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.3.8 User Context, Motivation, Incentives, and Benefits . . . . . . . . . . . . . . . . . 7
1.4 Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
1.5 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.6 Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2 Approaches Based on Self-Report Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.1 Self-Report Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.2 Advantages and Limitations of Self-Report Methods . . . . . . . . . . . . . . . . . . . . 12
2.2.1 Communication . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.2 Methodology Bias . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.2.3 Reliability and Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.3 Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.1 Types of Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.3.2 Applying Interviews to Measure User Engagement. . . . . . . . . . . . . . . . . 16
2.4 Think Aloud and Think After Protocols . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4.1 Think Aloud . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4.2 Think After . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.3 Relationship to User Engagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.5 Questionnaires . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.6 Questionnaires for Measuring User Engagement . . . . . . . . . . . . . . . . . . . . . . . . 22
2.6.1 Survey to Evaluate Engagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.6.2 Engagement and Influences on Questionnaire . . . . . . . . . . . . . . . . . . . . . 23
2.6.3 User Engagement Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.6.4 eHealth Engagement Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.7 Constructs Related to User Engagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.7.1 Mental Workload . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.7.2 Disorientation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.7.3 Playfulness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.7.4 Cognitive Absorption . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
2.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3 Approaches Based on Physiological Measurements . . . . . . . . . . . . . . . . . . . . . . . 31
3.1 Psychophysiological Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.2 Facial Expressions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3 Eye Tracking. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.3.1 Eye Tracking and Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3.2 Eye Tracking and Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.3.3 Eye Tracking and Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
3.3.4 Summary and Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.4 Cursor Tracking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.4.1 Aligning Eye Gaze and Mouse Movement . . . . . . . . . . . . . . . . . . . . . . . 39
3.4.2 Mouse Movement in Search . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
3.4.3 Mouse Movement Elsewhere . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
4 Approaches Based on Web Analytics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.1 Intra-Session vs. Inter-Session Engagement . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
4.2 Some Dimensions of Online Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.2.1 Dependence on the Type of Website . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
4.2.2 Dependence on the Type of User . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
4.2.3 Dependence on the Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.3 Large-Scale Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
4.4 Intra-Session Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.4.1 Dwell Time and Similar Measures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
4.4.2 Revisits to a Site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
4.4.3 Clickthrough Rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.4.4 Number of Pages Viewed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
4.4.5 Other Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.5 Inter-Session Measurements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
4.5.1 Direct Value Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.5.2 Total Use Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
4.5.3 Return-Rate Measurement . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
5 Beyond Desktop, Single Site, and Single Task . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5.1 Measuring for Online Multitasking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
5.2 Measuring on a Network of Sites. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
5.3 Measuring in Mobile Information Seeking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
5.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
6 Enhancing the Rigor of User Engagement Methods and Measures . . . . . . . . . . 73
6.1 Scale . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
6.2 Setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76
6.3 Temporality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
6.4 Objectivity and Subjectivity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
6.5 Process- and Product-Based . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
6.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
7 Conclusions and Future Research Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
7.1 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
7.2 Future Research Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
7.3 Take-Aways . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Authors’ Biographies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115

Preface

User engagement is not a new concept, but has bubbled to the surface over the past decade, attracting a number of academic and industry researchers from diverse fields, including information, computer, and learning sciences. We live in a highly connected society, as evidenced from the plethora of web-ready devices and how these have been integrated into our everyday work and personal lives. Engaging the user has become a goal — and to some extent, a necessity — for online search companies, e-commerce firms, news portals, etc. and a source of interest for those of us invested in understanding the relationship between people, technology and information. A major challenge in the study of user engagement continues to be the lack of a shared definition and understanding of user engagement and its measurement. The term “user engagement” is frequently referred to as a desired outcome of people’s interactions with information technology, but this means different things to different people; many people use the term without articulating their definition. This creates further dilemmas when we turn to measurement: if we cannot define user engagement, then how do we know whether we have actually measured it? Our impetus in writing this book was to shed light on these challenges, but also to highlight important existing and emerging work in this area. Although our focus here is on measurement, we ground the book by first defining user engagement and its characteristics. We also list important measurement attributes, such as scale and temporality. These measurement attributes are used to frame our discussion and to encourage researchers to consider factors of scale, time, etc. when determining their approach to measurement in a given context.

We then move into describing three approaches: self-report, web analytics, and physiological methods. In this book, we devote a chapter to each of these approaches. These chapters focus specifically on how self-report, web analytics, and physiological methods have been employed to measure engagement, and their advantages and limitations. Self-report approaches rely on asking users about their experiences though questionnaires, interviews, and so on. Alternatively, physiological measures, which can track user gaze and mouse movements can be performed in the lab and, increasingly, by Internet companies. The latter, however, still mostly relies on web analytic approaches such as calculating dwell time and return rate.

In the final chapters of the book, we share emerging measurement techniques in the areas of mobile search, the networked environment and multi-tasking. In doing so, we push the measurement agenda for user engagement “beyond the desktop, a single site, and a single task” to more closely reflect the ways in which technology users are interacting with web applications and their use of different devices to do so. We conclude by returning to the measurement attributes, and discuss these with respect to enhancing the rigor of measurement application.

This book makes strong references to our own work and relies on the expertise we have developed working in this area. It takes years to properly understand and talk confidently about what user engagement is, what it is not, and what a particular measurement tells us about it. Particularly important for us was to present the three main types of approaches in detail to fully comprehend what each can bring to the study of user engagement. Combining approaches will allow for a richer and deeper understanding of what user engagement is, how to measure it properly, and how to interpret the measurement outcomes to then be able to act upon them in an informed manner.

Our belief is that this text will provide a solid foundation for defining and measuring user engagement in both research and application. It is our intention to promote an active and robust agenda for user engagement and its measurement across various fields of inquiry.

CHAPTER 1 - Introduction and Scope

The nature of human-computer interactions has dramatically changed over the past several decades. In the 1980s the dominant interaction paradigm was of a solitary individual sitting in front of one workstation, most probably in an office, manipulating the interface through a keyboard and mouse. Fast forward to the 21st century where human-computer interaction’s (HCI) “third wave” [25] is dominated by discourse on User Experience (UX). UX is framed as “a consequence of a user’s internal state (predispositions, expectations, needs, motivation, mood, etc.), the characteristics of the designed system (e.g., complexity, purpose, usability, functionality, etc.) and the context (or the environment) within which the interaction occurs (e.g., organizational/social setting, meaningfulness of the activity, voluntariness of use, etc.).” [74]

…

References

;

	Author	volume	Date Value	title	type	journal	titleUrl	doi	note	year
2014 MeasuringUserEngagement	Mounia Lalmas Heather O'Brien Elad Yom-Tov			Measuring User Engagement				10.2200/S00605ED1V01Y201410ICR038		2014