About

Diary Studies are a type of qualitative longitudinal user research (Lallemand, C. 2012) used to record organic behavioural insights. Participants of diary studies self-report data about their experiences and activities, over a period of time. During this period participants will log data, collected in in a natural context, using a prescribed recording method. (Palen L., Salzman M. 2001)

Diary studies are also known as experience sampling or EMA (ecological momentary assessment) methodology (Wikipedia 2017, Experience sampling method.).

Diary studies can be either Elicitation Studies – where participants record material for later interviews, or Feedback Studies – where predefined topics, and question data are recorded by participants.

The most popular methods of recording diary study data are:

• Paper-based diaries

• Email-based diaries

• Diary websites/apps

• SMS / Messenger Apps

• Social media-based diaries

• Video

(Wikipedia, 2017,  Diary studies) (Sullivan B. K. 2012)

The two most common types of data collection are, In-Situ Logging and the Snippet technique. With In-Situ Logging, more suited to less frequent logging, participants record all relevant information about activities as soon as they occur.

Using the Snippet Technique, participants record fragments of data as they happen, then later they expand on each snippet with additional detail.

Entries are elicited in one of three ways:

• Interval-contingent protocol – participants record entries at predefined intervals, or times
• Signal-contingent protocol – participants record entries when prompted
• Event-contingent protocol – participants record entries when specific events happen

(Lallemand, C. 2012)

Diary studies can help understand behaviours, including:

• Habits – what times/intervals the participant engages with the product
• Attitudes and motivations for performing tasks
• Usage scenarios – recording the participant’s engagement and tasks, while engaged with the study topic
• Monitoring behaviors and perceptions including user loyalty and system learnability
• Customer journeys – while the participant is engaged with the researcher’s product or organisation

(Flaherty K. 2016)

Methodology

A diary study typically spans five distinct phases:

1. Planning and Preparation – defining the focus of the study, the behaviours being analysed, the timeframe, recruiting participants, choosing the tools to be used for data gathering, and analysis of pre-study data
2. Prestudy brief – meetings with each participant to discuss the study, expectations, and diary schedule
3. Logging period – discussing when, and what, information participants should log
4. Post-study interview  – follow-up interview to discuss data in more detail with participant
5. Data Analysis – evaluate data recorded during the study. Analyse for behaviour evolution, and influences

(Flaherty K. 2016)

Benefits

Diary studies benefits include:

• Reporting real events and experiences in context – pure data
• Research laboratory not needed
• Researcher not needed to be present – the absence of a researcher also lessens the risk of social desirability bias
• Wealth of readily-available data collection methods

Limitations

• Data quality dependant on participant quality
• Detailed training required for participants
• Data recall accuracy can be unpredictable, especially when running an Elicitation-type study
• Repeated diary updates place heavy demands on participants
• Reactance, habituation, increased complexity, and gradual entrainment, are possible, particularly in more socially reactive behaviors. (Bolger N., Davis A., Rafaeli E. 2003)

Case study

A Diary Study of Task Switching and Interruptions

Microsoft Research

http://research.microsoft.com/en-us/um/people/horvitz/taskdiary.pdf

Introduction

This case study, by Microsoft Research, describes a diary study, in Feedback format, which attempted to understand the effect of interruptions, on switching tasks, for 11 users of Microsoft products in their day-to-day job.

The week-long study saw participants record details including “Time of Task Start, Difficulty Switching to the Task, What Documents Were Included in the Task, What Was Forgotten If Anything, Comments, and the Number of Interruptions Experienced” (Czerwinski M., Horvitz E., Wilhite S.)

The result of the diary study being a product called the GroupBar – a organisable shortcut menu on the Windows taskbar, of project-related documents and applications.

Critical Analysis

As one would expect, from a company like Microsoft, the study was very well planned and executed, the only noteworthy issues being:

• One participant, a stockbroker reported, in his diary, that every phone call he made was a separate task, this was an incorrect assumption that resulted in skewed data – this issue suggests the prestudy brief stage wasn’t detailed enough, and the participant wasn’t briefed fully
• A Microsoft Excel spreadsheet was used by participants to record data, this non-standard method of recording diary data, along with the frequency of the  event-triggered diary entries, would doubtlessly have placed heavy demands on participants

References

Lallemand, C. (2012). Dear Diary: Using Diaries to Study User Experience. User Experience Magazine, 11(3). Retrieved February 1, 2017 from http://uxpamagazine.org/dear-diary-using-diaries-to-study-user-experience/

Palen L., Salzman M. (2001) Voice-Mail Diary Studies for Naturalistic Data Capture under Mobile Conditions. Retrieved February 1, 2017 from https://cmci.colorado.edu/~palen/Papers/voice.pdf

Wikipedia (2017) Experience sampling method. Retrieved February 1, 2017 from https://en.wikipedia.org/wiki/Experience_sampling_method

Wikipedia (2017) Diary studies. Retrieved February 1, 2017 from https://en.wikipedia.org/wiki/Diary_studies

Sullivan B. K. (2012, August 9). 5 Methods to Collect Data with Diary Studies. Retrieved February 3, 2017 from http://bigdesignevents.com/2012/08/5-methods-to-collect-data-with-diary-studies/

Flaherty K. (2016, June 5). Diary Studies: Understanding Long-Term User Behavior and Experiences. Retrieved February 3, 2017 from https://www.nngroup.com/articles/diary-studies/

Bolger N., Davis A., Rafaeli E. (2003) Diary Methods: Capturing Life as it is Lived. Retrieved February 4, 2017 from http://www.columbia.edu/~nb2229/docs/bolger-davis-rafaeli-arp-2003.pdf

Czerwinski M., Horvitz E., Wilhite S. A Diary Study of Task Switching and Interruptions. Retrieved February 7, 2017 from http://research.microsoft.com/en-us/um/people/horvitz/taskdiary.pdf