Tal August, Nigini Oliveira, Chenhao Tan, Noah A. Smith, and Katharina Reinecke.
In Proceedings of the Conference on Computer-Supported Cooperative Work and Social Computing (CSCW'2018).
Abstract:
Online experimentation with volunteers relies on participants' non-financial motivations to complete a study,
such as to altruistically support science or to compare oneself to others. Researchers rely on these motivations
to attract study participants and often use incentives, like performance comparisons, to encourage participation.
Often, these study incentives are advertised using a slogan (e.g., "What is your thinking style?"). Research on
framing effects suggests that advertisement slogans attract people with varying demographics and motivations.
Could the slogan advertisements for studies risk attracting only specific users? To investigate the existence of
potential sample biases, we measured how different slogan frames affected which participants self-selected
into studies. We found that slogan frames impact recruitment significantly; changing the slogan frame from a
'supporting science' frame to a 'comparing oneself to others' frame lead to a 9% increase in recruitment for
some studies. Additionally, slogans framed as learning more about oneself attract participants significantly
more motivated by boredom compared to other slogan frames. We discuss design implications for using frames
to improve recruitment and mitigate sources of sample bias in online research with volunteers.
[PDF]
@inproceedings{august+etal:18,
author = {Tal August and Nigini Oliveira and Chenhao Tan and Noah A. Smith and Katharina Reinecke},
title = {Framing Effect: The Choice of Slogans Used to Advertise Online Experiments Can Boost Recruitment and Lead to Sample Biases},
year = {2018},
booktitle = {Proceedings of CSCW}
}