Quality of online research

Improving the Quality of Your Online Research: Ten Questions to Ask Your Online Survey Provider

If you want to ensure or improve the quality of your online research, there are ten key questions you should be asking your online survey provider. The questions were developed by MRIA's Research and Development Committee:

1. What type of sample frame will you use for my project, and why is this frame the best way to approximate my target population? Is the best sample frame provided by a proprietary access panel, a proprietary panel augmented with other panels, a blended panel, a river sample, a customer list, or some other list or source?

2. What methods and types of sources will be used to recruit respondents to this sample frame? Some specific lists, websites, or recruiting methodologies could bias the sample in ways that affect the data for your particular product or service category.

3. How will my project sample be drawn or selected from this sample frame? Will a random method be used to select potential respondents for each cell or respondent category in your sample?

4. What incentives or inducements will be given to respondents? The incentive should be as neutral as possible in terms of the research project and the target population, thereby ensuring that it does not lead to response bias.

5. What question formats and techniques will you use to reduce respondent boredom and improve the quality of response? Ongoing research is identifying better ways to present questionnaires online. Is your supplier keeping up to date?

6. How do you ensure the sample is "deduped" – that the same respondent doesn't appear more than once? Simply searching for identical email addresses is not enough. This area is evolving rapidly, and a variety of new techniques are being employed to weed out dupes.

7. How do you strive to ensure that the respondents are who they say they are? Techniques are evolving to better ensure that each respondent is genuine and not "manufactured." Is your supplier on top of this?

8. How do you identify and remove fraudulent and inattentive respondents? A variety of planted questions and data checks should be used to clear out cheaters and satisficers.

9. What quota controls and/or weighting procedures will be used to make the final sample look more like my target population? With few exceptions, most online sample sources are not adequately proportioned to approximate the general population or your particular target population. Careful adjustments need to be made, before and after data collection, to achieve this approximation. Based on statistical science principles, MRIA's Code of Conduct and Good Practice prohibits statements about margins of sampling error or population estimates with regard to convenience samples, including most online panels. Therefore, you should neither request nor use a margin of sampling error statement for surveys that use nonprobability/convenience samples.

10. Is your firm a Corporate member of MRIA? Corporate members of MRIA have agreed to comply with critical ethical and legal standards as set out in MRIA's Code of Conduct and Good Practice – important to be sure that the projects conducted on behalf of your organization do not contravene accepted community standards, or the law.

Donald Williams is vice-president and research director at NADbank, the principal research arm of the Canadian newspaper industry. He has worked in market research, telecom pricing strategy, and business forecasting. He can be reached at dwilliams@nadbank.com.