2008-02-07T11:52:00.000-08:00

Technology considerations for improving market research data quality

Last year, the market research industry took a black eye, with things like the same provider delivering diametrically opposed results with two surveys a week apart - leaving advertisers and marketers questioning the quality of the market research industry. Because they rely so heavily on this data for product development and launches, their frustration undermines the advancement of our profession. So what can be done to slow the erosion of industry credibility?

A good first step occurred last fall when the ARF founded the Online Quality Research Council (OQRC). But we, as an industry, may need even more than best practices in order to resolve the data quality crisis.

Before examining some thoughts on what can be done to remedy this, let’s take a look at what is dragging research quality down.

Some fraudulent issues:

Professional survey respondents account for 30 percent of all surveys, when they actually compose 0.25 percent of survey-takers, leading to poor sample quality.
Then we have the straight-liners and satisficers who race through to the survey’s finish at the expense of quality.
What about safeguards against bots to prevent fraudulent and inaccurate data?
And lastly, the same, bad recruiting sources have yet to be discouraged via responsible panel management.

We are certain we should and can do more. Let’s start with the ARF OQRC standards and then go a step further by introducing technology-based standards to define, audit, measure and ultimately resolve challenges to data quality.

We submit the following points to be considered in order to improve data quality:

Identify unique respondents & computers
- This is one of the most fundamental requirements in measuring online traffic and data quality for analytics in online advertising, e-commerce and traffic measurement.
- Let’s move beyond easy-to-manipulate cookies and email addresses to embrace technologies like heuristic machine ID, IP addresses, or even the installation of ActiveX or Java controls.

De-Duplication Elimination
- Panelists who belong to several online access panels, or having multiple accounts within the same panel, need to be de-duped before going into the same survey.
- Their duplicate accounts also need to be restricted or deleted.

Researcher submission of patterned or fraudulent respondents
- Researchers should be able to submit respondent IDs or other non-invasive identifiers (that comply with privacy legislation) of bad respondents to create a history of bad respondents.
- These would include satisficing respondents, lying respondents, patterned respondents, inconsistent respondents and other categories of undesirable respondents.

Create and Maintain a Database or Blacklist of Bad Respondents across Panels
- The email world has a spammer black list. Let’s do it for bad respondents across all major online panels.
- Bad respondents would be uploaded into such a database by researchers and flagged automatically if they violate any automatic security controls.

Speeding and Straight-lining Prevention
- Survey platforms and softwares should have the built-in capability to detect and filter out speedsters.

We are convinced that the market research industry is capable of addressing many of the data quality issues it faces to restore the credibility the industry deserves. By coupling the ARF’s best practices with emerging technology standards, a roadmap to a future of quality data is visible, and ready to be implemented for our own good.

What do you think? What ideas do you have to restore quality and credibility in the market research industry?

2007-09-13T11:23:00.000-07:00

Peanut Labs is hiring!

http://peanutlabs.com/jobs.php

All talented engineers are welcome to apply.

2007-08-24T09:01:00.000-07:00

Something for the publishers....

Some of you might have noticed our new Flex interface for the iFrame. We're currently working on a similar interface for a publisher dashboard. Here's a sneak preview:

Publishers will now be able to keep a pulse on their panelists and best of all, be reminded of how much money they're making on the Peanut Labs network.

2007-08-21T17:17:00.000-07:00

The market research industry is in trouble. Someone HELP! We're shooting ourselves in the foot.

http://adage.com/article?article_id=119924

2007-08-12T11:27:00.001-07:00

Peanut Labs in Venturebeat

2007-08-12T09:00:00.000-07:00

Forrester also chips in. Interesting stuff.

Click here

2007-08-12T08:43:00.000-07:00

Very interesting presentation from e-rewards on quality issues in online panels. I'm a fan!

Click here

Uncensored ravings about Market Research - Peanut Labs