TGM RESEARCH BLOG
Fighting the Good Fight: Improving Research Quality and Tackling Online Survey Fraud
Learn how to improve the quality of your research and protect against online survey fraud. Don't let survey scams ruin your research sample and lead to wrong decisions.
If you’re an in-house research professional, I’m sure you don’t need me to tell you that we have a data quality problem in market research. We all know it, we’ve all experienced it, and we’ve all had that moment where we stare at a crosstab gone wrong and wonder if our survey data is actually nonsense.
The problem seems daunting. But there’s good news—we’re making progress in battling research fraud. New fraud detection technologies offer real improvements, buyers are getting serious about quality, and passionate advocates are moving the conversation forward. In this post, I want to walk through the key challenges, review some encouraging developments, and offer practical guidelines on how we can all help improve research integrity.
The problem seems daunting. But there’s good news—we’re making progress in battling research fraud. New fraud detection technologies offer real improvements, buyers are getting serious about quality, and passionate advocates are moving the conversation forward. In this post, I want to walk through the key challenges, review some encouraging developments, and offer practical guidelines on how we can all help improve research integrity.
The Root Causes: Why We Have a Quality Crisis
It’s easy to blame “bad panelists” or "bad actors" and bemoan falling response rates, but the roots of our quality crisis run deeper than that. Here are some of the core issues:
- The Incentives Are Out of Alignment — In an ideal world, sample providers would be rewarded for supplying the highest quality, most representative respondents. But in practice, it’s faster and cheaper to cut corners, with few immediate consequences. Buyers under budget pressure focus more on cost and speed than quality assurance. These incentives need realignment.
- Lack of Transparency and Accountability — Sample sources and quality protocols often lack transparency. And even when suppliers make impressive quality claims, there are no clear mechanisms to validate them. This murkiness enables bad actors while punishing those investing in integrity.
- An Arms Race We’re Losing — As fraud detection improves, the fraudsters seem to stay a step ahead. Cheap, scalable technology like bots and click farms combined with the anonymity of the web have created an environment where bogus respondents thrive.
- Surveys Can Be Boring! — Let’s be honest, surveys can be dull. Poorly designed surveys with long grids quickly lead to disengaged straighteners. We expect a lot from survey respondents in return for a small incentive. Keeping them meaningfully engaged is an uphill battle.
So in the face of these systemic problems, what can we as researchers do? While we may not be able to overhaul incentives across the entire industry, there are steps all of us can take to improve data quality and manage the impacts of fraud.
Guidelines for Battling Sample Fraud and Improving Research Integrity
The key is a layered defense—no single tactic can catch all the bad actors. By combining provider selection, technical fraud detection, rigorous data cleaning, and continuous optimization, we can make real headway:
1. Choose Sample Suppliers Carefully
- Thoroughly vet potential suppliers on their source recruitment, panel management, survey quality control protocols, and technologies used.
- Ask for recent metrics they can validate on removal rates, fraud flags, etc. to benchmark vendors.
- Ensure they have secure ISO certifications or industry memberships like ESOMAR and comply with codes of conduct and standards.
- Avoid vendors who cannot discuss quality transparently or claim unrealistic perfection.
2. Employ Technical Fraud Detection Services
- Work with your sample provider to layer on survey fraud detection tech like IP address detection and mapping, device fingerprinting, etc.
- Understand the techniques used and customize sensitivity thresholds based on survey needs.
- Review diagnostic metrics to analyze patterns and adapt approaches over time.
3. Rigorously Clean Data Post-Collection
- Don’t just run crosstabs! Pore through individual responses to uncover red flags.
- Use survey data validation techniques like speeder analysis, open text review, and trap questions tailored to the audience.
- Categorize removal reasons (e.g. straightlining, gibberish, inconsistent geo/demos) to refine cleaning.
4. Continuously Optimize Survey Design
- Place quality checks early to terminate fraudulent participants ASAP to save money.
- Limit grid questions and incorporate engaging formats like MaxDiff, video, and gamification when possible.
- Analyze behaviors in pre-testing to identify problematic questions and minimize noise.
5. Benchmark Data Quality Metrics and Report Back
- Track key metrics for every online survey project on removals, duration, and traps failed—and report back to suppliers on issues and fraudulent cases.
- Build internal benchmarks by survey type, audience, and vendor to drive continuous improvements.
With a rigorous, multi-pronged approach combining vendor vetting, fraud detection tech, meticulous data cleaning, optimized survey design, and consistent benchmarking against key metrics, we can make a real dent in the bad data penetrating our insights programs.
Does it take more work? Yes. Does it impact survey speed and cost? For sure. But isn’t high-quality, unbiased data the whole point of our business? By taking data integrity seriously, focusing relentlessly on quality, and working hand-in-glove with the best suppliers, we can raise the bar across the industry.
Encouraging Signs: New Solutions and Growing Advocacy
If reading the previous pages left you inspired but a little depressed about what we’re up against, take heart. As awareness spreads, the fight for survey quality is intensifying—and the cavalry has some impressive new weapons.
Does it take more work? Yes. Does it impact survey speed and cost? For sure. But isn’t high-quality, unbiased data the whole point of our business? By taking data integrity seriously, focusing relentlessly on quality, and working hand-in-glove with the best suppliers, we can raise the bar across the industry.
Encouraging Signs: New Solutions and Growing Advocacy
If reading the previous pages left you inspired but a little depressed about what we’re up against, take heart. As awareness spreads, the fight for survey quality is intensifying—and the cavalry has some impressive new weapons.
Targeted Survey Fraud-Fighting Innovation
A new breed of fraud detection firms offers powerful new techniques to identify the savviest bogus respondents or malicious responses. Leveraging vast device databases, IP address mapping, machine learning models, and vast experience fighting online security threats in other domains not only limited to survey responses, these solutions are light years beyond old-school deduping software.
And crucially, rather than a blunt yes/no on each respondent, most provide risk scoring and advanced diagnostics to help researchers evaluate tradeoffs and fine-tune the stringency. As the arms race against fraud accelerates, it’s encouraging to see sophisticated new tools bringing thoughtfulness, customization, continuous optimization, and usable intelligence.
And crucially, rather than a blunt yes/no on each respondent, most provide risk scoring and advanced diagnostics to help researchers evaluate tradeoffs and fine-tune the stringency. As the arms race against fraud accelerates, it’s encouraging to see sophisticated new tools bringing thoughtfulness, customization, continuous optimization, and usable intelligence.
Galvanized in Panel Fraud Battle
Different industry groups in the insights community understand the urgency, and many are stepping up, driving important conversations around technology, transparency, and building buyer trust. Key coalitions are also conducting research benchmarking fraud rates and evaluating detection solutions across the market research space. Their studies have revealed alarming statistics - including significant variation across detection vendors on who gets flagged as legitimate versus fraudulent.
Within leading organizations, fed-up research leaders are tackling the problem head-on - getting internal backing to enact reforms around auditing suppliers, bringing sampling in-house, and demanding better accountability. To enact change, these leaders emphasize aligning incentives through transparency, benchmarking vendor performance rigorously against threshold targets, applying fraud detection technologies religiously, and cleaning data meticulously before fielding.
Crucially, many are willing to pay more for online panels with better data integrity and accept the longer timelines required for research quality - bucking the temptation to rush analysis of questionable data. As one insight leader argued, “Bad data kills products.” If higher quality means a launch takes a quarter longer but prevents market failures, it’s worth it.
Within leading organizations, fed-up research leaders are tackling the problem head-on - getting internal backing to enact reforms around auditing suppliers, bringing sampling in-house, and demanding better accountability. To enact change, these leaders emphasize aligning incentives through transparency, benchmarking vendor performance rigorously against threshold targets, applying fraud detection technologies religiously, and cleaning data meticulously before fielding.
Crucially, many are willing to pay more for online panels with better data integrity and accept the longer timelines required for research quality - bucking the temptation to rush analysis of questionable data. As one insight leader argued, “Bad data kills products.” If higher quality means a launch takes a quarter longer but prevents market failures, it’s worth it.
Turning the Corner on Quality: 3 Reasons for Optimism
With advanced detection tools, coordinated industry action, and awakened advocates rallying organizations to higher standards, the tide may be turning on data quality. Here are three reasons I’m bullish:
1. We’re Following a Winning Playbook
In other domains like digital advertising and social media metrics, problems with viewability, attribution, bots, and more have caused crises of confidence. But over the last decade, major progress emerged on technical safeguards, industry standards, audited metrics, and rebuilding trust. With our new generation of fraud fighters, the insights space is poised for a similar step-change.
2. The Costs of Inaction Are Untenable
Frankly, the industry can’t afford not to tackle this existential threat. At a time when growth hangs in the balance, no firm can thrive selling unreliable insights derived from specious data. As margins tighten and skeptics question research value, our differentiator and lifeline is providing truth and certainty to our clients. Squandering that advantage would be suicidal.
3. Client Power is Mobilizing
Ultimately, the buyer holds the leverage needed to transform sampling ecosystems, drive transparency, and stimulate innovation. But wielding that power effectively takes coordination across clients, applying pressure collectively, and making sustained integrity a condition of purchase.
With frustration reaching a boiling point, signs indicate buyers are ready to act in concert and not take no for an answer. From contract language locking in accountabilities to payment policies denying suspect samples, their voices are essential to catalyzing change.
With frustration reaching a boiling point, signs indicate buyers are ready to act in concert and not take no for an answer. From contract language locking in accountabilities to payment policies denying suspect samples, their voices are essential to catalyzing change.
Here are 3 key ways you can contribute to improve online research integrity right now:
1. Lend Your Voice to Collective Action
Efforts by groups like CASE, ARF, and IA marking steady progress rely on people advocating priorities to accelerate reform. We need your help shaping the solutions. Join working groups, advocate internally for investment, provide technical insights on new validation techniques, and keep these conversations energized across conferences and industry gatherings.
2. Walk the Walk in Your Organization
Preach what we’ve covered far and wide across your company and functions. Actively collaborate with insights leads in Marketing, Sales, and Product, and across departments to educate colleagues on what’s at stake. Pitch the imperative of quality mandates in supplier contracts. And volunteer to help teams overhaul tracking studies or instrument key surveys to surface data quality problems early.
3. Evangelize Empowered Buying
Our power as buyers is only as strong as how broadly and uniformly we wield it. Beyond your own teams’ projects, help uphold higher standards as advocates in peer groups, network discussion forums, and more. Compare notes on vendor selection practices, terms and conditions used, and criteria for barring suppliers. Inspire others to see integrity as an unavoidable cost of insights done right.
By adding your energy to the groundswell through these channels and more, we collectively transform this from a research industry problem to a research industry opportunity—a chance to earn back lost trust in what we do and how we do it at a time when facts matter more than ever.
Who’s with me? The opportunity is enormous, and the time is now. Let’s work together to restore confidence, help our leaders succeed, and show why insights done the right way remain so vital. Our clients, our companies, and society at large depend on it.
Who’s with me? The opportunity is enormous, and the time is now. Let’s work together to restore confidence, help our leaders succeed, and show why insights done the right way remain so vital. Our clients, our companies, and society at large depend on it.
Implementing these responsible procedures ushers progress in studying society’s most wrenching issues to guide supportive policymaking and compassionately challenge stigma. Now more than ever, sensitive research that grasps people’s authentic realities provides ballast against reactionary resistance and sanctimonious prejudice, keeping equality, understanding, and justice ever out of reach for all. May conscientious surveys pave the way to progress, lifting up those whom ignorance, indifference, and intolerance marginalize needlessly.