The global thought leadership industry relies almost entirely on surveys as their primary method of capturing original data for studies. But there’s an issue that hardly anyone likes to talk about: the rapid rise of fraud within online survey panels.

The issue has been around for well over a decade, but a recent news story, and a wider trend, are pushing this more into the limelight.

The news story was a recent US Attorney’s Office announcement of charges against eight individuals in relation to a scheme to bill $10m in fraudulent market research data. The indictment centres on US market research companies Op4G and Slice, and alleges that senior leaders from the companies decided to boost revenues by fabricating survey data.

This is not an isolated incident. Survey fraud is most pervasive in the B2C polling space, but it affects the world of B2B too. By its very nature, it’s exceedingly difficult to know how widespread the issue is, but the trendline has been moving in the wrong direction for years: one academic study (with the fantastic title: “Yes stormtrooper, these are the droids you’re looking for”) found that in some cases up to 99% of respondents gathered via online panels have been fraudulent.

Another paper shows the wide-ranging extent of what the authors politely term ‘insincere’ respondents. The latter paper also highlights that these respondents don’t just answer at random, but typically opt for positive answer choices, which in turn introduces ‘small, systemic bias’. This is just one of the reasons why public polls of presidential candidates or approval ratings have so often been out of tune with reality.

Hey, I’m a global CEO too

The wider trend is of course AI, which is supercharging the issue of fraudulent data. This is thanks to the ability for bad actors to create AI bots or agents that respond automatically to surveys, in exchange for a survey incentive. While incentives usually aren’t huge, there has always been a cottage industry of ‘professional’ survey takers who repeatedly fill in surveys as a way of generating cash - and who are adept at gaming the survey and pretending to be a certain kind of respondent. But the economics get a lot better if you can simply build an AI bot to constantly fill in surveys for you.

ESOMAR, a market research industry body, has long agreed that research fraud is a big issue and is actively collaborating with other bodies, including The Insights Association, MRS and SampleCon, to try and find new ways to tackle it. Already last year, it quoted the chief executive of the MRS, Jane Frost, saying: “The threat of fraud in research isn’t new… However, fraudulent activity is becoming increasingly sophisticated, particularly in online research.” (My favourite line from this announcement is the last sentence: “Research, including fielding a survey about fraud detection, is being discussed.” – only the market research industry would propose to do a survey on fraud as part of their response to survey fraud.)

Which bot is best for B2B?

So if you’ve got a study you need to conduct and you plan to poll a B2B audience for it, what can you do about the issue of fraud? Inevitably, there’s no one magical remedy to resolve the issue, but there are various options that can be considered:

1. Poll your own audience. This would usually be the best option given first hand knowledge of the audience you’re surveying, but oftentimes it’s the one least likely to be viable. Simply having a large contact list is not enough: companies need to invest the effort and discipline into setting up the right tools, processes and governance for actually generating a response from their lists (and protecting them from being bombarded with requests). This typically requires a multi-year effort, including carefully priming their audience about why they’re being contacted and what’s in it for the recipient. All too often, companies simply carpet bomb a mailing list and are then surprised when they get little to no response. However, for those who do invest in getting this to work, it’s a huge boon - and far more likely to deliver a fraud-free response.
2. Skip the panel. For companies working with a research vendor, try to avoid those relying on their large existing database of survey panelists. The reality is that the underlying logic of a panel isn’t set up in your favour: people sign up for points or cash, and they need to get through a large number of surveys quickly in order to make it worth their while. This really isn’t what you’re looking for, especially in the world of B2B. Where possible, you’re likely to be better off pushing for a bottom-up approach - getting your research partner to reach out directly to relevant people, and convincing them to take a survey, along with a suitable incentive. Not all vendors are set up for this approach and the costs will be higher, but it allows for much better verification of your target audience.
3. Go qualitative only. This is usually harder, more expensive, and will typically only deliver a smaller sample, but it’s also an approach rich in depth and insight and unlikely to contain any fraudulent responses. Among some research providers, AI is also enabling qualitative surveying at a greater scale, although this remains new - and the best quality still comes from directly conducting the interviews yourself. To bolster the numbers, it helps to conduct these interviews anonymously, although of course having them on the record makes for a much richer report or set of insights at the end.
4. Revamp your checks and balances. If your only viable option is a panel, then there are a host of things that can be done to help weed out bad respondents. From logic checks (eg, if a person says in one question that they’re investing in something today, but later on says they’re not investing in that thing), through to simple sense checks (eg, is that 18-25 year old respondent really a CEO of a $1bn company - well, no), to checking the quality of the open-end responses, and so on. It’s a lot of work, and you’ll often be forced to ditch a lot of your sample, but this is a necessary evil. Importantly, though, the methods that worked in the past likely need to be evolved for the current reality, as an excellent new research paper makes clear. The authors reviewed and assessed 31 different fraud detection strategies, and helped outline which work most effectively. There’s no one silver bullet, but helpful methods range from email address analysis (apparently AI-powered bots often use emails with Capped Names, such as JohnDoe@Email.Com), through to analysis of IP location, consecutive start timings, and much more. One recommendation that particularly resonated: changing the incentives. Either use a location-based payment service (eg, Venmo for the US), or consider an invitation or physical gift. Years ago at the Economist Intelligence Unit, we used to post physical books to respondents as a thank you (a particular boon for any in-house authors looking to boost sales of their titles). This had the handy side-effect of requiring a real postal address, which helped with verification. And AI bots aren’t that keen on physical books.

_{Photo: Photo by Tom Barrett on Unsplash}