Survey Bots are like a stomach virus. Something doesn’t feel right, and the next thing you know, you have a big problem. And it stinks.
I learned this lesson the hard way.
I collected data for a science education CMO (we’ll call them “Alex”) who needed evidence to support or refute their gut instincts. The company had big decisions to make because they were considering a new vertical market. Making the wrong decision could be expensive for the bottom line and company morale. Revenue was flat for the prior three years, and profits were sinking into single-digit percentages as product costs increased.
Alex decided to test their assumptions on market and product trends before suggesting any big moves to their product team.
We talked to a LOT of educators—dozens, and dozens. Focus groups and in-depth interviews yielded strategic insights as we built a picture of the market landscape.
The Board spoke. They wanted numbers, so we shifted our focus from qualitative research (Qual) to quantitative data (Quant). Educator stories (Qual) painted part of the landscape, but we needed some hard numbers, too.
Good data makes my eyes twinkle, so I was excited to embark on this research journey for the Board and my client with a survey designed to elicit meaningful responses and eliminate bias. The survey we created was divinely simple, and as the data rolled in, we watched an accurate picture of the market landscape come into focus—until it went sideways.
Our data analyst, Michelle, texted me. “I think there is something wrong. I think we have a bot with a sense of humor.”
Now, there is nothing laughable about funny data. My stomach turned. Our first open response read like a word salad, and dozens of responses were recorded at precisely the same hour, minute, and second.
We learned a LOT that day, and this story ends with reliable data, and tips for those of you who want to ensure your market research data is reliable.
What is a Survey Bot?
A survey bot is a type of automated program specifically designed to fill out surveys on behalf of a human user. These bots can be malicious in nature, aiming to either gain monetary incentives, skew survey results, or simply harass.
Bots are people too. A person, often with malicious intent, writes a program for personal gain to take surveys. It is important to remember this as you set out to protect your survey because understanding motive helps you prevent a bot attack.
Some common motivations include:
– Earning commissions for survey fills.
– Claiming survey rewards like gift cards.
– Boosting chances to win in a survey contest.
– Sabotaging competitors by inserting misleading information.
– Enjoying harassment and pranks. 🤷
Bots are the bane of business. Our real-world research shows that a high percentage of fraudulent responses occur when the survey offers monetary incentives, like Amazon gift cards, and is promoted on platforms with wide reach, such as social media.
Five Tactics to Prevent Bots from Destroying Your Data
Use Fraud Detection Tools
No matter which survey tool you choose, look for embedded fraud detection tools.
These tools
– Discard bot responses.
– Redirect bot responses for separate analysis.
– Flag suspicious responses for additional checks.
– Prevent email scanning software from submitting a response.
– Block multiple submissions from the same user.
Preventing multiple submissions is one of the most common bot behaviors. The “Prevent multiple submissions” feature detects duplicate entries by placing a cookie on the browser during the first survey session. Returning users on the same device/browser are flagged as duplicates. Then those responses identified as likely duplicates are tagged under the field with a field making extraction easier for your data analyst.
Field Your Survey Wisely
It’s tempting to promote your survey as loudly as possible, shouting its existence on every social media platform that attracts your ideal respondent. Initially, you may be thrilled with the intense response rate; however, social media is one of the riskiest ways to attract respondents.
Consider
Sending to your house list of opt-in respondents
Purchasing a reliable list of contacts from companies such as MCH Strategic Data.
Cleaning your lists with programs such as ZeroBounce.net, a service that isolates suspicious emails, invalid emails, and greylisted emails that will reject any email from a sender it does not recognize.
Use CAPTCHA
💪 Be sure to ask your research agency to include a CAPTCHA before you even field your survey. CAPTCHAs can be your secret weapon in the fight against bots in your education market research surveys!
💂♂️So, what exactly are CAPTCHAs? At Catapult X, we like to think of CAPTCHAs like a digital bouncer, making sure only real humans get into your survey party. 🎉 Their main job is to prevent all kinds of spam, like promo spam, registration spam, and even sneaky data scraping. Bots are way less likely to mess with your surveys if you’ve got a CAPTCHA standing guard.
🤖❓Many websites out there are already using CAPTCHAs effectively to keep those pesky bots at bay. The cool thing is, CAPTCHAs are designed to be a piece of cake for humans, while leaving most sneaky bots scratching their digital heads.
⚡The newer CAPTCHAs are even smarter—they keep an eye on how users behave online to make sure they’re human. It’s like a secret handshake for the internet! 🤝 A normal CAPTCHA test will only pop up if the user starts acting suspicious, like requesting pages or clicking links at lightning speed.
Include Smart Incentives
We are big fans of incentives, especially for educators who contribute substantial time and dedication to market research for educational technology. It’s important. It’s part of respecting teachers and educators as professionals.
Catapult X Recommends
Don’t include the incentive in your subject line, until the second or third reminder email.
Do include the incentive in the body of your email.
Do include the incentive on the intro page of your survey.
Do keep the incentive small, and reward as many educators as your budget allows.
Do use a lottery system for choosing winners. There are a variety of free services.
Check Response Patterns
🕵️ Analyze Quality of Answers: Look for too perfect or unnatural responses that lack human variation. At Catapult X, we scan for responses that are so vague they could apply to any questions. If there are several of these, we then look at response times.
⏱️ Response Times: Keep an eye on impossibly fast replies that could indicate automated bot activities. Taking a five minute survey in 6 seconds triggers our fraud alerts.
🌐 Check IP Addresses: Investigate duplicate IP addresses to identify multiple accounts operated by bots. Sort your responses by IP address and look for large blocks of duplicates. These can also be from educators at the same building or district, so be sure to look for other signs of fraud as well.
🗣️ Examine Language Patterns: Watch out for unusual language usage or repetitive phrasing, typical of bot-generated content. Catapult X typically asks an open-ended question at the beginning of a survey to engage educators, elicit deep feedback, and screen for bots that include non-sensical language or responses like “idk.”
Summary
Always choose a reliable market research partner with expertise in educational technology, science, STEM, and CTE. The best agencies know that
🤖❌ Survey bots can wreak havoc on your market research, compromising data reliability and your ability to make informed decisions.
📊Combining qualitative and quantitative research is crucial for painting a complete market picture. Educator stories and hard numbers go hand-in-hand.
🎯Ensuring data accuracy and integrity is non-negotiable. Your strategic decisions depend on it.
📚 As seen in the real-life example, survey bots can disrupt the process. Learn from these lessons to safeguard your research.
🔒Implement the five practical ways to protect your surveys from bots. Secure reliable data and unlock the power of informed decision-making. 🔒🔑
Download a checklist for ensuring quality market research data from Catapult X.
