Survey Bots are like a stomach virus. Something doesn’t feel right, and the next thing you know, you have a big problem. And it stinks.
I learned this lesson the hard way.
I collected data for a science education CMO (we’ll call them “Alex”) who needed evidence to support or refute their gut instincts. The company had big decisions to make because they were considering a new vertical market. Making the wrong decision could be expensive for the bottom line and company morale. Revenue was flat for the prior three years, and profits were sinking into single-digit percentages as product costs increased.
Alex decided to test their assumptions on market and product trends before suggesting any big moves to their product team.
We talked to a LOT of educators—dozens, and dozens. Focus groups and in-depth interviews yielded strategic insights as we built a picture of the market landscape.
The Board spoke. They wanted numbers, so we shifted our focus from qualitative research (Qual) to quantitative data (Quant). Educator stories (Qual) painted part of the landscape, but we needed some hard numbers, too.
Good data makes my eyes twinkle, so I was excited to embark on this research journey for the Board and my client with a survey designed to elicit meaningful responses and eliminate bias. The survey we created was divinely simple, and as the data rolled in, we watched an accurate picture of the market landscape come into focus—until it went sideways.
Our data analyst, Michelle, texted me. “I think there is something wrong. I think we have a bot with a sense of humor.”
Now, there is nothing laughable about funny data. My stomach turned. Our first open response read like a word salad, and dozens of responses were recorded at precisely the same hour, minute, and second.
We learned a LOT that day, and this story ends with reliable data, and tips for those of you who want to ensure your market research data is reliable.
A survey bot is a type of automated program specifically designed to fill out surveys on behalf of a human user. These bots can be malicious in nature, aiming to either gain monetary incentives, skew survey results, or simply harass.
Bots are people too. A person, often with malicious intent, writes a program for personal gain to take surveys. It is important to remember this as you set out to protect your survey because understanding motive helps you prevent a bot attack.
Some common motivations include:
– Earning commissions for survey fills.
– Claiming survey rewards like gift cards.
– Boosting chances to win in a survey contest.
– Sabotaging competitors by inserting misleading information.
– Enjoying harassment and pranks. 🤷
Bots are the bane of business. Our real-world research shows that a high percentage of fraudulent responses occur when the survey offers monetary incentives, like Amazon gift cards, and is promoted on platforms with wide reach, such as social media.
No matter which survey tool you choose, look for embedded fraud detection tools.
These tools
– Discard bot responses.
– Redirect bot responses for separate analysis.
– Flag suspicious responses for additional checks.
– Prevent email scanning software from submitting a response.
– Block multiple submissions from the same user.
Preventing multiple submissions is one of the most common bot behaviors. The “Prevent multiple submissions” feature detects duplicate entries by placing a cookie on the browser during the first survey session. Returning users on the same device/browser are flagged as duplicates. Then those responses identified as likely duplicates are tagged under the field with a field making extraction easier for your data analyst.
It’s tempting to promote your survey as loudly as possible, shouting its existence on every social media platform that attracts your ideal respondent. Initially, you may be thrilled with the intense response rate; however, social media is one of the riskiest ways to attract respondents.
Consider
Sending to your house list of opt-in respondents
Purchasing a reliable list of contacts from companies such as MCH Strategic Data.
Cleaning your lists with programs such as ZeroBounce.net, a service that isolates suspicious emails, invalid emails, and greylisted emails that will reject any email from a sender it does not recognize.
💪 Be sure to ask your research agency to include a CAPTCHA before you even field your survey. CAPTCHAs can be your secret weapon in the fight against bots in your education market research surveys!
💂♂️So, what exactly are CAPTCHAs? At Catapult X, we like to think of CAPTCHAs like a digital bouncer, making sure only real humans get into your survey party. 🎉 Their main job is to prevent all kinds of spam, like promo spam, registration spam, and even sneaky data scraping. Bots are way less likely to mess with your surveys if you’ve got a CAPTCHA standing guard.
🤖❓Many websites out there are already using CAPTCHAs effectively to keep those pesky bots at bay. The cool thing is, CAPTCHAs are designed to be a piece of cake for humans, while leaving most sneaky bots scratching their digital heads.
⚡The newer CAPTCHAs are even smarter—they keep an eye on how users behave online to make sure they’re human. It’s like a secret handshake for the internet! 🤝 A normal CAPTCHA test will only pop up if the user starts acting suspicious, like requesting pages or clicking links at lightning speed.
We are big fans of incentives, especially for educators who contribute substantial time and dedication to market research for educational technology. It’s important. It’s part of respecting teachers and educators as professionals.
Catapult X Recommends
Don’t include the incentive in your subject line, until the second or third reminder email.
Do include the incentive in the body of your email.
Do include the incentive on the intro page of your survey.
Do keep the incentive small, and reward as many educators as your budget allows.
Do use a lottery system for choosing winners. There are a variety of free services.
🕵️ Analyze Quality of Answers: Look for too perfect or unnatural responses that lack human variation. At Catapult X, we scan for responses that are so vague they could apply to any questions. If there are several of these, we then look at response times.
⏱️ Response Times: Keep an eye on impossibly fast replies that could indicate automated bot activities. Taking a five minute survey in 6 seconds triggers our fraud alerts.
🌐 Check IP Addresses: Investigate duplicate IP addresses to identify multiple accounts operated by bots. Sort your responses by IP address and look for large blocks of duplicates. These can also be from educators at the same building or district, so be sure to look for other signs of fraud as well.
🗣️ Examine Language Patterns: Watch out for unusual language usage or repetitive phrasing, typical of bot-generated content. Catapult X typically asks an open-ended question at the beginning of a survey to engage educators, elicit deep feedback, and screen for bots that include non-sensical language or responses like “idk.”
Always choose a reliable market research partner with expertise in educational technology, science, STEM, and CTE. The best agencies know that
🤖❌ Survey bots can wreak havoc on your market research, compromising data reliability and your ability to make informed decisions.
📊Combining qualitative and quantitative research is crucial for painting a complete market picture. Educator stories and hard numbers go hand-in-hand.
🎯Ensuring data accuracy and integrity is non-negotiable. Your strategic decisions depend on it.
📚 As seen in the real-life example, survey bots can disrupt the process. Learn from these lessons to safeguard your research.
🔒Implement the five practical ways to protect your surveys from bots. Secure reliable data and unlock the power of informed decision-making. 🔒🔑
Download a checklist for ensuring quality market research data from Catapult X.
Educational Leader. Public Speaker. STEAM Advocate.
Dr. Linda Johnson-McClinton is a teacher-preneur and founder of LJM Educational Services, LLC, a consulting and EdTech influencing company. An educator for the last 27 years, she’s been a passionate advocate for STEMM and founded Enriching Explorations in Engineering to promote STEMM education programs.
Dr. Johnson-McClinton first fell in love with science in elementary school and, as a student, became a fixture of local and city-level science fairs. Now a grandmother, she uses her platform to become a change agent for a better world. Dr. Johnson-McClinton devotes time to volunteering at her local school district and is an active Zeta Phi Beta Sorority member.
Top National Strategist. Coalition Builder. Global STEM Leader.
Rita Ferrandino is a leading expert on STEM education, US education policy and politics, and Future of Work strategies. Educational policy has been at the center of her political activism, and for 30 years, she worked to steer opportunities for educators, students, and communities through innovation and social change. Ferrandino co-founded ARC Capital Development. She also consults with the University of Pennsylvania’s Graduate School of Education.
A naturally curious child, Ferrandino’s passion for science began early in school. But in high school, a chemistry teacher told her girls shouldn’t be scientists because they’d be taking jobs away from men. Ferrandino made it her mission to prove that teacher wrong and became a lifelong supporter of STEM education. Ferrandino is also a health enthusiast – she loves meeting clients for walks and often takes Zoom meetings from her fit bike.
Computer Science Entrepreneur. Technology Pioneer. Growth Strategist.
Nikki Navta is a pioneer in using multimedia and video games as educational tools. Through her EdTech company Zulama, she collaborated with Carnegie Mellon University faculty members to develop programs that teach computer science to middle and high school students through video game design.
Science was a natural part of Navta’s upbringing, as her father taught high school science. Her love for horses made her study veterinary medicine in college. Later, she realized that her true passion lies in mathematics and coding, so she switched to a multi-field major in mathematics and fine arts.
Content Marketing Master. Word Crafter. Author.
Annie Galvin Teich is the co-founder and CEO of The Teich Group, a company that creates marketing content in the EdTech industry. For over 25 years, she has worked with numerous EdTech publishers to discover the best words to promote and support their learning products.
Teich’s interest in science sparked when her father, a Navy pilot, taught her the major constellations under a starry sky. Since then, she’s been interested in people who developed “sky science,” like Copernicus, Galileo, and Newton. She’s also a self-proclaimed fangirl of Carl Sagan, Neil deGrasse Tyson, and Michio Kaku.
Creator. Writer. Project Manager.
Jessica Dunnivan contributes to Catapult X’s graphic design for social media, and Dunnivan contributes to writing and editing for clients. Her work with Catapult X has given her a deeper appreciation for STEM. It inspires her to explore research and development for marketing in today’s fast-paced and authenticity-driven environment.
Dunnivan is an equestrian with a deep interest in horses’ biology and social interactions. She dreams of having a self-sufficient farm where she can grow her food and raise horses and other livestock. Dunnivan is also an avid painter and fantasy writer.
Editor Extraordinaire. Writer. Citizen Scientist.
Ian Broderick is a member of Catapult X’s content creation team. As a freelance writer, he has a decade of experience in writing, editing, and copywriting.
One of Broderick’s earliest memories is of waking up one morning to find his backyard blanketed in 17-year cicadas. The discovery that the cicadas had been living under the ground he walked for longer than he’d been alive at the time fostered in him a deep fascination with biology. This fascination expanded to include a significant interest in botany and mycology.
Growth Marketer. Entrepreneur. Research Geek.
Daylene Long founded Catapult X to empower EdTech business leaders to create brands and products fueled by educators’ needs so they can develop tomorrow’s generation of scientists and engineers. Catapult X is the only ecosystem of brand experts, creatives, and business analysts who collaborate to build products and infuse STEM brands with the latest market research.
When she’s not curating market insights, Long nurtures her life-long curiosity in life sciences by reading, volunteering, and uncovering the hidden histories of female scientists. She relaxes through swimming and playing with her two Golden Retrievers. Most recently, a well-meaning friend introduced Long to hot yoga, and she’s been hooked ever since. Long moms a young computer scientist and finds joy in laughing with family and friends.