A Guide to Online Panel Surveys and Annotation on Qualtrics, Prolific, and Other Platforms

Students often ask me how to find survey respondents beyond their own networks, and the same few questions come up each time: which platform, how much it costs, and whether a reviewer will accept the sample. I am writing the answers down once here. I am not a survey methodologist, so please treat this as a starting point for your own reading and take the parts that apply to your study.

Online panels and self-distributed surveys

Most of us first run a survey by building it in Qualtrics or Google Forms and sending the link out ourselves, to a listserv, a class, a community organization, or a social media post. This costs little, and it is the right choice when the people you need already belong to a group you can contact. Its limits come from doing all the recruiting yourself. Nobody knows who saw the link, so there is no response rate, the sample is whoever chose to answer, and an open link with a payment attached draws fraud. When Pozzar and colleagues (2020) posted a survey with a $15 payment on Twitter and Facebook, they received 271 completed surveys within seven hours and classified 94.5% of them as fraudulent.

An online panel is a standing pool of people who have agreed in advance to take surveys; the AAPOR task force report on online panels uses the ISO definition, “a sample database of potential respondents” who agree to future data collection. The panel company recruits members, asks them profile questions when they join (age, gender, race, education, location, and often health or occupation), invites the members who match a study, pays their incentives, and removes members who fail quality checks. The AAPOR report lists these as five activities: recruitment, joining and profiling, study sampling, incentive programs, and panel maintenance. You write the survey, set the eligibility criteria and quotas, and pay per completed response. In return you receive respondents you could not have found yourself, usually within days, from a vendor that has already removed some bad actors, though not all of them (see the data quality section below).

Probability and opt-in panels

Every option below falls into one of two groups, and the difference decides what you can claim from your data.

A probability-based panel recruits households at random, usually from a list of postal addresses, and contacts them by mail, phone, or in person until they join. Each household has a known chance of selection, and that known chance is the basis for population estimates and margins of error. NORC’s AmeriSpeak, Ipsos KnowledgePanel, the SSRS Opinion Panel, the Gallup Panel, and Pew’s American Trends Panel work this way. Pew’s panel is not for hire, but its methods pages are the best free documentation of panel construction.

An opt-in panel is made up of people who signed up, usually through ads or referral sites. Prolific, CloudResearch Connect, Qualtrics panels (which resell sample from partner vendors such as Dynata and Cint), and YouGov are in this group. YouGov selects respondents from its opt-in panel to match a target population, which helps, but the starting pool is still self-selected.

Representativeness

The comparisons between the two groups point the same way across fifteen years. Yeager and colleagues (2011) found that probability samples “were consistently more accurate than the non-probability sample surveys, even after post-stratification with demographics,” and MacInnis and colleagues (2018) replicated the result with 40 benchmark variables. The clearest recent numbers come from Pew’s 2023 comparison of three probability panels and three opt-in samples fielded with the same questionnaire. Error on the opt-in samples was about twice as large overall, and much larger for young adults and Hispanic adults.

Pew attributed much of the subgroup error to bogus respondents who answer “Yes” to nearly everything: in the opt-in samples, 15% of 18 to 29 year olds and 19% of Hispanic adults said yes to at least 10 of 16 yes/no questions, against 1% to 2% on the probability panels. For social work research this is the uncomfortable part. The groups we most often oversample, young adults and Latino respondents, are the groups where opt-in estimates drift furthest, and Pew’s 2024 follow-up suggests some bogus respondents claim those identities to pass screeners.

Weighting does not fix much of it. Pew’s 2018 weighting study found that even the best adjustment removed about 30% of the bias in opt-in samples, and that “choosing the right variables for weighting is more important than choosing the right statistical method.” Its 2016 report adds that the two least accurate samples had some of the best unweighted demographic profiles, so a sample table that matches the Census tells you little about whether the answers are accurate.

Language coverage is a separate problem, and it applies to probability panels too. AmeriSpeak recruits and interviews in English and Spanish, as does KnowledgePanel through its Latino panel. As Pew put it in a 2020 note on Asian American estimates, those estimates “are based just on English (or Spanish) speakers,” and people who speak another language and are not proficient in English or Spanish generally cannot participate. For research on immigrants with limited English, no panel I know of covers the population, and I would state that coverage gap in the limitations section.

Experiments and prevalence estimates

None of this means an opt-in sample is useless. The evidence separates two kinds of question. For prevalence, such as the share of adults with unmet mental health need, the 2010 AAPOR task force wrote that researchers “should avoid nonprobability online panels when one of the research objectives is to accurately estimate population values.” For randomized survey experiments the record is better. Coppock, Leeper, and Mullinix (2018) replicated 27 experiments originally run on national probability samples using online convenience samples and found “very high correspondence,” which they attribute to low variation in treatment effects across people. That condition is an assumption, and the 2013 AAPOR report‘s answer to whether effects generalize is “It depends.”

So I would match the platform to the claim. A vignette experiment, a measurement study, or a pilot of a new scale can go on Prolific or Connect. A prevalence estimate you intend to generalize to U.S. adults needs a probability panel or an existing federal survey.

GSS comparisons and their limits

A common move with an opt-in sample is to place its estimates beside the General Social Survey (GSS) and argue that they look similar. NORC at the University of Chicago conducts the GSS, and the files are free from NORC or through the GSS Data Explorer. ICPSR at Michigan hosts cumulative files only through 2016, so for recent years I would download from NORC directly.

I would use the comparison with care, for four reasons.

• The GSS changed mode. The 2021 survey moved to a design that mailed invitations to a web survey with an AAPOR response rate of 17.4%, and the 2022 survey mixed in-person interviews, web, and phone. NORC’s mode sensitivity note reports lower confidence in institutions and much lower religious activity among web respondents, and NORC asks authors to note that recent changes in trends may reflect the methodological changes.
• Demographics are the wrong benchmark. If you weighted to age, sex, race, and education, matching the GSS on those variables only shows that the weights worked. Yeager and Pew use benchmarks that were not used in weighting, such as smoking, holding a driver’s license, or voting, and that is the comparison that tests accuracy.
• Subgroups are small. Recent single-year files have about 4,000 cases, and the AmeriSpeak oversamples of Black, Hispanic, and Asian respondents added in 2022 and 2024 are excluded from the cumulative file.
• Language coverage is limited. The 2021 GSS was offered in English and Spanish only.

I have not found a published study that benchmarks a Prolific sample against the GSS, but two studies compare Qualtrics panels with it. Boas, Christenson, and Glick (2018) recruited at least 1,000 U.S. respondents each through Facebook, MTurk, and Qualtrics in late 2015 and compared them mainly with the 2014 GSS. They concluded that “Qualtrics is most demographically and politically representative,” with the caveat that on age, education, income, race, and ethnicity the Qualtrics sample “was biased in the same direction as MTurk, but less severely so.” Zack, Kennedy, and Long (2019) compared an MTurk sample and a Qualtrics panel with the GSS and found that “Qualtrics outperforms MTurk in both univariate and multivariate analysis,” while reporting “a risk of false positives” with both. An older comparison, Simmons and Bobo (2015), found small demographic differences but more politically interested and more polarized respondents in the web sample.

Minority and hidden populations

Opt-in pools do not look like the U.S. population, and they do not look like each other. In Peer and colleagues (2022), with about 500 U.S. respondents per platform in September 2020, women were 33.5% of the MTurk sample, 52.7% of Prolific, 47.2% of Qualtrics, and 60.1% of Dynata, and in the Qualtrics sample “half of the sample were 64 years old or above.” In Douglas, Ewell, and Brauer (2023), women were 67.5% of the Prolific sample and 66.6% of the Qualtrics sample, whose mean age was 64.3 and in which 95.0% selected White. Older MTurk work found a younger, more liberal, and less religious pool; Levay, Freese, and Druckman (2016) describe their MTurk sample as “much less racially and ethnically diverse than the ANES sample.” Qualtrics says the same of its own panels: they “tend to have increased representation of 35-54 year old females, and lower representation of harder to reach demographics like 18-24 Hispanic males.” The makeup of any one sample depends on the platform and the month, so I would report it in a table next to a national benchmark instead of assuming it.

Each platform lets you choose who sees the study, in different ways.

• Prolific has “several hundred” filters, including ethnicity, country of birth, first language, fluent languages, gender identity, and sexual orientation. “Most filter answers are self-reported by participants,” so a filter is only as good as the answer someone gave when they joined. Its U.S. representative samples are stratified by age, sex, and “Ethnicity (simplified)” with five groups (White, Mixed, Asian, Black, Other), so Hispanic origin is not one of the strata, and they start at 300 participants. Prolific adds that “Very small demographic groups can be difficult to represent exactly.”
• CloudResearch Connect lets you target “age, race/ethnicity, gender, sexual orientation, political views, etc.” For smaller groups, its Prime Panels service pools several vendors, and CloudResearch notes that for a very small target group “a study may be feasible only when combining the resources of multiple platforms.”
• Qualtrics panels use quotas. When you build a panel, you “Click Add Quotas to set limits on each demographic,” set a percentage for each group, and the page shows the price and whether the panel is feasible. The targeting criteria “varies by country,” so confirm that race and ethnicity are offered for your country before you plan around them. The price goes up as the incidence rate falls and as the targeting criteria become more specific. Inside any Qualtrics survey you can also set quotas on your own screener answers, which closes a group once it is full.

Screening for a minority group invites a specific kind of fraud. In Pew’s 2024 analysis, 24% of opt-in respondents who said they were Hispanic also said they were licensed to operate a nuclear submarine, against 2% of non-Hispanic respondents, and Pew suggests bogus respondents may identify this way “in order to bypass screening questions.” I would rely on the platform’s stored profile filters where possible, avoid a screener that tells people which answer qualifies, and ask the identity question again inside the survey as a consistency check.

When the goal is an estimate for the group itself, probability panels with oversamples are the stronger choice. NORC’s AmeriSpeak overview lists “Amplify AAPI (Asians and Pacific Islanders), AmeriSpeak Latino (Spanish-language-dominant Hispanics),” and a young adult panel “with oversamples of African Americans, Hispanics, and Asians.” Pew’s American Trends Panel oversampled Hispanic, Black, and Asian adults in 2019, 2022, and 2023. For Asian Americans the interview language matters as much as the sample size. Pew’s 2022 to 2023 survey of Asian Americans was a separate address-based study “offered in six languages: Chinese (Simplified and Traditional), English, Hindi, Korean, Tagalog and Vietnamese,” and the California Health Interview Survey.aspx) is conducted in “Spanish, Chinese (Mandarin and Cantonese dialects), Korean, Vietnamese, and Tagalog” as well as English, which makes it one of the few public data sets where Asian Californians with limited English answer in their own language.

Some groups are missing from every online panel and from most household probability panels. Opt-in platforms require internet access, and Pew handles this in the ATP by giving households without internet “a data plan and tablets.” Address-based frames sample housing units, and the Census Bureau “classifies all people not living in housing units” as living in group quarters, which include shelters, correctional facilities, and nursing homes. People sleeping outside fall outside both frames. A review by Bonevski and colleagues (2014) lists “homeless people living on the streets” and people with low literacy among the groups that are hardest to recruit, describes snowball and respondent-driven sampling (Heckathorn, 1997), and advises researchers to “plan for higher resourcing costs and operate via community partnerships.” For groups that do use social media, paid advertising is another route. Pötzschke and Braun (2017) recruited 1,103 Polish migrants in four European countries through Facebook ads for a total of €500, and Guillory and colleagues (2018) recruited LGBT young adults on Facebook and Instagram and found that “Social media methods have more data quality issues but are faster and less expensive than intercept.”

So the same rule from the previous sections applies. An experiment or a measurement study within a group can use an opt-in platform with filters or quotas, reported in full in the methods. An estimate about the group needs a probability panel with an oversample, a survey in the group’s languages, or a recruitment design built with community partners.

Vendors and prices

The prices below come from each vendor’s own page or a university page, checked in October

1. Most probability panels quote custom studies only, so for those I list the omnibus

price, where you buy a few questions on a shared survey.

Option	Sample type	Published price
Prolific	Opt-in, about 300,000 active participants; U.S. and U.K. representative samples	Participant pay plus a 33.3% fee for academic accounts (42.8% corporate); minimum pay $8 per hour, recommended $12
CloudResearch Connect	Opt-in	Participant pay plus a 25% fee for academic accounts (40% otherwise)
Qualtrics panels	Opt-in, resold from partner panels	Quote based; online purchase up to $15,000. BYU lists $5 per complete with a $500 minimum
YouGov omnibus	Opt-in with sample matching	$1,500 setup plus $600 per question for 1,000 adults
NORC AmeriSpeak omnibus	Probability	$1,000 per question for the first five, three-question minimum, 1,000 adults
Ipsos KnowledgePanel	Probability	Quote only
U-M M-Panel	Probability, with a Michigan oversample	Shared-cost omnibus, three-question minimum; price on request

The calculator is the fastest way to budget an opt-in study. Remember that the fee is charged on top of what participants receive, so an academic study costs 1.33 times the participant pay before tax. Older university pages still quote a 30% fee and a $6.50 minimum, and both are out of date.

I have used Qualtrics panels several times in my own studies, and the company now also markets “research-grade synthetic respondents” alongside human panels. If you buy sample through Qualtrics, ask in writing whether any responses are synthetic. The U-M Qualtrics license covers survey building only, so panel sample is a separate purchase.

For a probability sample at Michigan, the newest option is ISR’s M-Panel, a “nationally representative online probability panel of U.S. adults, with an oversample of the state of Michigan.” Recruitment began in summer 2026 with a target of 6,750 members, researchers buy a minimum of three questions on a shared omnibus of about 3,000 respondents, and inquiries go to mpanel-inquiry@umich.edu. For state-level questions in Michigan this seems like the first place to ask.

Herman and colleagues (2024) fielded the same health survey on MTurk and on KnowledgePanel: the KnowledgePanel study cost $236,178 against $27,146 for MTurk, about nine times as much, though it took far less staff time.

MTurk and TESS in 2026

Amazon Mechanical Turk permanently closed on September 30, 2026. Requesters can still approve work and pay bonuses until October 30, and the closure FAQ says the shutdown also ends the MTurk worker option inside SageMaker Ground Truth and Amazon Augmented AI. CloudResearch retired its MTurk Toolkit on the same day. If a methods section, a lab manual, or an IRB template you inherited still says MTurk, it needs rewriting.

A second change matters for anyone who planned to apply for free fielding. Time-sharing Experiments for the Social Sciences (TESS), which ran accepted experiments at no cost on NORC’s probability panel, now says on its home page that it “has suspended operations and is not accepting new proposals.” The page gives no date or reason, and some NORC pages still describe TESS as open, so I would check the TESS site itself before writing it into a proposal.

Data quality and fraud

Among opt-in platforms, the independent comparisons favor Prolific and Connect. Douglas, Ewell, and Brauer (2023) ran the same study on five platforms and counted a response as high quality only if it passed attention checks, came from a unique ID and location, and gave a relevant open-ended answer. Peer and colleagues (2022) found the same ranking, though three of its authors work for Prolific.

The newer threat is software that answers surveys. Sean Westwood’s 2025 PNAS study built an AI agent that completed surveys with coherent answers that evaded nearly every detection method he tested. How often this happens in real studies is not yet known. Open recruitment through social media is worse still: in Pozzar and colleagues (2020), 256 of 271 responses collected within seven hours were fraudulent.

The checks I would build in before launch:

• Qualtrics fraud tools. Turn on bot detection, duplicate detection, and prevent multiple submissions before fielding; the Qualtrics fraud detection page explains each field. RelevantID was retired in 2025, so older guides that mention it are out of date.
• More than one attention check. Prolific allows rejection only when a participant fails at least two checks in a study of five minutes or longer, and Kennedy and colleagues found that attention checks alone missed much of the fraud on MTurk.
• Open-ended items tied to lived experience. Prolific’s own guidance on detecting LLM use recommends questions based on the participant’s experience and the study’s context.
• Exclusion rules written in advance. Decide and preregister what counts as a failed or fraudulent response before you see the data.

U-M’s IRB-HSBS has a tip sheet on fraudulent online responses, written mostly for open-link surveys with incentives. Its suggestion to collect mailing addresses conflicts with Prolific’s rule against collecting identifying information, so on Prolific I would follow the platform’s rule and describe that choice in the IRB application.

Annotation platforms

MTurk was also the default place to hire people to label text, and its closure leaves NLP projects in social work with the same question. The worker pools that remain for academic budgets are the same two platforms: Prolific runs annotation studies through its AI Task Builder under the same fee structure, and Connect accepts non-survey labeling projects. Commercial labeling vendors such as Scale, Surge, Toloka, and Appen do not publish prices and are built for enterprise contracts.

The annotation interface is a separate decision from the worker pool. Free, open-source tools include Label Studio, Doccano, INCEpTION, which computes inter-annotator agreement inside the tool, and Potato, built in David Jurgens’s lab at Michigan, which connects directly to Prolific and Connect. Prodigy is a paid alternative at $390 for a personal license.

For many projects the realistic plan now combines a large language model with a smaller human-coded set. Gilardi and colleagues (2023) found ChatGPT outperformed crowd workers on four of five annotation tasks, and Veselovsky and colleagues (2023) estimated that 33% to 46% of MTurk workers were already using language models on a text task. The cautions are just as strong: Baumann and colleagues (2025) found that LLM annotations produced incorrect conclusions in about 31% of the hypotheses they re-tested, and Egami and colleagues show that using model labels directly in a regression gives invalid confidence intervals even at 80% to 90% accuracy. I recommend hand-coding a gold-standard subset with trained coders, reporting agreement, and using that subset to correct the model labels. Student annotators at Michigan are paid at least the $15 hourly minimum for temporary staff.

If the text involves suicide, abuse, or other distressing content, tell annotators in specific terms what they will read; Prolific’s guidance on sensitive content says a general warning is not enough. Whether annotators count as human subjects depends on whether you collect information about them, and U-M’s human subjects FAQ does not address annotation directly, so I would ask the IRB.

Reporting

AAPOR’s guidance on the margin of sampling error says it “does not apply to opt-in online surveys and other non-probability based polls,” so I would not report one for a Prolific or Qualtrics sample. For the methods section, AAPOR’s disclosure standards list the items to report, including the recruitment method, the weighting, and the data quality procedures, and health journals often expect the CHERRIES checklist for web surveys. Naming the platform, the dates, the pay rate, the screening criteria, and every exclusion rule, with the number of cases each one removed, answers most of what a reviewer will ask.