AI for Survey Analysis: Coding Open-Ended Responses

Every guide about AI and survey data stops at the same sentence: paste the open-ended responses into ChatGPT and ask for the themes.

That is not analysis. It is a plausible-looking list, and you cannot defend a list. A researcher, a UX lead or a people-analytics analyst has to stand in a readout and say "31% of respondents raised this" — then answer the question that follows: how do you know?

This is a procedure you can answer that question with. Its fourth step is the whole point: you hand-code a random sample yourself and measure how far the model diverges from you.

The Challenge

Closed questions analyze themselves. A Likert scale is already a number. The free-text box is where the information you did not anticipate lives — and it is the part of the survey that quietly gets dropped from the readout.

The reason is arithmetic. Done properly, thematic coding means reading a sample, building a codebook, applying every code to every response, checking a second coder would have done the same, and only then counting. On 3,000 responses that is days of skilled work. Most teams do not have days, so the free text becomes a slide with six quotes on it — a slide that proves whatever the person who picked the quotes already believed.

The naive AI shortcut fails for a diagnosable reason: it answers a different question than the one you asked. Ask for "the main themes" and the model summarizes, and summarization gives you prose. A readout needs a column — every response assigned to categories, so they can be counted, cross-tabbed against tenure or NPS, and compared to last wave. Three things then break, all silently: the themes drift, because nothing holds the model to a fixed set of categories; the counts are unfalsifiable, because nothing says how wrong that 31% might be; and everything fits, because a model given a category list and no way to decline assigns something to every response — so the ones that belonged nowhere vanish into the nearest theme.

How AI Solves It

The reframe that makes everything else work:

This is a classification problem, not a summarization problem. You are not asking what the responses are about. You are asking, for each response, which of these specific, pre-agreed categories apply — and allowing the model to say none of them.

The evidence that this works — when the human review and the measurement below actually happen — is specific. Dunivin's study in EPJ Data Science coded dense, paragraph-long passages against nine socio-historical codes, comparing GPT-4 to a human-derived gold standard: substantial reliability (Cohen's κ at or above 0.6) on 8 of the 9 codes, and excellent reliability (κ at or above 0.79) on 3 of them. GPT-3.5, on the same codebook, collapsed — mean κ of 0.34. The paper also found fidelity improved considerably when the model was asked to justify each decision; that goes into the prompt below. Separately, Gilardi, Alizadeh and Kubli in PNAS (2023) annotated 6,183 tweets and news articles and found ChatGPT's zero-shot accuracy beat crowd workers by about 25 percentage points, with higher intercoder agreement than crowd workers and trained annotators.

Read that honestly. A frontier model applying a well-built codebook can reach the reliability you would demand of a trained human coder — in studies where somebody measured it. The same paper shows a weaker model, same codebook, at κ = 0.34, which is worthless.

Note the trap in reading those two results together. Gilardi's headline win was produced by GPT-3.5 — the same model tier that collapses in Dunivin. What changed is not the model but the task: Gilardi coded short tweets into a handful of flat categories; Dunivin coded dense, paragraph-long passages against nine interpretive socio-historical codes. So the lesson is not "big model good, small model bad" — it is that your codebook's difficulty, not the model's tier, decides whether this works. Which is why the cheapest model that passes your reliability check is the right one, and why you cannot know which that is in advance. Your model, on your data, is unmeasured until you measure it.

The model does not decide whether the coding is good. The kappa does.

Recommended Tools & Models

Two jobs. Building the codebook is interpretive — use the best model you have, in a chat interface, iteratively. Applying the codebook is rote — use the cheapest model that passes your reliability check, via an API, in batches.

Use a frontier model for the codebook — Claude Sonnet 5, GPT-5.5 or Gemini 3.5. Use a small, cheap one for classification: Claude Haiku 4.5 is the right call at $1 per million input tokens and $5 per million output, halving again through the Batch API (DeepSeek V4, Qwen3.7-Max and GLM-5.2 also work). Do not pick the cheap model on faith — run your reliability check against it, and pay for the bigger one if its kappa is materially worse.

Qualtrics, Thematic, NVivo and ATLAS.ti all ship AI coding features. What you buy from them is a contract that lets you put identifiable data in, an audit trail, and a familiar interface — not a better model.

Step-by-Step Implementation

1. Read a sample. Yourself. First.

Pull 30-50 responses at random and read them — not to code them, but to know what is in there. Twenty minutes of reading is what lets you see that a model-generated theme called "Communication" is hiding three different complaints. Note the responses that would be hard to code; those are your test cases.

2. Generate the codebook — inductive or deductive

Inductive: the model proposes categories from the data. Use it when exploring, on a first wave, or when you do not know what people will say. The risk is drift toward the generic.

Deductive: you supply the frame — an existing model, last wave's codebook, the hypotheses stakeholders want tested. Use it to compare across waves or answer pre-registered questions. The risk is that you only find what you went looking for. Most real projects are hybrid: bring last wave's codebook, then ask what the data contains that it has no home for.

Read that last block first. The uncovered responses are where the finding you did not expect is hiding.

3. Review the codebook — before a single response is coded

This is where your bias enters the analysis, and the only place you can still get it out. Once coding starts, every flaw in the codebook becomes a flaw in 5,000 rows.

• The generic test. Would this code fit any employee survey ever written? "Work-Life Balance", "Leadership", "Career Development" fit everything, so they carry no information about this survey. Split them into what people actually said.
• The wastebasket test. Is one code going to absorb 60% of responses? That is not a theme, it is a shrug.
• The boundary test. Take the hard responses from step 1. Can you place them using only the written definitions? If not, neither can the model — nor the human you will compare it against.
• The framing test. Does a code encode a conclusion? "Resistance to the return-to-office policy" has already decided that disagreement is resistance. "Objections to" has not.

Rewrite the definitions in your own words. The codebook you apply should be yours, not the model's.

4. Classify against a fixed schema

Every response now goes to the model with the same codebook and a schema it genuinely cannot deviate from. Four non-negotiables: structured JSON output; multi-label, since a response can raise three things; a rationale and an evidence span, which improved agreement in Dunivin's study and gives you something auditable regardless; and an explicit escape category.

On "cannot deviate": asking for JSON in the prompt constrains nothing — the model can invent a code_id that is not in your codebook, and a silently invented code is data corruption that per-code kappa will never surface. Use the API's real structured-output mode and pass your codebook's code_id list as a JSON Schema enum. Then the constraint is enforced by the decoder rather than requested politely.

That last one is the most important line in the prompt. A model that cannot say "none of these" will not say it. It picks the nearest code and moves on, and nothing in the output shows that it did. Your data is then silently wrong — and biased, because force-fits pile up in whichever code has the loosest definition.

{
  "results": [
    {
      "response_id": "R0412",
      "codes": [
        {
          "code_id": "rto_commute_cost",
          "evidence": "three hours a day on the train and I pay for it myself",
          "reasoning": "Names the commute and its personal cost explicitly.",
          "confidence": "high"
        }
      ],
      "does_not_fit": false,
      "flag_for_human": false
    },
    {
      "response_id": "R0413",
      "codes": [],
      "does_not_fit": true,
      "flag_for_human": true,
      "note": "Reports a specific payroll error. No code covers pay administration."
    }
  ]
}

Pilot on 100 responses first, and read the does_not_fit and low-confidence cases. If does_not_fit is at 0%, the model is force-fitting and rule 2 is not landing. If it is above 30%, your codebook does not describe your data — go back to step 2.

5. The reliability check — the step that makes this a method

Everything above this line is what other guides tell you. This is what lets you defend the numbers.

1. Draw a random sample — and draw more than you want to. Random, not the interesting ones: sampling hard cases inflates disagreement, sampling easy ones hides it. The binding constraint is not the total but the positive instances per code: aim for at least 20 of them for any code you intend to report a number on. With a dozen codes that usually means 150–200 responses, not 50. At n = 50, a code appearing in 8% of responses gives you four positives, and its kappa is noise wearing a decimal point. For genuinely rare codes, oversample responses the model coded positive, reweight, and say so in your methods.
2. Code them by hand, using only the codebook, before you look at the model's output. See the model's answer first and you will anchor on it; your "independent" coding becomes a rubber stamp. Better still, have a colleague code the same sample blind — you will also learn how reliable humans are on your codebook, which is often a humbling number. (With three coders, Cohen's κ no longer applies: it is defined for exactly two. Use Krippendorff's α.)
3. Split the sample. Tune the confidence threshold on one half; report per-code agreement on the other. Tune and report on the same responses and you have fitted your threshold to your sample, and the number you publish is optimistic by an amount you cannot estimate.
4. Compare code by code. Coding is multi-label, so you compute one kappa per code, treating each as a binary present/absent decision.

import pandas as pd
from sklearn.metrics import cohen_kappa_score

human = pd.read_csv("human.csv").set_index("response_id").sort_index()
model = pd.read_csv("model.csv").set_index("response_id").sort_index()

assert list(human.index) == list(model.index)
assert list(human.columns) == list(model.columns)

for code in human.columns:
    h, m = human[code], model[code]
    # Both constant AND agreeing = genuinely never applied. Both constant and
    # DISagreeing (human never applies it, model applies it to everything) is
    # total disagreement, kappa 0.0 — do not let it print as "n/a".
    if h.nunique() < 2 and m.nunique() < 2 and h.iloc[0] == m.iloc[0]:
        print(f"{code:32s}  n/a — never applied by either coder")
        continue
    k = cohen_kappa_score(h, m)
    agreement = (h == m).mean()
    base_rate = h.mean()
    print(f"{code:32s}  kappa={k: .2f}  raw={agreement:.0%}  base rate={base_rate:.0%}")

Reading the number. The convention everyone cites is Landis and Koch (1977):

Be honest about what that is: a convention from a 1977 biometrics paper, not a law of nature. McHugh (2012) argues it is too lenient — calling 0.41 "moderate" implies it is acceptable, when a κ in that band means a large share of your data is simply wrong. Her stricter scale puts 0.60-0.79 at moderate and 0.80-0.90 at strong, and her conclusion is blunt: any kappa below 0.60 means little confidence should be placed in the results. So treat 0.6 as the floor, aim for 0.7+, and report the number rather than the adjective — readers cannot un-hear "substantial agreement."

Never read kappa alone. This is the failure that catches everyone, and it will catch you on the codes you are proudest of. Cohen's κ is deflated by skewed marginals — the kappa paradox (Feinstein and Cicchetti, 1990). Take a code with a 1% base rate, exactly the kind of specific, non-generic code the escape-hatch discipline above is designed to produce. One false positive and one false negative in 100 responses gives you 98% raw agreement and a kappa of roughly zero. The definition is fine. The statistic is telling you nothing except that the code is rare.

This is why the script prints raw and base rate next to kappa, and it is why the rule at the end of this guide is stated the way it is. For codes below roughly a 10% base rate, report Gwet's AC1 alongside κ — it is prevalence-robust, and irrCAC computes it. If a third coder joins, κ no longer applies at all: use Krippendorff's α or Fleiss' κ.

And the part people get backwards. When a code with a healthy base rate comes back at κ = 0.38, the instinct is to blame the responses ("people write ambiguously") or the model ("it isn't smart enough"). Almost always, the codebook is the problem — a fuzzy boundary produces disagreement between any two coders, including you and yourself on a different Tuesday. Low kappa measures your definitions, not the model's intelligence.

So diagnose it: read the responses you and the model disagreed on, side by side. You will find the definition contains two ideas, or the inclusion and exclusion rules overlap, or the code is too rare for kappa to be stable. Fix, re-pilot, re-measure. A code going from 0.38 to 0.74 after a definition edit is the normal shape of this work, and it belongs in your methods section.

6. Report what you did

Disclose the model and version; whether coding was inductive, deductive or hybrid; the codebook itself, in an appendix; the validation sample (size, how drawn, coded by whom, and whether the human saw the model's output first — the answer should be no); per-code agreement figures, including the bad ones; and the does_not_fit rate. A codebook with one code at κ = 0.52 and a note saying so is more credible than one with a suspiciously uniform 0.8. If you have an IRB, check whether sending participant text to a third-party API needs review — often it does.

Real-World Examples

These are composites of the failures that recur, not case studies of a named survey. The numbers illustrate the mechanism; they are not measurements.

Example 1: The force-fit that hid the actual finding

Before. An employee survey, 2,400 responses to "Anything else you'd like to tell us?". Twelve-code codebook, and a prompt that says "assign the most appropriate code." No escape category.

All 2,400 come back coded. The distribution looks clean. "Management & Leadership" is the largest theme at 22%, and that goes on the slide. What was actually inside it:

"I'd rather not say." "n/a" "The payroll system has taken the wrong tax code off me for four months and three people have told me it's someone else's job to fix." "My manager is great, honestly no complaints."

Two non-answers, one operational failure no code covered, and one positive comment — all absorbed into a theme they have nothing to do with, because the model was given twelve boxes and no way to refuse.

After. Same prompt, plus does_not_fit and the sentence "I would much rather see 12% of responses land in does_not_fit than see them forced into the nearest theme."

Now 9% land there — 216 responses. The analyst reads them and finds 41 are variations of the payroll complaint. That is the finding. It was never going to appear in a theme chart, because it did not have a theme. It appeared because the model was allowed to say I don't know where this goes.

What changed: one code and one sentence. Not the model, not the data.

Example 2: The codebook that fit everything and meant nothing

Before. A first codebook pass on a 3,000-response survey returns: Communication · Leadership · Work-Life Balance · Career Development · Compensation · Culture · Workload. There is no escape category, so by construction 100% of responses "fit" — which tells you nothing at all.

It looks like a triumph; it is the opposite. Every one of those codes would fit an employee survey at any company, in any industry, in any year — so the codebook has learned nothing from this survey. "31% of responses mention Communication" is a sentence you could have written before running it.

After. Re-run with the anti-generic constraint, and the model returns codes like "Decisions are announced after they are made, not while they are being made" and "The return-to-office mandate was justified by collaboration benefits nobody has seen."

Now 71% fit the codes and 29% land in does_not_fit. Lower coverage, far higher information. "38% of responses describe decisions being announced rather than discussed" changes what a leadership team does on Monday. "31% mention Communication" changes nothing.

Example 3: The sentiment label that was exactly backwards

Before. A product survey. The team adds a sentiment field — positive, neutral, negative — because it is one more line of JSON and seems free.

"Well, at least it didn't crash this time." → positive

"I love the product and I will be cancelling because your support team never replied to me." → neutral

"It's not that it's slow, exactly." → negative

Sarcasm about a history of crashes, read as praise. A mixed response averaged into a label that describes nobody — this person is not neutral, they are both enthusiastic and leaving. A hedged partial defence, read as a complaint.

This is not a prompt-tuning problem. A 2024 benchmark of sarcasm understanding across eleven LLMs found current models still underperform smaller, supervised, purpose-trained classifiers: the best averaged 65.0 F1 against a supervised RoBERTa baseline's 71.3.

After. The sentiment field is deleted. Where valence matters, the codebook carries it explicitly: manager_praised and manager_criticised become two codes, each with an inclusion rule, each measurable with a kappa. The mixed response above gets both product_valued and support_failure — which is the truth about it.

Best Practices

• Read a sample yourself before you touch a model. It is what lets you review a codebook rather than accept one.
• Treat it as classification, not summarization. Fixed schema, structured output, multi-label, countable.
• Always include a "does not fit" category, and tell the model you want it used.
• Kill generic themes. A code that fits any survey of its type carries no information about yours.
• Ask for reasoning and an evidence span. It improves agreement and gives you something auditable.
• Hand-code your validation sample blind. Seeing the model's output first turns validation into confirmation.
• Report kappa per code, never an average. A mean of 0.72 can conceal a code at 0.35.
• Freeze the codebook once coding starts — and if you change a definition between waves, re-code the previous wave or say plainly that the series breaks.
• Anonymize before the text leaves your machine. Watch small cells too: a theme reported for a six-person team is not anonymous however the data was processed.
• If you have 40 responses, read them. AI's value begins where a human cannot read everything.

Common Pitfalls

Force-fitting. The defining failure (Example 1): with no escape category, every response gets a code — including the ones that belong nowhere — and nothing in the output tells you.

Themes that fit everything. Models drift toward the generic because generic is safe, and a code that fits any survey of its type tells you nothing about yours (Example 2).

Sentiment. Sarcasm, negation, hedging and mixed feeling are what survey free-text is full of, and what sentiment models handle worst. A "positive" label on "well, at least it didn't crash this time" is the median case, not an edge case.

Skipping the reliability check. A number with no error bar and no provenance; the first person who asks "how do you know?" ends the meeting.

Validating against another model. That measures whether two models agree, which they mostly do, for the same reasons. It is an echo — as is hand-coding your sample after seeing the model's labels.

PII in free text. People type their own names, colleagues' names, salaries and medical circumstances into "any other comments" — most of all when you promised anonymity. See privacy. Assume it is in there.

Trusting the fluency. A coding decision arrives in the same confident tone whether right or wrong — the property that makes hallucinations dangerous. The rationale field lets you audit it; it does not make it true.

Measuring Success

• Per-code Cohen's kappa against your hand-coded sample. Every code, not a mean — and never the kappa on its own. A code below 0.6 at a base rate above ~15% is one you have not defined properly yet. Below that, check the raw agreement and Gwet's AC1 before you touch the definition: a rare code can post 98% agreement and κ ≈ 0 with nothing wrong with it. Report a confidence interval; statsmodels' cohens_kappa returns one.
• The does_not_fit rate. Healthy is roughly 5-15%. 0% means force-fitting. Above 30% means the codebook does not describe your data.
• What was in does_not_fit. Read it — this is where the thing you did not know to ask about lives.
• Codebook specificity. Could each code have been written before you ran the survey? If so, you have measured your own assumptions.
• Human hours per survey. Hand-coding 3,000 responses is days. This is four to six hours.
• Whether the readout changed a decision. The point was never the coding.

Cost Analysis

A 5,000-response survey, a 12-code codebook, Claude Haiku 4.5 at $1.00 per million input tokens and $5.00 per million output — halved by the Batch API to $0.50 and $2.50 (July 2026 pricing). Batch 20 responses per call, so the codebook is sent once per 20 responses rather than once per response:

• 250 calls, each carrying ~800 tokens of codebook and instructions plus 20 responses at ~60 tokens = ~2,000 input tokens per call → 500K input tokens.
• Output at ~60 tokens per response = 300K output tokens.
• Input: 0.5M × $0.50/M = $0.25. Output: 0.3M × $2.50/M = $0.75.
• Total: about $1.00. Add pilot runs and a re-run after you fix the codebook and you are still under $5. With no batching and no discount, the same job lands near $6.

The model is not the cost. Your time is: reading a sample (20 minutes); codebook generation and, above all, review (2-3 hours on the first survey); hand-coding 100 responses (1-2 hours); reading the does_not_fit bucket (1 hour, usually the best one). Call it a day for the first survey and two to three hours for each one after.

What it does not buy. Judgment about what the themes mean, responsibility for the anonymity you promised, or permission to skip the reliability check. Skip that check and you have saved three days to produce a number you cannot defend — worth less than the free-text box you were ignoring before.

Related Tools

• Claude — codebook development, and Claude Haiku 4.5 via API for batch classification
• ChatGPT — codebook generation and arguing with your own themes
• Google Gemini — large context for very big samples
• Ollama / LM Studio — local models, for responses carrying PII
• n8n — wiring the batch pipeline once the prompts are stable
• Perplexity AI — checking the frameworks behind a deductive codebook

Related Models

• Claude Haiku 4.5 — the default for batch classification: cheap, fast, reliable JSON
• Claude Sonnet 5 — the codebook step, where interpretation actually matters
• GPT-5.5 — strong generalist for inductive codebook generation
• Gemini 3.5 — large context when the sample itself is enormous
• DeepSeek V4 — cheaper classification; check its kappa before trusting it

New to structuring prompts this precisely? Start with prompt engineering — the difference between "find the themes" and a fixed schema with an escape category is the difference between a list and a method. See also text analysis and clustering.