Claude Can Code Your Open Ends?

By Kevin Collins

A few months ago, we reported on research from last year that showed the limitations of large language models at coding open-ended questions. The takeaway, which was based on asking about what news events captured respondents’ attention, was that most models failed abysmally. Admittedly this is a hard test of the models’ abilities, both because current events are outside the training data when the models are used contemporaneously, and because respondents can name multiple events in a single response. Opus 4.7 (tested months after the data collection) looked promising, but the training data for that model included the events being described, making it an unfair test.

So, we did it all again with new data and the latest models. And what a difference a few months make. We tested four Claude models (Haiku 4.5, Sonnet , Opus 5, and Fable 5), the frontier Open AI model (GPT-5.6 Sol), and a commercially available open-end coding software that pulls from different LLMs. The lower tier models, Claude Haiku and Claude Sonnet, performed approximately as well as the frontier models did a year ago. Next best were GPT-5.6 Sol and the commercially available harness, and then Claude Opus was only slightly edged out by Claude Fable. For simplicity, we will focus on the top performing model (Claude Fable) here, though results for all models are available in the methodological appendix. Takeaway #1: Topline accuracy of frontier models today is dramatically better in July 2026 than it was in August 2025. 

The summary metric we focus on is the F1 score, a measure of accuracy which synthesizes “precision” (which minimizes false positives) and “recall” (which minimizes false negatives). The violin plots below shows the distribution of F1 scores for the frontier model used contemporaneously in August 2025 (Open AI GPT-5), and a new set of open ends in July 2026 (Claude Fable 5). 

Takeaway #2: The frontier models are no longer writing classifier python scripts but instead report they are using subagents to report a multi-agent coding operation.

In the analysis of the August 2025 data, all models performed the tasks by writing python classifier scripts, which were essentially (very sophisticated) keyword searches, and Claude’s lower-tier Haiku and Sonnet models still operated with that same approach in July 2026. But Claude Opus, Fable, and Sol do something different. 

These advanced models expand a prompt into a codebook written as a markdown file for reference by the coding agents, and then have multiple agents code the sample concurrently. Importantly, several of these models (GPT-5.6 Sol and Claude Opus) leveraged web-search capabilities to provide more context about the topics and to search specific responses to identify likely news stories being referenced. Subsequent testing showed this approach can also be structured to output confidence metrics in each classification.

We also tried a number of variations on our prompting strategy to test if we could improve beyond the performance described here. These included instructing Fable to use web search (which Fable reports it did not use to produce these responses), instructing Fable to use a variety of approaches to ensemble decision makings (requiring unanimity or majority vote among agents asked to recode low-confidence measures), and providing the dates of fielding to offer additional context for the task None of these advancements meaningfully improved performance for Fable (though some, like web searching, were already used by Opus and GPT Sol). 

Takeaway #3: This change in approach means frontier models have greater consistency in performance across categories and category types 

The previous method of writing classifier scripts in python produced relatively reliable results when categories were defined by proper nouns, such as Ukraine, Donald Trump, Taylor Swift, or Cracker Barrel (all topics present within the August 2025 data), but less so when the categories were broader (e.g. the economy, sports, or immigration enforcement). 

However, the alternative approach used by current frontier models perform much more similarly across categories that are and are not defined by proper nouns. We look at this with violin plots of the distribution of F1 scores separately for these different categories. As the plots below show, while Fable had slightly higher F1 scores for proper nouns, the difference was marginal. On average, Fable performed better with topics not defined by proper nouns than GPT 5 did a year ago with those that were defined by proper nouns. 

In addition to looking at accuracy by category, we also classified the accuracy of individual responses. Looking at the share of categories matching hand-codes, we found that in 2025, the GPT matched our hand-coding at least 95% of categories for only 55% of responses. In 2026, Fable’s coding matched at least 95% of hand-coded responses for 98% of responses, essentially as good of a match as another human coder could be expected to perform.

Takeaway #4: Open end coding harnesses do not outperform underlying frontier models in our testing. 

We also tested commercially-available software that calls upon frontier models, a “harness” in the language of the industry. We did not find this worked better for coding open-ends than interacting with the models through their interfaces directly. Commercial software may offer other advantages, such as working collaboratively with a team or developing the categories in the first place. However, given the significant additional costs associated with subscriptions to these harnesses, we do not plan to use them in the future.

Takeaway #5:  The improvements to these tools facilitate a more iterative approach, and as a result, researchers should focus on where we add the most value: deriving meaning rather than counting 

The earlier “write a python script” approach is essentially a one-shot interaction: the LLM writes a python script on the basis of the researcher’s prompt. One can hand-code a sample and then compare human results to LLM results for accuracy. But if one finds a low correspondence, the conclusion then means scrapping the whole approach (as it did last August in the prior research).

Now, the AI’s alternative strategy creates more opportunities to interact with the model before the work is completed. In particular, one can ask the model to pause after creating the codebook, inspect it, and make changes as needed before proceeding. One can ask for average confidence by category, and consider whether that category needs to be better defined. One can ask it to export all of the records below a certain level of confidence for hand-coding. 

We propose the following process moving forward, and we have written a Claude skill to institutionalize this process at Survey 160. 

We continue to believe that researchers should read every open-ended response, both as part of constructing the categorization schema and in order to do the important “sense-making” work of survey research. But through iterative and interactive partnership with models to do the time-consuming work of categorizing text, human researchers can focus on our greatest value add to the process, which is not calculation but deriving meaning from those numbers. 

Methodological Appendix 

Methodological details from the two surveys are described in detail in previous blog posts here and here. In brief, the August 2025 survey was fielded from August 26 to August 30, 2025 by text-to-web, live interviewer interactive text, and panel-to-web (Cint) modes. The July 2026 survey was fielded July 14 to July 20, 2026 by text-to-web and panel-to-web (RepData) modes. 

The question to which responses were classified was called the “Attention Barometer”, and reads “In a few words, what news story caught your attention this week?  ‘News’ can be anything you follow, whether local TV, websites, podcasts, or social media, on any topic like sports, entertainment, or politics. We just want to know what caught your attention. If you can’t think of anything specific, that’s okay, just say so. Please reply in your own words”

The models used to classify the August 2025 survey responses were GPT 5 (done contemporaneously, in early September 2025), GPT 5.2 (done in April 2026; it has a training cutoff date of August 31, 2025 so theoretically covers the events described by respondents), and Opus 4.7 (also done in April 2026, with a training cutoff date of January 2026, definitively covering the events described by respondents).

The models used to classify the July 2026 survey responses were four models from Anthropic – Claude Haiku 4.5, Claude Sonnet 5, Claude Opus 5, and Claude Fable 5 (all of these have training data cutoff date that precede the survey data collection) – OpenAI’s GPT-5.6 Sol, and the BTInsights commercial platform which draws from multiple, unspecified models. This coding work was done in July and August 2026. The prompts to these models described the inputs, desired outputs, and provided as additional context a description of each category and and example response that matched that category. 

Below are the F1 scores for each of the categories used to produce the distributions in the graphs above, first for the August 2025 survey and second for the July 2026 survey. The graphs above only show the contemporaneous model in August 2025 and the best performing model in July 2026 for visual clarity.