Classifying 33,000 Usenet Threads for Fifteen Dollars

Written by Claude (Fable 5.1), which also built the pipeline it describes. dested prompted, chose Jev, paid the bills, and edited. The first person below is dested's voice.

I've been building alt.tv.rewind: take the Internet Archive's dumps of the alt.tv.* newsgroups, line every thread up against the original air dates, and let you click an episode of Seinfeld and read what Usenet said the morning after. The Finale aired May 14, 1998; by breakfast there were hundreds of posts. That's the product.

Air dates get you most of the way. A thread that starts the day after an episode airs is almost certainly about that episode. But "almost" leaves a lot on the floor, and the interesting stuff — was this a rave or a pile-on, is this a prediction I can grade against what actually happened, which line in here is the one worth pulling — needs something that reads the text.

That's a classification job, not a generation job. Every thread needs the same six answers:

1. Which of the 180 episodes is it about, if any
2. What kind of thread it is: reaction, prediction, theory, question, trivia, quote, news, meta, off-topic, spam
3. How the opener feels: loved / liked / mixed / disliked / hated / neutral
4. Is it a hot take other fans would argue with
5. Is it junk
6. Which sentence is the most quotable

Nothing here wants prose. I want a label and, ideally, how sure the model is. Asking a chat model to emit JSON works, but you're paying output tokens for a format you then have to parse and validate, and the "confidence" it writes in the JSON is a number it made up.

#Jev

Jev, from TypeSafe AI, is a model that only answers typed questions. You hand it a JSON state (up to 32k tokens) and a set of questions, each a choice over labeled criteria or a noul (a probability that a statement is true). It returns, for every question, a calibrated distribution over the labels. It cannot generate text at all. Input is $0.042 per million tokens; output is free, because there effectively isn't any.

The SDK is @typesafe-ai/sdk. The whole classify call is this:

import { choice, noul, TypeSafeClient } from '@typesafe-ai/sdk'

const client = new TypeSafeClient() // reads TYPESAFE_API_KEY

const result = await client.systemOne({
  state,     // Record<string, JsonValue>
  questions, // Record<string, Question>
  model: 'jev-latest',
})
// result.answers.kind → { type: 'choice', choice: 'reaction', confidence: 0.91, probabilities: {...} }
// result.answers.hot_take → { type: 'noul', noul: 0.12 }

#The questions

Every criterion is a plain declarative sentence. Jev reads them literally, so this is where the work goes:

export function buildQuestions(episodes: EpisodeRecord[], candidateLines: CandidateLine[]): Questions {
  const episodeCriteria: Record<string, string> = { none: 'Not primarily about one specific episode' }
  for (const ep of episodes) {
    episodeCriteria[ep.key] = `${ep.title} — first aired ${ep.airDate}${ep.summary ? ' — ' + ep.summary.slice(0, 90) : ''}`
  }

  const questions: Questions = {
    episode: choice(
      'Which episode is this thread primarily about? Pick none when the thread is about the show in general, about several episodes, or about something else. A thread started within a few days after an air date that reacts to last night’s episode without naming it is about that episode.',
      episodeCriteria
    ),
    kind: choice('What kind of thread is this, judging by the opening post?', {
      reaction: 'Opinions about an episode or the show — what worked, what did not',
      prediction: 'Guesses about what will happen in future episodes or to the show',
      theory: 'An explanation of a plot point, continuity detail, or hidden meaning',
      question: 'Asks for facts, identification of an episode, or help remembering something',
      trivia: 'Shares facts about cast, production, references, or things noticed',
      quote: 'Posts or collects lines and dialogue from the show',
      news: 'Ratings, scheduling, press coverage, cast or network news',
      meta: 'About the newsgroup itself — FAQ, netiquette, posting rules',
      offtopic: 'Not about the show',
      spam: 'Advertising, adult content, scams, or chain letters',
    }),
    sentiment: choice('How does the opening poster feel about the episode or show being discussed?', {
      loved: 'Enthusiastic praise',
      liked: 'Positive overall',
      mixed: 'Both praise and complaints',
      disliked: 'Negative overall',
      hated: 'Strong dislike or anger',
      neutral: 'No opinion expressed — a question, fact, or announcement',
    }),
    hot_take: noul('The opening post takes a strong, provocative stance that other fans are likely to argue with', {
      true: 'A bold or contrarian opinion stated forcefully',
      false: 'Mild, balanced, factual, or a question',
    }),
    spam: noul('The opening post is unsolicited junk rather than a genuine post to this newsgroup', {
      true: 'Advertising, adult content, get-rich schemes, chain letters, or unrelated mass posting',
      false: 'A real message from a participant, even if off topic',
    }),
  }

  // Jev can't quote text back, so it picks from sentences we extracted ourselves.
  if (candidateLines.length > 0) {
    const pullCriteria: Record<string, string> = { none: 'Nothing here stands out' }
    for (const c of candidateLines) pullCriteria[c.label] = c.text
    questions.pull_quote = choice('Which line is the most striking, funny, or quotable, read on its own?', pullCriteria)
  }
  return questions
}

The pull_quote question is the trick I'm proudest of. Jev can't write, so it can't quote — but it can choose. I unwrap each post's hard-wrapped paragraphs, split them into sentences, keep the ones that look like sentences (25–240 chars, starts with a capital, ends with punctuation, no URLs, not all-caps), label them L1..L12, and ask which one stands out. The model returns a label; I look the sentence back up. No hallucinated quotes, by construction.

#The state

The state is the thread, trimmed: opener, up to four earliest replies, timing, and hints from the heuristic pass. A real one, from June 1992 (signature elided):

{
  "newsgroup": "alt.tv.seinfeld",
  "show": "Seinfeld",
  "thread_started": "Thursday, June 25, 1992 12:36 PM ET",
  "timing_hints": [
    "Mentions 'The Finale (1)' — associated with S09E23 The Finale (1)"
  ],
  "subject": "Season finale",
  "opening_post": "So what's the deal with the season finale? I heard there was going to be some grand scheme involving Kramer actually appearing on another show on another network while Seinfeld was on, but in the end there was just that brief bit with Kramer appearing on Murphy Brown. It was pretty amusing, but not the grand event I was expecting. Anyone know what happened there?",
  "replies": [
    { "hours_later": 35.9, "text": "Name some other shows that so much as *mention* another network. What happened on Seinfeld in the season finale was a pretty big deal..." },
    { "hours_later": 49.1, "text": "\"Murphy Brown\" mentioned an NBC show earlier this season..." }
  ],
  "total_replies": 11,
  "candidate_lines": {
    "L1": "Name some other shows that so much as *mention* another network.",
    "L2": "The fact that CBS cooperated to get the Kramer bit on Murphy Brown and then released it to NBC for their use seems like a pretty big deal to me."
  }
}

Two details in there cost me a day. First, 61% of the Date: headers in this archive are 1996/05/17 with no time of day — 1990s newsreaders. If you parse that as local midnight you get a thread that "started 7 hours before the episode aired," and the model dutifully reasons about it. Now those posts carry a dateOnly flag all the way through: the state drops the clock ("Friday, May 17, 1996", no time), and replies switch from hours_later to days_later. Second, the hint about the finale is wrong — the heuristic matched "season finale" in the subject to the 1998 episode titled "The Finale," and the thread is about the 1992 one. Jev answered S09E23 at 54%, with the rest of its mass spread across none and the actual season-three finale. That's the right shape of answer for a misleading hint, and it's why I store the confidence.

#Folding answers back

The answers come out stringly typed. They go straight through the zod schema every downstream stage already uses, so a label the model shouldn't be able to produce throws instead of leaking into Postgres:

return Classification.parse({
  episode: episode.choice === 'none' ? null : episode.choice,
  episodeConfidence: pct(episode.probabilities[episode.choice] ?? 0),
  secondaryEpisodes: Object.entries(episode.probabilities)
    .filter(([key, p]) => key !== 'none' && key !== episode.choice && p >= 0.15)
    .sort((a, b) => b[1] - a[1]).slice(0, 3).map(([key]) => key),
  kind: kind.choice,
  kindConfidence: pct(kind.probabilities[kind.choice] ?? 0),
  sentiment: sentiment.choice,
  sentimentConfidence: pct(sentiment.probabilities[sentiment.choice] ?? 0),
  hotTake: hotTake.noul >= 0.6,
  hotTakeProbability: pct(hotTake.noul),
  spamProbability: pct(spam.noul),
  pullQuote, pullQuoteMessageId, // resolved from the chosen L-label, ≥ 0.3 or nothing
})

Having a real distribution changes what you can build. secondaryEpisodes is free: any episode with ≥15% mass that isn't the winner. The "🔥 Controversial" tab is hot_take ≥ 0.6 combined with how many distinct people piled in and how deep the reply tree got. None of that exists if the model hands you one string.

#The 255-label problem

A choice takes at most 255 labels. Seinfeld has 180 episodes, so the whole series fits in one question. The Simpsons has 803.

The fix is a per-thread roster: the heuristic pass's candidates go in first (title and nickname matches, air-date window), then the most recently aired episodes before the thread started, until the list hits 254. A 2003 thread sees everything from the previous ~250 episodes plus whichever old episode it names. A retro thread about a 1991 episode reaches it only through a title match, which is how it would be found anyway.

#The bill

One call per thread, six questions per call. The four archives I've loaded so far:

Seinfeld is measured from the API's own usage counts — 11,300 input tokens per call, flat across the whole run. The other three are projected from that rate. (The pipeline's dry run, which builds every payload and divides characters by four, said 7,500 per call and $10.30 for Seinfeld; Jev's tokenizer bills about 1.5× that. Lesson noted — measure the first hundred calls before trusting an estimate.) At $0.042 per million input tokens and nothing for output, that's 395,010 Usenet posts sorted into episodes, kinds, sentiments, hot takes, spam and quotes for about the price of a nice lunch.

Why one call per thread and not per message. The unit the product cares about is the thread — that's what gets a badge and a place on an episode page — and a thread's character is set by its opener and the first few replies. Classifying all 157k Seinfeld messages individually would be 5× the calls for a worse signal, because most replies are "no it wasn't" with three lines of quoted context. So each call sees the opener (≤2,000 chars), the four earliest replies (≤800 each), and the timing. The count of calls is exactly the count of non-spam threads: 33,550 minus the 864 the heuristics already flagged as junk.

Why six questions per call and not six calls. Jev charges for the state once per call regardless of how many questions ride on it. Splitting episode / kind / sentiment / hot-take / spam / pull-quote into separate calls would multiply the bill by six for identical answers.

Why 11k tokens per call when the thread is 800 words. The thread is maybe a quarter of it. The rest is the episode question: 181 labels, each a title, an air date and 90 characters of TVMaze summary, re-sent on every call — there's no prompt caching, and the roster differs per show. I could shrink it (titles only, or a 30-episode window around the thread date), and for a show with 800 episodes I have to. For Seinfeld I left it, because the summaries are what let it match a thread that never names the episode, and the whole run was fifteen dollars.

Why 18 requests per second. The account limit is 1,200 per minute. Sixteen workers behind a token bucket at 18/s stays under it with no retries; the run finished in about 30 minutes with zero rate-limit errors. The only failures all night were 402s when the org's prepaid credits ran out mid-run — the stage appends as it goes and skips finished threads on restart, so topping up and re-running the same command resumed from thread 12,676.

The other two models. Jev is the cheap, wide layer. Haiku 4.5 only sees the ~⅓ of threads that earned a summary or carry a prediction, through the Batches API with the episode list cached in the system prompt; the measured rate from the smoke batch is about $0.0017 per thread, so roughly $18 for Seinfeld. Opus 5 writes one recap per episode — 180 calls for Seinfeld — and hasn't run yet; I'll report the invoice when it does. The shape is the point: the model that costs the most runs the fewest times, on inputs the cheaper models have already filtered.

For comparison, the version of this I sketched first — Haiku 4.5 doing everything through Batches — priced out at roughly 5× the Jev figure for the classification layer alone, without calibrated probabilities, and with a wait of thirty minutes to a day per batch.

#Results

What came back, over all 32,686 threads (June 1992 – June 2013):

Thirty-eight percent of everything ever posted to alt.tv.seinfeld is someone asking a question — "what episode was it where…" — which anyone who used Usenet will find exactly right. 3,355 threads cleared the hot-take bar; 904 that the regex spam filter missed came back with spam ≥ 0.9 and were dropped. 23,056 threads got a pull quote.

The thing I actually cared about — episode attribution — did the most. The heuristic pass had tied 2,043 threads to a specific episode as live reaction and 4,992 as retro. With Jev deciding, that's 4,845 live and 6,224 retro at a mean confidence of 82 — the gain is mostly threads that react to "last night" without ever naming the episode, exactly the case a date-plus-title matcher can't see.

And the reason to build any of this. The Invitations (S07E24, May 16, 1996) is the episode where Susan dies licking the cheap wedding envelopes George picked. The morning after, on alt.tv.seinfeld, the live threads with an opinion split mixed 27 · disliked 19 · hated 12 against liked 21 · loved 9. Two years later The Finale did the same thing at three times the volume: mixed 81 · disliked 77 · hated 53 against loved 68 · liked 58 — the most divisive night in the archive, by the numbers, which is how everyone remembers it. The biggest Invitations thread is titled "Killing off Susan UNACCEPTABLE" — 65 posts, 49 people. Jev's pull quote from it:

But killing off Susan to free George from marriage was unacceptable.

and from a rebuttal thread a few days later, the one signed by Kenny Kramer:

People have either loved the show or hated it, but nobody's been indifferent.

Both of those are sentences a person wrote in 1996, chosen by a model that cannot write sentences.

#What Jev can't do, and what I used instead

It can't summarize, and it can't grade a prediction against what actually happened on the show — both need generation and knowledge. So the pipeline is three models with a clean split:

• Jev — every thread, every label. $15 for the biggest show.
• Claude Haiku 4.5, Message Batches, structured output via output_config with a cached system prompt — one-line summaries and prediction claim/outcome for the ~⅓ of threads that qualify. Its first graded prediction was "Kramer will leave Seinfeld permanently after getting a job on Murphy Brown → did not happen," which is correct and also the thread from the JSON above.
• Claude Opus 5 — one recap paragraph per episode, written from that episode's live threads.

Each stage appends to a JSONL checkpoint and skips what's done, so the credit outage cost nothing but a restart. (One self-inflicted lesson: a --force smoke test on the same file while the real run was appending truncated 1,383 finished records. Don't share a checkpoint between a test and a run.)

#If you're evaluating it

Reach for Jev when the output is a decision, not a document: routing, moderation, tagging, extraction into a fixed vocabulary, "which of these N things," "is this true." Write criteria as declarative sentences, not instructions. Give it context in state and keep the questions about the state. If you need a quote, a name, or a number copied out of the text, extract candidates yourself and make it a choice. And keep the label count under 255, or build a roster.

The whole pipeline is in the repo — pipeline/lib/jev.ts is the wiring above, pipeline/stages/classify.ts is the runner. Next up are the other 22 alt.tv.* archives sitting on archive.org.