A fine-tuned text classifier that labels r/nba comments as analysis, hot_take, or meta — trained on real comment threads and compared against a zero-shot LLM baseline.
Community: r/nba — specifically, the comment section of a post titled "No ounce of Sportsmanship from Wemby or anyone from the Spurs", posted immediately after the Spurs lost the 2025 NBA Finals to the Knicks.
Why this community: r/nba is an ideal classification target because discourse quality varies dramatically within a single thread. Some comments cite specific game events, historical precedents, and tactical observations. Others are pure emotional reactions or hyperbole. A third distinct type — unique to large fan communities — consists of comments about the discourse itself: observations about how the fanbase is reacting, which narratives are being constructed, and how Reddit behavior is shaping the conversation. These three types are recognizable to anyone who reads NBA threads regularly, making the label taxonomy grounded in real community norms rather than invented from the outside.
The specific post was chosen because it generated a high volume of nested discussion (nearly 200 comments), covered a topic with genuine disagreement, and naturally produced all three label types in roughly balanced proportions.
The comment makes a structured argument supported by specific evidence: a named game event, a verifiable statistic, a historical comparison between players or eras, or a precise tactical observation. The evidence would support the claim even if the emotional framing were removed.
Example 1: "Knicks just posted a video showing quite a few spurs players congratulating them. Just not Wemby." — cites a specific, verifiable piece of evidence (a posted video) that directly counters the original post's claim.
Example 2: "In the WCF the Thunder players hung around to shake hands as the Spurs celebrated but the Spurs never stopped to acknowledge them. SGA literally went to Harper to congratulate him and he got brushed off." — cites a named event from a prior round with specific players named, establishing a pattern across series.
A bold or strong opinion asserted without supporting evidence. The comment may be confident, sarcastic, hyperbolic, reactive, or emotionally charged — but it asserts rather than argues. Jokes, brief reactions, and normative claims without evidence all belong here.
Example 1: "After losing to the Knicks, instead of congratulating their opponents they decided to head straight to the locker room. Total lack of class and maturity from the Spurs." — strong characterization with no supporting evidence beyond the assertion itself.
Example 2: "He just ended his media interview telling the reporters 'See y'all never' lmao he's salty af." — labels a player based on a quote snippet without engaging with context or making an argument.
The comment is primarily about the discourse, the fanbase, Reddit's behavior, or the media/league narrative — rather than about the players or the game. Meta comments discuss how people are reacting, which narratives are being pushed, how the subreddit behaves over time, or what the media/league is doing to manage perception.
Example 1: "This sub is bipolar as always. Somebody always has to be the bad guy. OKC, now Wemby, next year yall will find a reason to hate Brunson too." — explicitly about the subreddit's pattern of rotating villains.
Example 2: "The best part about Wemby these finals is if you like him you can make a case for him being gassed, and if you hate him you can make a case for him being lazy. Both are somewhat plausible." — an observation about the structure of the debate itself, not a position within it.
Source: A single r/nba post collected via the Reddit JSON API endpoint. All comments including nested replies (up to 7 levels deep) were extracted recursively.
Total comments collected: 194 raw comments
After filtering (removed comments under 4 words, deleted/removed accounts): 184 usable examples
Labeling process: Each comment was read individually and assigned one of the three labels using the definitions above. Comments were pre-annotated using Claude with the full label definitions and a one-sentence justification per comment. Every pre-annotation was reviewed and corrected where the label did not match the decision rules — approximately 15–20% required correction, primarily at the analysis/hot_take boundary for sarcastic comments that cited real events. The annotation_notes column in the CSV records the reasoning for each label.
Label distribution:
| Label | Count | Percentage |
|---|---|---|
analysis |
76 | 41.3% |
hot_take |
70 | 38.0% |
meta |
38 | 20.7% |
| Total | 184 | 100% |
Three genuinely difficult examples:
-
"He commited two of the biggest aura gambles ('i'm in your head' and 'everyone knows we're gonna do it') and neither paid off."
Could behot_take(mocking Wemby after the fact) oranalysis(cites two specific verifiable statements and evaluates their real outcomes). Labeledanalysis— the cited quotes are real and their failure is verifiable from the series result. Decision rule: if specific evidence supports the claim even when tone is stripped, label asanalysis. -
"I see that wemby has speed run his way from beloved nba star to nba villain."
Could bemeta(describing fan perception dynamics) orhot_take(the commenter making a characterization of Wemby). Labeledmeta— the comment is describing what fans/Reddit are doing collectively, not staking a personal claim about Wemby's character. -
"For real. Bird didn't shake the Bad Boys' hands when he lost. Bad Boys didn't shake MJ's hand when he beat them. These athletes are competitors."
Could behot_take(brief opinion) oranalysis(cites historical examples). Labeledanalysis— two specific named historical examples function as evidence for a normative claim, even though the conclusion is stated bluntly.
Base model: distilbert-base-uncased (HuggingFace) — a lightweight 66M-parameter transformer pre-trained on English text via masked language modeling. Chosen for speed on a free Colab T4 GPU and strong performance on short-text classification tasks.
Training setup:
- Framework: HuggingFace
transformers+datasets+scikit-learn - Dataset split: 70% train / 15% validation / 15% test (stratified)
- Hardware: Google Colab T4 GPU
- Training time: ~8 minutes
Hyperparameter decisions:
| Parameter | Value | Reasoning |
|---|---|---|
| Learning rate | 3e-5 | Increased from default 2e-5 to help model move faster on a very small dataset (128 training examples) |
| Epochs | 5 | Increased from default 3; validation loss was still decreasing at epoch 3, best validation accuracy (0.607) was reached at epoch 3–4 |
| Batch size | 8 | Reduced from default 16; with only 128 training examples, batch size 16 produces only 8 gradient updates per epoch — too few for stable learning |
| Max sequence length | 256 | Default in the starter notebook; nearly all comments fit within this limit |
The training curve showed genuine learning: loss decreased from 1.091 to 0.629 across 5 epochs and validation accuracy peaked at 0.607 in epochs 3–4 before the best model checkpoint was saved.
Model: Groq llama-3.3-70b-versatile (zero-shot)
Prompt used:
You are classifying comments from r/nba.
Assign each comment to exactly one of the following categories.
analysis: The comment makes a structured argument supported by specific evidence — a named game event,
a verifiable statistic, a historical comparison, or a tactical observation. The evidence supports the
claim even if emotional framing is removed.
Example: "Knicks just posted a video showing quite a few spurs players congratulating them. Just not Wemby."
hot_take: A bold or strong opinion asserted without supporting evidence. May be confident, sarcastic,
hyperbolic, or reactive, but asserts rather than argues. Jokes, brief reactions, and normative claims
without evidence are hot_take.
Example: "22 and already the biggest front runner in the NBA. quite impressive stuff."
meta: The comment is primarily about the discourse, the fanbase, Reddit behavior, or media narrative
rather than about the players or the game. Often discusses how people are reacting or which narratives
are being pushed.
Example: "This sub is bipolar as always. Somebody always has to be the bad guy."
Respond with ONLY the label name.
Do not explain your reasoning.
Valid labels:
analysis
hot_take
meta
Collection: Each test set example was sent to the Groq API individually at temperature 0. All 28 responses were parseable (0 unparseable).
| Model | Accuracy |
|---|---|
| Zero-shot baseline (Llama 3.3 70B) | 64.3% |
| Fine-tuned DistilBERT | 53.6% |
The fine-tuned model underperformed the zero-shot baseline by 10.7 percentage points. This is an honest result that reflects a real constraint: 128 training examples is insufficient for DistilBERT to learn a three-way functional distinction that a large language model with strong priors handles reasonably well out of the box. The result is itself informative — see the reflection section for analysis of what this gap reveals.
Fine-tuned DistilBERT:
| Label | Precision | Recall | F1 | Support |
|---|---|---|---|---|
analysis |
0.58 | 0.58 | 0.58 | 12 |
hot_take |
0.50 | 0.80 | 0.62 | 10 |
meta |
0.00 | 0.00 | 0.00 | 6 |
| macro avg | 0.36 | 0.46 | 0.40 | 28 |
Zero-shot baseline (Llama 3.3 70B):
| Label | Precision | Recall | F1 | Support |
|---|---|---|---|---|
analysis |
0.78 | 0.58 | 0.67 | 12 |
hot_take |
0.50 | 0.80 | 0.62 | 10 |
meta |
1.00 | 0.50 | 0.67 | 6 |
| macro avg | 0.76 | 0.63 | 0.65 | 28 |
The most striking finding: the fine-tuned model predicted meta exactly zero times across all 28 test examples (F1 = 0.00), while the baseline correctly identified 3 of 6 meta comments (F1 = 0.67). The baseline also shows better precision on analysis (0.78 vs 0.58). The fine-tuned model's only advantage is higher recall on hot_take (0.80), but this is partly because it over-predicts hot_take by routing all meta examples there.
Rows = true label, Columns = predicted label.
| pred: analysis | pred: hot_take | pred: meta | |
|---|---|---|---|
| true: analysis | 7 | 5 | 0 |
| true: hot_take | 2 | 8 | 0 |
| true: meta | 3 | 3 | 0 |
The right column is entirely zero — the model never once predicted meta. All 6 true meta examples were misclassified: 3 as analysis, 3 as hot_take. This is the defining failure of this model.
Wrong prediction 1:
"He hugged Brunson after the buzzer"
True label: analysis | Predicted: hot_take (confidence: 0.54)
This short comment cites a specific named behavior (Mitch Johnson hugging Brunson) as evidence that the sportsmanship narrative in the original post was wrong. It functions as evidence, not opinion. The model predicted hot_take, likely because the comment is only 7 words with no hedging language — it learned that brief, direct statements are hot_take. The boundary the model missed: brevity and directness are surface features; what matters is whether the content is evidence or assertion.
Wrong prediction 2:
"The Knicks just won a championship and you're finding a reason to be angry. Get over yourself and enjoy it"
True label: meta | Predicted: analysis (confidence: 0.83)
This comment is directed at the original poster's behavior — telling them they're engaging with the discourse in the wrong way. It's functionally meta. The model predicted analysis with high confidence (0.83), likely because the comment contains a factual claim ("The Knicks just won a championship") that looks like evidence. The model can't distinguish a factual statement used as rhetorical setup from one used as analytical support.
Wrong prediction 3:
"People may hate SGA cause he's a foul merchant. But he's a good dude, and shows good sportsmanship."
True label: analysis | Predicted: hot_take (confidence: 0.64)
This comment makes a structured distinction — separating SGA's on-court style (foul-drawing) from his character — and uses that distinction to argue a point. The model predicted hot_take, probably because the evaluative conclusion ("he's a good dude") sounds like an opinion. The model learned to key on the final evaluative claim rather than the argumentative structure leading to it.
The dominant failure is complete blindness to meta. With only 26 meta examples in training (after the 70% split), the model never learned to predict it at all. The meta examples were split between being predicted as analysis (3 cases) and hot_take (3 cases), suggesting the model is routing them based on surface features of the individual comment rather than the functional question of whether the comment is about the discourse.
A secondary pattern is low-confidence predictions across the board — nearly all predictions in the first run had confidence around 0.34–0.36, suggesting the model was barely moving off the prior distribution. After adjusting hyperparameters (5 epochs, batch size 8, learning rate 3e-5), confidence spread widened (0.46–0.83), but the meta blindness persisted.
The analysis/hot_take boundary improved meaningfully: the model correctly classifies most hot_take examples (8/10) and most analysis examples (7/12). The remaining analysis errors (5 misclassified as hot_take) are all short comments where the evidence is implicit rather than explicit.
Comments run through the fine-tuned model with predicted label and confidence:
| Comment (truncated) | True | Predicted | Confidence |
|---|---|---|---|
| "Knicks just posted a video showing quite a few spurs players..." | analysis |
analysis |
0.71 |
| "Wemby threw up the 3 and walked away so fast lol" | hot_take |
hot_take |
0.68 |
| "This sub is bipolar as always. Somebody always has to be the bad guy." | meta |
hot_take |
0.53 (wrong) |
| "Mitch plus Keldon were the only ones shaking hands with OKC players in the WCF" | analysis |
analysis |
0.62 |
| "He hugged Brunson after the buzzer" | analysis |
hot_take |
0.54 (wrong) |
Correct prediction explained: "Knicks just posted a video showing quite a few spurs players..." is correctly predicted as analysis with 0.71 confidence. This is the clearest analysis example in the dataset — it contains a specific, verifiable piece of evidence (a posted video) that directly contradicts the original post. The model's confidence here is justified: this comment has the exact structure it learned to associate with analysis.
My label definitions were built around function: does this comment argue (analysis), assert (hot_take), or observe the discourse (meta)? The model learned surface co-occurrence patterns from 128 examples instead.
The clearest evidence: meta F1 of 0.00 despite meta being 21% of the dataset and having a clear definition. The model didn't learn what makes a comment meta — it never learned to predict the label at all. With only ~26 meta training examples from a single thread on a single topic, the model saw too few examples of the category to extract a generalizable signal. Worse, many meta comments are superficially similar to hot_take comments — both use first-person language, both are often short, both react to something. The difference is functional (is the subject the game or the discourse about the game?), and 26 examples isn't enough to learn that functional distinction.
For analysis and hot_take, the model learned something real but incomplete. It learned that short, unhedged statements lean hot_take and that named-player references with specific claims lean analysis. This is approximately correct but misses the cases where a short unhedged statement is actually evidence (e.g., "He hugged Brunson after the buzzer") or where a named-player claim is actually an opinion ("Draymond has always faced the music").
The zero-shot baseline outperformed the fine-tuned model overall because Llama 3.3 70B has strong priors about what "argument" and "opinion" mean from its training data, and those priors generalize better than patterns learned from 128 examples. Fine-tuning would require substantially more data — likely 500+ examples across multiple threads and topics — to overcome the baseline on this task.
One way the spec helped: The requirement to write planning.md before annotating any data forced me to confront the analysis/hot_take boundary before I had 184 examples in front of me. Writing the decision rule ("if the evidence supports the claim when tone is stripped, it's analysis") before labeling prevented me from making inconsistent judgment calls mid-annotation.
One way implementation diverged from the spec: The spec warns that fine-tuning should meaningfully exceed the baseline, and recommends investigating if it doesn't. I did investigate — adjusting epochs, batch size, and learning rate — but the model still underperformed the baseline. Rather than treating this as a failure to hide, I documented it honestly. The result is informative: it shows that 128 training examples from a single thread is insufficient for a 70B zero-shot model to be beaten on a subjective three-way classification task. That is itself a valid finding about the limits of small-dataset fine-tuning.
Instance 1: Pre-annotation of all 184 comments.
I gave Claude the full label taxonomy (definitions, examples, and decision rules from planning.md) and asked it to assign one label and one justification sentence per comment. Claude returned labels for all 184 examples. I then reviewed every label against the decision rules, correcting approximately 30 cases — primarily sarcastic comments that Claude labeled as analysis because they named specific players or events, when the functional intent was clearly hot_take (e.g., "Pop would have this team as champions" — Claude labeled analysis, I corrected to hot_take because it's counterfactual speculation with no supporting argument). I also corrected several meta predictions that Claude made for comments that were first-person reactions rather than observations about the discourse.
Instance 2: Failure pattern analysis.
After running the model, I pasted the wrong predictions into Claude and asked it to identify common themes. Claude identified three candidate patterns: (1) short comments being mislabeled, (2) sarcasm confusing the boundary, and (3) the model never learning meta at all. I verified pattern 3 against the confusion matrix — confirmed, 0 meta predictions. Pattern 1 (length) was not consistently supported on close re-reading. Pattern 2 (sarcasm) was partially supported but less central than the meta failure. The final analysis reflects what I could verify from the actual outputs, not everything Claude suggested.