NLP
Reading 22,000 reviews at once: what customers talk about, and what the unhappy ones talk about.
10
Themes found
0.956
Tone model ROC-AUC
33%
Fabric & cut letdowns in 1–2★ reviews
22,641
Reviews analysed
Overview
Star ratings say how a customer felt, not why. The why is in the text, and nobody reads twenty thousand reviews. This is the problem a feedback tool has to solve: turn open text into a handful of themes a team can act on.
This rebuild finds 10 themes in clothing reviews with a topic model that is never told what to look for, links them to stars and the recommend box, and adds a tone model that flags unhappy reviews. Both run in your browser on any text you type.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Raw data
23,486 reviews
22,641 with text
- 02
Words
TF-IDF
2,000 words, stop words removed
- 03
Themes
NMF, k = 10
Number of themes chosen by coherence
- 04
Link
themes × stars
Which themes fill the 1–2★ reviews
- 05
Tone
AUC 0.956
Logistic model on words and phrases
- 06
Ship
8-bit weights
Themes and tone computed in the browser
Inside the model
Themes nobody tagged.
One model finds what reviews talk about, another reads their tone. Scroll through a real review.
A real review
10 themes found in 22,641 reviews, no labels given
Fabric & cut letdowns
fabric · look · fit · nice · didn
Quality & versatility
great · looks · quality · fits · fall
True to size
size · true · fits · fit · usual
Runs small or large
small · large · medium · runs · big
Sweaters & warmth
sweater · soft · warm · sleeves · long
Comfort & compliments
comfortable · cute · super · soft · flattering
Jeans & pants
jeans · pants · pair · skinny · fit
Length & season
perfect · length · summer · fall · fit
Skirts & waist
skirt · waist · hips · pencil · skirts
Colour
color · beautiful · blue · green · red
This review as a mix of themes
A separate neuron for tone
71%
chance this reviewer would not recommend it · ■ towards not · ■ towards recommend
Every theme: share of reviews · how many would not recommend
The data
23,486 clothing reviews, written by real customers.
22,641 have text, a median of 59 words. Nobody labelled the topics: the themes below were found by the model.
Women's E-Commerce Clothing Reviews
Anonymised reviews from a women's clothing e-commerce shop, with the shop's name replaced by "retailer" in the text. Each review has a star rating and whether the customer would recommend the item. The reviewer's age is in the data but is not used or shown here. Licence: CC0: Public Domain.
23,486
reviews
22,641
with text
59
median words
18.1%
would not recommend
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| Title · Review Text | text | What the customer wrote; themes come from the text, the tone model reads both |
| Rating | 1–5 | Stars given; used to read the themes, not to find them |
| Recommended IND | 0 / 1 | Would recommend the item: the target of the tone model |
| Division · Department · Class | category | Product grouping; used to compare themes across departments |
| Positive Feedback Count | number | Other shoppers who found the review helpful · not used |
| Clothing ID | id | Which item · not used or shown |
| Age | number | Reviewer's age · left out (personal attribute) |
Explore the data
Mostly five stars. The useful part is the rest.
Most reviews are glowing. Three stars is a coin flip: 59% of 3★ reviewers would not recommend the item.
A median of 59 words: enough to say what fits, what doesn't and why.
Tops and dresses dominate. Average ratings barely differ (4.14 to 4.28), so stars alone don't say what to fix.
Try it
Paste a review, see its themes and tone.
Themes come from an unsupervised topic model; the tone comes from a separate classifier trained on the recommend box. Both run in your browser.
Real rows from the dataset
click a row to try it in the demo ↓
| Review | Stars | Department | Main theme | Unhappy | Recommended |
|---|---|---|---|---|---|
| Cute dress but not for me. the waist is too high an… | 3★ | Dresses | Length & season | 71% | Yes |
| Bought this blouse in white. fits great. can be worn… | 5★ | Tops | Quality & versatility | 0% | Yes |
| I bought this dress online last week (in large part … | 5★ | Dresses | Fabric & cut letdowns | 45% | Yes |
| I really love this dress, just not on me! i am very … | 4★ | Dresses | Fabric & cut letdowns | 25% | Yes |
| I ordered a small. i'm usually a petite small. i lik… | 5★ | Intimate | Runs small or large | 3% | Yes |
| If you're expecting a sweatshirt when you order this… | 4★ | Tops | Jeans & pants | 2% | Yes |
| This dress is amazing. bought in orange and fits won… | 5★ | Dresses | Quality & versatility | 0% | Yes |
| These pants feel great. fit is true to size. the blu… | 5★ | Bottoms | True to size | 0% | Yes |
| I tried this top on with a pair of white shorts, hal… | 4★ | Tops | Jeans & pants | 0% | No |
| I love these leggings! the material is a bit thicker… | 5★ | Bottoms | Jeans & pants | 0% | No |
8 random reviews from the test split plus the 2 the tone model got most confidently wrong: both read as happy, yet the reviewer ticked 'would not recommend'.
Live demo
runs in your browser
What decided the tone
Beautiful color but the fabric is thin and see-through. Sending it back.
Tone
92% chance this reviewer would not recommend the item
Themes in this review
- Colour96%
- Fabric & cut letdowns4%
Theme words found: beautiful, color, fabric, sending
What it found
10 themes, and the one behind the bad reviews.
“Fabric & cut letdowns” takes up 33% of what low-rated reviews say, against 15% in high-rated ones.
Names are mine, read off each theme's top words and its most typical review. Bigger words weigh more in the theme.
Fabric & cut letdowns
fabriclookfitnicedidnmaterialgoodtried
Quality & versatility
greatlooksqualityfitsfallfitsummerpiece
True to size
sizetruefitsfitusualperfectlysmallerregular
Runs small or large
smalllargemediumrunsbigrunlittlenormally
Sweaters & warmth
sweatersoftwarmsleeveslongcozycoatitchy
Comfort & compliments
comfortablecutesupersoftflatteringcomfymaterialcompliments
Jeans & pants
jeanspantspairskinnyfitblackleggingslook
Length & season
perfectlengthsummerfallfitspringlongcolors
Skirts & waist
skirtwaisthipspencilskirtsworkflatteringknee
Colour
colorbeautifulbluegreenredpinkpersonpurchased
Fabric & cut letdowns more than doubles in low-rated reviews; Length & season is the mark of the happiest ones (4.69★ on average).
- Fabric & cut letdowns33% / 15%
- Runs small or large12% / 9%
- Sweaters & warmth6% / 5%
- Skirts & waist4% / 5%
- Colour12% / 13%
- True to size9% / 11%
- Jeans & pants6% / 9%
- Comfort & compliments8% / 13%
- Quality & versatility6% / 10%
- Length & season4% / 11%
Fabric & cut letdowns lead in tops, dresses and jackets; bottoms are about jeans and waists; intimates are mostly about comfort.
| Tops | Dresses | Bottoms | Intimate | Jackets | |
|---|---|---|---|---|---|
| Fabric & cut letdowns | 19% | 22% | 12% | 17% | 16% |
| Quality & versatility | 9% | 9% | 8% | 10% | 10% |
| True to size | 9% | 12% | 11% | 10% | 9% |
| Runs small or large | 11% | 10% | 7% | 12% | 13% |
| Sweaters & warmth | 9% | 2% | 1% | 4% | 8% |
| Comfort & compliments | 12% | 12% | 11% | 18% | 10% |
| Jeans & pants | 7% | 3% | 20% | 8% | 6% |
| Length & season | 8% | 11% | 9% | 9% | 11% |
| Skirts & waist | 2% | 5% | 13% | 2% | 2% |
| Colour | 14% | 14% | 8% | 10% | 14% |
Topic coherence (NPMI: do a theme's top words really appear together?) peaks at 10; at 12 the themes start repeating each other.
Flag reviews above 50%: 16% of reviews get flagged, 81% of those really wouldn't recommend, and 73% of all unhappy reviewers are caught.
Single words straight from the shipped model's weights; “not” is among the strongest warning signs. The model also uses two-word phrases, not shown here.
Towards “wouldn't recommend”
Towards “would recommend”
Words carry the signal: the theme mix alone reaches ROC-AUC 0.78, the words model 0.96. Regularisation picked by 3-fold CV on the training split.
| Model | ROC-AUC | Avg precision | F1 | Accuracy |
|---|---|---|---|---|
| Naive Bayes · words | 0.944 | 0.789 | 0.470 | 86.9% |
| Linear SVM · words | 0.958 | 0.828 | 0.750 | 91.6% |
| Logistic · words | 0.959 | 0.824 | 0.732 | 91.3% |
| Logistic · theme mix only | 0.782 | 0.441 | 0.271 | 82.5% |
| Shipped: 2,500 n-grams, 8-bit | 0.956 | 0.823 | 0.767 | 92.0% |
Build notes
How it's built, step by step.
The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.
Run it yourself
feedback() from ml/train.py + 4 helpers (r, save, _read_arff, FEEDBACK_THEME_NAMES)
pip install numpy pandas scikit-learn · data in ml/data/raw/ · dataset source
Training · Python
- 1
Load reviews with text
Reads the clothing reviews ARFF file, keeps reviews that have text and makes "would not recommend" the 0/1 target to predict.
ml/train.py:2606 · 18 lines
import base64 from sklearn.decomposition import NMF from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS, CountVectorizer, TfidfVectorizer from sklearn.metrics import f1_score from sklearn.naive_bayes import ComplementNB from sklearn.svm import LinearSVC print("feedback-themes") raw = _read_arff(RAW / "clothing" / "reviews.arff") d = raw[raw["Review_Text"].notna()].copy() d["rating"] = d["Rating"].astype(int) d["rec"] = d["Recommended_IND"].astype(int) d["neg"] = 1 - d["rec"] d["dept"] = d["Department_Name"].fillna("Unknown") d["body"] = d["Review_Text"].str.lower() d["full"] = (d["Title"].fillna("") + " " + d["Review_Text"]).str.lower() print(f" reviews with text: {len(d)} of {len(raw)}") - 2
Find themes with NMF
NMF on TF-IDF of review text is tried with 6, 8, 10 and 12 themes, and the count whose top words have the best NPMI coherence is kept.
ml/train.py:2626 · 31 lines
# ---------- themes: NMF on TF-IDF of the review body ---------- stop = sorted(ENGLISH_STOP_WORDS | {"dress", "dresses", "top", "shirt", "love", "loved", "really", "just", "like", "wear", "wearing", "worn", "bought", "ordered", "order", "retailer", "im", "ive", "did", "got", "usually", "think", "lbs", "petite"}) theme_tok = r"(?a)\b[a-z][a-z]+\b" tv = TfidfVectorizer(stop_words=stop, token_pattern=theme_tok, min_df=20, max_df=0.5, max_features=2000) X = tv.fit_transform(d["body"]) vocab = tv.get_feature_names_out() B = (X > 0).astype(float).tocsc() N = B.shape[0] dfreq = np.asarray(B.sum(axis=0)).ravel() def npmi(top): vals = [] for i in range(len(top)): for j in range(i + 1, len(top)): co = B[:, top[i]].multiply(B[:, top[j]]).sum() vals.append(-1.0 if co == 0 else np.log(co * N / (dfreq[top[i]] * dfreq[top[j]])) / -np.log(co / N)) - 3
Compress and profile the themes
Theme weights are stored as uint8 with the browser's update rule reproduced, and each theme gets top words, share, average rating and an example review.
ml/train.py:2659 · 49 lines
# browser copy: uint8 per theme, and the exact transform the browser runs (multiplicative updates) wscale = W.max(axis=1) / 255 Wq8 = np.round(W / wscale[:, None]).astype(np.uint8) Wq = Wq8.astype(float) * wscale[:, None] WWt = Wq @ Wq.T tcount = CountVectorizer(vocabulary=list(vocab), token_pattern=theme_tok) t_idf = np.round(tv.idf_, 5) def theme_mix(texts, iters=200): M = tcount.transform([t.lower() for t in texts]).astype(float).multiply(t_idf[None, :]).tocsr() out = [] for i in range(M.shape[0]): row = M.getrow(i) nrm = np.sqrt(row.multiply(row).sum()) if nrm == 0: out.append(np.zeros(K)) - 4
Split and tune the classifier
Reviews are split 80/20, stratified on the target, and the logistic C is chosen from 1, 3 and 10 by 3-fold cross-validated ROC AUC on training data.
ml/train.py:2710 · 13 lines
# ---------- "would not recommend": words → probability ---------- y = d["neg"].to_numpy() idx_tr, idx_te = train_test_split(np.arange(len(d)), test_size=0.2, stratify=y, random_state=SEED) Xtr, Xte, ytr, yte = d["full"].iloc[idx_tr], d["full"].iloc[idx_te], y[idx_tr], y[idx_te] word_tok = r"(?a)\b\w\w+\b" def wv(**kw): return TfidfVectorizer(ngram_range=(1, 2), token_pattern=word_tok, min_df=3, sublinear_tf=True, **kw) cv3 = StratifiedKFold(3, shuffle=True, random_state=SEED) cv_scores = {C: float(cross_val_score(Pipeline([("v", wv()), ("m", LogisticRegression(C=C, max_iter=3000))]), Xtr, ytr, cv=cv3, scoring="roc_auc").mean()) for C in [1, 3, 10]} C = max(cv_scores, key=cv_scores.get) print(f" sentiment CV AUC {cv_scores} -> C={C}") - 5
Compare four classifiers
Naive Bayes, linear SVM and logistic regression on words, plus logistic on the theme mix alone, are scored on the test set by AUC, average precision and F1.
ml/train.py:2725 · 19 lines
from sklearn.metrics import average_precision_score def met(score, pred): return {"roc_auc": r(roc_auc_score(yte, score), 4), "ap": r(average_precision_score(yte, score), 4), "f1": r(f1_score(yte, pred), 4), "acc": r(np.mean(pred == yte), 4)} comparison = [] nb = Pipeline([("v", TfidfVectorizer(ngram_range=(1, 2), token_pattern=word_tok, min_df=3)), ("m", ComplementNB())]).fit(Xtr, ytr) comparison.append({"model": "Naive Bayes · words", **met(nb.predict_proba(Xte)[:, 1], nb.predict(Xte))}) svm = Pipeline([("v", wv()), ("m", LinearSVC(C=0.5))]).fit(Xtr, ytr) comparison.append({"model": "Linear SVM · words", **met(svm.decision_function(Xte), svm.predict(Xte))}) lr = Pipeline([("v", wv()), ("m", LogisticRegression(C=C, max_iter=3000))]).fit(Xtr, ytr) p_lr = lr.predict_proba(Xte)[:, 1] comparison.append({"model": "Logistic · words", **met(p_lr, (p_lr >= 0.5).astype(int))}) th = LogisticRegression(max_iter=2000).fit(share[idx_tr], ytr) p_th = th.predict_proba(share[idx_te])[:, 1] comparison.append({"model": "Logistic · theme mix only", **met(p_th, (p_th >= 0.5).astype(int))}) for c_ in comparison: print(f" {c_['model']:28s} AUC={c_['roc_auc']:.4f} AP={c_['ap']:.4f} F1={c_['f1']:.4f}") chosen = "Logistic · words" - 6
Ship a compact int8 model
The 2500 strongest n-grams are refit and stored as int8 weights, re-scored exactly as the browser computes, plus flag rates per threshold and the strongest words.
ml/train.py:2746 · 36 lines
# ship: strongest n-grams, refit, int8 weights, evaluated exactly as the browser computes KS = 2500 coef = lr.named_steps["m"].coef_[0] sel = np.argsort(-np.abs(coef))[:KS] s_vocab = [str(v) for v in lr.named_steps["v"].get_feature_names_out()[np.sort(sel)]] lr2 = Pipeline([("v", wv(vocabulary=s_vocab)), ("m", LogisticRegression(C=C, max_iter=3000))]).fit(Xtr, ytr) s_idf = np.round(lr2.named_steps["v"].idf_, 5) w2 = lr2.named_steps["m"].coef_[0] sscale = float(f"{np.abs(w2).max() / 127:.8g}") sq = np.round(w2 / sscale).astype(np.int8) sb = r(lr2.named_steps["m"].intercept_[0], 5) scount = CountVectorizer(ngram_range=(1, 2), token_pattern=word_tok, vocabulary=s_vocab) def ship_prob(texts): M = scount.transform([t.lower() for t in texts]).astype(float).tocsr() M.data = 1 + np.log(M.data) - 7
Pick demo reviews and export
Eight random test reviews plus the two most confident mistakes get scores and theme mixes, and everything is written to feedback-themes.json.
ml/train.py:2784 · 65 lines
# samples from the test split: random reviews plus the two most confident mistakes rng = np.random.default_rng(SEED) pick = [int(i) for i in rng.choice(len(idx_te), size=8, replace=False)] wrong = np.where(((p_ship >= 0.5).astype(int)) != yte)[0] worst = [int(i) for i in wrong[np.argsort(-np.abs(p_ship[wrong] - 0.5))][:2]] pick += [i for i in worst if i not in pick] bodies = [str(d["Review_Text"].iloc[idx_te[i]]) for i in pick] mixes = theme_mix(bodies) samples = [] for j, i in enumerate(pick): row = d.iloc[idx_te[i]] h = mixes[j] samples.append( { "id": f"R-{int(row['Unnamed:_0']) + 1:05d}", "title": None if not isinstance(row["Title"], str) else str(row["Title"]),
In your browser · TypeScript
- 8
NMF themes in the browser
Solves for the review's theme mix with the same multiplicative updates as scikit-learn's NMF transform, against the exported theme matrix.
src/lib/feedback.ts:18 · 38 lines
/** Theme weights for one text: TF-IDF, then non-negative least squares by multiplicative updates against fixed themes. */ export function createThemer(m: ThemeModel) { const index = new Map(m.vocab.map((t, i) => [t, i])); const V = m.vocab.length; const K = m.scale.length; const raw = decode(m.w, false); const W = Array.from({ length: K }, (_, k) => Float64Array.from({ length: V }, (_, j) => raw[k * V + j] * m.scale[k])); const WWt = W.map((a) => W.map((b) => a.reduce((s, v, j) => s + v * b[j], 0))); return function mix(text: string) { const counts = new Map<number, number>(); for (const t of text.toLowerCase().match(/\b[a-z][a-z]+\b/g) ?? []) { const j = index.get(t); if (j !== undefined) counts.set(j, (counts.get(j) ?? 0) + 1); } let norm = 0; - 9
The tone model in the browser
Word TF-IDF and a logistic regression, from the exported vocabulary and weights.
src/lib/feedback.ts:59 · 42 lines
/** Probability the reviewer would NOT recommend, plus each word's push (positive = towards "not recommend"). */ export function createSentiment(m: SentimentModel) { const index = new Map(m.vocab.map((t, i) => [t, i])); const q = decode(m.q, true); return function score(text: string) { const lower = text.toLowerCase(); const toks = [...lower.matchAll(/\b\w\w+\b/g)].map((mt) => ({ t: mt[0], start: mt.index!, end: mt.index! + mt[0].length })); const grams: { j: number; owners: number[] }[] = []; toks.forEach((tk, i) => { const j = index.get(tk.t); if (j !== undefined) grams.push({ j, owners: [i] }); if (i + 1 < toks.length) { const j2 = index.get(`${tk.t} ${toks[i + 1].t}`); if (j2 !== undefined) grams.push({ j: j2, owners: [i, i + 1] }); }
Notes
The decisions behind it.
Approach
- 1
Turned each review into TF-IDF word weights, leaving out stop words and words like dress or love that say little on their own.
- 2
Factorised the reviews into themes with NMF; compared 6 to 12 themes by coherence and diversity and kept 10.
- 3
Read every theme against stars and the recommend box, overall and per department.
- 4
Trained a tone model on the recommend box (held-out 20% test), shrank it to 2,500 phrases with 8-bit weights, and re-tested that version.
Tone model: ROC-AUC · average precision (test split)
Honest note
A topic model finds word patterns, not meaning: the theme names are my reading of them, and "Fabric & cut letdowns" bundles fabric and fit complaints together. The recommend box is noisy (some 4–5★ reviewers ticked "would not recommend"), which caps how good any tone model can look. One anonymised retailer, English only; the reviewer's age is in the data but left out.
Reproduce: ml/train.py → feedback()
dataset sourceNext project
Telco Customer Churn