←BaldanAll skillsLet's talk
Machine Learning lab

NLP

Reading 22,000 reviews at once: what customers talk about, and what the unhappy ones talk about.

10

Themes found

0.956

Tone model ROC-AUC

33%

Fabric & cut letdowns in 1–2★ reviews

22,641

Reviews analysed

Overview

Star ratings say how a customer felt, not why. The why is in the text, and nobody reads twenty thousand reviews. This is the problem a feedback tool has to solve: turn open text into a handful of themes a team can act on.

This rebuild finds 10 themes in clothing reviews with a topic model that is never told what to look for, links them to stars and the recommend box, and adds a tone model that flags unhappy reviews. Both run in your browser on any text you type.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Raw data

    23,486 reviews

    22,641 with text

  2. 02

    Words

    TF-IDF

    2,000 words, stop words removed

  3. 03

    Themes

    NMF, k = 10

    Number of themes chosen by coherence

  4. 04

    Link

    themes × stars

    Which themes fill the 1–2★ reviews

  5. 05

    Tone

    AUC 0.956

    Logistic model on words and phrases

  6. 06

    Ship

    8-bit weights

    Themes and tone computed in the browser

Inside the model

Themes nobody tagged.

One model finds what reviews talk about, another reads their tone. Scroll through a real review.

Review

A real review

Cute dress but not for me. the waist is too high and the sleeves are tight. maybe on a different body type this dress would be perfect. i returned it.

10 themes found in 22,641 reviews, no labels given

Fabric & cut letdowns

fabric · look · fit · nice · didn

Quality & versatility

great · looks · quality · fits · fall

True to size

size · true · fits · fit · usual

Runs small or large

small · large · medium · runs · big

Sweaters & warmth

sweater · soft · warm · sleeves · long

Comfort & compliments

comfortable · cute · super · soft · flattering

Jeans & pants

jeans · pants · pair · skinny · fit

Length & season

perfect · length · summer · fall · fit

Skirts & waist

skirt · waist · hips · pencil · skirts

Colour

color · beautiful · blue · green · red

This review as a mix of themes

■ Length & season44%
■ Comfort & compliments25%
■ Fabric & cut letdowns20%
■ Skirts & waist9%

A separate neuron for tone

Cute dress. Cute dress but not for me. the waist is too high and the sleeves are tight. maybe on a different body type this dress would be perfect. i returned it.

71%

chance this reviewer would not recommend it · ■ towards not · ■ towards recommend

Every theme: share of reviews · how many would not recommend

■ Fabric & cut letdowns · 18%
40%
■ Runs small or large · 10%
21%
■ Sweaters & warmth · 6%
21%
■ Colour · 13%
17%
■ Skirts & waist · 5%
15%
■ True to size · 10%
14%
■ Comfort & compliments · 12%
12%
■ Jeans & pants · 8%
11%
■ Quality & versatility · 9%
10%
■ Length & season · 9%
5%

The data

23,486 clothing reviews, written by real customers.

22,641 have text, a median of 59 words. Nobody labelled the topics: the themes below were found by the model.

Women's E-Commerce Clothing Reviews

Anonymised reviews from a women's clothing e-commerce shop, with the shop's name replaced by "retailer" in the text. Each review has a star rating and whether the customer would recommend the item. The reviewer's age is in the data but is not used or shown here. Licence: CC0: Public Domain.

23,486

reviews

22,641

with text

59

median words

18.1%

would not recommend

Dataset (OpenML)

Column dictionary

● used by the model

ColumnTypeWhat it means
Title · Review TexttextWhat the customer wrote; themes come from the text, the tone model reads both
Rating1–5Stars given; used to read the themes, not to find them
Recommended IND0 / 1Would recommend the item: the target of the tone model
Division · Department · ClasscategoryProduct grouping; used to compare themes across departments
Positive Feedback CountnumberOther shoppers who found the review helpful · not used
Clothing IDidWhich item · not used or shown
AgenumberReviewer's age · left out (personal attribute)

Explore the data

Mostly five stars. The useful part is the rest.

Stars given

Most reviews are glowing. Three stars is a coin flip: 59% of 3★ reviewers would not recommend the item.

010000200008211 ★2 ★3 ★4 ★125405 ★
Review length

A median of 59 words: enough to say what fits, what doesn't and why.

05000100001–1920–3940–5960–79545680–991736100+
Reviews per department

Tops and dresses dominate. Average ratings barely differ (4.14 to 4.28), so stars alone don't say what to fix.

0100002000010048TopsDressesBottomsIntimate1002Jackets

Try it

Paste a review, see its themes and tone.

Themes come from an unsupervised topic model; the tone comes from a separate classifier trained on the recommend box. Both run in your browser.

Real rows from the dataset

click a row to try it in the demo ↓

ReviewStarsDepartmentMain themeUnhappyRecommended
Cute dress but not for me. the waist is too high an…3★DressesLength & season71%Yes
Bought this blouse in white. fits great. can be worn…5★TopsQuality & versatility0%Yes
I bought this dress online last week (in large part …5★DressesFabric & cut letdowns45%Yes
I really love this dress, just not on me! i am very …4★DressesFabric & cut letdowns25%Yes
I ordered a small. i'm usually a petite small. i lik…5★IntimateRuns small or large3%Yes
If you're expecting a sweatshirt when you order this…4★TopsJeans & pants2%Yes
This dress is amazing. bought in orange and fits won…5★DressesQuality & versatility0%Yes
These pants feel great. fit is true to size. the blu…5★BottomsTrue to size0%Yes
I tried this top on with a pair of white shorts, hal…4★TopsJeans & pants0%No
I love these leggings! the material is a bit thicker…5★BottomsJeans & pants0%No

8 random reviews from the test split plus the 2 the tone model got most confidently wrong: both read as happy, yet the reviewer ticked 'would not recommend'.

Live demo

runs in your browser

What decided the tone

Beautiful color but the fabric is thin and see-through. Sending it back.

towards “wouldn't recommend” towards “would recommend”

Tone

Sounds unhappy

92% chance this reviewer would not recommend the item

Themes in this review

  • Colour96%
  • Fabric & cut letdowns4%

Theme words found: beautiful, color, fabric, sending

What it found

10 themes, and the one behind the bad reviews.

“Fabric & cut letdowns” takes up 33% of what low-rated reviews say, against 15% in high-rated ones.

The themes

Names are mine, read off each theme's top words and its most typical review. Bigger words weigh more in the theme.

Fabric & cut letdowns

fabriclookfitnicedidnmaterialgoodtried

18% of all text3.51★ average40% wouldn't recommend

Quality & versatility

greatlooksqualityfitsfallfitsummerpiece

9% of all text4.46★ average10% wouldn't recommend

True to size

sizetruefitsfitusualperfectlysmallerregular

10% of all text4.26★ average14% wouldn't recommend

Runs small or large

smalllargemediumrunsbigrunlittlenormally

10% of all text3.97★ average21% wouldn't recommend

Sweaters & warmth

sweatersoftwarmsleeveslongcozycoatitchy

6% of all text4.14★ average21% wouldn't recommend

Comfort & compliments

comfortablecutesupersoftflatteringcomfymaterialcompliments

12% of all text4.38★ average12% wouldn't recommend

Jeans & pants

jeanspantspairskinnyfitblackleggingslook

8% of all text4.42★ average11% wouldn't recommend

Length & season

perfectlengthsummerfallfitspringlongcolors

9% of all text4.69★ average5% wouldn't recommend

Skirts & waist

skirtwaisthipspencilskirtsworkflatteringknee

5% of all text4.26★ average15% wouldn't recommend

Colour

colorbeautifulbluegreenredpinkpersonpurchased

13% of all text4.23★ average17% wouldn't recommend
What unhappy and happy reviews talk about

Fabric & cut letdowns more than doubles in low-rated reviews; Length & season is the mark of the happiest ones (4.69★ on average).

1–2★ reviews4–5★ reviews
  • Fabric & cut letdowns33% / 15%
  • Runs small or large12% / 9%
  • Sweaters & warmth6% / 5%
  • Skirts & waist4% / 5%
  • Colour12% / 13%
  • True to size9% / 11%
  • Jeans & pants6% / 9%
  • Comfort & compliments8% / 13%
  • Quality & versatility6% / 10%
  • Length & season4% / 11%
Themes by department

Fabric & cut letdowns lead in tops, dresses and jackets; bottoms are about jeans and waists; intimates are mostly about comfort.

TopsDressesBottomsIntimateJackets
Fabric & cut letdowns19%22%12%17%16%
Quality & versatility9%9%8%10%10%
True to size9%12%11%10%9%
Runs small or large11%10%7%12%13%
Sweaters & warmth9%2%1%4%8%
Comfort & compliments12%12%11%18%10%
Jeans & pants7%3%20%8%6%
Length & season8%11%9%9%11%
Skirts & waist2%5%13%2%2%
Colour14%14%8%10%14%
How many themes?

Topic coherence (NPMI: do a theme's top words really appear together?) peaks at 10; at 12 the themes start repeating each other.

Coherence (NPMI)Diversity (unique top words)
0.500.751.00681012Number of themes
Flagging unhappy reviews for follow-up

Flag reviews above 50%: 16% of reviews get flagged, 81% of those really wouldn't recommend, and 73% of all unhappy reviewers are caught.

PrecisionRecallShare flagged
0%50%100%10%30%50%70%90%Flag when the model is at least this sure
Words that move the tone

Single words straight from the shipped model's weights; “not” is among the strongest warning signs. The model also uses two-word phrases, not shown here.

Towards “wouldn't recommend”

disappointedpoorunflatteringcheapnotawfulhorriblemehreturnedhugeshapelessdisappointment

Towards “would recommend”

lovegreatperfectuniquebeautifulamazingfunnicecutecomfortablewithcomfy
All tone models

Words carry the signal: the theme mix alone reaches ROC-AUC 0.78, the words model 0.96. Regularisation picked by 3-fold CV on the training split.

ModelROC-AUCAvg precisionF1Accuracy
Naive Bayes · words0.9440.7890.47086.9%
Linear SVM · words0.9580.8280.75091.6%
Logistic · words0.9590.8240.73291.3%
Logistic · theme mix only0.7820.4410.27182.5%
Shipped: 2,500 n-grams, 8-bit0.9560.8230.76792.0%

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

feedback() from ml/train.py + 4 helpers (r, save, _read_arff, FEEDBACK_THEME_NAMES)

pip install numpy pandas scikit-learn · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Load reviews with text

    Reads the clothing reviews ARFF file, keeps reviews that have text and makes "would not recommend" the 0/1 target to predict.

    ml/train.py:2606 · 18 lines

    import base64
    
    from sklearn.decomposition import NMF
    from sklearn.feature_extraction.text import ENGLISH_STOP_WORDS, CountVectorizer, TfidfVectorizer
    from sklearn.metrics import f1_score
    from sklearn.naive_bayes import ComplementNB
    from sklearn.svm import LinearSVC
    
    print("feedback-themes")
    raw = _read_arff(RAW / "clothing" / "reviews.arff")
    d = raw[raw["Review_Text"].notna()].copy()
    d["rating"] = d["Rating"].astype(int)
    d["rec"] = d["Recommended_IND"].astype(int)
    d["neg"] = 1 - d["rec"]
    d["dept"] = d["Department_Name"].fillna("Unknown")
    d["body"] = d["Review_Text"].str.lower()
    d["full"] = (d["Title"].fillna("") + " " + d["Review_Text"]).str.lower()
    print(f"  reviews with text: {len(d)} of {len(raw)}")
  2. 2

    Find themes with NMF

    NMF on TF-IDF of review text is tried with 6, 8, 10 and 12 themes, and the count whose top words have the best NPMI coherence is kept.

    ml/train.py:2626 · 31 lines

    # ---------- themes: NMF on TF-IDF of the review body ----------
    stop = sorted(ENGLISH_STOP_WORDS | {"dress", "dresses", "top", "shirt", "love", "loved", "really", "just", "like", "wear", "wearing", "worn", "bought", "ordered", "order", "retailer", "im", "ive", "did", "got", "usually", "think", "lbs", "petite"})
    theme_tok = r"(?a)\b[a-z][a-z]+\b"
    tv = TfidfVectorizer(stop_words=stop, token_pattern=theme_tok, min_df=20, max_df=0.5, max_features=2000)
    X = tv.fit_transform(d["body"])
    vocab = tv.get_feature_names_out()
    B = (X > 0).astype(float).tocsc()
    N = B.shape[0]
    dfreq = np.asarray(B.sum(axis=0)).ravel()
    
    def npmi(top):
        vals = []
        for i in range(len(top)):
            for j in range(i + 1, len(top)):
                co = B[:, top[i]].multiply(B[:, top[j]]).sum()
                vals.append(-1.0 if co == 0 else np.log(co * N / (dfreq[top[i]] * dfreq[top[j]])) / -np.log(co / N))
  3. 3

    Compress and profile the themes

    Theme weights are stored as uint8 with the browser's update rule reproduced, and each theme gets top words, share, average rating and an example review.

    ml/train.py:2659 · 49 lines

    # browser copy: uint8 per theme, and the exact transform the browser runs (multiplicative updates)
    wscale = W.max(axis=1) / 255
    Wq8 = np.round(W / wscale[:, None]).astype(np.uint8)
    Wq = Wq8.astype(float) * wscale[:, None]
    WWt = Wq @ Wq.T
    tcount = CountVectorizer(vocabulary=list(vocab), token_pattern=theme_tok)
    t_idf = np.round(tv.idf_, 5)
    
    def theme_mix(texts, iters=200):
        M = tcount.transform([t.lower() for t in texts]).astype(float).multiply(t_idf[None, :]).tocsr()
        out = []
        for i in range(M.shape[0]):
            row = M.getrow(i)
            nrm = np.sqrt(row.multiply(row).sum())
            if nrm == 0:
                out.append(np.zeros(K))
  4. 4

    Split and tune the classifier

    Reviews are split 80/20, stratified on the target, and the logistic C is chosen from 1, 3 and 10 by 3-fold cross-validated ROC AUC on training data.

    ml/train.py:2710 · 13 lines

    # ---------- "would not recommend": words → probability ----------
    y = d["neg"].to_numpy()
    idx_tr, idx_te = train_test_split(np.arange(len(d)), test_size=0.2, stratify=y, random_state=SEED)
    Xtr, Xte, ytr, yte = d["full"].iloc[idx_tr], d["full"].iloc[idx_te], y[idx_tr], y[idx_te]
    word_tok = r"(?a)\b\w\w+\b"
    
    def wv(**kw):
        return TfidfVectorizer(ngram_range=(1, 2), token_pattern=word_tok, min_df=3, sublinear_tf=True, **kw)
    
    cv3 = StratifiedKFold(3, shuffle=True, random_state=SEED)
    cv_scores = {C: float(cross_val_score(Pipeline([("v", wv()), ("m", LogisticRegression(C=C, max_iter=3000))]), Xtr, ytr, cv=cv3, scoring="roc_auc").mean()) for C in [1, 3, 10]}
    C = max(cv_scores, key=cv_scores.get)
    print(f"  sentiment CV AUC {cv_scores} -> C={C}")
  5. 5

    Compare four classifiers

    Naive Bayes, linear SVM and logistic regression on words, plus logistic on the theme mix alone, are scored on the test set by AUC, average precision and F1.

    ml/train.py:2725 · 19 lines

    from sklearn.metrics import average_precision_score
    
    def met(score, pred):
        return {"roc_auc": r(roc_auc_score(yte, score), 4), "ap": r(average_precision_score(yte, score), 4), "f1": r(f1_score(yte, pred), 4), "acc": r(np.mean(pred == yte), 4)}
    
    comparison = []
    nb = Pipeline([("v", TfidfVectorizer(ngram_range=(1, 2), token_pattern=word_tok, min_df=3)), ("m", ComplementNB())]).fit(Xtr, ytr)
    comparison.append({"model": "Naive Bayes · words", **met(nb.predict_proba(Xte)[:, 1], nb.predict(Xte))})
    svm = Pipeline([("v", wv()), ("m", LinearSVC(C=0.5))]).fit(Xtr, ytr)
    comparison.append({"model": "Linear SVM · words", **met(svm.decision_function(Xte), svm.predict(Xte))})
    lr = Pipeline([("v", wv()), ("m", LogisticRegression(C=C, max_iter=3000))]).fit(Xtr, ytr)
    p_lr = lr.predict_proba(Xte)[:, 1]
    comparison.append({"model": "Logistic · words", **met(p_lr, (p_lr >= 0.5).astype(int))})
    th = LogisticRegression(max_iter=2000).fit(share[idx_tr], ytr)
    p_th = th.predict_proba(share[idx_te])[:, 1]
    comparison.append({"model": "Logistic · theme mix only", **met(p_th, (p_th >= 0.5).astype(int))})
    for c_ in comparison:
        print(f"  {c_['model']:28s} AUC={c_['roc_auc']:.4f} AP={c_['ap']:.4f} F1={c_['f1']:.4f}")
    chosen = "Logistic · words"
  6. 6

    Ship a compact int8 model

    The 2500 strongest n-grams are refit and stored as int8 weights, re-scored exactly as the browser computes, plus flag rates per threshold and the strongest words.

    ml/train.py:2746 · 36 lines

    # ship: strongest n-grams, refit, int8 weights, evaluated exactly as the browser computes
    KS = 2500
    coef = lr.named_steps["m"].coef_[0]
    sel = np.argsort(-np.abs(coef))[:KS]
    s_vocab = [str(v) for v in lr.named_steps["v"].get_feature_names_out()[np.sort(sel)]]
    lr2 = Pipeline([("v", wv(vocabulary=s_vocab)), ("m", LogisticRegression(C=C, max_iter=3000))]).fit(Xtr, ytr)
    s_idf = np.round(lr2.named_steps["v"].idf_, 5)
    w2 = lr2.named_steps["m"].coef_[0]
    sscale = float(f"{np.abs(w2).max() / 127:.8g}")
    sq = np.round(w2 / sscale).astype(np.int8)
    sb = r(lr2.named_steps["m"].intercept_[0], 5)
    scount = CountVectorizer(ngram_range=(1, 2), token_pattern=word_tok, vocabulary=s_vocab)
    
    def ship_prob(texts):
        M = scount.transform([t.lower() for t in texts]).astype(float).tocsr()
        M.data = 1 + np.log(M.data)
  7. 7

    Pick demo reviews and export

    Eight random test reviews plus the two most confident mistakes get scores and theme mixes, and everything is written to feedback-themes.json.

    ml/train.py:2784 · 65 lines

    # samples from the test split: random reviews plus the two most confident mistakes
    rng = np.random.default_rng(SEED)
    pick = [int(i) for i in rng.choice(len(idx_te), size=8, replace=False)]
    wrong = np.where(((p_ship >= 0.5).astype(int)) != yte)[0]
    worst = [int(i) for i in wrong[np.argsort(-np.abs(p_ship[wrong] - 0.5))][:2]]
    pick += [i for i in worst if i not in pick]
    bodies = [str(d["Review_Text"].iloc[idx_te[i]]) for i in pick]
    mixes = theme_mix(bodies)
    samples = []
    for j, i in enumerate(pick):
        row = d.iloc[idx_te[i]]
        h = mixes[j]
        samples.append(
            {
                "id": f"R-{int(row['Unnamed:_0']) + 1:05d}",
                "title": None if not isinstance(row["Title"], str) else str(row["Title"]),

In your browser · TypeScript

  1. 8

    NMF themes in the browser

    Solves for the review's theme mix with the same multiplicative updates as scikit-learn's NMF transform, against the exported theme matrix.

    src/lib/feedback.ts:18 · 38 lines

    /** Theme weights for one text: TF-IDF, then non-negative least squares by multiplicative updates against fixed themes. */
    export function createThemer(m: ThemeModel) {
      const index = new Map(m.vocab.map((t, i) => [t, i]));
      const V = m.vocab.length;
      const K = m.scale.length;
      const raw = decode(m.w, false);
      const W = Array.from({ length: K }, (_, k) => Float64Array.from({ length: V }, (_, j) => raw[k * V + j] * m.scale[k]));
      const WWt = W.map((a) => W.map((b) => a.reduce((s, v, j) => s + v * b[j], 0)));
    
      return function mix(text: string) {
        const counts = new Map<number, number>();
        for (const t of text.toLowerCase().match(/\b[a-z][a-z]+\b/g) ?? []) {
          const j = index.get(t);
          if (j !== undefined) counts.set(j, (counts.get(j) ?? 0) + 1);
        }
        let norm = 0;
  2. 9

    The tone model in the browser

    Word TF-IDF and a logistic regression, from the exported vocabulary and weights.

    src/lib/feedback.ts:59 · 42 lines

    /** Probability the reviewer would NOT recommend, plus each word's push (positive = towards "not recommend"). */
    export function createSentiment(m: SentimentModel) {
      const index = new Map(m.vocab.map((t, i) => [t, i]));
      const q = decode(m.q, true);
    
      return function score(text: string) {
        const lower = text.toLowerCase();
        const toks = [...lower.matchAll(/\b\w\w+\b/g)].map((mt) => ({ t: mt[0], start: mt.index!, end: mt.index! + mt[0].length }));
        const grams: { j: number; owners: number[] }[] = [];
        toks.forEach((tk, i) => {
          const j = index.get(tk.t);
          if (j !== undefined) grams.push({ j, owners: [i] });
          if (i + 1 < toks.length) {
            const j2 = index.get(`${tk.t} ${toks[i + 1].t}`);
            if (j2 !== undefined) grams.push({ j: j2, owners: [i, i + 1] });
          }

Notes

The decisions behind it.

Approach

  1. 1

    Turned each review into TF-IDF word weights, leaving out stop words and words like dress or love that say little on their own.

  2. 2

    Factorised the reviews into themes with NMF; compared 6 to 12 themes by coherence and diversity and kept 10.

  3. 3

    Read every theme against stars and the recommend box, overall and per department.

  4. 4

    Trained a tone model on the recommend box (held-out 20% test), shrank it to 2,500 phrases with 8-bit weights, and re-tested that version.

Tone model: ROC-AUC · average precision (test split)

Naive Bayes · words0.944 · 0.789
Linear SVM · words0.958 · 0.828
Logistic · words0.959 · 0.824best
Logistic · theme mix only0.782 · 0.441
Shipped: 2,500 n-grams, 8-bit0.956 · 0.823shipped

Honest note

A topic model finds word patterns, not meaning: the theme names are my reading of them, and "Fabric & cut letdowns" bundles fabric and fit complaints together. The recommend box is noisy (some 4–5★ reviewers ticked "would not recommend"), which caps how good any tone model can look. One anonymised retailer, English only; the reviewer's age is in the data but left out.

Reproduce: ml/train.py → feedback()

dataset source

Next project

Telco Customer Churn