←BaldanAll skillsLet's talk
Machine Learning lab

NLP

Reading a customer's message and sending it to the right team, in the browser, typos included.

96.4%

Right team

90.5%

Exact intent (of 77)

97.2%

Right intent in top 3

88%

Auto-routed at 60%

Overview

Every support inbox has the same first job: read the message, decide who should handle it. Done by hand, it is slow and inconsistent; done badly by a bot, tickets bounce between teams.

This rebuild learns that job from 13,083 labelled bank customer messages. It reads letter chunks rather than whole words, picks one of 77 intents, and maps it to one of 8 teams. Below a confidence you choose, it hands the message to a person instead of guessing.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Raw data

    13,083 messages

    77 intents, fixed public test split

  2. 02

    Teams

    8 desks

    Intents grouped by who would handle them

  3. 03

    Features

    char n-grams

    2–5 letters inside each word, TF-IDF weighted

  4. 04

    Tune

    C = 30

    3-fold CV on the training set only

  5. 05

    Compare

    4 models

    Naive Bayes, linear SVM, logistic (words, chars)

  6. 06

    Ship

    2,373 n-grams

    8-bit weights → 90.5% exact in the browser

Inside the model

77 neurons read a message.

Text becomes letter pieces, pieces become weighted numbers, and one small neuron per intent votes. Scroll through a real customer message.

Message
Am I able to exchange currencies?

A real message from the test set. The model has never seen it.

Pieces of 2 to 5 letters, inside each word

␣currencies?␣
␣aamm␣␣amam␣␣am␣␣ii␣␣i␣abbllee␣␣abablblele␣␣ablableble␣␣ableable␣␣ttoo␣␣toto␣␣to␣␣eexxcchhaanngge␣exexcxchchahanangngege␣␣excexchxchachanhangangenge␣␣exchexchaxchanchanghangeange␣␣ccuurrrreenncciieess??␣␣cucururrrrerenencncicieieses?s?␣␣curcurrurrerrenrencencincieciesies?es?␣

101 pieces; 82 of them are in the model's vocabulary of 2,373 (faded ones are not). Typos still share most pieces with the right word.

Each piece's weight (TF-IDF): rare pieces count more

es?es?␣ncierenciencienciesencinciciescierrencrencenc␣curr␣currrenurrenurrecurrecurrurrable␣␣cucurableablcurenxchanxch

77 neurons, one per intent

Exchange via app
93%
Exchange charge
5%
Fiat currency support ✓
2%
Wrong exchange rate for cash withdrawal
0%

The person who labelled it said: Fiat currency support

Every intent belongs to one team

Cards & delivery
Card security
Payments & disputes
Cash & ATM
Transfers
Top-ups
Exchange & currency · right team
Account & identity

Which words pushed towards “Exchange via app”

amiabletoexchangecurrencies?

■ towards this intent · ■ away from it

The data

13,083 customer messages, 77 intents.

Short questions to an online bank, a median of 10 words each. I grouped the 77 intents into the 8 teams a support desk might have.

BANKING77 (PolyAI)

Released by PolyAI with the paper “Efficient Intent Detection with Dual Sentence Encoders” (Casanueva et al., 2020). Each message carries one of 77 fine-grained intents, from a card that hasn't arrived to a top-up that was reverted. The authors fixed the test set at 40 messages per intent, so every number here is on data the model never saw. Licence: CC BY 4.0.

10,003

training messages

3,080

test messages

77

intents

8

teams (my grouping)

Dataset (PolyAI, GitHub)

Column dictionary

● used by the model

ColumnTypeWhat it means
texttextThe customer's message, as typed
categorytargetOne of 77 intents, e.g. card_arrival or top_up_reverted: what we predict
teamaddedWhich of 8 support teams handles that intent: my grouping, shown in full below

Explore the data

Cards, top-ups and a lot of “why”.

Training messages per team

Payments & disputes is the busiest desk. Exchange & currency and Account & identity get the least.

  • Cards & delivery · 13 intents1,539
  • Card security · 11 intents1,024
  • Payments & disputes · 11 intents1,849
  • Cash & ATM · 7 intents1,083
  • Transfers · 11 intents1,581
  • Top-ups · 11 intents1,413
  • Exchange & currency · 6 intents735
  • Account & identity · 7 intents779
Words per message

Most messages are 5 to 12 words: about as much as a chat bubble. 8% run past 25 words.

0250050002041–435785–89–1213–1617–2425+
How 77 intents become 8 teams

Each card lists the words that are most typical of that team compared with all the others, then the intents it owns.

Cards & delivery

virtualdisposableexpirevisaexpiresreactivate

Activate my card · Card arrival · Card delivery estimate · Card linking · Card about to expire · Get physical card · Order physical card · Getting spare card · Getting virtual card · Get disposable virtual card · Disposable card limits · Virtual card not working · Visa or Mastercard

Card security

appleunblockresetcontactlessgooglewatch

Lost or stolen card · Compromised card · Card not working · Contactless not working · Card swallowed · PIN blocked · Change PIN · Passcode forgotten · Lost or stolen phone · Card acceptance · Apple Pay or Google Pay

Payments & disputes

refundtwicerefundeddirectitemcontacted

Card payment fee charged · Card payment not recognised · Card payment wrong exchange rate · Declined card payment · Pending card payment · Reverted card payment · Transaction charged twice · Extra charge on statement · Direct debit payment not recognised · Request refund · Refund not showing up

Cash & ATM

withdrawalwithdrawwithdrewwithdrawingwithdrawals20

ATM support · Cash withdrawal charge · Cash withdrawal not recognised · Declined cash withdrawal · Pending cash withdrawal · Wrong amount of cash received · Wrong exchange rate for cash withdrawal

Transfers

beneficiarysalaryrecipientlandlordtransferredreceiver

Beneficiary not allowed · Cancel transfer · Declined transfer · Failed transfer · Pending transfer · Transfer fee charged · Transfer into account · Transfer not received by recipient · Transfer timing · Receiving money · Balance not updated after bank transfer

Top-ups

chequeautodisappearedtoppedautomaticallydeposited

Automatic top up · Pending top up · Top up by bank transfer charge · Top up by card charge · Top up by cash or cheque · Top up failed · Top up limits · Top up reverted · Topping up by card · Verify top up · Balance not updated after cheque or cash deposit

Exchange & currency

fiatexchangingcurrenciesaudeurexchanges

Exchange charge · Exchange rate · Exchange via app · Fiat currency support · Supported cards and currencies · Country support

Account & identity

identityagepersonalprovedeleteedit

Age limit · Edit personal details · Terminate account · Unable to verify identity · Verify my identity · Verify source of funds · Why verify identity

Examples per intent (training set)

Uneven: from 35 (contactless not working) to 187 (card payment fee charged). The test set is perfectly even at 40 each.

02550330–5960–8990–11925120–149150–189

Try it

Type a message, watch it get routed.

The model reads letter chunks (2 to 5 characters inside each word), so typos and word endings still count. Click a real test message or write your own.

Real rows from the dataset

click a row to try it in the demo ↓

MessageLabelledModel saysSureResult
Am I able to exchange currencies?Fiat currency supportExchange via app93%Right team
What do I need to do to change my PIN?Change PINChange PIN99%Exact
I have a top up that didn't go through. Why?Top up failedTop up failed86%Exact
Can I find the origination of my funds?Verify source of fundsVerify source of funds99%Exact
how can i top up?Topping up by cardTop up limits81%Right team
Can I transfer money to my credit card?Topping up by cardTopping up by card99%Exact
How man currencies can I hold?Fiat currency supportFiat currency support100%Exact
How do I find the verification code for my top-up …Verify top upVerify top up100%Exact
My transfer is pending.Balance not updated after bank transferPending transfer100%Right team
how many transactions can i make with a disposable…Get disposable virtual cardDisposable card limits100%Right team

8 random test messages plus the 2 mistakes the model was most sure about. Both still land with the right team, and both labels are arguable.

Live demo

runs in your browser

What the model read

I ordered my card two weeks ago and it still hasn't arrived

points to “Card arrival” points elsewhere126/169 letter chunks known to the model

Route to

Cards & delivery

Intent: Card arrival · 99% sure

Top 3 intents

Card arrival · Cards & delivery99.5%
Card delivery estimate · Cards & delivery0.2%
Reverted card payment · Payments & disputes0.1%

Auto-routed to Cards & delivery.

On the test set at this setting, 88% of messages are auto-routed and 99.0% of those reach the right team.

How well it works

90% exact intent, 96.4% to the right team.

On 3,080 test messages, choosing between 77 intents. Always guessing one intent would score 1.3%. 63% of the intent mistakes still reach the right team.

Four models, one test set

Character n-grams beat words (91.0% vs 89.1%). Shrinking the model to 2,373 n-grams with 8-bit weights for the browser costs 0.6 points.

Exact intentRight team
0%50%100%79%89%NB89%95%SVM89%96%LR words91%97%LR chars90%96%Shipped
Automate the sure ones, ask a human for the rest

At 60% confidence, 88% of messages are routed automatically and 99.0% of those reach the right team. The rest go to a person.

Share auto-routedRight team, of thoseExact intent, of those
50%75%100%0%25%50%75%95%Confidence needed to auto-route
More labelled messages, better routing

With 1,003 labelled examples the model is right 76% of the time; all 10,003 get it to 91%. Still climbing: more labels would help.

70%83%95%02.5k5.0k7.5k10.0kLabelled training messages
The intents it mixes up

Almost all near-misses are siblings: a virtual card that doesn't work vs getting one, a failed top-up vs a reverted one.

  • Virtual card not working → Get disposable virtual cardsame team8×
  • Top up failed → Top up revertedsame team6×
  • Balance not updated after bank transfer → Transfer not received by recipientsame team5×
  • Verify my identity → Why verify identitysame team5×
  • Card arrival → Card delivery estimatesame team4×
  • Fiat currency support → Exchange via appsame team4×
  • Top up reverted → Top up failedsame team4×
Where team routing goes wrong

Dark cells: share of each team's messages routed correctly. Orange: messages sent to another team.

Belongs to ↓ · sent to →12345678
1Cards & delivery96%114212
2Card security895%73211
3Payments & disputes396%62114
4Cash & ATM1596%23
5Transfers19195%81
6Top-ups213897%
7Exchange & currency112298%
8Account & identity100%
All models

Regularisation was chosen by 3-fold cross-validation on the training set (C = 30); the test set was only used for these final numbers.

ModelExact intentMacro-F1Right intent in top 3Right teamFeatures
Naive Bayes · words79.1%0.78693.3%88.5%10,292
Linear SVM · words89.1%0.89196.7%95.4%10,292
Logistic · words89.1%0.89197.4%95.5%10,292
Logistic · character n-grams91.0%0.91097.4%96.6%14,118
Shipped: char n-grams, 8-bit90.5%0.90497.2%96.4%2,373

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

complaints() from ml/train.py + 3 helpers (r, save, B77_TEAMS)

pip install numpy pandas scikit-learn scipy · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Load BANKING77 and map teams

    Loads the train and test CSVs, checks all 77 intents map to a team in B77_TEAMS, and sets up word and character n-gram TF-IDF helpers.

    ml/train.py:1876 · 29 lines

    import base64
    
    from scipy import sparse
    from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer
    from sklearn.metrics import f1_score
    from sklearn.naive_bayes import ComplementNB
    from sklearn.svm import LinearSVC
    
    print("complaint-router")
    tr = pd.read_csv(RAW / "banking77" / "train.csv")
    te = pd.read_csv(RAW / "banking77" / "test.csv")
    team_names = list(B77_TEAMS)
    team_of = {i: t for t, intents in B77_TEAMS.items() for i in intents}
    assert set(team_of) == set(tr.category) == set(te.category) and len(team_of) == 77
    X, y, Xt, yt = tr.text, tr.category.to_numpy(), te.text, te.category.to_numpy()
    yt_team = np.array([team_of[i] for i in yt])
  2. 2

    Tune regularisation on training data

    C is chosen from 3, 10 and 30 by 3-fold cross-validation of character n-gram logistic regression on the training set only, never on the test set.

    ml/train.py:1907 · 7 lines

    # regularisation picked on the training data only (3-fold CV), never on the test set
    cv3 = StratifiedKFold(3, shuffle=True, random_state=SEED)
    cv_scores = {}
    for C in [3, 10, 30]:
        cv_scores[C] = float(cross_val_score(Pipeline([("v", char()), ("m", LogisticRegression(C=C, max_iter=3000))]), X, y, cv=cv3).mean())
        print(f"  CV C={C}: {cv_scores[C]:.4f}")
    C = max(cv_scores, key=cv_scores.get)
  3. 3

    Compare four text classifiers

    Naive Bayes, linear SVM and logistic regression on words, plus logistic on character 2–5 grams, scored on test by accuracy, macro F1, top-3 and team accuracy.

    ml/train.py:1916 · 27 lines

    def scores_of(m, texts):
        return m.predict_proba(texts) if hasattr(m, "predict_proba") else m.decision_function(texts)
    
    def metrics(classes, S, truth):
        cls = np.asarray(classes)
        top = np.argsort(-S, axis=1)
        pred = cls[top[:, 0]]
        top3 = np.mean([truth[i] in cls[top[i, :3]] for i in range(len(truth))])
        return pred, {
            "acc": r(np.mean(pred == truth), 4),
            "macro_f1": r(f1_score(truth, pred, average="macro"), 4),
            "top3": r(top3, 4),
            "team_acc": r(np.mean(np.array([team_of[p] for p in pred]) == np.array([team_of[t] for t in truth])), 4),
        }
    
    models = {
  4. 4

    Shrink the model to int8

    Keeps the 60 strongest n-grams per intent, refits on them, stores the weights as int8 and re-scores the test set exactly as the browser will compute it.

    ml/train.py:1944 · 32 lines

    chosen = "Logistic · character n-grams"
    full = models[chosen]
    
    # what ships: the strongest n-grams per intent, refit, weights stored as int8 per intent
    K = 60
    sel = sorted(set(np.argsort(-np.abs(full.named_steps["m"].coef_), axis=1)[:, :K].ravel()))
    names = [str(s) for s in full.named_steps["v"].get_feature_names_out()[sel]]
    v2 = char(vocabulary=names)
    lr2 = LogisticRegression(C=C, max_iter=3000).fit(v2.fit_transform(X), y)
    classes = [str(c) for c in lr2.classes_]
    idf = np.round(v2.idf_, 5)
    scale = np.array([float(f"{s:.8g}") for s in np.abs(lr2.coef_).max(axis=1) / 127])
    q = np.round(lr2.coef_ / scale[:, None]).astype(np.int8)
    b = np.round(lr2.intercept_, 5)
    Wq = q.astype(float) * scale[:, None]
    counter = CountVectorizer(analyzer="char_wb", ngram_range=(2, 5), vocabulary=names)
  5. 5

    Check confidence and mistakes

    Measures accuracy against coverage at each confidence threshold, the 10 most common mix-ups, team-level confusion and a learning curve on 10% to 100% of the training data.

    ml/train.py:1977 · 27 lines

    pred_team = np.array([team_of[p] for p in pred])
    conf = P.max(axis=1)
    
    confidence = []
    for t in np.round(np.arange(0, 0.951, 0.05), 2):
        m_ = conf >= t
        confidence.append(
            {
                "t": float(t),
                "coverage": r(m_.mean(), 4),
                "acc": r((pred[m_] == yt[m_]).mean() if m_.any() else 1, 4),
                "team_acc": r((pred_team[m_] == yt_team[m_]).mean() if m_.any() else 1, 4),
            }
        )
    
    pairs = pd.Series([(a, p) for a, p in zip(yt, pred) if a != p]).value_counts().head(10)
  6. 6

    Pick demo queries

    Eight random test queries plus the two most confident mistakes become presets, each with its top 3 intents and whether the team was right.

    ml/train.py:2006 · 21 lines

    # sample test queries: a random draw plus the two most confident mistakes
    rng = np.random.default_rng(SEED)
    pick = [int(i) for i in rng.choice(len(te), size=8, replace=False)]
    wrong = np.where(pred != yt)[0]
    worst = [int(i) for i in wrong[np.argsort(-conf[wrong])][:2]]
    pick += [i for i in worst if i not in pick]
    top3 = np.argsort(-P, axis=1)[:, :3]
    samples = [
        {
            "id": f"Q-{i + 1:04d}",
            "text": str(Xt.iloc[i]),
            "actual": classes.index(yt[i]),
            "predicted": classes.index(pred[i]),
            "p": r(conf[i], 4),
            "top3": [[int(j), r(P[i, j], 4)] for j in top3[i]],
            "correct": bool(pred[i] == yt[i]),
  7. 7

    Export the JSON

    Word-count and per-intent histograms, the most telling words per team and the int8 model are written to complaint-router.json.

    ml/train.py:2029 · 60 lines

    # EDA
    wc = tr.text.str.split().str.len()
    word_bins = [(1, 5, "1–4"), (5, 9, "5–8"), (9, 13, "9–12"), (13, 17, "13–16"), (17, 25, "17–24"), (25, 10**4, "25+")]
    per_intent = tr.category.value_counts()
    cv_words = CountVectorizer(stop_words="english", min_df=10)
    Wc = cv_words.fit_transform(tr.text)
    vocab_w = cv_words.get_feature_names_out()
    tr_team = tr.category.map(team_of).to_numpy()
    telling = {}
    for t in team_names:
        a = np.asarray(Wc[tr_team == t].sum(axis=0)).ravel() + 1
        bb = np.asarray(Wc[tr_team != t].sum(axis=0)).ravel() + 1
        s = np.log(a / a.sum()) - np.log(bb / bb.sum())
        telling[t] = [str(vocab_w[j]) for j in np.argsort(-s)[:6]]
    
    save(

In your browser · TypeScript

  1. 8

    Character n-grams, as scikit-learn makes them

    The same char_wb n-grams as TfidfVectorizer: each word padded with spaces and cut into overlapping pieces, so typos still share most pieces.

    src/lib/text-router.ts:36 · 18 lines

    /** scikit-learn's char_wb analyzer: lowercase, split on whitespace, pad each word with spaces, n-grams inside the word. */
    export function charWbNgrams(text: string, min: number, max: number): { word: string; grams: string[] }[] {
      const words = text.toLowerCase().replace(/\s\s+/g, " ").split(/\s+/).filter(Boolean);
      return words.map((w) => {
        const c = Array.from(` ${w} `);
        const grams: string[] = [];
        for (let n = min; n <= max; n++) {
          let offset = 0;
          grams.push(c.slice(0, n).join(""));
          while (offset + n < c.length) {
            offset++;
            grams.push(c.slice(offset, offset + n).join(""));
          }
          if (offset === 0) break; // a short word is counted once
        }
        return { word: w, grams };
      });
    }
  2. 9

    TF-IDF and logistic regression in the browser

    Looks up each n-gram's IDF, normalises the vector and multiplies by the int8 weights of every team.

    src/lib/text-router.ts:55 · 54 lines

    export function createRouter(m: RouterModel) {
      const index = new Map(m.vocab.map((g, i) => [g, i]));
      const q = decodeInt8(m.q);
      const V = m.vocab.length;
      const K = m.classes.length;
      const weight = (k: number, j: number) => q[k * V + j] * m.scale[k];
    
      return function predict(text: string, explain?: number): RouterOutput {
        const words = charWbNgrams(text, m.ngram[0], m.ngram[1]);
        const counts = new Map<number, number>();
        let grams = 0;
        for (const w of words)
          for (const g of w.grams) {
            grams++;
            const j = index.get(g);
            if (j !== undefined) counts.set(j, (counts.get(j) ?? 0) + 1);

Notes

The decisions behind it.

Approach

  1. 1

    Kept the authors' fixed split: 10,003 messages to learn from, 3,080 (40 per intent) to test on.

  2. 2

    Grouped the 77 intents into 8 support teams and scored both the exact intent and the team.

  3. 3

    Compared word and character n-gram features with Naive Bayes, a linear SVM and logistic regression; tuned only on the training set.

  4. 4

    Kept the strongest n-grams per intent, refit, stored the weights as 8-bit integers, and re-tested exactly that version.

Exact intent · right team (test set)

Naive Bayes · words79.1% · 88.5%
Linear SVM · words89.1% · 95.4%
Logistic · words89.1% · 95.5%
Logistic · character n-grams91.0% · 96.6%best
Char n-grams, 2,373 features, 8-bit90.5% · 96.4%shipped

Honest note

These are short, tidy single questions, cleaner than real complaint emails, which are longer and mix several problems; real tickets would need their own labels. The 8 teams are my grouping, not part of the dataset, and some test labels are debatable: the model's most confident "mistakes" are arguably right. Neural sentence encoders in the original paper score a few points higher than this model's 91.0%, but they need a server; this one runs in your browser.

Reproduce: ml/train.py → complaints()

dataset source

Next project

Uplift Promo Targeting