NLP
Reading a customer's message and sending it to the right team, in the browser, typos included.
96.4%
Right team
90.5%
Exact intent (of 77)
97.2%
Right intent in top 3
88%
Auto-routed at 60%
Overview
Every support inbox has the same first job: read the message, decide who should handle it. Done by hand, it is slow and inconsistent; done badly by a bot, tickets bounce between teams.
This rebuild learns that job from 13,083 labelled bank customer messages. It reads letter chunks rather than whole words, picks one of 77 intents, and maps it to one of 8 teams. Below a confidence you choose, it hands the message to a person instead of guessing.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Raw data
13,083 messages
77 intents, fixed public test split
- 02
Teams
8 desks
Intents grouped by who would handle them
- 03
Features
char n-grams
2–5 letters inside each word, TF-IDF weighted
- 04
Tune
C = 30
3-fold CV on the training set only
- 05
Compare
4 models
Naive Bayes, linear SVM, logistic (words, chars)
- 06
Ship
2,373 n-grams
8-bit weights → 90.5% exact in the browser
Inside the model
77 neurons read a message.
Text becomes letter pieces, pieces become weighted numbers, and one small neuron per intent votes. Scroll through a real customer message.
A real message from the test set. The model has never seen it.
Pieces of 2 to 5 letters, inside each word
101 pieces; 82 of them are in the model's vocabulary of 2,373 (faded ones are not). Typos still share most pieces with the right word.
Each piece's weight (TF-IDF): rare pieces count more
77 neurons, one per intent
The person who labelled it said: Fiat currency support
Every intent belongs to one team
Which words pushed towards “Exchange via app”
■ towards this intent · ■ away from it
The data
13,083 customer messages, 77 intents.
Short questions to an online bank, a median of 10 words each. I grouped the 77 intents into the 8 teams a support desk might have.
BANKING77 (PolyAI)
Released by PolyAI with the paper “Efficient Intent Detection with Dual Sentence Encoders” (Casanueva et al., 2020). Each message carries one of 77 fine-grained intents, from a card that hasn't arrived to a top-up that was reverted. The authors fixed the test set at 40 messages per intent, so every number here is on data the model never saw. Licence: CC BY 4.0.
10,003
training messages
3,080
test messages
77
intents
8
teams (my grouping)
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| text | text | The customer's message, as typed |
| category | target | One of 77 intents, e.g. card_arrival or top_up_reverted: what we predict |
| team | added | Which of 8 support teams handles that intent: my grouping, shown in full below |
Explore the data
Cards, top-ups and a lot of “why”.
Payments & disputes is the busiest desk. Exchange & currency and Account & identity get the least.
- Cards & delivery · 13 intents1,539
- Card security · 11 intents1,024
- Payments & disputes · 11 intents1,849
- Cash & ATM · 7 intents1,083
- Transfers · 11 intents1,581
- Top-ups · 11 intents1,413
- Exchange & currency · 6 intents735
- Account & identity · 7 intents779
Most messages are 5 to 12 words: about as much as a chat bubble. 8% run past 25 words.
Each card lists the words that are most typical of that team compared with all the others, then the intents it owns.
Cards & delivery
Activate my card · Card arrival · Card delivery estimate · Card linking · Card about to expire · Get physical card · Order physical card · Getting spare card · Getting virtual card · Get disposable virtual card · Disposable card limits · Virtual card not working · Visa or Mastercard
Card security
Lost or stolen card · Compromised card · Card not working · Contactless not working · Card swallowed · PIN blocked · Change PIN · Passcode forgotten · Lost or stolen phone · Card acceptance · Apple Pay or Google Pay
Payments & disputes
Card payment fee charged · Card payment not recognised · Card payment wrong exchange rate · Declined card payment · Pending card payment · Reverted card payment · Transaction charged twice · Extra charge on statement · Direct debit payment not recognised · Request refund · Refund not showing up
Cash & ATM
ATM support · Cash withdrawal charge · Cash withdrawal not recognised · Declined cash withdrawal · Pending cash withdrawal · Wrong amount of cash received · Wrong exchange rate for cash withdrawal
Transfers
Beneficiary not allowed · Cancel transfer · Declined transfer · Failed transfer · Pending transfer · Transfer fee charged · Transfer into account · Transfer not received by recipient · Transfer timing · Receiving money · Balance not updated after bank transfer
Top-ups
Automatic top up · Pending top up · Top up by bank transfer charge · Top up by card charge · Top up by cash or cheque · Top up failed · Top up limits · Top up reverted · Topping up by card · Verify top up · Balance not updated after cheque or cash deposit
Exchange & currency
Exchange charge · Exchange rate · Exchange via app · Fiat currency support · Supported cards and currencies · Country support
Account & identity
Age limit · Edit personal details · Terminate account · Unable to verify identity · Verify my identity · Verify source of funds · Why verify identity
Uneven: from 35 (contactless not working) to 187 (card payment fee charged). The test set is perfectly even at 40 each.
Try it
Type a message, watch it get routed.
The model reads letter chunks (2 to 5 characters inside each word), so typos and word endings still count. Click a real test message or write your own.
Real rows from the dataset
click a row to try it in the demo ↓
| Message | Labelled | Model says | Sure | Result |
|---|---|---|---|---|
| Am I able to exchange currencies? | Fiat currency support | Exchange via app | 93% | Right team |
| What do I need to do to change my PIN? | Change PIN | Change PIN | 99% | Exact |
| I have a top up that didn't go through. Why? | Top up failed | Top up failed | 86% | Exact |
| Can I find the origination of my funds? | Verify source of funds | Verify source of funds | 99% | Exact |
| how can i top up? | Topping up by card | Top up limits | 81% | Right team |
| Can I transfer money to my credit card? | Topping up by card | Topping up by card | 99% | Exact |
| How man currencies can I hold? | Fiat currency support | Fiat currency support | 100% | Exact |
| How do I find the verification code for my top-up … | Verify top up | Verify top up | 100% | Exact |
| My transfer is pending. | Balance not updated after bank transfer | Pending transfer | 100% | Right team |
| how many transactions can i make with a disposable… | Get disposable virtual card | Disposable card limits | 100% | Right team |
8 random test messages plus the 2 mistakes the model was most sure about. Both still land with the right team, and both labels are arguable.
Live demo
runs in your browser
What the model read
I ordered my card two weeks ago and it still hasn't arrived
Route to
Intent: Card arrival · 99% sure
Top 3 intents
Auto-routed to Cards & delivery.
On the test set at this setting, 88% of messages are auto-routed and 99.0% of those reach the right team.
How well it works
90% exact intent, 96.4% to the right team.
On 3,080 test messages, choosing between 77 intents. Always guessing one intent would score 1.3%. 63% of the intent mistakes still reach the right team.
Character n-grams beat words (91.0% vs 89.1%). Shrinking the model to 2,373 n-grams with 8-bit weights for the browser costs 0.6 points.
At 60% confidence, 88% of messages are routed automatically and 99.0% of those reach the right team. The rest go to a person.
With 1,003 labelled examples the model is right 76% of the time; all 10,003 get it to 91%. Still climbing: more labels would help.
Almost all near-misses are siblings: a virtual card that doesn't work vs getting one, a failed top-up vs a reverted one.
- Virtual card not working → Get disposable virtual cardsame team8×
- Top up failed → Top up revertedsame team6×
- Balance not updated after bank transfer → Transfer not received by recipientsame team5×
- Verify my identity → Why verify identitysame team5×
- Card arrival → Card delivery estimatesame team4×
- Fiat currency support → Exchange via appsame team4×
- Top up reverted → Top up failedsame team4×
Dark cells: share of each team's messages routed correctly. Orange: messages sent to another team.
| Belongs to ↓ · sent to → | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 |
|---|---|---|---|---|---|---|---|---|
| 1Cards & delivery | 96% | 11 | 4 | 2 | 1 | 2 | ||
| 2Card security | 8 | 95% | 7 | 3 | 2 | 1 | 1 | |
| 3Payments & disputes | 3 | 96% | 6 | 2 | 1 | 1 | 4 | |
| 4Cash & ATM | 1 | 5 | 96% | 2 | 3 | |||
| 5Transfers | 1 | 9 | 1 | 95% | 8 | 1 | ||
| 6Top-ups | 2 | 1 | 3 | 8 | 97% | |||
| 7Exchange & currency | 1 | 1 | 2 | 2 | 98% | |||
| 8Account & identity | 100% |
Regularisation was chosen by 3-fold cross-validation on the training set (C = 30); the test set was only used for these final numbers.
| Model | Exact intent | Macro-F1 | Right intent in top 3 | Right team | Features |
|---|---|---|---|---|---|
| Naive Bayes · words | 79.1% | 0.786 | 93.3% | 88.5% | 10,292 |
| Linear SVM · words | 89.1% | 0.891 | 96.7% | 95.4% | 10,292 |
| Logistic · words | 89.1% | 0.891 | 97.4% | 95.5% | 10,292 |
| Logistic · character n-grams | 91.0% | 0.910 | 97.4% | 96.6% | 14,118 |
| Shipped: char n-grams, 8-bit | 90.5% | 0.904 | 97.2% | 96.4% | 2,373 |
Build notes
How it's built, step by step.
The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.
Run it yourself
complaints() from ml/train.py + 3 helpers (r, save, B77_TEAMS)
pip install numpy pandas scikit-learn scipy · data in ml/data/raw/ · dataset source
Training · Python
- 1
Load BANKING77 and map teams
Loads the train and test CSVs, checks all 77 intents map to a team in B77_TEAMS, and sets up word and character n-gram TF-IDF helpers.
ml/train.py:1876 · 29 lines
import base64 from scipy import sparse from sklearn.feature_extraction.text import CountVectorizer, TfidfVectorizer from sklearn.metrics import f1_score from sklearn.naive_bayes import ComplementNB from sklearn.svm import LinearSVC print("complaint-router") tr = pd.read_csv(RAW / "banking77" / "train.csv") te = pd.read_csv(RAW / "banking77" / "test.csv") team_names = list(B77_TEAMS) team_of = {i: t for t, intents in B77_TEAMS.items() for i in intents} assert set(team_of) == set(tr.category) == set(te.category) and len(team_of) == 77 X, y, Xt, yt = tr.text, tr.category.to_numpy(), te.text, te.category.to_numpy() yt_team = np.array([team_of[i] for i in yt]) - 2
Tune regularisation on training data
C is chosen from 3, 10 and 30 by 3-fold cross-validation of character n-gram logistic regression on the training set only, never on the test set.
ml/train.py:1907 · 7 lines
# regularisation picked on the training data only (3-fold CV), never on the test set cv3 = StratifiedKFold(3, shuffle=True, random_state=SEED) cv_scores = {} for C in [3, 10, 30]: cv_scores[C] = float(cross_val_score(Pipeline([("v", char()), ("m", LogisticRegression(C=C, max_iter=3000))]), X, y, cv=cv3).mean()) print(f" CV C={C}: {cv_scores[C]:.4f}") C = max(cv_scores, key=cv_scores.get) - 3
Compare four text classifiers
Naive Bayes, linear SVM and logistic regression on words, plus logistic on character 2–5 grams, scored on test by accuracy, macro F1, top-3 and team accuracy.
ml/train.py:1916 · 27 lines
def scores_of(m, texts): return m.predict_proba(texts) if hasattr(m, "predict_proba") else m.decision_function(texts) def metrics(classes, S, truth): cls = np.asarray(classes) top = np.argsort(-S, axis=1) pred = cls[top[:, 0]] top3 = np.mean([truth[i] in cls[top[i, :3]] for i in range(len(truth))]) return pred, { "acc": r(np.mean(pred == truth), 4), "macro_f1": r(f1_score(truth, pred, average="macro"), 4), "top3": r(top3, 4), "team_acc": r(np.mean(np.array([team_of[p] for p in pred]) == np.array([team_of[t] for t in truth])), 4), } models = { - 4
Shrink the model to int8
Keeps the 60 strongest n-grams per intent, refits on them, stores the weights as int8 and re-scores the test set exactly as the browser will compute it.
ml/train.py:1944 · 32 lines
chosen = "Logistic · character n-grams" full = models[chosen] # what ships: the strongest n-grams per intent, refit, weights stored as int8 per intent K = 60 sel = sorted(set(np.argsort(-np.abs(full.named_steps["m"].coef_), axis=1)[:, :K].ravel())) names = [str(s) for s in full.named_steps["v"].get_feature_names_out()[sel]] v2 = char(vocabulary=names) lr2 = LogisticRegression(C=C, max_iter=3000).fit(v2.fit_transform(X), y) classes = [str(c) for c in lr2.classes_] idf = np.round(v2.idf_, 5) scale = np.array([float(f"{s:.8g}") for s in np.abs(lr2.coef_).max(axis=1) / 127]) q = np.round(lr2.coef_ / scale[:, None]).astype(np.int8) b = np.round(lr2.intercept_, 5) Wq = q.astype(float) * scale[:, None] counter = CountVectorizer(analyzer="char_wb", ngram_range=(2, 5), vocabulary=names) - 5
Check confidence and mistakes
Measures accuracy against coverage at each confidence threshold, the 10 most common mix-ups, team-level confusion and a learning curve on 10% to 100% of the training data.
ml/train.py:1977 · 27 lines
pred_team = np.array([team_of[p] for p in pred]) conf = P.max(axis=1) confidence = [] for t in np.round(np.arange(0, 0.951, 0.05), 2): m_ = conf >= t confidence.append( { "t": float(t), "coverage": r(m_.mean(), 4), "acc": r((pred[m_] == yt[m_]).mean() if m_.any() else 1, 4), "team_acc": r((pred_team[m_] == yt_team[m_]).mean() if m_.any() else 1, 4), } ) pairs = pd.Series([(a, p) for a, p in zip(yt, pred) if a != p]).value_counts().head(10) - 6
Pick demo queries
Eight random test queries plus the two most confident mistakes become presets, each with its top 3 intents and whether the team was right.
ml/train.py:2006 · 21 lines
# sample test queries: a random draw plus the two most confident mistakes rng = np.random.default_rng(SEED) pick = [int(i) for i in rng.choice(len(te), size=8, replace=False)] wrong = np.where(pred != yt)[0] worst = [int(i) for i in wrong[np.argsort(-conf[wrong])][:2]] pick += [i for i in worst if i not in pick] top3 = np.argsort(-P, axis=1)[:, :3] samples = [ { "id": f"Q-{i + 1:04d}", "text": str(Xt.iloc[i]), "actual": classes.index(yt[i]), "predicted": classes.index(pred[i]), "p": r(conf[i], 4), "top3": [[int(j), r(P[i, j], 4)] for j in top3[i]], "correct": bool(pred[i] == yt[i]), - 7
Export the JSON
Word-count and per-intent histograms, the most telling words per team and the int8 model are written to complaint-router.json.
ml/train.py:2029 · 60 lines
# EDA wc = tr.text.str.split().str.len() word_bins = [(1, 5, "1–4"), (5, 9, "5–8"), (9, 13, "9–12"), (13, 17, "13–16"), (17, 25, "17–24"), (25, 10**4, "25+")] per_intent = tr.category.value_counts() cv_words = CountVectorizer(stop_words="english", min_df=10) Wc = cv_words.fit_transform(tr.text) vocab_w = cv_words.get_feature_names_out() tr_team = tr.category.map(team_of).to_numpy() telling = {} for t in team_names: a = np.asarray(Wc[tr_team == t].sum(axis=0)).ravel() + 1 bb = np.asarray(Wc[tr_team != t].sum(axis=0)).ravel() + 1 s = np.log(a / a.sum()) - np.log(bb / bb.sum()) telling[t] = [str(vocab_w[j]) for j in np.argsort(-s)[:6]] save(
In your browser · TypeScript
- 8
Character n-grams, as scikit-learn makes them
The same char_wb n-grams as TfidfVectorizer: each word padded with spaces and cut into overlapping pieces, so typos still share most pieces.
src/lib/text-router.ts:36 · 18 lines
/** scikit-learn's char_wb analyzer: lowercase, split on whitespace, pad each word with spaces, n-grams inside the word. */ export function charWbNgrams(text: string, min: number, max: number): { word: string; grams: string[] }[] { const words = text.toLowerCase().replace(/\s\s+/g, " ").split(/\s+/).filter(Boolean); return words.map((w) => { const c = Array.from(` ${w} `); const grams: string[] = []; for (let n = min; n <= max; n++) { let offset = 0; grams.push(c.slice(0, n).join("")); while (offset + n < c.length) { offset++; grams.push(c.slice(offset, offset + n).join("")); } if (offset === 0) break; // a short word is counted once } return { word: w, grams }; }); } - 9
TF-IDF and logistic regression in the browser
Looks up each n-gram's IDF, normalises the vector and multiplies by the int8 weights of every team.
src/lib/text-router.ts:55 · 54 lines
export function createRouter(m: RouterModel) { const index = new Map(m.vocab.map((g, i) => [g, i])); const q = decodeInt8(m.q); const V = m.vocab.length; const K = m.classes.length; const weight = (k: number, j: number) => q[k * V + j] * m.scale[k]; return function predict(text: string, explain?: number): RouterOutput { const words = charWbNgrams(text, m.ngram[0], m.ngram[1]); const counts = new Map<number, number>(); let grams = 0; for (const w of words) for (const g of w.grams) { grams++; const j = index.get(g); if (j !== undefined) counts.set(j, (counts.get(j) ?? 0) + 1);
Notes
The decisions behind it.
Approach
- 1
Kept the authors' fixed split: 10,003 messages to learn from, 3,080 (40 per intent) to test on.
- 2
Grouped the 77 intents into 8 support teams and scored both the exact intent and the team.
- 3
Compared word and character n-gram features with Naive Bayes, a linear SVM and logistic regression; tuned only on the training set.
- 4
Kept the strongest n-grams per intent, refit, stored the weights as 8-bit integers, and re-tested exactly that version.
Exact intent · right team (test set)
Honest note
These are short, tidy single questions, cleaner than real complaint emails, which are longer and mix several problems; real tickets would need their own labels. The 8 teams are my grouping, not part of the dataset, and some test labels are debatable: the model's most confident "mistakes" are arguably right. Neural sentence encoders in the original paper score a few points higher than this model's 91.0%, but they need a server; this one runs in your browser.
Reproduce: ml/train.py → complaints()
dataset sourceNext project
Uplift Promo Targeting