NLP
Sentiment for Indonesian and English, plus 28 emotions for English, from two small BiLSTMs running in your browser.
0.754
Indonesian test macro-F1
0.632
English test macro-F1 (4 sentiments)
0.386
English emotions macro-F1
28
Emotions it can name
Overview
Reading opinions at scale is one of the most common jobs for language models: product reviews, support tickets, comments. Most portfolio versions call a large hosted model. This one trains two small recurrent networks from scratch, one per language, and runs them on your device, with a view of which words decided each answer.
The Indonesian model learned from 11,000 labelled reviews and opinions (SmSA, IndoNLU); the English one from 43,410 Reddit comments labelled with 28 emotions (GoEmotions, Google Research), which also give it a sentiment. Each network is compared with a TF-IDF + logistic regression baseline and with averaged word embeddings on test data it never saw.
How it works
From raw file to prediction.
Every number below comes from the training run.
- 01
Raw text
12,760 + 54,263
Indonesian sentences + English comments
- 02
Tokens
8,000 · 10,000 words
Same tokenizer in Python and the browser
- 03
Baselines
F1 0.74 · 0.64
TF-IDF + logistic regression, test macro-F1
- 04
BiLSTM
578,563 · 710,304
Weights; English adds 28 emotion outputs
- 05
Ship
int8
8-bit weights, re-tested with the browser's forward pass
- 06
Explain
word by word
How much each word moves the answer
Inside the model
Reading a sentence both ways.
A bidirectional LSTM, step by step, on a real sentence from the data, with the network's real weights (loaded from the same file as the demo).
Loading the network…
The data
Two languages, two kinds of text.
Indonesian reviews and opinions, and English Reddit comments labelled with 28 emotions. Each language gets its own model, trained on its own data; the test sets stay untouched until the end.
SmSA: Indonesian sentiment
12,760 Indonesian sentences from reviews and opinions, labelled positive, neutral or negative. Collected by Prosa.ai; Purwarianti & Crisdayanti (2019), published in the IndoNLU benchmark under Apache-2.0.
11,000
train
1,260
validation
500
test
8,000
words known
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| text | sentence | A review or opinion in Indonesian, already lower-cased and split into words by the dataset authors |
| label | 3 classes | positive, neutral or negative, labelled by people |
GoEmotions: English emotions
54,263 English Reddit comments, each labelled by raters with one or more of 27 emotions or neutral. Published by Google Research; Demszky et al. (2020) in a repository under Apache-2.0. Reddit language is informal and sometimes rude; rude comments are left in training but never shown as samples here.
43,410
train
5,426
validation
5,427
test
16%
with 2+ emotions
Column dictionary
● used by the model
| Column | Type | What it means |
|---|---|---|
| text | comment | A Reddit comment in English; names are replaced by [NAME] |
| emotions | 1+ of 28 | admiration, gratitude, anger, curiosity… or neutral; a comment can have several |
| sentiment | 4 classes | Derived from the emotions with Google's own mapping: positive, negative, ambiguous or neutral |
| id | text | Reddit comment id |
01
Words
The text is split into words and punctuation, exactly the same way in Python and in your browser
02
Numbers
Each known word becomes a learned vector (64–64 numbers); unknown words share one vector
03
BiLSTM
Two LSTMs read the sentence, one left to right and one right to left, so each word is seen in context
04
Strongest signals
For every feature, the strongest value anywhere in the sentence is kept
05
Answer
Sentiment probabilities; for English also 28 emotions, each with its own threshold
Try it
Write something. Watch which words decide.
Type in Indonesian or English; the page guesses the language. The colours show how much each word moved the answer: remove it, and the probability changes by that much.
Live demo
runs in your browser
Loading the two models…
About 2 MB of 8-bit weights and vocabularies, downloaded once and run on your device.
What the data looks like
Unbalanced labels, short texts, telling words.
Positive dominates (58%) and neutral is rare, so every model was trained with balanced class weights and judged by macro-F1, which counts each class equally.
"Mixed" comments carry emotions from more than one group (say, gratitude and sadness). They have no single sentiment, so they train the emotion head only: 3,380 of 43,410.
The strongest single words in the logistic regression baseline. They are what any model latches onto first; the BiLSTM can also use word order, as in 'tidak enak' or 'not bad'.
Indonesian
English
neutral is the most common label (14,219); grief the rarest (77). Rare emotions are much harder to learn.
How good is it, honestly
For sentiment, the BiLSTM is level with the classic baseline in Indonesian and level with it in English.
Three models per language, compared on test data none of them saw: a TF-IDF + logistic regression baseline, averaged word embeddings and a BiLSTM, with sizes chosen on validation data only. For the 28 English emotions the baseline, with one classifier per emotion, stays ahead (macro-F1 0.462 vs 0.385). Macro-F1 averages every class's F1, so rare classes count as much as common ones.
Shipped: BiLSTM, 77.4% accuracy, macro-F1 0.754; its 8-bit browser version agrees with full precision on 100.0% of test sentences. Every model loses about 0.13 macro-F1 from validation to test: the official test set is more balanced than training.
| Model | Accuracy | Macro-F1 | Weights |
|---|---|---|---|
| TF-IDF + logistic regression | 78.4% | 0.740 | 127,257 |
| Word embeddings, averaged | 68.4% | 0.661 | 384,243 |
| BiLSTM ✓ | 77.4% | 0.754 | 578,563 |
Shipped: BiLSTM, 65.4% accuracy over 4 sentiments, macro-F1 0.632; emotions macro-F1 0.386 over all 5,427 test comments.
| Model | Accuracy | Macro-F1 | Emotions F1 |
|---|---|---|---|
| TF-IDF + logistic regression | 66.6% | 0.642 | 0.462 |
| Word embeddings, averaged | 63.5% | 0.612 | 0.360 |
| BiLSTM ✓ | 65.4% | 0.632 | 0.385 |
Rows are the human label, columns the model's answer. Neutral is the hardest class (F1 0.68): it is rare in training and sits between the other two.
Ambiguous (surprise, curiosity, confusion, realisation) is the hardest group, F1 0.53; positive is the easiest, F1 0.77.
gratitude (0.90) and amusement (0.79) are easy: they come with telling words. relief (0.06) and nervousness (0.07) are hard: rare, subtle or both.
Two small networks trained from scratch, not a large pretrained language model.
| Here | |
|---|---|
| Models | BiLSTMs trained from scratch in TensorFlow: 578,563 weights for Indonesian, 710,304 for English; int8 weights in the browser |
| Runs on | Your browser, in TypeScript; checked against the NumPy reference sentence by sentence. Nothing you type leaves the page |
| Indonesian | Trained on reviews and opinions; slang, typos and words it never saw weaken it. Neutral is its weakest class |
| English | Trained on Reddit comments; sarcasm and long, mixed texts fool it. Emotions are English only |
| Not comparable | The two test sets are different kinds of text with different labels, so Indonesian vs English scores say nothing about the languages |
| Better models exist | Pretrained transformers (IndoBERT, RoBERTa) score higher but are hundreds of MB; these run instantly in a page |
Build notes
How it's built, step by step.
The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.
Run it yourself
sentiment() from ml/train.py + 7 helpers (r, save, SENT_TOKEN, SENT_MAXLEN, SENT_CONFIGS, SENT_RUDE, SENT_PICKS)
pip install numpy pandas scikit-learn tensorflow · data in ml/data/raw/ · dataset source
Training · Python
- 1
Tokenise and define the networks
Uses the browser's token pattern, maps words to ids (0 for padding, 1 for unknown) and defines an averaged-embedding model and a BiLSTM with padding-aware max pooling.
ml/train.py:3645 · 62 lines
import base64 import re import keras import tensorflow as tf from keras import layers from sklearn.feature_extraction.text import TfidfVectorizer from sklearn.metrics import accuracy_score, confusion_matrix, f1_score print("bilingual-sentiment") keras.utils.set_random_seed(SEED) tf.config.experimental.enable_op_determinism() TOK = re.compile(SENT_TOKEN) tok = lambda t: TOK.findall(str(t).replace("’", "'").lower())[:SENT_MAXLEN] # noqa: E731 parity: dict = {} - 2
Write the export and training helpers
Int8 export with one scale per row or column, a NumPy copy of the browser's forward pass, plus training, rare-emotion loss, threshold and top-word helpers.
ml/train.py:3709 · 88 lines
def q8_cols(w): scale = np.abs(w).max(axis=0) / 127 scale = np.where(scale == 0, 1e-12, scale) scale = np.array([float(f"{s:.8g}") for s in scale]) q = np.round(w / scale).astype(np.int8) return {"shape": list(w.shape), "scale": [float(s) for s in scale], "q": base64.b64encode(q.tobytes()).decode("ascii")}, q.astype(np.float64) * scale def export(model, kind, head_names): E = model.get_layer("emb").get_weights()[0] rs = np.abs(E).max(axis=1) / 127 rs = np.where(rs == 0, 1e-12, rs) rs = np.array([float(f"{s:.6g}") for s in rs]) Eq = np.clip(np.round(E / rs[:, None]), -127, 127).astype(np.int8) out = {"kind": kind, "emb": {"shape": list(E.shape), "scale": [float(s) for s in rs], "q": base64.b64encode(Eq.tobytes()).decode("ascii")}} deq = {"emb": Eq.astype(np.float64) * rs[:, None]} if kind == "bilstm": - 3
Prepare the Indonesian reviews
Loads the SmSA train, validation and test files, tokenises each review and keeps words seen at least twice, up to 8000, as the vocabulary.
ml/train.py:3799 · 14 lines
payload: dict = {} # ================================================================ Indonesian: SmSA (IndoNLU, Apache-2.0) rd = lambda f: pd.read_csv(RAW / "smsa" / f, sep="\t", header=None, names=["text", "label"], quoting=3) # noqa: E731 tr, va, te = rd("train_preprocess.tsv"), rd("valid_preprocess.tsv"), rd("test_preprocess.tsv") ID_CLASSES = ["positive", "neutral", "negative"] ci = {c: i for i, c in enumerate(ID_CLASSES)} ytr, yva, yte = (d.label.map(ci).to_numpy() for d in (tr, va, te)) Ttr, Tva, Tte = ([tok(t) for t in d.text] for d in (tr, va, te)) words, index = vocab_of(Ttr, 2, 8000) V = len(words) + 2 Xtr, Xva, Xte = encode(Ttr, index), encode(Tva, index), encode(Tte, index) oov = float(np.mean([index.get(t, 1) == 1 for ts in Tte for t in ts])) print(f" id: vocab {V}, test OOV {oov:.3f}") - 4
Compare three Indonesian models
TF-IDF with logistic regression, averaged embeddings and a BiLSTM, each tuned on validation macro-F1 with balanced class weights, then scored on the test reviews.
ml/train.py:3814 · 30 lines
cmp_id = [] # 1) TF-IDF + logistic regression best = None for C in [1, 3, 10]: vec = TfidfVectorizer(tokenizer=tok, lowercase=False, token_pattern=None, ngram_range=(1, 2), min_df=2, sublinear_tf=True) A = vec.fit_transform(tr.text) lr = LogisticRegression(C=C, max_iter=3000, class_weight="balanced").fit(A, ytr) f = f1_score(yva, lr.predict(vec.transform(va.text)), average="macro") if best is None or f > best[0]: best = (f, C, vec, lr) _, C_id, vec_id, lr_id = best p = lr_id.predict(vec_id.transform(te.text)) cmp_id.append({"model": "TF-IDF + logistic regression", "val_f1": r(best[0], 4), "acc": r(accuracy_score(yte, p), 4), "f1": r(f1_score(yte, p, average="macro"), 4), "params": int(lr_id.coef_.size)}) # 2) + 3) neural models trained_id = {} for kind, label in [("avg", "Word embeddings, averaged"), ("bilstm", "BiLSTM")]: - 5
Ship the Indonesian model
Keeps the neural model with the best validation F1, exports it as int8 and re-scores the test reviews with the NumPy forward pass, checking it against the float model.
ml/train.py:3845 · 29 lines
kind_id = max(trained_id, key=lambda k: trained_id[k][2]) m_id, hist_id, _ = trained_id[kind_id] exp_id, deq_id = export(m_id, kind_id, ["sent"]) seqs_te = [[index.get(t, 1) for t in ts] or [1] for ts in Tte] P_ship = np.array([forward(deq_id, kind_id, s, ["sent"])["sent"] for s in seqs_te]) p_ship = P_ship.argmax(1) agree = float(np.mean(p_ship == m_id.predict(Xte, verbose=0).argmax(1))) gap = float(np.abs(m_id.predict(Xte[:200], verbose=0) - P_ship[:200]).max()) print(f" id shipped {kind_id}: acc {accuracy_score(yte, p_ship):.4f} f1 {f1_score(yte, p_ship, average='macro'):.4f}, agrees with float {agree:.4f}, max prob gap {gap:.4f}") parity["id"] = {"seqs": seqs_te, "probs": P_ship.tolist(), "texts": te.text.tolist()[:300], "tokens": [[index.get(t, 1) for t in ts] for ts in Tte[:300]]} pick = [{"id": f"id-{i}", "text": te.text[i], "label": ID_CLASSES[yte[i]], "pred": ID_CLASSES[p_ship[i]], "probs": [r(v, 4) for v in P_ship[i]]} for i in SENT_PICKS["id"]] payload["id"] = { "dataset": {"name": "SmSA (IndoNLU)", "source": "https://github.com/IndoNLP/indonlu", "license": "Apache-2.0", "origin": "Prosa.ai; Purwarianti & Crisdayanti (2019)", "train": len(tr), "val": len(va), "test": len(te)}, "classes": ID_CLASSES, "label_counts": {d: {c: int((x == i).sum()) for i, c in enumerate(ID_CLASSES)} for d, x in [("train", ytr), ("val", yva), ("test", yte)]}, "lengths": [int(v) for v in np.percentile([len(t) for t in Ttr], [10, 25, 50, 75, 90, 95])], - 6
Prepare the English comments
GoEmotions labels map to positive, neutral, negative or ambiguous; mixed-sentiment comments train only the emotion task; the vocabulary keeps up to 10000 words.
ml/train.py:3876 · 32 lines
# ================================================================ English: GoEmotions (Google Research, Apache-2.0) emo = open(RAW / "goemotions" / "emotions.txt", encoding="utf-8").read().split() mapping = json.load(open(RAW / "goemotions" / "sentiment_mapping.json")) group = {e: k for k, v in mapping.items() for e in v} group["neutral"] = "neutral" EN_CLASSES = ["positive", "neutral", "negative", "ambiguous"] rg = lambda f: pd.read_csv(RAW / "goemotions" / f, sep="\t", header=None, names=["text", "labels", "id"], quoting=3) # noqa: E731 gtr, gva, gte = rg("train.tsv"), rg("dev.tsv"), rg("test.tsv") def targets(d): E = np.zeros((len(d), len(emo)), dtype="float32") s = np.zeros(len(d), dtype="int32") keep = np.ones(len(d), dtype=bool) for i, ls in enumerate(d.labels): ids = [int(x) for x in str(ls).split(",")] E[i, ids] = 1 - 7
Compare English models with two heads
A TF-IDF baseline adds one logistic regression per emotion; neural models get a sentiment head and an emotion head, trying two positive-weight schemes for rare emotions.
ml/train.py:3909 · 44 lines
cmp_en = [] best = None for C in [1, 3, 10]: vec = TfidfVectorizer(tokenizer=tok, lowercase=False, token_pattern=None, ngram_range=(1, 2), min_df=2, sublinear_tf=True) A = vec.fit_transform(gtr.text) lr = LogisticRegression(C=C, max_iter=3000, class_weight="balanced").fit(A[Ktr], Str[Ktr]) f = f1_score(Sva[Kva], lr.predict(vec.transform(gva.text[Kva])), average="macro") if best is None or f > best[0]: best = (f, C, vec, lr, A) _, C_en, vec_en, lr_en, A_en = best p = lr_en.predict(vec_en.transform(gte.text[Kte])) # emotions with the baseline: one logistic regression per emotion from sklearn.multiclass import OneVsRestClassifier ovr = OneVsRestClassifier(LogisticRegression(C=C_en, max_iter=3000, class_weight="balanced"), n_jobs=-1).fit(A_en, Etr.astype(int)) th_lr = best_thresholds(ovr.predict_proba(vec_en.transform(gva.text)), Eva) - 8
Ship the English model
Keeps the neural model with the best validation sentiment F1, exports it as int8, tunes each emotion's cut-off on the int8 outputs and scores the test comments.
ml/train.py:3954 · 51 lines
kind_en = max(trained_en, key=lambda k: trained_en[k][2]) m_en, hist_en, _ = trained_en[kind_en] exp_en, deq_en = export(m_en, kind_en, ["sent", "emo"]) gseq_va = [[gindex.get(t, 1) for t in ts] or [1] for ts in Gva] gseq_te = [[gindex.get(t, 1) for t in ts] or [1] for ts in Gte] Fva = [forward(deq_en, kind_en, s, ["sent", "emo"]) for s in gseq_va] Fte = [forward(deq_en, kind_en, s, ["sent", "emo"]) for s in gseq_te] Sv, Ev = np.array([f["sent"] for f in Fva]), np.array([f["emo"] for f in Fva]) St, Et = np.array([f["sent"] for f in Fte]), np.array([f["emo"] for f in Fte]) th_ship = best_thresholds(Ev, Eva) # thresholds tuned on the shipped (int8) model's validation outputs ps = St.argmax(1) emo_f1 = f1_score(Ete, Et >= th_ship, average=None, zero_division=0) agree_en = float(np.mean(ps[Kte] == m_en.predict(GXte, verbose=0)[0][Kte].argmax(1))) print(f" en shipped {kind_en}: acc {accuracy_score(Ste[Kte], ps[Kte]):.4f} f1 {f1_score(Ste[Kte], ps[Kte], average='macro'):.4f} emo macro-F1 {emo_f1.mean():.4f}, agrees with float {agree_en:.4f}") parity["en"] = {"seqs": gseq_te[:600], "sent": St[:600].tolist(), "emo": Et[:600].tolist(), "texts": gte.text.tolist()[:300], "tokens": [[gindex.get(t, 1) for t in ts] for ts in Gte[:300]]} pick_en = [ - 9
Export weights and results
Saves parity data, moves the int8 weights and vocabularies to public/models/bilingual-sentiment-weights.json so the page stays light, then writes the results JSON.
ml/train.py:4007 · 10 lines
json.dump(parity, open(RAW / "sentiment_parity.json", "w")) # weights + vocabularies (~2 MB) go to a static file the demo fetches when it opens, so the page itself stays light weights = {lang: {"model": payload[lang].pop("model"), "vocab": payload[lang].pop("vocab")} for lang in ("id", "en")} for lang in ("id", "en"): payload[lang]["vocab_size"] = len(weights[lang]["vocab"]) wp = ROOT / "public" / "models" / "bilingual-sentiment-weights.json" wp.parent.mkdir(parents=True, exist_ok=True) wp.write_text(json.dumps({**weights, "token_pattern": SENT_TOKEN, "maxlen": SENT_MAXLEN}, separators=(",", ":")), encoding="utf-8") print(f" -> {wp.relative_to(ROOT)} ({wp.stat().st_size / 1024:.1f} KB)") save("bilingual-sentiment", {**payload, "token_pattern": SENT_TOKEN, "maxlen": SENT_MAXLEN, "tf": {"tensorflow": tf.__version__, "keras": keras.__version__}})
In your browser · TypeScript
- 10
Tokenizer, BiLSTM and explanations in TypeScript
Same tokens as Python, int8 embeddings, both LSTM directions, max over words, then sentiment and emotions; explain() leaves out one word at a time.
src/lib/text-lstm.ts:41 · 94 lines
export function createTextNet(m: TextModel, vocab: string[], maxlen = 100) { const index = new Map(vocab.map((w, i) => [w, i + 2] as const)); // 0 = padding, 1 = unknown word const D = m.emb.shape[1]; const E = perRow(m.emb); const dirs = m.kind === "bilstm" ? [m.fwd!, m.bwd!].map((d) => ({ U: d.units, k: perCol(d.kernel), r: perCol(d.recurrent), b: d.bias })) : []; const head = (h: Head) => ({ w: perCol(h.w), b: h.b, n: h.w.shape[1] }); const sent = head(m.sent); const emo = m.emo ? head(m.emo) : null; const ids = (tokens: string[]) => (tokens.length ? tokens.map((t) => index.get(t) ?? 1) : [1]); function lstm(xs: Float64Array[], d: (typeof dirs)[number]): Float64Array[] { const { U, k, r, b } = d; const G = 4 * U; let h = new Float64Array(U); let c = new Float64Array(U);
Notes
The decisions behind it.
Approach
- 1
Kept each dataset's official train, validation and test split; the test sets were scored once, at the end.
- 2
Wrote one tokenizer used by both Python and TypeScript; words seen at least twice in training form each vocabulary, the rest share an 'unknown' vector.
- 3
Compared three models per language: TF-IDF + logistic regression, averaged word embeddings and a BiLSTM (two sizes, chosen on validation data), all with balanced class weights.
- 4
For English, one network with two heads: a softmax over four sentiments and 28 sigmoid emotions, each emotion with its own threshold tuned on validation data.
- 5
Shipped 8-bit weights, re-tested them with the exact forward pass the browser runs, and explained answers by leaving each word out in turn.
Test macro-F1 · Indonesian · English (sentiment) · English emotions
Honest note
With thousands of short, labelled texts, a linear model on word counts is a strong rival. The BiLSTM only edges the TF-IDF baseline on Indonesian (macro-F1 0.754 vs 0.740, at slightly lower accuracy) and does not beat it on English sentiment (0.632 vs 0.642) or on the 28 emotions (0.386 vs 0.462), where the baseline trains one classifier per emotion. Every model scores lower on the official Indonesian test set than on validation; it is more balanced than the training data. Pretrained transformers such as IndoBERT or RoBERTa score clearly higher but weigh hundreds of megabytes; these networks are small enough to run instantly in a page. The two test sets are different kinds of text, so Indonesian and English scores should not be compared with each other.
Reproduce: ml/train.py → sentiment()
dataset sourceNext project
Doodle Duel