←BaldanAll skillsLet's talk
Deep Learning & Vision lab

NLP

Sentiment for Indonesian and English, plus 28 emotions for English, from two small BiLSTMs running in your browser.

0.754

Indonesian test macro-F1

0.632

English test macro-F1 (4 sentiments)

0.386

English emotions macro-F1

28

Emotions it can name

Overview

Reading opinions at scale is one of the most common jobs for language models: product reviews, support tickets, comments. Most portfolio versions call a large hosted model. This one trains two small recurrent networks from scratch, one per language, and runs them on your device, with a view of which words decided each answer.

The Indonesian model learned from 11,000 labelled reviews and opinions (SmSA, IndoNLU); the English one from 43,410 Reddit comments labelled with 28 emotions (GoEmotions, Google Research), which also give it a sentiment. Each network is compared with a TF-IDF + logistic regression baseline and with averaged word embeddings on test data it never saw.

How it works

From raw file to prediction.

Every number below comes from the training run.

  1. 01

    Raw text

    12,760 + 54,263

    Indonesian sentences + English comments

  2. 02

    Tokens

    8,000 · 10,000 words

    Same tokenizer in Python and the browser

  3. 03

    Baselines

    F1 0.74 · 0.64

    TF-IDF + logistic regression, test macro-F1

  4. 04

    BiLSTM

    578,563 · 710,304

    Weights; English adds 28 emotion outputs

  5. 05

    Ship

    int8

    8-bit weights, re-tested with the browser's forward pass

  6. 06

    Explain

    word by word

    How much each word moves the answer

Inside the model

Reading a sentence both ways.

A bidirectional LSTM, step by step, on a real sentence from the data, with the network's real weights (loaded from the same file as the demo).

Sentence

Loading the network…

The data

Two languages, two kinds of text.

Indonesian reviews and opinions, and English Reddit comments labelled with 28 emotions. Each language gets its own model, trained on its own data; the test sets stay untouched until the end.

SmSA: Indonesian sentiment

12,760 Indonesian sentences from reviews and opinions, labelled positive, neutral or negative. Collected by Prosa.ai; Purwarianti & Crisdayanti (2019), published in the IndoNLU benchmark under Apache-2.0.

11,000

train

1,260

validation

500

test

8,000

words known

IndoNLU on GitHub

Column dictionary

● used by the model

ColumnTypeWhat it means
textsentenceA review or opinion in Indonesian, already lower-cased and split into words by the dataset authors
label3 classespositive, neutral or negative, labelled by people

GoEmotions: English emotions

54,263 English Reddit comments, each labelled by raters with one or more of 27 emotions or neutral. Published by Google Research; Demszky et al. (2020) in a repository under Apache-2.0. Reddit language is informal and sometimes rude; rude comments are left in training but never shown as samples here.

43,410

train

5,426

validation

5,427

test

16%

with 2+ emotions

GoEmotions on GitHub

Column dictionary

● used by the model

ColumnTypeWhat it means
textcommentA Reddit comment in English; names are replaced by [NAME]
emotions1+ of 28admiration, gratitude, anger, curiosity… or neutral; a comment can have several
sentiment4 classesDerived from the emotions with Google's own mapping: positive, negative, ambiguous or neutral
idtextReddit comment id

01

Words

The text is split into words and punctuation, exactly the same way in Python and in your browser

02

Numbers

Each known word becomes a learned vector (64–64 numbers); unknown words share one vector

03

BiLSTM

Two LSTMs read the sentence, one left to right and one right to left, so each word is seen in context

04

Strongest signals

For every feature, the strongest value anywhere in the sentence is kept

05

Answer

Sentiment probabilities; for English also 28 emotions, each with its own threshold

Try it

Write something. Watch which words decide.

Type in Indonesian or English; the page guesses the language. The colours show how much each word moved the answer: remove it, and the probability changes by that much.

Live demo

runs in your browser

Loading the two models…

About 2 MB of 8-bit weights and vocabularies, downloaded once and run on your device.

What the data looks like

Unbalanced labels, short texts, telling words.

Indonesian training labels

Positive dominates (58%) and neutral is rare, so every model was trained with balanced class weights and judged by macro-F1, which counts each class equally.

05,00010,0006,416Positive1,148NeutralNegative
English training labels (sentiment from emotions)

"Mixed" comments carry emotions from more than one group (say, gratitude and sadness). They have no single sentiment, so they train the emotion head only: 3,380 of 43,410.

010,00020,00015,216PositiveNeutralNegativeAmbiguous3,380Mixed
Words that point each way (from the TF-IDF baseline)

The strongest single words in the logistic regression baseline. They are what any model latches onto first; the BiLSTM can also use word order, as in 'tidak enak' or 'not bad'.

Indonesian

Positiveenaktempatbagussangatmakananbangetkebaikannyamenu
Negativesajakecewatidakkurangmacetmahalituorangmalah

English

Positivelolthankslovegreatthankgoodhopehahaglad
Negativesorrysadhatestupiddisgustingworstworsebadawful
The 28 emotions in English training data

neutral is the most common label (14,219); grief the rarest (77). Rare emotions are much harder to learn.

010,00020,00014,219neutraladmirationapprovalgratitudeannoyanceamusementcuriositylovedisapprovaloptimismangerjoyconfusionsadnessdisappointmentrealizationcaringsurpriseexcitementdisgustdesirefearremorseembarrassmentnervousnessreliefpride77grief

How good is it, honestly

For sentiment, the BiLSTM is level with the classic baseline in Indonesian and level with it in English.

Three models per language, compared on test data none of them saw: a TF-IDF + logistic regression baseline, averaged word embeddings and a BiLSTM, with sizes chosen on validation data only. For the 28 English emotions the baseline, with one classifier per emotion, stays ahead (macro-F1 0.462 vs 0.385). Macro-F1 averages every class's F1, so rare classes count as much as common ones.

Indonesian test set (500 sentences)

Shipped: BiLSTM, 77.4% accuracy, macro-F1 0.754; its 8-bit browser version agrees with full precision on 100.0% of test sentences. Every model loses about 0.13 macro-F1 from validation to test: the official test set is more balanced than training.

ModelAccuracyMacro-F1Weights
TF-IDF + logistic regression78.4%0.740127,257
Word embeddings, averaged68.4%0.661384,243
BiLSTM ✓77.4%0.754578,563
English test set (5,027 comments with one sentiment)

Shipped: BiLSTM, 65.4% accuracy over 4 sentiments, macro-F1 0.632; emotions macro-F1 0.386 over all 5,427 test comments.

ModelAccuracyMacro-F1Emotions F1
TF-IDF + logistic regression66.6%0.6420.462
Word embeddings, averaged63.5%0.6120.360
BiLSTM ✓65.4%0.6320.385
Indonesian: where it goes wrong

Rows are the human label, columns the model's answer. Neutral is the hardest class (F1 0.68): it is rare in training and sits between the other two.

PosNeuNegPositiveNeutralNegative5192
English: where it goes wrong

Ambiguous (surprise, curiosity, confusion, realisation) is the hardest group, F1 0.53; positive is the easiest, F1 0.77.

PosNeuNegAmbPositiveNeutralNegativeAmbiguous161299
English: F1 for each of the 28 emotions (shipped model)

gratitude (0.90) and amusement (0.79) are easy: they come with telling words. relief (0.06) and nervousness (0.07) are hard: rare, subtle or both.

0.000.501.000.90gratitudeamusementloveneutralremorseadmirationjoyoptimismcuriosityfearsurpriseangersadnessdisgustdesireexcitementconfusiondisapprovalapprovalcaringannoyancerealizationprideembarrassmentdisappointmentgriefnervousness0.06relief
What this project is, and isn't

Two small networks trained from scratch, not a large pretrained language model.

Here
ModelsBiLSTMs trained from scratch in TensorFlow: 578,563 weights for Indonesian, 710,304 for English; int8 weights in the browser
Runs onYour browser, in TypeScript; checked against the NumPy reference sentence by sentence. Nothing you type leaves the page
IndonesianTrained on reviews and opinions; slang, typos and words it never saw weaken it. Neutral is its weakest class
EnglishTrained on Reddit comments; sarcasm and long, mixed texts fool it. Emotions are English only
Not comparableThe two test sets are different kinds of text with different labels, so Indonesian vs English scores say nothing about the languages
Better models existPretrained transformers (IndoBERT, RoBERTa) score higher but are hundreds of MB; these run instantly in a page

Build notes

How it's built, step by step.

The real code behind this page, taken from the repository when the site is built and split at each step of the training run. Copy any step, or the whole script with its imports and helpers.

Run it yourself

sentiment() from ml/train.py + 7 helpers (r, save, SENT_TOKEN, SENT_MAXLEN, SENT_CONFIGS, SENT_RUDE, SENT_PICKS)

pip install numpy pandas scikit-learn tensorflow · data in ml/data/raw/ · dataset source

Training · Python

  1. 1

    Tokenise and define the networks

    Uses the browser's token pattern, maps words to ids (0 for padding, 1 for unknown) and defines an averaged-embedding model and a BiLSTM with padding-aware max pooling.

    ml/train.py:3645 · 62 lines

    import base64
    import re
    
    import keras
    import tensorflow as tf
    from keras import layers
    from sklearn.feature_extraction.text import TfidfVectorizer
    from sklearn.metrics import accuracy_score, confusion_matrix, f1_score
    
    print("bilingual-sentiment")
    keras.utils.set_random_seed(SEED)
    tf.config.experimental.enable_op_determinism()
    TOK = re.compile(SENT_TOKEN)
    tok = lambda t: TOK.findall(str(t).replace("’", "'").lower())[:SENT_MAXLEN]  # noqa: E731
    parity: dict = {}
    
  2. 2

    Write the export and training helpers

    Int8 export with one scale per row or column, a NumPy copy of the browser's forward pass, plus training, rare-emotion loss, threshold and top-word helpers.

    ml/train.py:3709 · 88 lines

    def q8_cols(w):
        scale = np.abs(w).max(axis=0) / 127
        scale = np.where(scale == 0, 1e-12, scale)
        scale = np.array([float(f"{s:.8g}") for s in scale])
        q = np.round(w / scale).astype(np.int8)
        return {"shape": list(w.shape), "scale": [float(s) for s in scale], "q": base64.b64encode(q.tobytes()).decode("ascii")}, q.astype(np.float64) * scale
    
    def export(model, kind, head_names):
        E = model.get_layer("emb").get_weights()[0]
        rs = np.abs(E).max(axis=1) / 127
        rs = np.where(rs == 0, 1e-12, rs)
        rs = np.array([float(f"{s:.6g}") for s in rs])
        Eq = np.clip(np.round(E / rs[:, None]), -127, 127).astype(np.int8)
        out = {"kind": kind, "emb": {"shape": list(E.shape), "scale": [float(s) for s in rs], "q": base64.b64encode(Eq.tobytes()).decode("ascii")}}
        deq = {"emb": Eq.astype(np.float64) * rs[:, None]}
        if kind == "bilstm":
  3. 3

    Prepare the Indonesian reviews

    Loads the SmSA train, validation and test files, tokenises each review and keeps words seen at least twice, up to 8000, as the vocabulary.

    ml/train.py:3799 · 14 lines

    payload: dict = {}
    
    # ================================================================ Indonesian: SmSA (IndoNLU, Apache-2.0)
    rd = lambda f: pd.read_csv(RAW / "smsa" / f, sep="\t", header=None, names=["text", "label"], quoting=3)  # noqa: E731
    tr, va, te = rd("train_preprocess.tsv"), rd("valid_preprocess.tsv"), rd("test_preprocess.tsv")
    ID_CLASSES = ["positive", "neutral", "negative"]
    ci = {c: i for i, c in enumerate(ID_CLASSES)}
    ytr, yva, yte = (d.label.map(ci).to_numpy() for d in (tr, va, te))
    Ttr, Tva, Tte = ([tok(t) for t in d.text] for d in (tr, va, te))
    words, index = vocab_of(Ttr, 2, 8000)
    V = len(words) + 2
    Xtr, Xva, Xte = encode(Ttr, index), encode(Tva, index), encode(Tte, index)
    oov = float(np.mean([index.get(t, 1) == 1 for ts in Tte for t in ts]))
    print(f"  id: vocab {V}, test OOV {oov:.3f}")
  4. 4

    Compare three Indonesian models

    TF-IDF with logistic regression, averaged embeddings and a BiLSTM, each tuned on validation macro-F1 with balanced class weights, then scored on the test reviews.

    ml/train.py:3814 · 30 lines

    cmp_id = []
    # 1) TF-IDF + logistic regression
    best = None
    for C in [1, 3, 10]:
        vec = TfidfVectorizer(tokenizer=tok, lowercase=False, token_pattern=None, ngram_range=(1, 2), min_df=2, sublinear_tf=True)
        A = vec.fit_transform(tr.text)
        lr = LogisticRegression(C=C, max_iter=3000, class_weight="balanced").fit(A, ytr)
        f = f1_score(yva, lr.predict(vec.transform(va.text)), average="macro")
        if best is None or f > best[0]:
            best = (f, C, vec, lr)
    _, C_id, vec_id, lr_id = best
    p = lr_id.predict(vec_id.transform(te.text))
    cmp_id.append({"model": "TF-IDF + logistic regression", "val_f1": r(best[0], 4), "acc": r(accuracy_score(yte, p), 4), "f1": r(f1_score(yte, p, average="macro"), 4), "params": int(lr_id.coef_.size)})
    # 2) + 3) neural models
    trained_id = {}
    for kind, label in [("avg", "Word embeddings, averaged"), ("bilstm", "BiLSTM")]:
  5. 5

    Ship the Indonesian model

    Keeps the neural model with the best validation F1, exports it as int8 and re-scores the test reviews with the NumPy forward pass, checking it against the float model.

    ml/train.py:3845 · 29 lines

    kind_id = max(trained_id, key=lambda k: trained_id[k][2])
    m_id, hist_id, _ = trained_id[kind_id]
    exp_id, deq_id = export(m_id, kind_id, ["sent"])
    seqs_te = [[index.get(t, 1) for t in ts] or [1] for ts in Tte]
    P_ship = np.array([forward(deq_id, kind_id, s, ["sent"])["sent"] for s in seqs_te])
    p_ship = P_ship.argmax(1)
    agree = float(np.mean(p_ship == m_id.predict(Xte, verbose=0).argmax(1)))
    gap = float(np.abs(m_id.predict(Xte[:200], verbose=0) - P_ship[:200]).max())
    print(f"  id shipped {kind_id}: acc {accuracy_score(yte, p_ship):.4f} f1 {f1_score(yte, p_ship, average='macro'):.4f}, agrees with float {agree:.4f}, max prob gap {gap:.4f}")
    parity["id"] = {"seqs": seqs_te, "probs": P_ship.tolist(), "texts": te.text.tolist()[:300], "tokens": [[index.get(t, 1) for t in ts] for ts in Tte[:300]]}
    pick = [{"id": f"id-{i}", "text": te.text[i], "label": ID_CLASSES[yte[i]], "pred": ID_CLASSES[p_ship[i]], "probs": [r(v, 4) for v in P_ship[i]]} for i in SENT_PICKS["id"]]
    payload["id"] = {
        "dataset": {"name": "SmSA (IndoNLU)", "source": "https://github.com/IndoNLP/indonlu", "license": "Apache-2.0", "origin": "Prosa.ai; Purwarianti & Crisdayanti (2019)", "train": len(tr), "val": len(va), "test": len(te)},
        "classes": ID_CLASSES,
        "label_counts": {d: {c: int((x == i).sum()) for i, c in enumerate(ID_CLASSES)} for d, x in [("train", ytr), ("val", yva), ("test", yte)]},
        "lengths": [int(v) for v in np.percentile([len(t) for t in Ttr], [10, 25, 50, 75, 90, 95])],
  6. 6

    Prepare the English comments

    GoEmotions labels map to positive, neutral, negative or ambiguous; mixed-sentiment comments train only the emotion task; the vocabulary keeps up to 10000 words.

    ml/train.py:3876 · 32 lines

    # ================================================================ English: GoEmotions (Google Research, Apache-2.0)
    emo = open(RAW / "goemotions" / "emotions.txt", encoding="utf-8").read().split()
    mapping = json.load(open(RAW / "goemotions" / "sentiment_mapping.json"))
    group = {e: k for k, v in mapping.items() for e in v}
    group["neutral"] = "neutral"
    EN_CLASSES = ["positive", "neutral", "negative", "ambiguous"]
    rg = lambda f: pd.read_csv(RAW / "goemotions" / f, sep="\t", header=None, names=["text", "labels", "id"], quoting=3)  # noqa: E731
    gtr, gva, gte = rg("train.tsv"), rg("dev.tsv"), rg("test.tsv")
    
    def targets(d):
        E = np.zeros((len(d), len(emo)), dtype="float32")
        s = np.zeros(len(d), dtype="int32")
        keep = np.ones(len(d), dtype=bool)
        for i, ls in enumerate(d.labels):
            ids = [int(x) for x in str(ls).split(",")]
            E[i, ids] = 1
  7. 7

    Compare English models with two heads

    A TF-IDF baseline adds one logistic regression per emotion; neural models get a sentiment head and an emotion head, trying two positive-weight schemes for rare emotions.

    ml/train.py:3909 · 44 lines

    cmp_en = []
    best = None
    for C in [1, 3, 10]:
        vec = TfidfVectorizer(tokenizer=tok, lowercase=False, token_pattern=None, ngram_range=(1, 2), min_df=2, sublinear_tf=True)
        A = vec.fit_transform(gtr.text)
        lr = LogisticRegression(C=C, max_iter=3000, class_weight="balanced").fit(A[Ktr], Str[Ktr])
        f = f1_score(Sva[Kva], lr.predict(vec.transform(gva.text[Kva])), average="macro")
        if best is None or f > best[0]:
            best = (f, C, vec, lr, A)
    _, C_en, vec_en, lr_en, A_en = best
    p = lr_en.predict(vec_en.transform(gte.text[Kte]))
    # emotions with the baseline: one logistic regression per emotion
    from sklearn.multiclass import OneVsRestClassifier
    
    ovr = OneVsRestClassifier(LogisticRegression(C=C_en, max_iter=3000, class_weight="balanced"), n_jobs=-1).fit(A_en, Etr.astype(int))
    th_lr = best_thresholds(ovr.predict_proba(vec_en.transform(gva.text)), Eva)
  8. 8

    Ship the English model

    Keeps the neural model with the best validation sentiment F1, exports it as int8, tunes each emotion's cut-off on the int8 outputs and scores the test comments.

    ml/train.py:3954 · 51 lines

    kind_en = max(trained_en, key=lambda k: trained_en[k][2])
    m_en, hist_en, _ = trained_en[kind_en]
    exp_en, deq_en = export(m_en, kind_en, ["sent", "emo"])
    gseq_va = [[gindex.get(t, 1) for t in ts] or [1] for ts in Gva]
    gseq_te = [[gindex.get(t, 1) for t in ts] or [1] for ts in Gte]
    Fva = [forward(deq_en, kind_en, s, ["sent", "emo"]) for s in gseq_va]
    Fte = [forward(deq_en, kind_en, s, ["sent", "emo"]) for s in gseq_te]
    Sv, Ev = np.array([f["sent"] for f in Fva]), np.array([f["emo"] for f in Fva])
    St, Et = np.array([f["sent"] for f in Fte]), np.array([f["emo"] for f in Fte])
    th_ship = best_thresholds(Ev, Eva)  # thresholds tuned on the shipped (int8) model's validation outputs
    ps = St.argmax(1)
    emo_f1 = f1_score(Ete, Et >= th_ship, average=None, zero_division=0)
    agree_en = float(np.mean(ps[Kte] == m_en.predict(GXte, verbose=0)[0][Kte].argmax(1)))
    print(f"  en shipped {kind_en}: acc {accuracy_score(Ste[Kte], ps[Kte]):.4f} f1 {f1_score(Ste[Kte], ps[Kte], average='macro'):.4f} emo macro-F1 {emo_f1.mean():.4f}, agrees with float {agree_en:.4f}")
    parity["en"] = {"seqs": gseq_te[:600], "sent": St[:600].tolist(), "emo": Et[:600].tolist(), "texts": gte.text.tolist()[:300], "tokens": [[gindex.get(t, 1) for t in ts] for ts in Gte[:300]]}
    pick_en = [
  9. 9

    Export weights and results

    Saves parity data, moves the int8 weights and vocabularies to public/models/bilingual-sentiment-weights.json so the page stays light, then writes the results JSON.

    ml/train.py:4007 · 10 lines

    json.dump(parity, open(RAW / "sentiment_parity.json", "w"))
    # weights + vocabularies (~2 MB) go to a static file the demo fetches when it opens, so the page itself stays light
    weights = {lang: {"model": payload[lang].pop("model"), "vocab": payload[lang].pop("vocab")} for lang in ("id", "en")}
    for lang in ("id", "en"):
        payload[lang]["vocab_size"] = len(weights[lang]["vocab"])
    wp = ROOT / "public" / "models" / "bilingual-sentiment-weights.json"
    wp.parent.mkdir(parents=True, exist_ok=True)
    wp.write_text(json.dumps({**weights, "token_pattern": SENT_TOKEN, "maxlen": SENT_MAXLEN}, separators=(",", ":")), encoding="utf-8")
    print(f"  -> {wp.relative_to(ROOT)}  ({wp.stat().st_size / 1024:.1f} KB)")
    save("bilingual-sentiment", {**payload, "token_pattern": SENT_TOKEN, "maxlen": SENT_MAXLEN, "tf": {"tensorflow": tf.__version__, "keras": keras.__version__}})

In your browser · TypeScript

  1. 10

    Tokenizer, BiLSTM and explanations in TypeScript

    Same tokens as Python, int8 embeddings, both LSTM directions, max over words, then sentiment and emotions; explain() leaves out one word at a time.

    src/lib/text-lstm.ts:41 · 94 lines

    export function createTextNet(m: TextModel, vocab: string[], maxlen = 100) {
      const index = new Map(vocab.map((w, i) => [w, i + 2] as const)); // 0 = padding, 1 = unknown word
      const D = m.emb.shape[1];
      const E = perRow(m.emb);
      const dirs = m.kind === "bilstm" ? [m.fwd!, m.bwd!].map((d) => ({ U: d.units, k: perCol(d.kernel), r: perCol(d.recurrent), b: d.bias })) : [];
      const head = (h: Head) => ({ w: perCol(h.w), b: h.b, n: h.w.shape[1] });
      const sent = head(m.sent);
      const emo = m.emo ? head(m.emo) : null;
    
      const ids = (tokens: string[]) => (tokens.length ? tokens.map((t) => index.get(t) ?? 1) : [1]);
    
      function lstm(xs: Float64Array[], d: (typeof dirs)[number]): Float64Array[] {
        const { U, k, r, b } = d;
        const G = 4 * U;
        let h = new Float64Array(U);
        let c = new Float64Array(U);

Notes

The decisions behind it.

Approach

  1. 1

    Kept each dataset's official train, validation and test split; the test sets were scored once, at the end.

  2. 2

    Wrote one tokenizer used by both Python and TypeScript; words seen at least twice in training form each vocabulary, the rest share an 'unknown' vector.

  3. 3

    Compared three models per language: TF-IDF + logistic regression, averaged word embeddings and a BiLSTM (two sizes, chosen on validation data), all with balanced class weights.

  4. 4

    For English, one network with two heads: a softmax over four sentiments and 28 sigmoid emotions, each emotion with its own threshold tuned on validation data.

  5. 5

    Shipped 8-bit weights, re-tested them with the exact forward pass the browser runs, and explained answers by leaving each word out in turn.

Test macro-F1 · Indonesian · English (sentiment) · English emotions

TF-IDF + logistic regression0.740 · 0.642 · 0.462
Word embeddings, averaged0.661 · 0.612 · 0.360
BiLSTM0.754 · 0.632 · 0.385shipped

Honest note

With thousands of short, labelled texts, a linear model on word counts is a strong rival. The BiLSTM only edges the TF-IDF baseline on Indonesian (macro-F1 0.754 vs 0.740, at slightly lower accuracy) and does not beat it on English sentiment (0.632 vs 0.642) or on the 28 emotions (0.386 vs 0.462), where the baseline trains one classifier per emotion. Every model scores lower on the official Indonesian test set than on validation; it is more balanced than the training data. Pretrained transformers such as IndoBERT or RoBERTa score clearly higher but weigh hundreds of megabytes; these networks are small enough to run instantly in a page. The two test sets are different kinds of text, so Indonesian and English scores should not be compared with each other.

Reproduce: ml/train.py → sentiment()

dataset source

Next project

Doodle Duel