fastText

What is fastText

• fastText is just linear-classifier with explicit low-rank constraint (the embedding table)

1.1 Notation

• Vocabulary of features (tokens and n-grams): size $V$.
• Number of classes: $k$.
• Embedding dimension (hidden size): $h$.
• Document $n$ → multiset (because the same words/n-grams can appear multiple times) of features $S_n\subset \{1,\dots ,V\}$

fastText represents a document as a normalized bag-of-features vector:

$x_n\in {\mathbb{R}}^V,(x_n{)}_i=\{\begin{array}{ll}\frac{1}{|S_n|}& \text{if feature }i\in S_n\\ 0& \text{otherwise}\end{array}$

This is just like a one-hot vector but divided by the number of features in $S_n$.

Two parameter matrices:

• Embedding / input matrix $A\in {\mathbb{R}}^{h\times V}$
• Output matrix $B\in {\mathbb{R}}^{k\times h}$

The document representation (hidden feature) is:

$z_n=A{x}_n\in {\mathbb{R}}^h$

Concretely, because $x_n$​ is a normalized bag, $z_n=\frac{1}{|S_n|}{\sum }_{i\in S_n}{A}_{\bullet ,i}$ i.e. average of the embeddings of all features (words + n-grams).

The class scores are just a linear projection:

$s_n=B{z}_n\in {\mathbb{R}}^k$

To obtain the final probablities, we can just use softmax or hierarchical softmax if the number of classes is too large.

1.2 Low-rank view

If you define a single big weight matrix:

$W=BA\in {\mathbb{R}}^{k\times V}$ then $s_n=B{z}_n=B(A{x}_n)=W{x}_n$

So fastText is exactly a linear classifier in the bag-of-features space with weight matrix WWW, but constrained to have rank ≤ $h$.

N-gram features + hashing trick

The base model uses word unigrams, but fastText improves results significantly with word n-grams (bigrams, trigrams, etc.).

2.1 Bag of n-grams

Instead of just: $S=\{{\text{word}}_1,\dots ,{\text{word}}_T\}$

you also include:

• word bigrams: $(w_1{w}_2),(w_2{w}_3),\dots$
• maybe trigrams, up to some max order.

All of these are simply treated as additional “tokens” in the vocabulary and embedded in the same space:

• Each n-gram has its own column in $A$.
• The document representation is still the average of all present tokens and n-grams.

This partially injects word order information into an otherwise bag-of-words model.

2.2 Hashing trick for n-grams

Explicitly storing full n-gram vocabulary is memory-expensive (billions of possible n-grams), so they use the hashing trick:

• Choose a fixed number of bins $B$:
  • If only bigrams: $B=10\text{M}$
  • If higher n-grams: $B=100\text{M}$
• For each n-gram string $g$, compute a hash $h(g)$
• Map n-gram to a bucket index $b=h(g)\mathrm{mod}B$

This index behaves like a synthetic “token id” used to index the columns of AAA. Collisions between distinct n-grams mean their embeddings are shared, which introduces a bit of noise but works well in practice.

The final input feature set $S$ is then:

$S=\{\text{word tokens}\}\cup \{\text{hashed n-gram IDs}\}$

How to train it in practice

Deepseek-Math

• Deepseek-Math used it as binary classifier for math web pages
• They train it using the fastText library
• Task: binary classification: math vs non-math web pages
• Training data:
  • 500k positives from OpenWebMath
  • 500k negatives sampled from Common Crawl
• fastText hyperparameters (explicit in the paper):
  • dim = 256 (embedding size)
  • lr = 0.1
  • wordNgrams = 3 (max length of word n-gram)
  • minCount = 3 (minimum word occurrences)
  • epoch = 3
  • minn = 0, maxn = 0 (no char n-grams mentioned; they only talk about word n-grams)
  • Loss function isn’t specified; they just say “fastText (Joulin et al. 2016)” → in practice that usually means default supervised settings

FineWeb

FineWeb itself uses fastText only as a language detector, not a custom classifier:

• Task: language ID to keep (mostly) English text.
• Model: off-the-shelf fastText language classifier (e.g. lid.176.*).
• Filtering rule: keep samples where English probability ≥ 0.65