a 70m-parameter trilingual language model, trained from scratch on free gpus

i built a small english / spanish / french gpt end to end, with its own tokenizer, data pipeline, pretraining and fine-tuning, on kaggle t4s, and measured where it wins and where it doesn't.

question

can a 70m model trained from scratch on 1.25b tokens, with a tokenizer built for three languages, beat a much larger english-first model on spanish and french? and what does a tiny instruction-tuning pass cost?

what i expected going in:

  • i expect a shared trilingual tokenizer to beat gpt-2 small on spanish and french bits-per-byte.
  • i expect gpt-2 small to still win on english, since it saw far more english text.
  • i expect fine-tuning on a small instruction set to raise validation loss a little.
  • i expect a 70m model to be fluent but often factually wrong.

model

fieldvalue
parameters70.55m total (49.58m non-embedding)
architecturedecoder-only transformer, 10 layers, d_model 640, 10 heads, ffn 1728 (swiglu), rmsnorm, rope
context1,024 tokens
tokenizermy own 32,768-token byte-level bpe, shared across en/es/fr, with language tags
flops per token~502m

data

languagesourcetrain tokenslicence
englishfineweb-edu (huggingfacefw)650.0modc-by
spanishfineweb2-hq (epfml)325.0modc-by
frenchfineweb2-hq (epfml)325.0modc-by

the mix is 50/25/25. about 2m held-out validation tokens per language. fine-tuning used cohereLabs/aya_dataset filtered to eng/spa/fra (apache-2.0) plus self-made 'continue this text' tasks, 1,500 per language.

method

pretraining: 2,385 steps at 524,288 tokens per step (1.25b tokens total), peak learning rate 1e-3 decaying to 1e-4, on 2× tesla t4 with ddp and torch.compile, in fp16 with a gradient scaler.

fine-tuning: 2 epochs, lr 1e-4, loss on replies only, 832 steps, about 11 minutes.

evaluation: per-language perplexity on validation shards, and bits-per-byte on identical raw held-out documents (200 per language), which is fair across models with different tokenizers. every model gets the same 1,024-token window. a language-id check runs on the generations.

metricstatusnote
bits per byteusedmain comparison metric. tokenizer-independent.
perplexityused, own model onlynot comparable across tokenizers.
language-id of generationsuseddoes the model answer in the language it was asked in.
train lossnot quotedvalidation loss is quoted instead.

results

lower bits-per-byte is better.

pretrained base vs gpt-2 small
metricenglishspanishfrench
base val perplexity30.4328.7426.07
base bits per byte1.1051.0131.061
gpt-2 small (124m) bits per byte0.9861.9891.914
base language-id accuracy0.980.980.98

on spanish and french the base model roughly halves gpt-2 small's bits-per-byte at about half its size. gpt-2 still wins english.

after fine-tuning (sft)

metricenglishspanishfrench
sft val perplexity38.7036.5032.56
sft bits per byte1.1801.0871.137
sft language-id accuracy0.870.200.47

fine-tuning cost about 0.07–0.08 bits per byte, and the replies drift out of spanish and french. that is the main failure so far.

vs other small open models

bits per byte on the same 200 raw held-out docs per language, 1024-token window for every model. lower is better.

trilingua vs small open models, bits per byte
modelparamsenesfravg
bloom-560m559M0.9330.8640.8840.894
qwen2.5-0.5b494M0.8020.9420.9660.903
trilingua (base)71M1.1051.0131.0611.060
pythia-160m162M0.9841.2801.2951.186
smollm2-135m135M0.8401.3801.3901.204
pythia-70m70M1.1101.5031.5111.375
gpt-2 small124M0.9861.9891.9141.630
  • best average of every open model up to 162m params, and best spanish and french of that group.
  • only bloom-560m and qwen2.5-0.5b beat it, at 7–8x the parameters.
  • english is its weakest language: pythia-160m, smollm2-135m and gpt-2 all beat it there.
  • caveat: the test docs are held out from the same fineweb / fineweb2-hq distribution trilingua trained on, so it has an in-domain edge.

training curve

validation loss at the end of pretraining (step 2,385): english 3.454, spanish 3.124, french 3.254.

deviations

every change to the plan, and what it cost.

  • bf16 on t4 was a trapthe t4 reports bf16 as supported, but it is emulated. run 1 crawled at about 8k tokens/s (3% mfu).
  • switching to fp16 made it 7× fasterfp16 with a gradient scaler and torch.compile reached about 57k tokens/s (22% mfu). same model, one dtype change.
  • run 1 hit kaggle's time limitit stopped at step 615 of 2,385 after the 11-hour session budget. i resumed from its checkpoint in a second session.
  • the resume broke the first timea bf16 checkpoint has no gradient-scaler state, so loading it in fp16 needed a fix (commit b8be4c1).
  • fine-tuning drifts languagesafter sft, only 20% of spanish and 47% of french replies stayed in the right language. i am changing the fine-tuning mix before calling it a writing assistant.

limitations

  • one seed per run.
  • bits-per-byte uses 200 documents per language, so small gaps are noise.
  • gpt-2 saw far more english than this model, so the english comparison is not a like-for-like data budget.
  • 70m parameters is fluent but not factual. samples often state wrong facts.
  • no human evaluation yet.
  • spanish and french come from the same web corpus family, so results may not hold on other domains.

sources

  • fineweb-edu, hugging face
  • fineweb2-hq, epfml
  • aya dataset, cohere labs
  • gpt-2 small, openai
  • pytorch
  • kaggle for compute (2× t4)
  • ai assistants for code and analysis

code, notebooks and logs are in the repo.