Whisper tiny, 27% fewer errors on Beninese French: how we got there

Four billion people that machines barely hear

More than four billion people live in regions whose languages sit at the margins of the corpora used to train speech recognition systems. Wolof, Hausa, Swahili and Darija are hardly rare languages: each has tens of millions of speakers. What they lack is hours of transcribed speech.

National languages are not the only ones affected. French and English as spoken in Cotonou, Lagos or Dakar suffer just as much, with their own prosody and borrowings, and a model trained on the French of France transcribes them poorly.

We measured it. On everyday sentences read by two Beninese speakers, the official release of Whisper tiny gets 32% of words wrong, close to one word in three.

Here is what that looks like on a sentence from the test set. The speaker says:

Ne communiquez jamais votre code secret, même à un membre de votre famille.“Never share your PIN, not even with a family member.”

The official model hears:

Le communique est jamais votre cours de sucrée, même à un membre de votre famille.Roughly: “The press release is never your sugary class, even with a family member.”

The safety warning no longer forbids anything. After our training, the same sentence is transcribed without a single error.

At this error rate, dictation is unreadable and subtitles say something other than what was spoken.

The model still has to fit on a phone

In these regions, calling a remote service is not an answer: connectivity is intermittent, mobile data is expensive, and fieldwork happens precisely where the network does not reach. The model has to run on the phone people already carry, with no subscription and no latency.

Whisper tiny fits that brief: at 75 MB and 37.2 million parameters, the whole speech recognition system fits easily inside a mobile app.

Its weakness is that of every small model. With little capacity, it generalizes poorly beyond what it has seen, and underrepresented varieties are the first thing it stumbles on. That is also what makes it so sensitive to the quality of the data it is trained on.

What eight hours of recordings changed

We produced eight hours of Beninese French speech data from twenty-three voice-over sessions recorded in our studio with eight native speakers. We then trained Whisper tiny on 7.32 hours of that corpus, keeping the rest aside to monitor training. Nothing else changed: we started from the official weights, and the resulting file is exactly the same size as the original.

The gap is 8.7 points, with a 95% confidence interval of 6.6 to 10.8 points, which means 27% fewer errors. Across ten thousand bootstrap resamples of the test set, the fine-tuned model never loses. Nor does the gain hinge on one particular voice: both test speakers improve by the same proportion.

The effort comes down to two figures: eight hours of data, 7.3 of which the model actually saw, and seventeen minutes of training on a laptop. Nothing changes at inference time, since the model keeps its size.

That leaves the question of leakage, which any technical reader will raise straight away. The test set is a separate public corpus (labari-voice/fr-bj-speech-pilot), recorded independently, and none of its files was used for training. None of its 105 sentences appears in our corpus, none of its clips has an acoustic twin there, and every training run starts again from Whisper's official weights. The evaluation protocol is the one published with the test set, so any lab can reproduce the baseline.

It all comes down to the labels

A fine-tune is never better than its labels, and that is where our work lies. Most cheap speech corpora are transcribed by a machine, often by the very model you are trying to improve: you end up teaching it its own mistakes, and the gains level off quickly. Our transcripts come from the script. The voice actor reads a written text, and that text is the reference. Recognition models are only used to find where each sentence was spoken and to cut in the right place; they have no say over the content.

The rest is filtering, automated and unforgiving. A studio session is full of retakes and false starts, sentences read a second time because the first one did not work. Our pipeline aligns the script with the recording, detects these accidents and throws out any clip whose text does not match the audio exactly. Out of twelve hours of processed tracks, we kept eight: a third of the material goes in the bin, on purpose. The fine-tuned model improves 123 of the 209 test clips, degrades 33 and leaves 53 unchanged.

Everything we use comes from our studio in Benin. We write the scripts and record the voices ourselves. Nothing is scraped from the web or outsourced: we know who is speaking and under what contract.

What we offer: speech datasets in local French and English

At 23% errors, the model does not understand Beninese French, and we do not claim it does. It makes a quarter fewer errors without gaining a single byte, for eight hours of data and seventeen minutes of compute. The curve had not flattened when we stopped training, so there is room left.

This result took no new architecture and no compute budget, only eight hours of properly labeled data. The method works for any underrepresented variety, given a studio and scripts. That is what we sell: custom speech datasets, first of all in French and English as they are spoken locally, recorded and labeled the way this one was.

For an embedded product, the math is simple: the model still fits in 75 MB, runs offline with no server cost, and goes from one wrong word in three to one in four. That is still too many for open dictation, but enough for closed-vocabulary voice commands or a guided form.

Which leaves the question of where the scarcity lies. Compute can be bought and open models downloaded, whereas an hour of transcribed Beninese French still has to be recorded, where its speakers live.

We do not release the training corpus or the model weights, which are reserved for commercial use. The result, the method and the evaluation protocol are public, along with all 209 predictions, clip by clip: github.com/ahouetonde/whisper-tiny-beninese-french. So is the test set (huggingface.co/datasets/labari-voice/fr-bj-speech-pilot), on which the 32.07% baseline can be reproduced with a single command.