How we built the most accurate Somali speech-to-text model
, Impetik
Between 31 August and 3 September 2026 we trained five generations of a Somali speech-to-text model. The last one, whisper-somali-v5, is the most accurate Somali model we know of. You can test it with your own microphone on impetik.ai. This post explains how we got there, including the parts that went wrong.
Why Somali speech-to-text is hard
A speech recognition model needs many hours of audio paired with correct transcripts. For Somali, very little of that exists in public. What does exist is mostly read speech, where someone reads a script into a good microphone. Real conversation sounds different. People pause, talk over each other and mix in English and Arabic words. Dialects also differ between Somalia, Somaliland, Kenya, Ethiopia, Djibouti and the diaspora.
What already existed
We started by running five public Somali models on real Somali clips.
| Model | Result | License |
|---|---|---|
| mms-1b-all | Best transcripts, about 44% WER | CC-BY-NC |
| skydheere-mms | Close second | CC-BY-NC |
| mustafaa-wav2vec2 | Slightly worse | Apache 2.0 |
| paza-whisper-turbo | Drifted into Kikuyu and invented text | Unusable |
| steja-whisper-large | Empty output | Unusable |
WER is word error rate, the share of words the model gets wrong. Lower is better.
The best models carry a non-commercial license. Any fine-tune of them inherits it. The only model with a permissive license was worse than the rest. So nothing that existed could become a product.
Generations 1 and 2: MMS adapters
Our first fine-tunes used the mms-1b-all base with a small adapter layer. The data was African Next Voices Kenya, a conversational Somali corpus released under CC-BY-4.0 by Maseno University.
| Model | Training data | ANV test WER | FLEURS WER |
|---|---|---|---|
| stock mms-1b-all | none | 43.8% | 44.3% |
| adapter v1 | 12.5 h ANV | 37.6% | 47.3% |
| adapter v2 | 23.4 h ANV + FLEURS train | 35.4% | 46.3% |
The first run took 19 minutes on one A100 and cost about $3. It improved in-domain accuracy and got worse on formal read speech. Mixing in FLEURS recovered some of that. The lesson was that fine-tuning works, but the base model’s license made it a dead end.
Generation 3: Whisper
We switched the base to openai/whisper-large-v3-turbo. It is MIT licensed and it outputs punctuation and casing. We then scaled the training data in three steps.
| Model | ANV data | ANV test WER | Owner’s samples WER |
|---|---|---|---|
| whisper v1 | 12.5 h | 28.7% | 50.9% |
| whisper v2 | 23.4 h + FLEURS | 19.8% | 48.0% |
| whisper v3 | 45 h + FLEURS | 15.0% | 40.2% |
Each step cut the in-domain error by roughly a half, then a third. Whisper v3 was the first model to beat the stock models on the owner’s own phone recordings. An independent check on the thiomi-5k test split, which the model had never seen, gave 25.4% WER.
Keep the 15.0% figure in mind. It comes back later.
The clean-corpus experiment: a useful failure
By this point we had found two more datasets. Digital Divide Data had released a reviewed Somali corpus under CC-BY-4.0, and thiomi-5k offered a small human-recorded set. Both have carefully checked transcripts. Our conversational data does not.
So we tested the hypothesis that transcript quality beats quantity. We trained a model on reviewed read speech only, with no conversational data at all.
| Test set | Clean-corpus model | whisper v3 |
|---|---|---|
| Its own held-out split | 12.8% | |
| ANV conversational referee | 63.6% | 15.0% |
| thiomi test | 37.2% | 25.4% |
| Owner’s samples | 57.4% | 40.2% |
On its own held-out split the model looked spectacular. Everywhere else it collapsed. It was even worse on the thiomi test than v3, although thiomi’s training split was in its data. The answer was clear. Matching the register and acoustics of real speech matters more than polished transcripts. Conversational data is the scarce ingredient. That result shaped everything after it.
The audit
Before spending on the final run we audited the whole pipeline. The main finding was bad news. Our conversational benchmark had been sampled from the same corpus as the training data. As the training sets grew, more of the benchmark clips ended up inside them. The v3 figure of 15.0% was inflated by about 9 points. It measured memorisation, not recognition.
The fix was a shard-level holdout. Every ANV shard whose number ends in 9 is never trained on. We rebuilt the benchmark from those shards only, 845 clips, and rescored every model. On the clean benchmark v3 scores 23.7%. The old figures are retired.
The audit also caught a loss. An intermediate model, v4, reached 13.88% on its held-out split and then its weights were destroyed by a truncated file transfer that nobody had verified. Every artifact is now checksummed before a source is deleted. Crash resume, a distributed cache race and learning-rate scaling were fixed at the same time.
Generation 5
whisper-somali-v5 trains on every source combined, about 240 hours, with the holdout removed.
| Source | Hours | License |
|---|---|---|
| Digital-Divide-Data/Somali-ASR-Subset-68H (Maxaa Tiri, reviewed) | ~150 | CC-BY-4.0 |
| African Next Voices Kenya (conversational, shards ending in 9 held out) | 110 | CC-BY-4.0 |
| google/fleurs so_so train | ~10 | CC-BY-4.0 |
| thiomi/thiomi-5k som_Latn train | 8.7 | none published |
It trained for 3 epochs on two A100 GPUs, learning rate 2e-5 with 300 warmup steps. These are the results on the decontaminated benchmarks.
| Test set | v5 WER | v5 CER | v3 WER |
|---|---|---|---|
| ANV conversational referee (845 clips, never trained on) | 22.3% | 7.0% | 23.7% |
| thiomi-5k test (1,365 read-speech clips, train split in v5 data) | 18.9% | 6.0% | 25.4% |
| 25 real-world phone recordings, owner-verified transcripts | 37.9% | 9.9% | 41.2% |
| BBC Somali broadcast clip | 3.7% | 0.7% | 37.0% |
CER is character error rate, the same measure counted per character. On the same sets the stock mms-1b-all model scores around 44 to 47% WER. The best public Somali fine-tunes score 45 to 52% on the real-world set.
Total GPU and CPU rental for the whole project, including every mistake, was about $70.
About the word “most”
We say most accurate because that is what every measurement we could construct shows. It is still a claim with limits. One research lab in Mogadishu reports a lower WER for its own model on a private radio test set. Its weights are gated and we have not been able to run it on neutral benchmarks. We will update this post if that changes.
Limitations
- The training data is mostly Somali collected in Kenya. Dialects from Somalia and the diaspora are untested beyond the owner’s samples and one BBC clip.
- English words inside Somali speech get mangled. The training data is pure Somali.
- On silence, noise or badly degraded audio the model can invent fluent text instead of returning nothing.
- The model may stop early at long pauses, and the 30-second chunking can drop words at chunk boundaries.
- Numerals were filtered from training transcripts, so spoken numbers come out as words.
- The evaluation sets are small. The WER figures carry a few points of noise.
- thiomi-5k has no published license. Until that is cleared, v5 is a research preview and not a commercial product. A version without thiomi is one dataset filter and one retrain away.
Try it, and lend your voice
Every gap in that list is a data gap. The way to close it is more real Somali speech, from more places, with more accents and more code-switching.
That is what impetik.ai is for. Record a few seconds, read what the model heard, fix the mistakes and submit. Each corrected clip goes into a Somali speech dataset that we will publish on Hugging Face under CC-BY-4.0, free for anyone to use. Your voice will make the next model better for everyone who speaks Somali.
Training data credits: Maseno University and African Next Voices, Digital Divide Data, Google FLEURS and the thiomi-5k authors.