What a text-to-speech system cannot read

Two stages, and the second one is not where the problems are

Most text-to-speech systems work in two parts. A first model turns text into an intermediate acoustic representation: usually a mel-spectrogram, a time-frequency picture of the intended sound. A second model, the vocoder, turns that picture into a waveform. Newer systems compress the two into one by predicting tokens of a learned audio codec directly.

The vocoder is largely a solved problem: modern ones produce clean, natural-sounding audio. Almost everything that makes synthesised speech sound wrong happens in the first stage, and it is not about the voice at all. It is about prosody: the rhythm, stress and intonation that carry meaning beyond the words.

What the text does not contain

Say this sentence aloud seven times, stressing a different word each time:

I never said she stole my money.

Each version means something different. I never said it; someone else did. I never said it; I implied it. She stole my money, not my car. The written sentence contains none of that. The stress is the meaning, and it is absent from the input.

A TTS model has to guess, and it guesses from statistical regularities in its training data. On ordinary declarative prose it does well. On anything where the emphasis carries the point (a contrast, a correction, an aside, a list where one item is the surprising one) it produces a plausible reading of the wrong sentence.

The specific failures

Homographs. Words spelled the same and pronounced differently, disambiguated only by meaning:

• read: “I read it yesterday” against “I will read it”
• lead: the metal against the verb
• live: “live broadcast” against “where they live”
• bass, minute, wind, tear, close, record, desert, row

Models get these right most of the time from context and wrong in a way that stops a listener.

Numbers and dates. “1996” is nineteen ninety-six as a year and one thousand nine hundred and ninety-six as a quantity. “3/4” is a fraction, a date, or a ratio. “10-12” is a range or a subtraction. Phone numbers, version numbers, prices and measurements all have conventions the text does not state.

Names and jargon. Proper nouns are where the errors concentrate, and they are the words a listener is most likely to be listening for.

Acronyms. Some are spelled out, some are pronounced as words, and there is no rule in the text: NASA is a word, FBI is letters, SQL is both depending on who you ask.

Long-form structure. Most systems synthesise a sentence or a chunk at a time with no memory of the paragraph. So the emphasis does not build, a list does not fall at the end, and a long read has a flat, sentence-by-sentence sameness that listeners describe as tiring without being able to say why.

SSML: the controls that exist

Speech Synthesis Markup Language is the standard way of supplying what the text lacks. Support varies by engine, but the core is widespread:

<speak>
  The report was published in <say-as interpret-as="date" format="y">1996</say-as>.
  She <emphasis level="strong">never</emphasis> agreed to it.
  <phoneme alphabet="ipa" ph="ˈsiːkwəl">SQL</phoneme> is one option.
  <break time="400ms"/> The alternative is simpler.
  <prosody rate="92%" pitch="-1st">This part is an aside.</prosody>
</speak>

A pronunciation lexicon (a list of your names and terms with their phonetic spellings) is worth building once for any project with recurring vocabulary, and it removes the most damaging category of error permanently.

Practical technique

• Write for the ear. Shorter sentences, one idea each. A clause that a human reader would handle with a subtle pitch move will defeat the model.
• Punctuate for timing. A full stop produces a longer pause than a comma. Splitting a long sentence into two changes the delivery more reliably than any tag.
• Spell things out in the input. Writing “nineteen ninety-six” is often quicker than a tag and always works.
• Listen to the whole thing. Errors are local and a spot check will miss them.
• Chunk long reads at paragraph boundaries, and check the joins for a level or timbre shift.

The free path

• Piper: fast, local, neural, dozens of languages, runs on a Raspberry Pi. The sensible default for free self-hosted TTS.
• eSpeak NG: formant synthesis, very robotic, tiny, over a hundred languages, and still valuable for languages no neural model covers.
• Coqui TTS: a widely used open-source toolkit; note that the company behind it shut down, so the repository is no longer actively maintained even though the models and code remain available and usable. Check the project’s state before building on it.
• Mimic 3, MaryTTS: other open options.
• Whisper is the counterpart on the recognition side, covered in a later lesson.

The limitation

Synthesised speech is now good enough that the remaining errors are more dangerous than the old obvious ones. A robotic voice signalled clearly that this was a machine; a natural voice confidently mispronouncing a drug name or a person’s surname does not signal anything, and a listener may not notice.

There is also no reliable way for the model to know what it got wrong. It does not have a confidence score for prosody, and it will read a sentence with exactly the wrong emphasis in a perfectly convincing voice. The only check is a person listening to the output against the intended meaning, which is unglamorous and is the actual quality control step.