Koenecke et al. (FAccT 2024) found invented phrases in about 1% of Whisper API transcriptions from mid-2023, and 38% of them included explicit harms. They showed up disproportionately for speakers with longer non-vocal stretches, a common symptom of aphasia, which makes halting speech worth a place in that quiet-voice test set.
Have you ever caught Whisper or another AI transcription tool inventing words that nobody actually said? How did you notice?
Koenecke et al. (FAccT 2024) found invented phrases in about 1% of Whisper API transcriptions from mid-2023, and 38% of them included explicit harms. They showed up disproportionately for speakers with longer non-vocal stretches, a common symptom of aphasia, which makes halting speech worth a place in that quiet-voice test set.