A question comes up in almost every first conversation about turning phone calls or dictated notes into structured data: our recordings sound rough — does that break it? The honest answer is “it depends on how rough,” and that is measurable, so rather than guess, I measured it.
The trick that makes it measurable
You cannot compare a model’s transcript to the truth unless you know what was said. So start from clean recordings where the exact words are already known, then deliberately degrade the audio into the conditions a client actually has — a landline, a bad mobile connection, background machinery, a reverberant meeting room. What was said never changes, so the same known answer scores every version, and any difference between them is caused by the audio alone.
The model is whisper, running entirely on a local machine — no cloud, no data leaving the building. The score below is recall: the share of spoken words the model recovered.
What it says
The reassuring half: telephone audio costs essentially nothing. A landline, a compressed mobile call, clipping from a gain set too high, even dropped packets — the model shrugs them off. For anything built on phone calls, the recording quality is not the bottleneck.
The useful half: the meeting-room case is where it bends. Put the microphone at the far end of a table — distance, echo, and background noise together — and recall falls from near-perfect to 88%. That is the one condition worth spending on, and the fix is a better microphone, not a better model.
There was also a quieter signal. The model reports how confident it is, and that confidence fell in step with the audio quality — highest on the clean call, lowest on the far-field room. A number that drops exactly when accuracy drops is useful on its own: it lets the system flag the recordings a human should double-check, instead of failing silently.
Honest scope
This is a small, controlled test — twelve short read-aloud sentences, not spontaneous business speech with its interruptions and crosstalk. The room and noise conditions are faithful approximations, deliberately labelled apart from the real phone channels, because an approximation can rank options honestly but cannot stand in for one specific real room. So read the shape, not the decimal places: phone-grade audio is fine; far-field capture is the weak point. A number a client could take to the bank needs their own recordings — which, since the whole point is that the audio never leaves their building, is exactly the kind of thing measured on site.