Why '98% accurate' tells you nothing — and how to test it yourself

Why published figures are not comparable

Accuracy depends entirely on what audio was scored and how. One person reading a script into a headset scores well on every service. Four people talking over each other in a meeting room, using jargon, scores lower on the same service.

The scoring rules differ too. Whether particle errors, punctuation placement and script choice count as errors, or whether anything comprehensible counts as correct, moves the same result between 90% and 98%. With no third-party evaluation under common conditions, comparing published figures is meaningless.

The five factors that decide your result

1. Microphones and capture path

The largest single factor. An online meeting where everyone speaks into their own device gives the engine separated audio and scores well. One microphone in a room picking up several people gives distant, reverberant speech and scores badly.

This is also where you have the most leverage. Adding microphones to the room, or having everyone join online from their own machine, improves results more than switching vendors.

2. Number of speakers and overlap

Speaker separation degrades as participants increase. Two speakers is stable; beyond five with overlapping speech, attribution breaks down. Having the chair name the next speaker aloud gives you something to reconcile against afterwards.

3. Jargon and proper nouns

Company names, product names and industry abbreviations are not in a general dictionary. Most tools support a custom vocabulary — registering the thirty terms you use constantly changes perceived accuracy dramatically. Judging a tool as inaccurate before doing this is premature.

4. Ambient noise

Air conditioning, keyboards, conversation in the next room. Keyboards are the worst because they are closest to the microphone — the person typing the notes, unmuted, becomes the person hardest to transcribe.

5. How people speak

Frequent fillers, self-corrections and trailing sentence endings all make the transcript harder to read. That is not an accuracy problem; it is what speech looks like written down. Summarisation, not better transcription, is the fix.

A procedure for measuring it yourself

• 1. Take one real recording of about ten minutes, representative of your normal meetings
• 2. Register your thirty most common terms in each tool's custom vocabulary
• 3. Run the same audio through every candidate
• 4. Count only the errors that would mislead a reader — names, numbers, decisions — not particles or punctuation
• 5. Record how long correcting each transcript took; that number is the one that matters

Step four is the important discipline. Counting every deviation produces a number that correlates with nothing. Counting errors that would change what a reader understood gives you a figure you can act on.

Frequently asked questions