What is Word Error Rate?

WER measures ASR accuracy as the sum of insertions, deletions, and substitutions divided by reference word count. It is the standard baseline metric but does not capture business impact of specific error types.

Why is WER not sufficient for production ASR evaluation?

WER treats all errors equally regardless of business impact. Intent accuracy and slot error rate capture the downstream consequences of ASR errors more accurately for transactional applications.

How do you improve ASR accuracy for domain-specific applications?

Phrase boosting for domain vocabulary, acoustic model adaptation with production recordings, and post-processing correction models trained on your specific error distribution.

Evaluating ASR Accuracy in Production: Metrics, Methods, and What WER Actually Tells You

Why ASR Accuracy Is Not Just About Word Error Rate

Word Error Rate (WER) is the standard benchmark for automatic speech recognition accuracy. It measures the percentage of words in a transcript that differ from the reference transcript. A 5% WER sounds excellent until you realize that in a customer service context, a single wrong word in a phone number or order ID invalidates the entire transaction.

Production ASR evaluation requires understanding not just how many words the system gets wrong, but which words, in which contexts, and with what downstream consequences.

ASR Metric	What It Measures	Production Relevance
Word Error Rate (WER)	% of incorrect words vs reference	Baseline benchmark, context-blind
Character Error Rate (CER)	% of incorrect characters	Better for morphologically rich languages
Semantic Error Rate (SER)	% of utterances with changed meaning	More relevant for intent-based systems
Intent Accuracy	% of utterances correctly classified	Most business-relevant metric
Slot Error Rate	% of entity slots incorrectly filled	Critical for transactional applications

Building a Production Evaluation Dataset

Evaluating ASR against lab datasets does not predict production performance. Acoustic conditions, speaker demographics, ambient noise, domain vocabulary, and speaking styles in production differ substantially from clean lab recordings.

A useful production evaluation dataset requires real customer call recordings (with appropriate consent and privacy handling), a representative distribution of speakers, noise conditions that match your actual deployment environment, and domain-specific vocabulary that tests whether the ASR handles your terminology correctly.

Optimize your ASR for real-world production accuracy

Request an ASR Audit

The Sources of ASR Error in Production

Acoustic Conditions

Background noise, echo, bandwidth-limited phone audio, and variable microphone quality all degrade ASR accuracy. Systems that perform at 95% WER on clean audio can drop to 80% WER on typical mobile phone audio in a noisy environment.

Domain Vocabulary

General-purpose ASR models are not optimized for domain-specific terminology. A contact center handling technical support calls will see elevated error rates on product names, model numbers, and technical terms that appear rarely in general speech data.

Speaker Variation

ASR models trained primarily on one accent or dialect perform worse on speakers with different speech patterns. If your user base includes non-native English speakers or regional accents not well-represented in your ASR provider's training data, expect elevated WER for those speaker groups.

Continuous Evaluation in Production

ASR accuracy changes over time as user demographics shift, product vocabulary evolves, and ASR provider model updates occur. Continuous evaluation samples production transcripts against human-labeled references on a weekly or monthly cadence.

Sample 1-2% of production sessions for human review
Label samples across a stratified distribution of call types and speaker demographics
Track WER trend over time, not just point-in-time snapshots
Alert when WER increases more than 2 percentage points from baseline

Improving ASR Accuracy in Production

When evaluation identifies specific ASR failure patterns, several levers are available: custom vocabulary or phrase boosting for domain-specific terms, acoustic model adaptation using production recordings, post-processing correction models trained on your error distribution, and ASR provider evaluation using production-representative data to select the best-performing option.

Evaluating ASR accuracy in production