Voice AI Accuracy: Test Orders, Returns, and Codes Against Benchmarks

Expect near-human error rates on clean recordings from the best speech models, but clearly worse results on phone, noisy and conversational audio. A voice agent’s real accuracy also depends on whether it finishes the task and gets order numbers and codes right, not on transcription alone. Below you will find definitions, benchmark ranges, the factors that shift them, and a test plan.

TL;DR:

Reported WER changes with transcript normalization and dataset choice, while phone audio raises errors versus clean recordings, so compare only matched conditions.

VoiceBench found response quality could vary by roughly 1.5 points at identical WER, so score task completion, entity fidelity, and conversation quality too.

One benchmark reports a median 0.44 gap between pass@k and pass^k, so repeat scenarios and judge consistent success, not the best attempt.

Build a representative test set across noise, accents, short turns, and overlap, normalize every transcript consistently, and report confidence intervals before setting acceptance thresholds.

For store agents, test order lookup, returns, shipping estimates, and code readbacks; treat any incorrect order number or amount as a failed run.

Orphora AI

Test Voice Support Against Store Data

Orphora AI voice agents integrate with WooCommerce and use real-time order and customer data to answer order status and returns questions.

Visit Orphora AI

What do WER, CER, DER and pass@k actually measure?
What accuracy can you realistically expect?
Which factors move accuracy in deployment?
Why isn’t WER enough to judge a voice agent?
How do you test voice AI accuracy on your own audio?
What should you test for e-commerce voice agents?
Our view: judge reliability, not the best run
FAQ
Sources

What do WER, CER, DER and pass@k actually measure?

Metric choice decides which question you can answer, so start with the definitions:

Word error rate (WER): substitutions plus deletions plus insertions, divided by the number of words in the reference transcript. A substitution swaps a word, a deletion drops one, an insertion adds one that was never said.
Character error rate (CER): the same calculation at the character level, which is more useful for order numbers, codes and names.
Diarization error rate (DER): how often speech is attributed to the wrong speaker. It matters when you need to separate a caller from an agent.
pass@k: whether a system succeeds at least once across k attempts at the same scenario. It shows peak performance.
pass^k: whether it succeeds on all k attempts. It shows reliable performance.
EVA-A and EVA-X: composite scores from EVA-Bench, where EVA-A covers accuracy and EVA-X covers the conversational experience.

Two caveats apply to every number you read. Transcript normalization (how casing, punctuation and numerals are handled before scoring) changes reported WER. So does the dataset. A figure without its method is only a rough hint.

What accuracy can you realistically expect?

On clean, studio-quality audio, published benchmarks report low WER for the strongest models. Reproducible reports such as VoiceBench and vendor benchmark write-ups in the Gladia style point the same way. Those numbers are a ceiling, not a forecast for your call center.

Realistic ranges by condition:

Clean recordings: low WER for top models.
Phone calls: a material gap. Depending on dataset and conditions, expect WER to worsen, plus higher DER when speakers are labeled.
Conversational and long-form audio: wider variance, because of disfluencies, overlap and very short turns.
Intent and entity extraction: domain-tuned language understanding beats generic models, and structured fields such as order IDs are extracted more reliably than free-text answers.
Text to speech: modern synthetic voices score close to human on many listening tests, but that is a separate trust axis from recognition accuracy.

Historical evidence backs the telephone gap. NIST’s analysis of the DARPA Communicator systems found that WER correlates with task completion and that telephone-channel speech produces materially higher WER than high-quality recordings.

Raw WER alone is an insufficient metric for modern voice agents, and the Open ASR Leaderboard research recommends task-specific, multidimensional evaluation that accounts for noise and spontaneous speech.

Which factors move accuracy in deployment?

Four groups of variables explain most of the movement between a lab number and a live one.

Acoustic: signal-to-noise ratio, steady versus sudden noise, reverberation, and the low-pass filtering and compression of telephone codecs. Telephone channels can increase recognition error rates.
Speaker: accents, non-native speech, pitch variation and disfluencies all produce measurable drops. Overlapping speech causes outsized errors.
Content: jargon, order numbers, dates and money amounts are the riskiest entities. Code-switching between languages raises errors further.
Architecture: pipelines (separate recognition, language model and speech synthesis) and end-to-end models fail differently. VoiceBench found pipelines often outperform end-to-end models on spoken-instruction tasks.

The practical consequence: a model can look fine on average and still mangle the one string your customer cares about. Mitigations that help in agent scenarios include speaker adaptation, contextual biasing (telling the recognizer which words to expect, such as your product names), and ontology repair, which corrects a mis-heard value against known valid options.

Pro Tip: Build your hardest test cases around entities, not sentences: order numbers, ZIP codes, email addresses and dollar amounts fail first.

Why isn’t WER enough to judge a voice agent?

Because a transcript can score well and still produce a bad outcome. VoiceBench showed that ASR metrics alone do not predict response quality, with response quality varying by roughly 1.5 points at identical WER values. Treat WER as one input, not the verdict.

Three frameworks fill the gap:

EVA-Bench adds EVA-A and EVA-X composites and separates peak from reliable performance. Across systems it reports a median pass@k to pass^k gap of about 0.44, meaning single-run results often overstate what you will see in production.
VoiceBench compares architectures on spoken-instruction tasks rather than transcription alone.
Talk2Agent uses task-conditioned, execution-free measures that correlate better with downstream task completion than WER or CER, and can flag when transcription alters task-critical entities.

Framework	What it measures	Why you care
EVA-Bench	Accuracy, experience, reliability across repeated trials	Exposes the peak-versus-reliable gap
VoiceBench	Response quality for spoken instructions	Shows ASR metrics miss response quality
Talk2Agent	Task-critical entity retention	Catches transcription that changes what the agent does

A workable in-house rubric combines three weighted parts: task completion, entity fidelity (order IDs, confirmation codes, amounts), and conversation quality (turn-taking, conciseness). Weight entity fidelity heavily if wrong actions are costly.

How do you test voice AI accuracy on your own audio?

Follow these steps so your numbers are reproducible and comparable:

Collect representative calls. Pull in-scope audio covering clean and noisy lines, varied accents, short turns and overlap. Aim for enough samples per condition that a handful of bad calls cannot swing the result; a few dozen per condition is a sensible floor for a first pass.
Apply controlled perturbations. Step down the signal-to-noise ratio, simulate the telephone channel, add codec compression and insert background noise.
Repeat each scenario k times. Run independent trials so you can compute both pass@k and pass^k.
Normalize transcripts. Use one normalization recipe for every system you compare, then compute WER, CER and DER.
Score the task level. Record task completion, entity fidelity and conversation quality for every run.
Read the gaps. Report confidence intervals, not just means. A large pass@k to pass^k gap signals brittleness. Set acceptance thresholds before you look at results.

NIST’s DARPA findings add one more reading rule: task completion tracks WER roughly linearly, and user satisfaction becomes harder to predict as error rates rise.

Pro Tip: Keep a frozen test set. If you retune on it, build a fresh one before any procurement decision.

For capturing and monitoring real call audio, our guide on call quality monitoring covers the workflow. If you also want to connect call quality to ad results, this partner piece on measuring call quality for PPC is a useful read.

What should you test for e-commerce voice agents?

We build AI voice agents for WooCommerce stores, with real-time order lookup, agent customization and 24/7 phone support. We make claims about faster response times and customer satisfaction. Those are our own claims, so test them the way you would test anyone’s.

Four scenarios cover most store calls: order-status lookup, return initiation, shipping ETA, and confirmation-code readback. Score entity fidelity strictly. An order number or amount that is off by one character is a failed run, even if the conversation sounded fine. Our WooCommerce voice bot checklist walks through these checks before launch, and our escalation rules show when a failed entity should hand off to a person.

Our view: judge reliability, not the best run

Vendor numbers usually describe the best case. We would ask for pass@k and pass^k figures, plus a domain-specific entity fidelity test, and then run a small test on your own production audio before you buy anything. Consistent task success beats a single impressive WER every time.

— Orphora AI

FAQ

Is voice AI reliable?

Voice AI is reliable under good conditions and less so on noisy phone audio. Peak results often overstate everyday performance: EVA-Bench reports a median gap of about 0.44 between best-attempt and all-attempts success. Test repeated trials on your own calls.

Is any AI 100% accurate?

No. Even the strongest speech systems make occasional errors, and error rates rise with noise, accents, telephone compression and overlapping speakers. Plan for fallback paths, such as confirmation readbacks and handoff to a human, rather than assuming perfection.

How can you tell if a voice is AI generated?

Listen for unnatural pacing, flat emotion around interruptions, and odd handling of numbers or names. Detection is unreliable by ear alone, since modern text-to-speech approaches human quality in many listening tests. For anything high stakes, rely on verification steps instead of judgment.

Which AI voice is most realistic?

There is no single answer, because realism is judged by listening tests that vary by voice, language and context. Evaluate candidates on your own scripts, and keep in mind that a natural voice says nothing about recognition accuracy.