This article is on voice AI tuning, illustrating how voice AI is about many moving parts working together and failure is often hidden in nuance. Without clear and vigilant observability, a seemingly successful call can fail the business objective.

The business purpose: capture a caller's message as structured output for follow-up.

Mira, the message-taking assistant in this test, has a narrow job. She captures three fields when the caller volunteers them: caller name, callback number, and caller message. None of the three is mandatory.

Context: this is a typical speech-to-text, language-model, text-to-speech pipeline. Speech-to-text turns the audio into a transcript. This test used Deepgram Nova 3. Structured output is extracted from that transcript, not from the original audio. The audio is not interrogated. The transcript is the only reference.

If the transcript is inaccurate, or only seemingly accurate, the saved record can fail before any later step begins.

Presence is not accuracy. Each field is either captured or not, which gives eight possible outcomes: 2 × 2 × 2. That count only records whether a field is there. A captured message can still carry the wrong meaning.

Accuracy adds a third state. Each detail can be missing, captured accurately, or captured inaccurately: 3 × 3 × 3, so 27 possible outcomes. Accurate means the semantic meaning, not the caller's exact words.

Because extraction reads the transcript, and the transcript is not always accurate, the meaning can already be damaged before extraction. Copying the transcript verbatim would store that damage.

Solution: make Mira say the proposed message and ask only whether it is correct. Record whether the caller accepted that meaning.

A voice assistant can sound competent and still save the wrong message for your business.

We saw this while testing Mira. The caller asked the team to discuss an idea. The audio was clear.

Audio: "Yes. I'm Alex. Please ask the team to discuss my idea."

The stored transcript left discuss out.

Caller: "Yes. I'm Alex. Please ask the team to my idea."

Mira: "Thank you, Alex. I have your request for the team to discuss your idea. Is there anything specific you would like me to add?"

Caller: "No."

Structured output: "message": "Please ask the team to my idea."

The field follows the transcript. From the business side, it is a fail. The verb discuss is missing, so the meaning is gone. The team, at best, has to guess.

Mira's follow-on question did not help. It is not a confirmation of the message. It is a follow-on question about the action.

What we changed

We added confirmation gates to Mira's prompt. She says the proposed message and asks only: "Is that correct?"

Example: "You would like the team to discuss your idea. Is that correct?"

She waits for the caller's answer before collecting another detail.

We also added structured output fields for the read-back: whether the message was confirmed, and an accuracy judgement.

What the next call proved

The audio was the same. The transcript again omitted discuss. This time Mira read the meaning back and asked for a direct answer. The caller accepted it.

Audio: "Yes. I'm Alex. Please ask the team to discuss my idea."

Caller: "Yes. I'm Alex. Please ask the team to my idea."

Mira: "Thank you, Alex. You would like the team to discuss your idea. Is that correct?"

Caller: "Yes, correct."

Structured output: "message": "Alex would like the team to discuss Alex's idea."

Structured output: "message_confirmed": "Yes"

Structured output: "message_accuracy": "High"

Alex confirmed the meaning. The stored message is a semantic reading of that meaning, not a verbatim copy of the damaged transcript. High records that the meaning held.

The transcription problem had not disappeared. The confirmation gate now shows whether the meaning survived.

What to ask your voice-agent builder.

Ask them to show the recording, the transcript, the saved result and the evidence supporting that result. Then ask whether the receiving team can understand the intended action without listening to the call themselves.

That is observability in practical terms: being able to inspect what happened at each stage and see where meaning changed.

Voice combines transcription, conversation decisions, spoken responses and extraction. Each part can appear to work while the overall business task fails. Competent building and operation require checking those parts together, against the outcome the business actually needs.

Stop guessing which prompt lines are safe to remove. Book a free 30-minute discovery call and we will review your live voice agent with you: https://patrickmichael.co.za