Product

Nobody Talks Like the Demo: 10 Things Real Calls Throw at a Voice Agent

Nobody Talks Like the Demo: 10 Things Real Calls Throw at a Voice Agent

Published

•

Written by

Khush Bakht

5 min Read

Keep Reading

Nobody Talks Like the Demo: 10 Things Real Calls Throw at a Voice Agent

Voice AI demos usually work under ideal conditions. The speaker is clear, the background is quiet, the conversation follows a predictable path, and the agent responds almost immediately.

Production calls are different.

People interrupt, hesitate, switch languages, speak over background noise, lose connectivity, and ask questions the system wasn't expecting. Meanwhile, the infrastructure has to process audio, run inference, execute tools, and return speech quickly enough that the conversation still feels natural.

At AethexAI, we've found that building reliable voice agents is as much a systems engineering problem as it is a model problem.

Here are ten challenges that shape how we build and operate our voice infrastructure.

1. Background noise changes everything

A customer calls to check their account balance. They're standing beside a busy road, with traffic, other conversations, and intermittent noise competing with their voice.

The agent needs to recognise what they're saying without confusing background speech for the customer's instructions.

This is particularly challenging over telephone networks, where audio is often narrowband, compressed, and affected by packet loss.

How we approach it

Our Kora speech models are developed with real-world audio conditions in mind.

Rather than evaluating speech recognition only on clean recordings, we focus on performance across different acoustic environments, including noisy and degraded audio.

The challenge isn't simply removing noise. It's preserving the information that matters while avoiding false recognition of background speech.

2. People switch languages mid-sentence

A customer might say:

"I was trying to check my balance, pero la aplicación no funciona."

The sentence begins in English and ends in Spanish.

For a multilingual speech recogniser, this introduces several challenges. The model needs to recognise both languages, handle the transition, and preserve the meaning of the full utterance.

A language classifier that assumes one language per call can make the problem worse.

How we approach it

We design our speech systems around multilingual input and code-switching rather than requiring callers to select a language at the beginning of a conversation.

The important distinction is between recognising multiple languages independently and recognising them within the same utterance.

That difference becomes especially important when evaluating models on real conversations.

3. An interruption isn't always an interruption

Consider an agent explaining a payment schedule.

The customer says:

"Right... yeah... okay... mm-hmm."

Those responses indicate that they're listening.

Then the customer says:

"Wait, that's not what I meant."

Now the agent needs to stop speaking.

A simple voice activity detector cannot reliably distinguish between these situations. It can detect speech, but not necessarily whether the speaker intends to take the conversational turn.

How we approach it

We treat turn-taking as a separate problem from speech recognition.

The system needs to account for acoustic activity, timing, and conversational context when deciding whether to continue speaking, pause, or yield to the caller.

This matters because overly sensitive interruption handling makes agents feel jumpy, while insufficient sensitivity makes them difficult to interrupt.

4. Silence is part of conversation

"Give me a second, I'm looking for my account number."

The caller goes quiet.

A rigid timeout might interpret the silence as abandonment and end the conversation.

But the caller has already explained why they're silent.

How we approach it

Silence handling should depend on context rather than a single fixed timeout.

An agent waiting for a customer to retrieve information should behave differently from an agent waiting for an answer to a straightforward question.

We aim to balance responsiveness with patience, avoiding unnecessary prompts while still detecting when a conversation has stalled.

5. Natural conversation isn't a fixed script

Two customers can ask the same question in completely different ways.

One gets straight to the point. Another begins with greetings, explains the background, changes direction, and only then describes the problem.

A voice agent needs to accommodate both without losing track of the task.

How we approach it

We separate conversational behaviour from the underlying business workflow.

The agent can adapt its tone, pacing, and conversational style while preserving the rules governing what it can do, what information it needs, and when it should escalate.

The objective is flexibility in conversation without unpredictability in execution.

6. Multiple voices don't mean multiple authorised users

A customer calls about an account issue while several people are speaking nearby.

Someone else asks the agent to disclose a balance or change an account detail.

The system may hear the request clearly, but understanding the words isn't the same as having permission to act on them.

How we approach it

We keep identity, authorisation, and workflow permissions separate from conversational understanding.

An instruction heard during a call shouldn't automatically be treated as an authorised request.

This is especially important for agents interacting with customer records, financial information, or other sensitive systems.

7. Latency is an end-to-end problem

Human conversation moves quickly.

Research by Levinson and Torreira (2015) describes conversational turn transitions that frequently occur on the order of a few hundred milliseconds.

For a voice agent, generating a response involves several stages:

Audio input → Speech recognition → Reasoning and tool execution → Speech synthesis → Audio output

Each stage introduces latency.

A fast speech recogniser doesn't guarantee a fast conversation if the reasoning model, synthesis engine, or network is slow.

How we approach it

We optimise the complete pipeline rather than individual model benchmarks alone.

Our approach focuses on streaming intermediate outputs, overlapping processing stages, reducing unnecessary network hops, and keeping inference components ready to serve requests.

We also distinguish between time to first audio and the time required to complete a response.

These measurements tell us different things about how the conversation feels to the caller.

8. Concurrency changes system behaviour

An agent might respond quickly during a demo with a single caller.

The same system can behave very differently when hundreds of conversations are active simultaneously.

GPU scheduling, memory utilisation, queueing, network connections, and downstream APIs all become potential bottlenecks.

How we approach it

We treat concurrency as an infrastructure problem, not simply a model-serving parameter.

Capacity needs to scale with demand while preserving acceptable response times.

We also pay attention to tail latency. A low average response time is less useful if a significant number of callers experience long pauses during peak traffic.

9. Calls can end before the task does

A caller is halfway through explaining a problem when the connection drops.

The conversation has ended, but the underlying task may still be unresolved.

If the agent's state exists only inside the call session, the customer may need to repeat everything when they call back.

How we approach it

We separate call state from persistent workflow state.

With Aethex PULSE, we're building agents that can maintain relevant execution context beyond an individual conversation.

That allows workflows to continue across interruptions, callbacks, and human handoffs without depending entirely on a single live session.

10. Deployments shouldn't interrupt conversations

Production voice agents are updated continuously.

Models change, prompts are refined, integrations evolve, and new functionality is introduced.

But those changes can happen while conversations are active.

Switching an agent's configuration halfway through a call risks inconsistent behaviour or broken sessions.

How we approach it

We design around stable session ownership and version consistency.

Active conversations should continue with the configuration they started with, while new sessions can use updated versions.

Workflow state also needs to survive unexpected failures so that infrastructure changes don't automatically mean lost customer progress.

Under the hood: designing for real-time voice

These ten problems share a common theme: the quality of a voice agent depends on how its components behave together.

A speech model can perform well on a benchmark while the complete system still produces a poor experience.

The engineering challenge is coordinating audio transport, inference, orchestration, and state management under real-time constraints.

Five principles guide our approach.

1. Stream rather than wait

Traditional request-response systems wait for one stage to finish before starting the next.

Real-time voice benefits from overlapping stages.

Speech recognition can process incoming audio incrementally. Reasoning can begin as useful context becomes available. Speech synthesis can start producing audio before the entire response is complete.

The goal is to reduce the time spent waiting between stages.

2. Minimise unnecessary network hops

Every additional service boundary introduces network latency and another potential failure point.

Because we operate our own speech models and inference infrastructure, we can make decisions about where components run and how they communicate.

This gives us more control over the trade-offs between model performance, infrastructure efficiency, and response latency.

3. Keep critical components ready

Cold starts are particularly noticeable in voice applications.

Loading a model after a call begins can introduce delays that would be acceptable in a background task but disruptive in a live conversation.

Keeping frequently used inference components ready reduces that risk, although it introduces a trade-off between resource utilisation and responsiveness.

4. Treat external tools as unreliable

A voice agent may need to query a database, retrieve customer information, or call an external API.

Those systems won't always respond quickly.

Rather than allowing a slow dependency to freeze the conversation, the agent needs appropriate timeout handling, retries where safe, and clear conversational feedback.

The voice experience should remain coherent even when a downstream service is slow.

5. Measure the worst experiences

Average latency is useful, but it doesn't describe the full distribution.

We care about metrics such as:

  • Time to first audio.

  • End-to-end response latency.

  • P50, P95, and P99 latency.

  • Interruption response time.

  • Failed or dropped sessions.

  • Performance under concurrent load.

These measurements help identify where callers experience delays and which parts of the system need attention.

The objective isn't to optimise for a single impressive benchmark number. It's to make response times predictable under real production conditions.

What we've learned

Reliable voice AI isn't built by solving speech recognition, reasoning, or synthesis independently.

It's built by making all of those components work together while dealing with unpredictable human behaviour and imperfect infrastructure.

A good demo proves that a model can hold a conversation.

A production system has to demonstrate much more: consistent latency, reliable turn-taking, appropriate security boundaries, recoverable workflows, and the ability to operate under load.

That's why we're investing in the complete stack at AethexAI, from Kora speech models to inference infrastructure and stateful agent execution.

The real benchmark isn't how an agent performs in a quiet room. It's how it performs when the conversation doesn't go according to plan.

To see how AethexAI handles real-world voice workflows, build an agent in Agent Studio or get in touch.

Nobody Talks Like the Demo: 10 Things Real Calls Throw at a Voice Agent

Voice AI demos usually work under ideal conditions. The speaker is clear, the background is quiet, the conversation follows a predictable path, and the agent responds almost immediately.

Production calls are different.

People interrupt, hesitate, switch languages, speak over background noise, lose connectivity, and ask questions the system wasn't expecting. Meanwhile, the infrastructure has to process audio, run inference, execute tools, and return speech quickly enough that the conversation still feels natural.

At AethexAI, we've found that building reliable voice agents is as much a systems engineering problem as it is a model problem.

Here are ten challenges that shape how we build and operate our voice infrastructure.

1. Background noise changes everything

A customer calls to check their account balance. They're standing beside a busy road, with traffic, other conversations, and intermittent noise competing with their voice.

The agent needs to recognise what they're saying without confusing background speech for the customer's instructions.

This is particularly challenging over telephone networks, where audio is often narrowband, compressed, and affected by packet loss.

How we approach it

Our Kora speech models are developed with real-world audio conditions in mind.

Rather than evaluating speech recognition only on clean recordings, we focus on performance across different acoustic environments, including noisy and degraded audio.

The challenge isn't simply removing noise. It's preserving the information that matters while avoiding false recognition of background speech.

2. People switch languages mid-sentence

A customer might say:

"I was trying to check my balance, pero la aplicación no funciona."

The sentence begins in English and ends in Spanish.

For a multilingual speech recogniser, this introduces several challenges. The model needs to recognise both languages, handle the transition, and preserve the meaning of the full utterance.

A language classifier that assumes one language per call can make the problem worse.

How we approach it

We design our speech systems around multilingual input and code-switching rather than requiring callers to select a language at the beginning of a conversation.

The important distinction is between recognising multiple languages independently and recognising them within the same utterance.

That difference becomes especially important when evaluating models on real conversations.

3. An interruption isn't always an interruption

Consider an agent explaining a payment schedule.

The customer says:

"Right... yeah... okay... mm-hmm."

Those responses indicate that they're listening.

Then the customer says:

"Wait, that's not what I meant."

Now the agent needs to stop speaking.

A simple voice activity detector cannot reliably distinguish between these situations. It can detect speech, but not necessarily whether the speaker intends to take the conversational turn.

How we approach it

We treat turn-taking as a separate problem from speech recognition.

The system needs to account for acoustic activity, timing, and conversational context when deciding whether to continue speaking, pause, or yield to the caller.

This matters because overly sensitive interruption handling makes agents feel jumpy, while insufficient sensitivity makes them difficult to interrupt.

4. Silence is part of conversation

"Give me a second, I'm looking for my account number."

The caller goes quiet.

A rigid timeout might interpret the silence as abandonment and end the conversation.

But the caller has already explained why they're silent.

How we approach it

Silence handling should depend on context rather than a single fixed timeout.

An agent waiting for a customer to retrieve information should behave differently from an agent waiting for an answer to a straightforward question.

We aim to balance responsiveness with patience, avoiding unnecessary prompts while still detecting when a conversation has stalled.

5. Natural conversation isn't a fixed script

Two customers can ask the same question in completely different ways.

One gets straight to the point. Another begins with greetings, explains the background, changes direction, and only then describes the problem.

A voice agent needs to accommodate both without losing track of the task.

How we approach it

We separate conversational behaviour from the underlying business workflow.

The agent can adapt its tone, pacing, and conversational style while preserving the rules governing what it can do, what information it needs, and when it should escalate.

The objective is flexibility in conversation without unpredictability in execution.

6. Multiple voices don't mean multiple authorised users

A customer calls about an account issue while several people are speaking nearby.

Someone else asks the agent to disclose a balance or change an account detail.

The system may hear the request clearly, but understanding the words isn't the same as having permission to act on them.

How we approach it

We keep identity, authorisation, and workflow permissions separate from conversational understanding.

An instruction heard during a call shouldn't automatically be treated as an authorised request.

This is especially important for agents interacting with customer records, financial information, or other sensitive systems.

7. Latency is an end-to-end problem

Human conversation moves quickly.

Research by Levinson and Torreira (2015) describes conversational turn transitions that frequently occur on the order of a few hundred milliseconds.

For a voice agent, generating a response involves several stages:

Audio input → Speech recognition → Reasoning and tool execution → Speech synthesis → Audio output

Each stage introduces latency.

A fast speech recogniser doesn't guarantee a fast conversation if the reasoning model, synthesis engine, or network is slow.

How we approach it

We optimise the complete pipeline rather than individual model benchmarks alone.

Our approach focuses on streaming intermediate outputs, overlapping processing stages, reducing unnecessary network hops, and keeping inference components ready to serve requests.

We also distinguish between time to first audio and the time required to complete a response.

These measurements tell us different things about how the conversation feels to the caller.

8. Concurrency changes system behaviour

An agent might respond quickly during a demo with a single caller.

The same system can behave very differently when hundreds of conversations are active simultaneously.

GPU scheduling, memory utilisation, queueing, network connections, and downstream APIs all become potential bottlenecks.

How we approach it

We treat concurrency as an infrastructure problem, not simply a model-serving parameter.

Capacity needs to scale with demand while preserving acceptable response times.

We also pay attention to tail latency. A low average response time is less useful if a significant number of callers experience long pauses during peak traffic.

9. Calls can end before the task does

A caller is halfway through explaining a problem when the connection drops.

The conversation has ended, but the underlying task may still be unresolved.

If the agent's state exists only inside the call session, the customer may need to repeat everything when they call back.

How we approach it

We separate call state from persistent workflow state.

With Aethex PULSE, we're building agents that can maintain relevant execution context beyond an individual conversation.

That allows workflows to continue across interruptions, callbacks, and human handoffs without depending entirely on a single live session.

10. Deployments shouldn't interrupt conversations

Production voice agents are updated continuously.

Models change, prompts are refined, integrations evolve, and new functionality is introduced.

But those changes can happen while conversations are active.

Switching an agent's configuration halfway through a call risks inconsistent behaviour or broken sessions.

How we approach it

We design around stable session ownership and version consistency.

Active conversations should continue with the configuration they started with, while new sessions can use updated versions.

Workflow state also needs to survive unexpected failures so that infrastructure changes don't automatically mean lost customer progress.

Under the hood: designing for real-time voice

These ten problems share a common theme: the quality of a voice agent depends on how its components behave together.

A speech model can perform well on a benchmark while the complete system still produces a poor experience.

The engineering challenge is coordinating audio transport, inference, orchestration, and state management under real-time constraints.

Five principles guide our approach.

1. Stream rather than wait

Traditional request-response systems wait for one stage to finish before starting the next.

Real-time voice benefits from overlapping stages.

Speech recognition can process incoming audio incrementally. Reasoning can begin as useful context becomes available. Speech synthesis can start producing audio before the entire response is complete.

The goal is to reduce the time spent waiting between stages.

2. Minimise unnecessary network hops

Every additional service boundary introduces network latency and another potential failure point.

Because we operate our own speech models and inference infrastructure, we can make decisions about where components run and how they communicate.

This gives us more control over the trade-offs between model performance, infrastructure efficiency, and response latency.

3. Keep critical components ready

Cold starts are particularly noticeable in voice applications.

Loading a model after a call begins can introduce delays that would be acceptable in a background task but disruptive in a live conversation.

Keeping frequently used inference components ready reduces that risk, although it introduces a trade-off between resource utilisation and responsiveness.

4. Treat external tools as unreliable

A voice agent may need to query a database, retrieve customer information, or call an external API.

Those systems won't always respond quickly.

Rather than allowing a slow dependency to freeze the conversation, the agent needs appropriate timeout handling, retries where safe, and clear conversational feedback.

The voice experience should remain coherent even when a downstream service is slow.

5. Measure the worst experiences

Average latency is useful, but it doesn't describe the full distribution.

We care about metrics such as:

  • Time to first audio.

  • End-to-end response latency.

  • P50, P95, and P99 latency.

  • Interruption response time.

  • Failed or dropped sessions.

  • Performance under concurrent load.

These measurements help identify where callers experience delays and which parts of the system need attention.

The objective isn't to optimise for a single impressive benchmark number. It's to make response times predictable under real production conditions.

What we've learned

Reliable voice AI isn't built by solving speech recognition, reasoning, or synthesis independently.

It's built by making all of those components work together while dealing with unpredictable human behaviour and imperfect infrastructure.

A good demo proves that a model can hold a conversation.

A production system has to demonstrate much more: consistent latency, reliable turn-taking, appropriate security boundaries, recoverable workflows, and the ability to operate under load.

That's why we're investing in the complete stack at AethexAI, from Kora speech models to inference infrastructure and stateful agent execution.

The real benchmark isn't how an agent performs in a quiet room. It's how it performs when the conversation doesn't go according to plan.

To see how AethexAI handles real-world voice workflows, build an agent in Agent Studio or get in touch.

Nobody Talks Like the Demo: 10 Things Real Calls Throw at a Voice Agent

Voice AI demos usually work under ideal conditions. The speaker is clear, the background is quiet, the conversation follows a predictable path, and the agent responds almost immediately.

Production calls are different.

People interrupt, hesitate, switch languages, speak over background noise, lose connectivity, and ask questions the system wasn't expecting. Meanwhile, the infrastructure has to process audio, run inference, execute tools, and return speech quickly enough that the conversation still feels natural.

At AethexAI, we've found that building reliable voice agents is as much a systems engineering problem as it is a model problem.

Here are ten challenges that shape how we build and operate our voice infrastructure.

1. Background noise changes everything

A customer calls to check their account balance. They're standing beside a busy road, with traffic, other conversations, and intermittent noise competing with their voice.

The agent needs to recognise what they're saying without confusing background speech for the customer's instructions.

This is particularly challenging over telephone networks, where audio is often narrowband, compressed, and affected by packet loss.

How we approach it

Our Kora speech models are developed with real-world audio conditions in mind.

Rather than evaluating speech recognition only on clean recordings, we focus on performance across different acoustic environments, including noisy and degraded audio.

The challenge isn't simply removing noise. It's preserving the information that matters while avoiding false recognition of background speech.

2. People switch languages mid-sentence

A customer might say:

"I was trying to check my balance, pero la aplicación no funciona."

The sentence begins in English and ends in Spanish.

For a multilingual speech recogniser, this introduces several challenges. The model needs to recognise both languages, handle the transition, and preserve the meaning of the full utterance.

A language classifier that assumes one language per call can make the problem worse.

How we approach it

We design our speech systems around multilingual input and code-switching rather than requiring callers to select a language at the beginning of a conversation.

The important distinction is between recognising multiple languages independently and recognising them within the same utterance.

That difference becomes especially important when evaluating models on real conversations.

3. An interruption isn't always an interruption

Consider an agent explaining a payment schedule.

The customer says:

"Right... yeah... okay... mm-hmm."

Those responses indicate that they're listening.

Then the customer says:

"Wait, that's not what I meant."

Now the agent needs to stop speaking.

A simple voice activity detector cannot reliably distinguish between these situations. It can detect speech, but not necessarily whether the speaker intends to take the conversational turn.

How we approach it

We treat turn-taking as a separate problem from speech recognition.

The system needs to account for acoustic activity, timing, and conversational context when deciding whether to continue speaking, pause, or yield to the caller.

This matters because overly sensitive interruption handling makes agents feel jumpy, while insufficient sensitivity makes them difficult to interrupt.

4. Silence is part of conversation

"Give me a second, I'm looking for my account number."

The caller goes quiet.

A rigid timeout might interpret the silence as abandonment and end the conversation.

But the caller has already explained why they're silent.

How we approach it

Silence handling should depend on context rather than a single fixed timeout.

An agent waiting for a customer to retrieve information should behave differently from an agent waiting for an answer to a straightforward question.

We aim to balance responsiveness with patience, avoiding unnecessary prompts while still detecting when a conversation has stalled.

5. Natural conversation isn't a fixed script

Two customers can ask the same question in completely different ways.

One gets straight to the point. Another begins with greetings, explains the background, changes direction, and only then describes the problem.

A voice agent needs to accommodate both without losing track of the task.

How we approach it

We separate conversational behaviour from the underlying business workflow.

The agent can adapt its tone, pacing, and conversational style while preserving the rules governing what it can do, what information it needs, and when it should escalate.

The objective is flexibility in conversation without unpredictability in execution.

6. Multiple voices don't mean multiple authorised users

A customer calls about an account issue while several people are speaking nearby.

Someone else asks the agent to disclose a balance or change an account detail.

The system may hear the request clearly, but understanding the words isn't the same as having permission to act on them.

How we approach it

We keep identity, authorisation, and workflow permissions separate from conversational understanding.

An instruction heard during a call shouldn't automatically be treated as an authorised request.

This is especially important for agents interacting with customer records, financial information, or other sensitive systems.

7. Latency is an end-to-end problem

Human conversation moves quickly.

Research by Levinson and Torreira (2015) describes conversational turn transitions that frequently occur on the order of a few hundred milliseconds.

For a voice agent, generating a response involves several stages:

Audio input → Speech recognition → Reasoning and tool execution → Speech synthesis → Audio output

Each stage introduces latency.

A fast speech recogniser doesn't guarantee a fast conversation if the reasoning model, synthesis engine, or network is slow.

How we approach it

We optimise the complete pipeline rather than individual model benchmarks alone.

Our approach focuses on streaming intermediate outputs, overlapping processing stages, reducing unnecessary network hops, and keeping inference components ready to serve requests.

We also distinguish between time to first audio and the time required to complete a response.

These measurements tell us different things about how the conversation feels to the caller.

8. Concurrency changes system behaviour

An agent might respond quickly during a demo with a single caller.

The same system can behave very differently when hundreds of conversations are active simultaneously.

GPU scheduling, memory utilisation, queueing, network connections, and downstream APIs all become potential bottlenecks.

How we approach it

We treat concurrency as an infrastructure problem, not simply a model-serving parameter.

Capacity needs to scale with demand while preserving acceptable response times.

We also pay attention to tail latency. A low average response time is less useful if a significant number of callers experience long pauses during peak traffic.

9. Calls can end before the task does

A caller is halfway through explaining a problem when the connection drops.

The conversation has ended, but the underlying task may still be unresolved.

If the agent's state exists only inside the call session, the customer may need to repeat everything when they call back.

How we approach it

We separate call state from persistent workflow state.

With Aethex PULSE, we're building agents that can maintain relevant execution context beyond an individual conversation.

That allows workflows to continue across interruptions, callbacks, and human handoffs without depending entirely on a single live session.

10. Deployments shouldn't interrupt conversations

Production voice agents are updated continuously.

Models change, prompts are refined, integrations evolve, and new functionality is introduced.

But those changes can happen while conversations are active.

Switching an agent's configuration halfway through a call risks inconsistent behaviour or broken sessions.

How we approach it

We design around stable session ownership and version consistency.

Active conversations should continue with the configuration they started with, while new sessions can use updated versions.

Workflow state also needs to survive unexpected failures so that infrastructure changes don't automatically mean lost customer progress.

Under the hood: designing for real-time voice

These ten problems share a common theme: the quality of a voice agent depends on how its components behave together.

A speech model can perform well on a benchmark while the complete system still produces a poor experience.

The engineering challenge is coordinating audio transport, inference, orchestration, and state management under real-time constraints.

Five principles guide our approach.

1. Stream rather than wait

Traditional request-response systems wait for one stage to finish before starting the next.

Real-time voice benefits from overlapping stages.

Speech recognition can process incoming audio incrementally. Reasoning can begin as useful context becomes available. Speech synthesis can start producing audio before the entire response is complete.

The goal is to reduce the time spent waiting between stages.

2. Minimise unnecessary network hops

Every additional service boundary introduces network latency and another potential failure point.

Because we operate our own speech models and inference infrastructure, we can make decisions about where components run and how they communicate.

This gives us more control over the trade-offs between model performance, infrastructure efficiency, and response latency.

3. Keep critical components ready

Cold starts are particularly noticeable in voice applications.

Loading a model after a call begins can introduce delays that would be acceptable in a background task but disruptive in a live conversation.

Keeping frequently used inference components ready reduces that risk, although it introduces a trade-off between resource utilisation and responsiveness.

4. Treat external tools as unreliable

A voice agent may need to query a database, retrieve customer information, or call an external API.

Those systems won't always respond quickly.

Rather than allowing a slow dependency to freeze the conversation, the agent needs appropriate timeout handling, retries where safe, and clear conversational feedback.

The voice experience should remain coherent even when a downstream service is slow.

5. Measure the worst experiences

Average latency is useful, but it doesn't describe the full distribution.

We care about metrics such as:

  • Time to first audio.

  • End-to-end response latency.

  • P50, P95, and P99 latency.

  • Interruption response time.

  • Failed or dropped sessions.

  • Performance under concurrent load.

These measurements help identify where callers experience delays and which parts of the system need attention.

The objective isn't to optimise for a single impressive benchmark number. It's to make response times predictable under real production conditions.

What we've learned

Reliable voice AI isn't built by solving speech recognition, reasoning, or synthesis independently.

It's built by making all of those components work together while dealing with unpredictable human behaviour and imperfect infrastructure.

A good demo proves that a model can hold a conversation.

A production system has to demonstrate much more: consistent latency, reliable turn-taking, appropriate security boundaries, recoverable workflows, and the ability to operate under load.

That's why we're investing in the complete stack at AethexAI, from Kora speech models to inference infrastructure and stateful agent execution.

The real benchmark isn't how an agent performs in a quiet room. It's how it performs when the conversation doesn't go according to plan.

To see how AethexAI handles real-world voice workflows, build an agent in Agent Studio or get in touch.

The voice AI stack for emerging markets.

See agents in production, configured for your use case, your market, your systems.

Screenshot of the Aethex logo

© 2026, AethexAI Inc.

The voice AI stack for emerging markets.

See agents in production, configured for your use case, your market, your systems.

Screenshot of the Aethex logo

© 2026, AethexAI Inc.

The voice AI stack for emerging markets.

See agents in production, configured for your use case, your market, your systems.