Voice Control You Can Rely On
Developing a reusable engine and framework for voice-controlled applications, with AI interpretation kept inside defined workflows.
- Industry
- AI, Voice Technology
2
Speech interpretations reconciled for every command
0
Repeat text-to-speech calls for pre-generated prompts
Summary:
We developed a reusable engine and framework for building applications controlled through voice, combining configurable workflows, AI speech interpretation and an audio pipeline designed to reduce repeated speech generation costs. To show what the framework can do, we also built a mobile app that demonstrates navigation, content search, option selection and playback control through speech.
The Challenge:
Building a dependable voice application means balancing flexibility with control. Our framework had to address three needs:
- Stay on script: keep AI-interpreted speech within the actions a workflow actually permits.
- Understand reliably: resolve unclear or conflicting interpretations of the same utterance.
- Control speech costs: avoid paying for a new text-to-speech call every time a familiar prompt plays.
The Solution: Flexible Workflows with Controlled AI
A configurable state machine defines each step, its permitted actions, transitions and help. AI interprets the user’s speech, and the engine validates the resulting command against the current workflow before acting.
- Focused interaction: validation limits off-topic interaction and reduces the opportunity for prompt injection to redirect the workflow beyond its defined actions.
- Automatic help: help prompts are generated from the available commands.
- Change detection: configuration checks identify changes that require audio regeneration.
Reliable Voice Interaction
The framework reconciles two interpretations of the same speech: a Realtime model’s proposed action and a separately transcribed message. Confidence-based rules resolve disagreements, with a fallback classifier for unclear input.
- Voice controls: selection from options, semantic search, repeat and contextual help.
- Pause behavior: controls move from active playback to a listening pause and then to a manual-resume state.
- Audio feedback: sound effects provide feedback and timing cues.
Reusable Speech with Automated Quality Checks
Full prompts and constant parts of responses are generated in advance and reused, avoiding a new TTS API call each time they play. The pipeline also checks its own work:
- Automated QA: generated speech is transcribed automatically and compared with the intended text, and failed clips are regenerated.
- Better short prompts: a custom technique generates longer speech and extracts the required phrase using word timestamps.
- Prompt delivery: our own WebRTC streaming server reuses persistent connections to deliver audio promptly.
Delivery and Technologies
Deployment tooling covers code updates, data imports, audio assets, validation, backups and rollback.
- Framework: Python and FastAPI, Node.js, OpenAI Realtime and speech services, Whisper, WebRTC and aiortc, and FFmpeg.
- Data and infrastructure: MongoDB, Redis and Qdrant, with Docker Compose and GitHub Actions.
- Demo app: React Native and Expo.
Conclusions
Good voice experiences come from combining AI flexibility with firm engineering guardrails. By validating every command against a defined workflow, reconciling speech interpretations and reusing quality-checked audio, we built a framework that makes voice applications more reliable, more affordable to run and faster to start.