A deep dive into RapidSOS’s intelligent non-emergency automation
Special thanks to Piotr Stachowicz, AI Engineering Lead at RapidSOS, for partnering on this great story!
Problem
Emergency dispatchers are the backbone of public safety, yet they are increasingly overwhelmed by administrative volume. Nationwide, up to 70% of calls to 911 centers are non-emergency requests – such as noise complaints. The stakes are even higher during high-density events, such as New Year’s Eve, where call spikes can increase by over 300%. To solve this RapidSOS, the leading public safety AI company, deployed a purpose-built voice agent designed to handle non-emergency calls with high autonomy. This tech allows highly-trained 911 telecommunicators to focus on life-saving emergencies with the oversight required for public safety environments.
What we built
RapidSOS developed HARMONY AI, an intelligence layer built specifically for the emergency ecosystem. HARMONY acts as an AI co-pilot within the 911 center, synthesizing data from 700M+ connected devices to automate routine tasks and provide instant incident insights.
Unlike traditional voice bots, this system dynamically guides callers through agency-specific protocols, verifies critical location data, and automatically packages the information for downstream field processing.
The stack & solution
To meet the high stakes of emergency services, we moved beyond simple speech-to-text flows. By leveraging Gemini Live API on Gemini Enterprise Agent Platform, we built a sophisticated agentic architecture that can handle the instantaneous complexity of human dialogue in real time.
-
Gemini 2.5 Flash Live API: This audio-to-audio model powers the conversational interface with robust function calling. Because it is natively multimodal, it delivers low-latency voice interactions that better understand tone, emphasis, and intent. It seamlessly deciphers acoustic nuances (like pauses or background noise), ensuring the conversational experience is fluid and highly responsive.
-
Google Agent Development Kit (ADK): This provides the framework for our core conversational loop. It allows us to manage complex conversation states and ensures high observability as the agent moves through various stages of a report.
-
The “supervisor” model pattern: While the native audio model handles the fluid conversation, we use a secondary, high-reasoning Gemini model (gemini-2.5-flash) to observe the audio and transcripts in parallel. This supervisor acts as a fail-safe, verifying that the classification of a non-emergency is accurate before the agent proceeds.
Under the hood
The core of the RapidSOS solution is a repeatable loop designed for high-stakes accuracy. While the Gemini 2.5 Flash Live API manages the active dialogue, the supervisor model acts as a real-time “watchdog” to solve two critical challenges:
-
Dual-model “voter” Mechanism: To manage accidental emergency calls on non-emergency lines, we employ a multi-layered fail-safe system. This involves cross-referencing a library of critical keywords (ie. hurt, gun, bleeding) derived from historical 911 data with an analysis of the caller’s emotional state and general sentiment. If the AI identifies a potential emergency, it immediately captures a callback number for transfer. Our system is intentionally biased toward rapid escalation to ensure safety over conservatism.
-
Parallel Verification - human in the loop: The supervisor model monitors the live audio and transcripts in parallel to the main interaction. If it detects a shift in the caller’s tone or a specific “emergency” keyword, it can override the flow and trigger an immediate escalation to a human dispatcher.
-
Self-Healing Conversations: RapidSOS’s architecture uses the supervisor model to monitor for stalls or awkward pauses. If a gap is detected, the supervisor “nudges” the primary agent to resume the flow, mimicking the way humans naturally self-heal a conversation when there is a lapse in speech.
The architecture utilizes a “supervisor” pattern to act as a fail-safe.
Technical wins
-
High Completion Rate: Successfully handles 60-70% of non-emergency calls without human intervention.
-
Sub-100ms Performance: Optimized the “core loop” to ensure conversation remains fluid.
-
Safety-First Reliability: The dual-model “Voter” pattern successfully filtered out false positives, ensuring only true non-emergencies were handled by the AI.
What we learned
Treating voice AI as a production-grade safety tool requires moving beyond “happy path” interactions. Here is our advice for developers building in high-stakes environments:
-
Orchestrate a multi-model pipeline for high reliability: The biggest takeaway for mission-critical voice automation is that a single model rarely solves the entire problem. We achieved maximum reliability by combining specialized Google Cloud tools within the call:
-
Gemini 2.5 Flash Live API Used as the primary interface to handle fluid dialogue.
-
Chirp 3: Deployed as a parallel engine for near-real-time, stable transcription.
-
Gemini 2.5 Flash (with Thinking Budget): Utilized while the call is active for high-reasoning final evaluations and guardrail verification.
-
-
Instrument for audio underruns, not just average latency: Relying on average latency metrics can easily obscure the primary failure mode in voice AI – choppy or jittery audio that disrupts the caller experience. We deployed specialized internal watchdogs specifically to monitor for audio underruns, ensuring a smooth, uninterrupted conversation stream.
-
Detect caller silence at the audio layer, not the transcript layer: Transcript-based silence detection fails whenever a caller makes non-transcribable noise, such as heavy breathing, background chatter, or line glitches. Detecting silence directly at the audio layer is far more reliable for natural turn-taking.
-
Use a parallel transcriber: Maintain a second, stable transcript stream to decouple “what the model heard” from the data you feed to your supervisor model for safety verification.
-
Focus on Logic, Not Infrastructure: By leveraging the managed services on Gemini Enterprise Agent Platform, we completely removed the burden of server management and autoscaling.
If you’re building something similar be sure to:
Explore Gemini Enterprise Agent Platform
Check out the Google Agent Development Kit (ADK)
Thought starter for readers:
- We focused on noise complaints and fireworks for the Reno test. What other ‘high-volume, low-criticality’ workflows in your industry are ripe for native-audio automation?