Smart Voice AI Pipeline

---

​

What You’re Building

Build an intelligent Pipecat voice AI pipeline that goes beyond basic conversation. Your pipeline will process a live voice call with our Shilpa agent (Kotak Securities demat account opening) and add four real-time capabilities on top:

1. Human Escalation : Detect when the caller needs a human agent and execute a live SIP transfer via FreeSWITCH
2. Gender Detection : Infer the caller’s gender from the conversation transcript in real-time and make it available as pipeline metadata
3. Language Detection : Detect the caller’s language (Hindi, English, code-mixed) from the transcript in real-time and make it available as pipeline metadata
4. Prompt Optimization : Reduce latency and cost through one or more of: prompt compression, faster tool calling, or RAG-based dynamic knowledge injection

---

​

Pipeline Requirements

​

Base Pipeline

Build a working Pipecat pipeline from scratch with:

• STT (any provider)
• LLM (Gemini or any provider)
• TTS (any provider)
• The Shilpa agent prompt loaded and working

​

Agent

The agent is Shilpa (Kotak Securities demat account opening).

---

​

The Four Capabilities

​

1. Human Escalation

Detect in real-time when the conversation should be handed off to a human agent, then execute the transfer via FreeSWITCH SIP. Detection triggers (at minimum):

• Caller explicitly asks to speak to a human / manager / supervisor
• Caller expresses extreme frustration or anger (repeated objections, raised voice cues in transcript)
• Conversation is stuck in a loop (agent repeating itself, caller not progressing)
• Agent is unable to answer a question outside its domain

When escalation is triggered, the agent should inform the caller, execute a SIP transfer via FreeSWITCH ESL, and log the escalation reason, turn number, and transcript up to that point.

​

2. Gender Detection

Infer the caller’s likely gender from the conversation transcript in real-time using NLP/LLM analysis. Use linguistic cues — name mentions, pronoun usage, Hindi gendered verb forms (e.g., “मैं करता हूँ” vs “मैं करती हूँ”). Produce a classification (male, female, unknown) with a confidence score that updates as the conversation progresses.

​

3. Language Detection

Detect the caller’s language in real-time from the transcript. Classify each caller turn as hindi, english, code-mixed, or other. Track the dominant language across the conversation. Handle edge cases like single-word responses, and numbers-only responses.

​

4. Prompt Optimization

Reduce LLM latency and/or cost through intelligent prompt engineering at the pipeline level. Implement one or more of: prompt compression (reduce token count while preserving instruction fidelity), fast tool calling (parallel execution, caching, speculative selection), or RAG-based knowledge injection (index agent knowledge into a vector store, retrieve relevant chunks per turn instead of stuffing the full prompt).

---

​

Judging Criteria

---

​

Rules

1. Individual work only this is a solo competition
2. Claude Code is allowed and encouraged use it aggressively
3. Any programming language Python required (Pipecat is Python)
4. Any STT/TTS/LLM provider you provision your own keys (Gemini is available)
5. No pre-written code start from scratch at the hackathon start
6. Final demo: 5-minute live demo run a conversation showing all four capabilities in action

---