Skip to main content
Hackathon Date: March 31, 2026 • 2:00 PM - 7:00 PM IST Format: Individual

​
What You’re Building

Build an intelligent Pipecat voice AI pipeline that goes beyond basic conversation. Your pipeline will process a live voice call with our Shilpa agent (Kotak Securities demat account opening) and add four real-time capabilities on top:
  1. Human Escalation : Detect when the caller needs a human agent and execute a live SIP transfer via FreeSWITCH
  2. Gender Detection : Infer the caller’s gender from the conversation transcript in real-time and make it available as pipeline metadata
  3. Language Detection : Detect the caller’s language (Hindi, English, code-mixed) from the transcript in real-time and make it available as pipeline metadata
  4. Prompt Optimization : Reduce latency and cost through one or more of: prompt compression, faster tool calling, or RAG-based dynamic knowledge injection

​
Pipeline Requirements

​
Base Pipeline

Build a working Pipecat pipeline from scratch with:
  • STT (any provider)
  • LLM (Gemini or any provider)
  • TTS (any provider)
  • The Shilpa agent prompt loaded and working

​
Agent

The agent is Shilpa (Kotak Securities demat account opening).

​
The Four Capabilities

​
1. Human Escalation

Detect in real-time when the conversation should be handed off to a human agent, then execute the transfer via FreeSWITCH SIP. Detection triggers (at minimum):
  • Caller explicitly asks to speak to a human / manager / supervisor
  • Caller expresses extreme frustration or anger (repeated objections, raised voice cues in transcript)
  • Conversation is stuck in a loop (agent repeating itself, caller not progressing)
  • Agent is unable to answer a question outside its domain
When escalation is triggered, the agent should inform the caller, execute a SIP transfer via FreeSWITCH ESL, and log the escalation reason, turn number, and transcript up to that point.

​
2. Gender Detection

Infer the caller’s likely gender from the conversation transcript in real-time using NLP/LLM analysis. Use linguistic cues — name mentions, pronoun usage, Hindi gendered verb forms (e.g., “मैं करता हूँ” vs “मैं करती हूँ”). Produce a classification (male, female, unknown) with a confidence score that updates as the conversation progresses.

​
3. Language Detection

Detect the caller’s language in real-time from the transcript. Classify each caller turn as hindi, english, code-mixed, or other. Track the dominant language across the conversation. Handle edge cases like single-word responses, and numbers-only responses.

​
4. Prompt Optimization

Reduce LLM latency and/or cost through intelligent prompt engineering at the pipeline level. Implement one or more of: prompt compression (reduce token count while preserving instruction fidelity), fast tool calling (parallel execution, caching, speculative selection), or RAG-based knowledge injection (index agent knowledge into a vector store, retrieve relevant chunks per turn instead of stuffing the full prompt).

​
Judging Criteria

CriteriaWeightWhat We’re Looking For
Gender & Language Detection30%Do detectors produce correct results in real-time? Handle edge cases — code-mixed utterances, Romanized Hindi (“haan bolo”), ambiguous gender, single-word responses? Does confidence improve with more turns? Clean processor design with proper error handling and logging?
Human Escalation30%Does the detector trigger correctly. no false positives on mild complaints, no misses on explicit requests like “get me a manager”? Does the SIP transfer actually execute via FreeSWITCH? Does the agent deliver a smooth handoff message before transfer? Is the escalation logged with reason, transcript, and metadata (gender, language)?
Prompt Optimization20%Is there a measurable improvement in latency, token count, or cost? Is response quality preserved after compression/RAG? Is the approach technically sound not just truncating the prompt but intelligently reducing it? Before/after metrics shown?
Ambition & Polish20%How far beyond the core requirements did you go? Dynamic STT/TTS language switching? Escalation context handoff to human agent screen? Real-time dashboard? Prompt compression + RAG combined? Does the overall system feel production-ready?

​
Rules

  1. Individual work only this is a solo competition
  2. Claude Code is allowed and encouraged use it aggressively
  3. Any programming language Python required (Pipecat is Python)
  4. Any STT/TTS/LLM provider you provision your own keys (Gemini is available)
  5. No pre-written code start from scratch at the hackathon start
  6. Final demo: 5-minute live demo run a conversation showing all four capabilities in action
Good luck.