Theme — Build something that makes a customer forget they’re talking to an AI.A common piece of client feedback is that our agents still feel robotic: they complete the task, but customers can tell they’re talking to a bot. The challenge isn’t making the agent smarter — it’s making the interaction feel human. By the end, we want a clearer answer to one question: what makes a customer forget they’re talking to an AI?
The Challenge
Improve the perceived humanness of the Tala Hindi collections agent. The goal is not a better script — it’s a more human conversational experience.You may modify
- System prompts & prompt structure
- Behavioral instructions
- Memory & state management
- Conversation planning & context management
- Dialogue policies & response strategies
- Few-shot examples & evaluation frameworks
- Model layer: RAG, fine-tuning
You may not modify
- The underlying foundation models
- STT or TTS providers
- Core collections business logic
- Compliance requirements
Start with a Hypothesis
Every participant begins by choosing one specific hypothesis about what makes conversation feel human. A narrow, well-tested idea is worth more than a broad pile of random tweaks — the goal is to learn something about human conversation, not just to improve the agent.Human Conversation Dimensions
Your solution should improve one or more of these:Listening
Demonstrates understanding without parroting the customer’s words.
Adaptiveness
Changes behavior based on the customer’s situation, tone, and responses.
Conversational Flow
Natural rather than scripted — smooth transitions, pacing, and turn-taking.
Social Intelligence
Adjusts tone, formality, warmth, and language appropriately.
Memory & Continuity
Remembers and uses relevant context from earlier in the conversation.
Repair & Recovery
Handles interruptions, misunderstandings, and ASR errors gracefully.
Strategic Effectiveness
Moves toward a productive outcome without sounding robotic or aggressive.
Things we want less of
Repeating customer responses · generic empathy statements · long monologues · repetitive phrasing · overly formal language · scripted transitions · one-size-fits-all behavior · massive phrase banks and hardcoded branching.We’re looking for principles that generalize, not larger scripts.
Constraints
Performance & practicality. Naturalness must not come at the cost of severe latency, excessive verbosity, or impractical operational complexity. Solutions should remain realistic for production deployment at scale.
Evaluation
The primary objective is agent naturalness. Submissions are evaluated on how natural the agent feels during interactions — the specific techniques, tools, or approaches used to get there earn no extra points. Only the end result and overall conversational naturalness count.
Hardship
Recently lost their job and cannot pay the full amount.
Hostile
Frustrated and confrontational.
Evasive
Avoids committing to any payment date.
Cooperative
Willing to engage but needs flexibility.
Judging Criteria
How We Evaluate with Okareo
We measure naturalness with Okareo, which runs automated, recorded phone calls against your agent using a fixed set of customer personas — so every submission faces identical conversations.How a simulation runs
1
Automated call
Okareo places a live phone call to your agent’s number.
2
Synthetic caller
A simulated caller plays a fixed persona — a defined personality, situation, and goal — and holds a real multi-turn conversation in Hindi/Hinglish.
3
Capture
Audio, transcript, and trace are recorded together on a single timeline.
4
Scoring
Each call is graded by the automated checks below; naturalness is additionally scored by a blind human listening test.
Personas
Alongside the four scored customer personas (Hardship, Hostile, Evasive, Cooperative), five operational probes run as a correctness gate: Busy-in-intro, Wrong person, Already paid, STT name variant, and Silence / gibberish.Checks
Naturalness (scored) — applied to the four customer personas:
Integrity — applied to every call (the first three are disqualifying):
Operational routing — one per operational persona:
Disqualification & scoring
For non-disqualified submissions, each check feeds the judging criterion shown in its Feeds column. Pass/Fail components score as the percentage of passing calls; a check that a given call doesn’t exercise returns N/A and is excluded from the math (it neither helps nor hurts).Running Your Own Simulations
The Okareo workspace is a shared account used by all participants (engineers contribute individually).1
Share your number
Provide a phone number wired to your agent.
2
We verify
We run an initial simulation to confirm the setup connects and runs cleanly, with no issues.
3
Run it yourself
Once verified, trigger simulations on your own from the shared Okareo account and review the recording, transcript, and scorecard for each run.
Because the account is shared, label your runs with your name so results stay easy to find, and avoid editing the shared personas or checks.
Timeline
Demo Format
Each participant has 5 minutes.1
Hypothesis (1 min)
What theory about human conversation are you testing?
2
Approach (1 min)
What changes did you make, and why?
3
Before vs. After (2 min)
Show the baseline and the improved behavior on the same scenario.
4
Results & Learning (1 min)
Share evidence or observations, and complete the sentence: “We learned that conversations feel more human when ______.”
Winning
The winning project won’t necessarily be the most sophisticated one. It will be the one that teaches us something important about human conversation — a principle that can improve every Awaaz AI agent we build.