TELECALLER: A Real-Time AI Voice Calling and Monitoring Platform

Abstract

Telecaller is a real-time AI voice calling and monitoring platform designed to automate customer interaction through intelligent voice conversations over telephone networks. The system integrates Large Language Models (LLMs), Speech-to-Text (STT), and Text-to-Speech (TTS) technologies with Twilio-based telephony infrastructure to enable low-latency, bidirectional audio communication. Incoming call audio is streamed to a FastAPI backend through asynchronous WebSocket connections, transcribed using Deepgram STT, processed by OpenAI GPT-4o-mini and Groq Llama models for context-aware response generation, and converted into natural Hindi and English speech using Sarvam AI TTS. Ollama-based local models provide fallback support during cloud-service failures, improving platform reliability. The system also includes a web-based dashboard for AI agent management, knowledge base integration, call monitoring, analytics visualization, live transcript viewing, and audio recording management. A lightweight semantic retrieval module based on sentencetransformer embeddings supports knowledge-grounded responses during live conversations. The platform currently supports Hindi and English interactions and is designed modularly for future expansion into additional Indian regional languages.