Voice Module

Add real-time AI voice calls to your app. Talk to an AI agent, get live transcription, and control the microphone — all through WebRTC.

Overview

XpectrumVoice lets users have a real-time voice conversation with an AI agent. Under the hood it uses LiveKit (WebRTC) for low-latency audio streaming.

How it works

  1. Your app calls voice.connect()
  2. SDK requests a LiveKit token from the Xpectrum API
  3. SDK connects to a LiveKit room via WebRTC
  4. User's microphone is enabled (browser permission prompt)
  5. Agent audio plays automatically, transcriptions stream in real-time
  6. On disconnect, SDK notifies the server to end the call

1. Setup

typescript
import { XpectrumVoice } from 'xpectrum';

const voice = new XpectrumVoice({
  baseUrl: 'https://cloud.xpectrum.dev/v1',  // Same API base URL as chat
  apiKey: 'xpectrum_...',                         // Your app's API key
});
PropertyTypeRequiredDefaultDescription
baseUrlstringYesXpectrum API base URL — the same one used for chat.
apiKeystringYesYour app's API key. The voice agent is determined by this key.

Same credentials as chat

Voice uses the same baseUrl and API key as the chat endpoints — no separate voice configuration. The voice agent is determined by the API key, so there is nothing else to select.

Requires voice access

Calls connect only when Voice calls is enabled in the app's publish settings; otherwise connect() fails with 403 Forbidden. Ending a call is always allowed, so a live call can hang up cleanly even if voice is switched off mid-call.

2. Start a Voice Call

Call voice.connect() with callbacks to handle events:

typescript
await voice.connect({
  // Called when connection is established
  onConnected: (roomName) => {
    console.log('Call started! Room:', roomName);
    showCallUI();
  },

  // Called for every transcription update
  onTranscription: (segment) => {
    // segment.id      — unique ID (interim results share same ID)
    // segment.text    — transcribed text
    // segment.isFinal — true when transcription is finalized
    // segment.speaker — 'user' or 'agent'
    console.log(`${segment.speaker}: ${segment.text}`);
  },

  // Called when agent starts/stops speaking
  onAgentSpeaking: (isSpeaking) => {
    updateStatus(isSpeaking ? 'Agent speaking...' : 'Listening...');
  },

  // The agent's live audio stream — for audio-reactive visuals.
  // Feed it to a Web Audio AnalyserNode to drive a waveform or orb.
  onAgentAudio: (stream) => {
    startVisualizer(stream);
  },

  // Called when call ends
  onDisconnected: (reason) => {
    console.log('Call ended:', reason);
    hideCallUI();
  },

  // Called when connection state changes
  onConnectionStateChanged: (state) => {
    // 'disconnected' | 'connecting' | 'connected' | 'reconnecting' | 'failed'
    console.log('State:', state);
  },

  // Network reconnection events
  onReconnecting: () => showStatus('Reconnecting...'),
  onReconnected: () => showStatus('Reconnected!'),

  // Error handler
  onError: (err) => {
    console.error('Voice error:', err.message);
  },
});

Browser permission

The browser will prompt the user for microphone permission when connect() is called. This must be triggered by a user interaction (e.g. button click) — it can't happen automatically on page load.

3. Understanding Transcription

The onTranscription callback receives a TranscriptionSegment:

PropertyTypeRequiredDefaultDescription
idstringYesSegment ID. Interim results update the same ID, so you can replace in-place.
textstringYesThe transcribed text.
isFinalbooleanYestrue when the transcription is finalized and won't change.
speaker'user' | 'agent'YesWho said this — the user or the AI agent.

How interim results work

text
onTranscription: { id: "seg-1", text: "What",           isFinal: false, speaker: "user" }
onTranscription: { id: "seg-1", text: "What are your",   isFinal: false, speaker: "user" }
onTranscription: { id: "seg-1", text: "What are your hours?", isFinal: true, speaker: "user" }

Notice the same id is reused for interim updates. When building your UI, find the existing segment by ID and replace it — don't append duplicates.

React pattern for transcription

typescript
const [transcripts, setTranscripts] = useState<TranscriptionSegment[]>([]);

// In your connect callbacks:
onTranscription: (segment) => {
  setTranscripts(prev => {
    const idx = prev.findIndex(t => t.id === segment.id);
    if (idx >= 0) {
      // Update existing segment (interim → final)
      const updated = [...prev];
      updated[idx] = segment;
      return updated;
    }
    // New segment
    return [...prev, segment];
  });
},

4. Microphone Control

typescript
// Mute the mic
await voice.setMicEnabled(false);

// Unmute the mic
await voice.setMicEnabled(true);

// Check current state
const isMuted = !voice.isMicEnabled();

5. End a Call

typescript
await voice.disconnect();
// This:
//  1. Disconnects from the LiveKit room
//  2. Stops the microphone
//  3. Cleans up audio elements
//  4. Notifies the server via POST /voice/call-control/end-call

6. Connection State

typescript
// Get current state
const state = voice.getConnectionState();
// 'disconnected' | 'connecting' | 'connected' | 'reconnecting' | 'failed'

// Quick check
if (voice.isConnected()) {
  console.log('Currently in a call');
}

// Get room name
const roomName = voice.getRoomName(); // string | null
StateMeaning
disconnectedNo active call
connectingRequesting token and joining LiveKit room
connectedIn an active voice call
reconnectingTemporary network issue, trying to reconnect
failedConnection failed permanently

7. Event Emitter (Alternative to Callbacks)

Besides passing callbacks to connect(), you can subscribe to events directly:

typescript
// Subscribe — returns an unsubscribe function
const unsub = voice.on('transcription', (segment) => {
  console.log(`${segment.speaker}: ${segment.text}`);
});

// Later: unsubscribe
unsub();

// Or use off()
const handler = (segment) => { ... };
voice.on('transcription', handler);
voice.off('transcription', handler);

// Remove all listeners
voice.removeAllListeners();

All events

EventPayloadDescription
connected{ roomName: string }Call connected
disconnected{ reason: string }Call ended
transcriptionTranscriptionSegmentSpeech transcribed
agentSpeaking{ isSpeaking: boolean }Agent started/stopped speaking
agentAudio{ stream: MediaStream }The agent's live audio — feed it to a Web Audio analyser for audio-reactive visuals (waveforms, orbs)
connectionStateChanged{ state: VoiceConnectionState }State transition
reconnecting{}Network reconnecting
reconnected{}Network reconnected
error{ message: string; code?: string }Error occurred
microphoneChanged{ enabled: boolean }Mic toggled

8. Cleanup

typescript
// Disconnect + remove all event listeners
voice.destroy();

// In React / Next.js:
useEffect(() => {
  const voice = new XpectrumVoice({ ... });
  voiceRef.current = voice;
  return () => voice.destroy();
}, []);

Full Working Example

VoiceCall.tsx
'use client';

import { useEffect, useRef, useState } from 'react';
import { XpectrumVoice } from 'xpectrum';
import type { TranscriptionSegment } from 'xpectrum';

export default function VoiceCall() {
  const voiceRef = useRef<XpectrumVoice | null>(null);
  const [state, setState] = useState<string>('disconnected');
  const [micOn, setMicOn] = useState(true);
  const [transcripts, setTranscripts] = useState<TranscriptionSegment[]>([]);

  useEffect(() => {
    voiceRef.current = new XpectrumVoice({
      baseUrl: process.env.NEXT_PUBLIC_XPECTRUM_BASE_URL!,  // same as chat
      apiKey: process.env.NEXT_PUBLIC_XPECTRUM_API_KEY!,
    });
    return () => voiceRef.current?.destroy();
  }, []);

  const startCall = async () => {
    setTranscripts([]);
    await voiceRef.current?.connect({
      onConnected: () => setState('connected'),
      onDisconnected: () => setState('disconnected'),
      onConnectionStateChanged: (s) => setState(s),
      onTranscription: (seg) => {
        setTranscripts(prev => {
          const idx = prev.findIndex(t => t.id === seg.id);
          if (idx >= 0) {
            const updated = [...prev];
            updated[idx] = seg;
            return updated;
          }
          return [...prev, seg];
        });
      },
      onError: (err) => console.error(err.message),
    });
  };

  const endCall = async () => {
    await voiceRef.current?.disconnect();
  };

  const toggleMic = async () => {
    const next = !micOn;
    await voiceRef.current?.setMicEnabled(next);
    setMicOn(next);
  };

  const isConnected = state === 'connected';

  return (
    <div style={{ maxWidth: 500, margin: '0 auto', padding: 20 }}>
      <h1>Voice Call</h1>
      <p>Status: <strong>{state}</strong></p>

      <div style={{ display: 'flex', gap: 8, marginBottom: 20 }}>
        {!isConnected ? (
          <button onClick={startCall}>Start Call</button>
        ) : (
          <>
            <button onClick={endCall}>End Call</button>
            <button onClick={toggleMic}>
              {micOn ? 'Mute' : 'Unmute'}
            </button>
          </>
        )}
      </div>

      <div>
        {transcripts.map((t) => (
          <p key={t.id} style={{ opacity: t.isFinal ? 1 : 0.6 }}>
            <strong>{t.speaker}:</strong> {t.text}
          </p>
        ))}
      </div>
    </div>
  );
}