Skip to main content

Voice Features API Reference

Overview​

The Voice Features API provides endpoints for managing conversations, processing audio files, automation rules, and handling processing workflows. The API is divided into three main controllers:

  • ConversationController: CRUD operations for conversations and folders
  • ConversationProcessingController: Audio processing, streaming, and service operations
  • AutomationController: Management of automation rules and agent workflows

Base URLs​

  • Conversations: /api/conversations
  • Processing Services: /api/conversations/service
  • Automations: /api/automations

Authentication​

All endpoints require:

  • Client ID Middleware: Validates client identification
  • CORS: Cross-origin resource sharing enabled
  • JWT Token: For user authentication (in res.locals.clientData)

ConversationController Endpoints​

Folder Management​

Get Conversation Tree​

POST /api/conversations/get-convos-and-folders-tree

Request Body:

{
  "folderType": "string"
}

Response:

{
  "folders": [
    {
      "id": 123,
      "name": "Meeting Recordings",
      "parent_id": null,
      "conversations": [
        {
          "id": "conv_uuid",
          "name": "Team Meeting Jan 2024",
          "timestamp": 1705934400,
          "duration": "00:30:47",
          "transcript": "Full transcription...",
          "user_id": "user_123",
          "runresultsreference": "result_uuid",
          "media_key": "audio/2024/01/meeting.mp3",
          "file_checksum": "sha256_hash",
          "folder_id": 123,
          "blob_source": "azure",
          "custom_info": {}
        }
      ]
    }
  ]
}

Create Folder​

POST /api/conversations/create-folder

Request Body:

{
  "folderName": "string",
  "parentId": "number|null",
  "folderType": "string"
}

Response:

{
  "id": 124,
  "name": "New Folder",
  "parent_id": null,
  "created_at": "2024-01-15T14:30:00Z"
}

Edit Folder​

PUT /api/conversations/edit-folder

Request Body:

{
  "folderId": 123,
  "folderName": "Updated Folder Name"
}

Response:

{
  "id": 123,
  "name": "Updated Folder Name",
  "updated_at": "2024-01-15T14:35:00Z"
}

File Management​

Move File​

POST /api/conversations/move-file

Request Body:

{
  "fileId": "conv_uuid",
  "targetFolderId": 123
}

Response:

{
  "success": true,
  "file": {
    "id": "conv_uuid",
    "folder_id": 123
  }
}

Delete Files and Folders​

POST /api/conversations/delete-items

Request Body:

{
  "conversations": ["conv_uuid1", "conv_uuid2"],
  "folders": [123, 124]
}

Response: 200 OK

Conversation CRUD Operations​

Get Conversation by ID​

GET /api/conversations/:id

Parameters:

  • id (path): Conversation UUID

Response:

{
  "id": "conv_uuid",
  "timestamp": 1705934400,
  "duration": "00:30:47",
  "transcript": "Complete transcription text",
  "custom_info": {
    "sentiment": "positive",
    "summary": "Meeting summary"
  },
  "name": "Team Meeting Jan 2024",
  "user_id": "user_123",
  "runresultsreference": "result_uuid",
  "media_key": "audio/2024/01/meeting.mp3",
  "file_checksum": "sha256_hash",
  "folder_id": 123,
  "blob_source": "azure"
}

Update Conversation​

PUT /api/conversations/:id

Parameters:

  • id (path): Conversation UUID

Request Body:

{
  "name": "Updated Meeting Title",
  "folder_id": 124,
  "custom_info": {}
}

Response:

{
  "success": "Conversation updated successfully"
}

Delete Conversation​

DELETE /api/conversations/:id

Parameters:

  • id (path): Conversation UUID

Response:

{
  "message": "Conversation deleted successfully"
}

Check Conversation Exists by Checksum​

POST /api/conversations/checksum

Request Body:

{
  "checksum": "a1b2c3d4e5f6..."
}

Response:

[
  {
    "id": "conv_uuid",
    "name": "Existing File",
    "file_checksum": "a1b2c3d4e5f6..."
  }
]

ConversationProcessingController Endpoints​

Audio Upload and Processing​

Confirm Audio Uploaded​

POST /api/conversations/service/confirm-upload

Request Body:

{
  "filename": "meeting_recording.mp3",
  "mediaKey": "audio/2024/01/meeting_abc123.mp3",
  "file_checksum": "a1b2c3d4e5f6...",
  "duration": "00:30:47",
  "size": 15728640,
  "contentType": "audio/mpeg",
  "folderId": 123
}

Response:

{
  "conversationId": "conv_uuid",
  "checksum": "a1b2c3d4e5f6..."
}

Process Conversation​

POST /api/conversations/service/processing

Request Body:

{
  "conversationId": "conv_uuid",
  "type": "audio",
  "origin": "meeting_uploads",
  "metadata": {
    "filename": "meeting_recording.mp3",
    "originalname": "Team Meeting Jan 2024.mp3",
    "mimeType": "audio/mpeg",
    "language": "en-US"
  },
  "source": "azure",
  "folder_id": 123,
  "provider": "azure_speech",
  "serviceModel": "standard"
}

Response:

{
  "taskId": "processing_uuid"
}

Media Access​

Get Conversation by Media Key​

POST /api/conversations/service/media

Request Body:

{
  "mediakey": "audio/2024/01/meeting_abc123.mp3"
}

Response:

{
  "id": "conv_uuid",
  "name": "meeting_recording.mp3",
  "mediakey": "audio/2024/01/meeting_abc123.mp3",
  "runresultsreference": "result_uuid",
  "processing": [
    {
      "processing_id": "proc_uuid",
      "status": "COMPLETED",
      "config": {
        "type": "audio",
        "origin": "meeting_uploads",
        "metadata": {
          "filename": "meeting_recording.mp3",
          "mimeType": "audio/mpeg",
          "language": "en-US"
        },
        "provider": "azure_speech",
        "serviceModel": "standard"
      },
      "timestamp": 1705934500
    }
  ]
}

Processing Tasks​

Get Processing Tasks​

POST /api/conversations/service/processing-tasks

Response:

[
  {
    "id": "proc_uuid",
    "status": "COMPLETED",
    "type": "CONVERSATION_PROCESSING",
    "timestamp": 1705934400,
    "clientId": "client_123",
    "payload": {
      "mediaKey": "audio/2024/01/meeting.mp3",
      "conversationId": "conv_uuid",
      "type": "audio",
      "origin": "meeting_uploads",
      "metadata": {
        "filename": "meeting.mp3",
        "mimeType": "audio/mpeg",
        "language": "en-US"
      },
      "source": "azure",
      "provider": "azure_speech"
    }
  }
]

Run Processing Task​

POST /api/conversations/service/run-processing-task

Request Body:

{
  "mediaKey": "audio/2024/01/meeting.mp3",
  "taskId": "proc_uuid",
  "config": {
    "type": "audio",
    "origin": "meeting_uploads",
    "metadata": {
      "filename": "meeting.mp3",
      "mimeType": "audio/mpeg",
      "language": "en-US"
    },
    "provider": "azure_speech",
    "serviceModel": "standard"
  }
}

Response: 200 OK

Results and Flow Data​

Get Flow IDs by Run Results​

GET /api/conversations/service/run-results/:runresultsreference/flow-ids

Parameters:

  • runresultsreference (path): Run results reference UUID

Response:

{
  "flows": ["sentiment_agent", "summary_agent", "action_items"],
  "failedFlows": ["failed_agent_id"]
}

Audio Streaming and Download​

Stream Audio​

GET /api/conversations/service/run-results/:runresultsreference/stream

Parameters:

  • runresultsreference (path): Run results reference UUID
  • quality (query): Audio quality - "low", "medium", "high" (default: "high")

Headers:

  • Range (optional): For partial content requests (e.g., "bytes=0-1023")

Response:

  • 206 Partial Content: When Range header is provided
  • 200 OK: Full audio stream
  • Content-Type: audio/mpeg or detected MIME type

Get Audio File URL​

GET /api/conversations/service/run-results/:runresultsreference/audio-url

Parameters:

  • runresultsreference (path): Run results reference UUID
  • quality (query): Audio quality - "low", "medium", "high" (default: "high")

Response:

{
  "audioFileUrl": "https://storage.blob.core.windows.net/container/file.mp3?sastoken",
  "quality": "high",
  "expiresIn": 3600
}

Download File by Run Results​

GET /api/conversations/service/run-results/:runresultsreference/download

Parameters:

  • runresultsreference (path): Run results reference UUID

Response:

  • Binary file download with appropriate Content-Type and Content-Disposition headers

AutomationController Endpoints​

Automation Management​

Create Automation Rule​

POST /api/automations

Request Body:

{
  "name": "Meeting Analysis Rule",
  "type": "audio",
  "origin": "meeting_uploads",
  "agents": ["sentiment_agent", "summary_agent", "action_items"],
  "is_active": true,
  "client_id": "client_123"
}

Response:

{
  "id": 1,
  "name": "Meeting Analysis Rule",
  "type": "audio",
  "origin": "meeting_uploads",
  "agents": ["sentiment_agent", "summary_agent", "action_items"],
  "is_active": true,
  "client_id": "client_123",
  "created_at": "2024-01-15T14:30:00Z",
  "last_update": "2024-01-15T14:30:00Z"
}

Update Automation Rule​

PATCH /api/automations/:id

Parameters:

  • id (path): Automation rule ID

Request Body:

{
  "name": "Updated Meeting Analysis",
  "agents": ["new_agent", "existing_agent"],
  "is_active": false
}

Response:

{
  "success": "Automation updated successfully"
}

Delete Automation Rule​

DELETE /api/automations/:id

Parameters:

  • id (path): Automation rule ID

Response:

{
  "message": "Automation deleted successfully"
}

Get All Automations​

GET /api/automations?page=1&limit=10

Query Parameters:

  • page (optional): Page number (default: 1)
  • limit (optional): Items per page (default: 10)

Response:

[
  {
    "id": 1,
    "name": "Meeting Analysis Rule",
    "type": "audio",
    "origin": "meeting_uploads",
    "agents": ["sentiment_agent", "summary_agent"],
    "is_active": true,
    "client_id": "client_123",
    "created_at": "2024-01-15T14:30:00Z",
    "last_update": "2024-01-15T14:30:00Z"
  }
]

Search Automations by Type and Origin​

GET /api/automations/search?origin=meeting_uploads&type=audio

Query Parameters:

  • origin (required): Origin pattern to match
  • type (required): Media type to match

Response:

[
  {
    "id": 1,
    "name": "Meeting Analysis Rule",
    "type": "audio",
    "origin": "meeting_uploads",
    "agents": ["sentiment_agent", "summary_agent"],
    "is_active": true,
    "client_id": "client_123",
    "created_at": "2024-01-15T14:30:00Z",
    "last_update": "2024-01-15T14:30:00Z"
  }
]

Data Models​

Core Schemas​

ConversationSchema​

interface ConversationSchema {
  id: string;                        // UUID identifier
  timestamp: number;                 // Unix timestamp
  duration: string;                  // Formatted duration (HH:MM:SS)
  transcript: string;                // Full transcription text
  custom_info: Record<string, any> | null; // Parsed agent results
  name: string;                      // Display name/filename
  user_id: string;                   // Owner identifier
  runresultsreference: string | null; // Link to AutomationRunResult
  media_key: string;                 // Storage identifier
  file_checksum: string;             // SHA-256 hash
  folder_id: number | null;          // Optional folder assignment
  blob_source: string;               // Storage backend type
}

Automation​

interface Automation {
  id: number;                        // Auto-incremented ID
  name: string;                      // Rule display name
  type: string;                      // Media type (e.g., "audio")
  origin: string;                    // Source identifier
  agents: string[];                  // LangFlow agent IDs
  last_update: Date;                 // Last modification time
  is_active: boolean;                // Rule enabled/disabled
  client_id: string;                 // Client ownership
}

AutomationRunResult​

interface AutomationRunResult {
  id: string;                        // UUID identifier
  file_reference: string;            // Media key reference
  task_results: TaskResults;         // Agent execution results
  created_at: Date;                  // Creation timestamp
}

interface TaskResults {
  flows: ProcessedFlowResult[];      // Successful agent executions
  failed_flows: ProcessedFlowResult[]; // Failed agent executions
  finalOutput: string;               // Combined or error output
}

interface ProcessedFlowResult {
  flowId: string;                    // Agent identifier
  text?: string;                     // Agent output text
  timestamp: number;                 // Execution timestamp
  error?: string;                    // Error message if failed
}

ConversationProcessingTask​

interface ConversationProcessingTask {
  id: string;                        // Processing UUID
  status: TaskStatus;                // PENDING, PROCESSING, COMPLETED, FAILED
  type: "CONVERSATION_PROCESSING";
  timestamp: number;                 // Creation timestamp
  clientId: string;                  // Client identifier
  payload: {
    mediaKey: string;                // Storage identifier
    folderId: number | null;         // Optional folder assignment
    source: StorageType;             // Storage backend
    conversationId: string;          // Conversation UUID
    type: string;                    // Media type
    origin: string;                  // Upload source
    metadata: AudioMetadata;         // Audio file metadata
    provider: string;                // Transcription provider
    serviceModel?: string;           // Provider-specific model
  };
}

AudioMetadata​

interface AudioMetadata {
  filename: string;                  // System filename
  originalname: string;              // User-provided name
  mimeType: string;                  // MIME type (audio/mpeg, etc.)
  language: string;                  // Language code (en-US, etc.)
}

WebSocket Notifications​

Real-Time Processing Updates​

The system provides real-time notifications via WebSocket for processing status updates:

Connection​

const socket = io('/conversation-processing');

Notification Types​

enum ConversationNotificationType {
  PENDING = "pending",                    // Task queued
  PREPARE = "prepare",                    // Task preparation
  START = "start",                        // Processing started
  DOWNLOAD = "download",                  // Media download
  TRANSCRIBE_START = "transcribe_start",  // Transcription started
  TRANSCRIBE_IN_PROGRESS = "transcribe_in_progress", // Transcription ongoing
  TRANSCRIBE = "transcribe",              // Transcription completed
  FETCH_AUTOMATIONS = "fetching-automations-up", // Fetching rules
  AUTOMATION_PROCESSING = "automation-processing", // Processing automations
  RUNNING_FLOWS = "running - flows",      // Executing agents
  SAVE_DB = "save-db",                   // Saving results
  COMPLETED = "completed",                // All processing done

  // Error notifications
  FILE_UPLOAD_ERROR = "file-upload-error",
  TRANSCRIPTION_ERROR = "transcription-error",
  AUTOMATION_FETCH_ERROR = "automation-fetch-error",
  AUTOMATION_ERROR = "automation-error"
}

Notification Structure​

interface ConversationProcessingNotificationData {
  type: ConversationNotificationType;
  status: "SUCCESS" | "FAIL" | "WARNING";
  resourceId: string;                // Media key
  conversationId?: string;           // Conversation UUID
  timestamp: Date;
  taskId?: string;                   // Processing task UUID
  extra?: {
    reason?: string;
    processing_completed?: boolean;
    is_processing?: boolean;
    error?: string;
    context?: string;
    message?: string;
  };
}

Example Notification Flow​

socket.on('conversation-processing', (data) => {
  switch (data.type) {
    case 'pending':
      updateUI('Queued for processing...');
      break;
    case 'transcribe_start':
      updateUI('Converting speech to text...');
      break;
    case 'running - flows':
      updateUI('Analyzing content...');
      break;
    case 'completed':
      if (data.status === 'SUCCESS') {
        updateUI('Processing completed successfully');
      } else {
        updateUI(`Processing failed: ${data.extra?.reason}`);
      }
      break;
  }
});

Error Responses​

Standard Error Format​

{
  "error": "string",
  "message": "string"
}

Common HTTP Status Codes​

  • 200 OK: Request successful
  • 201 Created: Resource created successfully
  • 206 Partial Content: Range request successful
  • 400 Bad Request: Invalid request payload or parameters
  • 401 Unauthorized: Missing or invalid authentication
  • 404 Not Found: Resource not found
  • 500 Internal Server Error: Server-side error

Processing Error Types​

type ConversationProcessingErrorType =
  | "fetch"                    // Database retrieval errors
  | "transcribe"              // Transcription service errors
  | "automation"              // Automation rule errors
  | "service-restart"         // System restart during processing
  | "capacity"                // Resource limitation errors
  | "download"                // File retrieval errors
  | "upload"                  // File upload errors
  | "save_db"                 // Database save errors
  | "running_flows"           // Agent execution errors
  | "automation_processing"   // Automation processing errors
  | "aborted"                 // User-initiated cancellation
  | "unhandled_error"         // Unexpected errors
  | "other";                  // Miscellaneous errors

Error Examples​

Validation Error​

{
  "error": "Invalid payload or non-audio file uploaded"
}

Authentication Error​

{
  "error": "Unauthorized: clientId missing"
}

Not Found Error​

{
  "error": "Conversation not found by id"
}

Processing Error​

{
  "error": "Failed to process conversation",
  "message": "Transcription service timeout"
}

Request/Response Examples​

Complete Audio Processing Workflow​

1. Confirm Upload​

POST /api/conversations/service/confirm-upload Content-Type: application/json

{
  "filename": "team_meeting_jan_2024.mp3",
  "mediaKey": "audio/2024/01/team_meeting_abc123.mp3",
  "file_checksum": "a1b2c3d4e5f6789abcdef1234567890abcdef1234567890abcdef1234567890ab",
  "duration": "01:15:30",
  "size": 45000000,
  "contentType": "audio/mpeg",
  "folderId": 123
}

Response:

{
  "conversationId": "550e8400-e29b-41d4-a716-446655440000",
  "checksum": "a1b2c3d4e5f6789abcdef1234567890abcdef1234567890abcdef1234567890ab"
}

2. Start Processing​

POST /api/conversations/service/processing Content-Type: application/json

{
  "conversationId": "550e8400-e29b-41d4-a716-446655440000",
  "type": "audio",
  "origin": "meeting_uploads",
  "metadata": {
    "filename": "team_meeting_jan_2024.mp3",
    "originalname": "Team Meeting January 2024.mp3",
    "mimeType": "audio/mpeg",
    "language": "en-US"
  },
  "source": "azure",
  "folder_id": 123,
  "provider": "azure_speech",
  "serviceModel": "latest"
}

Response:

{
  "taskId": "proc_440e8400-e29b-41d4-a716-446655440001"
}

3. Monitor Processing (WebSocket)​

socket.on('conversation-processing', (notification) => {
  console.log('Processing update:', notification);
  // Example notifications received during processing:
  // { type: 'pending', status: 'SUCCESS', resourceId: 'audio/2024/01/team_meeting_abc123.mp3' }
  // { type: 'transcribe_start', status: 'SUCCESS', resourceId: 'audio/2024/01/team_meeting_abc123.mp3' }
  // { type: 'running - flows', status: 'SUCCESS', resourceId: 'audio/2024/01/team_meeting_abc123.mp3' }
  // { type: 'completed', status: 'SUCCESS', resourceId: 'audio/2024/01/team_meeting_abc123.mp3' }
});

4. Get Final Results​

GET /api/conversations/550e8400-e29b-41d4-a716-446655440000

Response:

{
  "id": "550e8400-e29b-41d4-a716-446655440000",
  "timestamp": 1705934400,
  "duration": "01:15:30",
  "transcript": "Welcome everyone to today's team meeting. Let's start with our quarterly review...",
  "custom_info": {
    "sentiment": "positive",
    "summary": "Quarterly review meeting covering performance metrics and upcoming goals",
    "action_items": [
      "Update project timeline by Friday",
      "Schedule follow-up meeting with client",
      "Review budget allocation for Q2"
    ],
    "participants": ["Alice", "Bob", "Charlie"],
    "key_topics": ["quarterly review", "performance metrics", "budget planning"]
  },
  "name": "team_meeting_jan_2024.mp3",
  "user_id": "user_123",
  "runresultsreference": "result_660e8400-e29b-41d4-a716-446655440002",
  "media_key": "audio/2024/01/team_meeting_abc123.mp3",
  "file_checksum": "a1b2c3d4e5f6789abcdef1234567890abcdef1234567890abcdef1234567890ab",
  "folder_id": 123,
  "blob_source": "azure"
}

5. Access Audio Stream​

GET /api/conversations/service/run-results/result_660e8400-e29b-41d4-a716-446655440002/stream?quality=high

Rate Limiting and Performance​

Concurrency Control​

  • Default Concurrent Tasks: 1 per service instance
  • Configurable Limits: Adjustable via maxConcurrentTasks
  • Queue Management: FIFO processing with client isolation
  • Resource Monitoring: Active task count tracking

File Size Limitations​

  • Maximum File Size: Configurable per storage backend
  • Streaming Support: Large file processing via streaming
  • Timeout Handling: Configurable processing timeouts per stage

Performance Optimization​

  • Direct URL Transcription: When supported by storage and transcription service
  • Blob Caching: Efficient storage access patterns
  • Parallel Agent Execution: Multiple agents process simultaneously
  • Progressive Notifications: Real-time status updates reduce polling

Security Considerations​

Data Protection​

  • Client Isolation: Complete data separation between clients
  • File Integrity: SHA-256 checksums for all uploaded files
  • Secure Storage: Encrypted storage backends with access controls
  • Temporary URLs: Time-limited SAS tokens for secure file access

Access Control​

  • Conversation Ownership: Users can only access their own conversations
  • Automation Rules: Client-specific rule execution
  • Processing Tasks: Client-scoped task management
  • Audit Logging: Complete activity tracking for security

API Security​

  • Input Validation: Comprehensive payload validation
  • Error Handling: Secure error messages without information leakage
  • Authentication: JWT-based user authentication
  • CORS Configuration: Proper cross-origin request handling

SDK Integration Examples​

JavaScript/TypeScript Client​

class VoiceFeaturesClient {
  constructor(private baseUrl: string, private authToken: string) {}

  async uploadAndProcess(audioFile: File, folderId?: number) {
    // 1. Calculate checksum
    const checksum = await this.calculateChecksum(audioFile);

    // 2. Upload to storage (implementation depends on storage backend)
    const mediaKey = await this.uploadToStorage(audioFile);

    // 3. Confirm upload
    const { conversationId } = await this.confirmUpload({
      filename: audioFile.name,
      mediaKey,
      file_checksum: checksum,
      duration: await this.getAudioDuration(audioFile),
      size: audioFile.size,
      contentType: audioFile.type,
      folderId
    });

    // 4. Start processing
    const { taskId } = await this.startProcessing({
      conversationId,
      type: "audio",
      origin: "web_upload",
      metadata: {
        filename: audioFile.name,
        originalname: audioFile.name,
        mimeType: audioFile.type,
        language: "en-US"
      },
      source: "azure",
      folder_id: folderId,
      provider: "azure_speech"
    });

    return { conversationId, taskId };
  }

  async getProcessingStatus(taskId: string) {
    return await this.fetch(`/api/conversations/service/processing-tasks`, {
      method: 'POST'
    });
  }

  async streamAudio(runResultsReference: string, quality = 'high') {
    return await this.fetch(
      `/api/conversations/service/run-results/${runResultsReference}/stream?quality=${quality}`
    );
  }
}

This comprehensive API reference provides complete documentation for integrating with the Voice Features system, including all endpoints, data models, error handling, and real-time notifications.