Display live transcripts

1. Integrate the toolkit

   Copy the convoaiApi folder to your project and import the toolkit before calling the toolkit API. Refer to Folder structure to understand the role of each file.

2. Create a toolkit instance

   Create a configuration object with the Video SDK and Signaling engine instances. Set the transcript rendering mode, then use the configuration to create a toolkit instance.

   // Create configuration objects for the RTC and RTM instances
   val config = ConversationalAIAPIConfig(
       rtcEngine = rtcEngineInstance,
       rtmClient = rtmClientInstance,
   
       // Set the transcript rendering mode. Options:
       // - TranscriptRenderMode.Word: render transcript word by word.
       // - TranscriptRenderMode.Text: render the full sentence at once.
       renderMode = TranscriptRenderMode.Word,
       enableLog = true
   )
   
   // Create component instance
   val api = ConversationalAIAPIImpl(config)

3. Subscribe to the channel

   Transcript data is delivered through Signaling channel messages. To receive transcript data, call subscribeMessage before starting the agent session.

   api.subscribeMessage("channelName") { error ->
       if (error != null) {
           // Handle error
       }
   }

4. Receive transcript

   Call the addHandler method to register your implementation of the transcription callback.

   api.addHandler(covEventHandler)

5. Implement UI rendering logic

   Inherit your UI module from the IConversationalAIAPIEventHandler interface. Implement the onTranscriptUpdated method to handle transcript rendering to the UI.

   private val covEventHandler = object : IConversationalAIAPIEventHandler {
       override fun onTranscriptUpdated(agentUserId: String, transcript: Transcript) {
           // Handle transcript data and update the UI here
       }
   }

6. Add a conversational AI agent to the channel

   To start a conversational AI agent, configure the following parameters in your POST request:

   Parameter	Description	Required
   advanced_features.enable_rtm: true	Starts the Signaling service	Yes
   parameters.data_channel: "rtm"	Enables Signaling as the data transmission channel	Yes
   parameters.enable_metrics: true	Enables agent performance data collection	Optional
   parameters.enable_error_message: true	Enables reporting of agent error events	Optional

   After a successful response, the agent joins the specified RTC channel and is ready to interact with the user.

7. Unsubscribe from the channel

   After an agent session ends, unsubscribe from channel messages to release transcription resources.

   api.unsubscribeMessage("channelName") { error ->
       if (error != null) {
           // Handle the error
       }
   }

8. Release resources

   At the end of each call, use the destroy method to clean up the cache.

   api.destroy()

1. Integrate the toolkit

   Copy the ConversationalAIAPI folder to your project and import the toolkit before calling the toolkit APIs. Refer to Folder structure to understand the role of each file.

2. Create a toolkit instance

   Create a configuration object with the Video SDK and Signaling engine instances. Set the transcript rendering mode, then use the configuration to create a toolkit instance.

   // Create a configuration object for the RTC and RTM instances
   let config = ConversationalAIAPIConfig(
       rtcEngine: rtcEngine,
       rtmEngine: rtmEngine,
       /**
        * Set the transcript rendering mode. Available options:
        * - .words: Word-by-word rendering mode.
        * - .text: Sentence-by-sentence rendering mode.
        */
       renderMode: .words,
       enableLog: true
   )
   
   // Create the component instance
   convoAIAPI = ConversationalAIAPIImpl(config: config)

3. Subscribe to the channel

   Transcript data is delivered through Signaling channel messages. To receive transcript data, call subscribeMessage before starting the agent session.

   convoAIAPI.subscribeMessage(channelName: channelName) { error in
       if let error = error {
           print("Subscription failed: \(error.message)")
       } else {
           print("Subscription successful")
       }
   }

4. Receive transcript

   Call the addHandler method to register and implement the transcript callback.

   convoAIAPI.addHandler(handler: self)

5. Implement UI rendering logic

   Implement the ConversationalAIAPIEventHandler protocol in your UI module, and use onTranscriptUpdated to handle and render transcript updates.

   extension ChatViewController: ConversationalAIAPIEventHandler {
       public func onTranscriptUpdated(agentUserId: String, transcript: Transcript) {
           // Handle transcript data and update the UI here
       }
   }

6. Add a conversational AI agent to the channel

   To start a conversational AI agent, configure the following parameters in your POST request:

   Parameter	Description	Required
   advanced_features.enable_rtm: true	Starts the Signaling service	Yes
   parameters.data_channel: "rtm"	Enables Signaling as the data transmission channel	Yes
   parameters.enable_metrics: true	Enables agent performance data collection	Optional
   parameters.enable_error_message: true	Enables reporting of agent error events	Optional

   After a successful response, the agent joins the specified RTC channel and is ready to interact with the user.

7. Unsubscribe from the channel

   After each agent session ends, unsubscribe from channel messages to release transcript-related resources.

   convoAIAPI.unsubscribeMessage(channelName: channelName) { error in
       if let error = error {
           print("Unsubscription failed: \(error.message)")
       } else {
           print("Unsubscribed successfully")
       }
   }

8. Release resources

   At the end of each call, use the destroy method to clean up the cache.

   convoAIAPI.destroy()

1. Integrate the toolkit

   Copy the conversational-ai-api file to your project and import the toolkit before calling its APIs. Refer to Folder structure to understand the role of each file.

2. Create a toolkit instance

   Before joining an RTC channel, create Video SDK and Signaling engine instances and pass them into the toolkit instance.

   // Initialize the component
   ConversationalAIAPI.init({
     rtcEngine,
     rtmEngine,
   
     /**
      * Set the rendering mode for transcript. Available options:
      * - ESubtitleHelperMode.WORD: render transcript word by word.
      * - ESubtitleHelperMode.TEXT: render the full transcript at once.
      */
     renderMode: ESubtitleHelperMode.WORD,
   })
   
   // Get the API instance (singleton)
   const conversationalAIAPI = ConversationalAIAPI.getInstance()

3. Set audio parameters

   In word-by-word rendering mode, you must receive audio timestamp metadata from RTC to synchronize subtitles with speech. Before creating the client object, configure the following parameter:

   AgoraRTC.setParameter('ENABLE_AUDIO_PTS_METADATA', true)
   const client = AgoraRTC.createClient({ mode: 'rtc', codec: 'vp8' })

4. Subscribe to the channel

   Agent-related events are delivered through Signaling messages. Before starting an agent session, call subscribeMessage to receive these events:

   conversationalAIAPI.subscribeMessage(channel_name)

5. Receive transcript

   Register an event listener to receive transcript updates:

   import * as React from 'react'
   import {
     type IUserTranscription,
     type IAgentTranscription,
     type ISubtitleHelperItem,
     EConversationalAIAPIEvents,
   } from '@/conversational-ai-api/type'
   import { ConversationalAIAPI } from '@/conversational-ai-api'
   
   export const ChatHistory = () => {
     const [chatHistory, setChatHistory] = React.useState<
       ISubtitleHelperItem<Partial<IUserTranscription | IAgentTranscription>>[]
     >([])
   
     const conversationalAIAPI = ConversationalAIAPI.getInstance()
     conversationalAIAPI.on(
       EConversationalAIAPIEvents.TRANSCRIPT_UPDATED,
       setChatHistory
     )
   
     return (
       <>
         {chatHistory.map((message) => (
           <div key={`${message.uid}-${message.turn_id}`}>
             {message.uid}: {message.text}
           </div>
         ))}
       </>
     )
   }

6. Add a conversational AI agent to the channel

   To start a conversational AI agent, configure the following parameters in your POST request:

   Parameter	Description	Required
   advanced_features.enable_rtm: true	Starts the Signaling service	Yes
   parameters.data_channel: "rtm"	Enables Signaling as the data transmission channel	Yes
   parameters.enable_metrics: true	Enables agent performance data collection	Optional
   parameters.enable_error_message: true	Enables reporting of agent error events	Optional

   After a successful response, the agent joins the specified RTC channel and is ready to interact with the user.

7. Unsubscribe from the channel

   After each agent session ends, unsubscribe from channel messages to release resources associated with callback events.

   conversationalAIAPI.unsubscribeMessage(channel_name)

8. Release resources

   At the end of each call, use the destroy method to clean up the cache.

   conversationalAIAPI.destroy()

Folder structure

• IConversationalAIAPI.kt: API interface and related data structures and enumerations
• ConversationalAIAPIImpl.kt: ConversationalAI API main implementation logic
• ConversationalAIUtils.kt: tool functions and event callback management
• subRender/
  • v3/: transcription module
  • TranscriptionController.kt: transcription controller
  • MessageParser.kt: message parser

API reference

• addHandler
• subscribeMessage
• unsubscribeMessage
• destroy
• onTranscriptUpdated

Folder structure

• ConversationalAIAPI.swift: API interface and related data structures and enumerations
• ConversationalAIAPIImpl.swift: ConversationalAI API main implementation logic
• Transcription/
  • TranscriptionController.swift: transcription controller

API reference

• addHandler
• subscribeMessage
• unsubscribeMessage
• destroy
• onTranscriptUpdated

Folder structure

• index.ts: API class
• type.ts: API interface and related data structures and enumerations
• utils/
  • index.ts: API utility functions
  • events.ts: event management class
  • sub-render.ts: transcription module

API reference

• IConversationalAIAPIEventHandler interface
• EConversationalAIAPIEvents
• subscribeMessage
• unsubscribeMessage
• destroy