Send images to the agent

When interacting with an agent, you may need to upload images or send image messages from the client to help the agent better understand the user's intent. This page describes how to use the Conversational AI Engine toolkit to send image messages to the large language model from your app. The LLM can then automatically reference the image content in subsequent conversations and generate more relevant responses.

Understand the tech

Agora provides a flexible, scalable, and standardized conversational AI engine toolkit. The toolkit supports iOS, Android, and Web platforms, and encapsulates scenario-based APIs. You can use these APIs to integrate the capabilities of the Agora Signaling SDK and Agora Video SDK to enable the following features:

Interrupt agents
Display live transcripts
Client-side events
Set optimal audio parameters for iOS and Android
Send picture messages

Call the toolkit's chat API to send a picture message, and listen to onMessageReceiptUpdated to receive the picture message receipt.

Prerequisites

Before you begin, ensure the following:

You have implemented the Conversational AI Engine quickstart.
Your app integrates Agora Video SDK v4.5.1 or later and includes the Video SDK quickstart.
You have enabled Signaling in the Agora Console and completed the Signaling quickstart for basic messaging.
You maintain active and authenticated RTC and Signaling instances that persist beyond the component lifecycle. The toolkit does not manage RTC or Signaling initialization, lifecycle, or authentication.

Info

Picture messaging is currently in beta and free for a limited time.
Image processing depends on the capabilities of the integrated LLM. Make sure the LLM you connect to Conversational AI Engine supports image input.

Implementation

This section explains how to send a picture message from your app.

Integrate the toolkit

Copy the convoaiApi folder to your project and import the toolkit before calling the toolkit API. Refer to Folder structure to understand the role of each file.

Create a toolkit instance

Create a configuration object with the Video SDK and Signaling engine instances. Use the configuration to create a toolkit instance.

// Create configuration objects for the RTC and RTM instances
val config = ConversationalAIAPIConfig(
    rtcEngine = rtcEngineInstance,
    rtmClient = rtmClientInstance,
    enableLog = true
)

// Create component instance
val api = ConversationalAIAPIImpl(config)

Register callback
```
api.addHandler(covEventHandler)
```
Subscribe to the channel

Agent-related events are delivered through Signaling channel messages. To receive these events, call subscribeMessage before starting the agent session.
```
api.subscribeMessage("channelName") { error ->
    if (error != null) {
        // Handle error
    }
}
```

Add a conversational AI agent to the channel

To start a conversational AI agent, configure the following parameters in your POST request:

Parameter	Description	Required
`advanced_features.enable_rtm: true`	Starts the Signaling service	Yes
`parameters.data_channel: "rtm"`	Enables Signaling as the data transmission channel	Yes
`parameters.enable_metrics: true`	Enables agent performance data collection	Optional
`parameters.enable_error_message: true`	Enables reporting of agent error events	Optional

After a successful response, the agent joins the specified RTC channel and is ready to interact with the user.

Send an image

val uuid = "unique-image-id-123"
val imageUrl = "https://example.com/image.jpg"

api.chat("agentUserId", ImageMessage(uuid = uuid, imageUrl = imageUrl)) { error ->
    if (error != null) {
        Log.e("Chat", "Failed to send image: ${error.errorMessage}")
    } else {
        Log.d("Chat", "Image send request successful")
    }
}

Info

The chat completion callback only indicates whether the sending request is successful, and does not reflect the actual processing status of the message.

Handle image sending status

Image send success is confirmed by onMessageReceiptUpdated. If sending fails, onMessageError is triggered. Use the uuid value in the callback to identify the uploaded picture.

Image sent successfully:

override fun onMessageReceiptUpdated(agentUserId: String, receipt: MessageReceipt) {
    if (receipt.chatMessageType == ChatMessageType.Image) {
        try {
            val json = JSONObject(receipt.message)
            if (json.has("uuid")) {
                val receivedUuid = json.getString("uuid")
                if (receivedUuid == "your-sent-uuid") {
                    Log.d("ImageSend", "Image sent successfully: $receivedUuid")
                }
            }
        } catch (e: Exception) {
            Log.e("ImageSend", "Failed to parse message receipt: ${e.message}")
        }
    }
}

Image sending failed:

override fun onMessageError(agentUserId: String, error: MessageError) {
    if (error.chatMessageType == ChatMessageType.Image) {
        try {
            val json = JSONObject(error.message)
            if (json.has("uuid")) {
                val failedUuid = json.getString("uuid")
                if (failedUuid == "your-sent-uuid") {
                    Log.e("ImageSend", "Image send failed: $failedUuid")
                }
            }
        } catch (e: Exception) {
            Log.e("ImageSend", "Failed to parse error message: ${e.message}")
        }
    }
}

Unsubscribe from the channel

api.unsubscribeMessage("channelName") { error ->
    if (error != null) {
        // Handle the error
    }
}

Release resources
```
api.destroy()
```

Integrate the toolkit

Copy the ConversationalAIAPI folder to your project and import the toolkit before calling the toolkit APIs. Refer to Folder structure to understand the role of each file.

Create a toolkit instance

Create a configuration object with the Video SDK and Signaling engine instances, then use the configuration to create a toolkit instance.

let config = ConversationalAIAPIConfig(
    rtcEngine: rtcEngine,
    rtmEngine: rtmEngine,
    enableLog: true
)

convoAIAPI = ConversationalAIAPIImpl(config: config)

Subscribe to the channel

Agent-related events are delivered through Signaling channel messages. To receive these events, call subscribeMessage before starting the agent session.

convoAIAPI.subscribeMessage(channelName: channelName) { error in
    if let error = error {
        print("Subscription failed: \(error.message)")
    } else {
        print("Subscription successful")
    }
}

Register callback
```
convoAIAPI.addHandler(handler: self)
```

Add a conversational AI agent to the channel

To start a conversational AI agent, configure the following parameters in your POST request:

Parameter	Description	Required
`advanced_features.enable_rtm: true`	Starts the Signaling service	Yes
`parameters.data_channel: "rtm"`	Enables Signaling as the data transmission channel	Yes
`parameters.enable_metrics: true`	Enables agent performance data collection	Optional
`parameters.enable_error_message: true`	Enables reporting of agent error events	Optional

Send an image

let uuid = UUID().uuidString
let imageUrl = "https://example.com/image.jpg"
let message = ImageMessage(uuid: uuid, url: imageUrl)
self.convoAIAPI.chat(agentUserId: "\(agentUid)", message: message) { [weak self] error in
    if let error = error {
        print("send image failed, error: \(error.message)")
    } else {
        print("send image success")
    }
}

Info

The chat completion callback only indicates whether the sending request is successful, and does not reflect the actual processing status of the message.

Receive image sending response

Success is confirmed by onMessageReceiptUpdated. If sending fails, onMessageError is triggered.

Image sent successfully:

struct PictureInfo: Codable {
    let uuid: String
}

public func onMessageReceiptUpdated(agentUserId: String, messageReceipt: MessageReceipt) {
    if messageReceipt.type == .context {
        guard let messageData = messageReceipt.message.data(using: .utf8) else {
            return
        }
        do {
            let imageInfo = try JSONDecoder().decode(PictureInfo.self, from: messageData)
            self.messageView.viewModel.updateImageMessage(uuid: imageInfo.uuid, state: .success)
        } catch {
            print("Failed to decode PictureInfo: \(error)")
        }
    }
}

Image sending failed:

struct ImageUploadError: Codable {
    let code: Int
    let message: String
}

struct ImageUploadErrorResponse: Codable {
    let uuid: String
    let success: Bool
    let error: ImageUploadError?
}

public func onMessageError(agentUserId: String, error: MessageError) {
    if let messageData = error.message.data(using: .utf8) {
        do {
            let errorResponse = try JSONDecoder().decode(ImageUploadErrorResponse.self, from: messageData)
            if !errorResponse.success {
                DispatchQueue.main.async { [weak self] in
                    self?.messageView.viewModel.updateImageMessage(uuid: errorResponse.uuid, state: .failed)
                }
            }
        } catch {
            addLog("Failed to parse error message JSON: \(error)")
        }
    }
}

Unsubscribe from the channel

convoAIAPI.unsubscribeMessage(channelName: channelName) { error in
    if let error = error {
        print("Unsubscription failed: \(error.message)")
    } else {
        print("Unsubscribed successfully")
    }
}

Release resources
```
convoAIAPI.destroy()
```

Integrate the toolkit

Copy the conversational-ai-api file to your project and import the toolkit before calling its API. Refer to Folder structure to understand the role of each file.
Create a toolkit instance

Before joining an RTC channel, create Video SDK and Signaling engine instances and pass them into the toolkit instance.
```
ConversationalAIAPI.init({
    rtcEngine,
    rtmEngine,
})

const conversationalAIAPI = ConversationalAIAPI.getInstance()
```
Subscribe to the channel

Agent-related events are delivered through Signaling messages. Before starting an agent session, call subscribeMessage to receive these events.
```
conversationalAIAPI.subscribeMessage(channel_name)
```

Register callbacks

conversationalAIAPI.on(EConversationalAIAPIEvents.MESSAGE_RECEIPT_UPDATED, handleMessageReceiptUpdated)
conversationalAIAPI.on(EConversationalAIAPIEvents.MESSAGE_ERROR, onAgentError)

Add a conversational AI agent to the channel

To start a conversational AI agent, configure the following parameters in your POST request:

Parameter	Description	Required
`advanced_features.enable_rtm: true`	Starts the Signaling service	Yes
`parameters.data_channel: "rtm"`	Enables Signaling as the data transmission channel	Yes
`parameters.enable_metrics: true`	Enables agent performance data collection	Optional
`parameters.enable_error_message: true`	Enables reporting of agent error events	Optional

Send an image

import { EChatMessageType } from '@/conversational-ai-api/type'

await conversationalAIAPI.chat(`${agent_rtc_uid}`, {
    messageType: EChatMessageType.IMAGE,
    url: 'https://example.com/image.jpg',
    uuid: genUUID()
})

Info

The chat completion callback only indicates whether the sending request is successful, and does not reflect the actual processing status of the message.

Receive image sending response

Image sent successfully:

conversationalAIAPI.on(EConversationalAIAPIEvents.MESSAGE_RECEIPT_UPDATED, (agentUserId: string, messageReceipt: TMessageReceipt) => {
    if (messageReceipt.moduleType !== EModuleType.CONTEXT) {
        return
    }
    try {
        const receiptMessage = JSON.parse(messageReceipt.message)
        const uuid = receiptMessage.uuid
        if (!uuid) return
        console.log(`Message sent successfully, UUID: ${uuid}`)
    } catch (error) {
        console.error('Failed to parse message:', error)
    }
})

Image sending failed:

conversationalAIAPI.on(EConversationalAIAPIEvents.MESSAGE_ERROR, (agentUserId, error) => {
    console.error(`Message error for agent ${agentUserId}:`, error)
    if (error.type === EChatMessageType.IMAGE) {
        try {
            const errorData = JSON.parse(error.message)
            if (errorData?.uuid) {
                console.warn(`Image error for agent ${agentUserId} with UUID: ${errorData.uuid}`)
            }
        } catch (e) {
            console.error(`Failed to handle image error for agent ${agentUserId}:`, e)
        }
    }
})

Unsubscribe from the channel

conversationalAIAPI.unsubscribeMessage(channel_name)

Release resources
```
conversationalAIAPI.destroy()
```

Reference

This section contains supporting information that completes the guidance on this page.

Folder structure

IConversationalAIAPI.kt: API interface, related data structures, and enumerations
ConversationalAIAPIImpl.kt: main implementation logic
ConversationalAIUtils.kt: utility functions and callback management
v3/
- TranscriptionController.kt: transcript rendering and synchronization
- MessageParser.kt: transcription and message parsing

API reference

Folder structure

ConversationalAIAPI.swift: API interface, data structures, and enumerations
ConversationalAIAPIImpl.swift: main implementation logic
Transcription/
- TranscriptionController.swift: transcript rendering and transcription control

API reference

Folder structure

index.ts: main API class
type.ts: API interfaces, data structures, and enumerations
utils/
- index.ts: general utility functions
- events.ts: event management
- sub-render.ts: transcript rendering module

API reference

Send images to the agent

Understand the tech

Prerequisites

Implementation

Reference

Contact Us

Contact Us