# Custom audio source (/en/realtime-media/voice/build/customize-audio-processing/custom-audio)
> For AI agents: see the complete documentation index at [llms.txt](/llms.txt).
<_PlatformTabsGroup groupMode="structured" canonicalPlatform="web" platforms="["android","ios","macos","web","windows"]" showTabs="true">
<_PlatformPanel platform="android">
<_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="android" />
The default audio module of Voice SDK meets the need of using basic audio functions in your app. For adding advanced audio functions, Voice SDK supports using custom audio sources and custom audio rendering modules.
Video SDK uses the basic audio module on the device your app runs on by default. However, there are certain use-cases where you want to integrate a custom audio source into your app, such as:
* Your app has its own audio module.
* You need to process the captured audio with a pre-processing library for audio enhancement.
* You need flexible device resource allocation to avoid conflicts with other services.
This page shows you how to capture and render audio from custom sources.
## Understand the tech [#understand-the-tech]
To set an external audio source, you configure the Agora Engine before joining a channel. To manage the capture and processing of audio frames, you use methods from outside the Video SDK that are specific to your custom source. Video SDK enables you to push processed audio data to the subscribers in a channel.
#### Capture custom audio [#capture-custom-audio]
The following figure illustrates the process of custom audio capture.
* You implement the capture module using external methods provided by the SDK.
* You call `pushExternalAudioFrame` to send the captured audio frames to the SDK.
#### Render custom audio [#render-custom-audio]
The following figure illustrates the process of custom audio rendering.
* You implement the rendering module using external methods provided by the SDK.
* You call `pullPlaybackAudioFrame` to retrieve the audio data sent by remote users.
## Prerequisites [#prerequisites]
Ensure that you have implemented the [SDK quickstart](../index.mdx) in your project.
## Implementation [#implementation]
This section shows you how to implement custom audio capture and render audio from a custom source.
### Custom audio capture [#custom-audio-capture]
Refer to the following call sequence diagram to implement custom audio capture in your app:
**Custom audio capture process**
Follow these steps to implement custom audio capture in your project:
1. After initializing `RtcEngine`, call `createCustomAudioTrack` to create a custom audio track and obtain the audio track ID.
Java
Kotlin
```java
AudioTrackConfig config = new AudioTrackConfig();
config.enableLocalPlayback = false;
customAudioTrack = engine.createCustomAudioTrack(Constants.AudioTrackType.AUDIO_TRACK_MIXABLE, config);
```
```kotlin
val config = AudioTrackConfig().apply {
enableLocalPlayback = false
}
customAudioTrack = engine.createCustomAudioTrack(Constants.AudioTrackType.AUDIO_TRACK_MIXABLE, config)
```
2. Call `joinChannel` to join the channel. In `ChannelMediaOptions`, set `publishCustomAudioTrackId` to the audio track ID obtained in step 1, and set `publishCustomAudioTrack` to `true` to publish the custom audio track.
To use `enableCustomAudioLocalPlayback` for local playback of an external audio source, or to adjust the volume of a custom audio track with `adjustCustomAudioPlayoutVolume`, set `enableAudioRecordingOrPlayout` to `true` in `ChannelMediaOptions`.
Java
Kotlin
```java
ChannelMediaOptions option = new ChannelMediaOptions();
option.clientRoleType = Constants.CLIENT_ROLE_BROADCASTER;
option.autoSubscribeAudio = true;
option.autoSubscribeVideo = true;
// In the audio self-collection use-case, the audio collected by the microphone is not published
option.publishMicrophoneTrack = false;
// Publish the custom audio track
publishCustomAudioTrack = true
// Set the custom audio track ID
publishCustomAudioTrackId = customAudioTrack
// Join the channel
val res = engine.joinChannel(accessToken, channelId, 0, option)
```
```kotlin
val option = ChannelMediaOptions().apply {
clientRoleType = Constants.CLIENT_ROLE_BROADCASTER
autoSubscribeAudio = true
autoSubscribeVideo = true
// In the audio self-collection use-case, the audio collected by the microphone is not published
publishMicrophoneTrack = false
// Publish the custom audio track
publishCustomAudioTrack = true
// Set the custom audio track ID
publishCustomAudioTrackId = customAudioTrack
}
// Join the channel
val res = engine.joinChannel(accessToken, channelId, 0, option)
```
3. Agora provides the [AudioFileReader.java](https://github.com/AgoraIO/API-Examples/blob/main/Android/APIExample/app/src/main/java/io/agora/api/example/utils/AudioFileReader.java) sample to demonstrate how to read and publish PCM-format audio data from a local file. In a production environment, you create a custom audio acquisition module based on your business needs.
4. Call `pushExternalAudioFrame` to send the captured audio frame to the SDK through the custom audio track. Ensure that the `trackId` matches the audio track ID you obtained by calling `createCustomAudioTrack`. Set `sampleRate`, `channels`, and `bytesPerSample` to define the sampling rate, number of channels, and bytes per sample of the external audio frame.
For audio and video synchronization, Agora recommends calling `getCurrentMonotonicTimeInMs` to get the system’s current monotonic time and setting the `timestamp` accordingly.
Java
Kotlin
```java
audioPushingHelper = new AudioFileReader(requireContext(), (buffer, timestamp) -> {
if (joined && engine != null && customAudioTrack != -1) {
// Push external audio frames to SDK
int ret = engine.pushExternalAudioFrame(buffer, timestamp,
AudioFileReader.SAMPLE_RATE,
AudioFileReader.SAMPLE_NUM_OF_CHANNEL,
Constants.BytesPerSample.TWO_BYTES_PER_SAMPLE,
customAudioTrack);
Log.i(TAG, "pushExternalAudioFrame times:" + (++\pushTimes) + ", ret=" + ret);
}
});
```
```kotlin
audioPushingHelper = AudioFileReader(requireContext()) { buffer, timestamp ->
if (joined && engine != null && customAudioTrack != -1) {
// Push external audio frames to SDK
val ret = engine.pushExternalAudioFrame(
buffer, timestamp,
AudioFileReader.SAMPLE_RATE,
AudioFileReader.SAMPLE_NUM_OF_CHANNEL,
Constants.BytesPerSample.TWO_BYTES_PER_SAMPLE,
customAudioTrack
)
Log.i(TAG, "pushExternalAudioFrame times: \${++pushTimes\}, ret=$ret")
}
}
```
5. To stop publishing custom audio, call `destroyCustomAudioTrack` to destroy the custom audio track.
Java
Kotlin
```java
// Destroy the custom audio track
engine.destroyCustomAudioTrack(customAudioTrack);
```
```kotlin
// Destroy the custom audio track
engine.destroyCustomAudioTrack(customAudioTrack)
```
### Custom audio rendering [#custom-audio-rendering]
This section shows you how to implement custom audio rendering. Refer to the following call sequence diagram to implement custom audio rendering in your app:
**Custom audio rendering workflow**
To implement custom audio rendering, use the following methods:
1. Before calling `joinChannel`, use `setExternalAudioSink` to enable and configure custom audio rendering.
Java
Kotlin
```java
rtcEngine.setExternalAudioSink(
true, // Enable custom audio rendering
44100, // Sampling rate (Hz). Set this value to 16000, 32000, 441000, or 48000
1 // Number of channels for the custom audio source. Set this value to 1 or 2
);
```
```kotlin
rtcEngine.setExternalAudioSink(
true, // Enable custom audio rendering
44100, // Sampling rate (Hz). Set this value to 16000, 32000, 441000, or 48000
1 // Number of channels for the custom audio source. Set this value to 1 or 2
)
```
2. After joining the channel, call `pullPlaybackAudioFrame` to get audio data sent by remote users. Use your own audio renderer to process the audio data and then play the rendered data.
Java
Kotlin
```java
private class FileThread implements Runnable {
@Override
public void run() {
while (mPull) {
int lengthInByte = 48000 / 1000 * 2 * 1 * 10;
ByteBuffer frame = ByteBuffer.allocateDirect(lengthInByte);
int ret = engine.pullPlaybackAudioFrame(frame, lengthInByte);
byte[] data = new byte[frame.remaining()];
frame.get(data, 0, data.length);
// Write to a local file or render using a player
FileIOUtils.writeFileFromBytesByChannel("/sdcard/agora/pull_48k.pcm", data, true, true);
try {
Thread.sleep(10);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
}
}
```
```kotlin
private class FileThread : Runnable {
override fun run() {
while (mPull) {
val lengthInByte = 48000 / 1000 * 2 * 1 * 10
val frame = ByteBuffer.allocateDirect(lengthInByte)
val ret = engine.pullPlaybackAudioFrame(frame, lengthInByte)
val data = ByteArray(frame.remaining())
frame.get(data, 0, data.size)
// Write to a local file or render using a player
FileIOUtils.writeFileFromBytesByChannel("/sdcard/agora/pull_48k.pcm", data, true, true)
try {
Thread.sleep(10)
} catch (e: InterruptedException) {
e.printStackTrace()
}
}
}
}
```
### Using raw audio data callback [#using-raw-audio-data-callback]
This section explains how to implement custom audio rendering.
To retrieve audio data for playback, implement collection and processing of raw audio data. Refer to [Raw audio processing](stream-raw-audio.mdx).
Follow these steps to call the raw audio data API in your project for custom audio rendering:
1. Retrieve audio data for playback using the `onRecordAudioFrame`, `onPlaybackAudioFrame`, `onMixedAudioFrame`, or `onPlaybackAudioFrameBeforeMixing` callback.
2. Independently render and play the audio data.
## Reference [#reference]
This section explains how to implement different sound effects and audio mixing in your app, covering essential steps and code snippets.
### Sample projects [#sample-projects]
Agora provides the following open-source sample projects for audio self-capture and audio self-rendering for your reference:
* [CustomAudioSource](https://github.com/AgoraIO/API-Examples/blob/main/Android/APIExample/app/src/main/java/io/agora/api/example/examples/advanced/customaudio/CustomAudioSource.java)
* [CustomAudioRender](https://github.com/AgoraIO/API-Examples/blob/main/Android/APIExample/app/src/main/java/io/agora/api/example/examples/advanced/customaudio/CustomAudioRender.java)
### API reference [#api-reference]
* [`createCustomAudioTrack`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_imediaengine.html#api_imediaengine_createcustomaudiotrack)
* [`destroyCustomAudioTrack`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_imediaengine.html#api_imediaengine_destroycustomaudiotrack)
* [`pushExternalAudioFrame`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_irtcengine_pushaudioframe2)
* [`setExternalAudioSink`](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_imediaengine_setexternalaudiosink)
* [`pullPlaybackAudioFrame` \[1/2\]](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_imediaengine_pullaudioframe)
* [`pullPlaybackAudioFrame` \[2/2\]](https://api-ref.agora.io/en/video-sdk/android/4.x/API/class_irtcengine.html#api_irtcengine_pullaudioframe2)
<_PlatformProcessedMarker close="true" />
<_PlatformPanel platform="ios">
<_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="ios" />
The default audio module of Voice SDK meets the need of using basic audio functions in your app. For adding advanced audio functions, Voice SDK supports using custom audio sources and custom audio rendering modules.
Video SDK uses the basic audio module on the device your app runs on by default. However, there are certain use-cases where you want to integrate a custom audio source into your app, such as:
* Your app has its own audio module.
* You need to process the captured audio with a pre-processing library for audio enhancement.
* You need flexible device resource allocation to avoid conflicts with other services.
This page shows you how to capture and render audio from custom sources.
## Understand the tech [#understand-the-tech-1]
To set an external audio source, you configure the Agora Engine before joining a channel. To manage the capture and processing of audio frames, you use methods from outside the Video SDK that are specific to your custom source. Video SDK enables you to push processed audio data to the subscribers in a channel.
#### Capture custom audio [#capture-custom-audio-1]
The following figure illustrates the process of custom audio capture.
* You implement the capture module using external methods provided by the SDK.
* You call `pushExternalAudioFrame` to send the captured audio frames to the SDK.
#### Render custom audio [#render-custom-audio-1]
The following figure illustrates the process of custom audio rendering.
* You implement the rendering module using external methods provided by the SDK.
* You call `pullPlaybackAudioFrame` to retrieve the audio data sent by remote users.
## Prerequisites [#prerequisites-1]
Ensure that you have implemented the [SDK quickstart](../index.mdx) in your project.
## Implementation [#implementation-1]
This section shows you how to implement custom audio capture and render audio from a custom source.
### Customize the audio source [#customize-the-audio-source]
Refer to the following call sequence diagram to implement custom audio capture in your app:
**Custom audio capture**
Follow these steps to implement custom audio capture in your project:
1. After initializing `AgoraRtcEngineKit`, call `createCustomAudioTrack` to create a custom audio track and retrieve the audio track ID.
```swift
let audioTrack = AgoraAudioTrackConfig()
audioTrack.enableLocalPlayback = true
trackId = agoraKit.createCustomAudioTrack(.mixable, config: audioTrack)
```
2. Use `joinChannelByToken` to join the channel. In `AgoraRtcChannelMediaOptions`, set the `publishCustomAudioTrackId` parameter to the audio track ID you obtained. Set `publishCustomAudioTrack` to `YES` to publish the custom audio track in the channel.
To play an external audio source locally using `enableCustomAudioLocalPlayback`, or to adjust the local playback volume of a custom audio track using `adjustCustomAudioPlayoutVolume`, set `enableAudioRecordingOrPlayout` to `YES` in `AgoraRtcChannelMediaOptions`.
```swift
let option = AgoraRtcChannelMediaOptions()
// Disable microphone track
option.publishMicrophoneTrack = false
// Enable the custom audio track
option.publishCustomAudioTrack = true
// Set the custom audio track ID
option.publishCustomAudioTrackId = Int(trackId)
option.clientRoleType = GlobalSettings.shared.getUserRole()
NetworkManager.shared.generateToken(channelName: channelName, success: { token in
// Join the channel
let result = self.agoraKit.joinChannel(byToken: token, channelId: channelName, uid: 0, mediaOptions: option)
if result != 0 {
self.showAlert(title: "Error", message: "Failed to join the channel: \(result). Check your parameters.")
}
})
```
3. Implement the custom audio acquisition module
Agora provides a sample project [CustomAudioSource.swift](https://github.com/AgoraIO/API-Examples/tree/main/iOS/APIExample/APIExample/Examples/Advanced/CustomAudioSource) to demonstrate how to read PCM audio data from a local file. In production, create a custom audio acquisition module tailored to your business needs.
4. Call `pushExternalAudioFrameRawData` to send captured audio frames to the SDK via the custom audio track. Ensure `trackId` matches the one used when joining the channel in step 2. Set `sampleRate`, `channels`, and `samples` to configure the external audio frame.
* For audio-video synchronization, Agora recommends using `getCurrentMonotonicTimeInMs` to obtain the system’s monotonic time and set the `timestamp`.
* To push audio frames in `CMSampleBuffer` format, use `pushExternalAudioFrameSampleBuffer`.
```swift
extension CustomAudioSource: AgoraPcmSourcePushDelegate {
func onAudioFrame(data: UnsafeMutablePointer) {
agoraKit.pushExternalAudioFrameRawData(data,
samples: samples,
sampleRate: Int(sampleRate),
channels: Int(audioChannel),
trackId: Int(trackId),
timestamp: 0)
}
}
```
5. To stop publishing custom audio, call `destroyCustomAudioTrack` to remove the custom audio track.
```swift
agoraKit.destroyCustomAudioTrack(Int(trackId))
```
### Custom audio rendering [#custom-audio-rendering-1]
This section shows you how to implement custom audio rendering.
To retrieve audio data for playback, implement collection and processing of raw audio data. Refer to [Raw audio processing](stream-raw-audio.mdx).
Follow these steps to call the raw audio data API in your project for custom audio rendering:
1. Retrieve the audio data to be played from `onRecordAudioFrame`, `onPlaybackAudioFrame`, `onMixedAudioFrame`, or `onPlaybackAudioFrameBeforeMixing`.
2. Independently render and play the audio data.
## Reference [#reference-1]
This section explains how to implement different sound effects and audio mixing in your app, covering essential steps and code snippets.
### Sample projects [#sample-projects-1]
Agora provides the following open-source sample projects for audio self-capture and audio self-rendering for your reference:
* [CustomAudioSource](https://github.com/AgoraIO/API-Examples/tree/main/iOS/APIExample/APIExample/Examples/Advanced/CustomAudioSource)
* [CustomAudioRender](https://github.com/AgoraIO/API-Examples/tree/main/iOS/APIExample/APIExample/Examples/Advanced/CustomAudioRender)
### API reference [#api-reference-1]
* [`createCustomAudioTrack`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginekit/createcustomaudiotrack\(_\:config:\))
* [`destroyCustomAudioTrack`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginekit/destroycustomaudiotrack\(_:\))
* [`pushExternalAudioFrameSampleBuffer`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcmediaplayerprotocol/play\(\))
* [`pushExternalAudioFrameRawData`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginekit/pushexternalaudioframesamplebuffer\(_:\))
* [`enableExternalAudioSink`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginekit/enableexternalaudiosink\(_\:samplerate\:channels:\))
* [`pullPlaybackAudioFrameRawData`](https://api-ref.agora.io/en/video-sdk/ios/4.x/documentation/agorartckit/agorartcenginekit/pullplaybackaudioframerawdata\(_\:lengthinbyte:\))
<_PlatformProcessedMarker close="true" />
<_PlatformPanel platform="macos">
<_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="macos" />
The default audio module of Voice SDK meets the need of using basic audio functions in your app. For adding advanced audio functions, Voice SDK supports using custom audio sources and custom audio rendering modules.
Video SDK uses the basic audio module on the device your app runs on by default. However, there are certain use-cases where you want to integrate a custom audio source into your app, such as:
* Your app has its own audio module.
* You need to process the captured audio with a pre-processing library for audio enhancement.
* You need flexible device resource allocation to avoid conflicts with other services.
This page shows you how to capture and render audio from custom sources.
## Understand the tech [#understand-the-tech-2]
To set an external audio source, you configure the Agora Engine before joining a channel. To manage the capture and processing of audio frames, you use methods from outside the Video SDK that are specific to your custom source. Video SDK enables you to push processed audio data to the subscribers in a channel.
#### Capture custom audio [#capture-custom-audio-2]
The following figure illustrates the process of custom audio capture.
* You implement the capture module using external methods provided by the SDK.
* You call `pushExternalAudioFrame` to send the captured audio frames to the SDK.
#### Render custom audio [#render-custom-audio-2]
The following figure illustrates the process of custom audio rendering.
* You implement the rendering module using external methods provided by the SDK.
* You call `pullPlaybackAudioFrame` to retrieve the audio data sent by remote users.
## Prerequisites [#prerequisites-2]
Ensure that you have implemented the [SDK quickstart](../index.mdx) in your project.
## Implementation [#implementation-2]
This section shows you how to implement custom audio capture and render audio from a custom source.
### Customize the audio source [#customize-the-audio-source-1]
Refer to the following call sequence diagram to implement custom audio capture in your app:
**Custom audio capture**
Follow these steps to implement custom audio capture in your project:
1. After initializing `AgoraRtcEngineKit`, call `createCustomAudioTrack` to create a custom audio track and retrieve the audio track ID.
```swift
let audioTrack = AgoraAudioTrackConfig()
audioTrack.enableLocalPlayback = true
trackId = agoraKit.createCustomAudioTrack(.mixable, config: audioTrack)
```
2. Use `joinChannelByToken` to join the channel. In `AgoraRtcChannelMediaOptions`, set the `publishCustomAudioTrackId` parameter to the audio track ID you obtained. Set `publishCustomAudioTrack` to `YES` to publish the custom audio track in the channel.
To play an external audio source locally using `enableCustomAudioLocalPlayback`, or to adjust the local playback volume of a custom audio track using `adjustCustomAudioPlayoutVolume`, set `enableAudioRecordingOrPlayout` to `YES` in `AgoraRtcChannelMediaOptions`.
```swift
let option = AgoraRtcChannelMediaOptions()
// Disable microphone track
option.publishMicrophoneTrack = false
// Enable the custom audio track
option.publishCustomAudioTrack = true
// Set the custom audio track ID
option.publishCustomAudioTrackId = Int(trackId)
option.clientRoleType = GlobalSettings.shared.getUserRole()
NetworkManager.shared.generateToken(channelName: channelName, success: { token in
// Join the channel
let result = self.agoraKit.joinChannel(byToken: token, channelId: channelName, uid: 0, mediaOptions: option)
if result != 0 {
self.showAlert(title: "Error", message: "Failed to join the channel: \(result). Check your parameters.")
}
})
```
3. Implement the custom audio acquisition module
Agora provides a sample project [CustomAudioSource.swift](https://github.com/AgoraIO/API-Examples/tree/main/macOS/APIExample/Examples/Advanced/CustomAudioSource) to demonstrate how to read PCM audio data from a local file. In production, create a custom audio acquisition module tailored to your business needs.
4. Call `pushExternalAudioFrameRawData` to send captured audio frames to the SDK via the custom audio track. Ensure `trackId` matches the one used when joining the channel in step 2. Set `sampleRate`, `channels`, and `samples` to configure the external audio frame.
* For audio-video synchronization, Agora recommends using `getCurrentMonotonicTimeInMs` to obtain the system’s monotonic time and set the `timestamp`.
* To push audio frames in `CMSampleBuffer` format, use `pushExternalAudioFrameSampleBuffer`.
```swift
extension CustomAudioSource: AgoraPcmSourcePushDelegate {
func onAudioFrame(data: UnsafeMutablePointer) {
agoraKit.pushExternalAudioFrameRawData(data,
samples: samples,
sampleRate: Int(sampleRate),
channels: Int(audioChannel),
trackId: Int(trackId),
timestamp: 0)
}
}
```
5. To stop publishing custom audio, call `destroyCustomAudioTrack` to remove the custom audio track.
```swift
agoraKit.destroyCustomAudioTrack(Int(trackId))
```
### Custom audio rendering [#custom-audio-rendering-2]
This section explains how to implement custom audio rendering.
To retrieve audio data for playback, implement collection and processing of raw audio data. Refer to [Raw audio processing](stream-raw-audio.mdx).
Follow these steps to call the raw audio data API and implement custom audio rendering in your project:
1. Retrieve the audio data to be played from `onRecordAudioFrame`, `onPlaybackAudioFrame`, `onMixedAudioFrame`, or `onPlaybackAudioFrameBeforeMixing`.
2. Independently render and play the audio data.
## Reference [#reference-2]
This section explains how to implement different sound effects and audio mixing in your app, covering essential steps and code snippets.
### Sample projects [#sample-projects-2]
Agora provides the following open-source sample projects for audio self-capture and audio self-rendering for your reference:
* [CustomAudioSource](https://github.com/AgoraIO/API-Examples/tree/main/macOS/APIExample/Examples/Advanced/CustomAudioSource)
* [CustomAudioRender](https://github.com/AgoraIO/API-Examples/tree/main/macOS/APIExample/Examples/Advanced/CustomAudioRender)
### API reference [#api-reference-2]
* [`createCustomAudioTrack`](https://api-ref.agora.io/en/video-sdk/macos/4.x/documentation/agorartckit/agorartcenginekit/createcustomaudiotrack\(_\:config:\))
* [`destroyCustomAudioTrack`](https://api-ref.agora.io/en/video-sdk/macos/4.x/documentation/agorartckit/agorartcenginekit/destroycustomaudiotrack\(_:\))
* [`pushExternalAudioFrameSampleBuffer`](https://api-ref.agora.io/en/video-sdk/macos/4.x/documentation/agorartckit/agorartcmediaplayerprotocol/play\(\))
* [`pushExternalAudioFrameRawData`](https://api-ref.agora.io/en/video-sdk/macos/4.x/documentation/agorartckit/agorartcenginekit/pushexternalaudioframesamplebuffer\(_:\))
* [`enableExternalAudioSink`](https://api-ref.agora.io/en/video-sdk/macos/4.x/documentation/agorartckit/agorartcenginekit/enableexternalaudiosink\(_\:samplerate\:channels:\))
* [`pullPlaybackAudioFrameRawData`](https://api-ref.agora.io/en/video-sdk/macos/4.x/documentation/agorartckit/agorartcenginekit/pullplaybackaudioframerawdata\(_\:lengthinbyte:\))
<_PlatformProcessedMarker close="true" />
<_PlatformPanel platform="web">
<_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="web" />
The default audio module of Voice SDK meets the need of using basic audio functions in your app. For adding advanced audio functions, Voice SDK supports using custom audio sources and custom audio rendering modules.
Voice SDK uses the basic audio module on the device your app runs on by default. However, there are certain use-cases where you want to integrate a custom audio source into your app, such as:
* Your app has its own audio module.
* You need to process the captured audio with a pre-processing library for audio enhancement.
* You need flexible device resource allocation to avoid conflicts with other services.
This page shows you how to capture and render audio from custom sources.
## Understand the tech [#understand-the-tech-3]
To set an external audio source, you configure the Agora Engine before joining a channel. To manage the capture and processing of audio frames, you use methods from outside the Video SDK that are specific to your custom source. Video SDK enables you to push processed audio data to the subscribers in a channel.
## Prerequisites [#prerequisites-3]
Ensure that you have implemented the [SDK quickstart](../index.mdx) in your project.
## Implementation [#implementation-3]
### Custom audio capture [#custom-audio-capture-1]
The SDK provides the `createCustomAudioTrack` method, which enables you to create a local audio track by passing in a browser-native `MediaStreamTrack` object. This method allows you to achieve custom audio capture. To use multiple audio sources, you can call the `createCustomAudioTrack` method multiple times to create separate local audio track objects.
The following example uses `getUserMedia` to obtain a [MediaStreamTrack](https://developer.mozilla.org/en-US/docs/Web/API/MediaStreamTrack) object and passes it to `createCustomAudioTrack` to create a local audio track object for use with the SDK.
```javascript
async function createAndPublishCustomAudioTrack(client) {
try {
// Get the audio media stream.
const mediaStream = await navigator.mediaDevices.getUserMedia({
video: false,
audio: true,
});
// Extract the audio track from the media stream.
const audioMediaStreamTrack = mediaStream.getAudioTracks()[0];
// Create a custom audio track.
const customAudioTrack = await AgoraRTC.createCustomAudioTrack({
mediaStreamTrack: audioMediaStreamTrack,
});
// Store the custom audio track for later use in a shared object.
rtc.localAudioTrack = customAudioTrack;
// Publish the custom audio track to the RTC channel.
await rtc.client.publish([rtc.localAudioTrack]);
console.log('Custom audio track published successfully!');
} catch (error) {
console.error('Failed to create or publish custom audio track:', error);
}
}
```
To implement custom audio processing, use the [Web Audio API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Audio_API) to obtain the `MediaStreamTrack`.
## Reference [#reference-3]
### API reference [#api-reference-3]
* [`createCustomAudioTrack`](https://api-ref.agora.io/en/video-sdk/web/4.x/interfaces/iagorartc.html#createcustomaudiotrack)
* [`ILocalAudioTrack`](https://api-ref.agora.io/en/video-sdk/web/4.x/interfaces/ilocalaudiotrack.html)
<_PlatformProcessedMarker close="true" />
<_PlatformPanel platform="windows">
<_PlatformProcessedMarker groupMode="structured" canonicalPlatform="web" platform="windows" />
The default audio module of Voice SDK meets the need of using basic audio functions in your app. For adding advanced audio functions, Voice SDK supports using custom audio sources and custom audio rendering modules.
Video SDK uses the basic audio module on the device your app runs on by default. However, there are certain use-cases where you want to integrate a custom audio source into your app, such as:
* Your app has its own audio module.
* You need to process the captured audio with a pre-processing library for audio enhancement.
* You need flexible device resource allocation to avoid conflicts with other services.
This page shows you how to capture and render audio from custom sources.
## Understand the tech [#understand-the-tech-4]
To set an external audio source, you configure the Agora Engine before joining a channel. To manage the capture and processing of audio frames, you use methods from outside the Video SDK that are specific to your custom source. Video SDK enables you to push processed audio data to the subscribers in a channel.
#### Capture custom audio [#capture-custom-audio-3]
The following figure illustrates the process of custom audio capture.
* You implement the capture module using external methods provided by the SDK.
* You call `pushExternalAudioFrame` to send the captured audio frames to the SDK.
#### Render custom audio [#render-custom-audio-3]
The following figure illustrates the process of custom audio rendering.
* You implement the rendering module using external methods provided by the SDK.
* You call `pullPlaybackAudioFrame` to retrieve the audio data sent by remote users.
## Prerequisites [#prerequisites-4]
Ensure that you have implemented the [SDK quickstart](../index.mdx) in your project.
## Implementation [#implementation-4]
### Custom audio capture [#custom-audio-capture-2]
Refer to the following call sequence diagram to implement custom audio capture in your app:
**Custom audio capture**
Follow these steps to implement custom audio capture in your project:
1. Enable and configure custom audio source:
Before calling `joinChannel` to join the channel, call `setExternalAudioSource` to enable and configure custom audio capture.
```cpp
// Specify the custom audio source
m_rtcEngine->setExternalAudioSource(true, m_capAudioInfo.sampleRate, m_capAudioInfo.channels);
// Local user joins the channel
ChannelMediaOptions option;
option.autoSubscribeAudio = true;
option.autoSubscribeVideo = true;
m_rtcEngine->joinChannel("Your token", szChannelId.c_str(), 0, option);
```
2. Implement audio capture and processing
Use methods outside the SDK to implement audio capture and processing yourself.
3. Send audio frames to SDK
Call `pushAudioFrame` to send the captured audio frames to the SDK for later use.
```cpp
mediaEngine->pushAudioFrame(AUDIO_RECORDING_SOURCE, &m_audioFrame);
```
### Custom audio rendering [#custom-audio-rendering-3]
This section shows you how to implement custom audio rendering. Refer to the following call sequence diagram to implement custom audio rendering in your app:
**Custom audio rendering workflow**
To implement custom audio rendering, use the following methods:
1. Enable and configure custom audio sink
Before calling `joinChannel` to join the channel, call `setExternalAudioSink` to enable and configure custom audio rendering.
```cpp
// Enable custom audio rendering
// Sample rate (Hz) can be set to 16000, 32000, 44100 or 48000
// Number of channels can be set to 1 or 2
nRet = m_rtcEngine->setExternalAudioSink(m_renderAudioInfo.sampleRate, m_renderAudioInfo.channels);
```
2. Pull and render remote audio data
* After joining the channel, call `pullAudioFrame` to get the audio data sent by the remote user.
* Use your own audio renderer to process the audio data and then play the rendered data.
```cpp
void CAgoraCaptureAduioDlg::PullAudioFrameThread(CAgoraCaptureAduioDlg * self)
{
int nRet = 0;
agora::util::AutoPtr mediaEngine;
mediaEngine.queryInterface(self->m_rtcEngine, AGORA_IID_MEDIA_ENGINE);
IAudioFrameObserver::AudioFrame audioFrame;
audioFrame.avsync_type = 0; // Reserved parameter
audioFrame.bytesPerSample = TWO_BYTES_PER_SAMPLE;
audioFrame.type = agora::media::IAudioFrameObserver::FRAME_TYPE_PCM16;
audioFrame.channels = self->m_renderAudioInfo.channels;
audioFrame.samplesPerChannel = self->m_renderAudioInfo.sampleRate / 100 * self->m_renderAudioInfo.channels;
audioFrame.samplesPerSec = self->m_renderAudioInfo.sampleRate;
audioFrame.buffer = new BYTE[audioFrame.samplesPerChannel * audioFrame.bytesPerSample];
while (self->m_extenalRenderAudio )
{
// Pull remote audio data
nRet = mediaEngine->pullAudioFrame(&audioFrame);
if (nRet != 0)
{
Sleep(10);
continue;
}
SIZE_T nSize = audioFrame.samplesPerChannel * audioFrame.bytesPerSample;
self->m_audioRender.Render((BYTE*)audioFrame.buffer, nSize);
}
delete audioFrame.buffer;
}
```
### Using raw audio data callback [#using-raw-audio-data-callback-1]
This section explains how to implement custom audio rendering.
To retrieve audio data for playback, implement collection and processing of raw audio data. Refer to [Raw audio processing](stream-raw-audio.mdx).
Follow these steps to call the raw audio data API in your project for custom audio rendering:
1. Retrieve audio data for playback using the `onRecordAudioFrame`, `onPlaybackAudioFrame`, `onMixedAudioFrame`, or `onPlaybackAudioFrameBeforeMixing` callback.
2. Independently render and play the audio data.
## Reference [#reference-4]
This section explains how to implement different sound effects and audio mixing in your app, covering essential steps and code snippets.
### Sample projects [#sample-projects-3]
Agora provides an open-source [CustomAudioCapture](https://github.com/AgoraIO/API-Examples/tree/main/windows/APIExample/APIExample/Advanced/CustomAudioCapture) project for your reference. Download the project or inspect the source code for a more detailed example.
### API reference [#api-reference-4]
* [`setExternalAudioSource`](https://api-ref.agora.io/en/video-sdk/cpp/4.x/API/class_irtcengine.html#api_imediaengine_setexternalaudiosource2)
* [`pushAudioFrame`](https://api-ref.agora.io/en/video-sdk/cpp/4.x/API/class_imediaengine.html#api_imediaengine_pushaudioframe0)
* [`setExternalAudioSink`](https://api-ref.agora.io/en/video-sdk/cpp/4.x/API/class_irtcengine.html#api_imediaengine_setexternalaudiosink)
* [`pullAudioFrame`](https://api-ref.agora.io/en/video-sdk/cpp/4.x/API/class_imediaengine.html#api_imediaengine_pullaudioframe)
<_PlatformProcessedMarker close="true" />