Agora Native SDK for Android Reference Manual

Introduction

Agora Native SDK is mobile-optimized for smartphones and devices, allowing access to the Agora Global Network along with device-specific mobile optimizations. The Agora Native SDK for Android allows you to perform the following operations:

  • Session setup: Join and leave shared Agora conferencing sessions (identified by unique channel names), where there may be many global users conferenced together or simply one other user for one-to-one communication. Your application code should create and manage unique channel names; these usually originate in user, session, and date/time information that your application is already managing.
  • Media control: Enable and disable voice and video (allowing muting) and set various voice and video parameters that help the Agora Native SDK optimize communications. Many of the SDK operations are automated and do not require developer intervention if these parameter settings are not provided.
  • Device control: Access the microphone or speakerphone, set the volume, select from alternative cameras, and set the video window display.
  • Session control: Receive events when other users join or leave a conferencing session (channel), and understanding who’s speaking, who’s muted, and who’s viewing the session. These events allow the application to decide when to make changes to the delivery of video and audio streams, and other application-specific user and status information.
  • Quality management: Obtain statistics and quality indicators about network conditions, run built-in tests, submit ratings and complaints about sessions and enable different levels of error logging.
  • Recording: Record audio or video and audio in one or multiple channels simultaneously. This function is only applicable to the users who have adopted the Recording Key schema. For details, refer to Agora Recording Server.
  • Data management: Encrypt the audio and video packets, modify the audio or video raw data and receive reliable and ordered packets via data channels.

The Agora Native SDK for Android provides two Java abstract classes to deliver the following features. For details, see API Reference - Agora Native SDK for Android.

  • RtcEngine class provides all the methods that can be invoked by your application.
  • IRtcEngineEventHandler class enables callback event notifications to your application.

The Agora Native SDK for Android provides two C++ classes to deliver the following features. For details, see Agora Native SDK in Raw Data API.

  • IAudioFrameObserver class modifies the audio raw data.
  • IVideoFrameObserver class modifies the video raw data.

The Agora Native SDK may return an error or warning message when calling API or during runtime. For details, see Error and Warning Messages.

Required Development Environments

Use Android Studio, the recommended Google Android integrated development environment, or IntelliJ IDEA. These two environments take a similar approach. If you plan to use other development environments or require additional support, contact support@agora.io. Standard requirements are:

  • Android SDK API Level> = 14
  • Android Studio 1.4 or later
  • Supports voice / video simulator or a real device
  • Apps require Android 4.0 and above devices

Note

If you want to use the startAudioMixing API, ensure that Android device is >=4.2, and API Level>=16

Required Libraries

The Agora Native SDK for Android is an NDK native library (written for efficiency in C++) with binaries for the armeabi-v7a and x86 platforms and a Java wrapper that provides the Java classes for developers documented API Reference - Agora Native SDK for Android.

To use the SDK, copy the following libraries from the libs/ folder of the Agora Native SDK to the libs folder of your project:

  • armeabi-v7a/
  • x86/
  • agora-rtc-sdk.jar

Required Runtime Permissions

For the SDK to run properly it requires the following permissions in the AndroidManifest.xml file for your application when deployed:

  • <uses-permission android:name=”android.permission.INTERNET” />
  • <uses-permission android:name=”android.permission.RECORD_AUDIO” />
  • <uses-permission android:name=”android.permission.CAMERA” />
  • <uses-permission android:name=”android.permission.MODIFY_AUDIO_SETTINGS” />
  • <uses-permission android:name=”android.permission.ACCESS_NETWORK_STATE” />

Getting Started

Obtaining SDK

Download the latest SDK from http://www.agora.io/downloads/ contact sales@agora.io.

About Keys

This section describes the concepts and use of App ID, App Certificate, Dynamic Key, Channel Key and Recording Key. For details, see Agora Keys User Guide.

Creating a New project

  1. Create a new project in Android Studio by selecting Start a new Android Studio project.

    ../_images/android1.png
  2. Configure your new project.

    ../_images/android2.png
  3. Select the form factors for your application devices.

    ../_images/android3.png
  4. Find libs/agora­rtc­sdk.jar and libs/armeabi­-v7a/ (and libs/x86/ if appropriate) in the Agora Native SDK for Android package and add them to your project.

    Note

    The Agora Native SDK for Android is based on C++ native libraries. Set up the jniLibs path as shown, or directly create a jniLibs folder and copy libs/armeabi­-v7a and libs/x86 to the created folder.

../_images/android4.png
  1. Click Sync Project With Gradle Files after the sync is complete. You can then use the Agora Native SDK for Android. The main API is io.agora.rtc.RtcEngine.

Note

Gradle is the official build system for Android Studio. The Agora demo application also uses Gradle. Because the Agora Native SDK includes an NDK, you may want to review the Gradle and NDK integration information at https://gradle.org/getting­-started­-android/.

Using the Demo Application

Agora provides the following demo application in the Agora Native SDK for Android zip file (describing here the full package with both video and voice):

AgoraDemo includes basic methods of entering and leaving a call, demonstrates the core operations of Agora Native SDK for Android and enables audio and video calls with simple method calls.

Directory structure:

../_images/android_directory_structure.png

Note

Visit https://github.com/AgoraLab/ for more open source demo applications.

Compling the Demo Application

  1. Run Android Studio and import the Gradle configuration file for the demo application.
../_images/android_demo_1.png
  1. Select your Eclipse project folder and click OK.
../_images/android_demo_2.png
  1. Open the project files and compile them.
../_images/android_demo_3.png

Executing the Demo Application

Run the demo program, AgoraDemo, after the compilation.

Below is the first page of the demo application. You need two phone devices to run the demo for voice and video calls:

../_images/android_demo_4.png
  1. Make a voice call:
  1. On two mobile phones, enter your same App ID and use the same meeting room name (channel name), for example, 2804, as you see below:

    ../_images/android_demo_5.png
  2. Click or tap the Join Voice Call button to enter the audio call page.

  3. The call page features the following functions: mute/unmute, enable/disable speaker, leave the room.

    ../_images/android_demo_6.png
  1. Make a video call:
  1. Click or tap on the Video Call button to enter the main call page.

  2. On two mobile phones, enter the same App ID and use the same meeting room name (channel name).

    ../_images/android_demo_7.png
  3. Click or tap Join Video Call to enter the video call page. The call page features the following functions: mute/unmute, enable/disable speaker, open/close camera, switch cameras, and leave the room.

    ../_images/android_demo_8.png

Switch between Video and Voice calls if necessary. Use the Video Call and Voice Call buttons at the bottom of the screen as shown in the above figure.

Note

The SDK and the AgoraDemo application support up to 5-way group video sessions. Multiple users join using the same App ID and room number (channel name in API).

Encrypting Data

The Agora Native SDK allows your application to encrypt audio and video packets in one of the following ways:

Note

If your application has integrated the built-in encryption of Agora SDK, then users are able to record an encrypted channel.

Using Agora Native SDK Built-in Encryption

Agora Native SDKs support built-in encryption using the AES-128 or AES-256 encryption algorithm. You can call setEncryptionSecret to enable the encryption function and then call setEncryptionMode to set the mode to be used. For details, see API Reference - Agora Native SDK for Android.

The following diagram depicts the built-in encryption/decryption process:

../_images/agora-encryption.png

Implementing Your Customized Data Encryption Algorithm

The Agora Native SDK allows your application to encrypt audio and video packets by implementing the customized data encryption algorithm of your application.

The following diagram below depicts the data encryption/decryption process:

../_images/developer-encryption.png

These are the steps:

Registering a Packet Observer

Agora Native SDK allows your application to register a packet observer to receive events whenever an audio or video packet is transmitting.

Register a packet observer on your application using the following API:

virtual int registerPacketObserver(IPacketObserver* observer);

The observer must inherit from agora::rtc::IPacketObserver and be implemented in C++. The following is the definition of the IPacketObserver class:

class IPacketObserver
{
public:

        struct Packet
        {
                const unsigned char* buffer;
                unsigned int size;
        };
        /**
        * called by sdk before the audio packet is sent to other participants
        * @param [in,out] packet:
        *      buffer *buffer points the data to be sent
        *      size of buffer data to be sent
        * @return returns true to send out the packet, returns false to discard the packet
        */
        virtual bool onSendAudioPacket(Packet& packet) = 0;
        /**
        * called by sdk before the video packet is sent to other participants
        * @param [in,out] packet:
        *      buffer *buffer points the data to be sent
        *      size of buffer data to be sent
        * @return returns true to send out the packet, returns false to discard the packet
        */


virtual bool onSendVideoPacket(Packet& packet) = 0;
        /**
        * called by sdk when the audio packet is received from other participants
        * @param [in,out] packet
        *      buffer *buffer points the data to be sent
        *      size of buffer data to be sent
        * @return returns true to process the packet, returns false to discard the packet
        */
        virtual bool onReceiveAudioPacket(Packet& packet) = 0;
        /**
        * called by sdk when the video packet is received from other participants
        * @param [in,out] packet
        *      buffer *buffer points the data to be sent
        *      size of buffer data to be sent
        * @return returns true to process the packet, returns false to discard the packet
        */
        virtual bool onReceiveVideoPacket(Packet& packet) = 0;
Implementing a Customized Data Encryption Algorithm

Inherit from agora::rtc::IPacketObserver to implement the customized data encryption algorithm on your application. The following example uses XOR for data processing. For Agora Native SDK, sending and receiving packets are handled by the different threads, which is why encryption and decryption can use different buffers:

class AgoraPacketObserver : public agora::rtc::IPacketObserver
 {
 public:
     AgoraPacketObserver()
     {
         m_txAudioBuffer.resize(2048);
         m_rxAudioBuffer.resize(2048);
         m_txVideoBuffer.resize(2048);
         m_rxVideoBuffer.resize(2048);
     }
     virtual bool onSendAudioPacket(Packet& packet)
     {
         int i;
         //encrypt the packet
         const unsigned char* p = packet.buffer;
         const unsigned char* pe = packet.buffer+packet.size;


                  for (i = 0; p < pe && i < m_txAudioBuffer.size(); ++p, ++i)
         {
             m_txAudioBuffer[i] = *p ^ 0x55;
         }
         //assign new buffer and the length back to SDK
         packet.buffer = &m_txAudioBuffer[0];
         packet.size = i;
         return true;
     }

     virtual bool onSendVideoPacket(Packet& packet)
     {
         int i;
         //encrypt the packet
         const unsigned char* p = packet.buffer;
         const unsigned char* pe = packet.buffer+packet.size;
         for (i = 0; p < pe && i < m_txVideoBuffer.size(); ++p, ++i)
         {
             m_txVideoBuffer[i] = *p ^ 0x55;
         }
         //assign new buffer and the length back to SDK
         packet.buffer = &m_txVideoBuffer[0];
         packet.size = i;
         return true;
     }

     virtual bool onReceiveAudioPacket(Packet& packet)
     {
         int i = 0;
         //decrypt the packet
         const unsigned char* p = packet.buffer;
         const unsigned char* pe = packet.buffer+packet.size;
         for (i = 0; p < pe && i < m_rxAudioBuffer.size(); ++p, ++i)
         {
             m_rxAudioBuffer[i] = *p ^ 0x55;
         }
         //assign new buffer and the length back to SDK
         packet.buffer = &m_rxAudioBuffer[0];
         packet.size = i;
         return true;
     }

     virtual bool onReceiveVideoPacket(Packet& packet)
     {
         int i = 0;
         //decrypt the packet
         const unsigned char* p = packet.buffer;
         const unsigned char* pe = packet.buffer+packet.size;


                 for (i = 0; p < pe && i < m_rxVideoBuffer.size(); ++p, ++i)
         {
             m_rxVideoBuffer[i] = *p ^ 0x55;
         }
         //assign new buffer and the length back to SDK
         packet.buffer = &m_rxVideoBuffer[0];
         packet.size = i;
         return true;
     }

 private:
     std::vector<unsigned char> m_txAudioBuffer; //buffer for sending audio data
     std::vector<unsigned char> m_txVideoBuffer; //buffer for sending video data

     std::vector<unsigned char> m_rxAudioBuffer; //buffer for receiving audio data
     std::vector<unsigned char> m_rxVideoBuffer; //buffer for receiving video data
 };
Registering Instance
  1. Implement a Java wrapper.

For example,

  JNIEXPORT jint JNICALL Java_io_agora_video_demo_RtcEngineEncryption_enableEncryption(JNIEnv *env, jclass clazz, jlong engineHandle)
{
   typedef jint (*PFN_registerAgoraPacketObserver)(void* engine, agora::rtc::IPacketObserver* observer);

   void* handle = dlopen("libagora-rtc-sdk-jni.so", RTLD_LAZY);
   if (!handle)
   {
      __android_log_print(ANDROID_LOG_ERROR, "agora encrypt demo",

"cannot find libagora-rtc-sdk-jni.so");
      return -1;
   }
   PFN_registerAgoraPacketObserver pfn = (PFN_registerAgoraPacketObserver)dlsym(handle, "registerAgoraPacketObserver");
   if (!pfn)
   {
      __android_log_print(ANDROID_LOG_ERROR, "aogra encrypt demo", "cannot find registerAgoraPacketObserver");
      return -2;
   }
   return pfn((void*)engineHandle, &s_packetObserver);
}

Java wrapper:
public class RtcEngineEncryption {
    static {
        System.loadLibrary("agora-encrypt-demo-jni");
    }
    public static native int enableEncryption(long rtcEngineHandle);
}
  1. Call registerPacketObserver to register the instance of agora::rtc::IPacketObserver class implemented by your application:

Modifying Agora Raw Data

Agora raw data interface is an advanced feature provided in the SDK library for users to obtain the audio/video raw data of the SDK engine. You, as a developer, can modify the audio or video data and create special effects to meet special needs of their applications.

You can insert a pre-processing stage before sending the data to the encoder, modifying the captured of video frames or audio signals. You can also insert a post-processing stage before sending the data to the decoder, modifying the received video frames or audio signals.

Agora raw data interface is a C++ interface, and you need to use the JNI and plug-in manager included in the SDK library.

Modifying Audio Raw Data

See IAudioFrameObserver Interface Class for detailed API description. For details, see Raw Data API.

  1. Define AgoraAudioFrameObserver by inheriting IAudioFrameObserver(The IAudioFrameObserver Class is defined in IAgoraMediaEngine.h). You need to realize the following virtual interfaces:

For example,

class AgoraAudioFrameObserver : public agora::media::IAudioFrameObserver
{
  public:
    virtual bool onRecordAudioFrame(AudioFrame& audioFrame) override
    {
      return true;
    }
    virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) override
    {
      return true;
     }
    virtual bool onPlaybackAudioFrameBeforeMixing(unsigned int uid, AudioFrame& audioFrame) override
     {
      return true;
     }
};

The above example returns true for audio pre-processing or post-processing interfaces. Users can modify the data if necessary:

class IAudioFrameObserver
{
  public:
      enum AUDIO_FRAME_TYPE {
      FRAME_TYPE_PCM16 = 0,  //PCM 16bit little endian
      };
  struct AudioFrame {
      AUDIO_FRAME_TYPE type;
      int samples;  //number of samples in this frame
      int bytesPerSample;  //number of bytes per sample: 2 for PCM 16
      int channels;  // number of channels (data are interleaved if stereo)
      int samplesPerSec;  //sampling rate
      void* buffer;  //data buffer
      };
  public:
      virtual bool onRecordAudioFrame(AudioFrame& audioFrame) = 0;
      virtual bool onPlaybackAudioFrame(AudioFrame& audioFrame) = 0;
      virtual bool onPlaybackAudioFrameBeforeMixing(unsigned int uid, AudioFrame& audioFrame) = 0;
};
  1. Register the audio frame observer to the SDK engine. After creating the IRtcEngine object, and before joining a channel, you can register the object of audio observer.
AgoraAudioFrameObserver s_audioFrameObserver;

agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
mediaEngine.queryInterface(*engine, agora::rtc::AGORA_IID_MEDIA_ENGINE);
if (mediaEngine)
{
  mediaEngine->registerAudioFrameObserver(&s_audioFrameObserver);
}

Note

Here *engine can be passed in by loadAgoraRtcEnginePlugin in Creating Agora Native SDK Plugin.

Modifying Video Raw Data

See IVideoFrameObserver Interface Class for detailed API explanation. For details, see Raw Data API.

  1. Define AgoraVideoFrameObserver by inheriting IVideoFrameObserver(The IVideoFrameObserver Class is defined in IAgoraMediaEngine.h). You need to realize the following virtual interfaces:

    For example,

    class AgoraVideoFrameObserver : public agora::media::IVideoFrameObserver
    {
      public:
        virtual bool onCaptureVideoFrame(VideoFrame& videoFrame) override
        {
          return true;
        }
        virtual bool onRenderVideoFrame(unsigned int uid, VideoFrame& videoFrame) override
        {
          return true;
        }
    };
    

The above example returns true for audio pre-processing or post-processing interfaces. Users can modify the data if necessary:

class IVideoFrameObserver
{
  public:
    enum VIDEO_FRAME_TYPE {
    FRAME_TYPE_YUV420 = 0,  //YUV 420 format
    };
  struct VideoFrame {
    VIDEO_FRAME_TYPE type;
    int width;  //width of video frame
    int height;  //height of video frame
    int yStride;  //stride of Y data buffer
    int uStride;  //stride of U data buffer
    int vStride;  // stride of V data buffer
    void* yBuffer;  //Y data buffer
    void* uBuffer;  //U data buffer
    void* vBuffer;  //V data buffer
    };
  public:
    virtual bool onCaptureVideoFrame(VideoFrame& videoFrame) = 0;
    virtual bool onRenderVideoFrame(unsigned int uid, VideoFrame& videoFrame) = 0;
};
  1. Register the video frame observer to the SDK engine. After creating the IRtcEngine object and enabling the video mode, and before joining a channel, you can register the object of video observer.
AgoraVideoFrameObserver s_videoFrameObserver;

agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
  mediaEngine.queryInterface(*engine,agora::rtc::AGORA_IID_MEDIA_ENGINE);
  if (mediaEngine)
  {
     mediaEngine->registerVideoFrameObserver(&s_videoFrameObserver);
  }

Note

Here *engine can be passed in by loadAgoraRtcEnginePlugin in Creating Agora Native SDK Plugin.

Creating Agora Native SDK Plugin

Agora Native SDK supports loading the third-party plugin by using dynamic link libraries. The usage is as follows:

  1. Create a shared library project, and the SO file must be created with libapm- as prefix and so as suffix. For example, libapm-encryption.so.
  2. Implement and export the related interface functions.
  • The following plugin interface function is mandatory which is to be called when the Agora SDK loads the plugin. It returns 0 upon a successful registration, and the registration fails when it does not return 0.

    extern "C" __attribute__((visibility("default"))) int loadAgoraRtcEnginePlugin(agora::rtc::IRtcEngine* engine);
    

    Sample Code:

    extern "C" __attribute__((visibility("default"))) int loadAgoraRtcEnginePlugin(agora::rtc::IRtcEngine* engine)
    {
        agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
        mediaEngine.queryInterface(*engine,agora::rtc::AGORA_IID_MEDIA_ENGINE);
        if (mediaEngine)
        {
            mediaEngine->registerVideoFrameObserver(&s_videoFrameObserver);
        }
      return 0;
    }
    
  • The following plugin interface function is optional which is to be called when the Agora SDK removes the plugin.

    extern "C" __attribute__((visibility("default"))) void unloadAgoraRtcEnginePlugin(agora::rtc::IRtcEngine* engine);
    

    Sample Code:

    extern "C" __attribute__((visibility("default"))) void unloadAgoraRtcEnginePlugin(agora::rtc::IRtcEngine* engine)
    {
        agora::util::AutoPtr<agora::media::IMediaEngine> mediaEngine;
        mediaEngine.queryInterface(*engine,agora::rtc::AGORA_IID_MEDIA_ENGINE);
        if (mediaEngine)
        {
            mediaEngine->registerVideoFrameObserver(NULL);
        }
    }
    

The process of Agora Native SDK loading the plugin is listed as follows:

  1. The Agora SDK engine searches all matched dynamic link libraries in the format of libapm-xxx.so in the directory that contains the native libraries during the initialization. For example, libapm-encryption.so.
  2. The Agora SDK loads the plugin using dlopen.
  3. The user must implement the plugin and export the loadAgoraRtcEnginePlugin interface according to the above instructions. The SDK obtains and calls loadAgoraRtcEnginePlugin for each plugin found. If it returns 0 after calling the loadAgoraRtcEnginePlugin function, then the execution succeeds. Otherwise the SDK will remove the plugin (FreeLibrary). The input parameter of the interface is agora:rtc:: IRtcEngine interface object, which can be used by the plugin in the loadAgoraRtcEnginePlug function to register IPacketObserver, IAudioFrameObserver, IVideoFrameObserver and etc.
  4. When the SDK engine is destroyed, call the optional entry function unloadAgoraRtcEnginePlugin of the plugin.