Smallest AI

Updated

Use Smallest AI for speech recognition (ASR) in a Conversational AI Platform agent.

Preview documentation

You're reading the Conversational AI Platform (Preview) docs. Switch to Main.

Smallest AI provides real-time streaming transcription through its Pulse model. Use it as the ASR model in a cascading pipeline with any supported LLM and TTS vendor.

Sample configuration

The following example shows how to configure Smallest AI as the agent's ASR when you create or update an agent. For a complete example, see Create an agent and start a session.

Smallest AI supports BYOK only. It isn't available in managed mode. Store your Smallest AI API key in a secret set, and refer to it from the agent. For details, see Use your own provider keys.

The SDK converts the boolean options to the "true" and "false" strings that Smallest AI expects.

from agora_agent import Agent, SmallestAISTT, secret_ref

# client is your configured Agora client
agent = (
    Agent(client=client)
    .with_stt(
        SmallestAISTT(
            api_key=secret_ref("support_prod", "smallestai"),
            language="zh",
            url="wss://api.us.smallest.ai/waves/v1/stt/live",
            sample_rate=16000,
            encoding="linear16",
            word_timestamps=True,
            sentence_timestamps=True,
            diarize=True,
            vad_events=True,
            endpointing=True,
            eou_timeout_ms=480,
            format=True,
            finalize_on_words=True,
            max_words="100",
            punctuate=True,
            capitalize=True,
            itn_normalize=True,
            full_transcript=True,
            keywords="Codex:2,Smallest AI:2",
            redact_pii=False,
            redact_pci=False,
        )
    )
    .with_llm(...)  # configure your LLM vendor
    .with_tts(...)  # configure your TTS vendor
)

The SDK converts the boolean options to the "true" and "false" strings that Smallest AI expects.

import { Agent, SmallestAISTT, secretRef } from 'agora-agents';

// client is your configured Agora client
const agent = new Agent({ client })
  .withStt(
    new SmallestAISTT({
      apiKey: secretRef('support_prod', 'smallestai'),
      language: 'zh',
      url: 'wss://api.us.smallest.ai/waves/v1/stt/live',
      sampleRate: 16000,
      encoding: 'linear16',
      wordTimestamps: true,
      sentenceTimestamps: true,
      diarize: true,
      vadEvents: true,
      endpointing: true,
      eouTimeoutMs: 480,
      format: true,
      finalizeOnWords: true,
      maxWords: '100',
      punctuate: true,
      capitalize: true,
      itnNormalize: true,
      fullTranscript: true,
      keywords: 'Codex:2,Smallest AI:2',
      redactPii: false,
      redactPci: false,
    }),
  )
  .withLlm(/* configure your LLM vendor */)
  .withTts(/* configure your TTS vendor */);

The SDK converts the boolean options to the "true" and "false" strings that Smallest AI expects.

sampleRate, eouTimeoutMs := 16000, 480

// client is your configured Agora client
agent := agentkit.NewAgent(client).
	WithStt(vendors.NewSmallestAISTT(vendors.SmallestAISTTOptions{
		APIKey:             agentkit.SecretRef("support_prod", "smallestai"),
		Language:           "zh",
		URL:                "wss://api.us.smallest.ai/waves/v1/stt/live",
		SampleRate:         &sampleRate,
		Encoding:           "linear16",
		WordTimestamps:     true,
		SentenceTimestamps: true,
		Diarize:            true,
		VADEvents:          true,
		Endpointing:        true,
		EOUTimeoutMs:       &eouTimeoutMs,
		Format:             true,
		FinalizeOnWords:    true,
		MaxWords:           "100",
		Punctuate:          true,
		Capitalize:         true,
		ITNNormalize:       true,
		FullTranscript:     true,
		Keywords:           "Codex:2,Smallest AI:2",
	})).
	WithLlm( /* configure your LLM vendor */ ).
	WithTts( /* configure your TTS vendor */ )

Set credential.mode to byok and credential.fields.api_key to a reference to your Smallest AI API key in a secret set. In the following example, the key is stored as smallestai in the support_prod set. Only credential and vendor are required:

"asr": {
  "credential": {
    "mode": "byok",
    "fields": {
      "api_key": "$secrets.support_prod.smallestai"
    }
  },
  "vendor": "smallestai",
  "language": "zh",
  "url": "wss://api.us.smallest.ai/waves/v1/stt/live",
  "params": {
    "sample_rate": 16000,
    "encoding": "linear16",
    "word_timestamps": "true",
    "sentence_timestamps": "true",
    "diarize": "true",
    "vad_events": "true",
    "endpointing": "true",
    "eou_timeout_ms": 480,
    "format": "true",
    "finalize_on_words": "true",
    "max_words": "100",
    "punctuate": "true",
    "capitalize": "true",
    "itn_normalize": "true",
    "full_transcript": "true",
    "keywords": "Codex:2,Smallest AI:2",
    "redact_pii": "false",
    "redact_pci": "false"
  }
}

Info

Smallest AI expects its boolean options as the strings "true" and "false", as shown in the example.

Key parameters

The asr block takes the following fields. In the SDKs, SmallestAISTT sets them from its options, and passes any other Smallest AI parameters from its additional parameters option to params.

asr required
ParameterDescription
vendorstring
required

Set to smallestai.

credentialobject
required

How the agent authenticates with Smallest AI. Set mode to byok and fields.api_key to a $secrets.<set>.<key> reference to your Smallest AI API key.

languagestring
optional

The language code for speech recognition, for example, zh.

urlstring
optional

The Smallest AI streaming ASR WebSocket endpoint, for example, wss://api.us.smallest.ai/waves/v1/stt/live. Omit it to use the default endpoint.

paramsobject
optional
  • Optional sample_rate?: integer

    Input audio sample rate in Hz. Defaults to 16000.

  • Optional encoding?: string

    Input audio encoding. Defaults to linear16.

  • Optional word_timestamps?: string

    Whether to include word-level timestamps in the transcription results.

  • Optional sentence_timestamps?: string

    Whether to include sentence-level timestamps in the transcription results.

  • Optional diarize?: string

    Whether to enable speaker diarization. Speaker labels are emitted on final transcription results.

  • Optional vad_events?: string

    Whether to emit voice activity detection events.

  • Optional endpointing?: string

    Whether to enable automatic end-of-utterance detection.

  • Optional eou_timeout_ms?: integer

    End-of-utterance timeout in milliseconds. For example, 480.

  • Optional format?: string

    Whether to enable Smallest AI's transcript formatting. Smallest AI defines the formatting applied.

  • Optional finalize_on_words?: string

    Whether to finalize transcription results based on word count, together with max_words.

  • Optional max_words?: string

    Maximum number of words per transcription result, passed as a string. For example, "100".

  • Optional punctuate?: string

    Whether to add punctuation to transcription results.

  • Optional capitalize?: string

    Whether to apply capitalization to transcription results.

  • Optional itn_normalize?: string

    Whether to apply inverse text normalization, which converts spoken forms such as numbers and dates into written form. Smallest AI applies it only to finalized transcripts, before redaction.

  • Optional full_transcript?: string

    Whether to return the full accumulated transcript.

  • Optional keywords?: string

    Keyword boosts that improve recognition accuracy for specific terms, as a comma-separated list in keyword:weight format. For example, Codex:2,Smallest AI:2.

  • Optional redact_pii?: string

    Whether to redact personally identifiable information from transcription results.

  • Optional redact_pci?: string

    Whether to redact payment card information from transcription results.

The table extends beyond the current width. Scroll horizontally to view more columns.

Caution

The parameters listed on this page are validated for use with the Conversational AI Platform. Required parameters must be provided as documented. Any additional parameters in params are passed through directly to Smallest AI without validation. For a full list of supported options, refer to the Smallest AI documentation.

This integration is fully supported for use with the Conversational AI Platform. While it has completed functional validation, it is newer to the platform, and additional provider-specific edge cases may be identified as usage scales across a broader range of applications and workloads.