Smallest AI
Updated
Use Smallest AI for speech recognition (ASR) in a Conversational AI Platform agent.
Preview documentation
Smallest AI provides real-time streaming transcription through its Pulse model. Use it as the ASR model in a cascading pipeline with any supported LLM and TTS vendor.
Sample configuration
The following example shows how to configure Smallest AI as the agent's ASR when you create or update an agent. For a complete example, see Create an agent and start a session.
Smallest AI supports BYOK only. It isn't available in managed mode. Store your Smallest AI API key in a secret set, and refer to it from the agent. For details, see Use your own provider keys.
The SDK converts the boolean options to the "true" and "false" strings that Smallest AI expects.
from agora_agent import Agent, SmallestAISTT, secret_ref
# client is your configured Agora client
agent = (
Agent(client=client)
.with_stt(
SmallestAISTT(
api_key=secret_ref("support_prod", "smallestai"),
language="zh",
url="wss://api.us.smallest.ai/waves/v1/stt/live",
sample_rate=16000,
encoding="linear16",
word_timestamps=True,
sentence_timestamps=True,
diarize=True,
vad_events=True,
endpointing=True,
eou_timeout_ms=480,
format=True,
finalize_on_words=True,
max_words="100",
punctuate=True,
capitalize=True,
itn_normalize=True,
full_transcript=True,
keywords="Codex:2,Smallest AI:2",
redact_pii=False,
redact_pci=False,
)
)
.with_llm(...) # configure your LLM vendor
.with_tts(...) # configure your TTS vendor
)The SDK converts the boolean options to the "true" and "false" strings that Smallest AI expects.
import { Agent, SmallestAISTT, secretRef } from 'agora-agents';
// client is your configured Agora client
const agent = new Agent({ client })
.withStt(
new SmallestAISTT({
apiKey: secretRef('support_prod', 'smallestai'),
language: 'zh',
url: 'wss://api.us.smallest.ai/waves/v1/stt/live',
sampleRate: 16000,
encoding: 'linear16',
wordTimestamps: true,
sentenceTimestamps: true,
diarize: true,
vadEvents: true,
endpointing: true,
eouTimeoutMs: 480,
format: true,
finalizeOnWords: true,
maxWords: '100',
punctuate: true,
capitalize: true,
itnNormalize: true,
fullTranscript: true,
keywords: 'Codex:2,Smallest AI:2',
redactPii: false,
redactPci: false,
}),
)
.withLlm(/* configure your LLM vendor */)
.withTts(/* configure your TTS vendor */);The SDK converts the boolean options to the "true" and "false" strings that Smallest AI expects.
sampleRate, eouTimeoutMs := 16000, 480
// client is your configured Agora client
agent := agentkit.NewAgent(client).
WithStt(vendors.NewSmallestAISTT(vendors.SmallestAISTTOptions{
APIKey: agentkit.SecretRef("support_prod", "smallestai"),
Language: "zh",
URL: "wss://api.us.smallest.ai/waves/v1/stt/live",
SampleRate: &sampleRate,
Encoding: "linear16",
WordTimestamps: true,
SentenceTimestamps: true,
Diarize: true,
VADEvents: true,
Endpointing: true,
EOUTimeoutMs: &eouTimeoutMs,
Format: true,
FinalizeOnWords: true,
MaxWords: "100",
Punctuate: true,
Capitalize: true,
ITNNormalize: true,
FullTranscript: true,
Keywords: "Codex:2,Smallest AI:2",
})).
WithLlm( /* configure your LLM vendor */ ).
WithTts( /* configure your TTS vendor */ )Set credential.mode to byok and credential.fields.api_key to a reference to your Smallest AI API key in a secret set. In the following example, the key is stored as smallestai in the support_prod set. Only credential and vendor are required:
"asr": {
"credential": {
"mode": "byok",
"fields": {
"api_key": "$secrets.support_prod.smallestai"
}
},
"vendor": "smallestai",
"language": "zh",
"url": "wss://api.us.smallest.ai/waves/v1/stt/live",
"params": {
"sample_rate": 16000,
"encoding": "linear16",
"word_timestamps": "true",
"sentence_timestamps": "true",
"diarize": "true",
"vad_events": "true",
"endpointing": "true",
"eou_timeout_ms": 480,
"format": "true",
"finalize_on_words": "true",
"max_words": "100",
"punctuate": "true",
"capitalize": "true",
"itn_normalize": "true",
"full_transcript": "true",
"keywords": "Codex:2,Smallest AI:2",
"redact_pii": "false",
"redact_pci": "false"
}
}Info
Smallest AI expects its boolean options as the strings "true" and "false", as shown in the example.
Key parameters
The asr block takes the following fields. In the SDKs, SmallestAISTT sets them from its options, and passes any other Smallest AI parameters from its additional parameters option to params.
vendorstringSet to smallestai.
credentialobjectHow the agent authenticates with Smallest AI. Set mode to byok and fields.api_key to a $secrets.<set>.<key> reference to your Smallest AI API key.
languagestringThe language code for speech recognition, for example, zh.
urlstringThe Smallest AI streaming ASR WebSocket endpoint, for example, wss://api.us.smallest.ai/waves/v1/stt/live. Omit it to use the default endpoint.
paramsobjectsample_rateintegerInput audio sample rate in Hz. Defaults to 16000.
encodingstringInput audio encoding. Defaults to linear16.
word_timestampsstringWhether to include word-level timestamps in the transcription results.
sentence_timestampsstringWhether to include sentence-level timestamps in the transcription results.
diarizestringWhether to enable speaker diarization. Speaker labels are emitted on final transcription results.
vad_eventsstringWhether to emit voice activity detection events.
endpointingstringWhether to enable automatic end-of-utterance detection.
eou_timeout_msintegerEnd-of-utterance timeout in milliseconds. For example, 480.
formatstringWhether to enable Smallest AI's transcript formatting. Smallest AI defines the formatting applied.
finalize_on_wordsstringWhether to finalize transcription results based on word count, together with max_words.
max_wordsstringMaximum number of words per transcription result, passed as a string. For example, "100".
punctuatestringWhether to add punctuation to transcription results.
capitalizestringWhether to apply capitalization to transcription results.
itn_normalizestringWhether to apply inverse text normalization, which converts spoken forms such as numbers and dates into written form. Smallest AI applies it only to finalized transcripts, before redaction.
full_transcriptstringWhether to return the full accumulated transcript.
keywordsstringKeyword boosts that improve recognition accuracy for specific terms, as a comma-separated list in keyword:weight format. For example, Codex:2,Smallest AI:2.
redact_piistringWhether to redact personally identifiable information from transcription results.
redact_pcistringWhether to redact payment card information from transcription results.
Caution
The parameters listed on this page are validated for use with the Conversational AI Platform. Required parameters must be provided as documented. Any additional parameters in params are passed through directly to Smallest AI without validation. For a full list of supported options, refer to the Smallest AI documentation.
This integration is fully supported for use with the Conversational AI Platform. While it has completed functional validation, it is newer to the platform, and additional provider-specific edge cases may be identified as usage scales across a broader range of applications and workloads.