Use a Local LLM with Ollama

TEN Agent supports any LLM that provides OpenAI-compatible APIs. This guide shows you how to run models locally using Ollama, eliminating the need for external API calls while maintaining the same interface.

Before starting, ensure you have:

• Completed the TEN Agent quickstart
• Sufficient hardware for your chosen model:
  • CPU: Modern multi-core processor
  • RAM: 8GB minimum (16GB+ recommended for larger models)
  • GPU: Optional but recommended for better performance

Begin by installing Ollama and downloading your preferred language model:

1. Download and install Ollama from ollama.com

2. Download your desired model using ollama pull {model name}. For example:

   
   

   _2

   # Run on your local machine

   _2

   ollama pull llama3.2

   
   

Launch the Ollama server with network access enabled to allow Docker containers to connect:

_1

OLLAMA_HOST=0.0.0.0 ollama serve

Since TEN Agent runs in Docker, you'll need your machine's private IP address:

_5

# Get your private IP on macOS/Linux

_5

ifconfig | grep "inet " | grep -v 127.0.0.1

_5

_5

# Get your private IP on Windows

_5

ipconfig | findstr "IPv4"

Connect TEN Agent to your local Ollama instance through the playground interface:

1. Open the playground at http://localhost:3000

2. Select the voice_assistant graph

3. In Module Picker, Choose OpenAI as the LLM provider

4. In Settings, configure the llm extension as follows:

   • Base URL: http://<your-private-ip>:11434/v1/
   • Model: Your downloaded model name. For example, llama3.2
   • API Key: Any non-empty string. Ollama doesn't require authentication.
   • Prompt: Your system prompt. For example, "You're a helpful assistant."

5. Save your configuration in the playground

Verify your setup by running a test conversation:

1. Start a conversation to test the connection
2. Monitor Ollama logs for requests

For more information on Ollama’s OpenAI compatibility, refer to the official blog post.

If you encounter issues while setting up Ollama with TEN Agent, refer to these common problems and their solutions:

Issue	Possible solutions
Connection refused	Verify Ollama is running with OLLAMA_HOST=0.0.0.0 Check your firewall settings Ensure you're using the correct private IP
Model not found	Confirm the model is downloaded: ollama list Use the exact model name in your configuration
Performance issues	Try a smaller model like mistral Ensure sufficient system resources Consider GPU acceleration if available