Ollama

Installation

brew install ollama

curl -fsSL https://ollama.com/install.sh | sh

# Start Ollama service (runs in background)
ollama serve

# Run a model
ollama run llama3.1

Popular Models

Python Library

pip install ollama  # v0.6.1+

import ollama

response = ollama.chat(
    model='llama3.1',
    messages=[
        {'role': 'user', 'content': 'Explain quantum computing in simple terms'}
    ]
)
print(response['message']['content'])

import ollama

stream = ollama.chat(
    model='llama3.1',
    messages=[{'role': 'user', 'content': 'Write a haiku about coding'}],
    stream=True
)

for chunk in stream:
    print(chunk['message']['content'], end='', flush=True)

import ollama

response = ollama.embed(
    model='nomic-embed-text',
    input='The sky is blue because of Rayleigh scattering'
)
embedding = response['embeddings'][0]  # 768-dim vector

OpenAI API Compatibility

Ollama exposes an OpenAI-compatible API at localhost:11434. Use existing OpenAI code with local models!

from openai import OpenAI

# Point to local Ollama
client = OpenAI(
    base_url="http://localhost:11434/v1",
    api_key="ollama"  # Any string works
)

response = client.chat.completions.create(
    model="llama3.1",
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "What is the capital of France?"}
    ],
    temperature=0.7
)
print(response.choices[0].message.content)

Tool Calling (Function Calling)

Since July 2024, Ollama supports tool calling with models like Llama 3.1+. Let models call functions and use external APIs.

import ollama

# Define tools
tools = [{
    "type": "function",
    "function": {
        "name": "get_weather",
        "description": "Get the current weather for a location",
        "parameters": {
            "type": "object",
            "properties": {
                "location": {"type": "string", "description": "City name"},
                "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
            },
            "required": ["location"]
        }
    }
}]

response = ollama.chat(
    model='llama3.1',
    messages=[{'role': 'user', 'content': 'What is the weather in Paris?'}],
    tools=tools
)

# Model returns tool_calls if it wants to use a tool
if response['message'].get('tool_calls'):
    tool_call = response['message']['tool_calls'][0]
    print(f"Function: {tool_call['function']['name']}")
    print(f"Arguments: {tool_call['function']['arguments']}")

Structured Outputs (JSON Schema)

Since December 2024, Ollama can constrain outputs to a specific JSON schema.

import ollama

schema = {
    "type": "object",
    "properties": {
        "name": {"type": "string"},
        "age": {"type": "integer"},
        "skills": {"type": "array", "items": {"type": "string"}}
    },
    "required": ["name", "age", "skills"]
}

response = ollama.chat(
    model='llama3.1',
    messages=[{'role': 'user', 'content': 'Generate a profile for a software developer'}],
    format=schema
)

# Output is guaranteed to match the schema
import json
profile = json.loads(response['message']['content'])
print(profile)
# {"name": "Alex Chen", "age": 28, "skills": ["Python", "TypeScript", "Docker"]}

Vision / Multimodal Models

Ollama supports vision models like Llama 3.2 Vision for image understanding.

import ollama
import base64

# Load image
with open('image.jpg', 'rb') as f:
    image_data = base64.b64encode(f.read()).decode()

response = ollama.chat(
    model='llama3.2-vision',
    messages=[{
        'role': 'user',
        'content': 'What do you see in this image?',
        'images': [image_data]
    }]
)
print(response['message']['content'])

Essential CLI Commands

Creating Custom Models (Modelfile)

# Modelfile
FROM llama3.1

# Set system prompt
SYSTEM """You are a Python expert. You always write clean, 
well-documented code with type hints and docstrings."""

# Set default parameters
PARAMETER temperature 0.2
PARAMETER top_p 0.9
PARAMETER stop "```"

# Create and run custom model
ollama create python-expert -f Modelfile
ollama run python-expert

LangChain Integration

from langchain_ollama import ChatOllama
from langchain_core.messages import HumanMessage

# Initialize
llm = ChatOllama(model="llama3.1", temperature=0.7)

# Simple chat
response = llm.invoke([HumanMessage(content="Explain RAG in one paragraph")])
print(response.content)

# With streaming
for chunk in llm.stream([HumanMessage(content="Write a poem")]):
    print(chunk.content, end="", flush=True)

Related Topics