Docs / Core Concepts / Local Models
⭐ Star on GitHub

🦙 Local Model Support

OrcBot v2.1 introduces first-class support for local LLMs, allowing you to run your agent with total privacy and zero per-token costs.

TUI Integration: You can manage your entire Ollama stack directly from the OrcBot TUI under Manage AI Models -> Ollama / Local Models.

Supported Providers

OrcBot's local model engine is designed around the OpenAI-compatible API standard. While Ollama is the primary supported platform, any server that exposes a /v1/chat/completions endpoint will work.

  • Ollama (Native integration with auto-start and model pull)
  • LM Studio
  • vLLM
  • LocalAI
  • Text Generation WebUI

Configuration

To use a local model, update your configuration using one of the methods below.

Using the TUI (Recommended)

  1. Run orcbot ui
  2. Navigate to Manage AI Models
  3. Select Ollama / Local Models
  4. Follow the prompts to start the server, pull a model, and set it as primary.

Using orcbot.config.yaml

llmProvider: ollama
ollamaApiUrl: http://localhost:11434
modelName: llama3

Using Environment Variables

ORCBOT_LLM_PROVIDER=ollama
ORCBOT_OLLAMA_API_URL=http://localhost:11434
ORCBOT_MODEL_NAME=llama3

Native Tool Calling

One of OrcBot's core strengths is its ability to use tools (shell, browser, etc.). Unlike many local agent implementations that rely on brittle text parsing, OrcBot uses Native Tool Calling with local models that support it.

When a tool-capable local model (like llama3 or mistral) is used, OrcBot sends the tool definitions as part of the structured API request, ensuring high reliability and complex task execution.

Performance Tuning

Running models locally requires significant hardware resources. Here are some tips for optimizing performance:

GPU Acceleration

Always ensure your GPU drivers are up to date. Ollama will automatically use NVIDIA (CUDA) or Apple (Metal) acceleration if available.

Model Sizing

For complex strategic planning, use at least a 7B parameter model. For better tool usage and reasoning, 13B or 70B models are recommended if your VRAM allows.

Security & Air-Gapping

By combining Local Model support with OrcBot's User-Space Restricted Shell, you can create a highly secure, air-gapped autonomous system. Since the model runs on your machine, sensitive data like logs, file paths, and internal reasoning never leave your local network.