Integrations

One local server. Your preferred tools.

Start a model in ToshLLM before opening the client. Most integrations need the same three values:

Use the exact id returned by GET /v1/models. Router mode exposes a stable alias for each downloaded model. If API protection is disabled but a client requires a value, enter toshllm-local; the local server will ignore it.

curl http://127.0.0.1:8080/v1/models \
  -H "Authorization: Bearer YOUR_TOSHLLM_KEY"

Choose a client that stays direct.

Route the agent directly to ToshLLM.

Claude Code officially supports Anthropic-compatible gateways through environment variables. ToshLLM exposes the required Messages API, authentication headers and model discovery endpoint locally.

export ANTHROPIC_BASE_URL=http://127.0.0.1:8080
export ANTHROPIC_AUTH_TOKEN=YOUR_TOSHLLM_KEY
export CLAUDE_CODE_ENABLE_GATEWAY_MODEL_DISCOVERY=1

claude

1. Start ToshLLM in Router mode or run the model you want to use.
2. Copy the API key from ToshLLM Settings. Use toshllm-local if protection is disabled.
3. Export the variables in the same terminal where you will launch Claude Code.
4. Open /model and choose the local model reported by ToshLLM.

Gateway model discovery requires Claude Code 2.1.129 or newer. Without it, select a model explicitly through Claude Code's model configuration.

Build against the Messages API you already know.

Set the Anthropic client's base URL to the ToshLLM server root and use any model ID returned by GET /v1/models. The API supports regular messages, streaming, token counting and client-defined tools.

from anthropic import Anthropic

client = Anthropic(
    base_url="http://127.0.0.1:8080",
    api_key="YOUR_TOSHLLM_KEY",
)

response = client.messages.create(
    model="MODEL_ID",
    max_tokens=512,
    messages=[{"role": "user", "content": "Review this function."}],
)

print(response.content[0].text)

The model name does not need to be a Claude model. It identifies the local GGUF model that ToshLLM should run. See the Local API guide for raw requests, streaming and token counting.

Continue, Cline or Roo Code.

All three extensions support OpenAI-compatible providers. Continue is a clean choice for chat and editing. Cline and Roo Code are better suited to agent workflows that call tools and modify a workspace.

name: ToshLLM Local
version: 1.0.0
schema: v1
models:
  - name: ToshLLM
    provider: openai
    model: MODEL_ID
    apiBase: http://127.0.0.1:8080/v1
    apiKey: YOUR_TOSHLLM_KEY
    roles:
      - chat
      - edit
      - apply

Use the native llama.cpp provider.

Zed includes a local llama.cpp provider that points at the server root rather than the /v1 base URL. Add the model in Zed's settings JSON.

{
  "language_models": {
    "llama.cpp": {
      "api_url": "http://127.0.0.1:8080",
      "auto_discover": false,
      "available_models": [
        {
          "name": "MODEL_ID",
          "display_name": "ToshLLM Local",
          "max_tokens": 8192,
          "supports_tools": false,
          "supports_images": false
        }
      ]
    }
  }
}

Match max_tokens to the context configured in ToshLLM. Set tool and image support to the actual capabilities of the running model.

Create a local custom provider.

Run /connect, choose Other, use toshllm as the provider ID and store the ToshLLM key. Then add this provider to opencode.json.

{
  "$schema": "https://opencode.ai/config.json",
  "provider": {
    "toshllm": {
      "npm": "@ai-sdk/openai-compatible",
      "name": "ToshLLM Local",
      "options": {
        "baseURL": "http://127.0.0.1:8080/v1"
      },
      "models": {
        "MODEL_ID": {
          "name": "ToshLLM Local Model"
        }
      }
    }
  }
}

OpenCode's OpenAI-compatible package uses /v1/chat/completions, which matches ToshLLM's local API.

Point the CLI at ToshLLM.

Set the OpenAI-compatible endpoint and launch Aider with the openai/ model prefix.

export OPENAI_API_BASE=http://127.0.0.1:8080/v1
export OPENAI_API_KEY=YOUR_TOSHLLM_KEY
aider --model openai/MODEL_ID

For a persistent setup, place the base URL, key and model in ~/.aider.conf.yml instead of exporting them for every terminal session.

Do not assume localhost remains private.

Cursor's current official bring-your-own-key flow supports listed cloud providers and states that requests are routed through Cursor's servers for final prompt construction. That path cannot reliably reach a ToshLLM server bound to 127.0.0.1, and it does not provide the same direct local connection as the integrations above.

Use the same connection contract.

A client is a candidate when it accepts an OpenAI-compatible base URL for /v1/chat/completions or /v1/responses, or an Anthropic-compatible base URL for /v1/messages. Use the exact model ID returned by ToshLLM and confirm which protocol the client expects.

Review the complete local API guide for network access, authentication and router mode.