Local API

A local endpoint for existing clients.

When the server is running, ToshLLM exposes an OpenAI-compatible API at http://127.0.0.1:8080 by default. The host remains limited to the Mac until local-network discovery is enabled.

Send a chat completion.

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "your-model",
    "messages": [
      {"role": "user", "content": "Explain Metal in one paragraph."}
    ]
  }'

Use the model identifier returned by GET /v1/models. In single-model mode, many compatible clients can use their default model field; router mode uses stable aliases derived from downloaded filenames.

Protect clients beyond one process.

Enable Protect the API with a key in Settings to generate a key stored in the macOS Keychain. ToshLLM then requires it on every request.

curl http://127.0.0.1:8080/v1/models \
  -H "Authorization: Bearer YOUR_TOSHLLM_KEY"

The in-app chat supplies the key automatically. Copy it only into clients you trust.

Expose only what you intend to expose.

Enabling Discoverable on local network changes the listener from 127.0.0.1 to 0.0.0.0 and advertises a ToshLLM API service through Bonjour. Use it only on a trusted network and enable API-key protection first.

Router mode publishes multiple downloaded models behind the same endpoint and automatically loads them as requests arrive. The configured model limit controls how many remain loaded at once.