AI models and LLM providers

Gluesync does not host a model for you. AI features call your LLM providers through an encrypted vault under Settings → LLM providers. This page is the setup guide for those providers, including a local Ollama server.

The same vault is reused by:

Add at least one provider as a Super admin before those surfaces will run. An empty vault returns HTTP 400 / SQLSTATE 55000 with a message that a Super admin must configure a provider.

Choose cloud or local

Path When to use What Gluesync needs from you

Cloud API

OpenAI, an OpenAI-compatible endpoint, Anthropic, or Azure OpenAI

Outbound HTTPS from Core Hub to the vendor endpoint, an API key (or Azure key), and a model that supports tool calling

Local (Ollama)

Models stay on your hardware; no vendor key

An Ollama process whose HTTP API Core Hub can reach (default port 11434), a pulled model that supports tool calling, and extra RAM/GPU on the Ollama host — not on Gluesync itself

Gluesync never proxies tokens through MOLO17. Keys are stored encrypted in Core Hub. Ollama is treated as zero-priced in estimated spend.

Hardware and network (add-on)

Replication hardware on System requirements does not include model inference.

  • Cloud providers. No extra CPU/RAM on the Gluesync host. Core Hub must be allowed to call the vendor HTTPS API. See Egress allowlist for cloud APIs.

  • Ollama on another machine. Size that machine for the model (VRAM or plenty of system RAM). Gluesync only needs HTTP access to the Ollama API.

  • Ollama on the same host as Gluesync. Add the model’s memory on top of the Gluesync baseline (16 GB RAM for a typical compose install). A 7B–8B instruct model is a common starting point; 70B-class models need a dedicated GPU box.

Add a provider in the Control Plane

  1. Sign in as a Super admin.

  2. Open Settings → LLM providers.

  3. Select Add provider.

  4. Fill the fields below, then Save.

  5. Run Test connection.

Managers can open the tab read-only (providers and estimated spend). Viewers and Monitors never see it.

Fields

Field Notes

Name

Display name in the vault and in AI Studio’s model picker (model · name).

Provider

OPENAI, OPENAI_COMPATIBLE, ANTHROPIC, AZURE_OPENAI, or OLLAMA.

Default model

Required. Used as the default for new AI Studio conversations. Type a name or pick from suggestions. After save, Refresh from provider loads the live list (GET /ai-studio/providers/{id}/models). For Azure OpenAI this is often the deployment name.

Base URL

Optional override. Leave empty for the vendor default. For Azure OpenAI, Core Hub requires the resource endpoint (for example https://your-resource.openai.azure.com). For Ollama, set this to a URL Core Hub can actually reach — see Local models with Ollama.

Azure deployment (azureDeployment)

Required for Azure OpenAI.

API version (apiVersion)

Required for Azure OpenAI.

Input / output price (USD / 1M tokens)

Optional overrides. Empty save clears a previous override and falls back to the bundled catalogue. Ollama stays zero-priced unless you set a price yourself.

API key

Required for OpenAI, Anthropic, and Azure OpenAI. For OPENAI_COMPATIBLE, supply a bearer key when the endpoint requires one. Not shown for Ollama (Not required). On edit, leave blank to keep the saved key. Core Hub never returns the key (hasApiKey only).

Connection test

Test connection is not a ping. Core Hub requires the provider to support tool calling. A reachable endpoint with a chat-only model still fails.

Connection tests are not counted in monthly spend.

Enable or disable a provider

A Super admin can disable a vaulted provider without deleting its credentials. Disabled providers are rejected by chat and AI helpers and are removed from the provider-synced canonical model catalog, so new platform routes cannot select them. Re-enable the provider when it is ready for traffic again.

Disabling is useful for a temporary outage, spend freeze, or model migration. It does not erase historical usage or conversations.

Cloud providers

OpenAI

  • Default endpoint: https://api.openai.com/v1

  • Create an API key at platform.openai.com.

  • Pick a model that supports tool calling (for example the current gpt-4o / gpt-4.1 class — use Refresh from provider for the live list).

  • Allow Core Hub egress to api.openai.com (HTTPS 443).

OpenAI-compatible

Use OPENAI_COMPATIBLE for a self-hosted or third-party endpoint that implements the OpenAI /models and /chat/completions contracts.

  • Default endpoint: http://localhost:8080/v1; normally replace it with a URL reachable from the Core Hub process or container.

  • The API key is optional at the Gluesync boundary. Add it when the endpoint expects a bearer token.

  • The configured model must support the operation used by the Gluesync surface. In-product Spark agents and provider connection tests require tool calling; published 2.3 platform runs use text completion only.

  • Compatibility varies by server. Run Test connection before publishing an agent that depends on it.

Anthropic

Azure OpenAI

  • Provider type is AZURE_OPENAI (not a generic "Azure" label).

  • Base URL — Azure resource endpoint.

  • Azure deployment — the deployment name in Azure AI Foundry / Azure OpenAI.

  • API version — required (apiVersion).

  • API key from the Azure resource.

  • Allow Core Hub egress to that resource hostname (HTTPS 443). See Azure OpenAI documentation for creating the resource and deployment.

Egress allowlist for cloud APIs

Add these from the Core Hub host/container, not from the operator’s laptop:

Provider Typical destination

OpenAI

https://api.openai.com

OpenAI-compatible

Your configured endpoint

Anthropic

https://api.anthropic.com

Azure OpenAI

https://<your-resource>.openai.azure.com (your resource)

Model-list / chat traffic

Same origins; no extra MOLO17 hop

Air-gapped sites that cannot allow those destinations should use Local models with Ollama instead.

Local models with Ollama

Ollama exposes an OpenAI-compatible HTTP API (Ollama API) on port 11434. Gluesync talks to that API. There is no API key.

1. Install Ollama and pull a model

On the machine that will run the model (often the Gluesync host, or a GPU box on the same LAN):

  1. Install Ollama from ollama.com.

  2. Pull a model that can call tools (instruct / chat models). A typical first pull:

    ollama pull llama3.1

    The exact tag is yours to choose. If Test connection fails with a tool-calling error, switch to a model whose card lists tools / function calling.

  3. Confirm the API locally:

    curl http://127.0.0.1:11434/api/tags

2. Bind the API so Core Hub can reach it

Ollama defaults to localhost. Core Hub almost always runs in Docker (or Kubernetes). From inside that container, http://localhost:11434 is the container itself — not the host, and not another VM.

Do both of the following:

  • Listen on all interfaces on the Ollama host:

    # Linux systemd user/environment example
export OLLAMA_HOST=0.0.0.0:11434

    Then restart Ollama. Confirm ss -lntp | grep 11434 (or equivalent) shows 0.0.0.0:11434 or [::]:11434.

  • In Settings → LLM providers, set Base URL to an address Core Hub can route to:

Where Ollama runs Base URL to save in Gluesync

Docker Desktop (Windows / macOS), Ollama on the same machine

http://host.docker.internal:11434

Linux Docker Engine, Ollama on the same host

http://host.docker.internal:11434 if you added host.docker.internal:host-gateway to the Core Hub compose service extra_hosts; otherwise the host’s LAN IP, for example http://192.168.1.10:11434

Ollama on another server

http://<ollama-host>:11434 (open 11434 between Core Hub and that host; TLS is optional and up to your reverse proxy)

Kubernetes

A ClusterIP/Service to the Ollama pod, or an egress NetworkPolicy to the Ollama host. Do not use localhost.

Leave Base URL empty only if Core Hub and Ollama truly share the same network namespace (rare). The Control Plane default http://localhost:11434 is the right Ollama default, not the right container default.

Restrict who can hit port 11434. Binding 0.0.0.0 without a firewall exposes the model API on the network.

3. Register it in Gluesync

  1. Provider: Ollama.

  2. Default model: the pulled name (for example llama3.1). Use Refresh from provider after the first save if the list is empty.

  3. API key: leave unset.

  4. Test connection.

GPU, CPU, and size

Inference cost sits on the Ollama host:

  • GPU (NVIDIA, or Apple Silicon via Ollama’s Metal path) is strongly preferred.

  • CPU-only works for small models and is slow for agent/tool loops.

  • Keep enough free RAM/VRAM for the model plus Gluesync if they share a box. If the node starts swapping, move Ollama off the Gluesync host.

Gluesync does not manage ollama pull, GPU drivers, or model files.

After the vault has a provider

  • AI Studio — Super admins create agents that pick a provider and an MCP tool allow-list. Anyone with Query Studio read can chat with those agents. See AI Studio.

  • Query Studio AI helper — same vault; no AI Studio agent required. See AI helper.

  • gluesync.ai — SQL surface using the same vault. See AI as SQL.

You cannot delete a provider that is still linked to an AI Studio agent. Reassign or delete the agent first.

Troubleshooting

Symptom What to check

Empty-vault / Super admin must configure a provider

No rows in Settings → LLM providers, or the caller is not using a configured vault. Add a provider as Super admin.

Test connection fails, endpoint looks up

Model lacks tool calling. Try another model; for Ollama pull an instruct model that documents tools.

Ollama: connection refused / timeout from Test connection

Core Hub cannot route to Base URL. Fix bind (OLLAMA_HOST) and use host.docker.internal or a LAN IP — not localhost from a container.

Ollama: empty model list

Nothing pulled, or Base URL points at the wrong process. Run ollama list on the Ollama host and Refresh from provider.

Cloud: timeout or TLS errors

Proxy/firewall. Allow HTTPS from Core Hub to the vendor host. Corporate MITM proxies need the Core Hub trust store to include the proxy CA.

Azure: 4xx on chat

Missing or wrong azureDeployment / apiVersion / Base URL. The default model is often the deployment name, not the base model name.

Managers cannot add a provider

By design. Vault writes are Super admin only.

Related