Support LocalLLMs (self‑deployed models such as Ollama, vLLM) and relay‑station models (third‑party API proxies)

Feature Requests
,
1

Problem

The system currently only supports public cloud LLM vendor APIs. It cannot connect to self‑hosted local large‑language model deployments (e.g. Ollama, vLLM) or relay‑station / third‑party API proxy endpoints. This prevents on‑premises, air‑gapped, private‑network and proxy‑routed LLM workloads from being integrated into the platform.

Solution

Extend the LLM connector layer to support two new model categories:

  1. Self‑deployed LocalLLMs: Compatible with OpenAI‑compatible API interfaces exposed by Ollama, vLLM and similar local inference services.
  2. Relay‑station / third‑party API proxies: Accept custom proxy gateway endpoints, pass through authentication headers and model routing parameters to upstream LLM backends.

Abstract the LLM client logic so that local self‑hosted services and proxy relay endpoints can be configured in the same UI/config entry as public cloud LLMs.

  1. Add a new model provider type option: Local / Relay Proxy.
  2. User inputs custom base endpoint URL (e.g. http://localhost:11434/v1 for Ollama, or proxy gateway address).
  3. Optional configuration fields: API‑key (for proxies requiring auth), custom model name string, request timeout, custom HTTP headers.
  4. The underlying client sends OpenAI‑format request payload to the configured endpoint. Streaming and non‑streaming completions are both supported.
  5. Health‑check capability: test connection to target local/relay endpoint during configuration save to validate reachability.
  6. Model selection: users manually specify the actual backend model identifier instead of picking from a fixed pre‑defined list.
  7. Observability: trace logs retain the configured endpoint address for troubleshooting local/relay invocation failures.

Use Case

  1. Connect to self‑hosted Ollama running on internal on‑premises servers.
  2. Integrate vLLM high‑performance local inference deployments for private workloads.
  3. Route LLM traffic through internal API relay/proxy gateways for audit, rate‑limiting, token masking and traffic forwarding.
  4. Air‑gapped / offline environments without outbound internet access to public LLM cloud services.
  5. Multi‑tenant scenarios where LLM access is mediated via intermediate proxy service.

Priority

  • :red_circle: High - Blocking issue
  • :yellow_circle: Medium - Important improvement
  • :green_circle: Low - Nice to have

Additional Info

(Optional: screenshots, examples, links)