Show HN: Self-hosted AI gateway – MCP, budget, PII, smart router, fallback

Deploying AI models to production brings critical operational challenges such as unpredictable costs, data privacy risks, and API outages. ILTER is an independent gateway that sits between your application and AI providers; transforming your entire AI traffic into a secure, optimized, and fully controlled infrastructure without requiring any architectural changes in your code.

All this infrastructure requires no external server, Node.js/Python environment, or complex dependencies. It comes as a zero-setup, single static executable (binary) ready to run in seconds.

./ilter serve # Proxy: http://localhost:8181/v1/chat/completions # Dashboard: http://localhost:9191 # Metrics: http://localhost:9192/metrics Dashboard Overview Live KPIs, daily cost trend, provider/model breakdown, and a one-click toggle hub for every feature. Chat Built-in playground to test prompts against any registered model without leaving the dashboard. MCP Marketplace Browse and one-click install community MCP servers — filesystem, GitHub, Postgres, and more. Smart Router Manage routing strategies — cost-first, quality-first, or custom rules — and switch the active one live. Logs Full request history with cost, latency, and status, filterable by time range and provider. Jobs Cron-scheduled AI workflows with next/last run status, right from the embedded scheduler. Features Every business integrating AI faces uncontrolled costs, privacy concerns, and unpredictable agent behaviors. ILTER solves these directly at the gateway level.

Connecting AI to your internal APIs or CRM usually requires heavy client-side modifications.

Using expensive models for every simple question wastes engineering time and money. ILTER routes requests dynamically based on context.

Don't get locked into a single vendor or disrupted by provider outages.

Vector Search: Redis Stack-backed similarity search to catch and serve repetitive queries. Exact-Match Fallback: SHA-256 caching works even without an embedding model. Async Operations: Cache writing does not block the primary request path. 🛡️ Prompt Guardrails Security: Prompt injection detection and toxicity filtering. Topic Control: Keyword-based and custom regex rules to block specific topics. Actions: Set per-rule severity to block , warn , or mask . 💰 Budget Control Per-token pricing creates end-of-month surprises. ILTER tracks costs in real-time for each API key and enforces hard limits.

Sending customer data to cloud providers violates GDPR and HIPAA. ILTER masks this data before it ever leaves your network.

Agentic workflows can enter infinite loops, generating massive bills overnight. ILTER catches them while you sleep.

Running periodic AI tasks (summaries, reports) usually requires external queues and workers. ILTER embeds this directly in the binary.

Metrics: Native Prometheus metrics on /metrics (Port 9192 ). Tracing: OpenTelemetry (OTLP push) ready for Grafana Cloud, Datadog, or Honeycomb. Audit Logs: Full request logs, cost trends, and PII event indicators. 📊 Others No npm, no build step, no separate server. A modern web UI embedded directly into the Go binary via embed .

# Requires nothing — it just works ./ilter serve For the interactive setup wizard:

./ilter init To see the dashboard filled with mock costs, requests, and PII events:

./ilter init --demo && ./ilter serve Your first real request (Point your existing OpenAI SDK to ILTER):

curl -X POST http://localhost:8181/v1/chat/completions \ -H " Authorization: Bearer <ilter-api-key> " \ -H " Content-Type: application/json " \ -d ' {"model": "gpt-4o-mini", "messages": [{"role": "user", "content": "Hello!"}]} ' Supported Providers Provider Notes OpenAI Native format Anthropic System message extraction, content block conversion Google Gemini OpenAI-compatible mode DeepSeek OpenAI-compatible OpenRouter HTTP-Referer + X-Title headers automatically injected Ollama Local inference, OpenAI-compatible mode Qwen (Alibaba) OpenAI-compatible, model name mapping OpenCode opencode_go and opencode_zen SDK endpoint mapping Mock Built-in mock provider for local testing Architecture Your Application │ OpenAI API format ▼ ILTER :8181 │ ├─ Auth — ilter-xxxx key (Argon2id/SHA-256 + LRU cache) ├─ Rate Limiter — Redis / in-memory token bucket, RPM/TPM per key ├─ Budget Enforcer — Monthly spend limit, hard reject ├─ Prompt Injection — Inject system prompts from DB ├─ PII Masker — Bloom + Aho-Corasick + Regex, <0.04ms ├─ Guardrails — Injection detection, topic blocking ├─ MCP Inject — Tool injection + tool_call interception + OAuth PKCE ├─ Smart Router — Real-time complexity scoring & tier selection ├─ Loop Detector — Rate / fingerprint / cost / session ├─ Semantic Cache — Redis Stack vector search + SHA256 fallback └─ Provider Router — Weighted round-robin, circuit breaker, fallback │ ├─ OpenAI / Anthropic / Gemini / DeepSeek ├─ OpenRouter / Ollama / Qwen / OpenCode └─ ... ILTER :9191 → Dashboard (Astro + React, Go embed) ILTER :9192 → /metrics (Prometheus, OpenTelemetry bridge) Language: Go 1.26.3, single binary, CGo-free ( CGO_ENABLED=0 ), goroutine concurrency. Router: chi v5. Database: SQLite — modernc.org/sqlite , pure Go, WAL mode. Cache: Redis Stack 7+ — optional, graceful degradation. Config: Compiled defaults + ILTER_* env vars — no configuration file required. → Architecture details: docs/architecture.md → Design decisions & FAQ: docs/faq.md → Gateway comparison: docs/comparison.md

# Single container (scratch-based image, <20MB) # Admin key + a provider key are enough to boot straight into `serve` — no # `ilter init` step needed. The SQLite DB is created on first start, and the # provider whose ILTER_PROVIDER_<NAME>_API_KEY is set is enabled automatically. docker run -d \ -p 8181:8181 -p 9191:9191 -p 9192:9192 \ -v $( pwd ) /data:/app/data \ -e ILTER_ADMIN_API_KEY= < your-own-random-secret > \ -e ILTER_PROVIDER_OPENAI_API_KEY=sk-... \ ghcr.io/ilter-ai/ilter:latest # Full local stack: ILTER + Redis Stack + Ollama docker compose up -d Without ILTER_ADMIN_API_KEY and a provider key both set, serve refuses to start — there'd be no way to authenticate or route requests — and exits with a message telling you to run ilter init or set both.

Image size: <20MB (3-stage build: Bun web → Go UPX → empty scratch base image). This applies to the Docker image specifically — the plain binary from Releases (no UPX) is ~35-40MB.

Zero configuration by default. Three levels of overrides:

Command Description ilter serve Start proxy + dashboard ilter init Interactive setup wizard ilter init --demo Dashboard demo with mock data ilter models Supported models — provider, tier, cost ilter models --update Model discovery from all providers Development make build # Go binary + web assets make check # Build + lint (Go + web) make test # go test -race -count=1 ./... make fix # gofumpt + biome format Contribution guidelines: CONTRIBUTING.md

Apache 2.0 with Commons Clause — see LICENSE .