Who spent what
Per-request cost attribution down to a single virtual key, in the currency each vendor actually invoices you in.
Safe LLM API Gateway
One gate for every prompt your company sends.
TORHAUS.AI sits between your applications and the model vendors you buy from, and answers the four questions that turn urgent the moment more than one team shares an API key: who spent what, what was sent, who may call which model, and where it went.
app.py
openai >= 1.0
# The only change your developers make.
from openai import OpenAI
client = OpenAI(
base_url="https://gateway.example.com/v1",
api_key="tor_...", # a virtual key, not a vendor key
)
client.chat.completions.create(
model="gateway-balanced", # a routing alias, not a model name
messages=[{"role": "user", "content": "Hello"}],
)
# Charged to a team. Checked against a budget.
# Screened by your guardrails. Written to the audit log.
→ 200 OK x-request-id: 01JD8P4W2Q…
1 binary
plus Postgres and Valkey. Templates, translations and every asset are embedded, and dependencies are vendored and committed.
0 calls home
No account, no licence key, no activation. No telemetry, analytics, crash reporting or update check of any kind.
9 timings
on every audit row, so "the gateway is slow" resolves to a stage rather than to a shrug.
402
not 429, when a budget runs out, so an SDK stops retrying instead of hammering a wall that will never clear.
What it is
A shared API key is not a plan.
The first team gets access. Then the second. Six months later one key is in eleven repositories, the invoice is a single line item, nobody can say which prompt cost what, and legal wants to know what has been sent to whom. A torii is the gate you pass through deliberately. That is the whole idea.
What was sent
An audit row for every proxied request, with an opt-in copy of the prompt and the response for the teams that need it.
Who may call what
Keys scoped to models, operations, rate limits, budgets and an expiry date. Issued in seconds, revoked in seconds.
Where it went
Explicit routing with failover, and a record of every target that was tried before one answered.
Sovereignty
Sovereignty you deploy, not sovereignty we promise.
TORHAUS.AI runs as one binary on infrastructure you control, in a region you chose, talking only to the model endpoints you configured. It has no home to phone. Where your data lives follows from where you install it and which providers you point it at, which is the only kind of sovereignty a piece of software can honestly offer you.
One instance belongs to one company and is administered through a single bootstrap account whose password the console forces you to rotate before anything else opens.
Hosted in the EU, or nowhere but yours
Outbound connections
- The provider endpoints you configure. Nothing is called that you did not enter yourself, and every call honours
HTTPS_PROXY. - One daily exchange-rate fetch from the ECB, so multi-currency spend rolls up at a published reference rate.
GATEWAY_FX_SOURCE=noneswitches it off and a static rate takes over, which is what an air-gapped install runs. - That is the list. No telemetry, no analytics, no crash reporting, no update check, no licence server.
- The console loads nothing from a CDN. Every asset is embedded in the binary and served same-origin under a
'self'Content-Security-Policy that never permits an inline script. - Dependencies are vendored and committed, and both provider dialects are hand-rolled, so the wire format lives in the repository rather than behind someone else's release.
Latency breakdown · one request
Authenticationlatency_auth_us
Rate limit and budgetlatency_ratelimit_us
Guardrails, requestlatency_guardrail_pre_us
Routinglatency_routing_us
Providerlatency_provider_us
Time to first tokenlatency_ttft_us
Guardrails, responselatency_guardrail_post_us
Gateway overheadlatency_overhead_us
Proportions illustrative. The figures are whatever your own traffic measures, per request.
Speed
The gateway's own cost is a number you can see.
Every stage of a request is timed separately and written to the audit row in microseconds, including the one that matters: total minus the upstream call, which is everything TORHAUS.AI itself spent. The request detail page draws it as a stacked bar, so an outlier stage is obvious rather than argued about.
- Written in Go. The routing table, the price list and the guardrail set are immutable snapshots behind an atomic pointer, so the request path takes no lock.
- Virtual keys resolve from a hot-path cache; costs are computed in integer arithmetic, not floating point.
- Audit writes are asynchronous, so recording history never sits on the request path of a latency-sensitive proxy. Back-pressure drops are counted and alertable, and the writer can be told to block instead if you would rather trade latency for completeness.
- Streaming is first class, and time to first token is one of the nine timings.
Audit trail
One row per request. Including the boring ones.
Every proxied request writes an audit row: who called, which key and team it belonged to, which alias resolved to which target, tokens, exact cost, every guardrail verdict, and where the microseconds went. Prompt and response bodies are captured only where you switch that on, per team or per virtual key.
- Payload capture is off by default. Turning it on for a team is a data protection decision, and the console says so in those words.
- Redactions run before storage, and a stored payload that was rewritten is flagged as non-verbatim, so a reader is never misled about what they are looking at.
- Two retention clocks: months for request metadata, a shorter window in days for captured payloads, enforced by dropping whole time partitions rather than by row-by-row deletes.
- The usage report exports as CSV and your entire guardrail policy set exports as YAML or JSON, so neither is trapped in the interface.
- Trusted proxy depth defaults to zero, so
X-Forwarded-Foris ignored entirely until you say how many reverse proxies really sit in front. Believing that header by default would let any caller write whatever address it liked into your audit log.
REQUEST LOG
01JD8P4W2Q7RX3K…
- team
- engineering · org acme-gmbh
- key
- tor_live_…f30c active
- alias → target
- gateway-balanced → eu-vllm-01
- attempts
- eu-vllm-00 breaker open, eu-vllm-01 ok
- tokens
- 1 284 in · 611 out
- cost
- 0.004137 EUR priced
- guardrails
- pii redact (2) · token-ceiling pass
- latency
- total 1 942 ms · overhead 3.1 ms
- payload
- stored, non-verbatim
Illustrative row. Field names and units are the ones the gateway actually writes.
Guardrails
Policy you write, enforced the same way for every team.
A guardrail is a type plus a JSON configuration, bound additively to global, organization, team or key scope. Rules import and export as YAML or JSON, so a policy set moves between a staging gateway and a production one as a file, not as an afternoon of clicking.
Evaluators that ship with it
regex
keywords
pii
token ceiling
cost ceiling
model allow-list
model deny-list
content length
tool-argument schema
webhook
- The PII evaluator detects email addresses, phone numbers, credit card numbers, IBANs, IP addresses, US social security numbers and API keys, checksum-verified with Luhn and mod‑97 so a sixteen-digit order number is not mistaken for a card.
- Rules are opt-in: you choose the detectors, the stage and the scope. A fresh gateway enforces nothing until you bind your first rule.
- Redaction rewrites the prompt text on its way to the provider, and again in whatever you store.
- The webhook evaluator hands the decision to your own HTTPS endpoint, so a classifier you already run stays the one that judges content.
- Set
fail_closedand a check that errors, an unreachable webhook say, blocks the request instead of letting it through.
Three things a rule can do
Block
The request never reaches the provider. The client gets an OpenAI-shaped error it already knows how to handle.
Redact
The match is rewritten before the request leaves, and before anything is written down.
Warn
The request passes and the verdict lands on the audit row. This is how you measure a rule before you enforce it.
Guardrails read the prompt text your users write. Whatever your policy needs beyond that, a house pattern or an internal project code, is a regex rule you add yourself.
Chargeback and budgets
An invoice your finance team can take apart.
Priced per vendor, in the vendor's currency
Prices and billing currency are set per provider endpoint, so a vendor invoicing in EUR and one invoicing in USD are each accounted natively, with nothing converted at write time.
- Costs are computed in integer nanos, never floats, so millions of fractional-cent requests still add up to the right number.
- A display currency rolls multi-currency spend up at the reference rate for the day the spend happened, so last quarter's total reads the same next quarter.
- Traffic with no price list is flagged unpriced rather than quietly counted as free.
Budgets at three levels
Spend caps on an organization, a team or a single virtual key, per day, week, month or lifetime.
- An exhausted budget returns
402, not429. A 429 makes SDK retry logic back off and try again against a condition that will never clear on its own. - Enforcement is continuous rather than end-of-month, so you find out at the cap instead of on the invoice.
- Rate limits sit beside them: requests per minute and tokens per minute, per key.
Tenancy that maps to your org chart
Organizations, teams under them, and any number of virtual keys per team, each with a cost centre you can hand to accounting.
- Per-key allow-lists for models and for operations, so a key issued for embeddings cannot quietly start generating images.
- A key carries 178 bits of entropy, is stored only as a digest, and is shown exactly once. The gateway cannot recover it.
- Revocation is immediate and permanent.
Alias · gateway-balanced
PRIORITY 1
eu-vllm-01 w 70
eu-vllm-00 w 30
PRIORITY 2
azure-westeu w 100
PRIORITY 3
anthropic w 100
Lower numbers are tried first. Targets sharing a number split traffic by weight. A struck-through target is one whose circuit breaker is open.
Routing
Your clients ask for an alias. You decide what that means.
A model alias maps to targets grouped into priority tiers and weighted within a tier. When a provider degrades, its circuit breaker opens, the balancer skips it, and traffic moves to the next tier, and the audit row shows exactly which targets were tried on the way. Recovery is a single probe, not a stampede.
- Clients speak the OpenAI API. Upstream, TORHAUS.AI talks to OpenAI and anything OpenAI-compatible (Azure OpenAI, vLLM, OpenRouter, Ollama) and to Anthropic, translating in both directions.
- Both dialects are hand-rolled rather than taken from a vendor SDK, so a vendor's release schedule is not your release schedule.
- A workload moves between vendors by changing what an alias points at. No client-side change, no redeploy.
- Per-endpoint timeout, retry count, failure threshold and cooldown, all set in the console.
Drop-in
Nothing your developers have to learn.
Point any OpenAI SDK at the gateway, give it a virtual key, and ask for an alias instead of a model name. That is the whole integration.
Client API
POST /v1/chat/completions
POST /v1/audio/speech
POST /v1/completions
POST /v1/audio/transcriptions
POST /v1/embeddings
POST /v1/images/generations
POST /v1/responses
GET /v1/models
POST /v1/moderations
GET /v1/models/{id}
Upstream
OpenAI
Anthropic
Azure OpenAI
vLLM
OpenRouter
Ollama
anything OpenAI-compatible
Names are listed to say what TORHAUS.AI connects to. None of these companies endorses or is affiliated with this product, and all trademarks belong to their owners.
- Console, error messages and end-user documentation in German and English, switchable from the interface, with EUR and USD accounting side by side.
- Errors come back OpenAI-shaped, so the retry and error handling your clients already have keeps working.
How you get it
Three ways to run it. The software is the same one.
Every route starts with the same conversation, because the right answer depends on what your DPO, your Betriebsrat and your platform team each need to be able to say.
Managed by us
We run TORHAUS.AI for you on infrastructure in the European Union, and you get the console on day one instead of a deployment project.
- Optionally use models from every provider we support without opening a single vendor account, billed to you through Lime Labs on one invoice.
- Your teams, keys, budgets and guardrails, configured with you during onboarding.
- Prompts reach whichever model provider you choose, under that provider's own terms. We will tell you exactly which ones apply before you send anything real.
Self-hosted in your cloud
The binary, in your account, in your region, under your own Kubernetes or Compose, with your own Postgres and Valkey.
- Nothing is licensed at runtime, so there is no service that can switch you off.
- The operator guide covers what changes for production: TLS termination in front, the split admin port, database SSL.
- Support and hardening review from the people who wrote it.
On-premise and air-gapped
Your own hardware, your own network, and a gateway that will happily never see the internet.
- Turn the exchange-rate fetch off, or pin a static rate, and the only outbound connections left are the model endpoints on your own network.
- Pair it with a local vLLM or Ollama fleet and no prompt has anywhere to go but your own racks.
- Split the admin console onto its own port and keep it off the network your developers use.
Questions we actually get
The awkward ones, answered plainly.
Does TORHAUS.AI guarantee that our data stays in Europe?
No, and be careful with anyone who says their software does. The gateway does not record or check the geography of any destination. Residency is a consequence of where you deploy it and which endpoints you configure, not something a binary can assert.
What it does give you is the control that makes residency yours to decide: it runs wherever you put it, it calls nothing on its own initiative, and every upstream it talks to is one you entered by hand. On the EU-hosted plan, we run it in the European Union and tell you exactly which model providers your prompts reach.
Is the audit log tamper-proof?
It is a Postgres table, not a blockchain. There is no hash chain and no write-once storage, so anyone with database access can change it. Treat it the way you treat your other production data, and control access to it the same way.
What it is: a complete, per-request record of identity, routing, tokens, cost, guardrail verdicts and latency, written asynchronously so it never slows a request. If the writer backs up under load it drops rows rather than blocking the proxy, and it counts every drop so you can alert on it, or you can configure it to block instead and trade latency for completeness.
Are our prompts encrypted?
Provider credentials are sealed with AES-256-GCM and bound to their own database row, so a credential copied elsewhere in the database simply fails to decrypt. Virtual keys are stored only as digests and shown exactly once.
Captured prompts are not separately encrypted by the gateway: they live in your database, under whatever encryption at rest you already run there. Capture is off until you enable it for a named team or key, and redaction runs before anything is written. Transport is yours to terminate: run it behind your reverse proxy, set the production flag, and require SSL on any database that is not on localhost.
Will it redact everything sensitive automatically?
No. A fresh gateway enforces nothing until you bind your first rule, and that is deliberate: a redactor nobody chose is a redactor nobody trusts.
Once bound, the PII evaluator finds and rewrites email addresses, phone numbers, card numbers, IBANs, IP addresses, US social security numbers and API keys in the prompt text, with checksum validation so it survives real traffic. It reads the prose your users write; anything specific to your business is a regex rule you add. On a streamed response, text already sent to the client is recorded as a verdict but cannot be recalled.
Can we give different administrators different permissions?
Not yet. An instance is administered through a single bootstrap account, and every administrator account has the same full access, including the ability to read stored prompts where a team has capture enabled. That is worth knowing before you enable capture broadly.
What you can separate today is the network. The -admin-port flag puts the console on its own socket, after which the data plane serves only /v1 and answers admin paths with a pointed 404. Expose the data plane to your developers and keep the console on an internal network or behind a VPN.
How long does it take to get running?
On a laptop, minutes: bring up Postgres and Valkey, initialise the database, start the gateway, and the console prints a one-time bootstrap password it then forces you to rotate. From there it is one provider endpoint with its price list, one model alias, one organization, one team, one key.
For production, add the reverse proxy, real datastore credentials with TLS, and the split admin port. The Compose file in the repository is a development stack and should never be mistaken for a hardened deployment.
What does it cost?
It depends on which of the three routes you take and how many teams sit behind it, so we quote rather than publish a table. Book the demo and you will have a number in the same conversation.
Ready when you are
Put a gate in front of it.
Thirty minutes, your questions, and a live gateway rather than slides. Bring the person who will ask what happens to the prompts. That is the part we like doing.
Built and supported from Rosenheim, Germany