Run on this machine
Connect a model server that already runs on your own hardware: vLLM, SGLang, llama.cpp, Ollama, or LM Studio, on a laptop, a workstation, an on-prem box, or a cloud VM. The InferCrane agent finds the server, reports its health, engine, model, and hardware, and relays its engine metrics so the endpoint page and the bottleneck diagnosis work. It only dials out over HTTPS, so nothing has to be reachable from the internet.Prompts and completions never leave the machine. The agent sends hardware and server details,
health, and engine metrics filtered to the families InferCrane reads. Request timings and token
counts are sent only if you route traffic through its local gateway. See
what leaves the machine.
1
Start your model server
Any of these, on its usual port, works as it is:vLLM serves engine metrics by default. Start SGLang with
--enable-metrics and llama.cpp with
--metrics to get them. Ollama and LM Studio do not expose engine metrics, so their queue and
cache internals stay unknown.2
Check what the agent would connect
detect looks on ports 8000, 30000, 8080, 11434, and 1234 and sends nothing:--server http://127.0.0.1:9000.3
Run the agent
qwen3-8b.
Keep the agent running (in a terminal, tmux, or a service manager); it syncs every 15 seconds.4
Open the endpoint
A server that needs an API key
Keep the key in an environment variable, name the variable, and name the server. The agent sends the key only to the servers you name with--server, never to InferCrane:
See request timings
The agent cannot see traffic that goes straight to the server, and it says so. To record request timings and token counts, start its local gateway and point your application at it:A machine where nobody signs in
Create an agent key once from a signed-in machine (or the API), then pass it to the agent:HTTPS_PROXY. It needs outbound HTTPS to the control plane and nothing inbound.
Stop or revoke
Ctrl-C stops the agent. To cut it off from InferCrane for good:
What it does not do yet
- It does not start, restart, or tune your server. Running candidate setups on your own GPUs is the next step; see candidate runs.
- Apple Silicon and AMD GPUs are reported as detected; their GPU telemetry is not collected.
- Requests through the InferCrane gateway cannot reach a server that only the agent sees. Use the agent’s local gateway instead.