Skip to main content

Run on this machine

Connect a model server that already runs on your own hardware: vLLM, SGLang, llama.cpp, Ollama, or LM Studio, on a laptop, a workstation, an on-prem box, or a cloud VM. The InferCrane agent finds the server, reports its health, engine, model, and hardware, and relays its engine metrics so the endpoint page and the bottleneck diagnosis work. It only dials out over HTTPS, so nothing has to be reachable from the internet.
Prompts and completions never leave the machine. The agent sends hardware and server details, health, and engine metrics filtered to the families InferCrane reads. Request timings and token counts are sent only if you route traffic through its local gateway. See what leaves the machine.
1

Start your model server

Any of these, on its usual port, works as it is:
vLLM serves engine metrics by default. Start SGLang with --enable-metrics and llama.cpp with --metrics to get them. Ollama and LM Studio do not expose engine metrics, so their queue and cache internals stay unknown.
2

Check what the agent would connect

detect looks on ports 8000, 30000, 8080, 11434, and 1234 and sends nothing:
Name a server on another port or host with --server http://127.0.0.1:9000.
3

Run the agent

The first run creates an agent key with your sign-in and saves it in the OS keychain. Each healthy server becomes an observe-only endpoint named after its model, for example qwen3-8b. Keep the agent running (in a terminal, tmux, or a service manager); it syncs every 15 seconds.
4

Open the endpoint

Running and waiting requests and KV-cache use appear with the first sync. Queue time and time to first token come from the difference between two scrapes, so they appear about 30 seconds later, once requests have been served.

A server that needs an API key

Keep the key in an environment variable, name the variable, and name the server. The agent sends the key only to the servers you name with --server, never to InferCrane:

See request timings

The agent cannot see traffic that goes straight to the server, and it says so. To record request timings and token counts, start its local gateway and point your application at it:
The gateway forwards every request unchanged. If you stop the agent, point your application back at the server.

A machine where nobody signs in

Create an agent key once from a signed-in machine (or the API), then pass it to the agent:
The agent honours HTTPS_PROXY. It needs outbound HTTPS to the control plane and nothing inbound.

Stop or revoke

Ctrl-C stops the agent. To cut it off from InferCrane for good:
A revoked agent stops at its next sync, within one poll. Its endpoints stay, marked unhealthy.

What it does not do yet

  • It does not start, restart, or tune your server. Running candidate setups on your own GPUs is the next step; see candidate runs.
  • Apple Silicon and AMD GPUs are reported as detected; their GPU telemetry is not collected.
  • Requests through the InferCrane gateway cannot reach a server that only the agent sees. Use the agent’s local gateway instead.

Next