Model-agnostic optimization engine
This design is the engineering boundary for the InferCrane optimization PoC. It is intentionally operator- and workload-driven. A model name can select a reviewed recipe, but it can never establish that an optimization is compatible, correct, or faster.Scope and claim
The generic flow works for any immutable open-weight model that a supported runtime can load and expose through the serving contract. Specialized optimizations work only when InferCrane can discover the model’s operators, match an exact runtime/hardware capability, and collect the required evidence. Unsupported architecture features produce a conservative baseline and an explicit missing-capability result—not a guessed recipe. The planner and evidence protocol cover NVIDIA, AMD ROCm, Google TPU, and AWS Neuron. That is not a blanket hardware-support claim. At startup a configured Accelerator Lab worker must declare its installed profiler, runtime, topology, modality, and generated-kernel capabilities. InferCrane accepts a run only when that live catalog covers the exact request, and a result becomes qualified only after the same worker returns evidence for the exact immutable tuple.Workload contract
Optimization begins with a distribution, not a benchmark slogan:- input/output token distributions, request rate, concurrency, arrival pattern, streaming, tool use, structured output, multimodal inputs, and prefix reuse;
- TTFT, inter-token latency, end-to-end latency, goodput, error rate, quality, cost, energy, and memory constraints;
- prefill/decode mix, batch size, context growth, KV-cache residency, and request cancellation behavior;
- exact model revision, tokenizer, runtime/image, accelerator, driver/CUDA, topology, and region; and
- metadata-only, redacted replay, or in-customer-boundary data policy.
Search dimensions
This table is the design space, not current coverage.
Implementation coverage shows what campaigns can
vary today; every other family needs its own compatibility and measurement work
before it becomes a candidate.
No candidate family is universally beneficial. For example, prefix caching has
no value without reusable prefixes; speculative decoding loses when acceptance
is low; quantization can violate quality; tensor parallelism can lose to
communication; and a faster microkernel may not move end-to-end latency.
Candidate-generation algorithm
- Freeze identity. Hash model revision, tokenizer, runtime image, hardware, workload contract, evaluation set, and policy.
- Create controls. Produce at least one conservative vendor/runtime baseline. Never compare only optimized candidates with one another.
- Discover capability. Read model configuration and runtime support, then observe the actual operator graph and serving phases. Feature detection wins over family-name matching.
- Generate one-change candidates. Start with scheduler, cache, precision, decoding, topology, and hardware candidates whose preconditions are true. Preserve interactions as later composite candidates rather than changing everything at once.
- Reject cheaply. Apply compatibility, memory, quality-risk, cost, and historical-rejection filters. Use analytical roofline and Amdahl bounds.
- Screen. Run small deterministic correctness and performance lanes on the exact target accelerator. Failed candidates become durable search evidence.
- Profile the best control. Attribute prefill, decode, communication, CPU, queue, memory, and kernel time. Kernel work begins only here.
- Search kernels. Translate a measured operator hotspot into a typed problem, generate candidates through reviewed compiler backends, and verify them against the runtime reference on randomized/adversarial shapes.
- Qualify the Pareto set. Repeat AIPerf replay and customer quality tests; compare confidence intervals, SLO-qualified goodput, and landed cost.
- Promote separately. A human-approved Release Guard action creates a canary. Monitoring invalidates evidence when workload or identity drifts.
Continual optimization loop
The candidate pipeline above is now wrapped byinternal/continualoptimizer.
This outer loop is deliberately slower than request admission and fleet
autoscaling. It decides when the evidence justifies another bounded experiment;
it does not tune production inline.
public_prior; no number of public samples makes it
production promotion evidence. Once a customer endpoint has enough requests,
the loop replaces that prior with a content-free replay carrying only arrival,
duration, token counts, streaming, hashed session/prefix relationships, and
tool-pause timing. Prompt and completion content are not stored.
Every evaluation joins four independent inputs:
- workload shape and drift;
- correctness, quality, SLO, and availability;
- cost per productive token, contribution margin, and productive utilization;
- durable experiment memory, including failures and superseded candidates.
internal/store/migrations/070_continual_optimization.sql persists reviewed
policy, customer baseline, exact normalized inputs, immutable decisions, and
the last experiment boundary. The authenticated deployment API exposes policy,
evaluation, and decision history. Evaluation records a decision only; paid GPU
mutation continues through the existing approved optimization-campaign
workflow.
Capacity loop for marketplace traffic
Optimization and autoscaling solve different problems. Recipe search runs over hours or days. Admission reacts per request, and fleet scaling reacts over seconds or minutes.internal/autoscale.EvaluateCapacityEnvelope sizes the
fleet against four exact-workload capacity dimensions independently:
- requests per second;
- prefill tokens per second;
- decode tokens per second; and
- concurrent requests.
internal/autoscale.CapacityController applies that safe admission envelope
before asking a provider to add or remove replicas. Ready, pending, and draining
capacity are tracked separately. Pending GPUs prevent duplicate scale requests
but never count as request-ready capacity. Provider mutation failures are
recorded and leave admission bounded by the currently ready fleet.
For example, 1,450 qualified output tokens/second is about 125 million output
tokens/day. It corresponds to roughly 1.1 billion total tokens/day only for the
specific 4,000-input/512-output mix and only if the same replica also sustains
the required prefill rate under the target TTFT. A 4.7-billion-total-token day
at that mix requires roughly 48,225 input and 6,173 output tokens/second. With
70% target productive utilization, a 1.15 demand multiplier, and one failure
reserve, decode alone calls for seven serving replicas plus one reserve. The
production decision must take the maximum across all four capacity dimensions.
Candidate ranking is constrained optimization, not one scalar benchmark:
Kernel opportunity gate
The newkernelplanner accepts a profile manifest with typed operator hotspots.
It is independent of the model repository name. Supported initial families are:
- residual-add + RMSNorm;
- quantized linear/dequant + GEMM or GEMV;
- SwiGLU activation;
- prefill and decode attention;
- KV-cache append/quantization;
- recurrent/linear-attention state updates for hybrid and state-space models;
- recurrent-state commit, scatter, and movement;
- MoE routing/grouped GEMM; and
- token sampling.
f, the maximum possible end-to-end speedup is:
S, the kernel itself must reach:
tools/kernel-lab/fused_residual_rmsnorm.py. Its dependency-free laptop path
emulates the fused dataflow. Its optional Triton path performs exact-GPU parity
and a microbenchmark. tools/kernel-lab/fused_residual_rmsnorm_cuda.py applies
the same contract to visible handwritten CUDA C++ through NVRTC. On the measured
H100 shapes that CUDA candidate was correct but slower than both Triton and the
vLLM vendor operator, so it was rejected. The result remains operator-level
evidence until a runtime-level AIPerf campaign passes.
Compiler/backend strategy
InferCrane owns the typed problem, evidence, and selection policy—not a single kernel language:- Triton for productive fusion and shape specialization;
- CUTLASS/CuTe DSL for Tensor Core GEMM, grouped GEMM, low precision, and architecture-specific control;
- FlashInfer as the vendor-grade attention/GEMM/MoE/sampling baseline and reusable generator;
- CUDA C++ or CUDA Rust when low-level SIMT control, a Rust systems stack, or Tile portability justifies it;
- KernelBench task and correctness methodology for generated-kernel evaluation; and
- Mirage as a later whole-graph or persistent-megakernel search backend, never as automatic evidence authority.
Autonomous kernel research incorporated
The implementation follows the common result from current kernel-agent work: generation is a search primitive, not an evidence authority.- KernelBench supplies the functional correctness plus target-device speed evaluation shape. InferCrane also uses the project’s stricter evaluation guidance: hidden/adversarial shapes, suspicious-speedup review, and independent verification.
- KernelBench-Verified and RealisticTritonBench reinforce why realistic operators, stronger tests, and contamination-resistant evaluation matter more than a single public-shape score.
- CUDA Agent, KernelPro, and CudaForge motivate iterative profile/generate/compile/measure loops. InferCrane adopts the loop but places generation behind Amdahl materiality, a hard spend/expiry lease, an isolated Brezel build, exact-target sanitizers, and full serving replay.
- AutoKernel contributes the deliberately simple inner loop: change one kernel revision, run five-stage correctness, benchmark it, retain only a material improvement, and move on after a plateau, roofline, time, iteration, or target-speedup limit. InferCrane uses that loop as a model-neutral worker contract rather than a Qwen-specific script. The retained kernel still has to pass the wider serving and Release Guard gates below.
- AIConfigurator informs the broader configuration search. Its proposals remain candidates until InferCrane’s model/runtime/hardware/workload evidence gates pass.
- NVIDIA SOL-ExecBench is a useful independent execution benchmark for generated GPU programs; it complements, rather than replaces, workload-specific end-to-end qualification.
internal/kernelresearch implements the reusable AutoKernel-style state
machine. Its intent binds the search to the exact input, hotspot candidate,
profile artifact, model revision, runtime image, hardware, and workload. Its
receipt records every keep/reject decision, all five correctness stages,
baseline and candidate timing, roofline utilization, cost, plateau state, and
the immutable winning source revision. The residual-RMSNorm evaluator computes
roofline utilization as the time the operator’s minimum DRAM traffic would take
at the GPU’s peak bandwidth, divided by the measured time. It reports 0, which
never stops a search, when the bandwidth is unknown or the timing was not
DRAM-bound. The receipt is content-addressed and tampering fails validation.
internal/acceleratorlab passes that intent through its opt-in generator
contract. A worker may iterate in Triton, CUDA C++, CuTe, Pallas, NKI, HIP, or
another declared backend; this is not tied to a model name. The generated
source bundle must match the retained winner, echo the input digest, candidate
identity, and exact profile artifact digest, include immutable revision,
license, content hashes, and sizes, build through the fixed Brezel runner, and
then pass the same target qualification gates as reviewed code. The system
never promotes it automatically.
Technique evidence map
The search space is grounded in primary work and maintained implementations:- FlashAttention: IO-aware exact attention.
- PagedAttention/vLLM: paged KV-cache memory management and serving throughput.
- SGLang/RadixAttention: structured-program execution and prefix reuse.
- Sarathi-Serve: chunked prefill and stall-free scheduling.
- Speculative decoding and EAGLE: verified parallel token generation.
- GPTQ, AWQ, and SmoothQuant: weight-only and W8A8 post-training quantization families.
- KIVI: asymmetric low-bit KV-cache quantization.
- vLLM, SGLang, and TensorRT-LLM: production references for runtime, scheduling, precision, graph, speculative, and distributed paths.
Implementation coverage
The Wafer GPU performance engineering resource map is a useful field checklist, not a feature checklist InferCrane can honestly mark complete. The current implementation boundary is:
For the MVP, the defensible product is the locally qualified control plane plus
the real tuples whose worker evidence has passed. An installed adapter or a
compiled contract is never rendered as a speed, capacity, hardware, or modality
claim.
Accelerator Lab product wiring
InferCrane is split into a durable, provider-neutral control service and small target-compute adapters. A campaign directory such astools/modal-openrouter-qwen38 is an executable example for one pinned tuple;
it is not the optimization algorithm.
cmd/infercrane-accelerator-worker is the deployable worker. It exposes the
existing acceleratorlab.WorkerClient contract, persists idempotency receipts
across restarts, brokers content-addressed build artifacts, and invokes an
absolute provider adapter executable without a shell. The adapter receives a
typed JSON request on stdin and returns typed evidence on stdout. Provider
credentials are explicitly supplied from an owner-only environment file; the
worker does not inherit the control-plane environment.
This supports previously unseen model repositories without adding a model-name
branch. “Model agnostic” does not mean every model is automatically runnable:
the pinned model must load in a declared runtime, fit a declared target, and
pass the workload, quality, and compatibility gates. Unsupported tuples fail
closed and remain useful negative evidence.
The authenticated surface is:
GET /api/v1/optimization/accelerator-lab/capabilities— shows the live worker catalog and whether execution is configured;POST /api/v1/optimization/accelerator-lab/runs— validates the tenant, immutable request, 24-hour authority ceiling, cost cap, and live worker capability before enqueueing a durable operation;POST /v1/profiles,POST /v1/kernel-generations, andPOST /v1/qualifications— server-to-worker calls, never browser calls; and/v1/artifacts/...— same-origin, content-addressed exchange between the trusted worker and a deny-by-default Brezel builder.
FastPath PoC migration
infercrane-fastpath-poc remains a research repository; it is not copied into
the control-plane binary or imported at runtime. Its production-worthy
concepts have been absorbed as typed Go policy:
The old Python
CandidateGenerator was only an interface and the PoC registry
kept generator adapters blocked until targeted NCU evidence existed. InferCrane
now has the full typed generation-to-evaluation orchestration, and
cmd/infercrane-kernel-adapter plus internal/kernelgen supply a generator.
Each deployment must still declare and prove its model endpoint, evaluator
image, and vendor workers. This is deliberate separation, not duplicate product
logic.
Kernel registry and search order
internal/kernelplanner owns a reviewed registry of implementation sources.
The registry is not a live GitHub crawler and it does not import code into the
API process. Each entry records the project, HTTPS source, license, backend,
operator families, hardware and dtype filters, revision policy, trust status,
and execution boundary.
Search is deterministic and existing-first:
- measure the implementation bundled in the pinned vLLM or SGLang image;
- inspect compatible specialized/vendor libraries such as FlashInfer and CUTLASS/CuTe;
- adapt or specialize a reviewed Triton/CuTe implementation;
- generate handwritten CUDA only after the prior candidates lose and the profiler-backed Amdahl ceiling justifies the experiment.
POST /api/v1/optimization/kernel-opportunities endpoint
returns this bounded plan with code_execution: false, provider_mutation: false, and performance_claims: false.
Deploy, observe, improve
Deployment and optimization are one lifecycle:starting_point. It is selected
from compatibility and declared objective fit, not an unmeasured speed claim.
After deployment, GET /api/v1/endpoints/{name}/optimization-readiness turns
fresh request aggregates into a conservative stage diagnosis such as queueing,
prefill/first-token, or decode/generation pressure. When a fresh serving-engine
scrape or DCGM profiling import exists, the same assessment applies explicit
KV-pressure, preemption, prefix-reuse, speculative-acceptance, and GPU memory,
compute, host, and interconnect rules and returns the technique families worth
measuring for the diagnosed bottleneck
(thresholds).
A proposal that names diagnosis_endpoint closes the loop: the control plane
reads that assessment and a recent GPU profile summary (kernel shares, plus the
fired findings of a v2 capture’s trace analysis with their measured values and
thresholds), binds them into the proposal input digest, and the planner’s typed
rule set (diagnosis-technique-rules-v4 in internal/optimizer) keeps only the
starting point and the one-change setups its rules choose, ordered by expected
benefit with the reason recorded on each. Flags equal to the engine’s resolved
default are dropped, and when no rule applies only the starting point is
measured while the plan names the evidence to collect
(rules).
The plan chooses what to measure; the campaign’s gates decide what wins.
No request, engine, or DCGM telemetry opens the kernel gate: exact target-GPU
profiling and the Amdahl policy remain mandatory.
Process boundary
Do not move compiler, profiler, or arbitrary candidate-source execution into the InferCrane API server.- InferCrane control plane: workload identity, registry metadata, policy, plans, approvals, budget leases, evidence, Release Guard, and audit.
- Brezel worker: isolated source checkout, license/provenance inspection,
source screening, CPU-side compilation and semantic tests, artifact signing,
and lifecycle receipts. Network access is deny-by-default and source commits
are explicit allowlisted inputs. The baked entrypoint is
tools/kernel-lab/brezel-runner/run-optimization-job; it never executes the source it inspects. - Kernel adapter: runs on the accelerator worker and calls the configured
model endpoint. The model has no tools and returns source text only. Drafts
execute solely through the isolated evaluator, a no-network container with no
credentials, and
kernelresearchalone decides keep or reject. - GPU worker: exact-SM build when required, correctness, sanitizers,
PyTorch profiler traces, microbenchmarks, AIPerf replay, quality tests, and
teardown under the campaign lease. An opt-in
timelineoperation wraps the serving process in Nsight Systems for one bounded CUDA-profiler range. It returns only a summary of GPU idle time, launch overhead, communication share, and the top kernels, and it refuses a runtime without a reviewed configuration or withoutnsysin its image.
internal/brezelexecutor is the implemented adapter boundary. It creates or
adopts a sandbox from a pinned Brezel environment revision, writes a typed
content-addressed job manifest, invokes only the baked-in
/opt/infercrane/bin/run-optimization-job entrypoint, validates a typed result,
deletes the sandbox, and retains the lifecycle receipt. Manifests cannot carry
shell commands, environment variables, credentials, or an alternate runner.
Brezel egress remains disabled; approved source access belongs in Brezel
connector policy.