Skip to main content

​
Reliability test environment

InferCrane uses an evidence ladder. Every failure is tested at the cheapest boundary that can faithfully reproduce it, while claims that depend on a real provider, GPU, runtime, or customer network remain explicitly unqualified until that boundary runs.

​
Test tiers

The machine-readable inventory is qualification/product-gates.json. Each result is bound to a Git commit and has its own status. Successful cleanup never overwrites a failed suite.

​
Provider simulation

AWS, GCP, RunPod, Kubernetes, external capacity, vLLM, SGLang, and custom OCI adapters use fault-injectable contracts. Fixtures can return lost create responses, delayed observations, mismatched identities, invalid success bodies, rate limits, malformed/truncated data, eventual deletion, and credential-bearing errors. The same conformance properties apply to every adapter:
  • stable intent identity and at-most-one owned resource;
  • restart-safe adoption after ambiguous create;
  • fail-closed immutable configuration matching;
  • idempotent deletion and independently verified inventory;
  • bounded, normalized, secret-free error evidence.
Run the complete hermetic cloud boundary directly with:
LocalStack is not a required dependency. It can be used as an optional AWS API compatibility probe, but its mock EC2 manager supplies CRUD semantics, Docker-backed EC2 has networking limitations on macOS, and IAM enforcement depends on configuration and plan. InferCrane’s deterministic adapter fixtures are faster and more precise for control-plane fault injection; real AWS remains the qualification boundary for IAM propagation, VPC behavior, capacity, quota, and eventual consistency. Google does not ship a Compute Engine emulator. Its official emulators cover data services such as Firestore, Spanner, Pub/Sub, Bigtable, and Datastore. InferCrane therefore keeps a deterministic Compute API/CLI model locally and requires an isolated GCP project for real VM evidence.

​
Kubernetes fidelity and scale

Kind boots a real Kubernetes API server, controller manager, scheduler, kubelet, and server-side apply implementation. InferCrane tests ownership, RBAC, stale observedGeneration, field conflicts, partial owner repair, restart adoption, asynchronous deletion, and clean inventory against the current API. The scheduled matrix repeats this lifecycle against Kubernetes 1.34, 1.35, and 1.36 using digest-pinned Kind images. KWOK adds a different boundary: hundreds of simulated nodes and InferCrane-managed workloads with very low resource usage. The fleet test proves that missing GPU capacity stays provisioning and unroutable, large inventory remains coherent, half-fleet node loss cannot manufacture readiness, and bulk cleanup reaches zero. KWOK does not run containers or GPUs, so it never qualifies runtime startup or performance.

​
Fuzz and soak

Go’s coverage-guided fuzz engine continuously mutates critical state-machine inputs. A discovered failure is minimized and committed to the corpus so ordinary test runs preserve it as a regression. The soak tier repeatedly shuffles and race-tests operation leasing, async execution, admission, quotas, autoscaling, routing, reconciliation, providers, and workflows. When INFERCRANE_TEST_DATABASE_URL is present it also repeats PostgreSQL fencing and contention cases.

​
Network and process faults

InferCrane prefers deterministic HTTP fixtures when an HTTP semantic matters and real process/TCP boundaries when connection behavior matters. The local recovery suite kills workers, routers, control-plane replicas, and PostgreSQL connections; provider fault proxies inject timeouts and lost responses. The scheduled Toxiproxy gate cuts every PostgreSQL connection in a full Docker stack and proves that readiness fails closed, an already-published inference route remains available without a database lookup, and the control plane recovers after connectivity returns. Future latency, reset, bandwidth, partial-transfer, and slow-close cases belong in the same boundary; this complements—not replaces—assertions about durable state and ownership.

​
Evidence retention and release policy

The nightly GitHub workflow uploads qualification evidence for 30 days. Release evidence remains under .infercrane/product-qualification/<commit> locally and should be attached to the RC. A release decision must inspect:
  1. all local gates for the exact commit;
  2. the most recent scheduled gates on the same lineage;
  3. every real-provider/runtime combination advertised as qualified;
  4. independent zero-resource inventory after paid suites;
  5. human terminal, dashboard, docs, and clean-install review.
No aggregate percentage can erase a missing boundary. For example, 100% fixture coverage still does not qualify AWS IAM or CUDA behavior.

​
Primary design references