​

PostgreSQL backup and restore drill

scripts/backup-postgres.sh infercrane-$(date +%Y%m%d).dump

export INFERCRANE_ALLOW_RESTORE=yes
export INFERCRANE_RESTORE_TARGET_DATABASE=infercrane_restore_drill
scripts/restore-postgres.sh infercrane-20260809.dump

​

Reconcile restored state with real infrastructure

psql "$INFERCRANE_DATABASE_URL" -X -v ON_ERROR_STOP=1 --csv \
  -c 'SELECT id,name,active_revision_id,candidate_revision_id,desired_state
      FROM deployments ORDER BY tenant_id,name' \
  > restored-deployments.csv

psql "$INFERCRANE_DATABASE_URL" -X -v ON_ERROR_STOP=1 --csv \
  -c 'SELECT tenant_id,deployment_id,revision_id,ordinal,external_key,provider,
             provider_request_id,provider_resource_id,lifecycle_state
      FROM replicas ORDER BY tenant_id,deployment_id,revision_id,ordinal' \
  > restored-replicas.csv

psql "$INFERCRANE_DATABASE_URL" -X -v ON_ERROR_STOP=1 --csv \
  -c 'SELECT id,kind,status,idempotency_key,lease_owner,lease_generation
      FROM operations
      WHERE status NOT IN (''succeeded'',''failed'',''cancelled'')
      ORDER BY created_at,id' \
  > restored-nonterminal-operations.csv

evidence="restore-evidence/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$evidence"
mv restored-deployments.csv restored-replicas.csv \
  restored-nonterminal-operations.csv "$evidence/"

runpod_key=$(tr -d '\r\n' < "$RUNPOD_KEY_FILE")
curl -fsS -H "Authorization: Bearer $runpod_key" \
  https://rest.runpod.io/v1/pods |
  jq -S '[.[] | select((.name // "") | startswith("infercrane-")) |
    {id,name,desiredStatus}] | sort_by(.id)' \
  > "$evidence/provider-pods.json"
curl -fsS -H "Authorization: Bearer $runpod_key" \
  'https://rest.runpod.io/v1/endpoints?includeWorkers=true' |
  jq -S '[.[] | select((.name // "") | startswith("infercrane-")) |
    {id,name,workersMin,workersMax}] | sort_by(.id)' \
  > "$evidence/provider-endpoints.json"
unset runpod_key

aws ec2 describe-instances --region "$INFERCRANE_AWS_REGION" \
  --filters Name=tag:infercrane:managed,Values=true \
    Name=instance-state-name,Values=pending,running,stopping,shutting-down \
  --query 'Reservations[].Instances[].{id:InstanceId,state:State.Name,external_key:Tags[?Key==`infercrane:external-key`]|[0].Value}' \
  --output json --no-cli-pager | jq -S 'sort_by(.id)' \
  > "$evidence/provider-inventory.json"

gcloud compute instances list \
  --project "$INFERCRANE_GCP_PROJECT" \
  --filter='labels.infercrane-managed=true' \
  --format=json --quiet |
  jq -S '[.[] | {id,name,status,zone,labels}] | sort_by(.id)' \
  > "$evidence/provider-inventory.json"

kubectl --context "$INFERCRANE_KUBERNETES_CONTEXT" \
  --namespace "$INFERCRANE_KUBERNETES_NAMESPACE" \
  get deployment,service \
  -l app.kubernetes.io/managed-by=infercrane -o json |
  jq -S '[.items[] | {
    kind,uid:.metadata.uid,name:.metadata.name,
    external_key_hash:.metadata.labels["infercrane.dev/external-key-hash"]
  }] | sort_by(.kind,.name)' \
  > "$evidence/provider-inventory.json"

if kubectl --context "$INFERCRANE_KUBERNETES_CONTEXT" \
  api-resources --api-group serving.kserve.io -o name |
  grep -qx inferenceservices; then
  kubectl --context "$INFERCRANE_KUBERNETES_CONTEXT" \
    --namespace "$INFERCRANE_KUBERNETES_NAMESPACE" \
    get inferenceservices.serving.kserve.io \
    -l app.kubernetes.io/managed-by=infercrane -o json |
    jq -S '[.items[] | {uid:.metadata.uid,name:.metadata.name}] | sort_by(.name)' \
    > "$evidence/provider-kserve-inventory.json"
fi

curl -fsS "$INFERCRANE_URL/readyz"
infercrane system instances --output json
infercrane deployments --output json
infercrane orphans --output json
infercrane inspect DEPLOYMENT --output json
infercrane inbox --output json
infercrane operation watch OPERATION_ID
infercrane events DEPLOYMENT --output json

​

Recover the active control plane after a failed migration

1. Retain the failed/migrated database read-only and export the available post-upgrade evidence using the evidence-preserving rollback procedure.
2. Have the database operator create a separate empty rollback database. Set the TLS-protected INFERCRANE_DATABASE_URL to that database, not to the failed database - and verify its name before restore.
3. Restore the pre-upgrade dump:
   export INFERCRANE_ALLOW_RESTORE=yes
   export INFERCRANE_RESTORE_TARGET_DATABASE=infercrane_rollback
   scripts/restore-postgres.sh /secure/path/infercrane-before-upgrade.dump
   
4. Keep every InferCrane replica stopped. Use the offline PostgreSQL exports and provider inventory procedure above to compare the restored active revision, candidate, nonterminal operations, provider identities, external keys, replica ordinals, and desired replica count with the failed-database exports and direct provider inventory. The pre-upgrade backup is authoritative through its timestamp; later provider state is an observation to reconcile, not permission to overwrite history.
5. Keep reconciliation and provider cleanup stopped while any identity is missing, duplicated, or mismatched. Never run delete, create a replacement deployment, or clear an orphan merely to make the restored database resemble current inventory.
6. Only after every owned resource maps unambiguously to one restored replica intent, start exactly one control-plane replica at the pre-upgrade application version. Do not start a mixed old/new set. Wait for /readyz, then collect API state:
   infercrane system instances --output json
   infercrane deployments --output json
   infercrane rollout inspect DEPLOYMENT --output json
   infercrane inspect DEPLOYMENT --output json
   infercrane orphans --output json
   
7. Confirm adoption without a provider create, send one bounded request through the unchanged active endpoint, inspect its X-Request-Id, and only then restore the normal same-version control-plane replica count.