Failures: don’t turn one timeout into ten requests

Picture it

When a massive GPU suddenly drops offline, it's like a bridge collapsing under heavy traffic. If every stranded driver immediately hits 'refresh' and blindly charges forward, the sudden surge will quickly overwhelm and collapse the neighboring bridges too. A robust system doesn't just blindly retry; it pauses, adds a little randomness to the delay, and carefully meters out the recovery traffic to ensure the surviving infrastructure isn't crushed by the echo of the failure.

What to understand

• OOM, GPU errors, worker death, network partitions, and health probes
• Exponential backoff with jitter, retry budgets, idempotency, and resumability

Follow the references

1. 1 · Book chapter

   Google SRE · Handling overload ↗

   Study retry amplification and cascading overload.

2. 2 · Documentation

   NVIDIA · DCGM feature overview ↗

   Find GPU health and diagnostic signals to connect application errors with hardware issues.

Make it concrete

Terminate a test worker mid-stream. Observe client errors, cache cleanup, retry volume, and healthy-replica latency. Document whether the client restarts, resumes, or surfaces failure.