DeepSeek API Reliability: Timeouts, Retries, and Backpressure

A reliable DeepSeek integration needs a policy for what happens after the easy path fails. A request can wait before inference begins, lose its connection halfway through a stream, receive an overload response, or finish after the person who asked for it has already left. Repeating the same call immediately can turn one delayed answer into several expensive attempts. If generated output later triggers a payment, an email, or a database change, a careless retry can also repeat a real action.

The practical starting point is to give each logical operation one deadline, a small attempt limit, and an explicit decision about whether replay is acceptable. Add a shared retry budget, bounded concurrency, and a queue that refuses work it cannot finish. Treat the response as complete only after its protocol and application checks pass. Those controls are useful independently: jitter spreads attempts over time, while admission control prevents your own application from creating more work than it can safely hold.

This guide develops that policy for production DeepSeek clients. It includes a transport-neutral Python controller and deterministic tests, with no API key, paid request, or claimed provider benchmark. It assumes you already know how to authenticate and send a basic request; the DeepSeek API guide covers that setup. The numbers below are example application settings unless explicitly identified as current provider documentation. They are starting hypotheses to test against your workload, not universal values that every deployment should copy.

Read the current DeepSeek contract before choosing limits

DeepSeek’s published limits and model names can change, so old advice should not become a permanent infrastructure assumption. On October 2, 2026, its Rate Limit documentation describes account-level concurrent-request caps of 2,500 for deepseek-flash and 500 for deepseek-v4-pro, with a 429 response when the relevant cap is exceeded. These are concurrent-request limits. They do not establish a universal requests-per-minute or tokens-per-minute allowance, and they do not tell you how many simultaneous requests your own service should admit.

A service with eight web workers, a small connection pool, and a tight response-time objective may need an application limit far below the provider ceiling. Other services using the same account can also consume its available concurrency. Keep provider limits in configuration and review the current documentation before a capacity change. Do not derive a single global number by adding the two documented model caps together, or assume a limit observed in one account is an allowance for every endpoint and deployment arrangement.

The same documentation explains a waiting behavior that matters for timeouts. When a request has been received but inference is waiting to start, the server can keep the connection alive with empty lines for non-streaming requests or SSE comment lines for streaming requests. If inference has not started after ten minutes, the server closes the connection. The condition is important: this is not a guarantee that every completion finishes within ten minutes, and it is not a recommendation that an interactive application should make a person wait that long.

Transport activity, meaningful model output, and a completed answer are therefore three different events. An SSE keepalive can demonstrate that a connection is active without showing that an answer is progressing. Your client, reverse proxy, and user interface may each impose their own timers. Document which timer is authoritative and check how intermediate components handle those keepalives. An idle timeout that keeps resetting on comments still needs an independent overall deadline.

A 429 should not be diagnosed from concurrency alone: the Chinese Error Codes page also describes TPM/RPM limits, but no numerical RPM/TPM entitlement was found in the reviewed pages. The public pages reviewed for this article do not establish a general DeepSeek idempotency-key contract, a promise that every retry receives a Retry-After header, or exactly-once execution of your downstream actions. The design below does not depend on any of those guarantees. If you integrate through a gateway, its documented contract may add different behavior; verify that separately rather than attributing gateway behavior to the provider.

Classify the failure before deciding to repeat it

An HTTP status gives you evidence about a particular attempt, but the decision to retry belongs to the logical operation. DeepSeek’s Error Codes page distinguishes malformed requests and invalid parameters from authentication, balance, rate-limit, server-error, and overload conditions. Preserve that distinction in code. Retrying a broken schema unchanged spends time without repairing it, while repeatedly submitting a rejected credential can bury the original configuration problem beneath hundreds of redundant log entries.

The final column is an application policy, not an additional provider guarantee. A 429 or 503 may remain unresolved longer than your user-facing deadline. A transport timeout is even less conclusive: your client may have stopped waiting after the server accepted and began processing the request. A failed connection attempt before anything was sent is different from losing a connection after sending the body, but many application-level error paths cannot prove exactly where that boundary fell.

Use a small allowlist of retry candidates rather than “retry every exception.” The controller later in this guide considers the documented 429, 500, and 503 statuses and an explicitly classified transport failure. It does not automatically retry parsing bugs, programming errors, or every other 4xx and 5xx response. Additional statuses require an intentional extension supported by the endpoint or intermediary you actually use. Keep the original error as the cause when a budget stops further attempts, so incident responders can see both the trigger and the reason the operation ended.

Finally, distinguish a recoverable dependency failure from a valid response that your application cannot use. Malformed model output, an incomplete JSON object, or a refusal needs its own handling. The structured-output guide explains schema validation and incomplete results. Feeding those cases into the network retry loop can hide a systematic prompt or validation problem and silently multiply generation cost.

Give the whole operation one deadline

A per-attempt time budget limits what your application is willing to spend on one attempt; the transport must be configured to enforce it. An operation deadline answers how long the user or background job is willing to wait for the final outcome. You need both. If every attempt receives a fresh thirty-second allowance, three attempts plus queueing and backoff can exceed a thirty-second product promise even though each individual request obeys its own timer. Propagate the original deadline through the layers instead of resetting it at each function boundary.

Start the budget when your application accepts responsibility for the operation. Include time spent in your admission queue, obtaining a connection, establishing a connection, uploading the request, waiting for output, reading the response, and sleeping between retries. A worker that receives an already queued job should get the remaining duration or a shared deadline, not a new full allowance. Across machines, clock differences need an explicit protocol; within a process, use a monotonic clock for elapsed time so wall-clock adjustments do not extend or shorten the budget unexpectedly.

Before each attempt, calculate the remaining time. The attempt receives the smaller of that remainder and its normal cap. Before a retry sleep, check that the proposed delay leaves enough useful time for another attempt. A one-millisecond remainder might technically be positive while still making a meaningful completion impossible. The minimum useful attempt budget in the example is configurable for this reason. Measure realistic connection and output costs to choose it; the tutorial’s small value makes offline behavior easy to inspect.

Timeout APIs deserve close reading. Connect, read, write, and connection-pool timeouts are often phase or inactivity limits, not a hard upper bound on the complete operation. A stream that keeps producing occasional bytes can outlive a read timeout measured between chunks. In an asynchronous client with cancellation-aware, nonblocking transport, a surrounding cancellation scope can enforce a caller’s deadline while the transport still has phase-specific protection. Wrapping a blocking synchronous sender directly in an asynchronous timeout does not make that sender preemptible. Cancellation is cooperative, and aborting the local await does not establish that the remote server stopped processing or that no billable work occurred.

There is also a product decision at the deadline. You might return a clear failure for an interactive request, keep a user-authorized background job running, or offer an explicit retry action. Do not quietly start unbounded work after reporting failure. If a late result must be discarded, record that as a late outcome rather than counting it as a successful on-time answer. The sample controller rejects a returned result after its deadline, but a synchronous callback that ignores its budget can still hold the calling thread until it returns. That limitation is stated in the code and must be solved in the live transport integration.

Limit retries per operation and across the service

A maximum attempt count prevents one operation from retrying forever. It does not prevent a fleet of operations from retrying together. Imagine one hundred original requests during an outage, each allowed three attempts. If they all fail in the same way, the service can issue three hundred attempts. Calling that setting “three retries” would allow four attempts instead. Use the word attempts consistently, count the initial request, and expose separate metrics for original work and retry work.

Layered retry policies multiply this effect. An SDK that performs its own retries inside an application retry loop can create more network attempts than the outer loop records. The current OpenAI Python SDK documentation says certain failures are retried twice by default, including connection errors, 408, 409, 429, and responses at or above 500. That is SDK behavior, not DeepSeek’s list of error codes. When one layer owns the policy, configure the SDK deliberately, commonly with max_retries=0 if the application owns all retries, and account for gateway or job-queue retries as well.

A shared retry budget limits the extra work the service can create. The example uses a token bucket: original attempts do not consume retry tokens, while every launched retry does. Tokens can replenish at a controlled rate. This is a teaching implementation scoped to one process, with a lock for threads sharing the same instance. Eight processes each holding an independent bucket have eight separate allowances. Account-wide protection requires coordination or carefully partitioned allowances, and it still needs a separate limit on original incoming work.

Choose the budget using an explicit failure scenario. Ask how much extra traffic you can tolerate while the dependency is unhealthy and how quickly recovery should refill retry capacity. A small budget allows occasional transient recovery without offering every failed request the full attempt allowance. Exhaustion should be a visible terminal reason, not an invitation to create a fresh operation automatically. Otherwise a caller can bypass the retry cap simply by renaming the same failed work.

Use exponential backoff with jitter and a deadline check

Exponential backoff increases the maximum waiting interval after successive failures. Full jitter chooses a delay uniformly between zero and that interval. For a base of half a second and a four-second cap, the first retry window is zero to half a second, then zero to one second, then zero to two seconds, until the cap applies. These are example client settings. The increasing window reduces repeated pressure, and randomness makes it less likely that many clients which failed together will return together.

Jitter is not a concurrency limiter or a guarantee of fairness. A random draw can be zero, and a large population still produces traffic throughout the window. Fixed backoff without randomness often produces visible waves after a shared incident. A constant small random delay may smooth traffic but does not expand the quiet period as failures continue. Select one clearly specified strategy, test its boundaries, and record actual waits so you can see whether the deployed policy behaves as intended.

If a trusted response includes a valid standard Retry-After value, the example treats it as a minimum delay by taking the larger of the server hint and the jittered delay. Standard HTTP permits a delay expressed as nonnegative integer seconds or an HTTP date. Parsing a date requires a wall-clock reference, unlike monotonic elapsed-time budgeting. Malformed values are ignored by the helper, and a valid delay that leaves no useful time before the operation deadline ends the operation instead of scheduling a futile retry. A syntactically valid numeric delay too large to represent raises RetryAfterTooLong; the caller must stop or defer, never catch and ignore it to fall back to a shorter wait.

This handling does not assert that DeepSeek supplies that header in every error response. It simply makes the client prepared to use a documented HTTP hint when one exists. Do not clamp a long server-requested delay down to your short local backoff cap and then retry early. If the requested wait exceeds your permitted waiting period, stop or move the work through an explicitly authorized background workflow. A retry budget is a limit on what the application may do, not a reason to disregard the server’s recovery signal.

Bound active requests, queued operations, and delayed retries

A semaphore around the network call limits active requests, but it does not necessarily limit waiting work. An unbounded queue can accumulate thousands of operations behind a modest semaphore. Sleeping retry tasks can create a second hidden queue. Memory consumption, expired deadlines, and retry storms can therefore grow even while your active-request chart looks comfortably below its limit. Track and bound the whole admitted population, including queued operations and those waiting for a retry timer.

Use two distinct concepts: an operation-admission limit and an active-attempt limit. Admission controls how many logical operations the service is willing to retain. The active-attempt limit controls how many network requests may run at once. An admitted operation may retain its place in the overall population while releasing its network permit during backoff. This keeps retry sleep from unnecessarily occupying a scarce connection slot without allowing an unlimited number of sleeping operations to build up.

Acquire a network permit close to dispatch and release it on success, failure, and cancellation. Check the remaining deadline before and after waiting for a permit. If the remaining budget cannot support an attempt, expire the operation instead of letting it consume provider capacity. Keep the queue small enough that its oldest items still have a chance to complete. At admission, reject or defer excess work with a response that accurately tells the caller what happened; do not claim that work was accepted when it was silently dropped.

Original attempts and retries also need a fairness policy. If a retry storm always jumps ahead of newly admitted work, new users can suffer even as older requests consume all the capacity. If retries always go to the back of a constantly full queue, they may expire before receiving another chance. A small reserved share, weighted scheduling, or separate bounded queues can be reasonable designs. Pick one that reflects your product priorities and test it under sustained arrivals rather than assuming a semaphore alone provides fairness.

For a starting capacity calculation, relate arrival rate to average in-flight time, then validate the tails. Five admitted operations per second with an average two-second in-flight request suggest about ten active requests in steady conditions. That arithmetic is a workload illustration, not a recommended DeepSeek limit. Longer outputs, slower reasoning, retries, and bursty traffic change the occupancy. The thinking-mode evaluation guide is useful when comparing quality and latency, but a benchmark average should not replace queue-depth and tail-latency measurements under your actual traffic.

A connected stream is not a completed answer

Streaming creates additional states that a non-streaming “request returned” abstraction can obscure. Your application might receive only keepalive comments, some answer text, a tool-call fragment, or a terminal chunk. Track those states separately. A first byte can be a comment rather than useful output. A first answer token is progress rather than proof that the final result will be complete. If you show partial output to a user, preserve the distinction in both the interface and your metrics.

The current DeepSeek Chat Completions reference documents terminal finish_reason values including stop, length, content_filter, tool_calls, insufficient_system_resource, and aborted. Receiving an HTTP 200 response does not make all those states equivalent. A length-limited answer may be incomplete. A resource-interrupted or aborted completion needs an explicit outcome classification. A tool-call result requires the next application-controlled stage rather than being presented as an ordinary final answer.

Do not blindly replay every incomplete stream. If the user has already seen part of an answer, a new generation can differ, repeat earlier text, or contradict the visible partial result. Appending the replacement stream to the first one can produce a misleading composite that no model actually generated. Choose an explicit restart experience, a background reconciliation path, or a clear interrupted state. Repeating the original request is not a documented token-by-token resume protocol.

Usage collection also depends on provider-specific stream behavior. The current DeepSeek reference states that usage appears in the last content chunk before [DONE], rather than in a separate empty-choices usage chunk. A client written around a different provider’s convention can miss that data. An interrupted stream may never deliver the final usage information you expect. Treat missing usage as unknown, preserve enough non-sensitive attempt metadata for reconciliation, and avoid reporting an unobserved cost as zero.

Validate the complete result before any business action uses it. Transport success, protocol completion, schema validity, and application acceptance are separate checks. An automatic network retry controller should not decide that an invalid answer deserves unlimited regeneration. If you introduce a repair or regeneration step, give it its own bounded purpose within the original operation budget and report it separately from transport retries. This makes it possible to distinguish provider availability from prompt quality in production.

Separate repeatable generation from irreversible actions

Consider an assistant that generates a draft response and then sends an email. Repeating the generation after a transient failure may be acceptable if the application discards unsuccessful drafts and chooses only one completed result. Repeating the email send after an ambiguous timeout can deliver the message twice. Putting both steps inside one generic retry wrapper loses that distinction. The unit of retry should be chosen around the actual side effect, not simply around the largest convenient function.

Give the business operation an application-generated identity before dispatch. Store its state durably when the workflow needs to survive restarts, and use the destination service’s supported idempotency mechanism when it has one. Define exactly which action the identity represents. A request ID used for tracing, a model-generated tool-call ID, and a destination’s idempotency key are different identifiers with different contracts. Equal request bodies are not proof that two requests express the same business intention.

After an uncertain remote write, reconcile before deciding to retry. Query an authoritative operation record or the destination’s status endpoint if its contract supports that. If the destination cannot tell you whether the write committed and offers no appropriate idempotency mechanism, exactly-once effects may be impossible to guarantee from the client alone. Escalate the unknown outcome instead of turning uncertainty into a second unverified action. A durable outbox and carefully designed transaction boundaries can improve coordination, but do not erase every cross-system ambiguity.

Even a generation-only retry has consequences. The replacement may differ, and the earlier attempt may already have consumed provider resources. Setting replay_safe=True in the tutorial means that the application accepts re-executing the attempted work under its own policy. It does not mean free retries, identical output, guaranteed remote cancellation, or provider-backed duplicate suppression. Use that flag only after deciding where output becomes visible and where business effects occur.

Cancellation follows the same rule. A browser tab closing or a worker timing out tells you that the local consumer stopped waiting. It does not undo an action already sent to another service. Keep enough operation state to distinguish cancelled-before-dispatch, cancelled-while-waiting, completed-but-not-delivered, and unknown remote outcome. Those states help support teams explain what happened without guessing from an HTTP timeout alone.

Build a small retry controller you can test

The following Python example makes the policy explicit without tying it to a particular HTTP library. The sender receives a per-attempt time budget and either returns a result or raises an intentionally classified AttemptFailure. Time, sleep, and random sampling are injected. The retry bucket is shared by calls that should compete for the same local retry allowance. Unexpected exceptions propagate so a programming defect cannot masquerade as a transient provider problem.

This is a policy harness, not a complete production HTTP adapter. It does not implement authentication, SSE parsing, connection pooling, an admission queue, a distributed limiter, or durable business-operation storage. Its synchronous callback cannot forcibly interrupt blocking I/O. In a live integration, configure transport phase timeouts and an outer cancellation boundary, translate only known transport exceptions, disable overlapping retry layers, and validate terminal output before marking the operation accepted. Keep the application deadline unchanged across those steps.

retry_policy.py · Offline Python policy example

"""Transport-neutral retry policy; no network calls or credentials.

The sender must honor its supplied time budget. This synchronous harness cannot
preempt a blocking sender. Add transport timeouts and an outer cancellation scope
in a live integration. Retry safety is an application decision, not a provider
idempotency guarantee. Times and random values are injectable for offline tests.
"""
from dataclasses import dataclass
from datetime import timezone
from email.utils import parsedate_to_datetime
import math
from numbers import Real
import random
import re
import threading
import time
from typing import Callable, TypeVar

T = TypeVar("T")


def _finite_number(value) -> bool:
    """Reject bool, non-numeric input and numbers too large for float time."""
    try:
        return (isinstance(value, Real) and not isinstance(value, bool)
                and math.isfinite(value))
    except OverflowError:
        return False


@dataclass(frozen=True)
class Policy:
    max_attempts: int = 3  # Includes the first request.
    per_attempt_seconds: float = 8.0
    base_delay_seconds: float = 0.5
    max_delay_seconds: float = 4.0
    min_attempt_seconds: float = 0.1

    def __post_init__(self):
        if type(self.max_attempts) is not int or self.max_attempts < 1:
            raise ValueError("max_attempts must be a positive integer")
        values = (self.per_attempt_seconds, self.base_delay_seconds,
                  self.max_delay_seconds, self.min_attempt_seconds)
        if any(not _finite_number(x) or x <= 0 for x in values):
            raise ValueError("time settings must be finite and positive")
        if self.min_attempt_seconds > self.per_attempt_seconds:
            raise ValueError("minimum attempt budget exceeds per-attempt cap")


class AttemptFailure(Exception):
    def __init__(self, kind: str, status: int | None = None,
                 retry_after_seconds: float | None = None):
        super().__init__(f"{kind} failure; status={status}")
        self.kind = kind
        self.status = status
        self.retry_after_seconds = retry_after_seconds

    @property
    def transient(self) -> bool:
        return self.kind == "transport" or (
            self.kind == "http" and self.status in {429, 500, 503})


class DeadlineExceeded(Exception):
    pass


class AttemptsExhausted(Exception):
    pass


class RetryBudgetExhausted(Exception):
    pass


class ReplayNotAllowed(Exception):
    pass


class RetryAfterTooLong(ValueError):
    """A syntactically valid delay cannot fit in finite float time. Defer it."""


def parse_retry_after(value: str | None, wall_time: float) -> float | None:
    """Parse HTTP Retry-After; raise RetryAfterTooLong on a huge valid delay.

    A malformed header is ignored. An unrepresentably long numeric delay must
    stop/defer work, never fall back to a short retry. Not promised by DeepSeek.
    """
    if not isinstance(value, str) or not value or not _finite_number(wall_time):
        return None
    value = value.strip()
    if value.isascii() and value.isdigit():
        try:
            seconds = float(value)
        except (ValueError, OverflowError) as error:
            raise RetryAfterTooLong("retry delay cannot be represented; defer work") from error
        if not math.isfinite(seconds):
            raise RetryAfterTooLong("retry delay cannot be represented; defer work")
        return seconds
    try:
        parsed = parsedate_to_datetime(value)
        if parsed.tzinfo is None:
            # The obsolete asctime HTTP-date form has no timezone field;
            # HTTP defines it as GMT. Reject other timezone-less strings.
            if not re.fullmatch(
                r"[A-Z][a-z]{2} [A-Z][a-z]{2} [ 0-9][0-9] "
                r"[0-9]{2}:[0-9]{2}:[0-9]{2} [0-9]{4}", value
            ):
                return None
            parsed = parsed.replace(tzinfo=timezone.utc)
        seconds = max(0.0, parsed.timestamp() - wall_time)
        return seconds if math.isfinite(seconds) else None
    except (ValueError, TypeError, OverflowError):
        return None


class RetryBucket:
    """Process-local token bucket. Share one instance across calls/threads.

    It is NOT a distributed/account-wide budget. Initial attempts use no tokens.
    """
    def __init__(self, capacity: float, refill_per_second: float,
                 now: Callable[[], float] = time.monotonic):
        if not _finite_number(capacity) or capacity < 1:
            raise ValueError("capacity must be finite and at least one")
        if not _finite_number(refill_per_second) or refill_per_second < 0:
            raise ValueError("refill rate must be finite and nonnegative")
        self.capacity = capacity
        self.rate = refill_per_second
        self.tokens = capacity
        self.now = now
        self.updated = now()
        self.lock = threading.Lock()

    def take(self) -> bool:
        with self.lock:
            current = self.now()
            elapsed = max(0.0, current - self.updated)
            self.updated = max(current, self.updated)
            self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
            if self.tokens < 1:
                return False
            self.tokens -= 1
            return True


def request_with_retry(
    send: Callable[[float], T], *, operation_seconds: float,
    replay_safe: bool, retry_bucket: RetryBucket,
    policy: Policy = Policy(),
    now: Callable[[], float] = time.monotonic,
    sleep: Callable[[float], None] = time.sleep,
    uniform: Callable[[float, float], float] = random.uniform,
) -> T:
    """Attempt only explicitly replay-safe operations after transient failures.

    Start this at admission if queue wait should count against the budget, or pass
    only the remaining budget after admission. All durations are example policy.
    """
    if not _finite_number(operation_seconds) or operation_seconds <= 0:
        raise ValueError("operation budget must be finite and positive")
    if type(replay_safe) is not bool:
        raise ValueError("replay_safe must be an explicit bool")
    started = now()
    deadline = started + operation_seconds
    if not _finite_number(deadline) or deadline <= started:
        raise ValueError("operation deadline must be finite and representable")
    previous = None
    ceiling = min(policy.base_delay_seconds, policy.max_delay_seconds)
    for attempt in range(1, policy.max_attempts + 1):
        remaining = deadline - now()
        if remaining < policy.min_attempt_seconds:
            raise DeadlineExceeded("insufficient time for another attempt") from previous
        if attempt > 1 and not retry_bucket.take():
            raise RetryBudgetExhausted("shared retry tokens exhausted") from previous
        # Acquiring a shared bucket lock can take time. Never hand the sender a
        # stale budget or start another attempt after that time was exhausted.
        remaining = deadline - now()
        if remaining < policy.min_attempt_seconds:
            raise DeadlineExceeded("insufficient time after retry admission") from previous
        try:
            result = send(min(policy.per_attempt_seconds, remaining))
        except AttemptFailure as error:
            previous = error
            if now() >= deadline:
                raise DeadlineExceeded("attempt crossed operation deadline") from error
            if not error.transient:
                raise
            if not replay_safe:
                raise ReplayNotAllowed("reconcile outcome before replay") from error
            if attempt == policy.max_attempts:
                raise AttemptsExhausted("attempt limit reached") from error
            delay = uniform(0.0, ceiling)  # Full jitter.
            if not _finite_number(delay) or not 0 <= delay <= ceiling:
                raise ValueError("random source returned an invalid delay")
            hint = error.retry_after_seconds
            if hint is not None:
                if not _finite_number(hint) or hint < 0:
                    raise ValueError("retry delay hint must be finite and nonnegative")
                delay = max(delay, hint)
            if delay + policy.min_attempt_seconds >= deadline - now():
                raise DeadlineExceeded("retry delay leaves no useful attempt") from error
            sleep(delay)
            ceiling = min(policy.max_delay_seconds, ceiling * 2.0)
        else:
            if now() >= deadline:
                raise DeadlineExceeded("response arrived after operation deadline")
            return result
    raise AssertionError("unreachable")

The code’s small defaults make the mechanics visible. They are not latency promises or tuned DeepSeek settings. The bucket contains only retry tokens, so it cannot replace an admission limit. The controller starts its deadline when called; if work already spent time in a queue, pass only the remaining operation allowance. In a production codebase, you may prefer passing an explicit deadline object through every layer so there is less opportunity to accidentally reset it.

The deterministic demonstration below simulates a 503, then a 429 with a delay hint, then an accepted local fixture. Each attempt advances a fake clock by two tenths of a second. Choosing the midpoint of each jitter window makes the result repeatable. The waits are a quarter second and one and a half seconds, giving three attempts and a simulated total of 2.35 seconds. These are arithmetic fixtures, not measurements of DeepSeek performance or evidence that a real outage would recover this quickly.

offline_demo.py · Offline Python policy example

"""Run: python offline_demo.py. Simulated clock and outcomes, no network."""
import json
from retry_policy import AttemptFailure, Policy, RetryBucket, request_with_retry

class FakeClock:
    def __init__(self):
        self.seconds = 0.0
        self.sleeps = []
    def now(self):
        return self.seconds
    def sleep(self, seconds):
        self.sleeps.append(seconds)
        self.seconds += seconds


def run_demo():
    clock = FakeClock()
    trace = []
    outcomes = iter([AttemptFailure("http", 503),
                     AttemptFailure("http", 429, retry_after_seconds=1.5),
                     {"answer": "validated local fixture"}])
    def send(attempt_budget):
        trace.append({"at": clock.now(), "budget": attempt_budget})
        clock.seconds += 0.2  # Simulated work, not measured API latency.
        value = next(outcomes)
        if isinstance(value, Exception):
            raise value
        return value
    result = request_with_retry(
        send, operation_seconds=5.0, replay_safe=True,
        retry_bucket=RetryBucket(2, 0, clock.now),
        policy=Policy(max_attempts=3, per_attempt_seconds=2),
        now=clock.now, sleep=clock.sleep,
        uniform=lambda low, high: high / 2,
    )
    return {"result": result, "attempts": trace,
            "waits": clock.sleeps, "elapsed": clock.now()}

if __name__ == "__main__":
    print(json.dumps(run_demo(), indent=2))

Test the failure paths without waiting or spending credits

Reliability tests should make rare failures ordinary. Injecting a clock lets a test jump across a deadline immediately. Injecting sleep records the intended delay without pausing the test runner. Injecting randomness lets you force both ends of a jitter interval, reproduce a suspicious sequence, and test invalid random output. A fake sender can produce a permanent error, a run of transient failures, an unexpectedly late result, or a success on the final permitted attempt.

Test the invariants rather than just one expected trace. No retry may begin when the useful-time budget is exhausted. The initial attempt must not consume a retry token. Every actual retry must require one. The maximum attempt count must include the first request. A permanent error must not be transformed into a delayed retry. A failure with replay disabled must not be resubmitted. A response returned after the deadline must not become an on-time success metric.

Header parsing deserves separate boundary tests for integer delays, HTTP dates, missing values, negative or fractional strings, invalid dates, and oversized numbers. Test the token bucket at capacity, just below a token, after replenishment, and when several callers contend for the final available token. Also test cancellation and connection cleanup in the real adapter once it exists. Passing the offline policy tests proves local decisions under simulated inputs; it does not validate an unimplemented transport layer or establish real provider availability.

The compact test file below demonstrates four important failure boundaries using the same injected clock. Run it with python -m unittest test_retry_examples.py after saving it beside the controller. Keep a broader regression suite beside the source as the policy evolves. When a production incident reveals a new failure mode, first turn its relevant behavior into a minimal fixture without copying credentials, prompts, or sensitive response bodies. Then change the policy and rerun the existing suite. This keeps the explanation, implementation, and incident learning aligned while allowing normal tests to remain deterministic and inexpensive.

test_retry_examples.py · Offline Python policy example

"""Run: python -m unittest -v test_retry_examples (from this directory)."""
import unittest
from retry_policy import (AttemptFailure, DeadlineExceeded, ReplayNotAllowed,
                          RetryBucket, RetryBudgetExhausted, request_with_retry)


class RetryExamples(unittest.TestCase):
    def setUp(self):
        self.seconds = 0.0
        self.calls = 0
        self.waits = []
        self.bucket = RetryBucket(2, 0, lambda: self.seconds)

    def sleep(self, delay):
        self.waits.append(delay)
        self.seconds += delay

    def failing_sender(self, error):
        def send(budget):
            self.calls += 1
            raise error
        return send

    def run_request(self, error, *, safe=True, budget=5):
        return request_with_retry(
            self.failing_sender(error), operation_seconds=budget, replay_safe=safe,
            retry_bucket=self.bucket, now=lambda: self.seconds,
            sleep=self.sleep, uniform=lambda low, high: high,
        )

    def test_permanent_failure_is_not_retried(self):
        error = AttemptFailure('http', 401)
        with self.assertRaises(AttemptFailure) as caught:
            self.run_request(error)
        self.assertIs(caught.exception, error)
        self.assertEqual(self.calls, 1)
        self.assertEqual(self.waits, [])

    def test_unsafe_replay_is_blocked(self):
        with self.assertRaises(ReplayNotAllowed):
            self.run_request(AttemptFailure('transport'), safe=False)
        self.assertEqual(self.calls, 1)
        self.assertEqual(self.waits, [])

    def test_retry_delay_cannot_fit_deadline(self):
        with self.assertRaises(DeadlineExceeded):
            self.run_request(AttemptFailure('http', 429, 10), budget=2)
        self.assertEqual(self.calls, 1)
        self.assertEqual(self.waits, [])

    def test_empty_shared_budget_blocks_retry(self):
        self.bucket.take()
        self.bucket.take()
        with self.assertRaises(RetryBudgetExhausted):
            self.run_request(AttemptFailure('http', 503))
        self.assertEqual(self.calls, 1)  # Initial attempt uses no retry token.
        self.assertEqual(self.waits, [0.5])  # Budget is checked after backoff.


if __name__ == '__main__':
    unittest.main()

Measure logical outcomes as well as attempts

Count accepted logical operations, dispatched attempts, retries, and terminal outcomes separately. A dashboard showing only a high request-success rate can hide users whose original operations failed after several attempts, or requests that succeeded only after their deadlines. Track the final outcome of each logical operation and link its attempts with a non-sensitive correlation identifier. Keep provider request identifiers when available, without inventing a guarantee that any particular header will always exist.

Useful timing measurements include queue delay, permit wait, connection establishment, first meaningful output, total stream duration, retry sleep, and end-to-end elapsed time. Report distributions, including tail latency, rather than relying entirely on averages. Separate time-to-first-byte from time-to-first-content so keepalive behavior does not make an overloaded service look responsive. Record which limit ended the work: attempt count, operation deadline, retry-token exhaustion, admission rejection, replay restriction, or a permanent error.

For cost analysis, distinguish provider-reported usage from estimates and missing data. Multiple attempts can represent multiple generations, including an attempt whose response you never received. Do not divide reported tokens by successful operations and assume you have measured every failed attempt’s cost. The context-caching measurement guide covers reading actual usage and avoiding optimistic savings claims. Cached inputs do not make retry storms harmless, and cache hits do not replace reliability controls.

Logs should be useful without becoming an accidental prompt archive. Prefer timestamps, status categories, model identifier, attempt number, remaining budget, wait reason, and safe correlation IDs. Keep credentials out of logs and avoid attaching full request or response bodies by default. If debugging requires content samples, use an approved retention and access policy. Aggregated signals usually answer the first incident questions: whether failures are widespread, which paths are affected, and whether your retries are amplifying pressure.

Roll out the policy against explicit failure scenarios

Begin with a representative workload and a baseline with retries disabled. Measure admission, completion, queueing, and latency before introducing recovery behavior. Then inject bounded failures in a test environment: isolated 503 responses, sustained overload, connection loss before output, interruptions after partial output, and requests that outlive their caller’s deadline. Use a small canary when changing production behavior. A more elaborate retry policy is only an improvement if it raises useful completion without violating latency, cost, or safety constraints.

During load testing, separate arrival rate from concurrency. If your test waits for every response before starting another, a slowdown can reduce offered load automatically and hide queue growth. Test the arrival patterns the service actually sees, including bursts and multiple independent clients. Watch queued and delayed operations as closely as active connections. Stop at predetermined safety limits rather than using a provider outage or a real customer workload as an uncontrolled experiment.

A circuit breaker can complement these controls by temporarily stopping attempts when a dependency is consistently failing, but it needs a scoped design. A failure limited to one model, route, or credential should not unnecessarily block every unrelated operation. Define the observation window, minimum sample size, opening condition, recovery probes, and how concurrent callers coordinate. A breaker is not a substitute for a deadline or a queue bound, and an overly eager one can prolong an incident after the dependency has recovered.

Write a short operational runbook before the first incident. For 401 and 402, route to the appropriate account or configuration owner instead of escalating retry volume. For sustained 429 or 503 responses, inspect shared-account traffic, reduce admitted concurrency, and check whether retries are consuming a disproportionate share of attempts. For long active connections with little output, distinguish provider queueing from an application parser or proxy problem. Preserve the original symptoms before changing several limits at once.

The final acceptance criteria should be concrete. The application must stop on its own deadline, keep waiting work bounded, avoid hidden retry multiplication, surface interrupted output honestly, and prevent uncertain remote writes from being blindly replayed. The exact timeout values can then evolve with evidence. This produces a client that remains understandable during failures: every extra attempt has a reason, a remaining budget, and a clearly defined boundary around what it is allowed to repeat.

Sources & References

Primary sources checked October 2, 2026. Provider behavior is attributed to DeepSeek; retry algorithms, HTTP parsing rules, and SDK defaults are identified separately. Example capacities and timings are application recommendations, not provider service-level guarantees.

1. DeepSeek API documentation: Rate Limit — concurrent-request limits, waiting keepalives, and connection closure before inference starts.
2. DeepSeek API documentation: Error Codes — documented HTTP error meanings and provider troubleshooting guidance; the Chinese Error Codes page supplies the TPM/RPM wording discussed above.
3. DeepSeek API reference: Create Chat Completion — streaming behavior, terminal reasons, and usage placement.
4. DeepSeek API documentation: Your First API Call — current compatibility and model naming context.
5. OpenAI Python SDK: Retries and timeouts — SDK-level defaults and configuration, separate from provider policy.
6. HTTP Semantics, RFC 9110: Retry-After — standard delay-seconds and HTTP-date forms.
7. Amazon Builders’ Library: Timeouts, retries, and backoff with jitter — general distributed-systems guidance on timeout selection, retry amplification, and side effects.
8. AWS Architecture Blog: Exponential Backoff and Jitter — comparison of backoff strategies and full jitter.