Rate limits and retries
See when a failed attempt runs again.
A failed attempt is either retried or finished. The engine decides from the error the executor reported. A check failure is finished: the same code over the same inputs fails the same check, so it is not tried again.
Ordinary failures
A retryable failure is a rate limit, an upstream outage (500, 502, 503, 504), or a dropped connection. A 400, a missing page, and a TerminalError thrown from run are terminal. The attempt status becomes failed, and error_class is retryable or terminal.
The ordinary budget counts retryable failures that are not rate limits. A source is tried 3 times. A raw asset, a model, and a destination are tried once. When the budget is spent, the error message gains a suffix that names that number:
(gave up after 3 attempts) A model, a destination, or a raw asset uses (gave up after 1 attempts). That is the string the engine writes.
Upstream blips
A connector’s HTTP client waits out a status from 500 to 599, or a request that hits its timeout, on GET, HEAD, and OPTIONS. The waits are 250ms and then 1 second, and they count toward the same 30 second cap as a rate limit. A third failure is an ordinary retryable error and spends the attempt budget above. PUT, DELETE, POST, PATCH, a 4xx, a missing page, and a dropped connection leave the client on the first response.
Rate limits
A connector’s HTTP client waits out a 429 inside the attempt, for up to 30 seconds. Retry-After sets the wait. With no header, the client waits 1 second, then 4 seconds, then 10 seconds. A Retry-After that would push the wait past 30 seconds is not slept in the client.
After the client stops waiting, the engine retries a rate limit on its own budget of 8. Those retries do not spend the ordinary attempt budget. Each wait is the server’s Retry-After, capped at 15 minutes. With no delay, the next attempt starts immediately. Past 8 retries the message is:
(gave up after 8 rate-limit retries) Read the result
puppi logs <run-id> --env prod Each attempt prints its status, error_class when one was set, the error message, and the ctx.log lines. A retry is a new attempt on the same asset, with a higher number. See Read a run.