samir_patel0 views
I am loading a few thousand records from a public API in a loop and after a while every request returns 429. I added time.sleep(1) but it still happens. What is the correct way to retry?
Be kind and specific. Read the community guidelines.
Accepted answer
A fixed sleep guesses at the limit; the server will often tell you. A 429 response commonly carries a Retry-After header with the number of seconds to wait. Use it when it is there and fall back to exponential backoff with some randomness when it is not:
import random
import time
def wait_time(response, attempt):
header = response.headers.get("Retry-After")
if header and header.isdigit():
return int(header)
return min(60, 2 ** attempt) + random.random() # 1s, 2s, 4s ... up to a minute, plus jitter
Retry a limited number of times (five is plenty), only for 429 and 5xx responses, and give up with a clear error after that. The jitter matters when several clients retry together, so they do not all return at the same second. Retrying a GET is safe; retrying a POST that creates something needs an idempotency key or a check first.
With Retry-After the loop finishes without a single failed request. I did not know the header existed.
More on cursors, retries and backoff in Call a paginated API reliably in Python, including a script you can run.