Before you start
- Python 3.8 or newer with requests installed (python -m pip install requests)
- The address of a paginated API, or the small test server below
Permissions and scopes you need
- Whatever credentials the API needs, passed in a header from an environment variable, never written in the code
Real APIs do not hand you everything in one response, and they do not always answer. Two habits make a client dependable: follow the pagination to the end, and retry failures sensibly. This guide builds both and tests them against a deliberately unreliable server.
1. A server to test against
Reading the retry code is not testing it. This small server serves ten items in pages of four, and is built to fail: the page starting at item 4 returns 503 Service Unavailable the first time, and the page starting at item 8 returns 429 Too Many Requests with a Retry-After: 1 header the first time.
import json
from http.server import BaseHTTPRequestHandler, HTTPServer
from urllib.parse import parse_qs, urlparse
ITEMS = [{"id": i, "name": f"item {i}"} for i in range(1, 11)]
PAGE = 4
failures = {"cursor=4": ["503"], "cursor=8": ["429"]} # fail once, then succeed
class Handler(BaseHTTPRequestHandler):
def do_GET(self):
query = parse_qs(urlparse(self.path).query)
cursor = int(query.get("cursor", ["0"])[0])
key = f"cursor={cursor}"
if failures.get(key):
kind = failures[key].pop()
self.send_response(int(kind))
if kind == "429":
self.send_header("Retry-After", "1")
self.end_headers()
return
chunk = ITEMS[cursor:cursor + PAGE]
body = {"results": chunk, "next_cursor": cursor + PAGE if cursor + PAGE < len(ITEMS) else None}
data = json.dumps(body).encode()
self.send_response(200)
self.send_header("Content-Type", "application/json")
self.send_header("Content-Length", str(len(data)))
self.end_headers()
self.wfile.write(data)
def log_message(self, *args):
pass
HTTPServer(("127.0.0.1", 8098), Handler).serve_forever()
Run it in one terminal. The response carries the items and a next_cursor that is null on the last page, which is how this API signals "that is everything".
2. The client
import random
import time
import requests
RETRY_STATUSES = {429, 500, 502, 503, 504}
def get_with_retries(session, url, params=None, attempts=5, base_delay=0.5):
for attempt in range(1, attempts + 1):
try:
response = session.get(url, params=params, timeout=(3.05, 20)) # connect and read timeouts
except (requests.ConnectionError, requests.Timeout):
response = None
if response is not None and response.status_code not in RETRY_STATUSES:
response.raise_for_status() # a 404 or 401 will not get better: fail now
return response
if attempt == attempts:
raise RuntimeError(f"giving up on {url} after {attempts} attempts")
retry_after = response.headers.get("Retry-After") if response is not None else None
delay = float(retry_after) if retry_after and retry_after.isdigit() else base_delay * 2 ** (attempt - 1)
delay += random.uniform(0, delay / 4) # jitter, so clients do not retry in step
status = response.status_code if response is not None else "no response"
print(f" attempt {attempt} failed ({status}); waiting {delay:.1f}s")
time.sleep(delay)
def fetch_all(url):
session = requests.Session()
items, cursor = [], None
while True:
params = {"cursor": cursor} if cursor is not None else None
data = get_with_retries(session, url, params=params).json()
items.extend(data["results"])
cursor = data["next_cursor"]
if cursor is None:
return items
random.seed(3)
everything = fetch_all("http://127.0.0.1:8098/items")
print(len(everything), "items;", "ids", everything[0]["id"], "to", everything[-1]["id"], "unique:", len({i["id"] for i in everything}) == len(everything))
Running it prints:
attempt 1 failed (503); waiting 0.5s
attempt 1 failed (429); waiting 1.1s
10 items; ids 1 to 10 unique: True
3. What each decision does
- Timeouts.
requestswaits forever by default.timeout=(3.05, 20)gives connect and read limits, so a stalled server cannot freeze your job. - Retry only what can recover.
429and the 5xx family are temporary, as are network errors. A401(wrong credentials) or404(no such thing) will fail the same way every time, soraise_for_status()stops at once. - Honour
Retry-After. When the server says how long to wait, believe it. Here it asked for one second and the client waited 1.1 (the extra tenth is jitter). - Exponential backoff. With no hint, wait 0.5 s, then 1 s, then 2 s and so on. The doubling gives a struggling server room to recover rather than hammering it.
- Jitter. A random addition spreads out clients that failed together.
- A limit. After five attempts the function raises, so a permanent outage becomes an error you can see instead of a loop that never ends.
- A
Session. It reuses the connection between pages, which is faster and kinder to the server.
4. Pagination traps
- Cursor beats offset. With
?page=2or?offset=100, rows inserted or deleted while you read shift the pages, so you skip or repeat items. A cursor marks a position that does not shift. Use it when the API offers one. - Do not assume the last page is short. Rely on the signal the API gives (here a null cursor; elsewhere a
nextlink or an empty page), not on counting. - De-duplicate on a stable id if the data can change while you read, and compare your count with any total the API reports.
- Respect rate limits proactively. If the API publishes a limit, pause between pages rather than waiting to be told off.
5. Before you rely on it
Send the credentials in a header from an environment variable, never in the code or the URL (URLs get logged). Make the whole job idempotent, so a failure on page 40 can be re-run safely. And log how many pages and retries happened: a client that quietly retries a lot is telling you something about the API.
Expected result
The script fetches all 10 items across three pages, survives one 503 and one 429 on the way, prints each failed attempt, and ends with a count of 10 unique items.
Troubleshooting
| Problem | Likely cause | Fix |
|---|---|---|
| The script hangs forever | No timeout was set, so a stalled connection waits indefinitely. | Always pass timeout=(connect, read) to requests. |
| Items appear twice, or are missing, across pages | The data changed while you paged, or the API uses offset pagination on a changing list. | Prefer cursor (keyset) pagination, de-duplicate by id, and compare totals when the API provides one. |
| You keep getting 429 Too Many Requests | You are sending requests faster than the API allows. | Honour Retry-After, slow down between pages, and ask for larger pages. |
| A 401 or 404 is retried five times | The retry logic treats every failure alike. | Retry only transient errors (429, 5xx and network failures). A 4xx like 401 or 404 will not get better. |
Frequently asked questions
Why add jitter to the delay?
If many clients fail together and all retry after exactly 2 seconds, they hit the server together again. A little randomness spreads them out.
Is it safe to retry a POST?
Only if the operation is idempotent or the API supports an idempotency key. Retrying a GET is always safe; a blind POST retry can create duplicates.
References
From Grokem: https://www.grokem.com/programming/apis-integrations/rest/paginated-api-python-retries
