API down
Retry, alert, backfill from the watermark
- Retry with growing waits: a blip should not page anyone
- Alert on the first run that still fails, and say “delayed” on the dashboard
- Load from the last good hour, not from “now”, so a gap fills itself
python · simplified pattern
def extract(since):
for wait in (1, 2, 4):
try:
return get(API, start=since, end=now())
except HTTPError:
sleep(wait)
alert("extract failing")
raise
since = min(watermark + HOUR, now() - 5 * HOUR)
batch = extract(since)