The shipped answer to "why not scrape it yourself" is usually about markup: a source site redesigns its page, your selectors stop matching, and your scraper breaks. That's true, but it skips the part that actually costs you money — not that it broke, but that you didn't find out. A campaign built on a quietly-empty scrape looks identical to a campaign built on real data, right up until someone asks why the numbers don't add up.
The more useful question isn't whether a source can change. It will. The question is what you actually see on your end when it does.
Does a broken run fail loudly, or fail silently?
Loudly, by contract. A run that can't produce valid data comes back with
an explicit failed status and a real error string — not a completed
status wrapped around zero or garbage rows. That distinction is the whole
point: a scraper that silently returns nothing looks, from your code's
point of view, exactly like a day with no results. A run that explicitly
fails does not.
{
"status": "failed",
"error": "..."
}
The guidance for handling it is equally direct: read the error field and
report it, don't blindly retry. That's a deliberate choice — an
automatic retry on a run that failed because the underlying request was
malformed just fails again, and masks the actual problem behind a delay.
What does a 'running' response with zero rows actually mean?
Not a failure. A job can sit at running with row_count: 0 for a while
before rows start landing — that's normal mid-run state, not a signal
anything broke. The state worth reacting to is the terminal one:
completed, failed, cancelled, or rejected_balance. All four are
terminal — stop polling on any of them, and treat completed with an
empty data array as a real answer (nothing matched), not an error to
retry.
rejected_balance is kept separate from failed on purpose: it means the
wallet couldn't cover the run, not that anything went wrong with the
request itself. A client that only checks for completed/failed and
doesn't recognize rejected_balance will poll it forever, waiting for a
state change that isn't coming.
What happens when only part of a run breaks?
Some endpoints carry the error at the row level instead of the job level.
Instagram Reels and Google Play reviews, for example, both return an
error and errorDescription field on each individual row alongside the
normal data fields — so a run can come back completed with 40 good rows
and 3 rows that explicitly say what went wrong on that specific item,
rather than either failing the whole batch over three bad rows or quietly
dropping them with no trace.
That's a real, deliberate tradeoff: partial success is common enough (one video got taken down mid-run, one review was deleted between search and fetch) that failing an entire 43-row job over 3 unavailable items would be worse than telling you exactly which 3 and why.
What does this not promise?
It's worth being precise about what's actually being described here.
This is about the shape of the response you get back — a real terminal
status and a real error message instead of silence — not a claim about
how quickly any particular broken source gets fixed upstream. That's
genuinely not something to promise a specific number on here; treat a
failed run as your signal to check back, not as something with a fixed
turnaround attached.
How should your own code treat a failed run?
The same way the guidance above treats it: as information, not noise.
Log the error string somewhere you'll actually see it, don't retry a
failed run automatically (a 400 on a search agent usually means a
required filter like a location is missing — fix the request, don't
resubmit it unchanged), and treat rejected_balance as a billing action
item rather than a data problem. A pipeline that swallows failed the
same way it swallows an empty-but-valid result is choosing to recreate
the exact silent-failure problem an explicit status code exists to solve.
The reference polling helper makes this concrete: it checks status at
every step, and the moment a job comes back failed, cancelled, or
rejected_balance it raises immediately with the real error message
attached, instead of returning an empty list that looks like "nothing
matched today":
if r.get("status") in ("failed", "cancelled"):
raise RuntimeError(r.get("error") or r["status"])
That one line is the actual difference this is about. A scraper that breaks against a redesigned page has no equivalent line to write — there's no status to check, because there was never a contract promising one. Whatever your own pipeline does with a raised error (alert someone, skip the source and continue, retry after fixing the input), it can only do it because the run told you, clearly, that it needed attention.
FAQ
No. Failed runs don't count toward billable rows — you're charged for rows actually returned, not for attempts.
A timeout on your own scraper usually looks the same as any other empty result unless you built explicit timeout handling yourself. A failed run here always comes back with a status and an error string, so the distinction between 'nothing matched' and 'something broke' isn't something you have to build detection for.
Read the error first. A malformed request or a missing required filter fails the same way every time you resubmit it unchanged — retrying without fixing the input just spends another attempt on the same failure.
Just some — it depends on whether a single run can plausibly contain a mix of valid and invalid items, like a batch of video or review URLs where one entry might be deleted. Check the specific endpoint's field list rather than assuming every response includes it.