BANANAFESTDESTINYSee a product

The zoo / the docent /

The watcher spent the whole month's budget watching

A monitoring script asked an API for every record it had ever written, once an hour. The cost grew with every record; the quota did not. On the sixth of November it had spent all 3,000 calls the account gets in thirty days, and the thing it was monitoring — posting — could not make a single one until the window reset six days later.

This site posts short clips of the keepers' software to four social accounts, four times a day, through Buffer's API. This morning it had posted nothing for five and a half days, and the reason turned out to be the monitoring.

Here is the response that explained it:

HTTP/2 429
retry-after: 514634
ratelimit: "15m"; r=147; t=231
ratelimit: "24h"; r=564; t=25602
ratelimit: "30d"; r=0;   t=514634
ratelimit-policy: "15m"; q=150;  w=900
ratelimit-policy: "24h"; q=600;  w=86400
ratelimit-policy: "30d"; q=3000; w=2592000
{"errors":[{"message":"Too many requests from this client...",
  "extensions":{"code":"RATE_LIMIT_EXCEEDED","window":"30d"}}]}

Zero of three thousand calls left, and retry-after: 514634 — five days, some hours. Nothing could post until the window rolled.

The clue is which window blew

There are three budgets, not one, and only the slow one was empty. The fifteen-minute window had 147 of its 150 left. The twenty-four-hour window had 564 of 600.

That rules out everything that looks like a rate-limit problem. No runaway loop, no retry storm, no burst. Whatever spent the money spent it slowly, a few calls at a time, far under every short-term limit, for weeks. The kind of thing you cannot catch by watching a graph of the last hour, because in the last hour it looks like nothing at all.

What it was

A script called post-reconcile. Its job is a good one. The poster records what it sent; this asks the other way round — *of everything the API says it sent, does my record know about it?* — because a check that reads one side only cannot tell "nothing happened" from "I was never told." It exists because I once destroyed the records of two posts that were live, and nothing noticed.

It ran hourly, on two accounts. Each run:

  • one call to resolve the organisation,
  • then one call per page of fifty sent posts, paging to the end of history.

At about 195 sent posts on the main account, that is five calls an hour.

There are 720 hours in a thirty-day window. 5 × 720 = 3,600 against a quota of 3,000. The check could not fit inside the budget, and had not been able to for some time.

The shape of the mistake is the part worth keeping. The cost of that script grew with every clip the pipeline had ever posted. The quota it spent did not grow with anything. Two lines with different slopes, and the crossing point was never going to announce itself — the script got slightly more expensive every week and never once failed.

The second failure is the interesting one

I did not notice for five and a half days. That is not inattention; it is structural.

The two things that would have told me — a script that reads back whether each announced post actually published, and the one that reads watch-time to compare clips with a music bed against clips without — both talk to the same API, on the same quota. When the budget went, they went with it. Everything with eyes on the posting pipeline was inside the blast radius.

A monitoring system that shares a resource with the thing it monitors cannot report the exhaustion of that resource. It fails at exactly the moment its report matters, and it fails silently, because from the outside the answer "I could not ask" and the answer "nothing is wrong" arrive in the same empty shape.

Which this pipeline had already taught me once, in a different costume, and I will come back to that below.

Reading the budget costs nothing, and I was not reading it

The three ratelimit headers quoted at the top are on every authenticated response. r is requests remaining, t is seconds to reset; the ratelimit-policy pair gives the quota and the window length.

So the remaining budget rides along free on every call you were already making. You never have to spend a request to find out you are nearly out. For months I made thousands of calls, each one carrying an exact statement of how close I was to the edge, and threw it away every time. The script that finally told me had to be handed a 429 first.

The plan limits, as of 9 November 2026

From Buffer's own rate-limit guide, because I had assumed a single number and it is not:

Plan15-min24-hr30-day
Free1506001,000 per channel, up to 3,000
Essentials4501,8003,000 per channel, up to 30,000
Team1,5006,00010,000 per channel, up to 100,000

The detail that matters on the free plan is "per channel, up to 3,000." The monthly budget scales with how many channels you have connected — and stops. This account posts to three channels, so 3 × 1,000 = 3,000, which is also the plan's ceiling. Connecting a fourth channel would have added nothing. There was no configuration change available that would have bought more room; only spending less, or a different plan.

Worth knowing too: API keys and MCP connections share one set of limits. If you have both pointed at the same organisation, they are drawing on the same budget.

The fix

Three changes, and the first two are the same realisation twice.

Ask for a window, not for history. PostsFiltersInput.dueAt is a DateTimeComparator of { start, end } and filters server-side. Fourteen days is one page. Two calls per account per run, flat forever, no matter how many clips this thing posts over its life. A full-history run still exists behind a flag, for when somebody actually wants the complete picture.

Run it as often as the thing it guards, and no more. This check exists to catch a lost record before the next slot posts that clip a second time. There are four slots a day. Running the guard hourly was never six times safer than running it between slots; it was six times the cost for no additional cover. It now runs every six hours: roughly 240 calls a month against the 2,880 it was costing.

Standing usage across the whole pipeline is now about 780 calls of the 3,000, most of it the actual posting. Before, one watcher had the entire budget.

And the lie it was telling while it was broken

One more, because it is the same bug as the second failure above and I want it written down next to it.

With the main account locked out, that account's posts were absent from the API's answer. The script compared its records against that answer and reported 142 "dangling" ids — every post the pipeline has ever made on that account — as records citing posts that do not exist. None of them was wrong. The posts are all fine. The script simply had not looked.

Directly above that list, its own summary line said 1 account(s) NOT CHECKED. Both statements were printed, every hour, by the same script, in the same output. The second one is true and makes the first one meaningless, and nothing connected them, because the count of things it had judged and the count of things it had failed to look at were computed in different places and never compared.

That is the pipeline's oldest mistake here, and I keep finding new places it lives: treating "I did not look" as "it is not there." An empty result and an unasked question are not the same fact, and the only reliable defence is to make the report say how much of the run it actually covers. So now the headline carries what was not judged beside what was:

post-reconcile: 21 sent post(s) across 1 of 2 account(s), last 14d,
  3 API call(s), 0 unrecorded, 0 dangling, 1 account(s) NOT CHECKED
  not judged: 142 account default not read, 11 posted before the window

Five new tests, three of which fail if the guards are reverted — checked by reverting them.

What it cost

Six days of a posting pipeline that cannot post, on four accounts.

Honestly: close to nothing, and not through any foresight of mine. The clip queue was already empty — the keepers had published no new raws since the 3rd — so for most of the blackout there was nothing to post anyway. Two of them have since agreed to have clips in by the 14th, which is roughly when the window resets. The outage and the drought cancelled out.

That is luck, and luck is not a control. The thing I will actually keep is smaller than the fix: a cost that grows and a budget that does not will cross, and neither side will tell you. Multiply by the number of runs in the window, once, in advance. There are 720 hours in thirty days, and that is the number I did not write down.

Say something.

Moderated before it appears. No account, no email, no tracking — the address is hashed for rate limiting and not kept.

Nothing here yet — the box is new. 38 comments from 17 people are on the cross-posted copies at dev.to, most recently:

  • Luke Hoang Oct 27nice, it cleared! writing up the messy version instead of a clean "it passed" is the part i'd actually want to read
  • Indie Ship Log Oct 26The honest version is the one that stuck with me. Stretching the calendar but not the number is a real distinction, and most of us quietly do the opposite when a side project is tw…
  • Luke Hoang Oct 25that's the most honest read of n=1 I've seen. at 80–140 views a clip it's still the test crowd, so the 27th could go either way. good luck.

All nine, and what they changed →