Error budgets for an ecommerce integration

The idea comes from site reliability engineering and it transfers cleanly. You pick an indicator, set a target for it, and the gap between the target and perfection is your budget for failure. If the target is that 99.5% of orders reach the ERP within fifteen minutes, then half a per cent of orders may not, and that half a per cent is a resource the team is allowed to spend.

The reason this is more useful than "monitor the integration" is that it forces two conversations to happen before an incident rather than during one. The first is about what failure costs: an order that reaches the warehouse two hours late costs something different from a price update that arrives late, and pretending both need the same reliability is how teams end up paging someone at 3am for a marketing sync.

The second conversation is about consequence. An error budget with no consequence is a chart on a wall. The consequence that works in an agency-and-client arrangement is simple and slightly uncomfortable: when the budget for an integration is spent, no further changes ship to that integration until it is back inside target. Not a punishment — a prioritisation rule that stops reliability work losing to feature work every sprint, which it otherwise always does.

The most common monitoring mistake on an ecommerce integration is measuring the thing that is easy to measure. HTTP status codes, queue depth, job success rate. All of those can be perfect while the business outcome is broken.

A worked example. An order webhook arrives, your endpoint returns 200, the job is queued, the job runs, and the ERP rejects the payload because a SKU is missing from its item master. Your dashboard is entirely green. The order is not in the ERP, and it will not be until somebody in the warehouse notices it is not on the pick list. Every technical indicator in that chain succeeded.

So define the indicator as a state, not an event: orders created in Shopify in the last hour that do not exist in the ERP. That query is harder to build and it is the only one that would have caught the example above. Do the same for the other direction — stock levels in the ERP that differ from Shopify by more than a tolerance, fulfilments recorded in the WMS with no tracking number back on the order, refunds issued in Shopify with no corresponding credit note.

Four of those, checked on a schedule, will tell you more than forty transport metrics. They are also the ones a client actually understands, which matters when you are asking for budget to fix something.

Part of your budget is set by the platform, and it is worth knowing the figures rather than inferring them.

Webhooks retry, but not forever. Shopify's documentation states that it "retries failed webhook calls up to eight times in a four-hour period", and that "after 8 failed delivery attempts, Shopify stops attempting delivery". That gives you a real recovery window: an outage shorter than four hours is largely self-healing, and one longer than four hours guarantees permanent loss for anything that failed at the start of it. Four hours is therefore the number your on-call arrangement has to beat, and it is a much more useful target than "respond quickly".

You have five seconds. "Your app must respond to the webhook within five seconds." That single line settles the architecture: validate the HMAC, persist the payload, return 200, and do the real work asynchronously. An endpoint that calls the ERP synchronously has tied your webhook reliability to your ERP's response time, which is the most common cause of the eight-failure cascade in the first place.

Failing subscriptions can be deleted. Shopify documents the difference between subscription types plainly: shop-specific subscriptions "will be deleted by Shopify" when they fail, while app-specific subscriptions "will not be deleted by Shopify". Worse still, the documentation notes that after multiple failures in a 24-hour period the subscription is removed. An integration can therefore go from degraded to silently disconnected, and the symptom is not errors — it is the complete absence of traffic. Monitor for the absence. A "no orders received in the last hour" alert during trading hours catches this; an error-rate alert never will.

Ordering is not guaranteed. "Shopify doesn't guarantee ordering within a topic, or across different topics." Handlers must therefore be idempotent and must not assume that an orders/updated they receive is newer than the last one they processed. The documentation points at the timestamps — X-Shopify-Triggered-At or updated_at — for sequencing, and at X-Shopify-Webhook-Id, described as "a unique composite key per delivery. Use to identify and deduplicate individual deliveries", for deduplication. Shopify also debounces deliveries with identical payloads arriving within a short window, dropping the later one, which is helpful and is not something to depend on.

The other half of the platform-imposed budget is the API quota, and it behaves as a leaky bucket: cost points accumulate against a bucket and refill continuously at your plan's restore rate.

The documented restore rates are 100 points per second on Standard, 200 on Advanced, 1,000 on Shopify Plus, and 2,000 on Enterprise via Commerce Components. Query cost is calculated rather than per-request: scalars and enums cost 0, objects cost 1, interfaces and unions cost the maximum of the possible selections, connections are sized by their first and last arguments, and mutations cost 10. A single query may not exceed a cost of 1,000 points regardless of plan.

Two operational consequences. The first is that a throttle does not look like an error: the GraphQL Admin API returns 200 OK with a THROTTLED entry in the errors array, described in the documentation as "the client has exceeded the rate limit. Similar to 429 Too Many Requests." Code that only checks the HTTP status will treat a throttled request as a success and lose the data. That bug is quiet, common, and exactly the kind of thing a business-event indicator catches and a transport indicator does not.

The second is that Shopify tells you your remaining budget on every response. extensions.cost carries requestedQueryCost, actualQueryCost, and a throttleStatus with maximumAvailable, currentlyAvailable and restoreRate. A client that reads currentlyAvailable and backs off before it runs out is a few lines of code and removes an entire class of incident. Retrying blindly after a throttle, by contrast, extends the outage.

Worth noting what this implies about bulk work. A nightly catalogue sync that hammers the same bucket as your order integration will throttle the order integration, and the order integration is the one with revenue attached. If you have both, either separate them or make the bulk job yield — and for genuinely large reads, the bulk operations API exists precisely so a big job does not spend an interactive budget.

Shopify is direct about this in its own guidance: "webhook delivery isn't always guaranteed", and it recommends periodic background reconciliation jobs that fetch recently modified objects using updated_at filters.

Take that seriously and a lot of anxiety goes away. You do not need webhook delivery to be perfect; you need it to be good enough that reconciliation has little to do, and you need reconciliation to exist. A job that runs hourly, asks Shopify for everything modified since the last run, and compares it with the downstream system will repair the gaps that retries did not.

It also gives the error budget a clean definition. The budget is not "failed deliveries"; it is records that reconciliation had to fix, or worse, that it could not. Reconciliation finding three orders an hour is a system working as designed. Reconciliation finding three hundred is a fault, and the difference between those two numbers is the thing to alert on.

There is no industry figure for how reliable an ecommerce integration should be, and any consultant who offers one is quoting a number they invented. Derive it instead, from two things you already know.

Take your order volume in the relevant period and the cost of one order going wrong — not the order value, the cost of the exception: the time to find it, the call, the late delivery, the goodwill. Multiply. That is what a one-per-cent failure rate costs you per month. Now ask what the team would spend to avoid it. If the cost of the failures is smaller than the cost of the engineering, your target is lower than you assumed and you should say so out loud rather than quietly building for five nines.

Then set the budget per integration, not per store. Orders to the ERP and stock to the storefront deserve tight targets because both cause customer-visible harm. A marketing platform sync deserves a loose one. Reviews, loyalty and analytics feeds deserve almost none — if those break for a day, write it down and fix it on Tuesday.

The honest version of this conversation usually ends with two integrations on a tight budget and six on a loose one, which is the correct answer and is much cheaper to run than treating all eight as critical.

A full observability stack for eight integrations. Tracing, log aggregation and a metrics platform are genuinely good and they are an ongoing cost, a learning curve and a thing to maintain. Four scheduled reconciliation queries that email a discrepancy count will catch more real problems in year one than a dashboard nobody has opened since it was built.

Alerting on error rates alone. The failure mode that hurts most is silence — a removed webhook subscription, a disabled API credential, a scheduled job that stopped being scheduled. None of those produce errors. Alert on expected volume falling to zero during hours when it should not be zero.

And a budget with no consequence. If nothing changes when the budget is spent, the budget is decoration. Agree the rule while everything is working, because the argument is impossible to win during an incident and unnecessary to have afterwards.

• Shopify integrations and back-end systems
• Webhooks, retries and the orders you never processed
• Inventory truth: the number three systems disagree about