INTEGRATIONS · OPS · DATA · 15 SEPTEMBER 2026 · 8 MIN READ
What peak revealed about your integrations
Peak does not break integrations. It removes the slack that was hiding a design fault, and the failure it produces tells you exactly which one.
That they were already wrong in March. Peak does not break a well-built integration; it removes the slack that was covering a design fault, and integration failures at peak fall into four shapes — throttled, timed out, duplicated, silent — each of which names a specific thing that was built on an assumption volume has now disproved. The useful output of a bad Friday is therefore not an incident list. It is four questions about the architecture, and the honest answer to most of them is cheaper than the middleware someone will propose buying in January.
IN SHORT
- Peak is a load test nobody designed, run against assumptions nobody wrote down — which is why it is the most informative four days of your year.
- Throttling at peak is almost never a platform limit problem; Shopify meters per app per store, so your own jobs were competing for one budget.
- Shopify documents a one-second connection timeout and a five-second total timeout on webhook delivery, so an endpoint that does work inline fails first under load.
- After eight consecutive failures — eight retries over four hours — Shopify deletes a webhook subscription created via the Admin API, so a bad evening can end deliveries entirely.
- Duplicate records are not a delivery bug. Webhooks are at-least-once and unordered by design, and peak is simply the first time the volume makes that visible.
- The gap between what Shopify holds and what your ERP holds is the only number that tells you whether anything was actually lost, and most teams cannot produce it.
- Fix the four faults in September. In November you will be under a code freeze and the answer will be "next year".
Peak is the load test you did not design
Every integration in your stack encodes assumptions. That orders arrive at a rate one consumer can absorb. That a webhook handler finishes before the caller gives up. That the nightly catalogue job has the API to itself at 2am. That an update always arrives after the create it refers to. None of these were decided; they were inherited from the conditions on the day the thing was built, which were quiet.
Peak is the only time of year those assumptions are all tested at once, by an adversary with no interest in your architecture. It is a better load test than anything you would have written, because it applies real concurrency to real data through every path at the same time — including the paths your staging environment does not have, like a customer service agent bulk-editing orders while the queue is already deep.
What it produces is not a verdict on whether your stack is good. It is a set of observations about which assumptions were wrong. Treat the incident log as a findings document rather than a record of a bad night, and it turns into a roadmap you did not have to write.
Four shapes of failure, and what each one is telling you
Sort what happened by symptom rather than by system. Nearly everything lands in one of four buckets, and each bucket points at a different design fault.
- Throttled. Your app hit the rate limit. Shopify meters per app, per store, using a leaky bucket, and the GraphQL Admin API returns
THROTTLEDinextensions.codewith the current state of the bucket inextensions.cost. The finding is not "we need a higher limit". It is that several of your jobs were spending one budget with no shared view of it. - Timed out. Requests that succeeded in October returned nothing in November. The finding is that a synchronous dependency was on a path that should not have had one — an endpoint doing real work before responding, or a storefront call waiting on an ERP that was busy.
- Duplicated. The same order was written twice, or an inventory adjustment applied twice. The finding is missing idempotency, not an over-eager retry. Something retried correctly and your consumer was not safe to call twice.
- Silent. No error, no record. This is the worst of the four, because nothing paged anybody. The finding is that you have no reconciliation — nothing that periodically compares what Shopify holds against what the downstream system holds and reports the delta.
The evidence worth pulling before it ages out
Three artefacts make the difference between a findings document and a story people tell. All three decay — log retention on most platforms is shorter than the gap to the next peak — so pull them now rather than in the January review.
The throttle trace. Every GraphQL Admin response carries extensions.cost with requestedQueryCost, actualQueryCost and a throttleStatus. If you were logging it, you can reconstruct which of your processes drained the bucket and when. If you were not, the first fix is to start, because it converts the vaguest incident class into an attributable one. Note that the Admin API returns most errors with HTTP 200 and the detail in the body — a client that only checks status codes recorded your throttling as success.
The webhook failure record. Shopify documents a one-second connection timeout and a five-second timeout on the whole request, and retries a failed delivery eight times over four hours. After eight consecutive failures a subscription created through the Admin API is deleted and the app's emergency contact is emailed. That sequence matters for the post-mortem: a handler that got slow under load, rather than erroring, can take itself off the distribution list without anyone noticing until the next quiet Tuesday.
The reconciliation delta. Count the orders Shopify recorded in the peak window and the orders the downstream system holds for the same window, and subtract. Most teams discover they cannot run that query, which is itself the finding. Shopify's own webhook guidance is to run reconciliation jobs that periodically fetch data so an app stays consistent — it is documented as an expectation of the design, not a belt-and-braces extra.
The fixes, in the order they are worth doing
These are ordered by value per day of work, which is not the order they usually get scheduled in.
- One queue per shop, with a concurrency limit. This single change fixes more throttling than any retry strategy, because it makes the shared budget visible to every job instead of letting four services discover it independently.
- Make every consumer idempotent. A natural key from the payload and an upsert removes the entire duplicate class permanently, and it is usually a day of work per consumer.
- Move batch work off the interactive path. Shopify's bulk operations are documented as not having the maximum cost or rate limits that single queries have. A catalogue sync that runs as a bulk operation stops competing with the customer-facing lookups.
- Return 200 first, work afterwards. If a handler cannot reliably answer inside five seconds under peak load, it needs to acknowledge and enqueue. Shopify's own guidance says to queue the payload and process asynchronously.
- Add the reconciliation job you did not have. Nightly is enough. It needs to report a number even when the number is zero, because a job that only speaks up when something is wrong is indistinguishable from a job that has stopped.
- Graph queue depth. Not error rate — depth over time. It is the one metric that shows a system falling behind while every individual request still succeeds, and it is the earliest warning you will get next year.
What not to conclude from one bad Friday
The January proposal is usually to buy something. An integration platform, a queueing product, an ERP upgrade, a middleware layer whose sales deck opens with a diagram of your stack looking tidier. Some of those are the right answer eventually. Almost none of them are the right answer to the findings above.
Middleware does not make a consumer idempotent, and it does not give your jobs a shared view of a rate limit budget — it becomes another client of the same bucket. If the fault was that four services spent one budget blindly, adding a fifth service in front of them relocates the problem and adds a vendor to the incident call. Do the queue and the idempotency work first; then see whether the platform is solving anything that is left.
Be similarly sceptical of raising concurrency. It is the intuitive response to a backlog and it is usually wrong: more parallel workers against a leaky bucket drain it faster and throttle sooner, converting a slow system into a failing one. The lever that helps a deep queue is either fewer, larger operations, or moving the work into a lane that is not metered the same way.
And resist the urge to conclude that the platform let you down. It is a comfortable finding and it ends the investigation, which is precisely why it is worth checking twice. In our experience the platform limits are the documented, predictable part of the system; the surprises are in what your own components do to each other when they are all busy.
Why this is September work
There is a narrow window for this and it is open now. In eight weeks you will be inside a code freeze, and the correct answer to "should we refactor the order consumer" will be no. Everything in this post is a change to the parts of the stack that a freeze exists to protect, which means it either lands with two months of ordinary trading to shake it out, or it waits a year.
That is also the argument for doing the smallest version of each fix rather than the complete one. An idempotent consumer that has run through eleven quiet weeks is worth more on the day than an elegant redesign that has run through none. Ship the boring version, let the autumn prove it, and freeze with the evidence in hand.
If the four questions above do not have owners by the end of the month, next peak will produce the same findings document, and the January meeting will have the same proposal in it.
Questions this raises
Was it Shopify that failed, or us?
Check the shape of the failure before answering. Throttling is metered per app per store, so a `THROTTLED` response is a statement about your own traffic, not about the platform's capacity. Webhook timeouts are measured against your endpoint. Genuine platform incidents exist and are published, but the failures most teams carry out of peak are attributable to their own components competing.
How do we tell whether webhooks were dropped or never sent?
You cannot, from your side alone, which is the point of reconciliation. Compare the records Shopify holds for the peak window with what your system holds and work from the delta. Shopify documents webhook delivery as not always guaranteed and recommends periodic reconciliation jobs, so a design that treats webhooks as the source of truth is being used outside its stated guarantees.
Do we need an integration platform after a peak like that?
Probably not yet. An iPaaS is worth buying when the problem is the number of connections you maintain, not when the problem is that four jobs shared one rate limit budget or a consumer was not safe to call twice. Both of those follow you into the new platform. Fix them first; the platform decision will be clearer and cheaper afterwards.
Should we increase concurrency so the queue drains faster?
Usually the opposite. Against a leaky bucket, more parallel workers drain the available budget faster and start throttling sooner, so a backlog that was moving slowly stops moving at all. Reduce the number of calls instead — batch them, or move the work to bulk operations, which Shopify documents as exempt from the per-query cost and rate limits.
How long is the peak evidence still usable?
Shorter than you think. Log retention on most hosting and observability plans is measured in weeks, and Shopify's webhook delivery history is not a long-term archive. Export the throttle traces, the failed-delivery records and the reconciliation counts while they exist; a finding you cannot evidence in January will lose to a roadmap item somebody can.
Is peak data representative enough to design around?
For rates and conversion metrics, no. For failure modes, it is the best data you will get, because the faults it exposes are real faults that were simply below the threshold of visibility the rest of the year. A consumer that duplicates under concurrency duplicates in March too — you just process few enough orders to never see it.
NEXT STEP
Free store audit
A senior Shopify engineer reviews your storefront, theme performance and checkout, then sends a prioritised list of fixes.