PagerDuty Alerts

PagerDuty alerts open an incident on your PagerDuty service when a product runs out of stock at a location — and close it again when stock comes back. They are switched off until someone on your account adds a PagerDuty destination, and they are included on the plan you are already on — there is no add-on to buy.

PagerDuty alerts are available on Shared and Dedicated instances alike. What decides is where your instance actually runs, not the plan name. If we are moving your instance between tiers, alerts keep running on the shared fleet until the move and resume on the new server once the move is complete; the daily check may not run on the days the move is in progress.

The short version#

Stock-outs only. An incident opens when a product that has a minimum level set at a location drops to zero quantity on hand there. Nothing else pages. A product that is below its minimum but not at zero, and a lot that is close to expiring, never open an incident — those stay in the daily stock alert email. If a product has no minimum level at that location, it is never checked, however low it gets.
From the same daily check as the email. We take one picture of your stock each day, in your instance's timezone, and incidents are opened and closed from that picture. This is not real time: a product that runs out at noon is not an incident until the next daily check sees it.
One incident per product and location. If the product is still at zero the next day, no second incident is opened. The incident resolves when the next daily check sees stock again — whoever is on call sees it close in PagerDuty. If the same product runs out again later, that is a new incident.
Severity is always warning. Never error or critical. A test send is info. How urgently PagerDuty notifies your on-call is governed by your PagerDuty service's urgency settings, not by us — a service defaults to high urgency whatever severity we send, unless you have turned on Dynamic Notifications.
At most 50 new incidents per destination per day. If more products than that run out on one day, we open 50 and then one extra incident saying how many more there are.
Anyone on your account can add, change or remove a destination, and can send a test — and a test pages whoever is on call.

Setting it up in PagerDuty#

You need three things on the PagerDuty side:

A service — the one whose on-call should be notified about stock-outs.
An Events API v2 integration on that service. In PagerDuty, open the service, go to Integrations, add an integration and choose Events API v2. This is what gives you the integration key (PagerDuty also calls it a routing key). Nothing else — not an API token, not an email integration — will work.
An escalation policy on the service. The service's escalation policy decides who is notified, and how urgently, when an incident opens. A service with no escalation policy opens incidents that notify nobody — useful for testing, useless for being paged.

Adding the destination#

Go to Integrations in the portal sidebar, then the Destinations tab.
Open PagerDuty and choose Add a PagerDuty destination.
Give it a name you will recognise, like "Warehouse on-call".
Pick the PagerDuty service region your account is in — United States (events.pagerduty.com) or European Union (events.eu.pagerduty.com). Most accounts are US, and US is the default. Choosing the wrong region means PagerDuty refuses every event.
Paste the integration key from the Events API v2 integration you created.
Save.

The integration key is entered once and never shown again. We store it encrypted, and there is no way to read it back — not in the portal, not through support. After saving, the panel shows a short fingerprint so you can tell one saved key from another; that fingerprint is not the key. To change the key, paste a new one; to keep it, leave the field blank.

You can have up to 5 PagerDuty destinations on an instance, each with its own key and region — every stock-out goes to all of them.

Anyone on your account can open this panel and change these settings — add a destination, change its key or region, switch it off, remove it, and send a test. It is a shared team surface, not an owner-only control. Our support team can see your destinations but can never change them.

Sending a test#

Once a destination is saved, its edit dialog has a Send test incident button. The test opens a [TEST] incident on your service at severity info and we resolve it straight away.

A test will page whoever is on call. A PagerDuty service is set to high urgency by default, so a test opens a real incident and notifies whoever is on call for that service right now. Sending it as an information-level event and resolving it immediately changes neither of those things: urgency is governed by your PagerDuty service settings, not by us. Severity only affects urgency if you have turned on Dynamic Notifications, which maps warning and info to Low urgency.

To test without paging anyone, do one of these first:

Put the service in a maintenance window. In PagerDuty's words: "While a service is in a maintenance window, its integrations are disabled and no new incidents will trigger."
Point the destination at a test service that has no escalation policy, so an incident opens and nobody is notified.
Add an Event Orchestration rule that suppresses events for the service.

Tests are rate-limited: one test per destination every 5 minutes. Pressing the button again inside that window sends nothing and tells you so — it is not a sign that the key is wrong.

What an incident looks like#

The incident title is "Out of stock: product at location (your instance)". It carries the product, the location, the quantity on hand (zero), the minimum level you set, the date of the check that found it, and a link that opens your instance in OpenBoxes. Severity is warning, the component is inventory and the class is stock-out.

If the daily cap is reached, the extra incident is titled "your instance: N more products out of stock than we can alert on today (T in total; the daily limit is 50 incidents)" and names the real numbers. It resolves on its own on the first day the cap is no longer reached.

When an incident closes#

We only ever resolve an incident under the same integration key that opened it, from the same destination. These are the ways one closes:

Stock came back. The next daily check sees a quantity above zero for that product at that location.
The product is no longer tracked. The minimum level was removed, or the product or location is gone, so there is nothing left to check.
We could not check for three days running. If your instance has had no usable daily check for three consecutive days — it was suspended, being rebuilt, or the data could not be read — we resolve the incident rather than leave it holding your on-call open with nothing behind it.
You changed the destination. See the next section.
Your instance was suspended or closed.

We never close an incident silently just because it is old. An incident that has been open for more than 30 days is listed in the panel under Open for a long time, so you can see it — but it stays open, because as far as the daily check can tell it is still real.

Changing or removing a destination closes its open incidents first#

An incident can only be resolved with the key that opened it, against the region it was sent to, through a destination that is still switched on. So each of these first resolves every incident the destination still has open — whoever is on call sees them close in PagerDuty — and only then makes the change:

Replacing the integration key. The resolves go out under the old key before the new one is stored. Pasting the same key again is not a replacement and closes nothing.
Changing the region.
Switching the destination off.
Removing the destination. Once it is gone there is no key left to close anything with, which is why this happens before removal.

The dialog says so before you save, and tells you how many incidents are open on that destination right now.

When PagerDuty stops accepting our requests#

If PagerDuty refuses an event — the integration key is wrong, the service was deleted, the region is wrong — and it happens three times in a row, we switch the destination off and stop sending to it. The panel shows it as Switched off after repeated failures. A PagerDuty outage or rate limit does not count: those are retried and never switch anything off.

To switch it back on, check the integration key and the service region in PagerDuty, then open the destination and save the key again.

When we switch a destination off this way, we try to resolve its open incidents at once. Any we cannot close — because the destination no longer accepts our requests — are listed in the panel under Close these in PagerDuty yourself, each with its incident id, so you can close them by hand in PagerDuty. They are still open on your service, and nothing on our side will close them. The same list covers an incident whose key was replaced before it could be resolved.

What is not included#

Below-minimum and nearing-expiry are not incidents. They are in the daily stock alert email only. PagerDuty is for the one thing that stops work: a product at zero.
No daily or weekly digest to PagerDuty. An incident that arrives in a batch is not an incident, and a resolve has no digest form.
Not a webhook event. Stock-outs are not available through the webhooks API; this is a notification channel of its own.
Slack is available too. The same stock-outs can be posted to a Slack channel through an incoming webhook from a Slack app, with one "stock returned" message when stock comes back — see Slack alerts. Microsoft Teams destinations are coming soon — we are building them, they are not available yet, and we give no date.

Stock alerts — the daily email, which is where below-minimum and expiring stock are reported.
Slack alerts — the same stock-outs as messages in a Slack channel.
Integrations overview — what is available today and what is coming.