Alerts
An alert from Alertmanager, Grafana or PagerDuty starts an investigation before anyone opens a laptop.
Connect your alerting to sremcp and each firing alert starts an investigation on its own. By the time the on-call engineer looks at their phone, the agent has gathered the evidence and suggested what to do.
Who can: Owners, Admins and SREs connect alert sources. Owners, Admins, SREs and Managers see the alerts and their investigations.
Connect a source
Create it in sremcp
Go to Alerts → New alert source. Choose what sends the alerts, name it, and optionally add instructions for these alerts (Check the deploys of the last hour first). Click Create source.
Copy the address
sremcp shows the source's webhook address once. It works like a password: anyone with it can send alerts, so keep it out of chat and tickets. New address replaces it; the old one stops working at once.
Point your alerting at it
Add a receiver, and route the alerts you want investigated to it:
receivers:
- name: sremcp
webhook_configs:
- url: https://api.sremcp.com/hooks/alerts/<token>
send_resolved: true
route:
routes:
- matchers: [severity="critical"]
receiver: sremcp
continue: trueWhat happens to an alert
- A new firing alert starts an investigation: the alert is the agent's question, with your team's notes, your runbooks, and the source's instructions. It runs with the tools of whoever created the source, like a report. If they leave the organization, the source pauses; whoever resumes it takes it over.
- The same alert again within two hours (same fingerprint, or PagerDuty incident) joins that investigation instead of starting another.
- A resolved alert is recorded with its investigation.
When the investigation is done:
- The findings go to the PagerDuty incident as a note, if the source has an API key.
- They're posted to the source's Slack channel, if it has one (Slack): the alert, then the findings in its thread. Mention @sremcp in that thread to ask more.
- Owners, Admins, SREs and Managers get a push notification on the iPhone app.
Alert investigations are in Alerts (each alert links to its investigation), in History → Team, and in Runs (filter Alerts). Anyone who sees them can carry on asking, with their own tools.
Change a source
Edit on a source changes its name, what sends its alerts, its instructions, its Slack channel and its PagerDuty settings. Its address stays the same, and PagerDuty secrets left empty are kept.
- A new Slack channel gets a short note from sremcp when you save: a mistyped id, or a private channel sremcp wasn't invited to, is caught then rather than at the first alert.
- Editing someone else's source makes it yours: from then on, its investigations run with your tools. The form says so before you save. Pausing it doesn't.
- Changing a PagerDuty source to another kind removes its PagerDuty secrets.
Limits
- At most 10 investigations an hour per source: an alert storm is recorded, not investigated 200 times. Alerts past the limit show as not investigated.
- At most 20 alerts per delivery, and they count toward the organization's daily model tokens.
- Pause a source to record its alerts without investigating them.