Using On-call Assistant for incidents
Last updated
Last updated
Cortex’s On-call Assistant leverages the PagerDuty integration to automatically surface the most vital information about entity health and metadata when an incident is triggered. On-call Assistant notifies the user(s) responsible for an incident via Slack, including information about the affected entity, recent deployments, ownership, and links to get more details, including dependencies, runbooks, and logs.
On-call Assistant helps users respond to incidents in real time, simplifying the incident response process and helping to reduce MTTR. It can also drive adoption and engagement through links to the catalogs and Scorecards.
When an incident is triggered in PagerDuty, On-call Assistant will notify relevant users via Slack. This alert will include information about the affected entity, deploy details, and ownership information so an on-call team member can reach out to other relevant parties about the incident.
Developers can access entity information that is already in Cortex directly from the Slack notification to quickly resolve issues. On-call Assistant provides a direct link to view the alert in PagerDuty, so you can also quickly access the incident from its source.
You must have the PagerDuty integration configured.
You must create an API key in PagerDuty with the Write permission.
If you create an API key with Read-only permissions, you will also need to configure a webhook to get the On-call Assistant working.
To enable, navigate to Settings > PagerDuty and toggle on Enable On-call Assistant.
If you added PagerDuty API key with Write permissions, enabling the On-Call Assistant will create a webhook subscription in PagerDuty, allowing Cortex to receive events when incidents are triggered, escalated, or unacknowledged.
If you added PagerDuty API key with Read-only permissions, you must also configure a webhook subscription.
Copy the webhook URL. You will need this in the next steps.
In PagerDuty, add a new webhook.
Paste the Cortex webhook URL into the Webhook URL field.
Choose Account for scope type.
Select the following in Event Subscription:
incident.escalated
incident.reopened
incident.triggered
incident.unacknowledged
A secret will be generated. Copy the secret. You will need it in the next step.
Click Save at the bottom of the side panel.
In Cortex, on the PagerDuty settings page, click Configure webhook.
Navigate back to the browser window where your Cortex instance is open. In the Webhook configuration Secret field, enter the secret that you generated in PagerDuty.
