Alert actionability
Decide whether a monitoring alert needs action now, is informational, is already tracked, or is flapping noise.
F@fLimits to keep in mind
Never suppress an alert permanently based on this answer alone; send noise to the alert's owner for tuning.
Problem
On-call engineers get paged for alerts that resolve themselves, duplicate an open incident, or were never meant to wake anyone. Real problems hide in the noise.
Goal
Bring actionable alerts to a person quickly and turn noise into tuning work instead of pages.
Input / state
The alert, its recent history, and open incidents for the same service.
- alert
- Alert name, service, message and current value.
- history
- How often it fired and resolved recently, counted in code.
- open_incidents
- Titles of open incidents for this service.
Decision question
actionabilityDoes this alert need someone to act on it now?
Options
actionableIndicates a current problem that someone needs to investigate or fix.informationalReports a state change or warning that doesn't need action now.already_trackedDescribes the same problem as an open incident.flapping_noiseFires and resolves repeatedly without a lasting problem.
Allowed outcomes
Decided in your code from the answer and its confidence. Thresholds are yours to set and test.
- ActionableChoice is actionable and confidence meets your threshold.
- Log onlyChoice is informational and confidence meets your threshold.
- Attach to incidentChoice is already_tracked and confidence meets your threshold.
- Tune alertChoice is flapping_noise and confidence meets your threshold.
- On-call reviewConfidence is below your threshold: probability is spread across options.
Example scenarios
Predefined demonstrations. No model is called; the outcome shown is an illustrative judgment.
Example state
{
"alert": {
"name": "checkout-5xx-rate",
"message": "5xx rate 12% (threshold 2%) for 6 min",
"service": "checkout"
},
"history": "first fire this week",
"open_incidents": []
}Outcome
IllustrativeChoice is actionable and confidence meets your threshold.
Why
A sustained error spike on a customer-facing service.
Using it well
- Count fire/resolve cycles in code and pass the numbers; the model shouldn't have to count timestamps.
- Keep hard rules in code: alerts marked critical by their owners can page regardless of this answer.
Request definition
POST https://api.typesafe.ai/v1/systemone{
"model": "jev-latest",
"state": {
"alert": "<alert>",
"history": "<history>",
"open_incidents": "<open_incidents>"
},
"questions": {
"actionability": {
"type": "choice",
"instructions": "Does this alert need someone to act on it now?",
"criteria": {
"actionable": "Indicates a current problem that someone needs to investigate or fix.",
"informational": "Reports a state change or warning that doesn't need action now.",
"already_tracked": "Describes the same problem as an open incident.",
"flapping_noise": "Fires and resolves repeatedly without a lasting problem."
}
}
}
}Request shape from the System One API reference; a Kev server accepts the same body. Replace the placeholder state with your data. Requests need your own API key or server.
Version History
Initial version