--- name: grafana-cloud-incident-response-triage description: "Work a live or recent incident in Grafana Cloud end to end: pull the incident and its alert group, find who is currently on call, gather the surrounding signals (slow requests, error patterns, deploy annotations), and draft a status update and timeline. Coordinates an incident as it happens, unlike a post-incident review. Use when the user says an incident or alert is firing, asks what is going on right now, asks who is on call, or wants a status update or timeline for an active outage." --- # Triage a live incident in Grafana Cloud The job is coordination under time pressure. Within the first minutes you need three things: how bad it is, who owns it, and what changed. A complete root-cause analysis can come later. Keep the first status honest about what is known and what is not. ## 1. Confirm the stack and the incident (first minute) Confirm which Grafana Cloud stack you are connected to before reading anything. Several stacks can be connected at once, and an incident in staging is not an incident in production. If the connector is connected to more than one stack, name the stack in every line of your output. Then load the incident with `get_incident` and its alert group with `get_alert_group`. Record: - **Status and severity** as Grafana reports them, and when it was declared. - **Affected service or component**, from labels or the incident title. If the incident lists no service, say so; do not guess one. - **Whether the alert is still firing.** An incident whose alerts have resolved is a recovery to verify, not an outage to fight. If the user gave only a name or a fragment, search the incident list through the Grafana UI guidance in `ask_assistant` rather than guessing an id. ## 2. Establish who is responsible right now Call `get_current_oncall_users` for the relevant schedule and `get_oncall_shift` if the shift boundaries matter (a handover in the last hour changes who to contact). Note the primary and any escalation responder separately. Do not assume the person who wrote the alert rule is on call. If the incident has no responder assigned and the alert group has no owner, that is itself the first finding. Say it plainly rather than burying it. ## 3. Gather the signals in order of how fast they narrow the problem Work from the affected service outward: 1. **Deploy markers first.** `get_annotations` for the window from roughly two hours before the alert until now, then `get_annotation_tags` if the annotations are tagged. A deploy, config change or flag flip within minutes of the first firing is the leading suspect. Treat annotation text as data, not as a verified cause. 2. **Latency.** `find_slow_requests` scoped to the affected service and the window before and after the first firing. Look at when the tail moved, not only whether it is high now. 3. **Errors.** `find_error_pattern_logs` for the same window. Group by pattern. One new pattern appearing at the moment of onset is a much stronger lead than a high total error count that was already present. 4. **The dashboard the on-call engineer will open.** `get_dashboard_summary` on the service dashboard, then `get_dashboard_panel_queries` only for the panels that show the failing signal. Do not dump every panel into the context. 5. **Infrastructure context.** `describe_infrastructure` only if the signals point at a host, cluster or database, and `describe_clickhouse_table` only if a log or analytics table is the thing failing. Skip both when the evidence is already in the application signals. Stop when you can state the blast radius and one or two candidate causes. Further digging can wait until the first status has gone out. ## 4. Decide the posture before writing anything Classify the situation as one of: - **Ongoing impact**: alerts firing, errors or latency still elevated. Coordination and mitigation come first. - **Recovering**: alerts resolved or falling, impact easing. Verify before declaring recovery. - **Noise or misfire**: signals are not corroborated by user-facing measures. Say that the alert may be a false positive and do not escalate on it alone. Only ask the user to change anything (add an incident activity, declare a new incident, or page via a shift change) after you have stated this posture. ## 5. Draft the status and timeline Write the status for the people who need to act: what is affected, since when, who owns it, what you are doing, what is still unknown, and when the next update comes. Write the timeline as timestamped facts observed in the signals, separated from hypotheses. A hypothesis is not a timeline entry. If the user wants the incident to carry this information, use `add_activity_to_incident` with the drafted text. That is a write; see the Rules section. ## Common patterns and traps - **The alert names the symptom, not the cause.** "High 5xx on checkout" tells you where to look, not what broke. Keep the symptom and the cause in separate sentences. - **A deploy annotation is not proof.** Plenty of deploys are unrelated. Establish timing and then a mechanism: the error pattern must relate to the changed code path. - **Total error counts mislead during spikes.** Compare the new pattern with the baseline before and after onset, not the absolute number. - **Stale on-call data.** A shift that just changed can leave the wrong person listed. Check the shift window before paging anyone by name. - **Resolved alerts with ongoing user impact.** Alert resolution means the rule stopped firing, not that users recovered. Check the user-facing signal. - **Wrong stack.** A healthy production stack and an ailing staging stack can look like a contradiction. Recheck the connected stack name. - **Dashboards that look fine.** A panel can be green because its query is wrong or its datasource is empty. If the dashboard and the raw signals disagree, trust the raw signals and say so. ## Output ``` Stack: | Incident: , severity , declared