--- name: dagster-mcp-pipeline-failure-triage description: "Diagnose one failed Dagster+ run or job the way an experienced data engineer does: find the failing run, read its logs from the failure point backwards, separate the root cause from downstream fallout, check what alerts already said, and recommend what to re-run. Use when the user asks why a run or job failed, wants to triage a failed pipeline, asks what broke last night, or shares a Dagster run id or job name with an error." --- # Triage a pipeline failure The goal is a **root cause for one failure**, with evidence, and a clear recommendation on what to re-run. It is not a list of every error in the logs. A run that fails at step 14 of 30 has usually one broken step; the other failures after it are fallout, and reporting them as separate problems sends people in the wrong direction. ## 1. Pin down the exact run Confirm which run you are diagnosing before reading anything. Use `get_run` with the run id if the user gave one. If they gave a job name or a time ("last night's ingest"), find the candidates with `list_alert_policies` only if you need context on the job, and otherwise ask for the run id rather than guessing. Note: - **Status and failure time.** A run that is still in progress is not a failure yet; say so and stop, or wait. - **Which job, which code location, which deployment.** A failure in a branch or staging deployment is a different conversation from production. Confirm the deployment with `get_deployment_info` first if there is any doubt. - **Is it the first failure or a retry?** A run launched as a retry of an earlier failure means the original failure is the evidence you need. If the run id does not resolve, say so and do not substitute a nearby run. ## 2. Read the logs from the failure point backwards Use `get_run_logs` for the run. Long runs produce long logs, so do not read them top to bottom: 1. **Start at the first error-level event** and read the exception: type, message and traceback. The top in-app frame (code the team owns) matters more than library frames. 2. **Then read the events in the minutes before it.** The last successful step, the last query, the last resource initialisation, and any warning are usually the trigger. 3. **Only then look at events after the failure.** Those are either skipped steps, cleanup, or the error propagating. Treat them as fallout until shown otherwise. If the logs are truncated or paginated, say which part you read and fetch the next page only when the failure point is not yet covered. ## 3. Separate root cause from fallout A run often fails in several places. Sort the failed steps into two groups: - **Originating step:** the first step that failed on its own terms (bad input, a timeout, a schema change, an auth error, a null that should not be null). - **Downstream failures:** steps that failed or were skipped because an upstream asset or op did not produce its output. Only the originating step gets a root-cause claim. Downstream failures get "blocked by ". If you cannot tell which came first from the logs, say that explicitly rather than picking one. ## 4. Match against known failure shapes Before theorising, check whether the evidence matches a common pattern: - **Upstream schema change:** a column is missing or renamed, or a type changed. Fails on the first read of the new shape, often at the first transform. - **Source timeout or rate limit:** the error is a connection timeout, a 429 or a "too many requests". Usually transient, but check whether it recurs at the same time each night. - **Auth or secret expired:** a 401/403 or "credentials invalid" appearing suddenly on a job that ran fine before. Check the timing against when credentials were rotated. - **Out-of-memory or killed process:** the logs stop abruptly with no Python exception, or mention a container or memory limit. Often data volume grew; check whether the input is larger than usual. - **Null or empty input:** a source produced zero rows or a null key, and a downstream step assumed data. The failure is at the consumer, but the cause is at the producer. - **Dependency or environment mismatch:** an import error or a missing module after a code location deploy. Check whether the failure began with a deployment. - **Partition or date boundary:** the failure only happens for one partition or a date range, such as a month end or a daylight-saving change. If the evidence matches none of these, say "no known pattern" and report what you have. ## 5. Check what the alerts already said Use `get_run_alert_notifications` for the run to see whether a policy fired, to whom, and when. Two things to report: - **Was the failure alerted?** If no alert fired for a failed run, that is a coverage gap worth noting. Do not audit the policies here; that is the job of the alert policy review skill. - **Did the alert fire on the same failure?** If there were several notifications for one run, the team may already be working on it. ## 6. Recommend the re-run, carefully State whether the run should be re-run and what it depends on: - **Transient cause** (timeout, rate limit): re-run is reasonable, possibly after a delay. - **Data cause** (bad or missing source data): re-running will fail again until the source is fixed. Say which upstream owner must act. - **Code or config cause**: a fix must be deployed first. Re-running the old code reproduces the failure. - **Partial success**: if some steps succeeded, say whether a re-run should target only the failed steps or the whole job. Check the asset and partition state before recommending which. Re-running is an action that changes state. Propose the exact run to launch with `launch_job_run` or `launch_asset_run`, explain why, and wait for the user's explicit yes. Do not launch anything from this skill on your own initiative, even if the cause looks clearly transient. ## Common patterns and traps - **Fixing the last error instead of the first.** The final traceback is often a consequence. Always read from the first error-level event. - **Calling a skipped step a failure.** Skipped steps after an upstream failure are not separate problems. - **Trusting the alert message.** An alert summarises the run; it does not show the cause. Verify against the logs. - **Re-running a deterministic failure.** If the same input will produce the same error, a re-run wastes time and may trigger downstream side effects twice. Confirm the cause is transient first. - **Ignoring the timing.** A failure at the same hour each night points at a source system schedule or a resource limit, not at the code. - **Assuming the latest code caused it.** Check whether the failure predates the most recent deployment before blaming it. - **Losing the run id.** Quote the run id in every line of the report so the user can find the exact run. ## Output ``` Run: , job , deployment , failed at