--- name: latchbio-workflow-run-triage description: "Triage LatchBio workflow executions in a workspace the way an experienced lab computing or bioinformatics engineer does: find failed and stalled runs first, pull the task logs that explain each failure, separate infrastructure problems from input and code problems, and report what to do next without launching anything. Use when the user asks why a Latch run failed, which executions are stuck or still running, what went wrong overnight, or wants a status check on their workflows." --- # Triage workflow executions The goal is to answer three questions for every run that matters: **did it finish, if not where and why did it stop, and what is the next step?** This is a read-only job. You inspect executions and logs; you do not relaunch, cancel or change anything. A run that failed is a diagnosis, not a reason to start a new one. ## 1. Confirm the workspace before reading anything Call `list_workspaces` and check which workspace the user means. If more than one is available and the user did not name one, ask. Executions in the wrong workspace will look like missing runs, and you will waste time explaining an absence that does not exist. Name the workspace in your report. ## 2. Pull the executions and sort them by what needs attention Call `list_executions` with the workspace and filter by status and workflow name when the user has given a scope. Keep the result small: a few dozen recent executions is enough for a status check. If the user asks about "everything", page through in batches and say how far you got. Sort the results yourself, in this order: 1. **Failed** (most urgent; someone is probably waiting on the output). 2. **Stalled**: still marked running but started a long time ago. Compare the start time with how long the same workflow usually takes. A run three times longer than its siblings is stalled in practice, even if the status says running. 3. **Running normally**: list only the count, unless the user asked for detail. 4. **Succeeded**: list only the count. Do not open them. Open failed and stalled runs one at a time with `get_execution`. Read the **task nodes**, not just the top-level status. A workflow usually has many tasks; the top-level "failed" often hides which single task broke the run. Note the first task that failed in time order. Later failures are usually downstream fallout and should be reported as such, not as separate problems. ## 3. Pull the logs that explain the failure For the first failed task, call `get_task_logs`. Inline logs are often enough; if they are truncated or the failure is near the end of a long log, use the presigned full-log link. Read the last 50 to 100 lines first, since the cause is almost always at the bottom, then scroll upward to the first error. Classify the failure before proposing anything: - **Infrastructure:** out of memory, out of disk, killed for exceeding a time limit, node preempted or lost, container image could not be pulled, network timeout to a remote resource. The inputs are probably fine; the resources were too small or the environment flaked. - **Input or data:** a file missing or unreadable, wrong file format, an empty or malformed sample sheet, a parameter value outside the expected range, a reference genome that does not match the samples. Fix the inputs, not the code. - **Code or workflow:** an exception in the workflow's own scripts, a version mismatch between tools, a bad assumption about data shape. Report it to whoever maintains the workflow. - **Transient:** one failure that a plain rerun would likely clear. Say this only if the logs support it, and say that a rerun is the user's decision. ## 4. Check whether the failure is isolated or systemic Run the same check across the other failed executions of the same workflow. If five of six failed on the same task with the same error, the cause is probably the workflow or a shared input, and one fix covers all of them. If the failures are scattered across different tasks and different error messages, treat each one separately and say so. Report the pattern count in one line: "7 failed: 6 out-of-memory on `align_reads`, 1 missing input file." ## 5. Hold the line on scope You are diagnosing, not repairing. If the next step is to relaunch with more memory or a corrected sample sheet, say what to change and that a relaunch would be a new, chargeable run. Hand that decision back to the user. The launch itself belongs to the separate launch-and-validate job. ## Common patterns and traps - **The top-level status is the wrong place to stop.** A run marked failed may have succeeded on 40 of 42 tasks. The two failing tasks determine the next step, so read them. - **Stalled is a judgement, not a status.** Compare the run's age with the typical duration of the same workflow in this workspace. An eight-hour run of a workflow that usually takes forty minutes is stalled; an eight-hour run of a workflow that usually takes ten hours is not. - **Memory failures often look like crashes.** A process killed with no Python traceback, or a signal-style exit code, usually means it ran out of memory. Check the resource usage in the logs before blaming the code. - **Reporting the last error instead of the first.** The final error in a cascade is often a consequence. Trace back to the earliest failing task. - **Assuming the workspace is the one the user means.** Confirm it in step 1, every time. - **Calling a run "fixed" because a later run succeeded.** A new execution with different inputs does not prove the old failure is resolved. Say what changed. - **Trusting a log line that says "success".** A task can log success and still produce empty output. If the user's question depends on results, check the result files reported by `get_execution`. ## Output ``` Workspace: Checked: executions, from to Summary: failed, stalled, running, succeeded Failed 1. , workflow , started