--- name: ultipa-instance-incident-diagnosis description: "Respond to a reported fault on one Ultipa instance the way an experienced on-call engineer does: narrow the timeline with logs and metric history, check whether a recent lifecycle or version change lines up with the symptom, and recommend a reversible next step before any change. Use when the user says an Ultipa instance is down, slow, erroring, refusing connections, restarting, or running out of resources, or asks what went wrong and when." --- # Diagnose a reported fault on an Ultipa instance The goal is a **cause hypothesis with evidence and a reversible next step**, not a restatement of the complaint. "The database is slow" is the symptom. The cause is "memory climbed steadily after the upgrade to version X and the restarts at 02:10 reset it", or whatever the evidence actually shows. Start by pinning down what the connector can see. In this connector the control plane (instance lifecycle, status, logs, metrics) and the data plane (queries against a graph) are separate. Diagnosis here uses the control-plane tools only, so say so in your report: you are reading the instance's state and telemetry, not the content of its graphs. ## 1. Identify the target and the time of onset Confirm which instance is meant. Use `get_instance` with the instance ID, or `list_instances` if the user gave a name. Confirm the target is Ultipa Cloud or a direct instance first (the setup decides which tools will work). If several instances are plausible, ask which one before reading telemetry. Looking at the wrong instance wastes the whole investigation. Record the **onset**: the time the user first noticed the fault, the time of the first error they quote, and whether it is ongoing. The onset is the anchor for every later comparison. If the user cannot give one, say so and use the earliest anomaly you find as a provisional onset. Read the current state from `get_instance` and check whether an operations lock is held with `get_operations_lock`. A lock or an in-flight operation means someone else may be changing the instance right now, which changes the diagnosis: the fault may be a maintenance window, not a defect. ## 2. Establish the instance's current condition Use `wait_for_instance_status` only as a read: if the status is transitional (starting, stopping, upgrading), do not treat it as settled. Report the status as observed, and note whether it is steady. Do not wait on a status to force a state change; a long wait on a fault adds delay for no information. ## 3. Build the timeline from logs Pull `get_instance_logs` for a window that starts a little before the onset and runs to now. Read it in order, not by severity alone. The pattern you need is the sequence: normal lines, then the first unusual line, then escalation. The first anomaly is more useful than the loudest error that follows it. Things to look for, without assuming the log format: - restarts, crashes, out-of-memory or resource messages, and the lines immediately before them - authentication or connection failures, which often point to credentials, network, or the instance refusing new sessions - lifecycle events (start, stop, pause, resume, upgrade) with their timestamps - repeated identical errors, which usually mean a retry loop rather than many separate faults If the logs are empty or sparse for the window, say so. Silence is evidence about logging, not proof that nothing happened. ## 4. Compare against metric history Pull `get_metrics_history` over the same window, and `get_live_metrics` for the present reading. Match the metric curves to the log timeline. Ask: - Did a resource climb steadily and then cliff (memory leak, then an OOM restart)? - Did a metric jump at a single moment (a change, a load spike, a config switch)? - Did the fault begin when a resource was already near its limit, or when it was comfortable? A metric that matches the onset is a lead. A metric that is flat across the onset is evidence against the resource explanation, and you should say that too. ## 5. Check whether a change lines up with the onset Faults after changes are the most common shape. Use `get_latest_version` to compare the running version against the newest available, and `get_instance` for the recorded version and last-modified information. Then check whether a lifecycle event in the logs, a version change, or a size or configuration change falls within the window before the onset. Also check `get_trial_status` and `get_enterprise_status` if the fault coincides with a plan boundary, since those can change what the instance is allowed to do. Treat these as possible contributors and name them as hypotheses until the evidence confirms them. If a change does line up, the working hypothesis becomes "the fault is caused by change X at time T". State it as a hypothesis and list the one fact that would disprove it, such as the fault predating the change in the logs. ## 6. Form a hypothesis, then try to break it State it in the form: "The fault happens because **A**, under **B**, since **T**, most likely triggered by **C**." Then look for one piece of evidence that contradicts it. Confidence is high only when the log timeline, the metric curve and a change event agree on the same onset. Two of three is medium. One source alone is low, and you should say which source. ## 7. Recommend a reversible next step Do not fix the instance. Recommend the least disruptive action that reduces harm or gathers more evidence. Prefer, in order: 1. more evidence: a longer log window, a metric window at a finer grain, or confirming the instance's credentials and state 2. a reversible operation that stops the bleeding, such as a restart, but only with the trade-off stated (a restart loses in-memory state and interrupts clients) 3. a change that can be undone, such as an upgrade, pause or resume, each with its rollback path Any of these that changes state (`restart_instance`, `pause_instance`, `resume_instance`, `upgrade_version`, `set_log_level`, `rename_instance`, `create_instance`, `delete_instance`, `reset_admin_password`) requires that you describe it, its effect and its rollback, and then wait for the user's explicit yes before doing it. Recommending one is not the same as running it. A diagnosis is complete without any state change. ## Common patterns and traps - **Restart loop.** Status flips between running and starting, with the same error each time. The cause is almost always upstream of the restart (a crash on startup, a bad config, memory exhaustion on load). Restarting again will not help; read the lines just before the crash. - **A fault that starts exactly at a lifecycle event.** Check the log for that event before blaming load. A resume or upgrade that never fully finished is a common cause. - **Metrics look fine, the user is failing.** The problem may sit between the client and the instance, in credentials, the connection string, or a permission scope. Say the instance looks healthy and the fault is probably outside it, and list what to check on the client side. Do not conclude "no fault". - **Empty or delayed logs.** The incident may be before the logging window, or logs may be delayed. Widen the window and say so. Do not read silence as health. - **The loudest error is a symptom.** A timeout or a failed connection is what the client sees. The first anomaly in the logs is usually earlier and more useful. - **Maintenance mistaken for a fault.** An operations lock or an in-flight upgrade explains a lot. Check `get_operations_lock` before you hypothesise a defect. - **Correlation is not cause.** A change near the onset is a lead. Confirm it with the timeline before naming it the cause. - **Assuming the data plane is reachable.** Some tools only work with the right connection type and scope. If a call is refused for permissions, report the refusal as a finding and do not work around it. ## Output ``` Instance: , connection: Reported onset: