It is 2:40 on a Tuesday afternoon. Your team’s credential agent, built on the Claude Agent SDK, is working through 30 services whose credentials expire this week. For each one it generates a new credential, writes it to the secret store, triggers a rolling restart, and checks health. It is on service 16 when the node it runs on is drained for a kernel patch, and the container is rescheduled.
The job restarts from the top with no memory of what it finished. Five services get a second new credential before anyone notices. Two had already cached the first one, and they start failing authentication. The on-call engineer spends an hour reading logs to work out which services were rotated once, which twice, and which not at all.
This is what happens when a long-running agent is treated like a short-lived script.
Why long-running agents break this way
An agent built with the Claude Agent SDK is a loop: the model reads context, calls a tool, gets the result, and decides again. A runbook or maintenance task can run dozens of turns and hundreds of tool calls, often for longer than you can count on a pod or container living.
The loop’s state lives in process memory. When the process dies, that state goes with it. The external effects do not.
The world has moved forward, and the agent’s view of it has been reset to zero. On restart, the model sees the original task again and repeats the same actions. That is the root cause: the record of what happened and the effects themselves live in different places, and only one survives a restart.
Several problems follow:
- Restarts repeat side effects. Any tool that changes a system can run twice.
- Abandoned runs. Teams who have been burned by restarts stop restarting, so interrupted jobs are simply dropped and finished by hand.
- No reliable audit trail. Without a durable record of each turn and tool call, it is hard to answer “what did the agent actually do?” after the fact.
- Fragile approvals. Rotating a production credential should not happen without a human saying yes, and a blocking prompt inside a process loses the request if the process restarts while waiting.
Why common workarounds fall short
Retrying the whole job. Kubernetes Jobs, CI pipelines, and cron wrappers will happily restart a failed run. That caused the opening incident. Restarting from the top is only safe if every action is harmless to repeat.
Asking the model to check first. A system prompt line like “check whether the credential was already rotated today” sometimes works. A safety property that holds most of the time is not a safety property.
Saving the conversation yourself. Serializing message history after each turn helps, but if the crash lands after a tool call ran and before the save, the record says the call never happened. You also own custom checkpoint code forever.
Pinning to a node that never goes away. This reduces interruptions without removing them, and fights how your platform is designed to operate.
Approval via a polling loop. This ties an hours-long wait to a process that can die at any time.
The general approach: make every step durable
The pattern that solves this is durable execution. Instead of treating the agent run as one long function call, you treat it as a workflow made of recorded steps. The principles are vendor-neutral:
- Record each step before moving on. Every LLM turn and every tool call becomes a discrete step whose input and output are written to durable storage when it completes.
- Resume by replay, not by restart. After a crash, the runtime replays the recorded history to rebuild the agent’s state, and continues from the first step that has not completed. Completed steps are not run again.
- Make the orchestration deterministic. The code that decides what happens next must produce the same decisions when replayed. All I/O, including model calls, belongs inside the recorded steps.
- Make tools idempotent anyway. A step can be interrupted after its side effect but before its result is recorded. Durable execution narrows that window; idempotent tools close it.
- Model approvals as durable waits. A risky action should pause the workflow on an event, with a timeout, so the wait survives restarts and costs nothing while it sits idle.
- Give tools an identity. Agents that act on production systems should call tools over authenticated, encrypted channels with explicit policies about who can call what.
With these in place, the opening scene plays out differently. A new instance picks up the workflow, replays the history for services 1 through 15, and resumes at 16. If that rotation was half-done, the idempotent tool finds the new credential already present and returns it.
Implementing it with the Claude Agent SDK
Dapr Workflow is one open-source way to get these properties. It stores an event-sourced history in a state store, replays it after a crash, and does not re-execute completed activities. Workflows are written as code, and the orchestrator must be deterministic, with I/O in activities.
The diagrid Python package includes an integration for the Claude Agent SDK that runs the entire agent invocation as one Dapr workflow. Each LLM turn and each tool call becomes its own durable activity. You keep writing your tools with the SDK’s @tool decorator and hand them to a runner:
from claude_agent_sdk import ClaudeAgentOptions, tool
from diagrid.agent.claude_agents import DaprWorkflowAgentRunner
@tool(“get_ticket”, “Fetch a ticket by ID”, {“ticket_id”: str})
async def get_ticket(args):
return {“content”: [{“type”: “text”, “text”: “…”}]}
options = ClaudeAgentOptions(system_prompt=”…”, model=”claude-sonnet-4-6″)
runner = DaprWorkflowAgentRunner(name=”ops-agent”, options=options, tools=[get_ticket], max_iterations=10)
runner.start()
async for event in runner.run_async(user_message=”…”, session_id=”s-001″): …
Install it with pip install “diagrid[claude_agents]”. After dapr init, run it with dapr run –app-id <id> –resources-path ./resources — python3 agent.py, with a state store component that has actorStateStore: “true”. Use a stable session_id, such as a change ticket ID, not a random one per process.
A few practical notes for credential rotation and similar work:
- Keep tools single-purpose. Splitting rotate_credential into generate_credential, store_credential, and restart_service gives finer recovery points and a clearer history.
- Pass idempotency keys. Derive them from the run and the target, for example the session ID plus the service name. Have store_credential check whether a secret version with that key already exists.
- Set max_iterations deliberately. It caps how many turns the agent can take; size it to the work. The 10 in the sample is far too low for a 30-service rotation.
For approvals, Dapr Workflow supports external events and durable timers. A common shape is to run the risky part of the process as a workflow that waits on an approval event raced against a timer of, say, 48 hours, using ctx.wait_for_external_event and wf.when_any. If the timer wins, the workflow escalates or stops. Because the wait is recorded, a restart during those 48 hours changes nothing. Check how your integration exposes this; often the cleanest option is a parent workflow that gates the agent run.
On identity, Dapr’s Sentry service issues SPIFFE-based workload identities, and sidecars use mTLS by default. Access-control policies can restrict which app IDs may call which operations, so the agent’s app ID can call the secret-rotation service, and nothing else can. Resiliency policies add timeouts, retries, and circuit breakers to those calls without changes to tool code.
One public example is close to this exact use case: an IT services and DevOps team used Dapr Workflow for Azure cost attribution, security key rotation, and legacy file integration with fault-tolerant pipelines.
If you would rather not run the workflow engine and state store yourself, managed options exist. Diagrid offers Catalyst, a durable execution platform for agents and workflows built on Dapr, which lists human-in-the-loop approvals, a cryptographic identity per agent and MCP server with mTLS by default, and OpenTelemetry tracing of each activity. It can be Diagrid-hosted, run in your own cloud, or self-hosted. As with any vendor, test failure scenarios in your own environment.
A rollout checklist
Before your next long-running Claude Agent SDK job goes to production, work through this list:
- ☐ Inventory every tool and mark which ones change external systems.
- ☐ Make each state-changing tool idempotent, using a key derived from the session and target.
- ☐ Split multi-effect tools into single-effect tools.
- ☐ Run the agent invocation as a workflow so each LLM turn and tool call is a recorded step.
- ☐ Use stable session IDs tied to the business task, not to the process.
- ☐ Put approvals for risky actions behind a durable wait with a timeout and an escalation path.
- ☐ Give the agent its own workload identity and restrict which services it can call.
- ☐ Add timeouts and retry policies on tool calls.
- ☐ Turn on tracing so every activity is visible after the fact.
- ☐ Kill the agent’s pod mid-run in a staging environment and confirm that it resumes without repeating completed work.
That last item matters most. Until a run survives a forced restart, you are trusting the design rather than verifying it.
Conclusion
Long-running agents fail in a specific, predictable way: the process forgets, and the world remembers. The fix is to make the agent’s record of its work as durable as its effects. Treat each turn and each tool call as a recorded step, make tools safe to repeat, and put human approvals on waits that survive restarts. You can build that with open-source tooling or adopt a durable execution platform for AI agents, but the principles are the same either way, and they are what keep the next node drain from turning into an outage.
