Cancellation & recovery

Four engine methods handle the endings that aren’t “every step completed”: one you call deliberately, one you run on a timer, one you reach for when a run failed for a reason you have since fixed, and one that deletes the runs about a record when the record itself has to go.

Cancelling a workflow

const cancelled = await engine.cancelWorkflow(workflowId);
if (!cancelled.ok) throw new Error(cancelled.error.message); // workflow_not_found

Every pending and waiting step is marked skipped, the workflow finishes cancelled, and a workflow.cancelled event is emitted. Three things worth knowing:

• It interrupts a running step only if that step beats. There is no signal into an in-flight handler on its own, so by default the step runs to completion and its result lands on a workflow that has already finished. A step with a heartbeat learns it was cancelled on its next beat: the engine fires ctx.signal and discards whatever the handler returns. Without one, cancellation remains a “stop scheduling more work” operation, not a kill.
• Compensation does not run. Saga rollback is wired to the failure path only (see Saga compensation). If you need completed steps undone on a cancel, do it yourself after the call returns.
• Cancelling a sub-workflow child fails the parent step, which cascades into the parent workflow exactly as a child failure would.

Calling it on a workflow that already reached a terminal state is a no-op that still returns ok — so a double-click on a cancel button is safe.

Recovering after a crash

A worker that dies mid-step leaves its step in running forever. Nothing detects that on its own: the queue’s job expiry releases the job, but the row stays claimed, and an atomic claim (markStepRunning) means a redelivered job can’t take it over.

recoverStuckWorkflows is the sweeper that clears them:

const { retriedSteps, recoveredSteps, recoveredWorkflows, expiredWorkflows } =
  await engine.recoverStuckWorkflows();

It scans every running workflow in the partition and does three things:

• Re-queues a crashed step that has been silent longer than its liveness window, provided its attempt budget has room. A dead pod costs an attempt, not the run.
• Fails a crashed step whose budget is spent, cascading the workflow to failed in the usual way (dependents skipped, compensation run).
• Fails a run past its deadline, which nothing else would notice while the run is suspended or idle.

Nothing calls this for you. Run it on a schedule in one process — a cron job, a setInterval, or a pg-boss schedule:

setInterval(() => {
  void engine.recoverStuckWorkflows().catch((e) => logger.error('sweep failed', e));
}, 60_000);

Once per partition is enough. Two sweepers racing on the same partition will not corrupt state — the step transition is a plain UPDATE and the cascade is recomputed from listSteps — but each one increments the workflow’s failedSteps counter, so a double sweep can inflate that progress number. Run it from one process, or accept the skew.

The stuck threshold

stuckThreshold = stepExpirySeconds + stuckStepBufferSeconds

This is the default window, measured from when a step started. A step type that declares heartbeatTimeoutMs overrides it with its own, measured from when the step last reported in — which is what lets the window be short without condemning work that is merely slow.

Both come from WorkflowEngineConfig, and both have defaults:

Option	Default	Meaning
stepExpirySeconds	600	What you told the dispatcher a step may occupy a worker for
stuckStepBufferSeconds	300	Grace on top, so a step that is merely slow isn’t swept
onStuckStep	'retry'	'retry' re-queues within the attempt budget; 'fail' always fails

Per step type, heartbeatTimeoutMs replaces the first two entirely for that type.

const engine = createWorkflowEngine({
  store, dispatcher, registry, partitionKey,
  config: { stepExpirySeconds: 900, stuckStepBufferSeconds: 300 }, // sweep at 20 min
});

What recovery does not cover

The sweeper only looks at steps stuck in running. A step stuck in pending with no job behind it is invisible to it — that is the dual-write window between persisting a transition and enqueueing the jobs it unlocks.

Closing that window is the job of transactional dispatch: with postgres + pg-boss on one Postgres, the write and its enqueues commit together and the window doesn’t exist. On a queue that lives elsewhere (SQS, Redis) the engine falls back to write-then-enqueue, and the repair path is the dispatcher redelivering the completed step’s job, which re-drives readiness.

Retrying a failed run

A workflow that used up its retries is failed, and failed is terminal. Starting a fresh run repeats every side effect the first one already committed — the charge, the email, the file that was written. retryWorkflow is the alternative: resume the existing run from where it stopped.

const retried = await engine.retryWorkflow(workflowId);
if (!retried.ok) throw new Error(retried.error.message);
retried.value.resetSteps;   // ['fulfil', 'notify'] — what will run again

Every step that failed, was skipped in the fallout, or had its work compensated away goes back to pending with a fresh attempt budget, its output, error and timestamps cleared. Steps that completed are left exactly as they are — they keep their output, and they do not run again. The workflow returns to running with recomputed counters, and the frontier is dispatched.

→ examples/16-deadlines-and-retry.ts

Details worth knowing:

• A compensated step does re-run. Saga rollback undid its effects, so its work has to happen again — that is why compensated is reset alongside failed and skipped.
• A map parent’s children are dropped before it re-runs, so it fans out afresh rather than aggregating two generations of items.
• A guard is re-evaluated. A step skipped by a when guard is reset too, so the branch decision is made again against the current data.
• It refuses a run that is not failed. A completed, running or cancelled workflow returns a workflow_not_retryable error rather than being restarted.
• It refuses a sub-workflow child, pointing you at the parent. Retrying a child cannot un-fail the parent step that was waiting on it; retry the parent, and it starts a fresh child.
• A workflow.retried event is emitted for your run history.

Deleting the runs about a record

A run keeps what its steps were given and what they returned — an application a model read, the draft it wrote, the answer a person gave. When the record the runs were about is deleted, or a person’s data is erased, those runs have to go too. They point at the record only by entityRef, with no foreign key, so nothing removes them on its own.

const purged = await engine.purgeWorkflows({ entityRef: `contact:${contactId}` });
if (!purged.ok) {
  // workflow_step_running: purged.error.running lists the steps a worker holds right now
  return conflict(purged.error.message);
}
purged.value.workflowIds; // every run deleted, its sub-workflow runs included

Every run of the engine’s partition whose entityRef matches goes, with every sub-workflow run below it (a child carries its parent’s id, not the entity), and with all of their steps, map children included — in one statement on octaflow/postgres, whatever foreign keys your schema declares.

• It refuses while a step is executing. A step that is running (a worker is inside its handler) or compensating would go on to write about something that no longer exists, so the call returns workflow_step_running and changes nothing. Try again once the step settles; the error names the runs and steps.
• Waiting does not stop it. A run suspended on a person (waiting), a sleep, a sub-workflow, or work that is only queued is cancelled first, exactly as cancelWorkflow would — parents before children — and then deleted. A pending review never blocks a deletion.
• What was queued for a purged run does nothing. A step job, a wait deadline or a late resumeStep finds no run and returns workflow_not_found; nothing is written back. The one window it cannot close is a worker claiming a step between the check and the delete: that handler’s outcome lands on rows that are gone and is dropped.
• workflow.cancelled, then workflow.purged is emitted per run.
• Runs only. Rows your app keeps elsewhere under the run’s id — the event sink’s flow_step_event, a ledger entry, a usage rollup — are yours to keep or delete; the returned ids are there for that. A ledger of who applied what is usually kept on purpose.

Delete the record after the runs, not before: while the runs exist, a step can still read the record; once they are gone, nothing will.