Durable Execution: Workflows That Survive Restarts

Every backend eventually runs into a process that refuses to fit into a single HTTP request. An order fulfillment flow that charges a card, reserves inventory, notifies a warehouse, and sends a confirmation email. An onboarding pipeline that provisions cloud resources, seeds a database, and sends a welcome sequence. These workflows span seconds to minutes, touch multiple external services, and must complete reliably — even if the server crashes halfway through.

The standard answer is a job queue plus a state machine persisted in a database. It works, until the state machine grows to 15 states, 40 transitions, and every developer on the team is afraid to touch it. There is a better model: durable execution.

The Problem with Ad-Hoc Multi-Step Jobs

A job queue handles discrete units of work well. It breaks down when a single "job" is actually a sequence of dependent steps with branching logic, external waits, and rollback requirements.

// ❌ Fragile — state lives in ephemeral memory; server restart = lost progress
async function processOrder(orderId: string) {
  await chargePayment(orderId);
  await reserveInventory(orderId);   // If the process crashes here...
  await sendConfirmationEmail(orderId); // ...this line never runs
}
 
// ✅ Durable — each step is checkpointed; restarts replay from the last successful step
export async function processOrderWorkflow(orderId: string): Promise<void> {
  await workflow.executeActivity(chargePayment, { args: [orderId] });
  await workflow.executeActivity(reserveInventory, { args: [orderId] });
  await workflow.executeActivity(sendConfirmationEmail, { args: [orderId] });
}

The difference is not just retry logic. It is that the workflow engine persists the execution history so a restarted worker can reconstruct exactly where the workflow was without re-running completed steps.

What Durable Execution Actually Means

In Temporal (and similar engines like Inngest or Restate), workflow code is not executed directly against external systems. Instead, the engine records every decision and activity result as an append-only event history. When a worker picks up a workflow, it replays that history to reconstruct state — fast-forwarding past completed activities.

This means your workflow function runs in a deterministic sandbox. A few rules follow:

• No direct I/O inside workflow code (fetch, fs, database queries)
• No Date.now() or Math.random() — use workflow.now() and workflow.random() instead
• All side effects happen in activities, which are ordinary async functions that run outside the sandbox

The mental model: workflow code is a coordinator that calls activities, waits on timers or signals, and makes decisions based on activity results.

Modeling Workflows in TypeScript

The Temporal TypeScript SDK lets you write workflows as plain async functions.

import * as workflow from "@temporalio/workflow";
import type { OrderActivities } from "./activities";
 
const { chargePayment, reserveInventory, sendConfirmationEmail, refundPayment } =
  workflow.proxyActivities<OrderActivities>({
    startToCloseTimeout: "30 seconds",
    retry: {
      maximumAttempts: 3,
      nonRetryableErrorTypes: ["PaymentDeclinedError"],
    },
  });
 
export async function processOrderWorkflow(orderId: string): Promise<string> {
  let paymentCharged = false;
 
  try {
    await chargePayment(orderId);
    paymentCharged = true;
 
    await reserveInventory(orderId);
    await sendConfirmationEmail(orderId);
 
    return "fulfilled";
  } catch (err) {
    // Compensate only if payment already completed
    if (paymentCharged) {
      await refundPayment(orderId);
    }
    throw err;
  }
}

Activities live in a separate module and have no restrictions — they can hit the database, call APIs, or write files:

// activities.ts — ordinary async functions, no sandbox constraints
export const orderActivities = {
  async chargePayment(orderId: string): Promise<void> {
    const order = await db.orders.findOrThrow(orderId);
    await stripe.paymentIntents.capture(order.paymentIntentId);
    await db.orders.update(orderId, { chargedAt: new Date() });
  },
 
  async reserveInventory(orderId: string): Promise<void> {
    const items = await db.orderItems.findAll({ orderId });
    await inventoryService.reserveBatch(items);
  },
};
 
export type OrderActivities = typeof orderActivities;

The separation looks ceremonial until you need to mock activities in tests or swap implementations without touching workflow logic.

Signals and Queries: Interacting with Running Workflows

Long-running workflows often need external input mid-execution — waiting for a human approval, a webhook callback, or a payment confirmation from a third-party provider.

import * as workflow from "@temporalio/workflow";
 
// Define signal and query handlers
const approveSignal = workflow.defineSignal<[{ approvedBy: string }]>("approve");
const rejectSignal = workflow.defineSignal<[{ reason: string }]>("reject");
const statusQuery = workflow.defineQuery<string>("status");
 
export async function expenseApprovalWorkflow(expenseId: string): Promise<void> {
  let status = "pending";
  let approvedBy: string | null = null;
  let rejectionReason: string | null = null;
 
  workflow.setHandler(approveSignal, ({ approvedBy: by }) => {
    status = "approved";
    approvedBy = by;
  });
 
  workflow.setHandler(rejectSignal, ({ reason }) => {
    status = "rejected";
    rejectionReason = reason;
  });
 
  workflow.setHandler(statusQuery, () => status);
 
  // Block until approved or rejected, or timeout after 7 days
  await workflow.condition(() => status !== "pending", "7 days");
 
  if (status === "approved" && approvedBy) {
    await processApprovedExpense({ expenseId, approvedBy });
  } else {
    await notifyRejection({ expenseId, reason: rejectionReason ?? "Timed out" });
  }
}

Signals are fire-and-forget messages sent to a running workflow. Queries return current state without advancing the workflow. Both are sent from application code via the Temporal client — no polling, no separate status table.

Error Handling and Compensation

Durable execution does not eliminate distributed systems failures — it gives you the tools to handle them cleanly. Temporal retries activities automatically on transient failures. For non-transient failures, the saga pattern maps naturally onto workflow code.

export async function provisionTenantWorkflow(tenantId: string): Promise<void> {
  const provisioned: string[] = [];
 
  try {
    await createDatabase(tenantId);
    provisioned.push("database");
 
    await createStorageBucket(tenantId);
    provisioned.push("bucket");
 
    await deployAppInstance(tenantId);
    provisioned.push("app");
 
    await sendWelcomeEmail(tenantId);
  } catch (err) {
    // Compensate in reverse order
    const rollbacks = provisioned.reverse().map((resource) => {
      if (resource === "app") return teardownAppInstance(tenantId);
      if (resource === "bucket") return deleteStorageBucket(tenantId);
      if (resource === "database") return dropDatabase(tenantId);
    });
 
    // Activities can also fail — Temporal retries them independently
    await Promise.allSettled(rollbacks);
    throw err;
  }
}

Compare this to implementing the same compensation logic with a database state machine. Each transition, each rollback path, each retry needs a row update. The workflow version reads like the happy path with explicit error handling — which is exactly what it is.

When Not to Use Durable Execution

Durable execution has real costs. The Temporal server is another piece of infrastructure to operate. Every activity result is serialized and stored — large payloads or high-throughput workflows can stress the history store.

The signal pattern alone is worth considering Temporal for any flow that parks waiting for a webhook. Polling a status column is a solved problem that generates unnecessary database load and still requires careful handling of timeouts.

Key Takeaways

1. Durable execution is not just retry logic — it is replayed event history that reconstructs workflow state across crashes and deployments without re-executing completed steps.
2. Workflow code is a coordinator, not an executor — all I/O happens in activities; workflow functions must be deterministic.
3. Signals and queries replace status polling — send a signal to unblock a waiting workflow, query it for current state, without touching the database.
4. Compensation logic reads like code, not migrations — saga patterns map directly onto try/catch blocks in workflow functions.
5. Evaluate fit before adopting — simple background jobs, sub-second latency requirements, and stateless fan-out are better served by traditional queues.
6. The infrastructure cost is real — Temporal (or a managed equivalent like Temporal Cloud) is the right tradeoff for workflows that span hours or days, not for every async task.