> ## Documentation Index
> Fetch the complete documentation index at: https://docs.kentron.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Jobs, lanes, and durable execution
> The four job lanes, the job lifecycle receipts, singleton and idempotency rules, the lease and recovery defaults, and how a job survives a worker dying.
Everything Receipt does in the background is a job: the run you start from a chat message, the task you file on the **Beetle Tasks** page (Beetle is the assistant's name in the interface), the objective you launch from the [developer CLI in the repository](/cli/from-source/factory-overview) — an objective being one written goal that [Factory](/core/factory-engine) plans into a graph of tasks, each of which becomes jobs of its own. A job is enqueued by appending a receipt, executed by a worker under a lease, and settled by appending a terminal receipt. Nothing about that path depends on a process staying alive.
That is what makes a run resumable. If the worker holding your task is killed mid-run, the job's state is still on its receipt stream, its lease eventually expires, and a recovery loop hands it back out.
## Lanes and statuses
There are **four** lanes:
`chat`, `collect`, `steer`, `follow_up`
`chat` is a real lane. Any document that lists three lanes is wrong. The [enqueue endpoint](/core/runtime-api) accepts all four, and the queue's own default is `collect`.
There are six job statuses:
`queued`, `leased`, `running`, `completed`, `failed`, `canceled`
Two narrower vocabularies sit alongside them: queue commands are typed `steer`, `follow_up` or `abort`, and the queue-command lanes are `steer` and `follow_up` only.
## Lifecycle receipts
A job's state is a fold over these ten event types on the stream `jobs/`:
`job.enqueued`, `job.leased`, `job.heartbeat`, `job.progress`, `job.completed`, `job.failed`, `job.canceled`, `queue.command`, `queue.command.consumed`, `job.lease_expired`
The reducer enforces three invariants worth designing around:
* `job.leased` is **ignored** unless the job is currently `queued`. A second lease on an already-leased job changes nothing.
* `job.heartbeat` promotes `leased` to `running`, but only when the heartbeat's worker id and attempt match the active lease. A heartbeat from a superseded attempt is inert. `job.progress`, `job.completed` and `job.failed` apply the same active-lease check.
* An event for a job that does not exist throws `Invariant: no job for `. There are two exceptions: `job.failed` and `job.lease_expired` are ignored rather than fatal, so a late settlement from a job that was never appended cannot break a fold.
## Singleton semantics and enqueue idempotency
`singletonMode` defaults to `allow`. Its other two values act on a `sessionKey`:
* `cancel` — every *active* job on that session key (`queued`, `leased` or `running`) is stopped with the reason `singleton cancel` before the new job is written. A `queued` job is canceled outright with `job.canceled`; a `leased` or `running` one receives a `queue.command` of type `abort` carrying that reason, because only the worker can stop work already in flight.
* `steer` — the most recent active job on the session key receives a `queue.command` of type `steer` carrying the new payload plus `fromSessionKey` and `fromEnqueue: true`, and **that existing job is returned instead of a new one being created**.
Lease expiry is evaluated before the session scan, so a dead worker's `running` job cannot swallow a steer.
**The idempotency contract.** An explicit `jobId` that already exists returns the existing job unchanged. Re-posting the same enqueue with the same `jobId` never creates a second job and never disturbs the first. This is the property callers rely on to make an enqueue safely retryable.
`maxAttempts` defaults to **2** and is clamped to the range 1..8.
## Payload kinds
Nine payload kinds are accepted:
`factory.dispatch`, `factory.run`, `factory.integration.publish`, `factory.integration.validate`, `factory.objective.audit`, `factory.objective.control`, `factory.objective.watchdog`, `factory.task.monitor`, `factory.task.run`
Every one of these except `factory.objective.watchdog` requires an `authContext` carrying a `userId` and an `organizationId` (with optional `workspaceId`, `receiptConnectGatewayUrl`, `sessionId` and `source`).
Each kind also declares a contract: a `workerGroup` of `chat`, `codex` or `control`, an optional durable workflow, and flags for durable activity, objective scoping, objective reconciliation on a terminal or expired lease, live execution and terminal objective audit. Only `factory.task.run` sets `hasDurableActivity`.
Actual Resonate routing, however, is decided agent-id first and kind second:
```
job.agentId === "codex" -> receipt-codex
job.payload.kind startsWith "factory.integration." -> receipt-codex
job.agentId === "factory-control" -> receipt-control
otherwise -> receipt-chat
```
These two do not always agree. A `factory.dispatch` job posted to agent id `factory` lands on the **chat** group even though its kind contract names `control` as its worker group. Treat this as an ambiguity in the implementation rather than assuming either one is the contract.
## The trap: only four agent ids have handlers
Exactly four agent ids have registered job handlers:
`factory`, `factory-control`, `factory-monitor`, `codex`
**Nothing rejects an unknown agent id at post time.** Posting to `/agents//jobs` still returns **202**, still appends `job.enqueued`, and still hands you back a job object with a stream and event URLs. The job is then routed to the chat worker group like any other unrecognised agent id, and the worker settles it as `failed` with the error `No handler for agent ''` — a non-retryable failure, so it is not attempted again. Where the worker roles are not running at all it stays `queued` instead. Check the agent id against the four above before you conclude a job is slow.
The payload is checked even though the agent id is not: a body whose `kind` is not one of the nine payload kinds above is rejected before the job is written, and the request returns **500** `Server error`.
## Who executes a job
A job runs on whichever process registered the worker function for its group, so nothing executes unless the deployment actually runs those roles; how many jobs each one takes at a time is per role, and `RECEIPT_RESONATE_EXECUTE_CONCURRENCY` overrides every default. See [processes, roles, and routing](/core/architecture) for the full table.
**A misspelled role is not an error.** A `RECEIPT_PROCESS_ROLE` value that is not one of the five role names resolves to `all`, which registers no Resonate function at all — it runs the maintenance loops below and then parks. Nothing rejects the typo, and jobs aimed at that group simply stay `queued`.
## The durable execution path
```mermaid theme={null}
%%{init: {'sequence': {'actorMargin': 15, 'width': 80, 'boxMargin': 5, 'messageMargin': 18, 'noteMargin': 5}}}%%
sequenceDiagram
participant C as Caller
participant A as api role
participant D as driver role
participant W as worker role
participant J as jobs/jobId
C->>A: POST /agents/:id/jobs
A->>J: append job.enqueued
A-->>C: 202 Accepted
A->>D: begins driver RPC
D->>J: read job, honour abort
D->>W: begins worker RPC
W->>J: append job.leased
W->>J: append job.heartbeat
W->>W: races execution
W->>J: re-read, then settle
W->>J: append terminal receipt
```
Step by step:
1. **Enqueue.** `job.enqueued` is appended to `jobs/`. The caller gets back a **202** carrying the job object and an `async` block with the job id, the stream name, the job and receipt event URLs, and the status.
2. **Dispatch outbox.** The queue receipt *is* the outbox. Enqueue never calls Resonate for the job it just created; it wakes a scanner pass, and that pass asks the driver starter to begin a driver RPC. Startup redrive, interval redrive and enqueue wake-ups therefore share one delivery path.
3. **Driver RPC.** The base dispatch key is the job id. If a previous driver or execute promise exists and is terminal, the key advances a *delivery generation*; that is bounded at 32, after which it errors with `resonate delivery recovery exceeded 32 generations for `. If a promise is still pending, the starter returns `created: false` with the reason `active_delivery` and does nothing.
4. **Driver executes.** The driver reads the job. A stale active lease ends the pass: the job is failed with `stale active Resonate job lost execution; retrying through Resonate driver` so the ordinary redrive path can pick it up again. Otherwise it honours `abortRequested` and begins the worker RPC under a fresh attempt fence with the id `:attempt:`. The driver deliberately does **not** lease the job — leasing belongs to the worker.
5. **Worker executes and settles.** It leases if the job is `queued`; work that was redelivered rather than leased here instead takes a start-fence heartbeat, and a heartbeat that does not match the current fence ends the attempt as `lease_lost`. It then checks `abortRequested`, races execution against the heartbeat loop and the execution timeout, re-reads the job, and completes, fails or cancels it under an attempt fence. Every settlement is wrapped in a storage retry: 5 attempts, 150 ms times attempt backoff, transient storage errors only.
6. **Callback.** The driver posts a JSON callback on each of `dispatched`, `completed`, `failed` and `canceled`, to `RECEIPT_RESONATE_CALLBACK_URL` / `RECEIPT_EVENT_CALLBACK_URL`, with a 5 s timeout and an optional `Authorization: Bearer` header from `RECEIPT_CALLBACK_TOKEN`.
Workers hold a long poll open against Resonate. That poll is issued with the request timeout disabled and reconnects with a backoff that starts at 1 s and doubles to a 30 s ceiling, so an idle worker keeps its delivery connection instead of dropping it between jobs.
### Failure statuses in a result
When an attempt does not complete, the worker's execution result carries one of these statuses:
`settlement_conflict`, `stale_attempt`, `lease_lost`, `terminal_state`, `canceled`, `execution_timeout`, `failed`
A failure that is marked non-retryable copies its status onto `job.result` in the `job.failed` receipt. A retryable one does not: the receipt records the error and requeues the job, and the status stays in the worker's own return value.
## Leases, timeouts and heartbeats
| Job | Lease |
| ----------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Default | `JOB_LEASE_MS`, default **300 000 ms** |
| `factory-control` running `factory.objective.control` | the larger of the default and `FACTORY_CONTROL_JOB_LEASE_MS`, itself defaulting to 900 000 ms |
| `factory-monitor` running `factory.task.monitor` with `executionPath` of `computer` | the larger of the default and `CODEX_JOB_LEASE_MS`, itself defaulting to 900 000 ms |
| `codex` | the larger of `CODEX_JOB_LEASE_MS` (default 900 000 ms) and the payload timeout plus 300 000 ms, capped at 3 600 000 ms; with no payload timeout it is `CODEX_JOB_LEASE_MS` |
The **execution timeout** is the lease minus 1 000 ms, with a floor of 5 000 ms. `FACTORY_CONTROL_JOB_EXECUTION_TIMEOUT_MS` may lower it, for objective control only.
The **heartbeat cadence** is derived rather than configured outright: it is one third of the smaller of the lease and `RECEIPT_RESONATE_ACTIVE_STALE_MS` (default 600 000 ms), floored at 1 000 ms. `RECEIPT_RESONATE_HEARTBEAT_INTERVAL_MS` can only lower that derived value, never raise it. Dividing by three gives a worker two renewal opportunities before the fence fires.
The driver's own invocation timeout is the lease plus 60 000 ms, with a floor of 120 000 ms.
## Four independent recovery loops
| Loop | Where it runs | Interval |
| -------------------------------- | --------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- |
| Queued-job redrive | Every process role | `RECEIPT_RESONATE_QUEUED_REDRIVE_INTERVAL_MS`, default 15 000 ms, first pass after a 5 000 ms startup delay |
| Objective-control outbox redrive | `worker-control` and `all` | `RECEIPT_OBJECTIVE_CONTROL_OUTBOX_REDRIVE_INTERVAL_MS`, default 5 000 ms |
| Factory objective watchdog | Whichever role holds a Resonate client; the worker runs on `worker-control` | Cron-scheduled, `RECEIPT_FACTORY_OBJECTIVE_WATCHDOG_CRON`, default `* * * * *` |
| Objective resume at startup | Every process role, once | After `RESONATE_STARTUP_SETTLE_MS`, default 1 000 ms |
The queued-job redrive is guarded by a minimum age (`RECEIPT_RESONATE_QUEUED_REDRIVE_MIN_AGE_MS`, 30 000 ms) and a cooldown (`RECEIPT_RESONATE_QUEUED_REDRIVE_COOLDOWN_MS`, 15 000 ms), and its dispatch key is `:redrive::` so a scanner pass is idempotent for one observed queue state.
The outbox redrive is single-flight. `RECEIPT_OBJECTIVE_CONTROL_OUTBOX_REDRIVE_TIMEOUT_MS` (default 15 000 ms) is an *observational* deadline only: a pass that runs longer keeps ownership until it settles and logs `factory.control_outbox_redrive_slow`, so interval ticks coalesce instead of piling duplicate database work on top of a slow pass.
The watchdog reconciles an objective for one of three reasons — `no_active_objective_work`, `phase_supersession_has_active_stale_job` or `active_objective_work_stalled` — by enqueuing an objective-control job with the reason `reconcile`. Sandboxed execution is treated as stale only after 15 minutes, deliberately longer than the UI's freshness window, because a computer-backed task can be quiet while its workspace syncs. The watchdog can be turned off with `RECEIPT_FACTORY_OBJECTIVE_WATCHDOG_ENABLED=0`; its timeout defaults to 60 000 ms with a 10 000 ms floor, and its scan limit to 200, clamped to 1..2000.
## The durable ledger is beside receipts, not above them
A Postgres-backed workflow and activity ledger tracks durable execution alongside the receipt chains. Every durable side effect is best-effort: a durable write failure is logged and does **not** fail the receipt mutation that has already been appended. Queue commands are a deliberate straight pass-through, so steer and follow-up never depend on workflow signal delivery.
The same rule governs Resonate itself. It is a durable delivery mechanism, not ownership state — the [receipt chain](/core/receipts-and-streams) is the authoritative queue. When delivery is lost after a job was leased, the queue appends a receipt-level `job.lease_expired` event so the ordinary redrive path can recover it.
Next step: [see how Factory turns an objective into these jobs](/core/factory-engine).