Skip to main content
Everything Receipt does in the background is a job: the run you start from a chat message, the task you file on the Beetle Tasks page (Beetle is the assistant’s name in the interface), the objective you launch from the developer CLI in the repository — an objective being one written goal that Factory plans into a graph of tasks, each of which becomes jobs of its own. A job is enqueued by appending a receipt, executed by a worker under a lease, and settled by appending a terminal receipt. Nothing about that path depends on a process staying alive. That is what makes a run resumable. If the worker holding your task is killed mid-run, the job’s state is still on its receipt stream, its lease eventually expires, and a recovery loop hands it back out.

​
Lanes and statuses

There are four lanes: chat, collect, steer, follow_up
chat is a real lane. Any document that lists three lanes is wrong. The enqueue endpoint accepts all four, and the queue’s own default is collect.
There are six job statuses: queued, leased, running, completed, failed, canceled Two narrower vocabularies sit alongside them: queue commands are typed steer, follow_up or abort, and the queue-command lanes are steer and follow_up only.

​
Lifecycle receipts

A job’s state is a fold over these ten event types on the stream jobs/<jobId>: job.enqueued, job.leased, job.heartbeat, job.progress, job.completed, job.failed, job.canceled, queue.command, queue.command.consumed, job.lease_expired The reducer enforces three invariants worth designing around:
  • job.leased is ignored unless the job is currently queued. A second lease on an already-leased job changes nothing.
  • job.heartbeat promotes leased to running, but only when the heartbeat’s worker id and attempt match the active lease. A heartbeat from a superseded attempt is inert. job.progress, job.completed and job.failed apply the same active-lease check.
  • An event for a job that does not exist throws Invariant: no job <jobId> for <type>. There are two exceptions: job.failed and job.lease_expired are ignored rather than fatal, so a late settlement from a job that was never appended cannot break a fold.

​
Singleton semantics and enqueue idempotency

singletonMode defaults to allow. Its other two values act on a sessionKey:
  • cancel — every active job on that session key (queued, leased or running) is stopped with the reason singleton cancel before the new job is written. A queued job is canceled outright with job.canceled; a leased or running one receives a queue.command of type abort carrying that reason, because only the worker can stop work already in flight.
  • steer — the most recent active job on the session key receives a queue.command of type steer carrying the new payload plus fromSessionKey and fromEnqueue: true, and that existing job is returned instead of a new one being created.
Lease expiry is evaluated before the session scan, so a dead worker’s running job cannot swallow a steer.
The idempotency contract. An explicit jobId that already exists returns the existing job unchanged. Re-posting the same enqueue with the same jobId never creates a second job and never disturbs the first. This is the property callers rely on to make an enqueue safely retryable.
maxAttempts defaults to 2 and is clamped to the range 1..8.

​
Payload kinds

Nine payload kinds are accepted: factory.dispatch, factory.run, factory.integration.publish, factory.integration.validate, factory.objective.audit, factory.objective.control, factory.objective.watchdog, factory.task.monitor, factory.task.run Every one of these except factory.objective.watchdog requires an authContext carrying a userId and an organizationId (with optional workspaceId, receiptConnectGatewayUrl, sessionId and source). Each kind also declares a contract: a workerGroup of chat, codex or control, an optional durable workflow, and flags for durable activity, objective scoping, objective reconciliation on a terminal or expired lease, live execution and terminal objective audit. Only factory.task.run sets hasDurableActivity. Actual Resonate routing, however, is decided agent-id first and kind second:
These two do not always agree. A factory.dispatch job posted to agent id factory lands on the chat group even though its kind contract names control as its worker group. Treat this as an ambiguity in the implementation rather than assuming either one is the contract.

​
The trap: only four agent ids have handlers

Exactly four agent ids have registered job handlers: factory, factory-control, factory-monitor, codex
Nothing rejects an unknown agent id at post time. Posting to /agents/<anything-else>/jobs still returns 202, still appends job.enqueued, and still hands you back a job object with a stream and event URLs. The job is then routed to the chat worker group like any other unrecognised agent id, and the worker settles it as failed with the error No handler for agent '<agentId>' — a non-retryable failure, so it is not attempted again. Where the worker roles are not running at all it stays queued instead. Check the agent id against the four above before you conclude a job is slow.
The payload is checked even though the agent id is not: a body whose kind is not one of the nine payload kinds above is rejected before the job is written, and the request returns 500 Server error.

​
Who executes a job

A job runs on whichever process registered the worker function for its group, so nothing executes unless the deployment actually runs those roles; how many jobs each one takes at a time is per role, and RECEIPT_RESONATE_EXECUTE_CONCURRENCY overrides every default. See processes, roles, and routing for the full table.
A misspelled role is not an error. A RECEIPT_PROCESS_ROLE value that is not one of the five role names resolves to all, which registers no Resonate function at all — it runs the maintenance loops below and then parks. Nothing rejects the typo, and jobs aimed at that group simply stay queued.

​
The durable execution path

Step by step:
  1. Enqueue. job.enqueued is appended to jobs/<jobId>. The caller gets back a 202 carrying the job object and an async block with the job id, the stream name, the job and receipt event URLs, and the status.
  2. Dispatch outbox. The queue receipt is the outbox. Enqueue never calls Resonate for the job it just created; it wakes a scanner pass, and that pass asks the driver starter to begin a driver RPC. Startup redrive, interval redrive and enqueue wake-ups therefore share one delivery path.
  3. Driver RPC. The base dispatch key is the job id. If a previous driver or execute promise exists and is terminal, the key advances a delivery generation; that is bounded at 32, after which it errors with resonate delivery recovery exceeded 32 generations for <jobId>. If a promise is still pending, the starter returns created: false with the reason active_delivery and does nothing.
  4. Driver executes. The driver reads the job. A stale active lease ends the pass: the job is failed with stale active Resonate job lost execution; retrying through Resonate driver so the ordinary redrive path can pick it up again. Otherwise it honours abortRequested and begins the worker RPC under a fresh attempt fence with the id <jobId>:attempt:<n>. The driver deliberately does not lease the job — leasing belongs to the worker.
  5. Worker executes and settles. It leases if the job is queued; work that was redelivered rather than leased here instead takes a start-fence heartbeat, and a heartbeat that does not match the current fence ends the attempt as lease_lost. It then checks abortRequested, races execution against the heartbeat loop and the execution timeout, re-reads the job, and completes, fails or cancels it under an attempt fence. Every settlement is wrapped in a storage retry: 5 attempts, 150 ms times attempt backoff, transient storage errors only.
  6. Callback. The driver posts a JSON callback on each of dispatched, completed, failed and canceled, to RECEIPT_RESONATE_CALLBACK_URL / RECEIPT_EVENT_CALLBACK_URL, with a 5 s timeout and an optional Authorization: Bearer header from RECEIPT_CALLBACK_TOKEN.
Workers hold a long poll open against Resonate. That poll is issued with the request timeout disabled and reconnects with a backoff that starts at 1 s and doubles to a 30 s ceiling, so an idle worker keeps its delivery connection instead of dropping it between jobs.

​
Failure statuses in a result

When an attempt does not complete, the worker’s execution result carries one of these statuses: settlement_conflict, stale_attempt, lease_lost, terminal_state, canceled, execution_timeout, failed A failure that is marked non-retryable copies its status onto job.result in the job.failed receipt. A retryable one does not: the receipt records the error and requeues the job, and the status stays in the worker’s own return value.

​
Leases, timeouts and heartbeats

The execution timeout is the lease minus 1 000 ms, with a floor of 5 000 ms. FACTORY_CONTROL_JOB_EXECUTION_TIMEOUT_MS may lower it, for objective control only. The heartbeat cadence is derived rather than configured outright: it is one third of the smaller of the lease and RECEIPT_RESONATE_ACTIVE_STALE_MS (default 600 000 ms), floored at 1 000 ms. RECEIPT_RESONATE_HEARTBEAT_INTERVAL_MS can only lower that derived value, never raise it. Dividing by three gives a worker two renewal opportunities before the fence fires. The driver’s own invocation timeout is the lease plus 60 000 ms, with a floor of 120 000 ms.

​
Four independent recovery loops

The queued-job redrive is guarded by a minimum age (RECEIPT_RESONATE_QUEUED_REDRIVE_MIN_AGE_MS, 30 000 ms) and a cooldown (RECEIPT_RESONATE_QUEUED_REDRIVE_COOLDOWN_MS, 15 000 ms), and its dispatch key is <jobId>:redrive:<attempt>:<updatedAt> so a scanner pass is idempotent for one observed queue state. The outbox redrive is single-flight. RECEIPT_OBJECTIVE_CONTROL_OUTBOX_REDRIVE_TIMEOUT_MS (default 15 000 ms) is an observational deadline only: a pass that runs longer keeps ownership until it settles and logs factory.control_outbox_redrive_slow, so interval ticks coalesce instead of piling duplicate database work on top of a slow pass. The watchdog reconciles an objective for one of three reasons — no_active_objective_work, phase_supersession_has_active_stale_job or active_objective_work_stalled — by enqueuing an objective-control job with the reason reconcile. Sandboxed execution is treated as stale only after 15 minutes, deliberately longer than the UI’s freshness window, because a computer-backed task can be quiet while its workspace syncs. The watchdog can be turned off with RECEIPT_FACTORY_OBJECTIVE_WATCHDOG_ENABLED=0; its timeout defaults to 60 000 ms with a 10 000 ms floor, and its scan limit to 200, clamped to 1..2000.

​
The durable ledger is beside receipts, not above them

A Postgres-backed workflow and activity ledger tracks durable execution alongside the receipt chains. Every durable side effect is best-effort: a durable write failure is logged and does not fail the receipt mutation that has already been appended. Queue commands are a deliberate straight pass-through, so steer and follow-up never depend on workflow signal delivery. The same rule governs Resonate itself. It is a durable delivery mechanism, not ownership state — the receipt chain is the authoritative queue. When delivery is lost after a job was leased, the queue appends a receipt-level job.lease_expired event so the ordinary redrive path can recover it. Next step: see how Factory turns an objective into these jobs.