Skip to content

The server ​

The server is the only authority. This page explains how a change travels from "someone asked for it" to "every device shows it", and the guarantees that hold along the way. If you read one page of these docs, read this one.

What you provide ​

ts
const server = createTwinServer(twin, {
  store: {                                            // one adapter per kind (see "The store adapter")
    note: { load, list, create, update, remove },     //   methods follow the kind's declaration
    // …
    twinstate: twinstateFiles('data/twinstate'),      // Twinstate's own records (see "Twinstate's own records")
  },
  transaction?: (run: (dbTx: DbTx) => Promise<void>) => db.transaction(run), // optional: a turn's writes, atomic (see Transactions)
  handlers: { pinNote: (args, tx) => … },             // required for each named mutation; `{ extend }` for a patch
  onFatal?: (err) => void,                            // default: log and process.exit(1) (see "On disk")
  onUnreadable?: ({ kind, id, reason }) => void,      // a stored record that cannot be read: quarantined, not fatal
  onError?: (err, { mutation, clientId }) => void,    // a handler threw something unexpected: rejected as `error` (default: log)
  onSlowTurn?: (turn, ms) => void,                    // telemetry only; nothing is checked in production
  keyRetention?: '7d',                                // how long keys of server mutations are remembered (see "Server code writes…")
  outbox?: serverOutbox,                              // several machines only: the server's outbox (see "Where a mutation goes"; designed, not built)
});

const stop = server.serve(wsServerTransport({ port: 8080 })); // attach a transport — the server runs without one

A server runs as soon as it is created: boot work (rebuilding a derived value, a migration) can send its mutations before any client connects, and flushes simply reach no one. serve attaches a transport and returns a function that detaches it; more than one may be attached, and fail-stop closes them all. The test kit attaches its in-memory transport the same way.

Every write goes through a turn handle, tx, and every turn is a mutation's: one a client sent, or one your own server code sent with server.mutate. A handler receives the handle. There is no other way to write:

ts
await tx.get(kind, id);                  // read, including this turn's staged changes
await tx.update(kind, id, (draft) => { draft.title = 'Plans' });  // the ONLY way to change a record
const made = await tx.create(kind, id, value);  // create-if-absent: made.created is false if it existed
made.onCreated(() => { … });             // follow-up work that runs once: only after a create that happened, once committed
await tx.remove(kind, id);
tx.emit(twin.events.notice, payload);    // an event (see concepts); the payload typed by the registry
return tx.reject('busy', detail);        // end the turn, rejected with a declared reason (it throws)
tx.idFor(kind, 'label');                 // an id derived from this mutation — the same on every resend
tx.childId(kind, parentId, localPart);   // an id that encodes its parent (see concepts)
tx.afterEveryCommit(() => { … });        // follow-up work that is safe to repeat on a resend (killing a process)

await server.mutate(twin.mutations.touchNote, { id }, { key }); // server code's write: a mutation, through the queue
await server.close();                    // stop: refuse client mutations, finish the queue (see "Shutting down")
await server.get(kind, id);              // read committed state, anywhere; records are DeepReadonly
server.onEvent(twin.events.notice, (p) => { … }); // hear events on the server too

You never broadcast anything yourself, and you never write records behind Twinstate's back: every record a read returns is DeepReadonly, so an assignment to one does not compile, and a function that holds no tx has nothing to write with. That is what lets Twinstate know, exactly, what changed.

tx.reject throws, and is typed never: nothing after it runs, and a handler that catches the throw cannot un-reject its turn. TypeScript does not narrow on a never call through a parameter it inferred, so write return tx.reject(…); the twinstate/reject-return lint rule requires it.

Why a mutation's handle has no plain afterCommit. A mutation can be resent, and a follow-up registered on every run would run again. So the handle offers the two honest choices: made.onCreated(fn), which runs only after a create that actually happened, or afterEveryCommit(fn), whose name says it repeats. Choosing between them is a type, not something to remember.

Server code writes the way clients do: it sends a mutation. A change no client asked for (a job's progress, a webhook, a timer, a streamed token) is a mutation declared from: 'server' (see concepts), sent with server.mutate:

ts
// in the registry
touchNote: mutation(z.strictObject({ id: idOf(note) }), { from: 'server' }),

// in the handlers
touchNote: ({ id }, tx) => tx.update(note, id, (n) => { n.updatedAt = now() }),

// anywhere in server code
await server.mutate(twin.mutations.touchNote, { id });
  • Clients cannot send it, it has no effect, and no policy checks it: it is the server's own work, as Convex's internalMutation is. Its handle has no ctx.
  • key makes a repeat harmless. A mutation sent with a key is applied at most once per key, and its tx.idFor derives from the mutation and its key, so a sender that sends it again names the same records. The key names the message, not what it is about: the same message sent again carries the same key (a webhook's event id, a job's id), while two different events about one payment carry two. Keys are scoped by mutation, so one webhook event may send two mutations under its id. Each is remembered in Twinstate's own records for keyRetention (seven days by default), then forgotten; a handler's own guard covers anything older. How exactly once it is depends on the store, as for clients' mutations (see idempotency): with plain files, a crash between the data and the key can apply it twice, so a keyed mutation that increments wants a transaction. A mutation sent without a key is never deduplicated, and its idFor ids are fresh each time.
  • It enqueues before it returns, so mutations sent one after another, without awaiting, keep their order; the promise resolves once the flush carrying the change has gone out, or rejects with the reason its handler gave. A mutation saved in the server's outbox instead resolves once it is saved: its result belongs to the process that applies it, which reports an unexpected throw through its own onError.
  • It is data, not code: a name and plain arguments. That is what lets it be applied by whichever process holds the partition (see where a mutation goes), and versioned by the lock file like any other mutation.

Where a mutation goes ​

A server instance is one partition, and knows its own state, so server.mutate looks up nothing: where the mutation goes follows from that state alone.

this instance's partition is…server.mutate
openruns it in this process's queue: the only case while the server runs
closed (close() was called; your own work is still finishing)saves it in the server's outbox, and whichever process starts the partition next takes it and applies it, once per key. Twinstate never knows where that is

The server's outbox is the client's outbox on the other side of the wire: there, mutations are saved until the server applies them; here, until their partition's owner does. On one server it needs nothing: it is kept in Twinstate's own records (outbox.<partition>.<seq>), and the next process takes what was saved when it starts the partition. On several machines it is an adapter you configure (outbox: a table with LISTEN/NOTIFY, a Redis stream), because the process saving into it has just lost the partition's lock, so a fenced store would refuse it, and because the new owner is already running and needs to hear about each mutation as it is saved. On one server, a webhook for a partition that is not running yet starts it, through your app's activation, and sends with server.mutate. On several machines, code that holds no server for the partition (a webhook landing on a pod that does not run it) saves straight into the configured outbox with saveToOutbox(outbox, partition, m, args, { key }). See running on many machines.

One way to write means one set of rules: every write is declared, validated, deduplicated, versioned and tested the same way, whoever sent it.

Shared logic that writes takes the handle, so it can only ever run inside a turn:

ts
function touch(tx: Turn, id: Id<'conversation'>) {
  return tx.update(conversation, id, (c) => {
    c.updatedAt = now();
  });
}

Turn is every mutation's handle; one sent by a client also has ctx and connection.

The store adapter ​

Each kind gets a small adapter. Its type is computed from the kind's declaration, and createTwinServer checks the store exactly — no missing method, no extra one (see below):

ts
note: {
  load:   (id) => Promise<Stored<'note'> | undefined>,   // what is on disk, as it was written
  list:   () => Promise<Id<'note'>[]>,                   // every id: an instant kind, read once at startup
  create: (id, stored, dbTx) => Promise<void>,
  update: (id, stored, dbTx) => Promise<void>,            // always the whole record
  updateFields?: (id, patches, dbTx) => Promise<void>,    // optional: write only what changed, when that is safe
  remove: (id, dbTx) => Promise<void>,
},
message: {                                               // ops: ['create'], a child of a transcript
  load:     (id) => Promise<Stored<'message'> | undefined>,
  children: (parentId) => Promise<Id<'message'>[]>,      // one root's children, in order — never "all messages"
  window:   ({ parent, order, first, after }) => Promise<Id<'message'>[]>, // it declares `query`
  create:   (id, stored, dbTx) => Promise<void>,          // no update or remove: the declaration has none,
},                                                       //   and it goes with its parent, whose adapter removes it

Store what you are given. Stored<'note'> is a note in its stored form — { value, meta }, plain JSON, the record as Twinstate wants it saved; how you store it (one JSON document, typed columns plus a meta column, keys in a file) is yours: value is the record's fields, and meta is Twinstate's own bookkeeping — the schema version that wrote it, and fields written by newer code that this code does not know (see changing a schema). meta is branded: an adapter can store it and hand it back, but not build one, so an adapter that maps value onto typed columns and forgets meta does not compile. It is one JSON blob (a meta column, a key in the file). load returns what is on disk, before migration, so value is typed as JSON, not as Note: turning it into a current Note is Twinstate's job. For files that must stay readable without Twinstate — files other code also reads — the flat helpers store meta as reserved top-level keys instead: load: async (id) => flat.read(await readJson(file(id))), update: (id, s) => writeJson(file(id), flat.write(s)). A file with none of those keys is version 1.

update always receives the whole record. An adapter that can write only what changed also provides updateFields, and Twinstate calls it only when the record's meta is unchanged — never after a migration, when the patches alone would leave the stored record half in the old shape.

dbTx is your store's transaction. Every write method's last argument is what your transaction passed to run (below) — your database's handle, not Twinstate's turn handle, which is always tx. Without a transaction, adapters get no such argument.

  • A kind without an adapter, a method the declaration requires or forbids, or an adapter for a memoryOnly or unknown kind, does not compile.memoryOnly kinds are never persisted and so have no key in the store's type at all.

  • A single-value kind (settings) has just load() and update(stored, dbTx), with no id: it always exists, starting from its schema defaults. A defaulted kind keyed by its parent is the same per parent: load returning undefined means "still the default", not absent; its ops are ['update'], and its update writes it the first time too (an upsert), as a single value's does.

  • Three reads, each required only where it is used:

    • list() — every id of the kind. Required for instant kinds, which are all loaded into memory once, at startup (snapshots are then served from memory); optional for any other kind, where having it is what lets server.migrateAll, twinstate check and --prune reach every record.
    • children(parentId) — one root's children, for a child kind.
    • window({ order, filter, first, after }) — one page of a query, for a kind that declares query (with parent, for a child kind). A window whose filter names a memoryOnly field never reaches the adapter: the store never sees those fields, and Twinstate answers it from memory.

    Looking a record up by id is always load — nothing is ever listed to find one record.

  • Removing a parent. When a record with children (or with records keyed by its id) is removed, Twinstate drops them all — and theirs, all the way down — from memory and from every client, and calls only the parent's remove: removing the rest from storage is that adapter's job — a folder deleted, a database cascade. What else should go with a record — links to it, embedded records it used — is yours too (see what your store and handlers own).

  • What types cannot check is whether load returns what create/update wrote, which is a question about your files or tables. twinstate/testing checks it: for every kind, a record written and loaded back is the same record, meta included.

  • Listing and scale. An instant kind is "every record on every device", so startup loads every record, and every snapshot sends every record. That is the cost of the promise, and it is why a kind that grows large stops being instant and becomes a query: a window answered by an index, never a full listing.

Twinstate's own records ​

Twinstate keeps a few small records of its own in your store, so they live and are backed up with your data: which mutations each client has had applied, which keyed server mutations were applied (see idempotency), and, on one server, the mutations saved in the server's outbox after a close(). They go through one adapter under the reserved name twinstate, a plain key-value store of JSON values whose ids Twinstate chooses:

ts
twinstate: {
  load: (id) => db.get('twinstate', id),            // undefined if absent
  save: (id, value, dbTx) => dbTx.put('twinstate', id, value),
  remove: (id, dbTx) => dbTx.delete('twinstate', id),
  list: () => db.ids('twinstate'),                  // read when a partition starts, and once a day to forget old keys
},

or one line: twinstateFiles(dir), a small JSON file per record. The ids always fit the id grammar, so they are safe as file names:

  • client.<clientId> for a client's record, removed when the client retires (a clientId is the client's own random id, refused above 64 characters, so the id stays well inside the grammar's 200);
  • key.<day>.<hash> for a keyed mutation: the UTC day it was applied (20261018), and a hash of the mutation's name and its key. A repeat may come on a later day, so a lookup tries each day within keyRetention — at most eight small loads by default (today and the seven days before), and the answer is cached. A day older than keyRetention is removed, found from list() by id alone, without loading anything;
  • outbox.<partition>.<seq> for a mutation reported after close(), on one server: saved outside any turn, in a transaction call of its own (so the adapter gets its dbTx as usual), and removed by the next process once it has applied it (see where a mutation goes).

Every other write goes through the turn's transaction, like any adapter write.

The transaction ​

Twinstate does not know what a database transaction is. transaction is a function you write that answers one question — how do I run this work inside one transaction of my store? — and Twinstate supplies the work:

ts
transaction: (run: (dbTx: DbTx) => Promise<void>) => db.transaction(run),

At the end of a turn, Twinstate calls it with run (the turn's writes, not yet executed); your store opens a transaction and calls run with its handle, which run passes to every adapter write, twinstate's included; your store commits, or rolls back if a write threw. Without it (plain files), the writes are made one after another, each atomic on its own. Transactions walks through all three setups — plain files, a database, and a hand-written file journal — with the crash cases of each.

Grouping commits across partitions. Within a partition, calls to transaction never overlap: a turn waits for its commit before the next one starts. Calls that do overlap therefore always come from different partitions, and a store shared by several partitions may combine them into one round trip or one commit. If a combined commit fails, every turn in it fails as if its own write had (fail-stop). Often there is nothing to do: Postgres already shares one disk flush between commits that finish together. Turns within one partition are never batched (see design notes).

Open question: synchronous drivers. run is async, which fits drivers whose transactions take an async callback (Drizzle, Postgres clients). Synchronous SQLite drivers (node:sqlite, better-sqlite3) cannot hold a transaction open across an await; transaction may need to accept a synchronous run for them. To be decided when a SQLite store is built.

What the compiler enforces ​

The adapter and handler types are computed from the registry (a mapped type over its kinds and mutations; kind() infers its options as literals, so ops: ['create'] is the type 'create', not string), and the store parameter is exact: every key the store — or any adapter in it — has beyond its type is required to be never. Exactness is computed from the value's type, not from how it was written, so it holds for adapters kept in variables or other files and for class instances alike (private members are not keys, so a class may keep its own state). There is no startup check in production: nothing the types already prove is checked again.

Adapter members are typed as function properties (create: (id, stored, dbTx) => Promise<void>), never as methods: TypeScript checks a method's parameters loosely in both directions, so a create expecting a field the kind lacks, or a children whose argument requires something Twinstate never passes, would compile. Property types are checked strictly under strictFunctionTypes. Twinstate requires strict and checks that it is on, in the types: createTwinServer's options type becomes an error that says "Twinstate needs strict: true" when function parameters are compared loosely or null is assignable to everything.

mistakecaught by
a kind with no adapter; an adapter for a memoryOnly or unknown kindthe compiler
a method ops requires is missing, or one it forbids is presentthe compiler
list, children or window missing where the declaration needs it, or mistypedthe compiler — for queries because the kind declares query (see concepts)
a value of the wrong type in create or update, narrower ones includedthe compiler
an adapter that drops metathe compiler (meta is branded, and load must return it)
an adapter whose dbTx is not the transaction'sthe compiler
no handler for a named mutation, or a handler for an undeclared onethe compiler
a patch mutation's handlerthe compiler: only { extend }, which runs after the patch is applied, in the same turn
a loose tsconfig silently weakening all of the abovethe compiler (the strictness check)
load returns something other than what was writtencheckStore — no type can see your files

A prototype on TypeScript 5.9 compiled the exactness rows as stated, with adapters written inline, kept in variables, and as classes.

Changing state: tracked updates ​

ts
await tx.update(conversation, id, (c) => {
  c.title = title.trim();
  c.titleLocked = true;
});

update is asynchronous — it may first load the record through its adapter — and every tx call that returns a promise must be awaited: @typescript-eslint/no-floating-promises is part of Twinstate's lint preset. It runs your function on a draft of the record (Immer's copy-on-write model): writes are recorded as they happen, and only the objects you touch — and their parents — are copied; everything else keeps its reference. When your function returns, Twinstate has the new value and a list of patches — [{ op: 'replace', path: ['title'], value: 'Plans' }, …] — without comparing anything. Then, in order:

  1. Validate. The new value is checked against the kind's schema and its validate hook. If either rejects, the update throws that rejection — and the turn is rejected, even if the handler catches the error: a caught validation failure cannot commit the rest of the turn.
  2. Stage the new value and its patches in the turn.

When the turn ends, all of its staged changes commit together: each is handed to its kind's adapter — create, update (or updateFields) or remove, inside your transaction if you gave one; memoryOnly fields and kinds are left out — and the turn's applied mutation ids are recorded (see idempotency); then their patches join the next flush. If the handler throws — an update's validation failing, a tx.reject, or any other error — the turn is discarded: its staged record changes, the emits it made and the follow-up work it registered all vanish, and the mutation is rejected with the reason — a declared one, invalid for a failed validation, or error for anything else (see rejections). In memory and on every device, a handler that creates a note and its body creates both or neither. Rollback is free because records are immutable.

On disk, a turn is as atomic as the store makes it. With a transaction, it is atomic. With plain files, each record is written atomically (write-then-rename), but a turn that spans several files can be cut between them. If a write fails, the server stops (fail-stop): it sends no reply for the turn's mutations (so their entries stay in every outbox and are resent after the restart — a failed write is never reported as a rejection, which would drop it), stops running further turns, logs loudly, and closes its connections; then calls onFatal, which by default exits the process (process.exit(1)) so that its supervisor restarts it — nothing depends on the host remembering to. Clients show themselves disconnected meanwhile. Disk is the truth, and after a partial write memory can no longer be trusted to match it; a restart reloads every record from disk. Persist failures (disk full, I/O errors) are rare and serious enough that stopping is safer than guessing. File adapters should keep what belongs together in one file.

Handlers must be safe to run again over their own partial result. That is what repairs a turn cut between files. Its mutation got no reply, so it is resent after the restart; a handler written as a sequence of create-if-absent and idempotent updates finishes the job — the note already exists, so its create does nothing, and the body's create writes the missing record. checkResend checks it, by failing a write in the middle of every fixture's turn. A mutation from server code has no outbox: unless its sender sends it again (a redelivered webhook, a retried job, with the same key), a turn cut between files stays partial until something writes those records again; keep one that must write several records together inside one record, or give the store a transaction.

Consequences worth knowing:

  • A change to a record cannot be missed. There is no other way to write a record, so there is no "remember to mark this dirty" call to forget. (State you keep outside records is simply not synced — keep synced state in records.)
  • Writes happen only inside a turn, by construction: they are methods of tx, and only a turn has one. Code that has no tx cannot write, and it does not compile if it tries.
  • Server-side derived values are explicit fields. A field such as "is a job running" is not computed at read time; the code that starts the job sets it (tx.update(job, id, (j) => { j.running = true })). A value that changes by itself — anything computed from the clock — does not belong in a record: send the timestamp and derive on the client.
  • Records are read-only by type (DeepReadonly), so mutating one outside update does not compile. Development and tests also freeze them (Immer's auto-freeze) to catch casts; production skips the freeze, which roughly doubles the cost of a large update in Immer's own benchmarks.
  • create is create-if-absent. It changes nothing when the record already exists (made.created is false), and follow-up work hangs off it with made.onCreated(fn), so work belonging to a create runs once however often its mutation is resent.

Cost ​

Measured by Immer: proxied updates run about 2–3× slower than hand-written immutable updates, which is negligible for writes made by people and agents. Memory stays close to 1×: only the changed path is duplicated, and "what clients were last sent" is simply the previous immutable value — no second copy of the data is kept for comparison. The flush compares nothing; its cost grows with the number of changes, not with the size of the data.

The queue, and one turn ​

Every change goes through one queue: client mutations, and every mutation your own code sends — background jobs, timers, events from a subprocess, a streaming reply. The queue runs one item at a time:

 ┌───────────────────── one turn ─────────────────────┐
 │ 1. take the next item from the queue                │  ← the WAIT rule: one at a time
 │ 2. run its handler, awaits included; its updates     │
 │    validate, then all of them persist together when  │
 │    it ends — or none do                              │
 └─────────────────────────────────────────────────────┘
   … further turns in the same tick …
 ┌──────────────────── the flush ─────────────────────┐
 │ 3. one broadcast of the patches recorded since the   │
 │    previous flush, per topic — each topic's message  │
 │    ending with the events emitted on it              │
 │ 4. reply "done" to each client whose mutation was    │  ← the ORDER rule: data, then done
 │    in that flush                                     │
 └─────────────────────────────────────────────────────┘

"Nothing else changes state" — without synchronous code ​

Handlers are asynchronous: when one awaits a disk write, Node's event loop keeps running, and an incoming frame or a subprocess event can arrive. Those do run — but they cannot apply anything, because they hold no tx. They enqueue their change with server.mutate, as a later item.

  • The queue is a promise chain: item n+1 starts only after item n's promise has resolved, all of its awaits included.
  • A tx is only good for its own turn. Work a handler starts without awaiting — a background job, a subprocess callback, follow-up work — must not capture the turn's tx: the twinstate/no-tx-escape lint rule flags a tx used inside a callback passed to onCreated, afterEveryCommit, setTimeout, .then or an event listener. Follow-up work that needs an id computes it inside the turn (tx.idFor) and captures the value, not the handle. The handle also carries a flag cleared when its turn ends, and a write through a closed handle throws — in production too. It is the one check production keeps for a programmer's mistake, because skipping it would write outside any turn, silently, and it costs a single branch. No AsyncLocalStorage, and no per-write lookup, is needed.
  • Never wait for a turn from inside one. Awaiting server.mutate inside a turn would wait for itself, so the twinstate/no-mutate-in-turn lint rule forbids calling server.mutate from a handler — follow-up work (onCreated, afterEveryCommit) runs after the turn and may call it freely, and so may background work outside any turn. A follow-up turn goes to the back of the queue, so it re-reads what it needs with tx.get: anything that must happen with nothing in between belongs in the same turn. server.mutate resolves only after the flush that carries its change.
  • A slow or stuck turn fails a test: the kit fails any handler that takes over 100ms, and any that never finishes. Production runs no watchdog; onSlowTurn is available if you want the telemetry.

This is the same model as Cloudflare Durable Objects' input gates: one writer per unit of state, requests held while it awaits storage, handlers kept fast.

Why one queue is enough ​

With handlers that do a short durable write, a turn takes milliseconds. Three devices acting "at once" each wait a few milliseconds for the others. Durable Objects serve around a thousand requests per second per object this way; Replicache's global-version strategy documents about fifty 20ms pushes per second through a single lock. A handful of devices come nowhere near either.

What one queue cannot tolerate is a slow handler — which is why the test kit fails one over 100ms ("move long work out of the handler: record the intent, then report back with a mutation"). If one queue is ever measurably too slow, the answer is a coarse partition (one queue per natural unit, like Replicache's "spaces") — never per-record locking.

Flush ​

A flush is how data leaves the server. It runs after the turns of a tick, at most once every 16ms, and it sends the patches recorded since the previous flush — nothing is re-read or compared.

Turns that share a flush share only the broadcast. Each turn commits on its own when it ends — its changes, and its mutation ids, are in the store before the next turn starts — and a turn that fails discards only its own changes. So an unrelated mutation that fails in the same tick as yours cannot revert it. The one event that affects every turn in a flush is fail-stop: if any turn's write fails, the server stops before broadcasting, so the turns already committed in that tick are durable but unannounced. Their changes are not lost: their mutations get no reply, stay in their outboxes, and are resent after the restart, where the durable record of applied ids answers each resend ok without running it again. What does not survive, as anywhere a server stops, is the events those turns emitted: they live in the rings, in memory only, so a restart drops them.

 for every record that received patches:
   to every topic that holds it:
     text field that only grew (appendable) → { append: suffix, at: oldLength }
     otherwise                              → { patch: [...] }  (field-level; serverOnly fields dropped)
     removed                                → { del }
   when a patch adds a ref to an embedded record it lacks      → { set } that record too
   when a child is created under a topic's root                 → { set } it there
   when the last ref to an embedded record goes                → { del } it from that topic
 → per topic, one message with the next revision
 → each topic's message carries, after its ops, the events emitted on it by those turns
   (kept in its ring with the changes; a resume replays them only for a subscribe with
   `events: true` — a reconnect — and a reset drops them)
  • Only what changed is sent, field by field. A rename sends the title, not the whole conversation. Big records stay cheap to change.
  • Bursts collapse — safely. Patches are sent in order and never merged (they address nested paths and array positions, so dropping an earlier one would corrupt the result); the only coalescing is that appends to the same text within one flush window are joined into one append, and a record whose patches in one flush outweigh its whole value may be sent as one set.
  • Snapshots are the one full read. A new subscriber (or a reset) gets the current records of its topic, from the in-memory store — taken at a flush boundary, between turns, so a device that connects mid-stream never sees a half-applied turn.

A subscription to a lazy root that does not exist yet is allowed — the client may have created it offline, with the create still in flight. Its reset says absent: true (not a deletion), the subscription stays open, and the first flush after the record appears sends it.

Memory ​

instant records stay loaded. Lazy roots, their children and the embedded records they ref are loaded through their adapters when a topic first needs them and evicted when no topic holds them (after the entries' grace period) — so an app's memory follows what is open, not what exists. Except records whose memoryOnly fields differ from their defaults (reloading them from disk would reset those fields — a turn marked running would look idle), and records of memoryOnly kinds, which exist only in memory and stay until removed. A record whose memory-only fields are all back at their defaults is evicted like any other. Anything that grows without bound — a thread's messages — is many records, never one growing value, so neither memory nor a store adapter ever handles a whole history at once; a store adapter can map create of such a record to an append in an append-only file.

Queries ​

A query subscribes to a live, ordered window over a kind instead of every record. The kind's adapter answers it through its window:

ts
activity.window({
  // the activity adapter's window:
  order: 'at desc',
  filter: { archived: false },
  first: 50,
  after: cursor,
});
// → the ids of the next 50 matching records after the cursor, in order
  • The cursor is the position of the last item the client holds — its sort key and id, e.g. (10:01, n7) — encoded as an opaque string. "After" means strictly after it in the order, with the id breaking ties. A database adapter turns this into WHERE (updatedAt, id) < (…) ORDER BY updatedAt DESC, id DESC LIMIT 50; a file adapter does the same comparison over records in memory.
  • The server keeps the window live. When a turn changes a record of the queried kind, the flush checks it against each active window on that kind: a record that now matches and sorts inside the window enters (a set, with its place), one that no longer matches leaves (a del from the window), and one whose sort key changed moves. The window's size is kept, so an item pushed past the end leaves too.
  • Load more widens the window: the client sends a window message for the same subscription (see the protocol), the server asks the adapter for the next items after the current end (after: endCursor) and sends them, and the wider window is kept live the same way. A query is identified by its kind, order and filter (and parent, for a child kind); the window's size is only a property of the subscription.
  • A child kind's window is the same, over one parent's children: { parent, order, first, after } — a long transcript read page by page.
  • The cost is one comparison per changed record per active window on its kind — changes, not data, again.

The ordering guarantees ​

Each client has one ordered connection (see the protocol). Over it, three things are guaranteed:

  1. Every client receives changes in the server's order. Whatever order two devices sent their mutations in, the queue decides one order, and every device sees that order.
  2. A mutation's reply arrives after the broadcast that contains its result. When you click, two messages come back: the data ("note x is pinned", sent to every device, which is what changes the screen) and the done ("your pin succeeded", sent to you, which ends the control's pending state). You always receive the data first, so when your pending preview is dropped, the real value is already on screen.
  3. Nothing is broadcast before it is durable, and nothing is confirmed before it is broadcast: persist → broadcast → reply.

Other updates may arrive before, between, or after your data and your done. Only your own done never overtakes your own data.

Worked example: clicking during a stream ​

A reply is streaming; you click a button that starts a job.

 server queue (one at a time)            what your device receives, in order
 ────────────────────────────            ───────────────────────────────────
 commit: append "…the fix is"            append "…the fix is"
 commit: append " in the"                append " in the"
 mutation: startJob → job running  ┐     one flush: append " index" + job {running}
 commit: append " index"           ┘
                                         done(startJob)        ← pending ends; "running" already shown
 commit: append " model."                append " model."
 … the job works outside the queue …
 commit: job done                        job {done, result}

The stream never pauses for your click, and your click never waits for the stream (beyond microseconds: each streaming commit is a tiny in-memory update).

Worked example: two devices at once ​

 queue:   [ Mac: pin x ]  [ phone: rename y ]

 Mac's turn     apply pin x      → broadcast "x pinned" to both → done to Mac
 phone's turn   apply rename y   → broadcast "y renamed" to both → done to phone
                (it waited during the Mac's turn: a few milliseconds)

If both land in the same flush window they share one broadcast, then each gets its done.

Worked example: both edit the same field ​

 Mac sets title "A"; phone sets title "B".

 B applied first:  broadcast B, done(B), broadcast A, done(A)   → everyone ends at "A"
 A applied first:  broadcast A, done(A), broadcast B, done(B)   → everyone ends at "B"

Every device ends on whichever write the server applied last. The Mac's pending "A" stays visible on the Mac until its done arrives (see the client).

Streaming while the queue is busy ​

A subprocess that streams output does not know or care that the queue is busy. Its events keep arriving; each becomes a queue item; while a turn runs they wait behind it, and when it ends they are applied at once and ride the next flush, merged into one append:

 queue:  [ Mac: pin x (running) ] [append " is"] [append " in"] [append " the"]
 next broadcast:  x pinned, { append: " is in the" }

There is no backpressure on the producer. At ~50 tokens per second and a few milliseconds per handler, a mutation delays one or two tokens by a few milliseconds. Enqueue synchronously, in arrival order: a producer whose events must stay in order calls server.mutate from its event callback without awaiting anything first, and the FIFO queue keeps them in order:

ts
// in the registry's mutations
appendReply: mutation(z.strictObject({ id: idOf(reply), text: z.string() }), { from: 'server' }),

// in the server's handlers
appendReply: ({ id, text }, tx) => tx.update(reply, id, (r) => { r.text += text }),
ts
session.on('delta', (text) => server.mutate(twin.mutations.appendReply, { id, text })); // the producer

Async and long-running work ​

A handler is a short, durable write. Anything longer — a payment, an agent run, an export, a call to another service — must not hold the queue. The pattern: record the intent on the record, return, and report the result back as a mutation. It is Akka's pipeToSelf: async work never touches the state directly; its result comes back through the queue like any other change.

ts
// cart: { coupon: z.string().nullable(), checking: memoryOnly(z.string().nullable().default(null)), … }
applyCoupon: async ({ cartId, code }, tx) => {
  await tx.update(cart, cartId, (d) => { d.checking = code });   // the intent, on the record
  tx.afterEveryCommit(() => checkCoupon(cartId, code));          // the work, after the commit
},

couponChecked: async ({ cartId, code, ok }, tx) => {          // from: 'server'
  await tx.update(cart, cartId, (d) => {
    if (d.checking !== code) return;                          // another code was typed, or a restart cleared it
    d.checking = null;
    if (ok) d.coupon = code;
  });
},

async function checkCoupon(cartId: Id<'cart'>, code: string) {
  const ok = await coupons.validate(code);                    // outside the queue, as slow as it likes
  await server.mutate(twin.mutations.couponChecked, { cartId, code, ok });
}

The handler, the follow-up and the result sit side by side in one file, in plain code. The state is a field of the record the work is about, so every device shows "checking…", then the outcome. "Checking" is this process's current activity, so it is memory-only: if the process stops mid-check, it is simply gone, the coupon was never applied, and the person applies it again, as after a failed POST. A mutation's pending covers only "has the server accepted it"; after that, the work is ordinary state.

Choosing how the work survives a crash ​

Follow-up work runs in the server's process, so a crash or a kill loses whatever was running. How much that matters decides the pattern — the same three answers a POST backend has:

if the work is lostthe patternwhat you write
it does not matter much (a coupon check, a preview)a plain follow-up, as above: what is in progress is memory-only (see memory-only state), so a crash clears it and the person simply tries againnothing more
it must happen, and another system is the source of truth (a payment)that system first, its webhook as the durable path. The provider knows about the work before anything can be lost, and redelivers its result until you accept it; a follow-up, if any, is only the fast paththe webhook, which the provider requires anyway
it must happen, and nothing else remembers it (an export, an agent run)a durable job queue (pg-boss, Graphile Worker, BullMQ, Inngest, Trigger.dev): enqueued in the turn's own transaction where the queue lives in your database, retried, drained on deploy, reporting back with server.mutatethe job, in the queue's API, and its enqueue in the job record's adapter

Twinstate builds no job system: a durable queue is a solved problem. The job is a record, and its kind's adapter enqueues it in the same transaction that stores it — a handler never sees dbTx, an adapter does (see transactions):

ts
startExport: async ({ exportId, noteId }, tx) => {
  await tx.create(exportJob, exportId, { noteId, status: 'running' });
},

// the exportJob adapter: the record and the job commit together, or neither does
exportJob: {
  create: async (id, stored, dbTx) => {
    await dbTx.insert(exportJobs).values({ id, doc: stored });
    await boss.send('export', { exportId: id }, { db: pgBossDb(dbTx) });   // pg-boss, inside your transaction
  },
  // load, update, remove …
},

boss.work('export', async ([job]) => {                       // one worker per job, retries, drains on deploy
  const url = await exporter.run(job.data.exportId);
  await server.mutate(twin.mutations.exportFinished, { exportId: job.data.exportId, url },
                      { key: `export:${job.data.exportId}` });
});

Worked example: a payment ​

Stripe is told first, so no crash can lose the payment; its webhook settles it.

ts
// client
const paymentId = newId(payment);
const { clientSecret } = await createPaymentIntent({ paymentId, orderId });  // your endpoint: Stripe now knows
pay({ paymentId, orderId });                       // "Processing…" on every device at once
await stripe.confirmPayment({ clientSecret });     // Stripe.js, 3-D Secure included

// server
app.post('/payment-intents', async (req) => {      // keyed: a retried request returns the same intent
  const pi = await stripe.paymentIntents.create(
    { amount, currency, metadata: { paymentId: req.body.paymentId, orderId: req.body.orderId } },
    { idempotencyKey: req.body.paymentId });
  return { clientSecret: pi.client_secret };
});

pay: async ({ paymentId, orderId }, tx) => {
  await tx.create(payment, paymentId, { orderId, status: 'paying', startedAt: now() });
},

app.post('/stripe-webhook', async (req) => {       // retried by Stripe until it is accepted
  const event = verifiedEvent(req);
  const outcome = event.type === 'payment_intent.succeeded' ? 'paid'
    : event.type === 'payment_intent.payment_failed' ? 'failed' : null;
  const pi = event.data.object;
  if (!outcome || !pi.metadata.paymentId) return;  // not one of ours: accepted, nothing to do
  await server.mutate(twin.mutations.paymentSettled,
    { paymentId: pi.metadata.paymentId, orderId: pi.metadata.orderId, outcome },
    { key: event.id });                            // the event's own id: a redelivery applies once
});

paymentSettled: async ({ paymentId, orderId, outcome }, tx) => {   // from: 'server'; outcome: 'paid' | 'failed'
  await tx.create(payment, paymentId, { orderId, status: 'paying', startedAt: now() }); // the webhook may beat `pay`
  await tx.update(payment, paymentId, (p) => {
    if (p.status !== 'paid') p.status = outcome;   // paid is final; a failed attempt may still succeed later
  });
},
 tap Pay ─► Stripe knows (the intent) ─► pay p1: {paying} on every device ─► Stripe.js confirms
 Stripe ─► webhook ─► paymentSettled (key: the event's id) ─► {paid} on every device
 a crash anywhere: Stripe still has the payment, and its webhook still comes

What the person sees is worked out on the client from startedAt: a moment of "Processing…", then "Still confirming…" while the person is free to leave, with the result arriving on every device (and as a push or an email, if the app sends one). Recovery is never the person's job: if automation has not settled it after a long while, the screen says so and points to support — not a retry button that could look like paying twice.

Safe to repeat ​

Work that leaves the queue is delivered at least once everywhere — Stripe says fulfilment "might be called multiple times, possibly concurrently"; job queues and webhooks redeliver. Most of it is safe to repeat by construction; the app writes two things, both at the edge with the outside world:

stepsafe to repeat becauseyou write
a client's mutation, resentapplied once, by its idnothing
a createcreate-if-absent, with an id chosen by the clientnothing
work that belongs to a createonCreated runs oncenothing
a server mutation sent again with its keyapplied once per key (exactly, with a transaction)the key: the message's own identity (a webhook's event id, a job's id)
a call to another serviceonly if the call is keyedits idempotency key (paymentId), or a flag on the record (receiptSent)
a result arriving late, or out of orderonly if it checks the state it findsa guard (if (p.status !== 'paid')), or a field policy on before and after that allows only forward transitions

A service with no idempotency key and no way to ask "did this happen?" cannot be made exactly-once by anyone: choose at most once (record "sending" first, and skip on a repeat) or at least once (send again), and say which. checkFollowUps runs every afterEveryCommit follow-up twice and concurrently, and applies every server mutation sent with a key twice and in reverse order, so a missing key or guard fails a test.

  • The client chooses exportId or paymentId (its argument schema is idOf(…)), and create-if-absent makes it the idempotency key: a resend after an app kill finds the record and starts nothing twice.
  • Cancelling is a mutation that sets status: 'cancelled'; a result arriving afterwards meets the guard, and the handler decides what that means (a payment made anyway is refunded).
  • Work that finishes after close() — during any deploy, on one server or several — still reports with server.mutate, which then saves it in the server's outbox for the next process (see where a mutation goes).

Shutting down ​

server.close() stops the partition at once; the async work your app started is the app's to wait for, as in any Node server (Fastify's onClose hooks, Terminus' beforeShutdown):

ts
process.on('SIGTERM', async () => {
  // SIGINT too, for Ctrl-C on a laptop
  await server.close(); // at once: never waits on your work
  await myWork.finish(25_000); // your own async work, however you track it
  process.exit(0);
});
  1. Client mutations are refused from now on: they stay in their outboxes, and are sent to the next process.
  2. The queue finishes its last turns and flush, and the sockets close.
  3. Your work keeps running. A result it reports with server.mutate now is saved in the server's outbox (where a mutation goes).
  4. The next process starts with a new epoch, applies what was saved, once per key, and clients reset and resend.

Twinstate does not track your follow-up work: a framework that waits on background promises (Vercel's waitUntil, Cloudflare's ctx.waitUntil) does so because its platform owns the process's lifetime, and here your app does. Work your app abandons, or a kill cuts off, is the crash case of async work.

Two processes must never hold a partition at once. Where the host starts the new process before the old one has stopped (Render, Heroku, a Kubernetes rolling update), switch the overlap off if the host allows it, or follow the several-machines recipe: a lock per partition, so the new process waits to start the partition until the old one has closed and released it, and a configured outbox, since the new process is then already running when the old one saves its results.

Set your host's grace period above how long you wait for your own work. Defaults are short, and some are zero:

hostold and new overlap?time between SIGTERM and SIGKILL
a laptop, a VMno: stop, then startyours
Railwaynot with a volume ("prevents multiple deployments from being active and mounted")RAILWAY_DEPLOYMENT_DRAINING_SECONDS, 0 by default
Fly.ionot by default (rolling); bluegreen cannot use volumeskill_timeout, 5s by default, at most 300s
Renderyes, without a persistent disk30s by default, up to 300s
Herokubriefly; Preboot keeps both up for up to three minutes30s
Kubernetesyes, in a rolling updateterminationGracePeriodSeconds, 30s by default
Durable Objectsthe platform keeps the old code until the object is idle, up to 300snot yours: close is not used, and work that must happen goes to Cloudflare Queues or Workflows
Vercel Functionsnot a Twinstate host: connections are pinned to an instance and capped (5 minutes by default), and instances come and go, so a partition cannot have one owner—

On several machines (designed, not built), the same shutdown lets the partition move at once: once close() returns and its lock is released, the next owner starts while this process finishes its work, and the results are saved in the configured outbox, which tells the running owner. See running on many machines.

Durable and memory-only state ​

"Persist before broadcast" applies to durable state. Some state is deliberately memory-only, declared with memoryOnly(schema) on a field (or memoryOnly: true on a whole kind): sent like anything else, never written to the store, and after a restart it starts from its schema default. It is for two things only, and should be rare:

  • derived values, recomputable from durable records but costly to compute every time (a count, a summary): a new process rebuilds them;
  • this process's current activity — "a turn is active", "a reply is streaming", "a prompt is waiting" — which a new process is not doing, so it starts empty: nothing can come back from disk claiming a turn is running when none is.

Nothing in memory crosses processes. A restart, a deploy or a move to another machine starts from the store, with a new epoch. A reply streamed into memory is lost if its process stops mid-stream, so the application writes what must survive durably as it settles (each completed block, the finished job). Persisting every streamed token instead would make each a disk write and slow the queue for everyone.

Revisions, epochs, rings ​

Streaming changes to clients uses three pieces of in-memory state:

  • rev — one counter for the whole server. Every flushed change gets the next number, whatever topic it belongs to, so changes can be ordered across topics.
  • Rings — each topic keeps its own recent changes (by default its last 1,000, at most 1MB and ten minutes old; tunable per kind, see scalability), with their global numbers, so a streaming topic cannot push another topic's history out.
  • epoch — an id chosen when the server boots: its boot time, then a random suffix. Revisions are only ever compared within one epoch; the time prefix only lets a client's persisted cache keep the newer of two saved copies.

A client keeps one position per topic — the (epoch, rev) of the last message it applied there — and sends it when it subscribes:

 same epoch, and the topic's ring still reaches back to the position → replay what was missed
                                                                        (its events too, if `events: true`)
 anything else (new epoch, position too old, none)                → reset: a snapshot of that topic

A restart, an upgrade, a restored backup, a partition moving to another machine or a client pointed at a different server all take the same path: a new epoch, so every client resets each of its topics once. Nothing about streaming is persisted or carried from one process to the next: a reset re-sends from the store, which is right however the store got there (see design notes). Without the epoch, a restarted server counting from 0 would have every client whose position says rev: 4815 ignore every message up to 4815 — silently frozen.

Idempotency ​

Clients resend unconfirmed mutations — after a reconnect, after the app was killed before its reply arrived, after a server restart, or when a person releases an entry that was held for a day. A resend must never apply twice.

  • Deduplication by mutation id, remembered durably. Each client numbers its mutations in order (clientId, mutationId). The server records each id when it is received, so a resend that arrives while the original is still queued is attached to it and answered with its result, never run twice. Once a turn runs, the server keeps, on disk, the highest id applied per client and every id whose result was not ok (rejections are rare, so this stays small; every other id at or below the mark was ok), and answers a resend with the original result. That record is one of Twinstate's own records, one per client (client.<clientId>, with the clientId checked against the id grammar first, so a client cannot choose a path), so a turn rewrites only its own client's record, however many clients there have been. It is read when a client first sends, written after the turn's data and inside the same transaction when there is one, and removed when the client retires. Like every reply, the answer goes out at the end of the next flush, after that flush's data.
  • How exact it is depends on the store. With a transaction, data and ids commit together, and a mutation applies at most once, across restarts and held entries alike. With plain files, a crash can fall between the data write and the id write; the data is written first, so the only effect is that the resend runs again. For updates and merges that is harmless (the next rule). For a create it is harmless too — unless another device removed the record before the resend arrived (a resend comes when the client reconnects, which can be hours later, or days for a held entry), in which case the create brings it back: the one residual case, and why a store that can have a transaction should. A missing or corrupt record is read as empty, never fatal.
  • Handlers are safe to re-run, and creates are idempotent by construction. In a mutation's turn, tx.create needs a branded Id<K>, and that turn's handle cannot mint a fresh one: ids come from the mutation's arguments (the client chose them, typed idOf(kind)), from records read, from tx.idFor(kind, 'label') — derived from the mutation's own identity, so the same on every resend and different for every other mutation — or from tx.childId. parseId is not available on the server at all. So a resent create always names the same record, and create-if-absent does nothing the second time; merges are idempotent (see concepts). A mutation sent by server code takes its ids the same ways; its idFor derives from the mutation and its key, so sending it again names the same records.
  • Follow-up work is chosen by type. A mutation's handle has made.onCreated for work that belongs to a create, and afterEveryCommit for work that is safe to repeat — no plain afterCommit that would quietly run again on a resend.
  • Clients are retired, not forgotten. Every tab has its own clientId, so the record grows by one short row per tab ever opened. A client whose outbox is empty and deleted sends retire, and its row is dropped. Rows are never dropped by age: an entry can be held for any length of time and released later, and its row is what answers it. A row is a few dozen bytes, so even thousands of never-retired tabs (closed by a crash, never adopted) stay small.

What the server rejects before a handler runs ​

  • a mutation whose arguments fail its schema — invalid. Arguments are parsed strictly: an unknown key is invalid, never stripped, so no typed text is ever dropped by a parse;
  • a patch the registry does not declare, or one naming a field it does not allow;
  • anything referring to a kind or mutation the registry does not declare — a protocolError, and the connection is closed (see the protocol).

A client the server cannot serve never gets this far: the hello handshake answers it incompatible (see running clients), so every mutation the server runs has arguments it accepts.

A handler can also reject with return tx.reject(reason, detail?). Reasons are declared on the mutation (mutation(args, { rejects: ['busy', 'stale'] })), so tx.reject accepts only those, and the client's useMutation(…).error is typed as that union plus invalid — the reason every schema or validate failure gives — unavailable, for a record that is quarantined (see changing a schema), and error. Nothing of a rejected turn is persisted.

An unexpected throw is error. A handler that throws anything other than a declared rejection — a bug, a store read that failed — has its turn discarded like any other rejection, and the mutation is rejected as error, with no detail: the message stays on the server, reported through onError, so internals never reach a client. It is not retried: a bug that threw once would throw again, and a retried entry would block every one behind it. The client treats it as any rejection — the effect stops, and the control shows it, or the outbox UI does, with the entry's content to copy. (A store write that fails is different: that is fail-stop, above.)