Skip to content

Changing a schema ​

A kind's schema will change as an application grows. The schema is the contract with every client, so Twinstate owns its versions: which shape each version has, and how a record of one version becomes the next. Where and how records are stored stays your store's: it keeps the version next to the value and hands both back. This page says what Twinstate does about old data, when you have to write a migration — and why the best migration is the one you avoid.

Prefer changes that need none ​

A migration is code that must keep working for as long as any record in the old shape might still exist. Until every stored record has been rewritten (below), each step stays in the codebase, is typed against a shape nobody uses any more, and has to be read by everyone who touches that kind. Most schema changes can be made additive instead, and additive changes need nothing at all:

  • add a field with a default (tags: z.array(z.string()).default([])) rather than a required one;
  • add a new field and stop writing the old one, rather than renaming it;
  • widen a type (string → string | null, a new enum value the code already handles) rather than changing it — no step, though not rollback-safe: after a rollback, every record written since is unreadable to the old code, whatever its values (see the table);
  • add a new kind rather than restructuring an old one.

Reach for a migration when the additive version would leave the model worse: two fields meaning the same thing, a type that no longer says what is true.

Where old data lives ​

Only one place ever needs a migration:

where old data liveswhat happens
records on the server's disk, written by older codemigrated when loaded, by the steps below — the only migration there is
clients running an older bundlenever migrated: an older client stays connected for as long as every change since its version is one it can read, and is told incompatible otherwise (see running clients)
outbox entries written by an older bundlesent by the old client while it is compatible; after an upgrade, checked strictly by the new bundle before sending, and an entry that no longer fits is held as "made by an older version", shown with its content to copy or discard (see the client)
the persisted cacheevery saved topic carries a cache key (its kinds' versions, stored fields and format); a tab uses it only under the same key, and otherwise deletes it and refetches the topic (see the client)

What needs a step, and what doesn't ​

Two different questions decide what a change costs:

  • Can the new code read the old records? If not, it needs a step — a function that turns an old record into a new one.
  • Can the old code read the new records? If not, the change is not rollback-safe: after a rollback, records written since become unreadable (quarantined, below) until the new code returns.
changestep needed?old code can still read new records?
a new field with a default (.nullable().default(null) for none)noyes — the field is kept as unknown, and survives its writes
a new kindnoyes — old code ignores it
loosening a type (string → string | null, a new enum value)nono — every record written since, whatever its values: readability is decided per version, not per record (checking each written value against every older schema would cost every write)
removing a fieldno step, but the kind must say dropped: ['field'], because that data is deleted — explicitly, whatever zod's unknown-key mode — and the name is retired for goodonly if the field had a default in the old code
adding a required field with no defaultyes (old records lack it)yes
changing a type, renaming, restructuringyes — and it does not compile until the step existsno

A change in what a value means with the same type — status: 'done' now also covering what 'archived' used to — is invisible to any tool. Treat it as a change of type: add a step that rewrites the old values, or add a new field.

How it works ​

  1. Every lock is a version. twinstate lock writes a checked-in file, twinstate.lock.ts: for each kind, its stored shape at every version, as types, and its retired field names; the same versions for every mutation's arguments and every event's payload; and a numbered revision per run, mapping each of them — and each query's orders and filters — to its version then (see running clients). It is regenerated only on purpose, so every schema change shows up in review as a diff to it. Two branches that each ran it conflict on the file when merged, like two migrations taking the same number: resolve it by running twinstate lock again on the merged code, never by hand-picking one side.
  2. Every field's handling is recorded, and confirmed. The lock also records, per kind and version, each field's effective settings, its defaults included: who sees it (clients or server), how long it lives (durable or memory), how its changes travel (replace or suffix), and its default value. A field nobody marked is still a decision: twinstate lock lists every new or changed field with its settings — "reply.usage: sent to clients, durable, replaced whole" — and does not write the lock until each is confirmed (--confirm reply.usage), so forgetting serverOnly is caught before it ships, and the decision is a line someone reads in review. CI fails when the registry and the lock disagree.
  3. The compiler asks the first question. If records of the latest locked version would not parse under the new schema (after its dropped fields are deleted), or a field disappeared without dropped, the kind does not compile until it declares a step for the new version.
  4. The lock answers the second. For each version it records the oldest version that can read its records — the smallest k such that every version from k up to it accepts them, checked against every older version, not just the one before.
  5. A retired name is never reused. A field that disappears — dropped, or removed by a step such as a rename — joins the kind's retired names, which the lock keeps for good (pruning old versions does not forget them). Declaring a field with a retired name does not compile. So a name always means one thing, and newer code, when it loads a record, deletes every retired field it finds in it — safely, whoever wrote it, because no current field can have that name.
  6. Every stored record says what shape it is in. Twinstate keeps two stamps in the record's meta (see the store adapter): _v, the version whose shape the record is in — every step up to it has run — and _c, the oldest version able to read it. A record with no stamps is version 1 — which is how data that existed before Twinstate (an application's existing files, say) enters the scheme. The stamps never go down: older code that writes a record it loaded with a newer _v (readable, because its _c allows it) keeps them. So the newer code never runs a step twice over data that is already migrated.
  7. Unknown fields survive. A field in a stored record that the running code does not know — written by newer code — is kept aside in meta on load, never sent to clients, and written back unchanged with the record. So older code can read, and change, records newer code wrote, without deleting what it cannot see. If the older code writes back a field that the newer code has retired, the newer code deletes it on its next load: an edit made during a rollback to a field the newer code no longer has is lost — the price of rolling back across that change.
  8. A step turns one version into the next, and its parameter is typed from the lock, so it cannot be wrong about what is on disk:
ts
kind('note', noteFields, {
  id: true,
  load: 'instant',
  ops: ['create', 'update', 'remove'],
  migrate: {
    2: (v1) => ({ ...v1, pinned: v1.pinned ? 'yes' : 'no' }), // v1 → v2: a boolean became an enum
    4: ({ labels, ...v3 }) => ({ ...v3, tags: labels }), // v3 → v4: labels renamed to tags (v3 only added a field)
  },
});
  1. Loading runs the steps. Twinstate reads a record's _v, applies each step above it up to the current version, deletes retired fields, then validates the result against the current schema. instant kinds are all loaded at boot, so they are migrated in memory at once; lazy ones when first opened.
  2. A record whose stored form changed is written whole. The first write to a record that loading changed — a step ran, or a retired field was deleted, so its meta differs from what is on disk — goes through the adapter's update with the whole record, never updateFields: the handler's patches alone would leave the stored record half in the old shape. checkStore checks it.
  3. Rewriting everything, then deleting the steps. server.migrateAll() — from a one-off script, or at boot — enumerates every record of every kind that has update in its ops and a list() (every id; required for instant kinds, optional for the rest), loads each through its steps and writes it back whole, in turns, through the normal queue. Then twinstate lock --prune drops the old versions and their steps can be deleted — but only after it has verified, through the same enumeration, that no record of the kind has a _v older than the oldest version kept, and none is quarantined. It refuses otherwise, and always for a kind it cannot enumerate. The retired names stay.

Create-only kinds keep their steps. A kind whose ops has no update — a sent message, a published revision — is never rewritten, so its old records stay in their old shape for good, and its steps can never be deleted (--prune refuses to prune one). For such kinds, additive changes are close to mandatory.

A worked example ​

pinned changes from a boolean to 'yes' | 'no':

  1. Edit the schema. tsc fails on the kind: stored notes would not parse, so a step for the new version is required.
  2. Add migrate: { 2: (v1) => … }. v1 is typed { title: string; pinned: boolean; … }, straight from the lock.
  3. Run twinstate lock. The lock gains version 2, readable only from version 2 (a v1 build expects a boolean); the diff shows the change.
  4. checkStore loads version-1 sample notes through the step, and checks that the result parses and round-trips, written whole.
  5. Deploy. Clients on the old bundle are answered incompatible (a v1 build cannot read the new type) and the app shows its update prompt; the server migrates each note as it loads.
  6. Later, run server.migrateAll(), delete step 2, and twinstate lock --prune.

Running clients ​

A deploy does not upgrade the tabs that are already open: for a while, older bundles keep talking to the new server. They are never migrated — a client has no steps — so the question is only whether they can keep working, and the lock already answers it. It is the same pair of questions as above, asked of the places where old and new code meet:

where old meets newwho reads whatneeds
the server loads stored recordsnew code, old recordsnew code reads old records — or a step
an older client receives recordsold code, new recordsold code reads new records
the server receives an older client's mutations and outbox entriesnew code, old argumentsnew code reads old arguments
a rollback: older server code reads records written sinceold code, new recordsold code reads new records (above)
the persisted cache is read by another tab versioneither, from a copy that may lack fieldsnothing: a topic is used only under the same cache key, otherwise refetched

The lock records both answers. For every kind it already keeps, per version, the oldest version able to read its records (_c); it keeps the same for every mutation's arguments (the oldest version whose arguments the current one accepts), every event's payload, and each query's declared orders and filters. Every twinstate lock is a numbered revision, mapping each kind, mutation, event and query (its orders and filters) to its version at that point, and a client's bundle carries the revision it was built with.

A client sends its revision in its hello, and the server decides once, there: the client is compatible if, for every kind and event the server still has, the client's version is at least the oldest one able to read the server's (a kind or event the client does not know is ignored, and one the server removed is simply never sent), and, for every mutation, kind and query it can send, the server still has it and accepts its arguments. The server computes the answer for every revision in its lock at startup, so a hello costs one lookup and nothing is checked per message. (On one device, only the newest version's tab writes at all — see the client.) A compatible client stays connected — records in fields it does not know reach it, and its schemas strip them — and an incompatible one is answered incompatible (see the client): Twinstate never reloads anything. A revision the server does not know (a client newer than the server, after a rollback, or one older than what --prune kept) is incompatible too.

So most deploys disturb no one:

changean older client
a new field with a default; a new kind, mutation or eventstays connected: records of a kind its bundle does not know are ignored (it has no screen for them), and so are events it does not know
a default added to an existing fieldstays connected
a new optional or defaulted mutation argumentstays connected: its mutations parse under the new arguments
a widened mutation argument (a new enum value accepted)stays connected: its arguments are still accepted
a widened field (a new enum value, string → string | null)incompatible: it could receive a value it cannot parse
a new rejection reason on a mutationstays connected: a reason its bundle does not know arrives as error, so its exhaustive handling still holds
a rename, a type change, a removed fieldincompatible
a removed kind, mutation, or query order or filterincompatible: it could send something the server no longer has (a removed event is not: the client only stops receiving it)

The decision is for the whole client: an incompatible change to any kind it could receive makes it incompatible, even if no screen it has open uses that kind. That is what comparable sync engines do; refusing only the affected kinds, so the rest of a tab keeps syncing, is on the roadmap. The protocol already reserves incompatible as the answer to one subscribe or one mutate, so that can be added without a protocol change.

What types cannot see. zod refinements and transforms do not change a schema's type, so the lock cannot compare them. Tightening a refinement makes an older client's mutations fail as invalid — a defined answer. Loosening one can produce records an older client's schema rejects: a client that fails to parse a record it received reports incompatible rather than render it, so the worst case is the same answer, late. A change of meaning with the same type is invisible too: when older clients must not keep using a kind across one, lock its new version as breaking (twinstate lock --breaking conversation), and they are answered incompatible.

Breaking changes in two deploys (expand, then contract). A change older clients cannot read is better split, as other sync engines recommend:

  1. Expand: add the new shape beside the old one — a new field, a new mutation — and have handlers write both. This is additive, so every open tab stays connected.
  2. Contract, once the old bundles are gone: stop writing the old shape, and drop it (with dropped, or a step). Only tabs older than the expand are incompatible now, and by then there should be none.

A rename becomes: add the new field, write both, contract later. A widened enum can be shipped the same way — first a release whose code handles the new value, then one that writes it.

Later, if needed: open enum fields (a client type with an "unknown" case, which the compiler makes the UI handle, so a new value is compatible), and per-kind translation to older shapes (a downgrade per version, as some public APIs do) — powerful, but code and tests for every change, so not built.

Splitting a kind ​

Moving a field into records of a kind of its own — a reply's steps array into step records it refs — moves data between kinds, which no migrate step does (a step turns one record into the next version of the same kind). It is the app's to write, with the expand-and-contract recipe above, and Twinstate's own tools:

  1. Expand. Declare the new kind, add the ref beside the old field — with a default ([] for a list), so existing records still parse — and have the handlers write both. This is additive: every open tab stays connected.
  2. Backfill. A one-off job — a server mutation sent for each record of the parent kind, as server.migrateAll() works through one — creates the new records from each record's old field, with ids derived from the parent (tx.childId(step, replyId, String(n))) so a re-run creates nothing twice, and fills the refs.
  3. Contract, once the old bundles are gone: stop writing the old field, drop it (dropped), and update the delete handler — embedded records do not go with the record that refers to them; your handlers remove them (see what your store and handlers own).

Folding records back into a field is the same recipe the other way. Splits are rarer when nested values that need their own handling start as kinds (see ref or a field).

Records that cannot be read ​

A record can be unreadable: its _c is newer than the running code (it was rewritten by newer code the running code cannot read, after a rollback), it fails to parse, or its file is corrupt. Such a record is quarantined, not fatal:

  • it is skipped at load, and reported through the server's onUnreadable option (({ kind, id, reason }) => …, logging by default);
  • it still exists as far as writes go: a create of its id returns false, and an update or remove of it is rejected (unavailable), so nothing ever overwrites a record the running code could not read;
  • it comes back the moment code that can read it is running again;
  • it is left out of snapshots, and a link to it resolves to null on clients, so nothing renders a record no one could read. The stored link keeps its id — a handler's draft sees the id, never the resolved value — so nothing is lost when it becomes readable again.

What else to do about unreadable data — keep it, repair it, remove it — is the store's business, not Twinstate's (see what your store and handlers own): Twinstate never overwrites it, but a parent's remove is your adapter's, so whether deleting a parent takes an unreadable child with it is yours to decide (a store that deletes a whole folder takes it along).

So one unreadable record never takes the server down, and newer data is never misread by older code.

Rolling back ​

Rolling back across changes that old code can read (the table's last column) just works: older code reads records newer code wrote (their _c is old enough), and keeps the fields it does not know. Rolling back across one it cannot leaves the records written since then quarantined until the newer code returns — none is misread, none is lost. To undo such a change for good, ship its reverse as the next version, with its own step.

Where the samples come from ​

The first samples that matter are the records that exist before Twinstate. Types describe what the schema accepts, not what is on disk, so adopting Twinstate over existing files starts with a read-only check of real data: twinstate check <store> loads every record through the adapter and the current schema, writes nothing, and lists what would be quarantined. Run it against a copy of each real store before the first deploy, and copy a few real records of each shape into the test dataset, so checkStore keeps covering them.

Enforced where ​

ruleenforced by
a change old records would not survive has a stepthe compiler (against the lock)
a removed field is acknowledged with droppedthe compiler
a step's input is the shape actually storedthe compiler (typed from the lock)
a step produces a value the current schema accepts, written wholecheckStore, over sample records of every version in your test dataset
the lock is regenerated only on purposereview: it is a checked-in file, and describeTwin fails if the registry and the lock disagree
a retired name is never reused, and retired fields are deleted on loadthe compiler (against the lock), and checkStore with records an older version wrote back
newer data is never misread or overwritten by older codethe loader: _c too new means quarantine, and a quarantined record is never written or pruned past
existing data adopted without surprisestwinstate check, run before the first deploy
an older client stays connected only while it can read what it receivesthe server, at hello, from the lock's revisions; a record it cannot parse after all makes it report incompatible