Changing a schema
A kind's schema will change as an application grows. The schema is the contract with every client, so Twinstate owns its versions: which shape each version has, and how a record of one version becomes the next. Where and how records are stored stays your store's: it keeps the version next to the value and hands both back. This page says what Twinstate does about old data, when you have to write a migration — and why the best migration is the one you avoid.
Prefer changes that need none
A migration is code that must keep working for as long as any record in the old shape might still exist. Until every stored record has been rewritten (below), each step stays in the codebase, is typed against a shape nobody uses any more, and has to be read by everyone who touches that kind. Most schema changes can be made additive instead, and additive changes need nothing at all:
- add a field with a default (
tags: z.array(z.string()).default([])) rather than a required one; - add a new field and stop writing the old one, rather than renaming it;
- widen a type (
string→string | null, a new enum value the code already handles) rather than changing it — no step, though not rollback-safe: after a rollback, every record written since is unreadable to the old code, whatever its values (see the table); - add a new kind rather than restructuring an old one.
Reach for a migration when the additive version would leave the model worse: two fields meaning the same thing, a type that no longer says what is true.
Where old data lives
Only one place ever needs a migration:
| where old data lives | what happens |
|---|---|
| records on the server's disk, written by older code | migrated when loaded, by the steps below — the only migration there is |
| clients running an older bundle | never migrated: an older client stays connected for as long as every change since its version is one it can read, and is told incompatible otherwise (see running clients) |
| outbox entries written by an older bundle | sent by the old client while it is compatible; after an upgrade, checked strictly by the new bundle before sending, and an entry that no longer fits is held as "made by an older version", shown with its content to copy or discard (see the client) |
| the persisted cache | every saved topic carries a cache key (its kinds' versions, stored fields and format); a tab uses it only under the same key, and otherwise deletes it and refetches the topic (see the client) |
What needs a step, and what doesn't
Two different questions decide what a change costs:
- Can the new code read the old records? If not, it needs a step — a function that turns an old record into a new one.
- Can the old code read the new records? If not, the change is not rollback-safe: after a rollback, records written since become unreadable (quarantined, below) until the new code returns.
| change | step needed? | old code can still read new records? |
|---|---|---|
a new field with a default (.nullable().default(null) for none) | no | yes — the field is kept as unknown, and survives its writes |
| a new kind | no | yes — old code ignores it |
loosening a type (string → string | null, a new enum value) | no | no — every record written since, whatever its values: readability is decided per version, not per record (checking each written value against every older schema would cost every write) |
| removing a field | no step, but the kind must say dropped: ['field'], because that data is deleted — explicitly, whatever zod's unknown-key mode — and the name is retired for good | only if the field had a default in the old code |
| adding a required field with no default | yes (old records lack it) | yes |
| changing a type, renaming, restructuring | yes — and it does not compile until the step exists | no |
A change in what a value means with the same type — status: 'done' now also covering what 'archived' used to — is invisible to any tool. Treat it as a change of type: add a step that rewrites the old values, or add a new field.
How it works
- Every lock is a version.
twinstate lockwrites a checked-in file,twinstate.lock.ts: for each kind, its stored shape at every version, as types, and its retired field names; the same versions for every mutation's arguments and every event's payload; and a numbered revision per run, mapping each of them — and each query's orders and filters — to its version then (see running clients). It is regenerated only on purpose, so every schema change shows up in review as a diff to it. Two branches that each ran it conflict on the file when merged, like two migrations taking the same number: resolve it by runningtwinstate lockagain on the merged code, never by hand-picking one side. - Every field's handling is recorded, and confirmed. The lock also records, per kind and version, each field's effective settings, its defaults included: who sees it (
clientsorserver), how long it lives (durableormemory), how its changes travel (replaceorsuffix), and its default value. A field nobody marked is still a decision:twinstate locklists every new or changed field with its settings — "reply.usage: sent to clients, durable, replaced whole" — and does not write the lock until each is confirmed (--confirm reply.usage), so forgettingserverOnlyis caught before it ships, and the decision is a line someone reads in review. CI fails when the registry and the lock disagree. - The compiler asks the first question. If records of the latest locked version would not parse under the new schema (after its
droppedfields are deleted), or a field disappeared withoutdropped, the kind does not compile until it declares a step for the new version. - The lock answers the second. For each version it records the oldest version that can read its records — the smallest
ksuch that every version fromkup to it accepts them, checked against every older version, not just the one before. - A retired name is never reused. A field that disappears —
dropped, or removed by a step such as a rename — joins the kind's retired names, which the lock keeps for good (pruning old versions does not forget them). Declaring a field with a retired name does not compile. So a name always means one thing, and newer code, when it loads a record, deletes every retired field it finds in it — safely, whoever wrote it, because no current field can have that name. - Every stored record says what shape it is in. Twinstate keeps two stamps in the record's
meta(see the store adapter):_v, the version whose shape the record is in — every step up to it has run — and_c, the oldest version able to read it. A record with no stamps is version 1 — which is how data that existed before Twinstate (an application's existing files, say) enters the scheme. The stamps never go down: older code that writes a record it loaded with a newer_v(readable, because its_callows it) keeps them. So the newer code never runs a step twice over data that is already migrated. - Unknown fields survive. A field in a stored record that the running code does not know — written by newer code — is kept aside in
metaon load, never sent to clients, and written back unchanged with the record. So older code can read, and change, records newer code wrote, without deleting what it cannot see. If the older code writes back a field that the newer code has retired, the newer code deletes it on its next load: an edit made during a rollback to a field the newer code no longer has is lost — the price of rolling back across that change. - A step turns one version into the next, and its parameter is typed from the lock, so it cannot be wrong about what is on disk:
ts
kind('note', noteFields, {
id: true,
load: 'instant',
ops: ['create', 'update', 'remove'],
migrate: {
2: (v1) => ({ ...v1, pinned: v1.pinned ? 'yes' : 'no' }), // v1 → v2: a boolean became an enum
4: ({ labels, ...v3 }) => ({ ...v3, tags: labels }), // v3 → v4: labels renamed to tags (v3 only added a field)
},
});- Loading runs the steps. Twinstate reads a record's
_v, applies each step above it up to the current version, deletes retired fields, then validates the result against the current schema.instantkinds are all loaded at boot, so they are migrated in memory at once; lazy ones when first opened. - A record whose stored form changed is written whole. The first write to a record that loading changed — a step ran, or a retired field was deleted, so its
metadiffers from what is on disk — goes through the adapter'supdatewith the whole record, neverupdateFields: the handler's patches alone would leave the stored record half in the old shape.checkStorechecks it. - Rewriting everything, then deleting the steps.
server.migrateAll()— from a one-off script, or at boot — enumerates every record of every kind that hasupdatein itsopsand alist()(every id; required for instant kinds, optional for the rest), loads each through its steps and writes it back whole, in turns, through the normal queue. Thentwinstate lock --prunedrops the old versions and their steps can be deleted — but only after it has verified, through the same enumeration, that no record of the kind has a_volder than the oldest version kept, and none is quarantined. It refuses otherwise, and always for a kind it cannot enumerate. The retired names stay.
Create-only kinds keep their steps. A kind whose ops has no update — a sent message, a published revision — is never rewritten, so its old records stay in their old shape for good, and its steps can never be deleted (--prune refuses to prune one). For such kinds, additive changes are close to mandatory.
A worked example
pinned changes from a boolean to 'yes' | 'no':
- Edit the schema.
tscfails on the kind: stored notes would not parse, so a step for the new version is required. - Add
migrate: { 2: (v1) => … }.v1is typed{ title: string; pinned: boolean; … }, straight from the lock. - Run
twinstate lock. The lock gains version 2, readable only from version 2 (a v1 build expects a boolean); the diff shows the change. checkStoreloads version-1 sample notes through the step, and checks that the result parses and round-trips, written whole.- Deploy. Clients on the old bundle are answered
incompatible(a v1 build cannot read the new type) and the app shows its update prompt; the server migrates each note as it loads. - Later, run
server.migrateAll(), delete step 2, andtwinstate lock --prune.
Running clients
A deploy does not upgrade the tabs that are already open: for a while, older bundles keep talking to the new server. They are never migrated — a client has no steps — so the question is only whether they can keep working, and the lock already answers it. It is the same pair of questions as above, asked of the places where old and new code meet:
| where old meets new | who reads what | needs |
|---|---|---|
| the server loads stored records | new code, old records | new code reads old records — or a step |
| an older client receives records | old code, new records | old code reads new records |
| the server receives an older client's mutations and outbox entries | new code, old arguments | new code reads old arguments |
| a rollback: older server code reads records written since | old code, new records | old code reads new records (above) |
| the persisted cache is read by another tab version | either, from a copy that may lack fields | nothing: a topic is used only under the same cache key, otherwise refetched |
The lock records both answers. For every kind it already keeps, per version, the oldest version able to read its records (_c); it keeps the same for every mutation's arguments (the oldest version whose arguments the current one accepts), every event's payload, and each query's declared orders and filters. Every twinstate lock is a numbered revision, mapping each kind, mutation, event and query (its orders and filters) to its version at that point, and a client's bundle carries the revision it was built with.
A client sends its revision in its hello, and the server decides once, there: the client is compatible if, for every kind and event the server still has, the client's version is at least the oldest one able to read the server's (a kind or event the client does not know is ignored, and one the server removed is simply never sent), and, for every mutation, kind and query it can send, the server still has it and accepts its arguments. The server computes the answer for every revision in its lock at startup, so a hello costs one lookup and nothing is checked per message. (On one device, only the newest version's tab writes at all — see the client.) A compatible client stays connected — records in fields it does not know reach it, and its schemas strip them — and an incompatible one is answered incompatible (see the client): Twinstate never reloads anything. A revision the server does not know (a client newer than the server, after a rollback, or one older than what --prune kept) is incompatible too.
So most deploys disturb no one:
| change | an older client |
|---|---|
| a new field with a default; a new kind, mutation or event | stays connected: records of a kind its bundle does not know are ignored (it has no screen for them), and so are events it does not know |
| a default added to an existing field | stays connected |
| a new optional or defaulted mutation argument | stays connected: its mutations parse under the new arguments |
| a widened mutation argument (a new enum value accepted) | stays connected: its arguments are still accepted |
a widened field (a new enum value, string → string | null) | incompatible: it could receive a value it cannot parse |
| a new rejection reason on a mutation | stays connected: a reason its bundle does not know arrives as error, so its exhaustive handling still holds |
| a rename, a type change, a removed field | incompatible |
| a removed kind, mutation, or query order or filter | incompatible: it could send something the server no longer has (a removed event is not: the client only stops receiving it) |
The decision is for the whole client: an incompatible change to any kind it could receive makes it incompatible, even if no screen it has open uses that kind. That is what comparable sync engines do; refusing only the affected kinds, so the rest of a tab keeps syncing, is on the roadmap. The protocol already reserves incompatible as the answer to one subscribe or one mutate, so that can be added without a protocol change.
What types cannot see. zod refinements and transforms do not change a schema's type, so the lock cannot compare them. Tightening a refinement makes an older client's mutations fail as invalid — a defined answer. Loosening one can produce records an older client's schema rejects: a client that fails to parse a record it received reports incompatible rather than render it, so the worst case is the same answer, late. A change of meaning with the same type is invisible too: when older clients must not keep using a kind across one, lock its new version as breaking (twinstate lock --breaking conversation), and they are answered incompatible.
Breaking changes in two deploys (expand, then contract). A change older clients cannot read is better split, as other sync engines recommend:
- Expand: add the new shape beside the old one — a new field, a new mutation — and have handlers write both. This is additive, so every open tab stays connected.
- Contract, once the old bundles are gone: stop writing the old shape, and drop it (with
dropped, or a step). Only tabs older than the expand are incompatible now, and by then there should be none.
A rename becomes: add the new field, write both, contract later. A widened enum can be shipped the same way — first a release whose code handles the new value, then one that writes it.
Later, if needed: open enum fields (a client type with an "unknown" case, which the compiler makes the UI handle, so a new value is compatible), and per-kind translation to older shapes (a downgrade per version, as some public APIs do) — powerful, but code and tests for every change, so not built.
Splitting a kind
Moving a field into records of a kind of its own — a reply's steps array into step records it refs — moves data between kinds, which no migrate step does (a step turns one record into the next version of the same kind). It is the app's to write, with the expand-and-contract recipe above, and Twinstate's own tools:
- Expand. Declare the new kind, add the
refbeside the old field — with a default ([]for a list), so existing records still parse — and have the handlers write both. This is additive: every open tab stays connected. - Backfill. A one-off job — a server mutation sent for each record of the parent kind, as
server.migrateAll()works through one — creates the new records from each record's old field, with ids derived from the parent (tx.childId(step, replyId, String(n))) so a re-run creates nothing twice, and fills the refs. - Contract, once the old bundles are gone: stop writing the old field, drop it (
dropped), and update the delete handler — embedded records do not go with the record that refers to them; your handlers remove them (see what your store and handlers own).
Folding records back into a field is the same recipe the other way. Splits are rarer when nested values that need their own handling start as kinds (see ref or a field).
Records that cannot be read
A record can be unreadable: its _c is newer than the running code (it was rewritten by newer code the running code cannot read, after a rollback), it fails to parse, or its file is corrupt. Such a record is quarantined, not fatal:
- it is skipped at load, and reported through the server's
onUnreadableoption (({ kind, id, reason }) => …, logging by default); - it still exists as far as writes go: a
createof its id returnsfalse, and anupdateorremoveof it is rejected (unavailable), so nothing ever overwrites a record the running code could not read; - it comes back the moment code that can read it is running again;
- it is left out of snapshots, and a
linkto it resolves tonullon clients, so nothing renders a record no one could read. The stored link keeps its id — a handler's draft sees the id, never the resolved value — so nothing is lost when it becomes readable again.
What else to do about unreadable data — keep it, repair it, remove it — is the store's business, not Twinstate's (see what your store and handlers own): Twinstate never overwrites it, but a parent's remove is your adapter's, so whether deleting a parent takes an unreadable child with it is yours to decide (a store that deletes a whole folder takes it along).
So one unreadable record never takes the server down, and newer data is never misread by older code.
Rolling back
Rolling back across changes that old code can read (the table's last column) just works: older code reads records newer code wrote (their _c is old enough), and keeps the fields it does not know. Rolling back across one it cannot leaves the records written since then quarantined until the newer code returns — none is misread, none is lost. To undo such a change for good, ship its reverse as the next version, with its own step.
Where the samples come from
The first samples that matter are the records that exist before Twinstate. Types describe what the schema accepts, not what is on disk, so adopting Twinstate over existing files starts with a read-only check of real data: twinstate check <store> loads every record through the adapter and the current schema, writes nothing, and lists what would be quarantined. Run it against a copy of each real store before the first deploy, and copy a few real records of each shape into the test dataset, so checkStore keeps covering them.
Enforced where
| rule | enforced by |
|---|---|
| a change old records would not survive has a step | the compiler (against the lock) |
a removed field is acknowledged with dropped | the compiler |
| a step's input is the shape actually stored | the compiler (typed from the lock) |
| a step produces a value the current schema accepts, written whole | checkStore, over sample records of every version in your test dataset |
| the lock is regenerated only on purpose | review: it is a checked-in file, and describeTwin fails if the registry and the lock disagree |
| a retired name is never reused, and retired fields are deleted on load | the compiler (against the lock), and checkStore with records an older version wrote back |
| newer data is never misread or overwritten by older code | the loader: _c too new means quarantine, and a quarantined record is never written or pruned past |
| existing data adopted without surprises | twinstate check, run before the first deploy |
| an older client stays connected only while it can read what it receives | the server, at hello, from the lock's revisions; a record it cannot parse after all makes it report incompatible |