<). You don't write SQL or manage table structure.
A few boundaries to know first:
- **Data belongs to this gem only.** Other gems can't read it. It is deleted entirely on uninstall, and reinstalling does not bring back the old data.
- **Storage is not a permission.** Every gem and every one of its tools has it. You don't request it in the manifest, and the user doesn't need to turn it on.
- **Anything meant for the user or for other gems goes in the knowledge base** ([Knowledge API](/en/dev/knowledge.md)). `ctx.data` holds the gem's own working data.
- **Collections, Logs and Blobs are stored encrypted; documents are plaintext files.** See [Encryption and Isolation](#encryption-and-isolation).
- **One copy per organization.** The same gem installed in two organizations has two separate sets of data that don't connect.
`ctx.data` provides the following, each explained below:
| Interface | What it stores |
| --- | --- |
| `ctx.data.collection(name)` | Tables: records that can be modified and queried |
| `ctx.data.log(name)` | Logs: append-only entries that are never modified, such as each reading or each action |
| `ctx.data.docs` | Documents: Markdown, editable and referenceable by block |
| `ctx.data.blobs` | Blobs: files (images, PDFs, ...), deduplicated by content |
| `ctx.data.refs` | References: links between the objects above |
| `ctx.data.transact(fn)` | Transactions: several writes either all succeed or none do |
Methods that read or write data are asynchronous and need `await`; `collection()`, `log()`, the methods that build a query, and `ctx.data.newId()` are synchronous.
## Declaring Data
In the packaged manifest (`notochord.json`), collections and Logs go in the top-level `data` section.
```json
{
"data": {
"collections": {
"notes": {
"kind": "table",
"fields": {
"title": { "type": "string", "required": true },
"status": { "type": "string", "enum": ["open", "done"] },
"due": { "type": "timestamp" },
"project": { "type": "ref", "to": "projects" },
"photo": { "type": "blob" },
"tags": { "type": "array", "items": "string" },
"extra": { "type": "object" }
},
"indexes": [["status", "due"], ["project"]]
},
"projects": {
"kind": "table",
"fields": {
"name": { "type": "string", "required": true, "unique": true },
"open_count": { "type": "integer" }
}
},
"readings": {
"kind": "log",
"fields": { "value": { "type": "number" }, "note": { "type": "ref", "to": "notes" } },
"retention": { "days": 365 }
}
}
}
}
```
- **`ctx.data` needs no request**: every tool has it once the gem is installed. If you only use documents, Blobs and references and need no collections, you can omit the `data` section.
- **A tool that declares `sideEffect: "read"` must not write any storage**: plan mode runs such tools before the user approves, and the platform does not check what they actually wrote. See [Permission Model](/en/dev/permissions.md#storage-is-not-a-permission).
- For how to write this in a Gem Studio draft, see [Testing in Gem Studio](#testing-in-gem-studio).
### Collections
Each entry in `data.collections` is a collection. The key is the name used in `collection(name)` and `log(name)`. There are two kinds of collection: `table` and `log`.
| Key | Table | Log | Notes |
| --- | --- | --- | --- |
| `kind` | `"table"` | `"log"` | Required; cannot be changed later |
| `fields` | Required | Required | 1–128 fields, see below |
| `description` | Optional | Optional | Up to 500 characters |
| `version` | Optional | Not allowed | Integer, defaults to 1, can only go up, never down; see [Versions and Migrations](#versions-and-migrations) |
| `indexes` | Optional | Not allowed | Up to 16; each is an array of 1–8 field names; see [Index Rules](#index-rules) |
| `migrations` | Required when `version` is greater than 1 | Not allowed | The file that holds the migration functions |
| `retention` | Not allowed | Optional | `{ "days": 1–36500, "maxEntries": 1–10000000 }`, write at least one; see [Log](#log) |
Collection names and field names: start with a lowercase letter, contain only lowercase letters, digits and underscores, at most 64 characters. A gem can have at most 64 collections.
The `data` section can also contain `eventRetentionDays` (1–3650, default 180): how many days the platform keeps the change records it writes for every modification.
### Fields
| `type` | Value | Can be queried, sorted, indexed |
| --- | --- | --- |
| `string` | String | Yes |
| `number` | Finite number | Yes |
| `integer` | Safe integer | Yes |
| `boolean` | `true` or `false` | Yes |
| `timestamp` | Millisecond timestamp (integer) | Yes |
| `ref` | The id of a record in some table; `to` names the table | Yes |
| `blob` | The hash of a Blob that has already been stored | Yes |
| `array` | Array; `items` gives the element type | Only with `array-contains` queries, and only when `items` is set |
| `object` | Plain object; contents are not checked | No |
| `any` | Any JSON | No |
The first seven are called scalar fields. A field can also have:
| Key | Notes |
| --- | --- |
| `required` | Required. |
| `unique` | The value must not repeat. Only allowed on scalar fields of tables, not on Logs. |
| `enum` | Allowed values, 1–256, no duplicates, with a type matching the field. Only allowed on `string`, `number` and `integer`. |
| `to` | Required for `ref` fields (and arrays whose `items` is `ref`): the name of the table pointed to, which must be a `table` in the same manifest. |
| `items` | Only allowed on `array`; one of the scalar types. Optional: if omitted, elements are not checked, but `array-contains` queries cannot be used either. |
| `description` | Up to 500 characters. |
Data that doesn't match the declaration is rejected on write (`E_INPUT_INVALID`); the platform never converts anything silently. A wrong type, a missing required field, an undeclared field, or a `blob` field whose hash has not been stored yet all raise an error. A `ref` field only has its id format checked, not whether that record exists. `object`, `any` and arrays without `items` can nest at most 32 levels deep.
There is one convention: **a value of `null` is the same as the field being absent**. On insert, fields whose value is `null` are dropped; on update, setting a field to `null` deletes it.
When the `data` section is written incorrectly (an invalid name, a `to` that points to a table that doesn't exist, a `version` above 1 with no `migrations`, ...), manifest validation fails with `E_PERM_INVALID`. This is checked when packaging, installing and testing in Gem Studio. When packaging and installing, shape errors (unknown keys, a wrong `type`, an over-long description) are caught first by the manifest's JSON Schema, with the message `notochord.json failed schema validation: …`; other errors (such as the ones listed above) have messages starting with `data:`. In Gem Studio tests, all of them start with `data:`.
## Collections: Writing and Querying
```js
export default async function (input, ctx) {
const notes = ctx.data.collection('notes')
const { id } = await notes.insert({ title: input.title, status: 'open', due: Date.now() })
const note = await notes.get(id) // { id, version, createdAt, updatedAt, data }, or null if there is none
// Merge by top-level field; fields set to null are deleted; ifVersion: only update if nobody has changed it
await notes.update(id, { status: 'done', due: null }, { ifVersion: note.version })
const page = await notes.where('status', '==', 'open').orderBy('due').limit(20).get()
// page = { items: [...], nextCursor?: '...' }
if (page.nextCursor) {
const next = await notes.where('status', '==', 'open').orderBy('due').limit(20).startAfter(page.nextCursor).get()
}
const open = await notes.where('status', '==', 'open').count()
return { open }
}
```
| Method | Notes |
| --- | --- |
| `insert(data, { id? })` | Creates a record and returns `{ id, version, nodeId }`. If `id` is not given, one is generated; if the given `id` already exists, it raises `E_DATA_CONFLICT`. An id may contain only letters, digits, `_` and `-`, up to 128 characters. |
| `get(id)` | `{ id, version, createdAt, updatedAt, data }`, or `null` if there is no such record. |
| `update(id, patch, { ifVersion? })` | Merges by top-level field: fields in `patch` replace the old values, fields set to `null` are deleted, and the merged record is checked against the declaration as a whole. Returns `{ version }`. Raises `E_DATA_NOT_FOUND` if the record doesn't exist. |
| `delete(id, { ifVersion? })` | Returns `{ deleted: true \| false }`; `false` if the record didn't exist, except that with `ifVersion` it raises `E_DATA_CONFLICT`. |
| `where(field, op, value)`, `orderBy(field, 'asc' \| 'desc')`, `limit(n)`, `startAfter(cursor)` | Build a query. Each step returns a new query; nothing runs until you call `get()` or `count()`. |
| `get()` | Runs the query and returns `{ items, nextCursor? }`; each item in `items` is the same as what `get(id)` returns. |
| `count()` | The number of records that match. Only the `where` conditions count; ordering, `limit` and `startAfter` have no effect. |
- **`version`** starts at 1 and goes up by 1 with every write. If you pass `ifVersion` to `update` or `delete` and the record has been changed since you read it, it raises `E_DATA_CONFLICT` and changes nothing.
- **Deletion is a soft delete**: the record disappears from the interface, but its content stays in the data folder until uninstall. See [Encryption and Isolation](#encryption-and-isolation). A deleted id can be used to insert again.
### Query Rules
- Operators: `==`, `!=`, `<`, `<=`, `>`, `>=`, `in` (1–30 values), `array-contains`. `where('due', '==', null)` finds records without the field, and `!= null` finds records with it.
- Only scalar fields can be filtered and sorted, and the value compared must match the field's type; `array` fields can only use `array-contains`.
- A query can have at most 16 conditions and 8 sort fields, and at most one `in`.
- Range conditions (`<`, `<=`, `>`, `>=`, `!=`) can only be used on one field; when there is an `orderBy`, the first sort field must be that field.
- All `orderBy` directions are the same.
- If the field with an `in` condition also takes part in sorting, it can only be the first sort field.
- Without `orderBy`: if there is a range condition, results are sorted by that field; otherwise by id.
- **Records missing a field** don't appear in results sorted by that field, and don't satisfy range conditions or `!=` on that field (except `!= null`).
- Strings are compared by Unicode code point and are case-sensitive.
- `limit` defaults to 100 and is at most 1000. A single result can be at most 16 MB; beyond that it raises `E_PAYLOAD_TOO_LARGE`, so lower `limit`.
- When there is another page, the result contains `nextCursor`; pass it unchanged to `startAfter` on the same query.
## Index Rules
Once a collection holds **more than 1000 records**, every query must have a declared index it can use, or it raises `E_DATA_NEEDS_INDEX`, and the message says which index to add:
```
E_DATA_NEEDS_INDEX: query on 'notes' needs an index: add ["status","due"] to data.collections.notes.indexes
```
Up to 1000 records nothing is checked, so everything works during development and only fails once the user's data grows. **Declare the index each query needs from the start.**
An index can be used for a paginated query (`get()`) if and only if its fields are **exactly**:
1. The fields of the `==` and `in` conditions in the query, in any order;
2. followed by its sort fields, in the same order as `orderBy`. If there is only a range condition and no `orderBy`, the sort field is that range field; fields pinned by `==` (other than `== null`) or `in` don't count as sort fields.
One field too many or too few doesn't work. `count()` is more lenient: the index may start with the `==` and `in` fields (in any order), followed by the range field (if there is one), and any further fields after that don't matter.
| Query | Index that works |
| --- | --- |
| `where('status', '==', 'open').orderBy('due').get()` | `["status", "due"]` |
| `where('status', '==', 'open').get()` | `["status"]` (`["status", "due"]` does not work) |
| `where('due', '<', t).get()` | `["due"]` |
| `orderBy('due', 'desc').get()` | `["due"]` |
| `where('project', '==', p).where('status', 'in', ['open', 'done']).orderBy('due').get()` | `["project", "status", "due"]` or `["status", "project", "due"]` |
| `where('status', '==', 'open').count()` | Any index starting with `"status"`, such as `["status", "due"]` |
| No conditions, no sorting (paging by id) | No index needed |
- **`array-contains` can't use an index** (arrays can't be indexed). On a large collection it has to be combined with an indexed `==` or range condition; used alone it raises `E_DATA_NEEDS_INDEX`.
- **`unique` only prevents duplicates; queries can't use it.** To query by a `unique` field, also declare it in `indexes`.
- Only scalar fields can be indexed. Adding or removing indexes doesn't require a version bump; new indexes are built the next time the gem starts (after install or upgrade).
- **Every call has a scan limit**: a paginated query reads at most about 64 MB of records, and a count counts at most 1 million records. Going over also raises `E_DATA_NEEDS_INDEX`. When too many records match, add conditions to narrow the range.
## Transactions
```js
const noteId = ctx.data.newId()
await ctx.data.transact(async (tx) => {
const project = await tx.get('projects', input.projectId) // Read: runs immediately, version is recorded
if (!project) throw new Error('No such project') // Throwing: nothing is committed
tx.update('projects', input.projectId, { open_count: (project.data.open_count ?? 0) + 1 }) // Write: collected first
tx.insert('notes', { title: input.title, project: input.projectId }, { id: noteId })
tx.log('readings').append({ value: 1, note: noteId })
}) // Committed together after the function returns
```
- Available in a transaction: `get`, `insert`, `update`, `delete`, `log(name).append`, `link`, `unlink`. Only `get` is asynchronous and runs immediately; writes are collected first and committed in one go after the function returns, at most 500 of them.
- **The function can't read its own uncommitted writes.**
- At commit, the platform checks whether the records the function read have been changed by someone else (including being deleted, or not existing when read but existing now). If so, it **reruns the whole function**, at most 5 times by default (`transact(fn, { maxAttempts })` can change this), and then throws `E_DATA_CONFLICT`. So don't do anything with external side effects inside the function: requests, file writes and model calls all belong outside the transaction.
- When the function throws, nothing is committed and the error is rethrown as is; when it returns normally, `transact` returns its return value.
- To know a new record's id ahead of time inside a transaction, use `ctx.data.newId()`.
- Document and Blob operations can't go in a transaction; each is atomic on its own. Every write outside a transaction is also atomic.
## Log
A Log stores append-only entries that are never modified, such as each reading or each processed email.
```js
const readings = ctx.data.log('readings')
await readings.append({ value: 4.2, note: noteId }) // { id, nodeId }
const { items, nextCursor } = await readings.query({ since: Date.now() - 86400e3, limit: 100 })
// items: [{ id, at, data }], newest first by default
```
| Method | Notes |
| --- | --- |
| `append(entry)` | Appends an entry. Checked against the declared fields like a table; at most 1 MiB. Returns `{ id, nodeId }`. |
| `query({ since?, until?, limit?, order?, startAfter? })` | `since` and `until` are millisecond timestamps, inclusive at both ends; `order` defaults to `'desc'` (newest first); `limit` defaults to 100, at most 1000. Returns `{ items: [{ id, at, data }], nextCursor? }`, where `at` is the time of appending. A single result is at most 16 MB. |
- **There is no interface to modify or delete a single entry.** The platform cleans up old entries according to `retention`: `days` by time, `maxEntries` by count (keeping the newest); if both are written, both apply. Without `retention`, entries are kept until uninstall. Cleanup runs at most once a day.
- A Log can't have `version`, `indexes` or `migrations`, and its fields can't be `unique`. Its field declaration only governs entries written afterward and can be changed at any time without a version bump; old entries stay as they were written.
- `ref` and `blob` fields in entries maintain references automatically, just like in tables: a Blob referenced by an entry won't be cleaned up until the entry itself is cleaned up.
- To append inside a transaction: `tx.log(name).append(entry)`.
## Documents
Documents are Markdown, edited by block (paragraph, heading, list, code block, ...), and each block has a stable id.
```js
const { id, blockIds } = await ctx.data.docs.create({ title: 'Spec', markdown: '# Spec\n\nOil 50%.' })
const doc = await ctx.data.docs.get(id) // { id, title, markdown, blocks: [{ id, type, markdown }], createdAt, updatedAt }
await ctx.data.docs.replaceBlock(id, blockIds[1], 'Oil 55%.')
await ctx.data.docs.insertAfter(id, blockIds[1], 'A new paragraph\n\nAnd another')
await ctx.data.docs.deleteBlock(id, blockIds[0])
```
| Method | Notes |
| --- | --- |
| `create({ id?, title?, markdown })` | Creates a document and returns `{ id, nodeId, blockIds }`. |
| `get(id)` | Returns `{ id, title, markdown, blocks, createdAt, updatedAt }`, or `null` if there is none. `markdown` is the whole text without block ids; each item of `blocks` is `{ id, type, markdown }`. |
| `list({ limit?, startAfter? })` | `{ items: [{ id, title, updatedAt }], nextCursor? }`. |
| `update(id, markdown, { title? })` | Replaces the whole document; blocks whose content didn't change keep their old ids. Returns `{ blockIds }`. |
| `insertAfter(docId, blockId, markdown)` | Inserts after a block; `markdown` may contain several blocks; when `blockId` is `null`, inserts at the very beginning. Returns the new blocks' `{ blockIds }`. |
| `replaceBlock(docId, blockId, markdown)` | Replaces a block; if the new content has several blocks, the first keeps the original id. Returns `{ blockIds }`. |
| `deleteBlock(docId, blockId)` | Deletes a block. |
| `delete(id)` | Returns `{ deleted: true \| false }`. |
- Block ids look like `b_…`, don't change after edits, and can be referenced (see [References and Node IDs](#references-and-node-ids)). A block's `type` is `heading`, `paragraph`, `list`, `code`, `quote`, `table`, `thematic_break` or `html`.
- Each document is one `.md` file in the data folder, with blocks separated by blank lines, and each block's id written as `{#b_…}` at the end of the block's last line (for a code block, on the opening line; for a setext heading, on the text line). If the file is changed outside the app, the platform notices on the next read and rebuilds the blocks from the file.
- **Document files are not encrypted.** Put sensitive content in collections, not in documents.
- A document is at most 2 MiB, and a title at most 500 characters. Id rules are the same as for records; two ids that differ only in case can't exist at the same time (the later one raises `E_DATA_CONFLICT`), and deleted documents count too (a soft-deleted document keeps holding its id until uninstall); an id that is identical, including case, can be created again. `get()` matches the id exactly, so a different case finds nothing.
- If the document or block to modify doesn't exist, it raises `E_DATA_NOT_FOUND`.
- Document operations can't go in a transaction; each call is atomic on its own.
- Deletion is a soft delete: the `.md` file stays on disk until uninstall.
## Blobs
Blobs store files: images, PDFs, audio, ...
```js
const { hash } = await ctx.data.blobs.put(base64, { mime: 'image/jpeg', name: 'photo.jpg' }) // { hash, size, nodeId }
await ctx.data.collection('notes').update(id, { photo: hash })
const blob = await ctx.data.blobs.get(hash) // { base64, mime, name, size }, or null if there is none
const info = await ctx.data.blobs.stat(hash) // { mime, name, size, createdAt }, or null if there is none
```
- Content goes in and out as base64, at most 20 MiB each. The hash is the SHA-256 of the content (64 lowercase hex characters), and identical content is stored only once.
- **`put` first, then store the hash in a `blob` field.** A hash written to a `blob` field must already be stored, or it raises `E_INPUT_INVALID`.
- **There is no delete interface.** A Blob that nothing references (via a `blob` field of a record or Log entry, or via `refs.link` pointing at it) is cleaned up by the platform 7 days after the most recent `put`. Putting the same content again restarts the 7 days; `createdAt` from `stat()` is the time of the most recent `put`.
- Blobs are stored encrypted and can't go in a transaction.
## References and Node IDs
Every object in a gem has a node id:
```
gem:<gemId>/rec/<collection>/<id> a record
gem:<gemId>/doc/<docId> a document
gem:<gemId>/doc/<docId>#<blockId> a block in a document
gem:<gemId>/blob/<sha256> a Blob
gem:<gemId>/log/<name>/<entryId> a Log entry
```
`insert`, `docs.create`, `blobs.put` and `log(name).append` all return `nodeId`. You can also construct one yourself:
```js
const noteNode = await ctx.data.nodeId({ kind: 'rec', collection: 'notes', id })
const blockNode = await ctx.data.nodeId({ kind: 'doc', docId, blockId }) // without blockId, it's the whole document
// The other two: { kind: 'blob', hash }, { kind: 'log', name, eventId } (eventId is the id returned by append)
await ctx.data.refs.link(noteNode, blockNode, 'cites')
await ctx.data.refs.backlinks(blockNode) // who references it: [{ from, to, kind, detached, createdAt }]
await ctx.data.refs.outgoing(noteNode) // who it references
await ctx.data.refs.unlink(noteNode, blockNode, 'cites')
```
| Method | Notes |
| --- | --- |
| `link(from, to, kind)` | Creates a reference and returns `{ linked }`; `false` if the same reference already exists. |
| `unlink(from, to, kind)` | Deletes a reference and returns `{ unlinked }`. |
| `backlinks(to, { kind?, limit? })` | References pointing at `to`. `limit` defaults to 100, at most 1000. |
| `outgoing(from, { kind?, limit? })` | References starting from `from`. |
- `kind` is a name you choose, matching `^[a-z][a-z0-9_.:-]{0,63}$`. Names starting with `field:` are reserved for the platform: `ref` and `blob` fields maintain references automatically (with `kind` set to `field:<field name>`), so you don't need to `link` them yourself.
- When the target is deleted, references to it are not deleted but marked `detached: true`; they are restored when the target reappears. Referencing an object that doesn't exist yet is also allowed, and is likewise marked `detached`.
- **Only objects of this gem can be referenced**; using another gem's node id raises `E_PERM_DENIED`.
- `link` and `unlink` are writes: tools declared read-only must not use them; they can be used inside a transaction.
## Versions and Migrations
A table's fields can change, but the records already stored must still be valid. So:
- **Within the same version you can only "loosen"**: add optional fields, remove `required` or `unique`, widen an `enum` or remove it, add or remove indexes, change descriptions. Adding `unique` is also allowed, but if duplicate values already exist, the gem fails to start (`E_DATA_UNIQUE`).
- **Every other change needs a `version` bump and a migration**: deleting a field, changing `type`, changing `to` or `items`, adding `required` to an existing field, narrowing an `enum` or adding one to a field that had none, adding a new required field. Making these changes within the same version makes the gem raise `E_DATA_SCHEMA` at startup.
- **A collection's `kind` can't change; `version` can't go down.**
- Logs are not subject to these restrictions; their fields can change at any time.
- A collection removed from the manifest keeps its data but is no longer reachable through the interface, and the data is deleted on uninstall; if you add it back later, the old data is still there (checked by the same rules above).
- Upgrades and reinstalls of the same version both keep the data.
```json
"notes": {
"kind": "table",
"version": 2,
"migrations": "migrations/notes.js",
"fields": {
"title": { "type": "string", "required": true },
"status": { "type": "string", "enum": ["open", "done"] }
}
}
```
```js
// migrations/notes.js: one function per step, v1_to_v2, v2_to_v3, ...
export function v1_to_v2(data, { id }) {
return { title: data.title, status: data.done ? 'done' : 'open' }
}
```
- **Migrations run when the gem starts after an upgrade**: in a sandbox, the platform runs the needed steps in order on every record (going from version 1 to 3 runs `v1_to_v2` and then `v2_to_v3`), checks the result against the new declaration and writes it back. Only records that haven't been deleted are migrated. Each record's migration leaves a change record.
- **Each step receives a record's `data` and `{ id }` and must return a plain object**, so a migration can't delete records.
- **A migration function must give the same result every time**: a migration may be interrupted and rerun. It has no `ctx`, and calling `Date.now()`, `new Date()` without arguments or `Math.random()` raises an error immediately.
- `migrations` is a `.js`, `.mjs` or `.ts` file in the package, with no `..` in the path; it is required when `version` is greater than 1, and installation checks that it is actually in the package. If a needed step is missing, the upgrade fails before any data is changed.
- **Time limits**: 60 seconds per batch (up to 500 records), 10 minutes for the whole migration.
- **If any step fails** (the function throws, the result doesn't match the new declaration, a function is missing, a timeout), **the whole upgrade is rolled back**: the data is restored as it was and the old version keeps running. The user sees `E_ACTIVATION_FAILED`, with the cause (`E_DATA_SCHEMA: …`) included in the message.
- After a successful migration, a copy of the pre-migration `data.db` is kept in the data folder (only the most recent one, also encrypted).
## Testing in Gem Studio
### Writing It in a Draft
A Gem Studio draft's `manifest.json` is written differently from the packaged manifest:
```json
{
"data": {
"collections": {
"notes": { "kind": "table", "fields": { "title": { "type": "string", "required": true } } }
}
},
"exports": [
{ "name": "save_note", "kind": "tool", "entry": "tools/save_note.js", "...": "..." },
{ "name": "list_notes", "kind": "tool", "entry": "tools/list_notes.js", "side_effect": "read", "...": "..." }
]
}
```
- You don't write storage: it isn't among the permissions on the overview, nor on the tool page.
- The top-level `data`: written exactly like the packaged manifest, and has to be written by hand in `manifest.json`.
- Read-only tools declare `"side_effect": "read"`.
- When the `data` section is wrong, testing raises `E_PERM_INVALID` (`data: …`) just like installing.
### Test Data
- **Tests use this draft's own test data**: testing a tool on its own, the agent's test conversation and testing an SOP all use the same copy. It is completely separate from the data of the gem installed from Gem Studio, and neither affects the other.
- **Test data is kept**, across tests and across app restarts. Only when the `data` section or the files under `db/` change does the next test reopen it under the new declaration.
- **So migrations run on the next test**: raise `version` in the draft, write the migration file, then run any test. It is a good idea to write some data with the old version first, then raise the version and check that the migrated data is correct.
- **Once the version is raised, there is no going back**: after the test data has been migrated to the new version, changing `version` back raises `E_DATA_SCHEMA`.
- **Changing the migration file after a successful migration does nothing**, since the data is already on the new version. To fix a mistaken step, raise the version again and write the fix as the next step.
- If a migration fails, the test data stays as it was, and the next test tries again.
- **Only deleting the draft clears its test data** (together with its key).
- To test an upgrade the way a user would: install from Gem Studio, write some data with the installed gem, then raise the collection's `version` in the draft, write the migration file, and install again. Every install from Gem Studio automatically gets a higher version number and goes through the same upgrade flow as the store: the migration runs on the installed copy's data, and a failure rolls everything back.
## How SOPs Access Data
An SOP has only `ctx.call`, no `ctx.data`.
- **Collections, Logs, documents, Blobs: call your own tool**, and let the tool read and write.
- **`kv` and `rawSqlite` can be called directly**, with no declaration.
```js
export default async function (input, ctx) {
// Data in collections: hand it to your own tool
const { id } = await ctx.call('tool.save_note', { title: input.title })
// kv: call directly
const previous = await ctx.call('storage.kv.get', { key: 'last_note' }) // null if there is none
await ctx.call('storage.kv.set', { key: 'last_note', value: id })
return { id, previous }
}
```
| Call target | Parameters | Returns | Storage accessed |
| --- | --- | --- | --- |
| `storage.kv.get` | `{ key }` | The value, or `null` if there is none | `kv` |
| `storage.kv.set` | `{ key, value }` | `{ ok: true }` | `kv` |
| `storage.sql.query` | `{ sql, params? }` | An array of rows | `rawSqlite` |
| `storage.sql.insert` | `{ table, row }` | `{ id }` | `rawSqlite` |
- Call targets are scanned at install time like any other `ctx.call`; see [Events and Triggers](/en/dev/events.md#after-the-trigger-a-run).
- A tool called by an SOP may be rerun once after an app crash (see [Tools API](/en/dev/tools.md#how-a-call-runs)). A tool that writes data should preferably survive a rerun, for example by inserting with a deterministic id: on a rerun you get `E_DATA_CONFLICT` instead of an extra record.
## Other Storage Kinds
Besides `ctx.data` there are two more kinds of storage. Every gem has them and needs no request:
| Kind | Interface | Good for | Cost |
| --- | --- | --- | --- |
| `kv` | `ctx.storage.kv.get(key)`, `.set(key, value)` | Small settings, sync cursors | Values must be strings, at most 1 MiB |
| `rawSqlite` | `ctx.storage.sql.query(sql, params)`, `.insert(table, row)` | Things `ctx.data` really can't do | You write the SQL yourself; time limits and a 256 MB cap; no change records, no references, and the platform doesn't check the data |
### kv
```js
const since = Number((await ctx.storage.kv.get('cursor')) ?? 0)
// ... process what came after since ...
await ctx.storage.kv.set('cursor', String(cursor))
```
- Stored in `data.db` (encrypted); every `set` leaves a change record like any other write. Every tool has it, with no declaration.
- Values must be strings; to store an object, `JSON.stringify` it first. Keys are 1–512 characters and can't contain control characters.
- SOPs can call `storage.kv.get` and `storage.kv.set` directly, see the previous section.
### rawSqlite
```js
const { id } = await ctx.storage.sql.insert('notes', { title: input.title, created_at: Date.now() })
const rows = await ctx.storage.sql.query('SELECT id, title FROM notes WHERE created_at > ? LIMIT 50', [since])
```
- A private SQLite database, `raw.db`, encrypted with the gem's key and running in a separate process. Every gem has it, with no declaration. `query` returns an array of rows, and `insert` returns `{ id }` (the new row's rowid).
- **When it is opened**: a gem whose package has `db/schema.sql` or `db/migrations/` opens it at startup, and if the table structure or a migration has an error the gem can't start, so this is found at install time. Other gems open it on the first query; for a gem that never uses it, there is no such file on disk.
- **Table structure goes in the package's `db/` folder.** You can also `CREATE TABLE` in a query, but writing it in `db/schema.sql` is recommended:
- `db/schema.sql` runs every time the database is opened, so it may only contain statements that can be run repeatedly (`CREATE TABLE IF NOT EXISTS …`), and it can't change existing tables.
- To change the table structure, write `db/migrations/<number>_<name>.sql` (for example `0002_add_lang.sql`). Each migration runs only once, after `db/schema.sql`, in ascending order of number. Numbers must not repeat; don't change or reuse a number that has been published, and write fixes as a new, larger number. A file name that isn't "number_name.sql" makes installation fail.
- `ATTACH`, `DETACH`, `PRAGMA` (including `pragma_*` functions) and `VACUUM` can't be used; statements containing them are rejected.
- The cost: the platform doesn't know what is stored here. There are no change records, no record of who wrote it, no references, and the data isn't checked. Use it only when `ctx.data` really can't do the job. For time and size limits see [Limits](#limits).
## Encryption and Isolation
All of a gem's data is in its own data folder (`gems/store/<gem id>/` under the organization's data folder):
| Content | Where | Encrypted |
| --- | --- | --- |
| Collections, Logs, `kv`, change records | `data.db` | Encrypted |
| Blobs | `blobs/` | Encrypted |
| `rawSqlite` | `raw.db` | Encrypted |
| Copy left from before a migration | `data.db.bak-…` | Encrypted |
| Documents | `docs/<id>.md` | **Not encrypted** |
| The [run folder](/en/dev/permissions.md#run-folder) of scheduled and event-triggered SOPs | `runs/` | **Not encrypted** |
- **Encryption uses a key per gem**: a randomly generated 256-bit key, stored in the app's credential store, which the gem's code can't reach. With only these files and not the key, the encrypted parts can't be read.
- **Other gems can't read it.** A gem can only access its own data through `ctx.data` and `ctx.storage`; there is no interface for reading another gem's data, and another gem's node id is rejected (`E_PERM_DENIED`). Code in the sandbox also can't directly read or write files in the data folder.
- **Each gem's code runs in its own sandbox process**, not sharing a process with other gems.
- **Everything is deleted on uninstall**: the whole data folder (including the plaintext documents and the run folder) is deleted together with this key. Reinstalling gives a new, empty set of data.
- **Deletion is a soft delete.** After `delete`, records and documents disappear from the interface, but the content is still in the data folder (a document's `.md` file too) and is deleted together on uninstall; change records keep the content from before and after the change, and are cleaned up according to `eventRetentionDays`. So don't promise users that "deleting wipes it from disk".
- Gem Studio's test data is a separate copy (see [Testing in Gem Studio](#testing-in-gem-studio)), and is deleted together with its key when the draft is deleted.
### Isolation by Organization
A user can set up several [organizations](/en/guide/organizations.md) on this machine, and each organization is a completely independent set of data. Gems are installed per organization: the same gem installed in two organizations is two installations that don't connect.
- **What is independent per organization**: the gem's own data (`ctx.data`, `ctx.storage`), downloaded resource files ([`resources`](/en/dev/manifest.md#resources) in the manifest), credential connection grants, knowledge base grants, and the content written to the knowledge base.
- **Other organizations are invisible**: at run time a gem can only access the current organization's data, can't access other organizations, and can't tell whether other organizations exist. Interfaces such as `ctx.knowledge` neither accept nor return an organization id.
- **Switching organizations restarts the app**: tool calls in progress are interrupted. Unfinished SOP runs and scheduled tasks that came due continue the next time that organization is opened. Tools must not depend on the process staying alive; write any intermediate state that needs to be kept into the data.
- **Deleting an organization**: the gem data inside that organization is permanently deleted with it.
## Limits
### ctx.data Limits
| Item | Limit |
| --- | --- |
| One record (JSON) | 1 MiB |
| One Log entry | 1 MiB |
| One document | 2 MiB; title 500 characters |
| One Blob | 20 MiB |
| Result of one query (table or Log) | 1000 items, 16 MB |
| Data scanned by one call | About 64 MB for paginated queries, 1 million for counts |
| Conditions / sort fields in one query | 16 / 8; at most one `in`, with at most 30 values |
| One transaction | 500 write operations; at most 5 runs on conflict by default |
| Collections | 64 per gem; each has at most 128 fields and 16 indexes (1–8 fields each) |
| `enum` | 256 values |
| Collection names, field names | 64 characters |
| Record and document ids | 1–128 characters, only letters, digits, `_` and `-` |
| Nesting depth of field values (`object`, `any`, arrays without `items`) | 32 levels |
| Reference `kind` | Matches `^[a-z][a-z0-9_.:-]{0,63}$`, can't start with `field:` |
| Migrations | 60 seconds per batch, 10 minutes in total |
| `kv` | Keys 1–512 characters with no control characters; values are strings, at most 1 MiB |
| Whole data folder | No limit. Above 4 GiB a warning is written to the app log, and again each time it doubles (8 GiB, 16 GiB, ...); writes are not blocked |
### rawSqlite Limits
- `raw.db` is at most 256 MB.
- A query returns at most 100,000 rows and 16 MB; add `LIMIT` if there would be more.
- Each call takes at most 10 seconds; when the database is opened, `db/schema.sql` and each migration get at most 30 seconds each. A statement that times out is stopped and the call raises an error; if `db/schema.sql` times out, the gem can't start this time. So don't write queries that never finish (a recursive `WITH` with no termination condition, triggers that fire each other).
## Error Codes
Error messages from `ctx.data` and `ctx.storage.kv` start with the error code, and tool code can branch on it:
```js
try {
await notes.update(id, patch, { ifVersion: note.version })
} catch (err) {
if (!err.message.startsWith('E_DATA_CONFLICT')) throw err
// Someone else changed it first: read again and then decide
}
```
| Error code | Meaning |
| --- | --- |
| `E_INPUT_INVALID` | Invalid parameters or record: wrong type, missing required field, undeclared field or collection, malformed query |
| `E_DATA_CONFLICT` | `ifVersion` mismatch; deleting a nonexistent record with `ifVersion`; the inserted record id or created document id already exists; a record read by a transaction was changed |
| `E_DATA_NOT_FOUND` | The record, document or block to modify doesn't exist |
| `E_DATA_UNIQUE` | `unique` violated |
| `E_DATA_NEEDS_INDEX` | No usable index on a large collection, or a call scanned too much data |
| `E_PAYLOAD_TOO_LARGE` | A size limit was exceeded: record, Log entry, document, Blob, `kv` value, result of one query |
| `E_PERM_DENIED` | Another gem's node id was used |
| `E_DATA_SCHEMA` | Version went backward, a change needing a version bump was made within the same version, a collection's `kind` was changed, a migration function is missing or the migration failed |
| `E_DATA_CLOSED` | The gem has been disabled or uninstalled |
| `E_PERM_INVALID` | The `data` section is wrong; raised when packaging, installing and testing in Gem Studio. The message starts with `data:`; when packaging and installing, shape errors start with `notochord.json failed schema validation:` |
| `E_ACTIVATION_FAILED` | The gem failed to start after an upgrade or a same-version reinstall, with the cause (such as `E_DATA_SCHEMA`, `E_DATA_UNIQUE`) in the message; a failed upgrade is rolled back entirely to the old version. On the first install, the original error is raised instead |
Errors from `ctx.storage.sql` don't carry these error codes.
]]>