Stream of decoded documents with stored keys, a shared key layout, and a direction. Use reader.table(...).stream(...) for standard indexes; search indexes use search.
A first query
After setting up Confect, define afeed paginated query spec with the notes document as its item schema. Its handler receives paginationOpts automatically. This handler merges notes from two roles, removes hidden notes, and returns a page:
confect/notes.impl.ts
reader.table("notes").stream(...) is different from
reader.table("notes").index(...).stream() on the
Reading page. The latter is a plain
Stream over one query; the former is a composable query stream.Operations at a glance
The last column compares each operation with Effect’sStream operations.
The ordering contract
Every query stream visits its keys in its direction. Keys are stored separately from emitted values, so mapping a document doesn’t change its position. Compatible streams can be merged in key order, and pagination resumes after a saved key rather than an offset. TheKeyLabels and OrderDirection type parameters enforce this contract: TypeScript rejects known label or direction mismatches in merge. Runtime checks also require matching directions and key layouts, including implicit-ID positions.
flatMap appends the inner key to the outer key; renameKey replaces visible labels without changing key values. reverse changes the direction, not the key.
Example data and diagram notation
The examples use these two tables.n1 and c1 are readable stand-ins for real Convex document IDs:
notes has the indexes by_text on ["text"] and by_role on ["author.role"]; comments has by_note on ["noteId"] and by_body on ["body"]. Runtime keys include _creationTime and _id tiebreakers, so by_text orders by text, creation time, then ID.
Reading the diagrams
A stream is drawn as a track, read left to right in key order—the way a marble diagram reads left to right in time, with the index’s ordering as the axis: Diagram keys are abbreviated:[apple,1] means ["apple", 1, "n1"], with _id omitted for space. Joined keys omit both outer and inner IDs: [1,7] means [1, "n1", 7, "c1"]. Cursor labels such as cursor: [apple,3] stand for schematic strings, not arrays accepted by the API. Pass returned cursors back unchanged; see Cursor handling.
- The stream’s name is printed above its track, and
╰joins the two. ─ n1 ─is an element: the documentn1. Its key, the values the stream is ordered by, is printed beneath it.─ (n6) ─is an element that was read but filtered out. It emits nothing, but it still counts toward read budgets, and cursors still advance past it.- A blank stretch of track is key space the stream never read (skipped by a seek).
┤is the end of the stream,╎a cursor: a position between two keys.╞═ … ═╡is the operation between the input track (above) and its output (below). Tracks in one diagram share columns, so elements that line up vertically hold the same position in the output’s order.
Creating a stream
stream takes an index name, an optional range callback, and an optional order ("asc" by default). It returns a QueryStream whose elements are decoded documents, in index order.
Creating or composing a stream does not read documents. Reads begin when you run a consuming effect, such as Stream.runCollect or QueryStream.paginate; each run executes the queries again.
Every note, sorted by text
Every note, sorted by text, descending
Notes whose text is at least "banana", sorted by text
Notes whose text is exactly "apple", sorted by creation time
eq removes it:
Range callbacks
The range callback mirrors Convex’s index range builder with one addition: it tracks which fields are still varying after the range is applied.eqpins the next index field to a value and consumes it—it no longer varies within the stream.gt,gte,lt, andltebound the next field without consuming it, and must come last (a lower bound may be followed by an upper bound on the same field).
Keys, labels, and layouts
An key contains the values that determine an element’s position. Every element has its own key values; the stream has one shared key layout describing the positions in those keys: their sequence, their labels, and where implicit document-ID tiebreakers occur. For a note ordered by text:
Labels start as index field paths, but
renameKey can replace them with aliases. Labels can repeat, especially after a join. An implicit ID has no label; it still participates in ordering, bounds, and pagination. An explicit ID, such as the _id in by_id, has a label.
Pinning text with eq removes its position: the key becomes [1, "n1"], the labels become ["_creationTime"], and the layout contains creation time followed by the implicit ID. Range bounds such as gte keep the position.
A join concatenates the full outer and inner keys, retaining both IDs. For a note joined to its comments, both ordered by creation time:
distinct label prefix that reaches into the inner key also includes that intervening ID, so it groups within each outer document.
Layouts are compatible when they have the same total width, the same labels in the same positions, and the same implicit-ID positions. Direction is stored in stream.orderDirection and checked separately. A layout contains no key values, value-type schema, table identity, or equality-pinned values, so different indexes or tables can have compatible layouts. When merging them, choose positions whose values have the same meaning; matching layouts alone does not establish that.
Reuse a stream’s keyLayout when calling empty or supplying flatMap’s innerKeyLayout. Creating a stream to obtain its layout reads no documents. The type is normally inferred, but you can import the layout module directly when you need an annotation:
narrow bounds, pass key values, including any intervening implicit IDs. For distinct, pass a prefix of the labels; for renameKey, pass replacement labels for every labeled position.
Empty streams
QueryStream.empty is a query stream with no documents but a known key layout and direction, for the places where a stream is required and there is nothing to put in it. Its usual job is a merge over a list of streams that may turn out empty, since merge needs at least one input.
keyLayout. Creating that stream does not read documents. The generated document types name the document:
Consuming a stream
A query stream is a genuineStream, so everything in Effect’s Stream module applies (QueryStream.isQueryStream tells the two apart at runtime):
Stream.runCollect returns an effect that collects all results; Stream.runHead returns an effect producing the first result as an Option. Stream.take(n) is a transform, not a consumer: it returns a plain stream of at most n results, which you can then collect.
Stream combinator the result is an ordinary Stream: it can be consumed, but it no longer retains the key layout and stored keys, so it can’t be merged or paginated further. Use the QueryStream combinators below when you need to keep that ability.
QueryStream.unique consumes a stream expected to hold at most one document, returning an Option and failing with NotUniqueError if there are two or more:
Composing streams
Keep callbacks deterministic and read-only: pagination, seeks, and reversal can reevaluate them for the same document. Effects should be safe to run again, not perform writes or other one-time side effects.Merge
QueryStream.merge interleaves streams with matching key layouts into one ordered stream. Merging is how you query over several index ranges at once, such as the notes by two roles in the example at the top of this page.
It is an ordered merge, the step of merge sort that combines sorted runs, always emitting the smallest next key (the largest, descending). It is not Stream.merge, which interleaves inputs in arrival order.
author.role leaves both streams with visible labels ["_creationTime"] and a layout containing creation time followed by an implicit ID, so the merge interleaves them by creation time and then ID:
Filter and map
QueryStream.filter removes emitted values; QueryStream.map transforms them. Both preserve the original stored keys. A rejected document still counts as read, so cursors can advance past it.
note.text to its length still orders by the original text, creation time, and ID—not by length.
When the predicate or mapper needs to run an effect—read another table, use a service, or fail with a typed error—use QueryStream.filterEffect and QueryStream.mapEffect instead. They behave the same way, and the effect’s error and requirement types flow into the stream’s:
{ concurrency } to run several at once; elements are still emitted in stream order, so the result is the same query stream, just faster when each effect is a database read:
Distinct
QueryStream.distinct keeps the first document for each distinct value of a prefix of the key.
After a group’s first document, distinct seeks past the group with a fresh index read instead of scanning the rest of it.
n1 is found, the stream seeks straight to the first key past apple; n3 is never read. The same happens after n2 for banana:
flatMap result, a prefix reaching into the inner key includes the outer ID, so it groups within each outer document. Implicit IDs after the last selected label are excluded from the grouping prefix.
Operation order determines which document represents each group. Apply filter (or filterEffect) before distinct to choose the first matching document; apply it after distinct to test the chosen representative, omitting the group if that document fails. Reversing a distinct stream keeps its representatives and reverses their output order. Reversing the input before applying distinct instead chooses the first document from the other direction.
Narrow to a key range
QueryStream.narrow restricts a stream to the keys between start and end. Each endpoint has keyValues and a required inclusive flag, so you can choose any combination of inclusive and exclusive bounds. Provide at least one endpoint; omit the other to leave that side unbounded.
Endpoints follow stream order: on an ascending stream, start is the lower key; on a descending stream, it is the upper key. Narrowing intersects existing bounds, so repeated calls can only restrict the range further.
distinct, may still read outside the output bounds to find the correct representative.
For this direct index stream, the bounds become index-range predicates, so the elements outside them are not read at all:
_id:
startTime and excludes every note at endTime. Adjacent windows can share an endpoint without overlapping. Apply the same bounds to a merged stream when its leading key value is also a creation time.
When both endpoints use the same key, the range is empty unless both are inclusive. Two inclusive endpoints select that key, or the entire group for a prefix key.
On a distinct stream, full-key bounds filter the original representatives exactly; they do not select a replacement from within the narrowed range. Explicit prefix bounds still include or exclude whole groups. Bounds applied before distinct constrain its input and can change which document represents a group; bounds applied after it constrain only the output, preserving that original input range and its representatives.
To resume pagination, pass the previous page’s continueCursor directly to QueryStream.paginate as cursor.
Flat-map
QueryStream.flatMap runs an inner stream for each document of an outer stream and concatenates the results, ordered by the outer key and then the inner key.
An outer document whose inner stream is empty contributes no elements. Use onEmpty to emit a placeholder for it.
Supply innerKeyLayout from a stream whose layout matches every stream returned by the callback. The joined stream needs its layout at construction, before any outer document is available to pass to the callback—even if the outer stream is empty. The layout determines the joined key positions and the width of markers for empty inner streams.
Here, pinning noteId in by_note leaves creation time followed by an implicit ID. A by_creation_time stream has the same layout and supplies the template without needing a note ID. Creating that template reads no documents.
n5 has no comments, so it contributes only a filtered marker that keeps cursors moving:
innerKeyLayout and run in the outer stream’s direction. The template’s direction is separate from its layout. TypeScript rejects known label or direction mismatches; runtime mismatches die with InnerStreamLayoutMismatchError or InnerStreamOrderMismatchError when the join runs.
The joined key keeps both IDs: comment c1 has key [1, "n1", 7, "c1"].
For renamed inner streams, apply the same renaming to the template and the streams returned by the callback:
["_creationTime", "commentTime"]; both implicit IDs keep their positions. For nested joins, build a template with the same nested composition and reuse its resulting keyLayout. A single-index template cannot describe the extra key positions introduced by that composition.
Keep outer documents without inner documents
By default an outer document whose inner stream is empty contributes nothing. PassonEmpty to flatMap to keep it: the document is emitted as onEmpty(outer), and the element type widens to include that placeholder.
∅n5, that is onEmpty(n5)) takes the position the filtered marker had without onEmpty, so it sorts first within its outer document and pagination steps past it like any element:
innerKeyLayout check and the direction rules, is unchanged.
Rename ordering labels
QueryStream.renameKey replaces a stream’s visible labels positionally. It does not sort: the elements and their order are untouched.
Use it to merge streams from different indexes or tables that sort by the same kind of value under different field names. Since merged streams must also share a document type, map each side to a common shape first:
flatMap result relabels by its combined key, outer labels first.
Reverse
QueryStream.reverse runs a composed stream in the opposite direction. Its typical use is bidirectional pagination: the stream that loads a feed’s later pages, reversed, loads its earlier ones.
It uses index scans and seeks rather than collecting and reversing the results, and the result is still a query stream.
distinct, renameKey, and empty all reverse. reverse(distinct(q)) keeps the same first representative of each group and flips only their output order. distinct(reverse(q)) instead chooses each group’s representative from the other direction. Preserving representatives can require extra index reads to discover groups and seek their original first documents.
Paginating a stream
QueryStream.paginate returns an effect producing one page of a composed stream. Pass the handler’s paginationOpts as in the first example; the options follow Convex’s pagination protocol, with key-based page boundaries and read budgets as described below.
Options
Result
For a one-shot read inside a handler, consume the result’s
page. For sequential reads, preserve the returned cursor rather than reconstructing it from the last visible document:
page.
Page boundaries
Three pages of two over the filtered notes follow each prior cursor. In this direct filtered scan, earlier documents aren’t reread; the hidden note is read on the last page, counted, and not returned:endCursor, a page covers exactly the range between its two cursors, however many documents that range holds as data changes, which is how reactive clients keep adjacent pages gap-free:
Read budgets
Cursor bounds are pushed into underlying index queries where possible, but compositions may read more documents than they return, including documents read by an earlier page. Both budgets count filtered documents, distinct-group discovery, repeated seeks, merge prefetch, and outer and innerQueryStream reads in joins. They do not track arbitrary I/O inside callbacks, such as a separate database lookup in mapEffect.
A budget-limited page stops at a safe key position so resuming cannot skip an unfinished group or join. If the budget prevents safe progress or a strictly interior split of a pinned page, paginate fails instead of returning a non-advancing cursor or repeating the same split. Increase the budget or change the query to reduce the reads needed for progress. Byte accounting happens after each document is read, so the last document can take the total over maximumBytesRead.
Cursor handling
Save cursors returned byQueryStream.paginate and pass them back unchanged. Use cursor: null for the first page, then its continueCursor for the next. Do not construct cursors from documents or decode them into narrow bounds.
Malformed or incompatible cursors fail with a ConvexError whose data is { paginationError: "InvalidCursor" }; Confect’s pagination clients restart from the first page. Restart pagination when changing the query’s filters, ordering, or other semantics, even if an old cursor is still accepted.
On the client
React’suseStreamPaginatedQuery handles page pinning, splitting, and invalid-cursor resets. Foldkit’s PaginatedQuery machine also works with stream-paginated queries: it pins each page it navigates away from, so going back reloads the same range.
The QueryStream type
TypeScript normally infers these parameters fromstream and its composition:
For example, the notes
by_text query carries branded labels for ["text", "_creationTime"], emits decoded notes in "asc" order, and can fail with DocumentDecodeError. For a type alias, prefer typeof notes from an existing stream rather than manually supplying its parameters. The KeyLabels parameter requires the brand; a plain string tuple is not a labels type. It retains the database access supplied when it was created, so consuming it doesn’t require providing DatabaseReader again. mapEffect and other effectful operations can add errors and service requirements.
A QueryStream<Doc, KeyLabels, OrderDirection, E, R> is also a Stream<Doc, E, R>; only QueryStream carries the visible-label and direction types and the runtime key layout.