Triggers & Change Data Capture¶
Uni can react to committed writes in three ways. They differ in what they see and what they guarantee:
| Mechanism | What it receives | Guarantee | Surfaces |
|---|---|---|---|
| Triggers | One Arrow batch of mutation events per matching commit, with pre-images | Before* triggers can veto the commit; After* triggers run once the commit has landed |
Rust TriggerPlugin; Cypher uni.plugin.declareTrigger (usable from Python) |
| CDC outputs | One CdcBatch per commit, with the same event rows |
Contiguous: a stream halts on a gap instead of skipping one. Checkpointed | Rust CdcOutputProvider + CdcStream |
| Commit notifications | Commit metadata only (version, labels, counts), no rows | Best effort: a slow consumer loses the oldest notifications | Rust Session::watch(), Python Session.watch() |
Use a trigger to act on a change, or to refuse it. Use CDC to ship every
change to another system. Use watch() to learn that something changed. For
watch(), see Commit notifications.
Triggers and CDC outputs are plugin surfaces. Only native Rust plugins can
author them; the WASM, Extism, Rhai and PyO3 loaders cannot. The plugin
mechanics (manifest, capabilities, Uni::add_plugin) are covered in
Authoring (Rust). From Python, triggers are
available through the Cypher declareTrigger procedure. Python has no CDC
API.
Mutation events¶
Triggers and CDC both receive mutation events as an Arrow RecordBatch with
one fixed schema:
| Column | Type | Contents |
|---|---|---|
event_kind |
UInt8 |
1 node create, 2 node update, 3 node delete, 4 edge create, 5 edge update, 6 edge delete |
vid_or_eid |
Int64 |
Vertex id for node events, edge id for edge events |
label |
Utf8 |
The label for node events, the edge type for edge events |
property |
Utf8 |
Always the empty string |
old_value |
LargeBinary, nullable |
The pre-image: the entity's committed user properties, serialized as JSON. Null for a create |
new_value |
LargeBinary, nullable |
Always null |
properties_new |
LargeBinary, nullable |
Post-mutation values of the properties that some trigger predicate references (n.x), as a CypherValue-encoded map. Otherwise null |
properties_old |
LargeBinary, nullable |
Pre-image values of the properties that some trigger predicate references (old.x). Otherwise null |
Each written or deleted entity produces one row; there is no row per property. A vertex with several labels produces one row per label.
The event-kind masks are NODE_CREATE, NODE_UPDATE, NODE_DELETE,
EDGE_CREATE, EDGE_UPDATE and EDGE_DELETE. The TriggerEventMask type
also defines PROPERTY_CHANGE, LABEL_ADDED and LABEL_REMOVED, but
event extraction never produces those kinds, so a trigger subscribed only to
them never fires.
Create, update or delete¶
A write to an entity that already existed before the transaction is an update and carries a pre-image. A write to a new entity is a create and carries none. A delete carries the pre-image of what it deleted. This holds whether the entity's committed state is still in memory (L0) or has already been flushed to storage (L1). See Pre-images.
Pre-images¶
old_value is the entity's committed state before the transaction, read
from the newest place it lives:
- Vertices. The engine first checks the in-memory L0 chain (current and
pending-flush buffers, newest first). A tombstone in a newer buffer counts
as a delete, so stale properties in an older buffer cannot come back. Any
vertex the transaction touched that L0 does not hold is looked up in
storage, in batched
_vid IN (...)scans per label. Either way the pre-image is the user property map: declared properties,ext_id, and undeclared properties. It never contains an internal_-prefixed column. - Edges. Every edge the transaction writes or deletes is read in one
batched lookup over the committed L0 chain layered on storage. Newer L0
state wins over storage. The pre-image holds the edge's whole committed
property set (declared, undeclared and schemaless). A property that was
removed is absent from it. Edge-delete rows carry the edge type name, so a
delete trigger filtered on
-[:Type]-fires.
A pre-image read is never served from the transaction's own uncommitted buffer.
Known gaps:
- A vertex that has no label in the transaction's buffer skips the storage lookup. If its committed state was already flushed, it can be reported as a create with no pre-image.
- If the storage lookup for vertices fails, the affected vertices are reported as creates and the commit still succeeds.
- An edge whose type id cannot be resolved to a name is skipped for writes.
On delete it is reported with the placeholder label
type:<id>, which no edge-type filter matches.
Triggers¶
Phases and fire modes¶
A trigger subscribes to one TriggerPhase. All four phases run at commit
time; none runs per statement.
| Phase | Runs | Can veto |
|---|---|---|
BeforeMutation, then BeforeCommit |
Before the commit is written | Yes, in Synchronous mode |
AfterMutation, then AfterCommit |
After the commit is published | No |
FireMode sets how the trigger runs:
Synchronous. Runs on the commit path, beforecommit()returns.- In a
Before*phase, returningTriggerOutcome::Reject { reason }or anErraborts the transaction.commit()then fails withUniError::TriggerRejected { trigger, reason }and none of the transaction's writes land. - In an
After*phase, an error or panic is logged and the commit still succeeds. Async. Runs after the commit on a blocking worker, once per matching commit, and cannot reject. Errors and panics are logged.shutdownanddropwait for in-flight async fires.EventualConsistency. Matching batches are coalesced per trigger and fired as one combined batch. A bucket is flushed when it is older thanUniConfig::ec_flush_interval(default 1 s) or reachesUniConfig::ec_flush_thresholdrows (default 10,000). A bucket that grows past four times the threshold is drained immediately. The combined fire reports the session and transaction id of the first commit in the window.
Async and EventualConsistency triggers subscribed to a Before* phase
are skipped.
A trigger can also return TriggerOutcome::Defer { until }, built with
TriggerDeferral::from_payload (re-fire on the next tick) or
TriggerDeferral::after (re-fire after a delay). The re-fire calls
on_deferred, which defaults to fire. Deferring again is capped at ten
attempts, after which the item is dropped with a warning.
- A deferral returned in a
Before*phase is queued only if everyBefore*trigger passes, so an aborted transaction leaves none behind. - With a local data path, the deferral queue is mirrored to
<data_path>/_system/deferred_triggers.jsonand reloaded on open. Otherwise it is in memory only. - The queue file is not written in the same transaction as the commit. A crash between a commit and the next queue write can lose a deferred item.
Filters¶
A TriggerSubscription selects events with:
events: aTriggerEventMask; combine masks with.union().labels: node events only. Exact match;Nonemeans every label. Write bare names ("Account"); the Cypher form":Account"names the same label.edge_types: edge events only. Exact match;Nonemeans every type. A leading:is accepted here too.predicate_source: a Cypher boolean expression evaluated per event row.n.<prop>reads the post-mutation value andold.<prop>the pre-image.- The event columns (
event_kind,vid_or_eid,label, ...) are also in scope. - The predicate is compiled at commit time. One that does not compile
makes the commit fail with
TriggerRejected. - If evaluation fails, a
Before*phase aborts the commit; anAfter*phase logs the error and skips the trigger.
The properties field of TriggerSubscription is not enforced: every
matching event is delivered regardless of which properties changed. To react
to a specific property, reference it in the predicate, for example
old.balance <> n.balance.
Rust: a TriggerPlugin¶
Implement TriggerPlugin (uni_plugin::traits::trigger) and register it
from a plugin whose manifest declares Capability::Trigger:
use std::sync::Arc;
use uni_plugin::FnError;
use uni_plugin::traits::trigger::{
FireMode, MutationBatch, TriggerContext, TriggerEventMask, TriggerOutcome,
TriggerPhase, TriggerPlugin, TriggerSubscription,
};
struct NoNegativeBalance {
sub: TriggerSubscription,
}
impl NoNegativeBalance {
fn new() -> Self {
Self {
sub: TriggerSubscription {
phase: TriggerPhase::BeforeCommit,
events: TriggerEventMask::NODE_CREATE.union(TriggerEventMask::NODE_UPDATE),
labels: Some(vec!["Account".into()]),
edge_types: None,
properties: None,
predicate_source: Some("n.balance < 0".into()),
fire_mode: FireMode::Synchronous,
// The first line of `docs` names the trigger in errors.
docs: "no-negative-balance".into(),
},
}
}
}
impl TriggerPlugin for NoNegativeBalance {
fn subscription(&self) -> &TriggerSubscription {
&self.sub
}
fn fire(&self, _ctx: TriggerContext<'_>, events: &MutationBatch) -> Result<TriggerOutcome, FnError> {
// `events.events` holds only the rows that matched the predicate.
Ok(TriggerOutcome::Reject {
reason: format!("{} account(s) would go negative", events.events.num_rows()),
})
}
}
// In your `Plugin::register`:
// r.trigger(Arc::new(NoNegativeBalance::new()))?;
// then install the plugin with `db.add_plugin(plugin)?`.
fire is synchronous and must not call block_on on the host runtime. A
panic inside it is caught at the dispatcher. TriggerContext has the
public fields session_id and tx_id, and a host() accessor.
Triggers are read from the database's instance plugin registry
(Uni::plugin_registry()), which forked sessions share (see
Forks).
Cypher: uni.plugin.declareTrigger¶
A declared trigger runs a Cypher action after a matching commit. It needs no compiled code, so it works from any binding:
CALL uni.plugin.declareTrigger(
'mycorp.audit', // qualified name
'CREATE ON :Person WHEN n.age > 18', // event filter
'CREATE (:AuditLog {person: $vid, kind: $event_kind})' // action
)
from uni_db import Uni
db = Uni.in_memory()
tx = db.session().tx()
tx.execute("""
CALL uni.plugin.declareTrigger(
'mycorp.audit', 'CREATE ON :Person', 'CREATE (:AuditLog {person: $vid})')
""")
tx.commit()
The arguments are (qname, event_filter, body), all strings, plus an
optional fourth deps_json (default '[]'). The procedure yields
registered. The event filter grammar is:
- Verbs are
CREATE(orINSERT),UPDATE(orSET) andDELETE(orREMOVE). Keywords are case-insensitive. ON :Labelrestricts the trigger to node events andON -[:Type]-to edge events. With noON, both match.WHENtakes the same predicate language aspredicate_sourceabove.- An empty filter, or one with no recognised verb, matches every node and edge create, update and delete, and logs a warning.
How a declared trigger behaves:
- It always runs as
AfterCommit+Async. Declaring[SYNC]is an error and stores nothing. A declared trigger therefore cannot veto a commit; use a RustTriggerPluginfor that. - The action runs once per matching event row, with the parameters
$vid(vid_or_eid),$labeland$event_kind(the integer kind above). - The action's writes go straight to the database's in-memory buffer and WAL, not through a new commit, so they do not fire triggers again. A depth guard (four levels) stops any further nesting.
- A failing action is logged; the commit that fired it has already succeeded.
- Declarations are persisted and reinstalled when the database is reopened.
List them with
CALL uni.plugin.listDeclared().CALL uni.plugin.dropDeclared(qname)removes the persisted declaration and yieldsremoved.
CDC outputs¶
A CDC output is a push-style sink. Implement CdcOutputProvider
(uni_plugin::traits::cdc) and register it from a plugin whose manifest
declares Capability::Cdc, via PluginRegistrar::cdc_output. Capability::Cdc
cannot be granted to a guest plugin.
pub trait CdcOutputProvider: Send + Sync {
fn name(&self) -> &str;
fn start(&self, ctx: CdcStartContext<'_>) -> Result<Box<dyn CdcStream>, FnError>;
fn resumes_from_checkpoint(&self) -> bool { true }
}
pub trait CdcStream: Send {
fn deliver(&mut self, batch: &CdcBatch) -> Result<(), FnError>;
fn checkpoint(&mut self) -> Result<CdcLsn, FnError>;
fn shutdown(&mut self) -> Result<(), FnError>;
}
Each commit becomes one CdcBatch:
mutations: the event rows described in Mutation events. With CDC alone, no trigger predicate references properties, soproperties_newandproperties_oldare null. The pre-image inold_valueis still populated.lsn_start: the database version the transaction started from.lsn_end: the database version after the commit.commit_timestamp.
A commit with no event rows is delivered as an empty batch with the same schema.
Batches arrive in commit order: lsn_end never decreases from one batch to
the next. A commit's notification is ordered at the moment it becomes visible,
under the writer's commit lock, so a commit that finishes its post-commit work
first still cannot be delivered ahead of an earlier one. (Before this was
fixed, a commit landing in that window could make an earlier commit report
the later commit's version and be delivered after it.)
A bulk load (bulk_writer, appender) is delivered as one or more batches,
all carrying the load's LSN range (lsn_start the load's version, lsn_end
the database version when it became visible, which can equal the previous
commit's lsn_end). Its rows are split across them in chunks of at most 4096,
and each stream is checkpointed once, after the last. See
Bulk loads.
Delivery¶
- After commit, off the commit path. A commit does not wait for sinks. One background driver delivers each commit to every stream in turn, then asks the stream for its checkpoint. Both calls run on the blocking pool.
- Halt on a gap, never skip. A stream is halted when:
- its
delivererrors or panics; - its
checkpointpanics; - a commit's mutation batch could not be built;
- the driver falls more than
UniConfig::commit_channel_capacitycommits behind (default 256). This one halts every stream.
A halted stream receives nothing more until the database is reopened. Each
halt logs an error and increments uni_cdc_stream_halted_total. A panic
in one stream does not affect the others.
- Checkpoints. The latest LSN each stream reports is persisted to
<data_path>/_system/cdc_checkpoints.json by a background writer, never
past a gap. The persisted position can lag the last delivered commit. On
reopen, start receives that position as CdcStartContext::from_lsn.
The host delivers only commits made after the stream starts. Any replay
of commits between the checkpoint and the restart must come from the
provider itself.
- A provider whose start ignores from_lsn should return false from
resumes_from_checkpoint(), and its checkpoint is not persisted.
- A database opened from a remote object-store URI has no local data path
and persists no CDC checkpoints.
- Late registration. A provider added after the database is open is
started at the next commit it can be given rows for. Earlier commits are
not backfilled. A commit made while no CDC output was registered captures
no rows. A provider that registers after such a commit, but before the
driver reaches it, starts after that commit and does not checkpoint past
it: its first checkpoint is the first commit it received.
No CDC provider is registered by default. For tests, the built-in in-memory provider is opt-in:
use uni_plugin_builtin::extras::{MemoryCdcOutputProvider, MemoryCdcPlugin};
let captured = MemoryCdcOutputProvider::new();
db.add_plugin(MemoryCdcPlugin::with_provider(captured.clone()))?;
// ... commit ...
let batches = captured.delivered();
It keeps every batch until shutdown, ignores from_lsn and does not resume
from a checkpoint. Do not leave it on in a long-running process. When it was
registered by default it cost 236.6 MB of RSS over 20,000 small commits,
against 55.8 MB without it.
Bulk loads¶
A bulk load (tx.bulk_writer(), tx.appender()) writes straight to storage,
not through a transaction, and commits on its own. Once it is visible it emits:
- one commit notification, with
tx_idthe load id andsession_id"bulk-load"; - a
NODE_CREATErow per vertex and label and anEDGE_CREATErow per edge, read back from storage in chunks of at most 4096 rows. A load's events are never held in memory at once.old_valueandproperties_oldare null;properties_newfollows the same predicate-referenced rule as a commit. - After-phase triggers fire once per chunk, so a large load reaches a trigger as several batches.
- CDC gets the chunks as batches of one announcement, as described under CDC outputs.
A synchronous BeforeMutation / BeforeCommit trigger cannot veto a load: its
events exist only after the load is visible. A load that would write rows such
a trigger subscribes to (by event kind and label or edge type; a predicate is
not evaluated, so a predicated trigger counts as matching) is refused with
UniError::BulkLoadBlockedByTrigger before those rows are staged, or at
commit() for a trigger registered mid-load, and nothing of it becomes
visible. BulkWriterBuilder::skip_before_triggers(true) (and the same option on
the appender builder) loads anyway, without running those triggers.
Once a load is visible, its events are delivered by a background task of
their own, which commit() waits for. Dropping the commit() future does not
stop the delivery: every event still reaches triggers and CDC. (The dropped
commit skips the load's index and snapshot finalization; the next open
completes it and delivers the load's events again, so consumers see them
twice.) A shutdown during the delivery stops it; CDC then halts with its
checkpoint before the load, and the next open delivers the whole load again.
A load completed in the background after BulkCommitIncomplete emits its events
when it becomes visible. A load the database completes while opening, after a
crash that came before its events were delivered, emits them from the open:
announced before build() returns, so ahead of any commit made through the
handle, with its rows streamed in the background. Only consumers that exist
during the open see them: built-in triggers and triggers declared with
uni.plugin.declareTrigger (the open reactivates those), and CDC outputs
already registered -- which the public API offers no way to do before
build() returns. A trigger or CDC output registered after the open does not
get the load. Delivery is at least once: a crash during it delivers the load
again at the next open. A fork's recovered load fires triggers when the fork is
next opened. See Data ingestion.
A transaction commit that is abandoned after its commit point (its commit()
future dropped, or failing, before the notification goes out) is announced with
mutations_failed: true instead of not at all, so CDC halts rather than skip
it.
Crash recovery¶
A commit is durable once it is written to the write-ahead log, and announced (its commit notification sent, its after-phase triggers fired) just after. If the process dies in between, the next open announces it:
- before
build()returns, ahead of any commit made through the handle, withtx_id"wal-recovery-<lsn>"andsession_id"wal-recovery"; - with the same events the commit would have produced: each write is classified (create, update, delete) and its pre-image read against the state the commit was made on;
- only to consumers that exist during the open: built-in triggers and triggers
declared with
uni.plugin.declareTrigger. Before-phase triggers do not run; the commit is already durable.
Delivery is at least once: a crash just after a commit was announced can
announce it again at the next open. A database closed normally, by
shutdown() or by dropping it, records which commits were announced, so a
clean reopen announces nothing twice.
Forks¶
Forked sessions share the primary's plugin registry:
- Triggers fire on fork commits. A trigger registered on the database
fires for commits on every fork as well as on the primary; there is no
per-fork scoping. The pre-image probe and a declared trigger's action use
the fork's own buffers, so a declared action triggered by a fork commit
writes to that fork.
shutdownanddropof the database wait for async fires started on a fork. - CDC sees primary commits only. The CDC driver subscribes to the primary's commit channel, so fork commits are never delivered to a CDC output. They still pay for building the event batch (see Costs).
watch()is per fork. On a forked session it reports only that fork's commits.- Scratch transactions refuse
commit(), so no trigger fires for them.
Costs¶
With no triggers and no CDC outputs registered, a commit does no event extraction at all. Registering any trigger or any CDC output turns the following on for every commit, on the primary and on every fork:
| Work | Cost |
|---|---|
| Event extraction | One event row per written or deleted entity, built under a short read of the transaction buffer. |
| Pre-image probe | An L0 scan, plus a storage (L1) lookup for every vertex and every edge the commit writes, including entities that turn out to be new. Lookups are chunked IN scans, not one per entity. |
| CDC batch | Built only when a CDC output is registered. |
| CDC checkpoint | Each persisted checkpoint is an atomic file write with two fsyncs, on the blocking pool. Writes coalesce while one is in flight. Skipped for providers that do not resume from a checkpoint. |
| Synchronous triggers | Run on the commit path and add their own time to commit(). |
Measured: 1000 flushed edges updated in one commit with a trigger registered took 124–383 ms with the edge pre-image probe (3 runs) and 109–282 ms without it (6 runs). At this size the difference is inside run-to-run noise.
See also¶
- Plugin concepts and Authoring (Rust): the plugin surfaces and registration.
- Transactions & Consistency: commit semantics.
- Forks.
- Rust API: commit notifications.