Modules
Embedded Trellis for Elixir: define streaming transforms over Postgres
tables from the BEAM, through the trellis Rust crate itself
(docs/decisions/0010-embeddable-clients.md).
What Trellis.apply/2 did, one shape per statement form
A build that keeps failing: how many attempts have failed, the latest
error, and when it is tried again. It retries on its own, except that a
build failing for a reason no row explains pauses the transform after a
few attempts (next_attempt_at is then when it paused). Fix the cause,
resume a paused transform, and the next attempt goes through.
Why capture of source_table is broken, or why the drain (or a build's
merge) halted on the transform: kind is :capture, :halt or
:disruption, error
is a sentence naming the cause, columns the columns it is about (empty
when it isn't about a column, and for a halt), and detected_at when it
was first found.
What a transform's capture waits on: the staging worker's install, widen or
uninstall (operation) of the capture triggers on table needs
lock_mode, and another session holds or is queued for a conflicting lock.
blockers has one line per such session. Nothing cancels them; the wait
ends when they let go.
The configuration a handle connected with, as Trellis.config/1 returns
it: connect/1's :schema and :target_schema, and the connection pool's
size cap and how long a call waits for a free connection.
A partition of a partitioned source a transform is being corrected for, as
Trellis.status/2 reports it: the schema-qualified node, when the staging
worker queued its correction (since), and the keys still to re-derive
(keys_remaining, nil until the correction has counted them; an estimate
while it reads the node's own rows).
A registered transform definition, as Trellis.definitions/1 lists it.
One thing Trellis.self_check/3 found out of step, by kind
A drain page that keeps failing with nothing charged or paused, as
Trellis.status/2 and Trellis.self_check/3 report it: the segment whose
page fails (seg_seq), the qualified source tables it holds changes to
(tables), the latest failure's error and its sqlstate (nil when it
didn't come from Postgres), when a drain first and last failed on it
(since, last_seen), and how many drain passes have (attempts).
An error from Trellis: a stable code and the engine's message, and
nothing else (ADR-0010 decision 4).
The keys a transform holds in quarantine, as Trellis.status/2 and
Trellis.self_check/3 report them: how many (count), and when the one
held longest was last poisoned (oldest_poisoned_at).
Forwards the engine's log lines to Logger.
The engine's metrics, in Prometheus text exposition format: the Rust
crate's trellis::Metrics::render_prometheus
(docs/decisions/0009-observability-decisions.md).
Trellis statements in an Ecto migration, so a transform is defined, and dropped, the way the tables it reads are created.
One source row the apply path gave up on, as Trellis.poisoned_since/2
lists it: the transform it's held for (every other transform reading the
row keeps applying it), its fully-qualified src_table, its key as
Trellis renders it, the error that poisoned it, and when.
One quarantined source row, as Trellis.sample_quarantined/3 pages them:
its fully-qualified src_table, its key as Trellis renders it, and the
error it failed with.
One target's quarantine state, as Trellis.quarantined/1 lists it and
Trellis.quarantine_status/2 reports it.
One page of Trellis.sample_quarantined/3, ordered by (src_table, key).
A background check of a target table, as Trellis.self_check/3 starts it
and Trellis.self_check_job/2 reads it back.
What a finished Trellis.self_check/3 job found auditing the whole of a
target table against a fresh recompute of its definition from the source.
A source index a GROUP BY or JOIN target needs and lacks, as
Trellis.status/2 reports it: the qualified table, the columns the
index should lead with, the part of the definition that needs it (role:
:group_by for the GROUP BY columns, :join_from or :join_joined for
a JOIN's ON columns on its from or joined side), whether the transform
opted out with WITHOUT SOURCE INDEX (opted_out), and a create index
statement to run (fix).
Trellis doesn't create indexes on your tables.
Where a transform is in its lifecycle, as Trellis.status/2 reports it.
Appends Trellis's definitions to the structure file mix ecto.dump writes,
so mix ecto.load builds a database that has its transforms.
Helpers for a test suite that loads fixtures or bulk data into tables Trellis captures. Call them from test code only.
Raised by Trellis.TestSupport.resume_capture_disruptions!/2 when a
RESUME is refused or a resumed definition pauses again. refused maps
each such target ("schema.table") to the reason. Those definitions stay
paused; every other disrupted definition was resumed, and waited for,
before this is raised.
Mix Tasks
Runs mix ecto.dump and appends the repo's Trellis definitions to the
structure file it wrote, so mix ecto.load builds a database that has
its transforms (see Trellis.StructureDump).