Configuration
Configure a project, its connections, and its assets.
A project has two files. puppi.config.ts is committed. It names the entry, the build output, and every connection’s type. bindings.local.json stays on your machine. It names the environment you develop against, and the host and secrets for today.
puppi try reads bindings.local.json. The engine never reads that file. An environment that is already running gets the same facts from puppi env bind and puppi env secret.
puppi.config.json is not read. The build expects defineConfig() exported from puppi.config.ts.
Write puppi.config.ts
import * as postgres from "@puppi/adapter-postgres";
import { defineConfig } from "@puppi/sdk";
export default defineConfig({
entry: "src/pipeline.ts",
out: "out",
defaultStorage: "warehouse",
connections: {
warehouse: postgres,
},
}); | Field | Default | Meaning |
|---|---|---|
entry | src/pipeline.ts | The pipeline module. |
out | dist if omitted | Where bundle.js and manifest.json are written. puppi init sets out to out. puppi-env.d.ts is always written next to the config. |
defaultStorage | none | Where an asset writes when it does not set storage. A string is one connection for every role. A map may set source, raw, model, and destination. |
connections | {} | Connection name to a connector module, or to a project-local { type, Client, clientModule }. |
defaultStorage roles other than those four are a build error. An asset’s storage field wins over the default. One string, such as "warehouse", is one connection for every role.
entry and out are read by evaluating the config module, so they may be computed. Code at the top of that file runs during the build.
Declare a connection
A connection is a connector module. http is @puppi/adapter-http. A storage connector and a catalog connector are the same shape. See How connectors work.
import * as filesystem from "@puppi/adapter-filesystem";
import * as http from "@puppi/adapter-http";
import * as postgres from "@puppi/adapter-postgres";
connections: {
api: http,
warehouse: postgres,
lake: filesystem,
} Overrides when a binding is narrower than the adapter allows:
| Field | Meaning |
|---|---|
config_schema | Map of config field to a type string. Wins over the adapter’s schema. |
secret_fields | Secret field names. Wins over the adapter’s list. |
clientModule | Module specifier for Client in puppi-env.d.ts, such as "./src/clearbit.js". A connector module defaults to @puppi/adapter-<type>. |
puppi build writes config_schema and secret_fields from the client the bundle registered. A connection whose type that bundle does not register fails the build, and the message names the import to add. The engine rejects that deployment.
A project-local client is { type, Client, clientModule }. clientModule is required. A client declared in the pipeline uses adapter() from @puppi/sdk with type, Client, and optional configSchema, secretFields, and clientModule. Register a type in one place. defineConfig and adapter() both register it, and a second client for the same type is refused.
Write bindings.local.json
puppi try reads this shape. database is your database name.
{
"environment": "hello",
"bindings": {
"warehouse": {
"config": {
"host": { "env": "PGHOST", "default": "127.0.0.1" },
"port": { "env": "PGPORT", "default": 5432 },
"database": { "env": "PGDATABASE", "default": "app" },
"user": { "env": "PGUSER", "default": "postgres" }
},
"secret_refs": { "password": "warehouse_password" }
}
},
"secrets": {
"warehouse_password": { "env": "PGPASSWORD", "default": "postgres" }
}
} A value is a literal or { "env": "NAME", "default": <value> }. The default’s type decides the variable’s type: a numeric default binds a number. puppi try resolves values in the process that runs the command. A variable that is set wins over the default.
puppi init writes a filesystem binding: connection files, root set to out/files, and no secrets.
Environment variables
| Variable | Used by |
|---|---|
PUPPI_ENV | Commands that need an environment, when --env is omitted. |
PUPPI_ENGINE_URL | Engine base URL. Default https://app.puppi.dev. |
Declare an asset
Declare assets in the entry module. The key is a string you choose, usually group/name.
Shared fields:
| Field | Meaning |
|---|---|
key | Identity. Required. |
schema | Column contract. A { columns: [{ name, type }] } list, or a Standard Schema (Zod, Valibot, ArkType) whose output is an object. The manifest stores the column list either way. |
storage | Connection this asset writes to. Wins over defaultStorage. |
schedule | When the engine runs this asset on its own. See below. |
deps | Extra asset keys the literal scan cannot see. |
reads / writes | Connection names the literal scan cannot see, with a direction. |
partitioning | { by: "none" } (default), { by: "time", grain: "day" | "hour" }, or { by: "key", column }. An edge is rejected when both ends are partitioned and the specs differ. An unpartitioned end is compatible with either. |
checks | schema_contract and custom. See Quarantine. |
group | A value returned by group({ name, visibility }). Omitted assets use the implicit default group. |
visibility | group, project, or public. Can only widen the group’s visibility. |
Column type strings the contract compares are string, integer, float, boolean, json, and null, with |null for nullable. A Standard Schema number becomes float. Declare integer columns as integers (z.int()), because a whole-number value is observed as integer.
Source
source({
key: "demo/customers",
schema: Customer,
mode: "full_refresh",
run: async (ctx) => {
const api = ctx.connection("api");
await ctx.emit([]);
},
}); | Field | Meaning |
|---|---|
run | Required. (ctx) => Promise<void>. |
reads / writes | Connections the producer uses that the compile pass cannot see. Storage is storage or defaultStorage. |
mode | full_refresh (default) or append. |
cursor | Required when mode is append. The watermark column. |
full_refresh stores a complete copy. append stores a delta in one table and needs a watermark.
Model
A model is produced by run. Put SQL inside that function with ctx.query, the way the model below does.
The SDK also accepts sql: (ctx) => string in place of run. The executor refuses that form. The attempt error is <key> is a sql model; sql models are not supported.
model({
key: "core/customer",
schema: Customer,
run: async (ctx) => {
const upstream = ctx.input("demo/customers");
const { rows } = await ctx.query(`select * from ${upstream}`);
await ctx.emit(rows);
},
}); | Field | Meaning |
|---|---|
run | (ctx) => Promise<void>. This is the producer the executor calls. |
mode | table (default), view, or incremental. |
cursor | Required when mode is incremental. The upstream asset key the delta is over. |
storage | Optional. Most models use defaultStorage. |
table is a full rebuild. incremental stores a delta. view is its own shape: puppi rollback refuses it.
Destination
destination({
key: "warehouse/customer",
from: "core/customer",
storage: "warehouse",
write: "replace",
}); | Field | Meaning |
|---|---|
from | The asset this destination delivers. Added to deps. |
storage | Required. Where it delivers. |
write | Required. merge, append, or replace. Stored on the asset. |
A destination has no run. All three write values are delivered the same way: the destination reads the upstream named by from and emits those rows.
Group
import { group, source } from "@puppi/sdk";
const core = group({ name: "core", visibility: "project" });
source({ key: "core/customer", group: core, run: async () => {} }); visibility on the group is group, project, or public. The default group is project until you declare group({ name: "default", visibility }).
Schedule an asset
schedule: { every: "15m" }
schedule: { cron: "0 9 * * mon", timezone: "America/New_York" } every is <n>m (1–59), <n>h (1–23), or <n>d (1–31). It aligns to a clock boundary. It is not an interval from now, or from the last run. The engine expands it to cron: 15m is */15 * * * * (minute 0, 15, 30, 45), 2h is 0 */2 * * * (00:00, 02:00, 04:00, and so on), and 1d is 0 0 */1 * * (00:00 each day). 23h is 0 */23 * * *, which fires at 00:00 and 23:00 — the hours divisible by 23 — not 23 hours after the previous fire.
cron is five fields. Names such as mon and jan are accepted. Sunday is 0 or 7. A bad expression fails puppi build. timezone is checked by the engine at deploy.
Use ctx
ctx is supplied when the attempt runs.
| Member | Meaning |
|---|---|
input(key) | Physical table name of an upstream asset, or a file ref (stream(), meta, sha256, bytes, contentType) when that asset stores a file. The key is a string literal. This is the read that becomes a dependency. |
query(sql, params?) | Runs SQL against the asset’s storage. The type argument is your assertion about the select list. |
connection(name) | The client for a bound connection. The name is a string literal. |
emit(rows) | Buffers rows for this materialization. Type-checked against schema. The schema contract runs after the write. |
emitFile(stream, options?) | Stores one object as this materialization. options is { contentType, meta }. One attempt cannot call both emit and emitFile. A file asset is full_refresh. append, incremental, view, a destination, and a table store are build errors. |
checkpoint(state) | In-progress state for a retry. Cleared when the producer finishes. The committed watermark lives on the materialization. |
checkpointState | The checkpoint from a previous attempt, or null. |
cursorFrom | Where an append source or incremental model should resume, or null. |
latest | Table name of this asset’s current latest materialization, or null on the first run. A file asset’s latest is { meta, sha256, bytes, contentType }. |
unchanged() | Ends the attempt without writing. The engine records skipped and stamps verified_at on the current latest. A call with no latest fails the attempt. |
log(message, level?) | A line on the attempt. Levels: debug, info, warn, error. |
telemetry | span, event, and gauge for the attempt’s trace. |
processPages({ fetch, transform, write }) | Fetch, transform, and write one page at a time, with a checkpoint after write resolves. |
A custom check receives a smaller context: rows, output, query, and log. It has no input().