Skip to Content
Puppi
Reference

Configuration

Configure a project, its connections, and its assets.

A project has two files. puppi.config.ts is committed. It names the entry, the build output, and every connection’s type. bindings.local.json stays on your machine. It names the environment you develop against, and the host and secrets for today.

puppi try reads bindings.local.json. The engine never reads that file. An environment that is already running gets the same facts from puppi env bind and puppi env secret.

puppi.config.json is not read. The build expects defineConfig() exported from puppi.config.ts.

Write puppi.config.ts

import * as postgres from "@puppi/adapter-postgres";
import { defineConfig } from "@puppi/sdk";

export default defineConfig({
  entry: "src/pipeline.ts",
  out: "out",
  defaultStorage: "warehouse",
  connections: {
    warehouse: postgres,
  },
});
FieldDefaultMeaning
entrysrc/pipeline.tsThe pipeline module.
outdist if omittedWhere bundle.js and manifest.json are written. puppi init sets out to out. puppi-env.d.ts is always written next to the config.
defaultStoragenoneWhere an asset writes when it does not set storage. A string is one connection for every role. A map may set source, raw, model, and destination.
connections{}Connection name to a connector module, or to a project-local { type, Client, clientModule }.

defaultStorage roles other than those four are a build error. An asset’s storage field wins over the default. One string, such as "warehouse", is one connection for every role.

entry and out are read by evaluating the config module, so they may be computed. Code at the top of that file runs during the build.

Declare a connection

A connection is a connector module. http is @puppi/adapter-http. A storage connector and a catalog connector are the same shape. See How connectors work.

import * as filesystem from "@puppi/adapter-filesystem";
import * as http from "@puppi/adapter-http";
import * as postgres from "@puppi/adapter-postgres";

connections: {
  api: http,
  warehouse: postgres,
  lake: filesystem,
}

Overrides when a binding is narrower than the adapter allows:

FieldMeaning
config_schemaMap of config field to a type string. Wins over the adapter’s schema.
secret_fieldsSecret field names. Wins over the adapter’s list.
clientModuleModule specifier for Client in puppi-env.d.ts, such as "./src/clearbit.js". A connector module defaults to @puppi/adapter-<type>.

puppi build writes config_schema and secret_fields from the client the bundle registered. A connection whose type that bundle does not register fails the build, and the message names the import to add. The engine rejects that deployment.

A project-local client is { type, Client, clientModule }. clientModule is required. A client declared in the pipeline uses adapter() from @puppi/sdk with type, Client, and optional configSchema, secretFields, and clientModule. Register a type in one place. defineConfig and adapter() both register it, and a second client for the same type is refused.

Write bindings.local.json

puppi try reads this shape. database is your database name.

{
  "environment": "hello",
  "bindings": {
    "warehouse": {
      "config": {
        "host": { "env": "PGHOST", "default": "127.0.0.1" },
        "port": { "env": "PGPORT", "default": 5432 },
        "database": { "env": "PGDATABASE", "default": "app" },
        "user": { "env": "PGUSER", "default": "postgres" }
      },
      "secret_refs": { "password": "warehouse_password" }
    }
  },
  "secrets": {
    "warehouse_password": { "env": "PGPASSWORD", "default": "postgres" }
  }
}

A value is a literal or { "env": "NAME", "default": <value> }. The default’s type decides the variable’s type: a numeric default binds a number. puppi try resolves values in the process that runs the command. A variable that is set wins over the default.

puppi init writes a filesystem binding: connection files, root set to out/files, and no secrets.

Environment variables

VariableUsed by
PUPPI_ENVCommands that need an environment, when --env is omitted.
PUPPI_ENGINE_URLEngine base URL. Default https://app.puppi.dev.

Declare an asset

Declare assets in the entry module. The key is a string you choose, usually group/name.

Shared fields:

FieldMeaning
keyIdentity. Required.
schemaColumn contract. A { columns: [{ name, type }] } list, or a Standard Schema (Zod, Valibot, ArkType) whose output is an object. The manifest stores the column list either way.
storageConnection this asset writes to. Wins over defaultStorage.
scheduleWhen the engine runs this asset on its own. See below.
depsExtra asset keys the literal scan cannot see.
reads / writesConnection names the literal scan cannot see, with a direction.
partitioning{ by: "none" } (default), { by: "time", grain: "day" | "hour" }, or { by: "key", column }. An edge is rejected when both ends are partitioned and the specs differ. An unpartitioned end is compatible with either.
checksschema_contract and custom. See Quarantine.
groupA value returned by group({ name, visibility }). Omitted assets use the implicit default group.
visibilitygroup, project, or public. Can only widen the group’s visibility.

Column type strings the contract compares are string, integer, float, boolean, json, and null, with |null for nullable. A Standard Schema number becomes float. Declare integer columns as integers (z.int()), because a whole-number value is observed as integer.

Source

source({
  key: "demo/customers",
  schema: Customer,
  mode: "full_refresh",
  run: async (ctx) => {
    const api = ctx.connection("api");
    await ctx.emit([]);
  },
});
FieldMeaning
runRequired. (ctx) => Promise<void>.
reads / writesConnections the producer uses that the compile pass cannot see. Storage is storage or defaultStorage.
modefull_refresh (default) or append.
cursorRequired when mode is append. The watermark column.

full_refresh stores a complete copy. append stores a delta in one table and needs a watermark.

Model

A model is produced by run. Put SQL inside that function with ctx.query, the way the model below does.

The SDK also accepts sql: (ctx) => string in place of run. The executor refuses that form. The attempt error is <key> is a sql model; sql models are not supported.

model({
  key: "core/customer",
  schema: Customer,
  run: async (ctx) => {
    const upstream = ctx.input("demo/customers");
    const { rows } = await ctx.query(`select * from ${upstream}`);
    await ctx.emit(rows);
  },
});
FieldMeaning
run(ctx) => Promise<void>. This is the producer the executor calls.
modetable (default), view, or incremental.
cursorRequired when mode is incremental. The upstream asset key the delta is over.
storageOptional. Most models use defaultStorage.

table is a full rebuild. incremental stores a delta. view is its own shape: puppi rollback refuses it.

Destination

destination({
  key: "warehouse/customer",
  from: "core/customer",
  storage: "warehouse",
  write: "replace",
});
FieldMeaning
fromThe asset this destination delivers. Added to deps.
storageRequired. Where it delivers.
writeRequired. merge, append, or replace. Stored on the asset.

A destination has no run. All three write values are delivered the same way: the destination reads the upstream named by from and emits those rows.

Group

import { group, source } from "@puppi/sdk";

const core = group({ name: "core", visibility: "project" });

source({ key: "core/customer", group: core, run: async () => {} });

visibility on the group is group, project, or public. The default group is project until you declare group({ name: "default", visibility }).

Schedule an asset

schedule: { every: "15m" }
schedule: { cron: "0 9 * * mon", timezone: "America/New_York" }

every is <n>m (1–59), <n>h (1–23), or <n>d (1–31). It aligns to a clock boundary. It is not an interval from now, or from the last run. The engine expands it to cron: 15m is */15 * * * * (minute 0, 15, 30, 45), 2h is 0 */2 * * * (00:00, 02:00, 04:00, and so on), and 1d is 0 0 */1 * * (00:00 each day). 23h is 0 */23 * * *, which fires at 00:00 and 23:00 — the hours divisible by 23 — not 23 hours after the previous fire.

cron is five fields. Names such as mon and jan are accepted. Sunday is 0 or 7. A bad expression fails puppi build. timezone is checked by the engine at deploy.

Use ctx

ctx is supplied when the attempt runs.

MemberMeaning
input(key)Physical table name of an upstream asset, or a file ref (stream(), meta, sha256, bytes, contentType) when that asset stores a file. The key is a string literal. This is the read that becomes a dependency.
query(sql, params?)Runs SQL against the asset’s storage. The type argument is your assertion about the select list.
connection(name)The client for a bound connection. The name is a string literal.
emit(rows)Buffers rows for this materialization. Type-checked against schema. The schema contract runs after the write.
emitFile(stream, options?)Stores one object as this materialization. options is { contentType, meta }. One attempt cannot call both emit and emitFile. A file asset is full_refresh. append, incremental, view, a destination, and a table store are build errors.
checkpoint(state)In-progress state for a retry. Cleared when the producer finishes. The committed watermark lives on the materialization.
checkpointStateThe checkpoint from a previous attempt, or null.
cursorFromWhere an append source or incremental model should resume, or null.
latestTable name of this asset’s current latest materialization, or null on the first run. A file asset’s latest is { meta, sha256, bytes, contentType }.
unchanged()Ends the attempt without writing. The engine records skipped and stamps verified_at on the current latest. A call with no latest fails the attempt.
log(message, level?)A line on the attempt. Levels: debug, info, warn, error.
telemetryspan, event, and gauge for the attempt’s trace.
processPages({ fetch, transform, write })Fetch, transform, and write one page at a time, with a checkpoint after write resolves.

A custom check receives a smaller context: rows, output, query, and log. It has no input().