Workers
Run attempts on a machine you control. Outbound 443 only.
A remote environment queues each attempt until a worker in your pool takes it. Run a Puppi worker with puppi worker start, or with the image on this page. Puppi does not support a worker you write against the attempt claim routes.
Run only pipeline code you trust; the worker executes that code with access to its configured connections and secrets. The worker opens one outbound connection to the engine. It has no inbound port, and the engine never dials it.
puppi worker start is the development path. It runs that worker in the foreground. The container on this page is the worker you leave running.
What the host needs
| Need | Minimum |
|---|---|
| Machine | Linux, x86-64 or arm64, with Docker. |
| Network | DNS, and outbound TCP 443 to the engine. No inbound port, public IP, or VPN. |
| Disk | Nothing that has to persist. A read-only root and a /tmp tmpfs are enough. |
A private subnet reaches the engine through a NAT gateway. A firewall that inspects TLS sometimes drops the WebSocket upgrade or cuts a long-lived connection. Allow WebSockets to the engine’s hostname when that happens.
The worker also reaches the databases, object stores, and APIs your pipelines bind. Those stay on your network.
Sign what the worker will check
Create the key before the environment. puppi env create signs only a new environment. An environment that already exists keeps the signature it has. When a key is configured and that signature is not this key, the command exits 2 before it changes placement.
puppi keys create writes an Ed25519 private key to puppi-signing-key in the project directory (mode 0600) and prints the public key. Standard output is only that public key. In CI, set PUPPI_SIGNING_KEY to the same one-line private key (ed25519: and the base64 seed). The command then prints that key’s public id and does not mint a new one.
export PUPPI_ENGINE_URL="${PUPPI_ENGINE_URL:-https://app.puppi.dev}"
export PUPPI_ENV="${PUPPI_ENV:-prod}"
export PUPPI_TRUSTED_KEYS="$(puppi keys create)" --out <file> chooses the path. --force overwrites a file that is already there. When a key is configured, puppi env create, puppi env bind, and puppi deploy sign with it. The lookup order is PUPPI_SIGNING_KEY, then --signing-key, then PUPPI_SIGNING_KEY_FILE, then puppi-signing-key in --dir when that file exists. With no key, a write that leaves the row unsigned prints NOT SIGNED and still exits 0. A hosted executor prints unsigned. A hosted executor does not check signatures. A command that keeps an existing signature does not. A worker with trusted keys refuses an unsigned lease.
A deploy whose key no connected worker trusts prints a warning and still succeeds. The worker is what refuses the attempt.
Place the environment
puppi env create "$PUPPI_ENV" --executor remote puppi place remote sets remote on every environment you already have. An attempt that has started keeps the placement it started with.
Choose hosted for Puppi-managed execution or remote for your worker pool. The commands on this page use remote. A public API request that selects local is refused with 403 local_not_allowed. puppi try previews one producer on your machine and does not set placement.
Declare a secret the engine never stores
puppi env secret warehouse_password --on-worker The engine stores the name. The value stays in a file on the worker. On the machine:
install -d -m 700 /etc/puppi/secrets
printf '%s\n' "$WAREHOUSE_PASSWORD" > /etc/puppi/secrets/warehouse_password
chmod 600 /etc/puppi/secrets/warehouse_password The file name is the secret name. One trailing newline is stripped. The fallback is PUPPI_SECRET_ plus the name in upper case (PUPPI_SECRET_WAREHOUSE_PASSWORD), used only when the file is absent.
Bind and deploy while the signing key is configured. The signature covers the connection type from puppi.config.ts. The connection has to be declared there, and --type has to be that type. A re-bind with no key that changes the config exits 2. An unchanged re-bind keeps the stored signature.
puppi env bind warehouse \
--config '{"host":"warehouse.internal","port":5432}' \
--secret-ref password=warehouse_password \
--type postgres
puppi deploy Create a token
The token is printed once. Standard output is only the token.
export PUPPI_WORKER_TOKEN="$(puppi worker token create droplet-1)" puppi worker token list shows id, name, and last use, without the token. puppi worker token revoke <id> revokes it and closes the sockets that presented it.
Pull the image
The image is private. PUPPI_WORKER_IMAGE is the name you pull and run. The default is ghcr.io/tzelon/puppi-worker:0.4. A personal access token with read:packages logs Docker in. On Kubernetes the same credential is an imagePullSecret.
export PUPPI_WORKER_IMAGE="${PUPPI_WORKER_IMAGE:-ghcr.io/tzelon/puppi-worker:0.4}"
echo "$GHCR_PAT" | docker login ghcr.io -u "$GHCR_USER" --password-stdin
docker pull "$PUPPI_WORKER_IMAGE" umask 077
auth="$(printf '%s:%s' "$GHCR_USER" "$GHCR_PAT" | base64 | tr -d '\n')"
cfg="${XDG_CONFIG_HOME:-$HOME/.config}/puppi"
mkdir -p "$cfg"
cat > "$cfg/ghcr.json" <<EOF
{"auths":{"ghcr.io":{"username":"$GHCR_USER","password":"$GHCR_PAT","auth":"$auth"}}}
EOF
kubectl create secret generic ghcr \
--type=kubernetes.io/dockerconfigjson \
--from-file=.dockerconfigjson="$cfg/ghcr.json" Run the container
These commands do not set PUPPI_EGRESS_FILTER. They do not enable an egress filter. Restrict outbound destinations with your network firewall. Set PUPPI_EGRESS_FILTER=1 only when the host can load that filter. A hosted run cannot reach a private database. Give the database a public address, or use this self-hosted worker.
docker run -d --name puppi-worker --restart unless-stopped --stop-timeout 900 \
--user node \
-e PUPPI_ENGINE_URL="$PUPPI_ENGINE_URL" \
-e PUPPI_WORKER_TOKEN \
-e PUPPI_WORKER_ID=droplet-1 \
-e PUPPI_WORKER_SLOTS=4 \
-e PUPPI_RESIDENCY=strict \
-e PUPPI_TRUSTED_KEYS="$PUPPI_TRUSTED_KEYS" \
-e PUPPI_SECRETS_DIR=/run/puppi/secrets \
-v /etc/puppi/secrets:/run/puppi/secrets:ro \
--read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges \
"$PUPPI_WORKER_IMAGE" --stop-timeout 900 matters. docker stop waits 10 seconds by default, and a worker that is draining needs the 15-minute execution deadline. TimeoutStopSec=900 under systemd and terminationGracePeriodSeconds: 900 on Kubernetes are the same setting.
PUPPI_WORKER_ID is the id puppi worker list prints. It defaults to the hostname. The token name droplet-1 is not that id, so the command sets it. PUPPI_RESIDENCY=strict requires PUPPI_TRUSTED_KEYS. A strict worker also refuses a lease that still carries a secret value.
puppi worker list then shows droplet-1, its free slots, version, residency, trusted key, and any declared worker secret it does not have. --env or PUPPI_ENV limits the comparison to that environment. This page set PUPPI_ENV, so the command above is that environment only. Unset PUPPI_ENV and omit --env to compare every environment.
puppi worker list Keep the token out of EC2 user-data
User-data is readable from the instance metadata service. The boot script installs Docker and starts the container. The token comes from SSM Parameter Store, written by the instance role, and the container reads the file.
install -d -m 700 /run/puppi
aws ssm get-parameter --name /puppi/droplet-1/token --with-decryption \
--query Parameter.Value --output text > /run/puppi/worker-token
chmod 600 /run/puppi/worker-token
docker run -d --name puppi-worker --restart unless-stopped --stop-timeout 900 \
--user node \
-e PUPPI_ENGINE_URL="$PUPPI_ENGINE_URL" \
-e PUPPI_WORKER_TOKEN_FILE=/run/puppi/worker-token \
-e PUPPI_WORKER_ID=droplet-1 \
-e PUPPI_WORKER_SLOTS=4 \
-e PUPPI_RESIDENCY=strict \
-e PUPPI_TRUSTED_KEYS="$PUPPI_TRUSTED_KEYS" \
-e PUPPI_SECRETS_DIR=/run/puppi/secrets \
-v /run/puppi/worker-token:/run/puppi/worker-token:ro \
-v /etc/puppi/secrets:/run/puppi/secrets:ro \
--read-only --tmpfs /tmp --cap-drop ALL --security-opt no-new-privileges \
"$PUPPI_WORKER_IMAGE" On Kubernetes, mount the token Secret at PUPPI_WORKER_TOKEN_FILE, set image to $PUPPI_WORKER_IMAGE, name the pull secret ghcr, and set terminationGracePeriodSeconds: 900.
Allow outbound, and nothing else
The host’s firewall allows TCP 443 to the engine and to your data hosts, plus DNS, and no inbound traffic. Security groups and cloud firewalls match addresses, not hostnames. Resolve the engine, and resolve it again when the worker can no longer connect.
EC2 security group
A new group allows all egress. Revoke that, including ::/0 when the VPC added it. Then allow 443 to each address and DNS to the resolver. Repeat the 443 rule for each data host. A private subnet uses a NAT gateway for this traffic. It does not get an inbound rule or a load balancer.
ENGINE_HOST="${PUPPI_ENGINE_URL#*://}"
ENGINE_HOST="${ENGINE_HOST%%/*}"
aws ec2 revoke-security-group-egress --group-id "$SG" \
--ip-permissions '[{"IpProtocol":"-1","IpRanges":[{"CidrIp":"0.0.0.0/0"}]}]'
err="$(aws ec2 revoke-security-group-egress --group-id "$SG" \
--ip-permissions '[{"IpProtocol":"-1","Ipv6Ranges":[{"CidrIpv6":"::/0"}]}]' 2>&1)" || {
case "$err" in
*InvalidPermission.NotFound*) ;;
*) printf '%s\n' "$err" >&2; exit 1 ;;
esac
}
for ip in $(dig +short "$ENGINE_HOST" A); do
aws ec2 authorize-security-group-egress --group-id "$SG" \
--ip-permissions "[{\"IpProtocol\":\"tcp\",\"FromPort\":443,\"ToPort\":443,\"IpRanges\":[{\"CidrIp\":\"${ip}/32\"}]}]"
done
for ip in $(dig +short "$ENGINE_HOST" AAAA); do
aws ec2 authorize-security-group-egress --group-id "$SG" \
--ip-permissions "[{\"IpProtocol\":\"tcp\",\"FromPort\":443,\"ToPort\":443,\"Ipv6Ranges\":[{\"CidrIpv6\":\"${ip}/128\"}]}]"
done
aws ec2 authorize-security-group-egress --group-id "$SG" \
--ip-permissions '[{"IpProtocol":"udp","FromPort":53,"ToPort":53,"IpRanges":[{"CidrIp":"0.0.0.0/0"}]}]' DigitalOcean cloud firewall
Outbound rules replace the default allow. Leave inbound empty, and add one 443 rule per data host.
doctl compute firewall create --name puppi-worker \
--outbound-rules "protocol:tcp,ports:443,address:${ENGINE_IP}/32 protocol:udp,ports:53,address:0.0.0.0/0" \
--droplet-ids "$DROPLET_ID" Development
puppi worker start --slots 4 runs the pool worker in the foreground. The command needs PUPPI_WORKER_TOKEN or PUPPI_WORKER_TOKEN_FILE, and it passes the engine URL it resolved as PUPPI_ENGINE_URL.